Detecting_Fake_News.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detecting Fake News\n",
    "\n",
    "**What is Fake News?**\n",
    "\n",
    "A type of yellow journalism, fake news encapsulates pieces of news that may be hoaxes and is generally spread through social media and other online media. This is often done to further or impose certain ideas and is often achieved with political agendas. Such news items may contain false and/or exaggerated claims, and may end up being viralized by algorithms, and users may end up in a filter bubble."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is a TfidfVectorizer?**\n",
    "\n",
    "**TF (Term Frequency):** The number of times a word appears in a document is its Term Frequency. A higher value means a term appears more often than others, and so, the document is a good match when the term is part of the search terms.\n",
    "\n",
    "**IDF (Inverse Document Frequency):** Words that occur many times a document, but also occur many times in many others, may be irrelevant. IDF is a measure of how significant a term is in the entire corpus.\n",
    "\n",
    "\n",
    " \n",
    "The TfidfVectorizer converts a collection of raw documents into a matrix of TF-IDF features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is a PassiveAggressiveClassifier?**\n",
    "\n",
    "Passive Aggressive algorithms are online learning algorithms. Such an algorithm remains passive for a correct classification outcome, and turns aggressive in the event of a miscalculation, updating and adjusting. Unlike most other algorithms, it does not converge. Its purpose is to make updates that correct the loss, causing very little change in the norm of the weight vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Project Outcome**\n",
    "\n",
    "The project of detecting fake news deals with fake and real news. Using sklearn, we build a TfidfVectorizer on our dataset. Then, we initialize a PassiveAggressive Classifier and fit the model. In the end, the accuracy score and the confusion matrix tell us how well our model fares."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import itertools\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import PassiveAggressiveClassifier\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "import matplotlib.pyplot as plt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8476</td>\n",
       "      <td>You Can Smell Hillary’s Fear</td>\n",
       "      <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10294</td>\n",
       "      <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
       "      <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3608</td>\n",
       "      <td>Kerry to go to Paris in gesture of sympathy</td>\n",
       "      <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10142</td>\n",
       "      <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
       "      <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>875</td>\n",
       "      <td>The Battle of New York: Why This Primary Matters</td>\n",
       "      <td>It's primary day in New York and front-runners...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                              title  \\\n",
       "0        8476                       You Can Smell Hillary’s Fear   \n",
       "1       10294  Watch The Exact Moment Paul Ryan Committed Pol...   \n",
       "2        3608        Kerry to go to Paris in gesture of sympathy   \n",
       "3       10142  Bernie supporters on Twitter erupt in anger ag...   \n",
       "4         875   The Battle of New York: Why This Primary Matters   \n",
       "\n",
       "                                                text label  \n",
       "0  Daniel Greenfield, a Shillman Journalism Fello...  FAKE  \n",
       "1  Google Pinterest Digg Linkedin Reddit Stumbleu...  FAKE  \n",
       "2  U.S. Secretary of State John F. Kerry said Mon...  REAL  \n",
       "3  — Kaydee King (@KaydeeKing) November 9, 2016 T...  FAKE  \n",
       "4  It's primary day in New York and front-runners...  REAL  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Read the data\n",
    "df=pd.read_csv('/Users/petramala/Desktop/news.csv')\n",
    "#Get shape and head\n",
    "df.shape\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    FAKE\n",
       "1    FAKE\n",
       "2    REAL\n",
       "3    FAKE\n",
       "4    REAL\n",
       "Name: label, dtype: object"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get column label\n",
    "labels=df.label\n",
    "labels.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the dataset\n",
    "\n",
    "x_train,x_test,y_train,y_test=train_test_split(df['text'], labels, test_size=0.2, random_state=7)\n",
    "# assigns 20% to test and 80 to train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s initialize a `TfidfVectorizer` with stop words from the English language and a maximum document frequency of 0.7 (terms with a higher document frequency will be discarded). Stop words are the most common words in a language that are to be filtered out before processing the natural language data. And a TfidfVectorizer turns a collection of raw documents into a matrix of TF-IDF features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize a TfidfVectorizer\n",
    "tfidf_vectorizer=TfidfVectorizer(stop_words='english', max_df=0.7)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Fit and transform train set, transform test set\n",
    "tfidf_train=tfidf_vectorizer.fit_transform(x_train) # this step generates word counts for the words in x_train\n",
    "\n",
    "tfidf_test=tfidf_vectorizer.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "حلب\n"
     ]
    }
   ],
   "source": [
    "print(tfidf_vectorizer.get_feature_names()[-10])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5068, 61651)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_train.shape\n",
    "# 5 068 rows (5 068 docs) and 61 651 columns (61 651 unique words, minus single character words):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1267, 61651)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 56381)\t0.03622223988286098\n",
      "  (0, 16314)\t0.053492157980948106\n",
      "  (0, 19620)\t0.030351855107005405\n",
      "  (0, 52607)\t0.04266045446208797\n",
      "  (0, 14900)\t0.039165339742818085\n",
      "  (0, 53749)\t0.029756205182552464\n",
      "  (0, 15211)\t0.07772572986248194\n",
      "  (0, 61154)\t0.06726619958695557\n",
      "  (0, 59042)\t0.047893261248723944\n",
      "  (0, 42972)\t0.03152542343098286\n",
      "  (0, 54232)\t0.038673616329284524\n",
      "  (0, 59249)\t0.04106143649018827\n",
      "  (0, 28891)\t0.06514397995138038\n",
      "  (0, 41708)\t0.03983513460128018\n",
      "  (0, 50192)\t0.045331181477256094\n",
      "  (0, 44691)\t0.0318676439567658\n",
      "  (0, 11820)\t0.046381950858248124\n",
      "  (0, 7682)\t0.04137048243377956\n",
      "  (0, 50343)\t0.10196965191544219\n",
      "  (0, 48095)\t0.021092647294770877\n",
      "  (0, 17916)\t0.03674587236023286\n",
      "  (0, 46027)\t0.10236534701241509\n",
      "  (0, 16993)\t0.02775494464904786\n",
      "  (0, 55006)\t0.03368300200002207\n",
      "  (0, 51389)\t0.03397042876291898\n",
      "  :\t:\n",
      "  (5067, 32909)\t0.09429823872256275\n",
      "  (5067, 59221)\t0.11305513144362901\n",
      "  (5067, 14649)\t0.03772971846597005\n",
      "  (5067, 55827)\t0.2218263076177088\n",
      "  (5067, 10398)\t0.029198031075976353\n",
      "  (5067, 46158)\t0.037013973826002855\n",
      "  (5067, 60684)\t0.022935168393493133\n",
      "  (5067, 53139)\t0.025628375412833703\n",
      "  (5067, 14556)\t0.04989667741743244\n",
      "  (5067, 59249)\t0.09370008504693801\n",
      "  (5067, 48095)\t0.09626467139586602\n",
      "  (5067, 54706)\t0.02438296419332449\n",
      "  (5067, 54235)\t0.18013712617861882\n",
      "  (5067, 23649)\t0.11715980750719378\n",
      "  (5067, 60291)\t0.02511088091736429\n",
      "  (5067, 30025)\t0.07803429885512414\n",
      "  (5067, 4919)\t0.039597295646358985\n",
      "  (5067, 24041)\t0.031458613144788115\n",
      "  (5067, 51148)\t0.03759621365205542\n",
      "  (5067, 47648)\t0.051281746021764794\n",
      "  (5067, 40793)\t0.12848334208123888\n",
      "  (5067, 57811)\t0.028627812267104712\n",
      "  (5067, 51968)\t0.04662099560930727\n",
      "  (5067, 57236)\t0.05398142449766798\n",
      "  (5067, 24260)\t0.029565269989545447\n"
     ]
    }
   ],
   "source": [
    "print(tfidf_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we’ll initialize a PassiveAggressiveClassifier. We’ll fit this on tfidf_train and y_train.\n",
    "\n",
    "Then, we’ll predict on the test set from the TfidfVectorizer and calculate the accuracy with accuracy_score() from sklearn.metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 92.9%\n"
     ]
    }
   ],
   "source": [
    "# Initialize a PassiveAggressiveClassifier\n",
    "\n",
    "pac=PassiveAggressiveClassifier(max_iter=50)\n",
    "pac.fit(tfidf_train,y_train)\n",
    "\n",
    "# Predict on the test set and calculate accuracy\n",
    "\n",
    "y_pred=pac.predict(tfidf_test)\n",
    "score=accuracy_score(y_test,y_pred)\n",
    "print(f'Accuracy: {round(score*100,2)}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let’s print out a confusion matrix to gain insight into the number of false and true negatives and positives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[590,  48],\n",
       "       [ 42, 587]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test,y_pred, labels=['FAKE','REAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We have 589 true positives, 587 true negatives, 42 false positives, and 49 false negatives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fake News Detection - DataCamp tutorial\n",
    "\n",
    "Bayesian models\n",
    "\n",
    "### Data Exploration\n",
    "\n",
    "Pandas DataFrame, check the shape, head and apply any necessary transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import `fake_or_real_news.csv` \n",
    "df = pd.read_csv(\"https://s3.amazonaws.com/assets.datacamp.com/blog_assets/fake_or_real_news.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6335, 4)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect shape of `df` \n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8476</td>\n",
       "      <td>You Can Smell Hillary’s Fear</td>\n",
       "      <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10294</td>\n",
       "      <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
       "      <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3608</td>\n",
       "      <td>Kerry to go to Paris in gesture of sympathy</td>\n",
       "      <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10142</td>\n",
       "      <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
       "      <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>875</td>\n",
       "      <td>The Battle of New York: Why This Primary Matters</td>\n",
       "      <td>It's primary day in New York and front-runners...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                              title  \\\n",
       "0        8476                       You Can Smell Hillary’s Fear   \n",
       "1       10294  Watch The Exact Moment Paul Ryan Committed Pol...   \n",
       "2        3608        Kerry to go to Paris in gesture of sympathy   \n",
       "3       10142  Bernie supporters on Twitter erupt in anger ag...   \n",
       "4         875   The Battle of New York: Why This Primary Matters   \n",
       "\n",
       "                                                text label  \n",
       "0  Daniel Greenfield, a Shillman Journalism Fello...  FAKE  \n",
       "1  Google Pinterest Digg Linkedin Reddit Stumbleu...  FAKE  \n",
       "2  U.S. Secretary of State John F. Kerry said Mon...  REAL  \n",
       "3  — Kaydee King (@KaydeeKing) November 9, 2016 T...  FAKE  \n",
       "4  It's primary day in New York and front-runners...  REAL  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print first lines of `df` \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8476</th>\n",
       "      <td>You Can Smell Hillary’s Fear</td>\n",
       "      <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10294</th>\n",
       "      <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
       "      <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3608</th>\n",
       "      <td>Kerry to go to Paris in gesture of sympathy</td>\n",
       "      <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10142</th>\n",
       "      <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
       "      <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>875</th>\n",
       "      <td>The Battle of New York: Why This Primary Matters</td>\n",
       "      <td>It's primary day in New York and front-runners...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                        title  \\\n",
       "Unnamed: 0                                                      \n",
       "8476                             You Can Smell Hillary’s Fear   \n",
       "10294       Watch The Exact Moment Paul Ryan Committed Pol...   \n",
       "3608              Kerry to go to Paris in gesture of sympathy   \n",
       "10142       Bernie supporters on Twitter erupt in anger ag...   \n",
       "875          The Battle of New York: Why This Primary Matters   \n",
       "\n",
       "                                                         text label  \n",
       "Unnamed: 0                                                           \n",
       "8476        Daniel Greenfield, a Shillman Journalism Fello...  FAKE  \n",
       "10294       Google Pinterest Digg Linkedin Reddit Stumbleu...  FAKE  \n",
       "3608        U.S. Secretary of State John F. Kerry said Mon...  REAL  \n",
       "10142       — Kaydee King (@KaydeeKing) November 9, 2016 T...  FAKE  \n",
       "875         It's primary day in New York and front-runners...  REAL  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set index \n",
    "df = df.set_index(\"Unnamed: 0\")\n",
    "\n",
    "# Print first lines of `df` \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the training data\n",
    "\n",
    "Now that the DataFrame looks closer to what you need, you want to separate the labels and set up training and test datasets.\n",
    "\n",
    "For this notebook, I decided to focus on using the longer article text. Because I knew I would be using bag-of-words and Term Frequency–Inverse Document Frequency (TF-IDF) to extract features, this seemed like a good choice. Using longer text will hopefully allow for distinct words and features for my real and fake news data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set `y` \n",
    "y = df.label "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8476</th>\n",
       "      <td>You Can Smell Hillary’s Fear</td>\n",
       "      <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10294</th>\n",
       "      <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
       "      <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3608</th>\n",
       "      <td>Kerry to go to Paris in gesture of sympathy</td>\n",
       "      <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10142</th>\n",
       "      <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
       "      <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>875</th>\n",
       "      <td>The Battle of New York: Why This Primary Matters</td>\n",
       "      <td>It's primary day in New York and front-runners...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6903</th>\n",
       "      <td>Tehran, USA</td>\n",
       "      <td>\\nI’m not an immigrant, but my grandparents ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7341</th>\n",
       "      <td>Girl Horrified At What She Watches Boyfriend D...</td>\n",
       "      <td>Share This Baylee Luciani (left), Screenshot o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>‘Britain’s Schindler’ Dies at 106</td>\n",
       "      <td>A Czech stockbroker who saved more than 650 Je...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4869</th>\n",
       "      <td>Fact check: Trump and Clinton at the 'commande...</td>\n",
       "      <td>Hillary Clinton and Donald Trump made some ina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2909</th>\n",
       "      <td>Iran reportedly makes new push for uranium con...</td>\n",
       "      <td>Iranian negotiators reportedly have made a las...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1357</th>\n",
       "      <td>With all three Clintons in Iowa, a glimpse at ...</td>\n",
       "      <td>CEDAR RAPIDS, Iowa — “I had one of the most wo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>988</th>\n",
       "      <td>Donald Trump’s Shockingly Weak Delegate Game S...</td>\n",
       "      <td>Donald Trump’s organizational problems have go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7041</th>\n",
       "      <td>Strong Solar Storm, Tech Risks Today | S0 News...</td>\n",
       "      <td>Click Here To Learn More About Alexandra's Per...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7623</th>\n",
       "      <td>10 Ways America Is Preparing for World War 3</td>\n",
       "      <td>October 31, 2016 at 4:52 am \\nPretty factual e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1571</th>\n",
       "      <td>Trump takes on Cruz, but lightly</td>\n",
       "      <td>Killing Obama administration rules, dismantlin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4739</th>\n",
       "      <td>How women lead differently</td>\n",
       "      <td>As more women move into high offices, they oft...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7737</th>\n",
       "      <td>Shocking! Michele Obama &amp; Hillary Caught Glamo...</td>\n",
       "      <td>Shocking! Michele Obama &amp; Hillary Caught Glamo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8716</th>\n",
       "      <td>Hillary Clinton in HUGE Trouble After America ...</td>\n",
       "      <td>0 \\nHillary Clinton has barely just lost the p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3304</th>\n",
       "      <td>What's in that Iran bill that Obama doesn't like?</td>\n",
       "      <td>Washington (CNN) For months, the White House a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3078</th>\n",
       "      <td>The 1 chart that explains everything you need ...</td>\n",
       "      <td>While paging through Pew's best data visualiza...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2517</th>\n",
       "      <td>The slippery slope to Trump’s proposed ban on ...</td>\n",
       "      <td>With little fanfare this fall, the New York de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10348</th>\n",
       "      <td>Episode #160 – SUNDAY WIRE: ‘Hail to the Deplo...</td>\n",
       "      <td>November 13, 2016 By 21wire Leave a Comment \\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>778</th>\n",
       "      <td>Hillary Clinton Makes A Bipartisan Appeal on S...</td>\n",
       "      <td>Hillary Clinton told a Staten Island crowd tod...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3300</th>\n",
       "      <td>New Senate majority leader’s main goal for GOP...</td>\n",
       "      <td>Mitch McConnell has an unusual admonition for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6155</th>\n",
       "      <td>‘Inferno’ and the Overpopulation Myth</td>\n",
       "      <td>Mises.org November 1, 2016 Inferno is a great ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>636</th>\n",
       "      <td>Anti-Trump forces seek last-ditch delegate revolt</td>\n",
       "      <td>Washington (CNN) The faction of the GOP that i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>Sanders Trounces Clinton in W. Va. -- But Will...</td>\n",
       "      <td>Meanwhile, Democrat Bernie Sanders picked up m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>626</th>\n",
       "      <td>Donald Trump Is Changing His Campaign Slogan t...</td>\n",
       "      <td>After a week of nonstop criticism from Democra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>691</th>\n",
       "      <td>Pure chaos: Donald Trump’s campaign management...</td>\n",
       "      <td>If you want a glimpse into a presidential cand...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5743</th>\n",
       "      <td>Syrian War Report – November 1, 2016: Syrian M...</td>\n",
       "      <td>Syrian War Report – October 31, 2016: Al-Nusra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6457</th>\n",
       "      <td>Colin Kaepernick hosts ‘Know Your Rights’ camp...</td>\n",
       "      <td>Print \\n[Ed. – Now teaching the gospel of raci...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7030</th>\n",
       "      <td>Wikileaks Emails Disclose Aliens Linked to Vat...</td>\n",
       "      <td>Sound too “strange” to be true? We have proof!...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9013</th>\n",
       "      <td>US abstains from UN vote calling for end to Cu...</td>\n",
       "      <td>US abstains from UN vote calling for end to Cu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9509</th>\n",
       "      <td>West Ham fans laud aerodynamic properties of n...</td>\n",
       "      <td>Tuesday 1 November 2016 by Formelia Alberthine...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3825</th>\n",
       "      <td>How the Obama White House runs foreign policy</td>\n",
       "      <td>When Susan E. Rice took over as President Obam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4515</th>\n",
       "      <td>ISIS claims responsibility for Garland, Texas,...</td>\n",
       "      <td>(CNN) ISIS has claimed responsibility for the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2747</th>\n",
       "      <td>The “blame the left” crew: What the right’s ne...</td>\n",
       "      <td>It was inevitable that liberals would end up b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6516</th>\n",
       "      <td>ADHD NATION: How Big Pharma Created the ADHD E...</td>\n",
       "      <td>By Kalee Brown\\nWhile I was at university, man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9636</th>\n",
       "      <td>Donald Trump claims the election will be 'rigg...</td>\n",
       "      <td>Email \\nDonald Trump is again riling up his vo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7398</th>\n",
       "      <td>REPORT: Dirty Reporter Blackmails Montel… Help...</td>\n",
       "      <td>BREAKING: Trump Jumps in FL, Takes 4 Point Lea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3717</th>\n",
       "      <td>Police Arrest Suspect In Charleston Church Sho...</td>\n",
       "      <td>Police in Charleston, S.C., say a man they sus...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5205</th>\n",
       "      <td>Donald Trump’s collapse was caused by one big ...</td>\n",
       "      <td>Silver of FiveThirtyEight.com has laid out fou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6696</th>\n",
       "      <td>FINA suspends Russian swimmer for 8 years over...</td>\n",
       "      <td>This post was originally published on this sit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7991</th>\n",
       "      <td>BREAKING : Hillary Campaign Manager Deletes hi...</td>\n",
       "      <td>BREAKING : Hillary Campaign Manager Deletes hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1303</th>\n",
       "      <td>Why Ted Cruz Has the Most to Lose in New Hamps...</td>\n",
       "      <td>Ted Cruz took first prize in the Iowa caucuses...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9051</th>\n",
       "      <td>“Nothing Good Can Come of This Election”–and T...</td>\n",
       "      <td>Posted on November 4, 2016 by Charles Hugh Smi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10200</th>\n",
       "      <td>List of Republicans opposing Trump | OffGuardian</td>\n",
       "      <td>Charlie Baker , Massachusetts (2015–present)[3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10009</th>\n",
       "      <td>Putin: Use of 'mythical' Russian military thre...</td>\n",
       "      <td>vladimir putin , Valdai , sochi , RBTH Daily R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4214</th>\n",
       "      <td>Bernie Sanders says private meeting with Pope ...</td>\n",
       "      <td>ROME —  U.S. Democratic presidential candidate...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2316</th>\n",
       "      <td>Alabama Lawmaker: Same-Sex Couples Don’t Deser...</td>\n",
       "      <td>Most conservatives who oppose marriage equalit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8411</th>\n",
       "      <td>Will the Media Reset After the Election or Are...</td>\n",
       "      <td>Written by Peter Van Buren   venerable New Yor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6143</th>\n",
       "      <td>DOJ COMPLAINT: Comey Under Fire Over Partisan ...</td>\n",
       "      <td>DOJ COMPLAINT: Comey Under Fire Over Partisan ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3262</th>\n",
       "      <td>GOP Senator David Perdue Jokes About Praying f...</td>\n",
       "      <td>The freshman senator from Georgia quoted scrip...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9337</th>\n",
       "      <td>Radio Derb Is On The Air–Leonardo And Brazil’s...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8737</th>\n",
       "      <td>Assange claims ‘crazed’ Clinton campaign tried...</td>\n",
       "      <td>Julian Assange has claimed the Hillary Clinton...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4490</th>\n",
       "      <td>State Department says it can't find emails fro...</td>\n",
       "      <td>The State Department told the Republican Natio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8062</th>\n",
       "      <td>The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...</td>\n",
       "      <td>The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8622</th>\n",
       "      <td>Anti-Trump Protesters Are Tools of the Oligarc...</td>\n",
       "      <td>Anti-Trump Protesters Are Tools of the Oligar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4021</th>\n",
       "      <td>In Ethiopia, Obama seeks progress on peace, se...</td>\n",
       "      <td>ADDIS ABABA, Ethiopia —President Obama convene...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4330</th>\n",
       "      <td>Jeb Bush Is Suddenly Attacking Trump. Here's W...</td>\n",
       "      <td>Jeb Bush Is Suddenly Attacking Trump. Here's W...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6335 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                        title  \\\n",
       "Unnamed: 0                                                      \n",
       "8476                             You Can Smell Hillary’s Fear   \n",
       "10294       Watch The Exact Moment Paul Ryan Committed Pol...   \n",
       "3608              Kerry to go to Paris in gesture of sympathy   \n",
       "10142       Bernie supporters on Twitter erupt in anger ag...   \n",
       "875          The Battle of New York: Why This Primary Matters   \n",
       "6903                                              Tehran, USA   \n",
       "7341        Girl Horrified At What She Watches Boyfriend D...   \n",
       "95                          ‘Britain’s Schindler’ Dies at 106   \n",
       "4869        Fact check: Trump and Clinton at the 'commande...   \n",
       "2909        Iran reportedly makes new push for uranium con...   \n",
       "1357        With all three Clintons in Iowa, a glimpse at ...   \n",
       "988         Donald Trump’s Shockingly Weak Delegate Game S...   \n",
       "7041        Strong Solar Storm, Tech Risks Today | S0 News...   \n",
       "7623             10 Ways America Is Preparing for World War 3   \n",
       "1571                         Trump takes on Cruz, but lightly   \n",
       "4739                               How women lead differently   \n",
       "7737        Shocking! Michele Obama & Hillary Caught Glamo...   \n",
       "8716        Hillary Clinton in HUGE Trouble After America ...   \n",
       "3304        What's in that Iran bill that Obama doesn't like?   \n",
       "3078        The 1 chart that explains everything you need ...   \n",
       "2517        The slippery slope to Trump’s proposed ban on ...   \n",
       "10348       Episode #160 – SUNDAY WIRE: ‘Hail to the Deplo...   \n",
       "778         Hillary Clinton Makes A Bipartisan Appeal on S...   \n",
       "3300        New Senate majority leader’s main goal for GOP...   \n",
       "6155                    ‘Inferno’ and the Overpopulation Myth   \n",
       "636         Anti-Trump forces seek last-ditch delegate revolt   \n",
       "755         Sanders Trounces Clinton in W. Va. -- But Will...   \n",
       "626         Donald Trump Is Changing His Campaign Slogan t...   \n",
       "691         Pure chaos: Donald Trump’s campaign management...   \n",
       "5743        Syrian War Report – November 1, 2016: Syrian M...   \n",
       "...                                                       ...   \n",
       "6457        Colin Kaepernick hosts ‘Know Your Rights’ camp...   \n",
       "7030        Wikileaks Emails Disclose Aliens Linked to Vat...   \n",
       "9013        US abstains from UN vote calling for end to Cu...   \n",
       "9509        West Ham fans laud aerodynamic properties of n...   \n",
       "3825            How the Obama White House runs foreign policy   \n",
       "4515        ISIS claims responsibility for Garland, Texas,...   \n",
       "2747        The “blame the left” crew: What the right’s ne...   \n",
       "6516        ADHD NATION: How Big Pharma Created the ADHD E...   \n",
       "9636        Donald Trump claims the election will be 'rigg...   \n",
       "7398        REPORT: Dirty Reporter Blackmails Montel… Help...   \n",
       "3717        Police Arrest Suspect In Charleston Church Sho...   \n",
       "5205        Donald Trump’s collapse was caused by one big ...   \n",
       "6696        FINA suspends Russian swimmer for 8 years over...   \n",
       "7991        BREAKING : Hillary Campaign Manager Deletes hi...   \n",
       "1303        Why Ted Cruz Has the Most to Lose in New Hamps...   \n",
       "9051        “Nothing Good Can Come of This Election”–and T...   \n",
       "10200        List of Republicans opposing Trump | OffGuardian   \n",
       "10009       Putin: Use of 'mythical' Russian military thre...   \n",
       "4214        Bernie Sanders says private meeting with Pope ...   \n",
       "2316        Alabama Lawmaker: Same-Sex Couples Don’t Deser...   \n",
       "8411        Will the Media Reset After the Election or Are...   \n",
       "6143        DOJ COMPLAINT: Comey Under Fire Over Partisan ...   \n",
       "3262        GOP Senator David Perdue Jokes About Praying f...   \n",
       "9337        Radio Derb Is On The Air–Leonardo And Brazil’s...   \n",
       "8737        Assange claims ‘crazed’ Clinton campaign tried...   \n",
       "4490        State Department says it can't find emails fro...   \n",
       "8062        The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...   \n",
       "8622        Anti-Trump Protesters Are Tools of the Oligarc...   \n",
       "4021        In Ethiopia, Obama seeks progress on peace, se...   \n",
       "4330        Jeb Bush Is Suddenly Attacking Trump. Here's W...   \n",
       "\n",
       "                                                         text  \n",
       "Unnamed: 0                                                     \n",
       "8476        Daniel Greenfield, a Shillman Journalism Fello...  \n",
       "10294       Google Pinterest Digg Linkedin Reddit Stumbleu...  \n",
       "3608        U.S. Secretary of State John F. Kerry said Mon...  \n",
       "10142       — Kaydee King (@KaydeeKing) November 9, 2016 T...  \n",
       "875         It's primary day in New York and front-runners...  \n",
       "6903          \\nI’m not an immigrant, but my grandparents ...  \n",
       "7341        Share This Baylee Luciani (left), Screenshot o...  \n",
       "95          A Czech stockbroker who saved more than 650 Je...  \n",
       "4869        Hillary Clinton and Donald Trump made some ina...  \n",
       "2909        Iranian negotiators reportedly have made a las...  \n",
       "1357        CEDAR RAPIDS, Iowa — “I had one of the most wo...  \n",
       "988         Donald Trump’s organizational problems have go...  \n",
       "7041        Click Here To Learn More About Alexandra's Per...  \n",
       "7623        October 31, 2016 at 4:52 am \\nPretty factual e...  \n",
       "1571        Killing Obama administration rules, dismantlin...  \n",
       "4739        As more women move into high offices, they oft...  \n",
       "7737        Shocking! Michele Obama & Hillary Caught Glamo...  \n",
       "8716        0 \\nHillary Clinton has barely just lost the p...  \n",
       "3304        Washington (CNN) For months, the White House a...  \n",
       "3078        While paging through Pew's best data visualiza...  \n",
       "2517        With little fanfare this fall, the New York de...  \n",
       "10348       November 13, 2016 By 21wire Leave a Comment \\n...  \n",
       "778         Hillary Clinton told a Staten Island crowd tod...  \n",
       "3300        Mitch McConnell has an unusual admonition for ...  \n",
       "6155        Mises.org November 1, 2016 Inferno is a great ...  \n",
       "636         Washington (CNN) The faction of the GOP that i...  \n",
       "755         Meanwhile, Democrat Bernie Sanders picked up m...  \n",
       "626         After a week of nonstop criticism from Democra...  \n",
       "691         If you want a glimpse into a presidential cand...  \n",
       "5743        Syrian War Report – October 31, 2016: Al-Nusra...  \n",
       "...                                                       ...  \n",
       "6457        Print \\n[Ed. – Now teaching the gospel of raci...  \n",
       "7030        Sound too “strange” to be true? We have proof!...  \n",
       "9013        US abstains from UN vote calling for end to Cu...  \n",
       "9509        Tuesday 1 November 2016 by Formelia Alberthine...  \n",
       "3825        When Susan E. Rice took over as President Obam...  \n",
       "4515        (CNN) ISIS has claimed responsibility for the ...  \n",
       "2747        It was inevitable that liberals would end up b...  \n",
       "6516        By Kalee Brown\\nWhile I was at university, man...  \n",
       "9636        Email \\nDonald Trump is again riling up his vo...  \n",
       "7398        BREAKING: Trump Jumps in FL, Takes 4 Point Lea...  \n",
       "3717        Police in Charleston, S.C., say a man they sus...  \n",
       "5205        Silver of FiveThirtyEight.com has laid out fou...  \n",
       "6696        This post was originally published on this sit...  \n",
       "7991        BREAKING : Hillary Campaign Manager Deletes hi...  \n",
       "1303        Ted Cruz took first prize in the Iowa caucuses...  \n",
       "9051        Posted on November 4, 2016 by Charles Hugh Smi...  \n",
       "10200       Charlie Baker , Massachusetts (2015–present)[3...  \n",
       "10009       vladimir putin , Valdai , sochi , RBTH Daily R...  \n",
       "4214        ROME —  U.S. Democratic presidential candidate...  \n",
       "2316        Most conservatives who oppose marriage equalit...  \n",
       "8411        Written by Peter Van Buren   venerable New Yor...  \n",
       "6143        DOJ COMPLAINT: Comey Under Fire Over Partisan ...  \n",
       "3262        The freshman senator from Georgia quoted scrip...  \n",
       "9337                                                           \n",
       "8737        Julian Assange has claimed the Hillary Clinton...  \n",
       "4490        The State Department told the Republican Natio...  \n",
       "8062        The ‘P’ in PBS Should Stand for ‘Plutocratic’ ...  \n",
       "8622         Anti-Trump Protesters Are Tools of the Oligar...  \n",
       "4021        ADDIS ABABA, Ethiopia —President Obama convene...  \n",
       "4330        Jeb Bush Is Suddenly Attacking Trump. Here's W...  \n",
       "\n",
       "[6335 rows x 2 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the `label` column\n",
    "df.drop(\"label\", axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make training and test sets \n",
    "X_train, X_test, y_train, y_test = train_test_split(df['text'], y, test_size=0.33, random_state=53)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building Vectorizer Classifiers\n",
    "\n",
    "Now that you have your training and testing data, you can build your classifiers. To get a good idea if the words and tokens in the articles had a significant impact on whether the news was fake or real, you begin by using `CountVectorizer and TfidfVectorizer`.\n",
    "\n",
    "You’ll see the example has a max threshhold set at .7 for the TF-IDF vectorizer `tfidf_vectorizer` using the `max_df argument`. This removes words which appear in more than 70% of the articles. Also, the built-in `stop_words` parameter will remove English stop words from the data before making vectors.\n",
    "\n",
    "There are many more parameters available and you can read all about them in the scikit-learn documentation for TfidfVectorize and CountVectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Initialize the `count_vectorizer` \n",
    "count_vectorizer = CountVectorizer(stop_words='english')\n",
    "\n",
    "# Fit and transform the training data \n",
    "count_train = count_vectorizer.fit_transform(X_train) \n",
    "\n",
    "# Transform the test set \n",
    "count_test = count_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the `tfidf_vectorizer` \n",
    "tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7) \n",
    "\n",
    "# Fit and transform the training data \n",
    "tfidf_train = tfidf_vectorizer.fit_transform(X_train) \n",
    "\n",
    "# Transform the test set \n",
    "tfidf_test = tfidf_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have vectors, you can then take a look at the vector features, stored in count_vectorizer and tfidf_vectorizer.\n",
    "\n",
    "Are there any noticable issues? (Yes!)\n",
    "\n",
    "There are clearly comments, measurements or other nonsensical words as well as multilingual articles in the dataset that you have been using. Normally, you would want to spend more time preprocessing this and removing noise, but as this tutorial just showcases a small proof of concept, you will see if the model can overcome the noise and properly classify despite these issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['حلب', 'عربي', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']\n",
      "['00', '000', '0000', '00000031', '000035', '00006', '0001', '0001pt', '000ft', '000km']\n"
     ]
    }
   ],
   "source": [
    "# Get the feature names of `tfidf_vectorizer` \n",
    "print(tfidf_vectorizer.get_feature_names()[-10:])\n",
    "\n",
    "# Get the feature names of `count_vectorizer` \n",
    "print(count_vectorizer.get_feature_names()[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermezzo: Count versus TF-IDF Features\n",
    "\n",
    "I was curious if my count and TF-IDF vectorizers had extracted different tokens. To take a look and compare features, you can extract the vector information back into a DataFrame to use easy Python comparisons.\n",
    "\n",
    "As you can see by running the cells below, both vectorizers extracted the same tokens, but obviously have different weights. Likely, changing the `max_df` and `min_df` of the TF-IDF vectorizer could alter the result and lead to different features in each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_df = pd.DataFrame(count_train.A, columns=count_vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_df = pd.DataFrame(tfidf_train.A, columns=tfidf_vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "set()"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "difference = set(count_df.columns) - set(tfidf_df.columns)\n",
    "difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "print(count_df.equals(tfidf_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>0000</th>\n",
       "      <th>00000031</th>\n",
       "      <th>000035</th>\n",
       "      <th>00006</th>\n",
       "      <th>0001</th>\n",
       "      <th>0001pt</th>\n",
       "      <th>000ft</th>\n",
       "      <th>000km</th>\n",
       "      <th>...</th>\n",
       "      <th>حلب</th>\n",
       "      <th>عربي</th>\n",
       "      <th>عن</th>\n",
       "      <th>لم</th>\n",
       "      <th>ما</th>\n",
       "      <th>محاولات</th>\n",
       "      <th>من</th>\n",
       "      <th>هذا</th>\n",
       "      <th>والمرضى</th>\n",
       "      <th>ยงade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 56922 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   00  000  0000  00000031  000035  00006  0001  0001pt  000ft  000km  ...  \\\n",
       "0   0    0     0         0       0      0     0       0      0      0  ...   \n",
       "1   0    0     0         0       0      0     0       0      0      0  ...   \n",
       "2   0    0     0         0       0      0     0       0      0      0  ...   \n",
       "3   0    0     0         0       0      0     0       0      0      0  ...   \n",
       "4   0    0     0         0       0      0     0       0      0      0  ...   \n",
       "\n",
       "   حلب  عربي  عن  لم  ما  محاولات  من  هذا  والمرضى  ยงade  \n",
       "0    0     0   0   0   0        0   0    0        0      0  \n",
       "1    0     0   0   0   0        0   0    0        0      0  \n",
       "2    0     0   0   0   0        0   0    0        0      0  \n",
       "3    0     0   0   0   0        0   0    0        0      0  \n",
       "4    0     0   0   0   0        0   0    0        0      0  \n",
       "\n",
       "[5 rows x 56922 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>0000</th>\n",
       "      <th>00000031</th>\n",
       "      <th>000035</th>\n",
       "      <th>00006</th>\n",
       "      <th>0001</th>\n",
       "      <th>0001pt</th>\n",
       "      <th>000ft</th>\n",
       "      <th>000km</th>\n",
       "      <th>...</th>\n",
       "      <th>حلب</th>\n",
       "      <th>عربي</th>\n",
       "      <th>عن</th>\n",
       "      <th>لم</th>\n",
       "      <th>ما</th>\n",
       "      <th>محاولات</th>\n",
       "      <th>من</th>\n",
       "      <th>هذا</th>\n",
       "      <th>والمرضى</th>\n",
       "      <th>ยงade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 56922 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    00  000  0000  00000031  000035  00006  0001  0001pt  000ft  000km  ...  \\\n",
       "0  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...   \n",
       "1  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...   \n",
       "2  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...   \n",
       "3  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...   \n",
       "4  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...   \n",
       "\n",
       "   حلب  عربي   عن   لم   ما  محاولات   من  هذا  والمرضى  ยงade  \n",
       "0  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
       "1  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
       "2  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
       "3  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
       "4  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
       "\n",
       "[5 rows x 56922 columns]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing Models\n",
    "\n",
    "Now it's time to train and test your models.\n",
    "\n",
    "Here, you'll begin with an NLP favorite, `MultinomialNB`. You can use this to compare TF-IDF versus bag-of-words. My intuition was that bag-of-words (aka `CountVectorizer`) would perform better with this model. (For more reading on `multinomial` distribution and why it works best with integers, check out this fairly succinct explanation from a UPenn statistics course).\n",
    "\n",
    "I personally find Confusion Matrices easier to compare and read, so I used the scikit-learn documentation to build some easily-readable confusion matrices (thanks open source!). A confusion matrix shows the proper labels on the main diagonal (top left to bottom right). The other cells show the incorrect labels, often referred to as false positives or false negatives. Depending on your problem, one of these might be more significant. For example, for the fake news problem, is it more important that we don't label real news articles as fake news? If so, we might want to eventually weight our accuracy score to better reflect this concern.\n",
    "\n",
    "Other than Confusion Matrices, scikit-learn comes with many ways to visualize and compare your models. One popular way is to use a ROC Curve. There are many other ways to evaluate your model available in the scikit-learn metrics module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    See full source and example: \n",
    "    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
    "    \n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Drawing Confusion Matrix With TfIdf Vectorizer**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:   0.857\n",
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxVdf3H8dd7BkEREAERFBUXXIoU96VUTEPFBTM1t1yL9JdWmhX5s9SysuXnbhpmbpViWWquuWaaUmoomqAgLiiyiCCyCAyf3x/nO3hBmLkznJl77sz76eM85p5zvvecz5mLn/ne7/me71cRgZmZ5aem0gGYmbU1TqxmZjlzYjUzy5kTq5lZzpxYzcxy5sRqZpYzJ1ZbKUlrSPqrpNmS/rgKxzlG0t/yjK1SJO0uaXyl47Bik/uxVj9JRwNnAlsCc4AxwI8j4vFVPO6XgNOB3SJi8SoHWnCSAhgQERMqHYtVN9dYq5ykM4FLgJ8A6wIbAr8ChuVw+I2Al9tDUi2HpA6VjsGqRER4qdIFWAv4ADi8gTKdyBLv22m5BOiU9g0GJgPfAqYBU4AT077zgYXAonSOk4HzgN+VHLs/EECHtH4C8CpZrXkScEzJ9sdL3rcb8G9gdvq5W8m+R4EfAU+k4/wN6LWSa6uP/zsl8R8CDAVeBmYCZ5eU3wl4EpiVyl4BdEz7HkvXMjdd7xdLjv9d4B3gpvpt6T2bpnNsl9bXA2YAgyv9b8NLZRfXWKvbrsDqwF8aKPO/wC7AIGAbsuRyTsn+PmQJen2y5HmlpLUj4lyyWvCoiOgSEdc2FIikNYHLgP0joitZ8hyzgnI9gLtT2Z7ARcDdknqWFDsaOBHoDXQEzmrg1H3IfgfrAz8ArgGOBbYHdgd+IGmTVLYOOAPoRfa72xv4H4CI2COV2SZd76iS4/cgq70PLz1xREwkS7q/l9QZuA64PiIebSBeawecWKtbT2BGNPxV/RjghxExLSKmk9VEv1Syf1Havygi7iGrrW3RzHiWAAMlrRERUyLixRWUOQB4JSJuiojFEXEzMA44qKTMdRHxckTMB24l+6OwMovI2pMXAbeQJc1LI2JOOv+LwNYAEfFMRDyVzvsa8GtgzzKu6dyI+DDFs4yIuAZ4BRgN9CX7Q2btnBNrdXsX6NVI2996wOsl66+nbUuPsVxingd0aWogETGX7OvzKcAUSXdL2rKMeOpjWr9k/Z0mxPNuRNSl1/WJb2rJ/vn175e0uaS7JL0j6X2yGnmvBo4NMD0iFjRS5hpgIHB5RHzYSFlrB5xYq9uTwAKydsWVeZvsa2y9DdO25pgLdC5Z71O6MyLuj4jPkdXcxpElnMbiqY/prWbG1BRXkcU1ICK6AWcDauQ9DXabkdSFrN36WuC81NRh7ZwTaxWLiNlk7YpXSjpEUmdJq0naX9LPU7GbgXMkrSOpVyr/u2aecgywh6QNJa0FfK9+h6R1JR2c2lo/JGtSqFvBMe4BNpd0tKQOkr4IfAK4q5kxNUVX4H3gg1SbPnW5/VOBTT72roZdCjwTEV8mazu+epWjtKrnxFrlIuIisj6s5wDTgTeB04DbU5ELgKeB54GxwLNpW3PO9QAwKh3rGZZNhjVkvQveJrtTvifpxtByx3gXODCVfZfsjv6BETGjOTE10VlkN8bmkNWmRy23/zzgBkmzJB3R2MEkDQP2I2v+gOxz2E7SMblFbFXJDwiYmeXMNVYzs5w5sZqZ5cyJ1cwsZ06sZmY5a7ODSnTovFZ07N6n8YJWGBv17Nx4ISuUKW+9wayZ7zbWF7hJarttFLH4Yw+5fUzMn35/ROyX57nz0mYTa8fufdh8+K8qHYY1wcjjdqh0CNZExw8bnPsxY/F8Om3RaG83Foy5srGn5iqmzSZWM6tWAlV3K6UTq5kVi4Ca2kpHsUqcWM2seJRrs22rc2I1s4JxU4CZWf5cYzUzy5Go+hprdUdvZm2QshprY0s5R5J+K2mapBdKtvWQ9ICkV9LPtdN2SbpM0gRJz0varuQ9x6fyr0g6vrHzOrGaWfHU1Da+lOd6sqEdS40AHoqIAcBDaR1gf2BAWoaTDYxeP0/bucDOZHPGnVufjFcafrnRmZm1jnTzqrGlDBHxGNn4wKWGATek1zfw0Qwcw4AbI/MU0F1SX2Bf4IGImBkR7wEP8PFkvQy3sZpZsYiWvnm1bkRMAYiIKZJ6p+3rkw0UX29y2ray7SvlxGpmxVNejbSXpKdL1kdGxMhVOesKtkUD21fKidXMCqbsfqwzIqI5A0xMldQ31Vb7AtPS9snABiXl+pFNNTQZGLzc9kcbOoHbWM2seGrU+NJ8dwL1d/aPB+4o2X5c6h2wCzA7NRncDwyRtHa6aTUkbVsp11jNrFhyHCtA0s1ktc1ekiaT3d2/ELhV0snAG8Dhqfg9wFBgAjAPOBEgImZK+hHw71TuhxGx/A2xZTixmlnB5PdIa0QctZJde6+gbABfW8lxfgv8ttzzOrGaWfH4kVYzs5xV+SOtTqxmVixNeGS1qJxYzax4XGM1M8uTPIOAmVnu3BRgZpajNjAeqxOrmRWMp2YxM8ufmwLMzHLmm1dmZjmSmwLMzPLnpgAzs3zJidXMLD/ZzCxOrGZm+RErngylijixmlnBiJoa37wyM8uVmwLMzHLmxGpmlie3sZqZ5UvINVYzs7w5sZqZ5cy9AszM8uQ2VjOz/LkpwMwsR755ZWbWApxYzczyVt151YnVzApG7hVgZpY7NwWYmeXIN6/MzFpCdedVJ9ai2ahnZ35++MCl6+uvvQZXPfIqa62xGoO37EUEzJy7kB/c/l+mz1lI19U7cP6wrejXYw0WLl7CuXe8xMRpcyt4Be3P1Lcnc95ZpzBzxjRUU8MhXzyeI088FYBbb/g1f7zpGmo7dODTg4dw+ogfsmjhQn56zjcZN3YMqhFnfv9Ctt9l9wpfRYHITQGWs9ffnccXr/4XADWCv33rMzz80nTeX7CIXz3yKgBH7dyP4XtuzI/vGs+Xd+/P+Hc+4MxRY+nfqzPfG7oFX73xP5W8hHantkMHvnH2BWw5cBBzP5jD8cMGs9Nn9mLmjGk89uA9/P7uJ+jYqRMzZ0wH4PZRNwDwh3v/ycwZ0/nmSYdx/e2PVP0NmzxVe2L1J1lgO2/Sg8kz5zNl9gLmfli3dPsaq9USkb3eZJ01GT1pJgCvzZjHet1Xp8eaHSsRbrvVq3cfthw4CIA1u3Sl/2abM33qFP78h99y3Cln0LFTJwB69FoHgEkTxrPjbnsu3da121q8NNZ/DEupRo0uRebEWmD7DlyXe1+YunT9tM9uwn1nfJqhW/fhqlR7fXnqB+y9VW8ABq7fjb7dV2fdbp0qEq/B25Nf5+UXx/LJbbbnjUkTGPPvf3LSoXtzylFD+e/zzwIwYMuBPPbgPSxevJi333yNcS+MYeqUyRWOvFgkNbqUeZwzJL0o6QVJN0taXdLGkkZLekXSKEkdU9lOaX1C2t+/ufG3WGKVVCdpTMnSv2TfpZLeklRTsu0ESVek1zWSbpD0W2VekzS25FiXtVTcRdGhVuy5RS8eeHHa0m1XPPwq+138BPc8/w5H7tQPgN8+/hrdVu/AqFN24sid+jF+ygfULYlKhd2uzZv7ASP+5zjO+P5P6NK1G3WL65gzexbX3vYgp4/4EWeffgIRwUGHH0vvPutxwiGDueiC7/Gp7XamttatcvXKSarlJFZJ6wNfB3aIiIFALXAk8DPg4ogYALwHnJzecjLwXkRsBlycyjVLS36a8yNi0PIbUzL9PPAmsAfw6HL7BVwNrAacGBGRfol7RcSMFoy3UD6zWU/GTZnDzLkLP7bv3rFTufyYbbjq0UnM/bCOc+94aem+e765G2/Nmt+aoRqweNEiRnztOPYbdjh77XswAL37rMfgfQ9CEp/cZntqamqYNfNd1u7ZizPO+enS9375sCFs0H/TSoVeSDm2sXYA1pC0COgMTAE+Cxyd9t8AnAdcBQxLrwH+BFwhSRHR5JpKJZoC9gJeILuQo1aw/1KgJ3BcRCxpzcCKZL9P9eG+sR81A2zYY42lr/fcoheTZswDoOvqHehQm/0jPHS79Xjm9VnLtMday4sILhhxGv033ZyjTz5t6fY9hxzA008+BsAbkyawaOEiuvfoyYL585g/L+u5MfrxR6jtUMsmA7asSOxFVWaNtZekp0uW4aXHiIi3gF8Cb5Al1NnAM8CsiFicik0G1k+v1yer8JH2zybLRU3WkjXWNSSNSa8nRcTn0+ujgJuBO4CfSFotIhalfUcDLwGDSy683iOS6jPGDRFx8fInTL/Y4QCrrdU7x0tpXauvVsMum/Tggr9+VBP9+j6b0b9XZ5ZEMGXWAn5813gANu7VmQs+/0nqInh1+lzOK6m9Wut47pmnuPf2UWy2xSc49sDPAHDqt37AQYcdywUjTuOo/XZltY6rce4vfoUkZr47nW+c8AVqampYZ92+nPd/v67wFRRPmTenZkTEDis9hrQ2WS10Y2AW8Edg/xUUra+RruikzWpXa9WmgNRIPBQ4IyLmSBoNDAHuTkWeBbYEdgKeWO54jTYFRMRIYCRA5/W2qNqGxgWLljD4548ts+2sW8eusOzzk9/n4MufbI2wbCUG7bAroyfOWuG+8y8a+bFt6/XbiD8++HRLh1W98uvHug9ZpW46gKQ/A7sB3SV1SJW3fsDbqfxkYANgsqQOwFrAzOacuLWbAvYjC3aspNeAz7Bsc8A44AhglKRPtnJsZlYAAqTGlzK8AewiqXO6d7M38F/gEeCwVOZ4sm/PAHemddL+h5vTvgqtn1iPAr4cEf0joj9ZFX2IpM71BSLin8ApwN2SNmzl+Mys4vLpFRARo8luQj0LjCXLdyOB7wJnSppA1oZ6bXrLtUDPtP1MYERzr6DV+nik5Lkv8NX6bRExV9LjwEGlZSPiLknrAPdJqn/Wr7SN9fmIOK414jaz1pdXp4CIOBc4d7nNr5I1Ny5fdgFweB7nbbHEGhFdllufB/RYQblDS1avL9l+HXBdWu2ff4RmVlTV/kireyWbWaFIUFvrxGpmlqsqr7A6sZpZ8bgpwMwsT+V3pyosJ1YzK5SsH2t1Z1YnVjMrGM95ZWaWu5qCD2TdGCdWMysWt7GameXLbaxmZi2gyvOqE6uZFY9rrGZmOavyvOrEambFIrlXgJlZztyP1cwsd1WeV51Yzax4XGM1M8uTHxAwM8uXgJqa1p6OL19OrGZWOK6xmpnlzG2sZmZ5churmVm+5H6sZmb5q/K86sRqZsVT60dazczyI7Xhm1eSujX0xoh4P/9wzMygyiusDdZYXwSCrL9uvfr1ADZswbjMrB1rszXWiNigNQMxM6tX5XmVsp4bk3SkpLPT636Stm/ZsMysvRKpy1Uj/xVZo4lV0hXAXsCX0qZ5wNUtGZSZtWMStTWNL0VWTq+A3SJiO0n/AYiImZI6tnBcZtaOVXtTQDmJdZGkGrIbVkjqCSxp0ajMrN0SUFPlmbWcNtYrgduAdSSdDzwO/KxFozKzdk1qfCmyRmusEXGjpGeAfdKmwyPihZYNy8zas7y6W0nqDvwGGEj2rfskYDwwCugPvAYcERHvKTvppcBQsntJJ0TEs805b7mjydYCi4CFTXiPmVmTlVNbbULevRS4LyK2BLYBXgJGAA9FxADgobQOsD8wIC3Dgauaew3l9Ar4X+BmYD2gH/AHSd9r7gnNzBpTKzW6NCY9PboHcC1ARCyMiFnAMOCGVOwG4JD0ehhwY2SeArpL6tuc+Mu5eXUssH1EzEvB/hh4Bvhpc05oZtaYMpsCekl6umR9ZESMLFnfBJgOXCdpG7K89Q1g3YiYAhARUyT1TuXXB94sef/ktG1KU+MvJ7G+vly5DsCrTT2RmVk5sl4BZRWdERE7NLC/A7AdcHpEjJZ0KR997V/ZqZcXZUWyghOv+AzSxemg84AXJd2f1oeQ9QwwM8ufchvoejIwOSJGp/U/kSXWqZL6ptpqX2BaSfnSR/n7AW8358QN1Vjr7/y/CNxdsv2p5pzIzKxceeTViHhH0puStoiI8cDewH/TcjxwYfp5R3rLncBpkm4BdgZm1zcZNFVDg7Bc25wDmpmtqhxHtzod+H16WvRV4ESym/a3SjoZeAM4PJW9h6yr1QSyb+onNvekjbaxStoU+DHwCWD1+u0RsXlzT2pmtjIivxkEImIMsKJ22L1XUDaAr+Vx3nL6pF4PXEd2vfsDtwK35HFyM7MVURlLkZWTWDtHxP0AETExIs4hG+3KzCx3UjZWQGNLkZXT3erD9KjXREmnAG8BvRt5j5lZsxU8bzaqnMR6BtAF+DpZW+taZM/bmpm1iDY7NUu9kj5gc/hosGszsxYhij+QdWMaekDgLzTw1EFEHNoiEZlZ+1YFwwI2pqEa6xWtFkUL2KpvV54452M9KqzA1t7xtEqHYE304cS3WuS4bbYpICIeas1AzMzqVfvYpOXcvDIzazWiDddYzcwqpcrvXZWfWCV1iogPWzIYMzMpv0daK6WcGQR2kjQWeCWtbyPp8haPzMzarRo1vhRZOW3ElwEHAu8CRMRz+JFWM2tBbX6WVqAmIl5frjG5roXiMbN2LptBoOCZsxHlJNY3Je0EhKRasvENX27ZsMysPWsP3a1OJWsO2BCYCjyYtpmZtYgqr7CWNVbANODIVojFzAypDY8VUE/SNaxgzICIGN4iEZlZu1flebWspoAHS16vDnyeZefeNjPLTbu4eRURo0rXJd0EPNBiEZlZu1flebVZj7RuDGyUdyBmZgBUwQMAjSmnjfU9PmpjrQFmAiNaMigza99U+OkCG9ZgYk1zXW1DNs8VwJI0RayZWYsQ0KHKO7I2GH5Kon+JiLq0OKmaWYuT1OhSZOX8XfiXpO1aPBIzM+p7BVT3ICwNzXnVISIWA58BviJpIjCX7LojIpxszSx/VTDISmMaamP9F7AdcEgrxWJmBrTtfqwCiIiJrRSLmdnSpoBq1lBiXUfSmSvbGREXtUA8Ztbuido2XGOtBbpAlXcoM7Oqkk0mWOkoVk1DiXVKRPyw1SIxM4M2/+RVlV+amVWrtnzzau9Wi8LMLGnTTQERMbM1AzEzq1ftA11X+RO5ZtbWiCwxNbaUfTypVtJ/JN2V1jeWNFrSK5JGSeqYtndK6xPS/v7NvQYnVjMrFuU+VsA3gJdK1n8GXBwRA4D3gJPT9pOB9yJiM+DiVK5ZnFjNrHBUxlLWcaR+wAHAb9K6gM8Cf0pFbuCjp0uHpXXS/r3VzNFemjPQtZlZi2nC1Cy9JD1dsj4yIkYuV+YS4DtA17TeE5iVxkEBmAysn16vT5p2KiIWS5qdys9o6jU4sZpZ4ZRZTZwRETus9BjSgcC0iHhG0uAGDh1l7GsSJ1YzKxhRk0+vgE8DB0saSjYRajeyGmz3ktH7+gFvp/KTgQ2AyZI6AGuRzZjSZG5jNbNCyatXQER8LyL6RUR/4Ejg4Yg4BngEOCwVOx64I72+M62T9j/c3MH9nVjNrHBaeAaB7wJnSppA1oZ6bdp+LdAzbT+TVZjbz00BZlY4eT8eEBGPAo+m168CO62gzALg8DzO58RqZsWS+rFWMydWMyuU+jbWaubEamaF05ZHtzIzq4gqz6tOrGZWLFlTQHVnVidWMysc11jNzHIl5BqrmVm+XGM1M8uRRJue/trMrCKqPK86sZpZ8biN1VrMggUL2GevPVj44YcsrlvM5w89jO+fez5XXXkFV1x+Ca9OnMibU6bTq1evSofa7lx97jHsv8dAps+cww6H/wSAtbt15qafncRG6/Xg9bdncux3rmXWnPnsvv0A/njxcF57+10A7nh4DD8deR/91u3Ob350HOv27MaSCH572xNcefOjFbyqYsgGuq50FKvGibXAOnXqxH0PPEyXLl1YtGgRn93zMwzZd3923e3TDD3gQIbsM7jSIbZbN/31Ka4e9Xd+86Pjlm4768TP8ei/xvPL6x7grBM/x1knDuGcy7IR6Z74z0S+8I2rlznG4roljLjoz4wZN5kunTvxzz98l4dGj2Pcq++06rUUUbXXWKv9kdw2TRJdunQBYNGiRSxetAhJDNp2Wzbq37+ywbVzTzw7kZmz5y2z7cDBW/O7v44G4Hd/Hc1Be23d4DHemfE+Y8ZNBuCDeR8ybtI7rLdO95YJuMpIjS9F5sRacHV1dey8/SA2XK83n93nc+y0886VDslWonfPrrwz430gS5rr9Oi6dN/OW2/M6FEjuP2KU9lqkz4fe++GfXswaIt+/PuF11or3MISWa+AxpYia7HEKqlO0hhJL0j6q6TuaXt/SfPTvvrluJL3bSspJO273PE+aKlYi6y2tpbRz4xhwmuTefrf/+LFF16odEjWRGPGvckWQ7/Pzl+8kKtu+Tu3Xjx8mf1rrtGRm3/5Zb79y9uYM3dBhaIsEpX1X5G1ZI11fkQMioiBZPPGfK1k38S0r365sWTfUcDj6acl3bt3Z489B/O3v91X6VBsJaa9O4c+vboB0KdXN6bPnAPAnLkLmDt/IQD3P/5fVutQS8/uawLQoUMNN//yK4y692nuePi5ygReNGU0AxS8wtpqTQFP8tEUsyuV5vA+DDgBGCJp9RaOq9CmT5/OrFmzAJg/fz4PP/QgW2yxZYWjspW5++9jOfagrKnm2IN25q5Hnwdg3Z4fNQns8MmNqJF4d9ZcIOtdMH7SO1z2u4dbP+ACUxlLkbV4rwBJtcDefDSvDMCmksaUrJ8eEf8gm1VxUkRMlPQoMBT4cxPONRwYDrDBhhuuaugV986UKXzlpOOpq6tjSSzhC4cdwdADDuTKyy/jov/7OVPfeYcdt9ua/fYbylUjf1PpcNuVG356ArtvP4Be3bsw4b4f8aOr7+GX1z3A7352EscfsitvTnmPY76T/ZP//D7b8pXDd2dxXR0LFiziuO9dB8BugzbhmAN3ZuzLb/HULdn0SudecSf3P/7fil1XEWTdrYqeOhumZk5C2PiBpTpgLNAfeAYYEhF1kvoDd6UmguXfcyUwJiKukXQw8KWIODzt+yAiupR7/u233yGeGP30ql+ItZq1dzyt0iFYE304/laWzJuWaxbc6lPbxnW3P9JouV03W/uZiNghz3PnpcXbWIGNgI4s28b6Malm+wXgB5JeAy4H9pfUtaH3mVnb45tXjYiI2cDXgbMkrdZA0X2A5yJig4joHxEbAbcBh7R0jGZWLL55VYaI+A/wHHBk2rTpct2tvk7WC+Avy731NuDo9LqzpMkly5mtEbuZtT7fvFqJ5dtDI+KgktU1yjzGncCd6bUfZjBrL4qeORvhsQLMrFCyGml1Z1YnVjMrFnl0KzOz/DmxmpnlqfjdqRrjxGpmhVP07lSNcWI1s0Kphu5UjXFiNbPiqfLM6sRqZoVT7YOwOLGaWeFUd1r11CxmVjTlPM9aRuaVtIGkRyS9JOlFSd9I23tIekDSK+nn2mm7JF0maYKk5yVt19xLcGI1s8LJaXSrxcC3ImIrYBfga5I+AYwAHoqIAcBDaR1gf2BAWoYDVzU3fidWMysUkc/oVhExJSKeTa/nAC+RzWQyDLghFbuBj0bQGwbcGJmngO6S+jbnGpxYzaxw8h7dKg2wvy0wGlg3IqZAlnyB3qnY+sCbJW+bTBlTSq2Ib16ZWeGovF4BvSSVThMyMiJGruBYXciGIP1mRLzfwLFXtKNZU6w4sZpZ4ZTZ22pGY1OzpMH1bwN+HxH18+dNldQ3Iqakr/rT0vbJwAYlb+8HvN2kwBM3BZhZ4eTRFJBmfb4WeCkiLirZdSdwfHp9PHBHyfbjUu+AXYDZ9U0GTeUaq5kVTz4dWT8NfAkYWzIr9NnAhcCtkk4G3gAOT/vuIZsZegIwDzixuSd2YjWzQslroOuIeJyVp+i9V1A+aGTS03I5sZpZsXigazOzFuDEamaWJw90bWaWuyof3MqJ1cyKxQNdm5m1hCrPrE6sZlY4HujazCxn1Z1WnVjNrGjKHBawyJxYzayAqjuzOrGaWaHUD3RdzZxYzaxwqjyvOrGaWfG4V4CZWd6qO686sZpZ8VR5XnViNbNiKXcW1iJzYjWzwvHoVmZmeavuvOrEambF4xkEzMxy5YGuzcxy1RaevKqpdABmZm2Na6xmVjjVXmN1YjWzwnEbq5lZjiT3CjAzy58Tq5lZvtwUYGaWM9+8MjPLWZXnVSdWMyseVXmV1YnVzAqlLTx5pYiodAwtQtJ04PVKx9ECegEzKh2ENUlb/sw2ioh18jygpPvIfmeNmRER++V57ry02cTaVkl6OiJ2qHQcVj5/Zu2PxwowM8uZE6uZWc6cWKvPyEoHYE3mz6ydcRurmVnOXGM1M8uZE6uZWc6cWKucpJ6VjsHMluXEWsUkDQEukbS2qv0ZwHbCn1P74MRapVJS/QVwbUS8hx9PrhY9AST5/702zB9uFZK0H1lS/WpEPCppA+BsSeU8BmgVoExv4HVJB0fEEifXtssfbHXaGegcEU9JWgf4CzAtItrq8+hVLzLTgBOB6yQNrU+ukmorHZ/ly18fq4ikTwN7RsT5kjaR9CTZH8dfR8Q1JeU2iIg3KxaorVRE3CppIXCLpKMi4u76mqukg7IicVdlo7RV5RprFSj5yjgEWAsgIo4HHgPWXi6pHgNcJqlrqwdqHyNpP0nfl7Rr/baIuJ2s5nqLpANTzfWrwNXAuErFavlxjbU6rAW8BywAln5tjIjvSlpH0iMRsZekLwBnAMdFxJwKxWrL2hM4BdhP0ovAFcCkiLgt9RC4XtJdwE7A0IiYUMFYLSeusRacpI2Bn0raBJgKdE3b1wCIiJOAVyVNAc4mS6r/rVS89jF3Ag8CXwDmAUcCN0naJCL+BBwBHAwcHRHPVS5My5NrrMW3OjAN+CqwDjA5be8kaUG6KXKypLOAe5xUK0/SlsCHETEpIp6U1An4ZkR8U9LRwAigi6TJwKVAn4hYWMmYLV8ehKUKSBoI7AecBmxIVgvaFngbWATMAQ6JiEUVC9IAkDQU+D7wpfqv9ZIGAF8BxpN9q/gy2We3G/BoREyqULjWQlxjLSBJg8k+m8ciYmFEvCBpEdAZ2Aq4HhgLrAl0IzvRYEEAAAXbSURBVOtq5aRaYZL2JUuq50XEBEldgCCblmUj4GvA/hHxWCr/crhm0ya5xlowktYC7gY2Bi4B6iLiorRvU+CLQF/gpoj4V8UCtWVI+hTwHLBPRDycPqtfA2dGxPOStib7g3hYRLxawVCtFfjmVcFExGzgLmAh8AowVNL1kg4ha2u9kqyHwBGSVvez55VV8vt/jexBjSMk9Scb3Pr+lFRrIuJ54B/AXn4goO1zYi0ISX1K/if9P+BeYE5E7AN0BC4i67e6Z/r5k4hY4K+SFdcRIHVvOwboAkwEbo+IX6SkukTSILImgfsioq5y4VprcGItAEkHkN2Q6pUeBhBZ7XTb1M1qF7IO5ZcAhwL/iYiZlYrXMmkgnFsknSfp0IhYQNZ74w/ArgApqZ4MXAZcExFvVS5iay1uY62wNKDK/wI/joj7JHWMiIVpYJVnyGpAR9Q/5iipc0TMq2DIxtLP7XzgRqA3sB7w84h4JT319iuyG1d/I3tA4JSIeKFS8VrrcmKtIEk9yL4eHhoRt6cbHj8Avh0R0yQNB7aOiNPqE25FAzZgmc9tWET8VVI/4MfAVRHxVCrTERhF9hjyju5f3L64KaCC0tf5g4AfpLvGI8m+5k9LRZ4D9pa0uZNqcZR8bhdK6hYRk8ke3rhQ0iWSvkXWFe5kYDMn1fbH/VgrLI1uVAeMAc6OiEsk1UZEXUSMlvSHSsdoH5c+tyXAM5LuI7uJdSXQg+wBgK3Iulq5LbwdclNAQUj6HHA5sHNEzJbUKSI+rHRc1jBJ+5C1o/aNiKlpWw3Qw+Pjtl9uCiiIiHiAbGSqf0nq4aRaHSLiQeAA4GFJ66ZtS5xU2zc3BRRIRNybbno8KGkH0sDzlY7LGlbyud0raYeIWFLpmKyy3BRQQJK6RMQHlY7Dmsafm9VzYjUzy5nbWM3McubEamaWMydWM7OcObGameXMibWdklQnaYykFyT9UVLnVTjW4DTTKJIOljSigbLdJf1PM85xXprXq6zty5W5XtJhTThXf0keMMWazYm1/ZofEYMiYiDZoNqnlO5Upsn/PiLizoi4sIEi3YEmJ1azauLEapCNbL9Zqqm9JOlXwLPABpKGSHpS0rOpZtsFsmHzJI2T9DjZGLGk7SdIuiK9XlfSXyQ9l5bdgAuBTVNt+Rep3Lcl/VvS85LOLznW/0oaL+lBYIvGLkLSV9JxnpN023K18H0k/UPSy5IOTOVrJf2i5NxfXdVfpBk4sbZ7kjoA+5NNTghZArsxIrYF5gLnkM3jtB3wNHCmpNWBa8hGeNod6LOSw18G/D0itgG2A14km/p5YqotfzsNFj0A2AkYBGwvaQ9J2wNHks1GeyiwYxmX8+eI2DGd7yWy0aXq9SebfeEA4Op0DScDsyNix3T8r0jauIzzmDXIj7S2X2tIGpNe/wO4lmyw5tfrxxQlm7ngE8ATadaYjsCTwJbApIh4BUDS74DhKzjHZ4HjANJ0JLMlrb1cmSFp+U9a70KWaLsCf6kf1FvSnWVc00BJF5A1N3QB7i/Zd2t61PQVSa+maxgCbF3S/rpWOvfLZZzLbKWcWNuv+RExqHRDSp5zSzcBD0TEUcuVG0Q2On4eBPw0In693Dm+2YxzXA8cEhHPSToBGFyyb/ljRTr36RFRmoBRNhmgWbO5KcAa8hTwaUmbQTYtjKTNgXHAxmnGA4CjVvL+h4BT03trJXUD5pDVRuvdD5xU0na7vqTeZBMmfl7SGmmqk4PKiLcrMEXSamQT+5U6XFJNinkTYHw696mpPJI2l7RmGecxa5BrrLZSETE91fxultQpbT4nIl5WNm3M3ZJmAI8DA1dwiG8AI5VNplcHnBoRT0p6InVnuje1s24FPJlqzB8Ax0bEs5JGkQ0A/jpZc0Vjvg+MTuXHsmwCHw/8HViXbP6pBZJ+Q9b2+qyyk08HDinvt2O2ch6ExcwsZ24KMDPLmROrmVnOnFjNzHLmxGpmljMnVjOznDmxmpnlzInVzCxn/w9gnE8/gJJ/qQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "clf = MultinomialNB() \n",
    "clf.fit(tfidf_train, y_train)\n",
    "pred = clf.predict(tfidf_test)\n",
    "score = accuracy_score(y_test, pred)\n",
    "print(\"accuracy:   %0.3f\" % score)\n",
    "cm = confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drawing Confusion Matrix With Bag Of Words (Count Vectorizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:   0.893\n",
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3debxVZb3H8c+XUREVZBZQ0FAzR5w1pyxSlCDTnAei0FIbzNLU63C7lV1vapRlec0hy7TMCc1S1EpSuKComAMooiCKoOKAyPS7f6zn4GY6e5/DOmevfc737Wu9zt7PevZav302/s6zn/Ws51FEYGZm+WlT7QDMzFoaJ1Yzs5w5sZqZ5cyJ1cwsZ06sZmY5c2I1M8uZE6utlaT1Jd0laYGkP67DcY6T9Lc8Y6sWSftKeq7acVixyeNYa5+kY4EzgW2Ad4EpwA8i4uF1PO4JwBnA3hGxdJ0DLThJAQyKiOnVjsVqm1usNU7SmcAVwA+BXsBmwC+A4TkcfnPg+daQVCshqV21Y7AaERHeanQDNgbeA46sp05HssT7atquADqmfQcAs4BvA3OBOcDItO9iYDGwJJ1jFHARcGPJsQcAAbRLz08GXiRrNc8Ajispf7jkdXsD/wcsSD/3Ltn3EPB9YHw6zt+A7mt5b3Xxf7ck/hHAUOB54E3g3JL6uwOPAG+nuj8HOqR9/0jv5f30fo8qOf7ZwGvAb+vK0mu2TOcYnJ5vCswDDqj2vw1v1d3cYq1tewHrAbfVU+c8YE9gJ2BHsuRyfsn+3mQJui9Z8rxSUteIuJCsFXxzRHSOiGvqC0TSBsAY4JCI2JAseU5ZQ71NgLtT3W7AZcDdkrqVVDsWGAn0BDoAZ9Vz6t5kv4O+wAXA1cDxwC7AvsAFkrZIdZcB3wK6k/3uDgK+BhAR+6U6O6b3e3PJ8Tcha72PLj1xRLxAlnR/J6kTcC1wXUQ8VE+81go4sda2bsC8qP+r+nHAf0bE3Ih4g6wlekLJ/iVp/5KIuIestbZ1I+NZDmwnaf2ImBMRT6+hzqHAtIj4bUQsjYibgGeBYSV1ro2I5yPiA+AWsj8Ka7OErD95CfAHsqT504h4N53/aWAHgIiYHBGPpvO+BPwK2L+C93RhRHyY4llJRFwNTAMmAH3I/pBZK+fEWtvmA93L9P1tCswseT4zla04xiqJeSHQuaGBRMT7ZF+fTwXmSLpb0jYVxFMXU9+S5681IJ75EbEsPa5LfK+X7P+g7vWStpI0VtJrkt4ha5F3r+fYAG9ExKIyda4GtgN+FhEflqlrrYATa217BFhE1q+4Nq+SfY2ts1kqa4z3gU4lz3uX7oyIv0bEZ8habs+SJZxy8dTFNLuRMTXEL8niGhQRGwHnAirzmnqHzUjqTNZvfQ1wUerqsFbOibWGRcQCsn7FKyWNkNRJUntJh0j671TtJuB8ST0kdU/1b2zkKacA+0naTNLGwPfqdkjqJelzqa/1Q7IuhWVrOMY9wFaSjpXUTtJRwLbA2EbG1BAbAu8A76XW9FdX2f86sMVqr6rfT4HJEfFlsr7jq9Y5Sqt5Tqw1LiIuIxvDej7wBvAKcDpwe6ryX8Ak4EngKeCxVNaYc90H3JyONZmVk2EbstEFr5JdKd+fdGFolWPMBw5LdeeTXdE/LCLmNSamBjqL7MLYu2St6ZtX2X8RcL2ktyV9sdzBJA0HDibr/oDscxgs6bjcIraa5BsEzMxy5harmVnOnFjNzHLmxGpmLZak30iaK2lqSdkmku6TNC397JrKJWmMpOmSnpQ0uOQ1J6X60ySdVO68Tqxm1pJdR3aBsdQ5wLiIGASMS88BDgEGpW002fC8ursFLwT2ILtz8cK6ZLw2LXZSCXXcMNpsUG7stxXJ9gM8BLTWvDJzJvPnzys3FrhB2m60ecTS1W5yW0188MZfI2LVpLlynYh/SBqwSvFwsjkfAK4nm5/i7FR+Q2RX9B+V1EVSn1T3voh4E0DSfWTJ+qa1nbfFJtY2G3Sn05CLqh2GNcDfrj622iFYAw3Zf8/cjxlLP6Dj1mVHu7FoypXbSJpUUvTriPh1BafoFRFzACJijqSeqbwv2XDFOrNS2drK16rFJlYzq1UCVdRLOS8ids33xKuJesrXyn2sZlYsAtq0Lb813uvpKz7p59xUPgvoX1KvH9kNL2srXysnVjMrHqn81nh3AnVX9k8C7igpPzGNDtgTWJC6DP4KDJHUNV20GpLK1spdAWZWMBV3BZQ/knQT2cWn7pJmkV3dvwS4RdIo4GXgyFT9HrJJ0qeTzao2EiAi3pT0fbJJ2SGbZvPN+s7rxGpmxbNuLdIVIuKYtew6aA11AzhtLcf5DfCbSs/rxGpmxSJya7FWixOrmRXMOvehVp0Tq5kVz7pd9a86J1YzK5j8Ll5VixOrmRWLcFeAmVnu3GI1M8uTuwLMzPLXxl0BZmb5qZsroIY5sZpZwbgrwMwsfx4VYGaWM7dYzcxytO7TAladE6uZFY9brGZmeZJHBZiZ5c5dAWZmOfJ8rGZmefM4VjOz/LkrwMwsZ754ZWaWI7krwMwsf+4KMDPLl5xYzczyk63M4sRqZpYfpa2GObGaWcGINm188crMLFfuCjAzy5kTq5lZntzHamaWLyG3WM3M8ubEamaWM48KMDPLk/tYzczy564AM7Mc+eKVmVkTcGI1M8tbbedVJ1YzKxjV/qiA2o7ezFokSWW3Co/zLUlPS5oq6SZJ60kaKGmCpGmSbpbUIdXtmJ5PT/sHNDZ+J1YzK5S6i1frmlgl9QW+DuwaEdsBbYGjgR8Dl0fEIOAtYFR6ySjgrYj4GHB5qtcoTqxmVjyqYKtMO2B9Se2ATsAc4FPAn9L+64ER6fHw9Jy0/yA18iqaE2vBnDZ0WyZeOpwJl36O35yxHx3bZx/RBUftzOOXj2DST4Zz6sHbAPDJbXsx6zfHMP6SYYy/ZBhnH75DNUNvtb7xta+w7RZ92W+PnVbb94sxl9Frow7Mnz8PgL/cfScH7DWYT+2zK0P235MJj4xv7nCLTxV3BXSXNKlkG116mIiYDfwP8DJZQl0ATAbejoilqdosoG963Bd4Jb12aarfrTFvwRevCqRP106cevA27PbtO1i0ZBnXf2N/jth7IEL07bYBg8+8nQjovtF6K17zyLOvc+R/P1DFqO3o405k1OivcfopI1cqnz3rFf7+wDj69d9sRdl++3+Kg4cOQxJPT32S0Scdy/jJU5s75MKrsKE4LyJ2recYXclaoQOBt4E/AoesoWrUvaSefQ3iFmvBtGvbhvU7tKVtG9GpY1vmvPUBoz6zNT++9QkifcTz3llU3SBtJXvtsy9dunZdrfyC753FBd//4UpJYoPOnVc8X/j+wpofr9lU1EZltwp8GpgREW9ExBLgz8DeQJfUNQDQD3g1PZ4F9AdI+zcG3mxM/E6sBTLnrYWMGfs0/77yCKZf9UUWLFzCA0++yha9OnP4XgP4+w8O5dZzDmLL3huueM3ug3rwrx8P49ZzDmKbfl2qGL2Vuveeu+jdpy+f2H7H1fbdc9ft7LPLdhx/5HAuv/LqKkRXfDmNCngZ2FNSp9RXehDwb+BB4IhU5yTgjvT4zvSctP+BiChWi1XSMklTSrYBJft+Kmm2pDYlZSdL+nl63EbS9ZJ+o8xLkp4qOdaYpoq7mrps0IFDd+nP9mfcyqCv3sIGHdtx1Ce3oEP7tny4ZBn7n3c314+bxi9O3QeAJ2a8yban38reZ9/Fr+59lpu+fWCV34EBLFy4kCsuvYSzz7twjfuHDhvB+MlTue6mP/HjH1zUvMHVgEqSaiWJNSImkF2Eegx4iizf/Ro4GzhT0nSyPtRr0kuuAbql8jOBcxr7Hpqyj/WDiFitNz8l08+TdRLvBzy0yn4BVwHtgZEREemXeGBEzGvCeKvugO36MPON95j37ocA3DlxJnts1YNX5y/kjokzs7L/e5lffDVLrO9+sGTFa/82ZTaXjWpDtw07Mj+93qrjpRkv8PLMl/jUPln336uzZ/GZfffg3gfH07NX7xX19tpnX16a8SLz58+jW7fu1Qq3kPLqIomIC4FV/8K9COy+hrqLgCPzOG81ugIOBKYCvwSOWcP+n5L9FTkxIpY3Z2DVNmv+++z2sR6s36EtkCXa52YvYOykl9n/E32AbCTA9DnvANBz448uYu2yZXfaCCfVAtj2E9vz7xdnM2nqNCZNncamfftx3z8n0LNXb2a8MJ26b5dPTnmcJYsXs8kmjbrw3KLldYNAtTRli3V9SVPS4xkR8fn0+BjgJrJ+jR9Kap86lgGOBZ4BDigZDlHnQUnL0uPrI+LyVU+YhluMBlCn2vvHOmn6PG6f8BIP/2gYS5cv54mX3uTacc+zfoe2XHP6fpw2dFveX7SE03/1LwBG7DmAL396a5YuX86ixcsYOeYfVX4HrdMpI4/nXw//gzfnz2OnbQbynXMv4LgTR66x7tg7b+OPN91Iu/btWW+99fn1db8rfJKohgovThWWGtk3W/7A0nsR0XmVsg7AS8DWEfGupD8D10TE3ZJOBo4HtgGOiojxJa97iezuiYq7AtpuMjA6Dblond+HNZ8Xrj622iFYAw3Zf0+mPDY51yzYsfeg6Hdc+csoL142dHJ9w62qqbm7Ag4mG8LwVEqWn2Tl7oBngS8CN0v6RDPHZmYFIEAqvxVZcyfWY4AvR8SAiBhANnB3iKROdRUi4l/AqcDdkjZb82HMrOXKZ1RANTXbnVcpeX4WOKWuLCLel/QwMKy0bkSMldQDuFfSvqm4tI/1yYg4sTniNrPmV/C8WVaTJdZV+1cjYiGwyRrqHV7y9LqS8muBa9PTAflHaGZFVfQWaTmeK8DMCkWCtm2dWM3MclXjDVYnVjMrHncFmJnlqQaGU5XjxGpmhZKNY63tzOrEamYFU/xxquU4sZpZ4bSp8bkCnFjNrFjcx2pmli/3sZqZNYEaz6tOrGZWPG6xmpnlrMbzqhOrmRWL5FEBZmY58zhWM7Pc1XhedWI1s+Jxi9XMLE++QcDMLF8C2rRp7uX48uXEamaF4xarmVnO3MdqZpYn97GameVLHsdqZpa/Gs+rTqxmVjxtfUurmVl+pBZ88UrSRvW9MCLeyT8cMzOo8QZrvS3Wp4EgG69bp+55AJs1YVxm1oq12BZrRPRvzkDMzOrUeF6lovvGJB0t6dz0uJ+kXZo2LDNrrUQaclXmvyIrm1gl/Rw4EDghFS0ErmrKoMysFZNo26b8VmSVjArYOyIGS3ocICLelNShieMys1as1rsCKkmsSyS1IbtghaRuwPImjcrMWi0BbWo8s1bSx3olcCvQQ9LFwMPAj5s0KjNr1aTyW2XHURdJf5L0rKRnJO0laRNJ90maln52TXUlaYyk6ZKelDS4sfGXTawRcQNwPvA/wJvAkRHxh8ae0MysHElltwr9FLg3IrYBdgSeAc4BxkXEIGBceg5wCDAobaOBXzY2/kpnk20LLAEWN+A1ZmYNVklrtZK8mm5y2g+4BiAiFkfE28Bw4PpU7XpgRHo8HLghMo8CXST1acx7qGRUwHnATcCmQD/g95K+15iTmZlVoq1UdgO6S5pUso1e5TBbAG8A10p6XNL/StoA6BURcwDSz56pfl/glZLXz0plDVbJxavjgV0iYiGApB8Ak4EfNeaEZmblVPhVf15E7FrP/nbAYOCMiJgg6ad89LV/jaddQ1lUEsiqKvlaP5OVE3A74MXGnMzMrJxsVED5rQKzgFkRMSE9/xNZon297it++jm3pH7pHaf9gFcb8x7WmlglXS7pMrIbAp5OzeirgaeAtxtzMjOzsiq4cFVJizYiXgNekbR1KjoI+DdwJ3BSKjsJuCM9vhM4MY0O2BNYUNdl0FD1dQVMTT+fBu4uKX+0MScyM6tUjsNYzwB+l25qehEYSdagvEXSKOBl4MhU9x5gKDCdrEE5srEnrW8Slmsae1Azs3WR1+xWETEFWFM/7EFrqBvAaXmct+zFK0lbAj8AtgXWKwliqzwCMDMrJWp/BYFKLl5dB1xL9n4PAW4BfIOAmTUZVbAVWSWJtVNE/BUgIl6IiPPJZrsyM8udlM0VUG4rskrGsX6orMPjBUmnArP5aECtmVnuCp43y6oksX4L6Ax8nayvdWPgS00ZlJm1bi12aZY6JYNr3+Wjya7NzJqEKP5E1uXUt0rrbdRzO1dEHN4kEZlZ69aAaQGLqr4W68+bLYomsNPAboy/8aTyFa0wuu52erVDsAb68LlXyldqhBbbFRAR45ozEDOzOrU+N2klF6/MzJqNaMEtVjOzaqnxa1eVJ1ZJHSPiw6YMxsxMagW3tEraXdJTwLT0fEdJP2vyyMys1cppPtaqqaSPeAxwGDAfICKewLe0mlkTymuV1mqppCugTUTMXKUzeVkTxWNmrVy2gkDBM2cZlSTWVyTtDoSktmQTxz7ftGGZWWvWGoZbfZWsO2Az4HXg/lRmZtYkarzBWtFcAXOBo5shFjMzpBY8V0CdtIDganMGRMSqa3ibmeWixvNqRV0B95c8Xg/4PNA0NwibWavXKi5eRcTNpc8l/Ra4r8kiMrNWr8bzaqNuaR0IbJ53IGZmANTADQDlVNLH+hYf9bG2Ad4EzmnKoMysdVPhlwusX72JNa11tSPZOlcAy9Pa22ZmTUJAuxofyFpv+CmJ3hYRy9LmpGpmTU5S2a3IKvm7MFHS4CaPxMyMulEBtT0JS31rXrWLiKXAJ4GvSHoBeJ/sfUdEONmaWf5qYJKVcurrY50IDAZGNFMsZmZAyx7HKoCIeKGZYjEzW9EVUMvqS6w9JJ25tp0RcVkTxGNmrZ5o24JbrG2BzlDjA8rMrKZkiwlWO4p1U19inRMR/9lskZiZQYu/86rG35qZ1aqWfPHqoGaLwswsadFdARHxZnMGYmZWp8VPdG1m1pxE61jzysys+YjCzwVQjhOrmRVObafV2m9xm1kLU7c0S7mt4uNJbSU9Lmlsej5Q0gRJ0yTdLKlDKu+Ynk9P+wc09j04sZpZ4aiCrQG+ATxT8vzHwOURMQh4CxiVykcBb0XEx4DLU71GcWI1s4IRbdqU3yo6ktQPOBT43/RcwKeAP6Uq1/PRRFPD03PS/oPUyM5eJ1YzK5S6UQHlNqC7pEkl2+g1HO4K4LvA8vS8G/B2mhIVYBbQNz3uS1qBOu1fkOo3mC9emVnhVNhQnBcRu9ZzjMOAuRExWdIBdcVrqBoV7GsQJ1YzK5ycRgXsA3xO0lBgPWAjshZsl5KJ/PsBr6b6s4D+wCxJ7YCNyRZPbTB3BZhZsSifNa8i4nsR0S8iBgBHAw9ExHHAg8ARqdpJwB3p8Z3pOWn/A41d58+J1cwKpQF9rI11NnCmpOlkfajXpPJrgG6p/EzgnMaewF0BZlY4ec9uFREPAQ+lxy8Cu6+hziLgyDzO58RqZoVT43e0OrGaWbFkXQG1nVmdWM2scNxiNTPLlZBbrGZm+XKL1cwsRxItevlrM7OqqPG86sRqZsVT632svvOq4MZccTmDd/wEu+y0HScefwyLFi3ipRkz2HfvPdju44M4/tijWLx4cbXDbHWuuvA4Zo77EZP+eO6Ksq4bdWLsL0/nqTsuYOwvT6fLhuuv2PeT7x7B1DsuZOLN32OnbfoBsFmfroz/3Xd59A/nMPlP5/HlIz7Z7O+jiLKJrstvRebEWmCzZ8/mF1eOYfyjk5g8ZSrLli3jjzf/gfPOPZszvvEtpj4zja5dunLdb64pfzDL1W/vepThp125UtlZIz/DQxOfY/vh/8lDE5/jrJFDAPjsJ7dly816sN3wizn9v25izLlHAzDnjXc48OTL2PPoS9jvhEs5a+Rn6NNj42Z/L0WkCv4rMifWglu6dCkffPBB9nPhQnr36cPfH3yAw7+QzSFx3Akncdedt1c5ytZn/GMv8OaChSuVHXbADtx41wQAbrxrAsMO3CEr338Hfj92IgATn3qJjTdcn97dN2LJ0mUsXpJNC9qxQ/vcb+OsZVL5rcicWAusb9++fPNbZ7HVFpsxsH8fNtpoY3YevAsbd+lCu3ZZ93jffv149dXZVY7UAHp225DX5r0DwGvz3qHHJhsCsGnPLsx67a0V9Wa//jab9uwCQL9eXZh48/eY9pfv85Pr7mfOGwuaP/CCEdmogHJbkTVZYpW0TNIUSVMl3SWpSyofIOmDtK9uO7HkdTtLCkmfXeV47zVVrEX11ltvMfauO3hm2gxefPlV3l/4Pn+79y+r1Sv616LWbk05oG42ulmvv83uR/2I7YZfzPHDdqdnSsatWyUdAcX+N9+ULdYPImKniNiObLLY00r2vZD21W03lOw7Bng4/WzVHhh3PwMGDKRHjx60b9+eESMO59FH/sWCt99m6dLsK+TsWbPos+mmVY7UAObOf5fe3TcCoHf3jXjjzXeBrIXar3fXFfX69uqyWst0zhsL+PcLr7HP4C2bL+CiqqAboOAN1mbrCniEj9aVWau0cNcRwMnAEEnrNXFchda//2ZMnPgoCxcuJCJ48IFxbPPxbdnvgAP5863ZWmi/++31HDZseJUjNYC7//4Uxw/bA4Djh+3B2IeeXFF+7GHZLHW7bz+Ad977gNfmvUPfnl1Yr2N7ALpsuD577bQFz780tzrBF0zOq7Q2uyYfxyqpLXAQH00mC7ClpCklz8+IiH+SLaUwIyJekPQQMBT4cwPONRoYDdB/s83WNfSq232PPfj84Uew1+6DadeuHTvuuDOjvjKaQ4YeygnHHc3FF57PjjvtzMlfGlX+YJar6390MvvuMojuXToz/d7v8/2r7uF/rr2PG3/8JU4asRevzHmL476b/ZO/9+Gn+ewnP8HTd17IwkVLOOWiGwHYemBvLjnz8wSBEFfcMI6np79a32lbhWy4VdFTZ/3UyJUHyh9YWgY8BQwAJgNDImKZpAHA2NRFsOprrgSmRMTVkj4HnBARR6Z970VE50rPv8suu8b4CZPW/Y1Ys+m62+nVDsEa6MPnbmH5wrm5ZsGPb79zXHv7g2Xr7fWxrpPrW0ywmpq8jxXYHOjAyn2sq0kt2y8AF0h6CfgZcIgk9+abtTK+eFVGRCwAvg6cJal9PVU/DTwREf0jYkBEbA7cCoxo6hjNrFh88aoCEfE48ATZSomQ+lhLtq+TjQK4bZWX3gocmx53kjSrZDuzOWI3s+bni1drsWp/aEQMK3m6PhWIiDvJlqQlInwzg1lrUfTMWYZntzKzQslapLWdWZ1YzaxYamD2qnKcWM2seJxYzczyVPzhVOU4sZpZ4RR9OFU5TqxmVii1MJyqHCdWMyueGs+sTqxmVji1PgmLE6uZFU5tp1UnVjMrmhbQyerEamaF4+FWZmY5Eh5uZWaWuxrPq06sZlY8qvEmqxOrmRVOjedVJ1YzK54az6tOrGZWQDWeWT0rv5kVSt1E1+u6mKCk/pIelPSMpKclfSOVbyLpPknT0s+uqVySxkiaLulJSYMb+x6cWM2sWNJE1+W2CiwFvh0RHwf2BE6TtC1wDjAuIgYB49JzgEOAQWkbDfyysW/BidXMiieH1QQjYk5EPJYevws8A/QFhgPXp2rX89FK0MOBGyLzKNBFUp/GhO/EamYFU0lHgAC6S5pUso1e6xGlAcDOwASgV0TMgSz5Aj1Ttb7AKyUvm5XKGswXr8yscCocbjUvInYtfyx1Bm4FvhkR79QzRnZNO6KiSFbhFquZFUolvQCVDhqQ1J4sqf4uIv6cil+v+4qffs5N5bOA/iUv7we82pj34MRqZsWTQ2ZV1jS9BngmIi4r2XUncFJ6fBJwR0n5iWl0wJ7Agroug4ZyV4CZFU5OE13vA5wAPCVpSio7F7gEuEXSKOBl4Mi07x5gKDAdWAiMbOyJnVjNrHDySKsR8XA9hzpoDfUDOC2HUzuxmlnByHMFmJk1gdrOrE6sZlYonujazKwJ1HhedWI1s+Lx8tdmZnmr7bzqxGpmxVPjedWJ1cyKRR5uZWaWv0omsi4yJ1YzK57azqtOrGZWPBWuEFBYTqxmVjCVrWlVZE6sZlYoLeHOK8/HamaWM7dYzaxwar3F6sRqZoXjPlYzsxxJHhVgZpY/J1Yzs3y5K8DMLGe+eGVmlrMaz6tOrGZWPKrxJqsTq5kVSku480rZUtotj6Q3gJnVjqMJdAfmVTsIa5CW/JltHhE98jygpHvJfmflzIuIg/M8d15abGJtqSRNiohdqx2HVc6fWevjuQLMzHLmxGpmljMn1trz62oHYA3mz6yVcR+rmVnO3GI1M8uZE6uZWc6cWGucpG7VjsHMVubEWsMkDQGukNRVtX4PYCvhz6l1cGKtUSmpXgpcExFv4duTa0U3AEn+f68F84dbgyQdTJZUT4mIhyT1B86VVMltgFYFyvQEZkr6XEQsd3JtufzB1qY9gE4R8aikHsBtwNyIaKn3o9e8yMwFRgLXShpal1wlta12fJYvf32sIZL2AfaPiIslbSHpEbI/jr+KiKtL6vWPiFeqFqitVUTcImkx8AdJx0TE3XUtV0nDsioxtrpR2rpyi7UGlHxlHAJsDBARJwH/ALquklSPA8ZI2rDZA7XVSDpY0n9I2quuLCJuJ2u5/kHSYanlegpwFfBstWK1/LjFWhs2Bt4CFgErvjZGxNmSekh6MCIOlPQF4FvAiRHxbpVitZXtD5wKHCzpaeDnwIyIuDWNELhO0lhgd2BoREyvYqyWE7dYC07SQOBHkrYAXgc2TOXrA0TEl4AXJc0BziVLqv+uVry2mjuB+4EvAAuBo4HfStoiIv4EfBH4HHBsRDxRvTAtT26xFt96wFzgFKAHMCuVd5S0KF0UGSXpLOAeJ9Xqk7QN8GFEzIiIRyR1BL4ZEd+UdCxwDtBZ0izgp0DviFhczZgtX56EpQZI2g44GDgd2IysFbQz8CqwBHgXGBERS6oWpAEgaSjwH8AJdV/rJQ0CvgI8R/at4stkn93ewEMRMaNK4VoTcYu1gCQdQPbZ/CMiFkfEVElLgE7Ax4HrgKeADYCNyIZaOalWmaTPkiXViyJiuqTOQJAty7I5cBpwSET8I9V/PtyyaZHcYi0YSRsDdwMDgSuAZRFxWdq3JXAU0Af4bURMrFqgtplqY5cAAAWZSURBVBJJ2wNPAJ+OiAfSZ/Ur4MyIeFLSDmR/EI+IiBerGKo1A1+8KpiIWACMBRYD04Chkq6TNIKsr/VKshECX5S0nu89r66S3/9LZDdqfFHSALLJrf+akmqbiHgS+CdwoG8IaPmcWAtCUu+S/0l/AvwFeDciPg10AC4jG7e6f/r5w4hY5K+SVdcBIA1vOw7oDLwA3B4Rl6akulzSTmRdAvdGxLLqhWvNwYm1ACQdSnZBqnu6GUBkrdOd0zCrPckGlF8BHA48HhFvVitey6SJcP4g6SJJh0fEIrLRG78H9gJISXUUMAa4OiJmVy9iay7uY62yNKHKecAPIuJeSR0iYnGaWGUyWQvoi3W3OUrqFBELqxiyseJzuxi4AegJbAr8d0RMS3e9/YLswtXfyG4QODUiplYrXmteTqxVJGkTsq+Hh0fE7emCxwXAdyJirqTRwA4RcXpdwq1qwAas9LkNj4i7JPUDfgD8MiIeTXU6ADeT3Ya8m8cXty7uCqii9HV+GHBBumr8a7Kv+XNTlSeAgyRt5aRaHCWf2yWSNoqIWWQ3b1wi6QpJ3yYbCjcK+JiTauvjcaxVlmY3WgZMAc6NiCsktY2IZRExQdLvqx2jrS59bsuByZLuJbuIdSWwCdkNAB8nG2rlvvBWyF0BBSHpM8DPgD0iYoGkjhHxYbXjsvpJ+jRZP2qfiHg9lbUBNvH8uK2XuwIKIiLuI5uZaqKkTZxUa0NE3A8cCjwgqVcqW+6k2rq5K6BAIuIv6aLH/ZJ2JU08X+24rH4ln9tfJO0aEcurHZNVl7sCCkhS54h4r9pxWMP4c7M6TqxmZjlzH6uZWc6cWM3McubEamaWMydWM7OcObG2UpKWSZoiaaqkP0rqtA7HOiCtNIqkz0k6p566XSR9rRHnuCit61VR+Sp1rpN0RAPONUCSJ0yxRnNibb0+iIidImI7skm1Ty3dqUyD/31ExJ0RcUk9VboADU6sZrXEidUgm9n+Y6ml9oykXwCPAf0lDZH0iKTHUsu2M2TT5kl6VtLDZHPEkspPlvTz9LiXpNskPZG2vYFLgC1Ta/nSVO87kv5P0pOSLi451nmSnpN0P7B1uTch6SvpOE9IunWVVvinJf1T0vOSDkv120q6tOTcp6zrL9IMnFhbPUntgEPIFieELIHdEBE7A+8D55Ot4zQYmAScKWk94GqyGZ72BXqv5fBjgL9HxI7AYOBpsqWfX0it5e+kyaIHAbsDOwG7SNpP0i7A0WSr0R4O7FbB2/lzROyWzvcM2exSdQaQrb5wKHBVeg+jgAURsVs6/lckDazgPGb18i2trdf6kqakx/8EriGbrHlm3ZyiZCsXbAuMT6vGdAAeAbYBZkTENABJNwKj13COTwEnAqTlSBZI6rpKnSFpezw970yWaDcEbqub1FvSnRW8p+0k/RdZd0Nn4K8l+25Jt5pOk/Rieg9DgB1K+l83Tud+voJzma2VE2vr9UFE7FRakJLn+6VFwH0Rccwq9XYimx0/DwJ+FBG/WuUc32zEOa4DRkTEE5JOBg4o2bfqsSKd+4yIKE3AKFsM0KzR3BVg9XkU2EfSxyBbFkbSVsCzwMC04gHAMWt5/Tjgq+m1bSVtBLxL1hqt81fgSyV9t30l9SRbMPHzktZPS50MqyDeDYE5ktqTLexX6khJbVLMWwDPpXN/NdVH0laSNqjgPGb1covV1ioi3kgtv5skdUzF50fE88qWjblb0jzgYWC7NRziG8CvlS2mtwz4akQ8Iml8Gs70l9TP+nHgkdRifg84PiIek3Qz2QTgM8m6K8r5D2BCqv8UKyfw54C/A73I1p9aJOl/yfpeH1N28jeAEZX9dszWzpOwmJnlzF0BZmY5c2I1M8uZE6uZWc6cWM3McubEamaWMydWM7OcObGameXs/wGD/QAqTlfgWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "clf = MultinomialNB() \n",
    "clf.fit(count_train, y_train)\n",
    "pred = clf.predict(count_test)\n",
    "score = accuracy_score(y_test, pred)\n",
    "print(\"accuracy:   %0.3f\" % score)\n",
    "cm = confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And indeed, with absolutely no parameter tuning, your count vectorized training set count_train is visibly outperforming your TF-IDF vectors!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing Linear Models\n",
    "\n",
    "There are a lot of great write-ups about how linear models work well with TF-IDF vectorizers (take a look at word2vec for classification, SVM reference in scikit-learn text analysis, and many more).\n",
    "\n",
    "So you should use a SVM, right?\n",
    "\n",
    "Well, I recently watched Victor Lavrenko's lecture on text classification and he compares Passive Aggressive classifiers to linear SVMs for text classification. We'll test this approach (which has some significant speed benefits and permanent learning disadvantages) with the fake news dataset.\n",
    "\n",
    "**Drawing Confusion Matrix With Passive Agressive Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:   0.934\n",
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3debxVZb3H8c8XEJSYBUQGxQElMwc05zELh1SwnGey1K7aYJamlVq3q91upqZpesmxFNNKwinn0quQGE45gAOJokAqooBw8Hf/WM/BzZFz9j6Hdc5e+5zv29d6nb2f9ey1fvts/J1nP+tZz6OIwMzM8tOp2gGYmbU3TqxmZjlzYjUzy5kTq5lZzpxYzcxy5sRqZpYzJ1ZrlKQ1JP1Z0nxJv1+F4xwh6S95xlYtknaW9Hy147Bik8ex1j5JhwOnAiOBBcA04CcR8dAqHvco4BRgh4ioW+VAC05SACMiYka1Y7Ha5hZrjZN0KnAh8F/AWsA6wK+AMTkcfl3ghY6QVCshqUu1Y7AaERHeanQDegPvAQc1UacbWeJ9PW0XAt3Svt2AWcC3gTnAbGBc2ncusARYms5xHHAOcH3JsYcDAXRJz48FXiJrNb8MHFFS/lDJ63YA/g7MTz93KNn3APBj4OF0nL8A/Rt5b/Xxf7ck/rHAPsALwFvAmSX1twEeAd5JdS8BuqZ9f03v5f30fg8pOf7pwBvAdfVl6TUbpHOMSs8HA/OA3ar9b8NbdTe3WGvb9sDqwB+bqHMWsB2wBbA5WXL5fsn+QWQJeghZ8rxUUt+IOJusFTwhInpExPimApH0CeBiYO+I6EmWPKetpF4/4LZUd03gAuA2SWuWVDscGAcMBLoCpzVx6kFkv4MhwA+BK4Ejga2AnYEfSlo/1V0GfAvoT/a72wP4D4CI2CXV2Ty93wklx+9H1no/vvTEEfEiWdL9raTuwFXA1RHxQBPxWgfgxFrb1gTmRdNf1Y8AfhQRcyJiLllL9KiS/UvT/qURcTtZa23jFsbzIbCppDUiYnZEPLOSOl8ApkfEdRFRFxE3AM8B+5XUuSoiXoiIRcBNZH8UGrOUrD95KXAjWdK8KCIWpPM/A2wGEBFTI+LRdN5XgF8Du1bwns6OiA9SPCuIiCuB6cBkYG2yP2TWwTmx1rZ/A/3L9P0NBmaWPJ+ZypYfo0FiXgj0aG4gEfE+2dfnE4HZkm6TNLKCeOpjGlLy/I1mxPPviFiWHtcnvjdL9i+qf72kjSRNkvSGpHfJWuT9mzg2wNyIWFymzpXApsAvI+KDMnWtA3BirW2PAIvJ+hUb8zrZ19h666Sylngf6F7yfFDpzoi4KyI+T9Zye44s4ZSLpz6m11oYU3NcRhbXiIjoBZwJqMxrmhw2I6kHWb/1eOCc1NVhHZwTaw2LiPlk/YqXShorqbuk1STtLem/U7UbgO9LGiCpf6p/fQtPOQ3YRdI6knoD36vfIWktSfunvtYPyLoUlq3kGLcDG0k6XFIXSYcAmwCTWhhTc/QE3gXeS63przXY/yaw/sde1bSLgKkR8RWyvuPLVzlKq3lOrDUuIi4gG8P6fWAu8CpwMvCnVOU/gceAJ4GngMdTWUvOdTcwIR1rKismw05kowteJ7tSvivpwlCDY/wb2DfV/TfZFf19I2JeS2JqptPILowtIGtNT2iw/xzgGknvSDq43MEkjQH2Iuv+gOxzGCXpiNwitprkGwTMzHLmFquZWc6cWM3McubEambtlqTfSJoj6emSsn6S7pY0Pf3sm8ol6WJJMyQ9KWlUyWuOSfWnSzqm3HmdWM2sPbua7AJjqTOAeyNiBHBveg6wNzAibceTDc+rv1vwbGBbsjsXz65Pxo1pt5NKaLXuodX7VDsMa4YtNxpcvpIVysyZrzBv3rxyY4GbpXOvdSPqPnaT28fEorl3RUTDpLlinYi/ShreoHgM2ZwPANeQzU9xeiq/NrIr+o9K6iNp7VT37oh4C0DS3WTJ+obGztt+E+vqfeg26oRqh2HN8PC9Z1c7BGumHbfdOvdjRt0ium1cdrQbi6ddOlLSYyVFV0TEFRWcYq2ImA0QEbMlDUzlQ8iGK9ablcoaK29Uu02sZlarBKqol3JeROSZ2VfW8o4myhvlPlYzKxYBnTqX31ruzfQVn/RzTiqfBQwrqTeU7IaXxsob5cRqZsUjld9abiJQf2X/GODWkvKj0+iA7YD5qcvgLmC0pL7potXoVNYodwWYWcFU3BVQ/kjSDWQXn/pLmkV2df984CZJxwH/Ag5K1W8nmyR9BtmsauMAIuItST8mm5Qdsmk232rqvE6sZlY8q9YiXS4iDmtk1x4rqRvASY0c5zfAbyo9rxOrmRWLyK3FWi1OrGZWMKvch1p1TqxmVjyrdtW/6pxYzaxg8rt4VS1OrGZWLMJdAWZmuXOL1cwsT+4KMDPLXyd3BZiZ5ad+roAa5sRqZgXjrgAzs/x5VICZWc7cYjUzy9GqTwtYdU6sZlY8brGameVJHhVgZpY7dwWYmeXI87GameXN41jNzPLnrgAzs5z54pWZWY7krgAzs/y5K8DMLF9yYjUzy0+2MosTq5lZfpS2GubEamYFIzp18sUrM7NcuSvAzCxnTqxmZnlyH6uZWb6E3GI1M8ubE6uZWc48KsDMLE/uYzUzy5+7AszMcuSLV2ZmrcCJ1cwsb7WdV6ntS29m1v4oGxVQbqvoUNK3JD0j6WlJN0haXdJ6kiZLmi5pgqSuqW639HxG2j+8pW/BidXMCkdS2a2CYwwBvg5sHRGbAp2BQ4GfAr+IiBHA28Bx6SXHAW9HxIbAL1K9FnFiNbNCqb94taqJNekCrCGpC9AdmA18Frg57b8GGJsej0nPSfv3UAs7e51Yzax4VMEG/SU9VrIdX3qIiHgN+B/gX2QJdT4wFXgnIupStVnAkPR4CPBqem1dqr9mS8L3xauCOelL2zJu31FIcNWkx7nk5smcdeyufHnfUcx9ZyEAZ195L3dNnkG/Xmvwux8dxFYbD+H6O6fxrYvuqHL0BrDxhsPp2aMnnTt3pkuXLjw8+TGOPPwQpj//PADvzH+HPr37MHnqtCpHWlCqeFTAvIjYutHDSH3JWqHrAe8Avwf2XknV+OjMje5rFifWAtlkvQGM23cUO594JUvqljHxv4/kjkemA/DL3z/KhRMeWaH+4iV1/Gj8/Wyy3kA+td7AaoRsjbjznvvp37//8ufX/27C8senf+fb9O7duxph1Yychlt9Dng5IuamY/4B2AHoI6lLapUOBV5P9WcBw4BZqeugN/BWS07sroACGbnuAKb8cxaLPqhj2bLgb0/MZMwuIxutv3DxUv7vqVdZvKSu0TpWLBHBLTffxMGHHFbtUApNnVR2q8C/gO0kdU99pXsA/wTuBw5MdY4Bbk2PJ6bnpP33RUSLWqxOrAXyzMtz2GnzdenXaw3W6NaFvbbbkKEDs5bNiQdsw5TfnMjlp+9Pnx6rVzlSa4ok9tt7NDtssxXjr7xihX0PP/Q31hq4FhuOGFGl6GpDHhevImIy2UWox4GnyPLdFcDpwKmSZpD1oY5PLxkPrJnKTwXOaGn8rdYVIGkZ2ZupNzYiXkn7LiL7izAsIj5MZceSDYs4WVIn4CpgGdkQiJeBBek5wF8j4uutFXu1PD9zHj//3cNM+vlRvL9oCU/OeJO6ug+58tbHOO/avxIRnH3cZzn/pNGc+NOJ1Q7XGnHfgw8zePBg5syZw757fZ6NR45kp513AeCmG2/goEPdWm1KM6/6NykizgbOblD8ErDNSuouBg7K47yt2ce6KCK2aFiYkuYBZFffdgEeaLBfwOXAasC4iIj0S949Iua1YryFcM3t/+Ca2/8BwLlf/SyvzX2XOW+/v3z/byZN5Q/nHV6t8KwCgwcPBmDgwIHsP/YA/v73Key08y7U1dVx65/+wMOTp1Y5wuKr9Vtaq9EVsDvwNHAZsLI/3ReRNc+Prm/NdiQD+nQHYNjAXozZ+ZPcdM/TDOrXY/n+MTt/kn++PKda4VkZ77//PgsWLFj++J67/8KnPrUpAPfdew8bbTySoUOHVjPEmpDjONaqaM0W6xqS6seTvBwRB6THhwE3kHUY/5ek1SJiadp3OPAssFvJOLN696fuBYBrIuIXDU+YxrFlY9m61eZV1xt+fDD9enVnad0yvnnh7bzz3mLGnzWWzTYcRATMfOMdTvmfScvrP3fjN+j5iW507dKZ/XYayb6nXcdzM9t9w76w5rz5JoccmP1Tr1tWxyGHHs7oPfcC4PcTbvRFqwpVeHGqsNTCi17lDyy9FxE9GpR1BV4BNo6IBWn4w/iIuC31sR4JjAQOiYiHS173Cln/a8UZo1PPwdFt1Amr/kaszbx9b8OuMCu6HbfdmqlTH8s1C3YbNCKGHnFx2XovXbDP1KbGsVZTW3cF7EU2NuyplCx3YsXugOeAg4EJkj7VxrGZWQEIkMpvRdbWifUw4CsRMTwihpPdETFaUvf6ChHxf8CJwG2S1mnj+Mys6nKdK6Aq2uzOq5Q89wSWfz+PiPclPQTsV1o3IiZJGgDcKWnnVFzax/pkRBzdFnGbWdsreN4sq9USa8P+1YhYCPRbSb0vljy9uqT8KrKxrADD84/QzIqq6C3ScjxXgJkVigSdOzuxmpnlqsYbrE6sZlY87gowM8tTDQynKseJ1cwKJRvHWtuZ1YnVzAqm+ONUy3FiNbPC6VTjcwU4sZpZsbiP1cwsX+5jNTNrBTWeV51Yzax43GI1M8tZjedVJ1YzKxbJowLMzHLmcaxmZrmr8bzqxGpmxeMWq5lZnnyDgJlZvgR06tTWy/Hly4nVzArHLVYzs5y5j9XMLE/uYzUzy5c8jtXMLH81nledWM2seDr7llYzs/xI7fjilaReTb0wIt7NPxwzM6jxBmuTLdZngCAbr1uv/nkA67RiXGbWgbXbFmtEDGvLQMzM6tV4XqWi+8YkHSrpzPR4qKStWjcsM+uoRBpyVea/IiubWCVdAuwOHJWKFgKXt2ZQZtaBSXTuVH6r7FDqI+lmSc9JelbS9pL6Sbpb0vT0s2+qK0kXS5oh6UlJo1r6Fippse4QEScAiwEi4i2ga0tPaGZWjlR+q9BFwJ0RMRLYHHgWOAO4NyJGAPem5wB7AyPSdjxwWUvjrySxLpXUieyCFZLWBD5s6QnNzJoioJNUdit7nGxk0y7AeICIWBIR7wBjgGtStWuAsenxGODayDwK9JG0dkveQyWJ9VLgFmCApHOBh4CftuRkZmaVqLDF2l/SYyXb8Q0Osz4wF7hK0j8k/a+kTwBrRcRsgPRzYKo/BHi15PWzUlmzlb1BICKulTQV+FwqOiginm7JyczMKlHhcKt5EbF1E/u7AKOAUyJisqSL+Ohr/0pPu5KyqCSQhiqdTbYzsBRY0ozXmJk1WyWt1Qr7WGcBsyJicnp+M1mifbP+K376Oaekfukw06HA6y15D5WMCjgLuAEYnE70O0nfa8nJzMwq0Vkqu5UTEW8Ar0raOBXtAfwTmAgck8qOAW5NjycCR6fRAdsB8+u7DJqrkrkCjgS2ioiFAJJ+AkwFzmvJCc3MysnxzqtTgN9K6gq8BIwja1DeJOk44F/AQanu7cA+wAyyYaXjWnrSShLrzAb1uqQAzcxyl40KyOdYETENWFk/7B4rqRvASXmct6lJWH5B1nG7EHhG0l3p+WiykQFmZvlT+57ouv7K/zPAbSXlj7ZeOGZmtT9XQFOTsIxvy0DMzOq15xYrAJI2AH4CbAKsXl8eERu1Ylxm1kGJ2l9BoJIxqVcDV5G9372Bm4AbWzEmM+vgVMFWZJUk1u4RcRdARLwYEd8nm+3KzCx3Uj5zBVRTJcOtPlDW4fGipBOB1/jo3lozs9wVPG+WVUli/RbQA/g6WV9rb+DLrRmUmXVs7f7iVcl9tgv4aLJrM7NWISqfyLqomrpB4I80MbNLRHyxVSIys46teRNZF1JTLdZL2iyKVrDFRoN58K4fVDsMa4a+nzm52iFYM33w/L9a5bjttisgIu5ty0DMzOrV+tyklVy8MjNrM6Idt1jNzKqlxq9dVZ5YJXWLiA9aMxgzM6kD3NIqaRtJTwHT0/PNJf2y1SMzsw6rk8pvRVZJH/HFwL7AvwEi4gl8S6uZtaKc1ryqmkq6AjpFxMwGncnLWikeM+vgshUECp45y6gksb4qaRsgJHUmW0PmhdYNy8w6so4w3OprZN0B6wBvAvekMjOzVlHjDdaK5gqYAxzaBrGYmSG147kC6km6kpXMGRARx7dKRGbW4dV4Xq2oK+CekserAwcAr7ZOOGbW0XWIi1cRMaH0uaTrgLtbLSIz6/BqPK+26JbW9YB18w7EzAyAGrgBoJxK+ljf5qM+1k7AW8AZrRmUmXVsKvxygU1rMrGmta42J1vnCuDDiGh08mszs1UloEuND2RtMvyURP8YEcvS5qRqZq1OUtmtyCr5uzBF0qhWj8TMjPpRAbU9CUtTa151iYg6YCfgq5JeBN4ne98REU62Zpa/GphkpZym+linAKOAsW0Ui5kZ0L7HsQogIl5so1jMzJZ3BdSyphLrAEmnNrYzIi5ohXjMrMMTndtxi7Uz0ANqfECZmdWUbDHBakexappKrLMj4kdtFomZGbT7O69q/K2ZWa1qzxev9mizKMzMkvbQFdDoDQIR8VZbBmJmVq9zJ5XdKiWps6R/SJqUnq8nabKk6ZImSOqayrul5zPS/uEtjb/G78g1s/ZGZImp3NYM3wCeLXn+U+AXETECeBs4LpUfB7wdERsCv0j1WsSJ1cyKRfnNFSBpKPAF4H/TcwGfBW5OVa7ho5ugxqTnpP17qIWTEjixmlnhqIKtQhcC3wU+TM/XBN5Jt+sDzAKGpMdDSKujpP3zU/1mc2I1s0KpX5ql3Ab0l/RYybbCOnyS9gXmRMTUBodvKCrY1ywtWUHAzKxVVdginRcRWzexf0dgf0n7kK3X14usBdunZJKpocDrqf4sYBgwS1IXoDfZxP7N5harmRWM6NSp/FZORHwvIoZGxHDgUOC+iDgCuB84MFU7Brg1PZ6YnpP239fSOaidWM2sUFphVEBDpwOnSppB1oc6PpWPB9ZM5aeyCktQuSvAzAon7xUCIuIB4IH0+CVgm5XUWQwclMf5nFjNrHBq/MYrJ1YzKxjl32Jta06sZlYo9X2stcyJ1cwKpz3PbmVmVhU1nledWM2sWLKugNrOrE6sZlY4brGameVKyC1WM7N8ucVqZpYjiXa9/LWZWVXUeF51YjWz4qn1PtZav8Gh3bvk4gvZZtSn2XarzRh39OEsXryYV155md133p4tNt2YY488lCVLllQ7zA7n8rOPYOa95/HY789cXta3V3cmXXYyT936QyZddjJ9eq6xfN/Pv3sgT996NlMmfI8tRg4FYJ21+/Lwb7/LozeewdSbz+IrB+7U5u+jiLKJrstvRebEWmCvv/Yav/7VL3nw4SlMnvokHy5bxi2/v5GzzzqDk075BtOefp4+ffty7dXjyx/McnXdnx9lzEmXrlB22rjP88CU5/n0mB/xwJTnOW3caAD23GkTNlhnAJuOOZeT//MGLj7zUABmz32X3Y+9gO0OPZ9djvoZp437PGsP6N3m76WIVMF/RebEWnB1dXUsWrSIuro6Fi5ayFqD1ubBB+9n7BezeXoPO+JoJv351jJHsbw9/PiLvDV/4Qpl++62Gdf/eTIA1/95MvvtvllWvutm/G7SFACmPPUKvXuuwaD+vVhat4wlS7Oll7p1Xa3mb+PMk1R+KzIn1gIbPGQIp3zz23xqo+GMWG8IvXr1Zsstt6J37z506ZJ1jw8ZMpTZr79e5kjWFgau2ZM35r0LwBvz3mVAv54ADB7Yh1lvvL283mtvvsPggX0AGLpWH6ZM+B7T7/gxP7/6HmbPnd/2gReMyEYFlNuKrNUSq6RlkqZJelrSnyX1SeXDJS1K++q3o0tet6WkkLRng+O911qxFtXbb7/N7ZMm8tSzL/LCS7NY+P77/OUvd3ysXq1PsdberezjqV/xY9ab77DNIeex6ZhzOXK/bRiYknHHVklHQLH/zbdmi3VRRGwREZuSLch1Usm+F9O++u3akn2HAQ+lnx3aA/fdw7rDh9N/wABWW2019ht7AFMefYT589+hri77Cvnaa7MYtPbaVY7UAOb8ewGD+vcCYFD/Xsx9awGQtVCHDuq7vN6Qtfp8rGU6e+58/vniG+w4aoO2C7ioKugGKHpboq26Ah7ho7W7G6Ws6XUgcCwwWtLqrRxXoQ0dtg5/nzKZhQsXEhE8eP99bDxyE3bZZTf+9IebAbjht9fyhX3HVDlSA7jtwac4cr9tAThyv22Z9MCTy8sP3zdbCWSbTw/n3fcW8ca8dxkysA+rd1sNgD4912D7LdbnhVfmVCf4glEFW5G1+jhWSZ2BPfhowS6ADSRNK3l+SkT8jWy52pcj4kVJDwD7AH9oxrmOB44HGDZsnVUNveo+s822jDngS+y8/dZ06dKFzTbfgnHHfZU9996HcUcdzo/P/SGbb74FRx/75WqH2uFcc96x7LzVCPr36cGMO3/Mjy+/nf+56m6u/+mXOWbs9rw6+22O+G72T/7Oh55hz50+xTMTz2bh4qWccM71AGy83iDOP/UAgkCIC6+9l2dmuL88G25V9NTZNLVwddfyB5aWAU8Bw4GpwOiIWCZpODApdRE0fM2lwLSIuFLS/sBREXFQ2vdeRPSo9Pyjtto6Hnx4yqq/EWszA7f/erVDsGb64Pmb+HDhnFyz4Cc/vWVc9af7y9bbfsO+UyNi6zzPnZdW72MF1gW6smIf68eklu2XgB9KegX4JbC3JPfmm3UwvnhVRkTMB74OnCZptSaqfg54IiKGRcTwiFgXuAUY29oxmlmx+OJVBSLiH8ATwKGpaIMGw62+TjYK4I8NXnoLcHh63F3SrJLt1LaI3czani9eNaJhf2hE7FfydA0qEBETgYnpsW9mMOsoip45y/DsVmZWKFmLtLYzqxOrmRVLDcxeVY4Tq5kVjxOrmVmeij+cqhwnVjMrnKIPpyrHidXMCqUWhlOV48RqZsVT45nVidXMCqfWJ2FxYjWzwqnttOrEamZF0w46WZ1YzaxwPNzKzCxHovaHW3liEzMrnDxmt5I0TNL9kp6V9Iykb6TyfpLuljQ9/eybyiXpYkkzJD0paVRL43diNbPCkVR2q0Ad8O2I+CSwHXCSpE2AM4B7I2IEcG96DrA3MCJtxwOXtTR+J1YzK5w8JrqOiNkR8Xh6vAB4lmxR0zHANanaNXw0mf4Y4NrIPAr0kdSiJZCdWM2scPKe6DqttbclMBlYKyJmQ5Z8gYGp2hDg1ZKXzaKC1aVXxhevzKx4Ksuc/SU9VvL8ioi44mOHknqQrUbyzYh4t4luhJXtaNFqq06sZlYozZjoel65VVrTOnu3AL+NiD+k4jclrR0Rs9NX/TmpfBYwrOTlQ4EWrUfurgAzK5Y00XW5rexhsqbpeODZiLigZNdE4Jj0+Bjg1pLyo9PogO2A+fVdBs3lFquZFU8+41h3BI4CnpI0LZWdCZwP3CTpOOBfwEFp3+3APsAMYCEwrqUndmI1s4LJZ6LriHiIxlP0HiupH8BJq3xinFjNrIBq/c4rJ1YzK5R2MAeLE6uZFVCNZ1YnVjMrHE90bWaWs9pOq06sZlY0Fc4FUGROrGZWQLWdWZ1YzaxQ2sNE106sZlY4NZ5XnVjNrHg8KsDMLG+1nVedWM2seGo8rzqxmlmxVLr0SpE5sZpZ4eQxu1U1ObGaWfHUdl51YjWz4qlkhYAic2I1s4LJZ6LranJiNbNCaQ93XnkxQTOznLnFamaFU+stVidWMysc97GameVI8qgAM7P8ObGameXLXQFmZjnzxSszs5zVeF51YjWz4lGNN1mdWM2sUNrDnVeKiGrH0CokzQVmVjuOVtAfmFftIKxZ2vNntm5EDMjzgJLuJPudlTMvIvbK89x5abeJtb2S9FhEbF3tOKxy/sw6Hs8VYGaWMydWM7OcObHWniuqHYA1mz+zDsZ9rGZmOXOL1cwsZ06sZmY5c2KtcZLWrHYMZrYiJ9YaJmk0cKGkvqr1ewA7CH9OHYMTa41KSfVnwPiIeBvfnlwr1gSQ5P/32jF/uDVI0l5kSfWEiHhA0jDgTEmV3AZoVaDMQGCmpP0j4kMn1/bLH2xt2hboHhGPShoA/BGYExHt9X70mheZOcA44CpJ+9QnV0mdqx2f5ctfH2uIpB2BXSPiXEnrS3qE7I/jryPiypJ6wyLi1aoFao2KiJskLQFulHRYRNxW33KVtF9WJSZVN0pbVW6x1oCSr4yjgd4AEXEM8Fegb4OkegRwsaSebR6ofYykvST9QNL29WUR8SeyluuNkvZNLdcTgMuB56oVq+XHLdba0Bt4G1gMLP/aGBGnSxog6f6I2F3Sl4BvAUdHxIIqxWor2hU4EdhL0jPAJcDLEXFLGiFwtaRJwDbAPhExo4qxWk7cYi04SesB50laH3gT6JnK1wCIiC8DL0maDZxJllT/Wa147WMmAvcAXwIWAocC10laPyJuBg4G9gcOj4gnqhem5ckt1uJbHZgDnAAMAGal8m6SFqeLIsdJOg243Um1+iSNBD6IiJcj4hFJ3YBvRsQ3JR0OnAH0kDQLuAgYFBFLqhmz5cuTsNQASZsCewEnA+uQtYK2BF4HlgILgLERsbRqQRoAkvYBfgAcVf+1XtII4KvA82TfKr5C9tntADwQES9XKVxrJW6xFpCk3cg+m79GxJKIeFrSUqA78EngauAp4BNAL7KhVk6qVSZpT7Kkek5EzJDUAwiyZVnWBU4C9o6Iv6b6L4RbNu2SW6wFI6k3cBuwHnAhsCwiLkj7NgAOAdYGrouIKVUL1FYg6dPAE8DnIuK+9Fn9Gjg1Ip6UtBnZH8QDI+KlKoZqbcAXrwomIuYDk4AlwHRgH0lXSxpL1td6KdkIgYMlre57z6ur5Pf/CtmNGgdLGk42ufVdKal2iogngb8Bu/uGgPbPibUgJA0q+Z/058AdwIKI+BzQFbiAbNzqrunnf0XEYn+VrLquAGl42xFAD+BF4E8R8bOUVD+UtAVZl8CdEbGset1ziUkAAAT3SURBVOFaW3BiLQBJXyC7INU/3QwgstbplmmY1XZkA8ovBL4I/CMi3qpWvJZJE+HcKOkcSV+MiMVkozd+B2wPkJLqccDFwJUR8Vr1Ira24j7WKksTqpwF/CQi7pTUNSKWpIlVppK1gA6uv81RUveIWFjFkI3ln9u5wLXAQGAw8N8RMT3d9fYrsgtXfyG7QeDEiHi6WvFa23JirSJJ/ci+Hn4xIv6ULnj8EPhORMyRdDywWUScXJ9wqxqwASt8bmMi4s+ShgI/AS6LiEdTna7ABLLbkD/j8cUdi7sCqih9nd8P+GG6anwF2df8OanKE8AekjZyUi2Oks/tfEm9ImIW2c0b50u6UNK3yYbCHQds6KTa8Xgca5Wl2Y2WAdOAMyPiQkmdI2JZREyW9Ltqx2gflz63D4Gpku4ku4h1KdCP7AaAT5INtXJfeAfkroCCkPR54JfAthExX1K3iPig2nFZ0yR9jqwfde2IeDOVdQL6eX7cjstdAQUREXeTzUw1RVI/J9XaEBH3AF8A7pO0Vir70Em1Y3NXQIFExB3posc9krYmTTxf7bisaSWf2x2Sto6ID6sdk1WXuwIKSFKPiHiv2nFY8/hzs3pOrGZmOXMfq5lZzpxYzcxy5sRqZpYzJ1Yzs5w5sXZQkpZJmibpaUm/l9R9FY61W1ppFEn7Szqjibp9JP1HC85xTlrXq6LyBnWulnRgM841XJInTLEWc2LtuBZFxBYRsSnZpNonlu5Uptn/PiJiYkSc30SVPkCzE6tZLXFiNchmtt8wtdSelfQr4HFgmKTRkh6R9Hhq2faAbNo8Sc9JeohsjlhS+bGSLkmP15L0R0lPpG0H4Hxgg9Ra/lmq9x1Jf5f0pKRzS451lqTnJd0DbFzuTUj6ajrOE5JuadAK/5ykv0l6QdK+qX5nST8rOfcJq/qLNAMn1g5PUhdgb7LFCSFLYNdGxJbA+8D3ydZxGgU8BpwqaXXgSrIZnnYGBjVy+IuBByNic2AU8AzZ0s8vptbyd9Jk0SOAbYAtgK0k7SJpK+BQstVovwh8poK384eI+Ew637Nks0vVG062+sIXgMvTezgOmB8Rn0nH/6qk9So4j1mTfEtrx7WGpGnp8d+A8WSTNc+sn1OUbOWCTYCH06oxXYFHgJHAyxExHUDS9cDxKznHZ4GjAdJyJPMl9W1QZ3Ta/pGe9yBLtD2BP9ZP6i1pYgXvaVNJ/0nW3dADuKtk303pVtPpkl5K72E0sFlJ/2vvdO4XKjiXWaOcWDuuRRGxRWlBSp7vlxYBd0fEYQ3qbUE2O34eBJwXEb9ucI5vtuAcVwNjI+IJSccCu5Xsa3isSOc+JSJKEzDKFgM0azF3BVhTHgV2lLQhZMvCSNoIeA5YL614AHBYI6+/F/haem1nSb2ABWSt0Xp3AV8u6bsdImkg2YKJB0haIy11sl8F8fYEZktajWxhv1IHSeqUYl4feD6d+2upPpI2kvSJCs5j1iS3WK1RETE3tfxukNQtFX8/Il5QtmzMbZLmAQ8Bm67kEN8ArlC2mN4y4GsR8Yikh9NwpjtSP+sngUdSi/k94MiIeFzSBLIJwGeSdVeU8wNgcqr/FCsm8OeBB4G1yNafWizpf8n6Xh9XdvK5wNjKfjtmjfMkLGZmOXNXgJlZzpxYzcxy5sRqZpYzJ1Yzs5w5sZqZ5cyJ1cwsZ06sZmY5+38kBN5FOqYqVwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "linear_clf = PassiveAggressiveClassifier(max_iter=50)\n",
    "\n",
    "linear_clf.fit(tfidf_train, y_train)\n",
    "pred = linear_clf.predict(tfidf_test)\n",
    "score = accuracy_score(y_test, pred)\n",
    "print(\"accuracy:   %0.3f\" % score)\n",
    "cm = confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix looks different and the model classifies our fake news a bit better. We can test if tuning the alpha value for a MultinomialNB creates comparable results. You can also use parameter tuning with grid search for a more exhaustive search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MultinomialNB(alpha=0.1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alpha: 0.00 Score: 0.88140\n",
      "Alpha: 0.10 Score: 0.89766\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/petramala/anaconda3/lib/python3.7/site-packages/sklearn/naive_bayes.py:485: UserWarning: alpha too small will result in numeric errors, setting alpha = 1.0e-10\n",
      "  'setting alpha = %.1e' % _ALPHA_MIN)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alpha: 0.20 Score: 0.89383\n",
      "Alpha: 0.30 Score: 0.89000\n",
      "Alpha: 0.40 Score: 0.88570\n",
      "Alpha: 0.50 Score: 0.88427\n",
      "Alpha: 0.60 Score: 0.87470\n",
      "Alpha: 0.70 Score: 0.87040\n",
      "Alpha: 0.80 Score: 0.86609\n",
      "Alpha: 0.90 Score: 0.85892\n"
     ]
    }
   ],
   "source": [
    "last_score = 0\n",
    "for alpha in np.arange(0,1,.1):\n",
    "    nb_classifier = MultinomialNB(alpha=alpha)\n",
    "    nb_classifier.fit(tfidf_train, y_train)\n",
    "    pred = nb_classifier.predict(tfidf_test)\n",
    "    score = accuracy_score(y_test, pred)\n",
    "    if score > last_score:\n",
    "        clf = nb_classifier\n",
    "    print(\"Alpha: {:.2f} Score: {:.5f}\".format(alpha, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not quite... At this point, it might be interesting to perform parameter tuning across all of the classifiers, or take a look at some other scikit-learn Bayesian classifiers. You could also test with a Support Vector Machine (SVM) to see if that outperforms the Passive Aggressive classifier.\n",
    "\n",
    "But I am a bit more curious about what the Passive Aggressive model actually has learned. So let's move onto introspection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introspecting models\n",
    "\n",
    "So fake news is solved, right? We achieved 93% accuracy on my dataset so let's all close up shop and go home.\n",
    "\n",
    "Not quite, of course. I am wary at best of these results given how much noise we saw in the features. There is a great write-up on StackOverflow with this incredibly useful function for finding vectors that most affect labels. It only works for binary classificaiton (classifiers with 2 classes), but that's good news for you, since you only have FAKE or REAL labels.\n",
    "\n",
    "Using your best performing classifier with your TF-IDF vector dataset (`tfidf_vectorizer`) and Passive Aggressive classifier (`linear_clf`), inspect the top 30 vectors for fake and real news:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FAKE -4.970871305842294 2016\n",
      "FAKE -4.158056665411209 october\n",
      "FAKE -4.10132333509434 hillary\n",
      "FAKE -3.203092643866443 share\n",
      "FAKE -3.0037703441504586 article\n",
      "FAKE -2.967123407154177 november\n",
      "FAKE -2.6189790669116313 oct\n",
      "FAKE -2.574531254941327 print\n",
      "FAKE -2.4352932878866342 email\n",
      "FAKE -2.4083421241436156 advertisement\n",
      "FAKE -2.3893397455445204 source\n",
      "FAKE -2.2689707230229135 mosul\n",
      "FAKE -2.088464394815847 war\n",
      "FAKE -2.0870875430058877 wikileaks\n",
      "FAKE -2.0795967342065915 establishment\n",
      "FAKE -1.978347105373653 corporate\n",
      "FAKE -1.9769696120411098 com\n",
      "FAKE -1.973445179332854 podesta\n",
      "FAKE -1.9542729616747192 election\n",
      "FAKE -1.8992365072732829 nov\n",
      "FAKE -1.8881411391076262 pipeline\n",
      "FAKE -1.8094255989742816 brexit\n",
      "FAKE -1.7197909311862556 ayotte\n",
      "FAKE -1.6962221480860573 26\n",
      "FAKE -1.6927925548700604 donald\n",
      "FAKE -1.690429866897319 snip\n",
      "FAKE -1.672278306841482 photo\n",
      "FAKE -1.6433962889785454 uk\n",
      "FAKE -1.6365865595323195 entire\n",
      "FAKE -1.615784841795114 jewish\n",
      "\n",
      "REAL 4.8751577389747 said\n",
      "REAL 2.613636036759295 says\n",
      "REAL 2.505012066805682 tuesday\n",
      "REAL 2.4690590946559876 gop\n",
      "REAL 2.368283463007904 marriage\n",
      "REAL 2.3068897785133435 islamic\n",
      "REAL 2.2550392505475405 monday\n",
      "REAL 2.22306595788944 friday\n",
      "REAL 2.2099059586160217 cruz\n",
      "REAL 2.1944920677580924 conservative\n",
      "REAL 2.1718656698590983 sunday\n",
      "REAL 2.125401710092977 attacks\n",
      "REAL 2.098760765527483 jobs\n",
      "REAL 2.091196362660998 debate\n",
      "REAL 2.08253600830322 rush\n",
      "REAL 2.025410713474804 sen\n",
      "REAL 2.015224771338318 conservatives\n",
      "REAL 1.940492329920454 fox\n",
      "REAL 1.932542021445429 candidates\n",
      "REAL 1.8714272648084431 abortion\n",
      "REAL 1.85171516892976 continue\n",
      "REAL 1.847453935576769 presumptive\n",
      "REAL 1.830228254154281 attack\n",
      "REAL 1.822780984543137 march\n",
      "REAL 1.7905365801622997 recounts\n",
      "REAL 1.7380836058243316 convention\n",
      "REAL 1.7326016984200479 hispanic\n",
      "REAL 1.727323671729206 paris\n",
      "REAL 1.6760916474238805 deal\n",
      "REAL 1.6501703794881477 security\n"
     ]
    }
   ],
   "source": [
    "def most_informative_feature_for_binary_classification(vectorizer, classifier, n=100):\n",
    "    \"\"\"\n",
    "    See: https://stackoverflow.com/a/26980472\n",
    "    \n",
    "    Identify most important features if given a vectorizer and binary classifier. Set n to the number\n",
    "    of weighted features you would like to show. (Note: current implementation merely prints and does not \n",
    "    return top classes.)\n",
    "    \"\"\"\n",
    "\n",
    "    class_labels = classifier.classes_\n",
    "    feature_names = vectorizer.get_feature_names()\n",
    "    topn_class1 = sorted(zip(classifier.coef_[0], feature_names))[:n]\n",
    "    topn_class2 = sorted(zip(classifier.coef_[0], feature_names))[-n:]\n",
    "\n",
    "    for coef, feat in topn_class1:\n",
    "        print(class_labels[0], coef, feat)\n",
    "\n",
    "    print()\n",
    "\n",
    "    for coef, feat in reversed(topn_class2):\n",
    "        print(class_labels[1], coef, feat)\n",
    "\n",
    "\n",
    "most_informative_feature_for_binary_classification(tfidf_vectorizer, linear_clf, n=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do this in a pretty obvious way with only a few lines of Python, by zipping your coefficients to your features and taking a look at the top and bottom of your list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = tfidf_vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(-6.257361214701583, 'trump'),\n",
       " (-6.494453094312678, 'said'),\n",
       " (-6.6539784739838845, 'clinton'),\n",
       " (-7.037944662867073, 'obama'),\n",
       " (-7.146539983381228, 'sanders'),\n",
       " (-7.215376008647511, 'president'),\n",
       " (-7.266562805741618, 'campaign'),\n",
       " (-7.2875931446681514, 'republican'),\n",
       " (-7.341118458599064, 'state'),\n",
       " (-7.341357110247905, 'cruz'),\n",
       " (-7.378312441985425, 'party'),\n",
       " (-7.44688067245789, 'new'),\n",
       " (-7.476288801154588, 'people'),\n",
       " (-7.547225599514773, 'percent'),\n",
       " (-7.5553074094582335, 'bush'),\n",
       " (-7.580150633909893, 'republicans'),\n",
       " (-7.5855405012652435, 'house'),\n",
       " (-7.634478172520314, 'voters'),\n",
       " (-7.648482443695299, 'rubio'),\n",
       " (-7.6734836186463795, 'states')]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Most real\n",
    "sorted(zip(clf.coef_[0], feature_names), reverse=True)[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(-11.349866225220305, '0000'),\n",
       " (-11.349866225220305, '000035'),\n",
       " (-11.349866225220305, '0001'),\n",
       " (-11.349866225220305, '0001pt'),\n",
       " (-11.349866225220305, '000km'),\n",
       " (-11.349866225220305, '0011'),\n",
       " (-11.349866225220305, '006s'),\n",
       " (-11.349866225220305, '007'),\n",
       " (-11.349866225220305, '007s'),\n",
       " (-11.349866225220305, '008s'),\n",
       " (-11.349866225220305, '0099'),\n",
       " (-11.349866225220305, '00am'),\n",
       " (-11.349866225220305, '00p'),\n",
       " (-11.349866225220305, '00pm'),\n",
       " (-11.349866225220305, '014'),\n",
       " (-11.349866225220305, '015'),\n",
       " (-11.349866225220305, '018'),\n",
       " (-11.349866225220305, '01am'),\n",
       " (-11.349866225220305, '020'),\n",
       " (-11.349866225220305, '023')]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Most fake\n",
    "\n",
    "sorted(zip(clf.coef_[0], feature_names))[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, clearly there are certain words which might show political intent and source in the top fake features (such as the words corporate and establishment).\n",
    "\n",
    "Also, the real news data uses forms of the verb \"to say\" more often, likely because in newspapers and most journalistic publications sources are quoted directly (\"German Chancellor Angela Merkel said...\").\n",
    "\n",
    "To extract the full list from your current classifier and take a look at each token (or easily compare tokens from classifier to classifier), you can easily export it like so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_with_weights = sorted(list(zip(feature_names, clf.coef_[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermezzo: HashingVectorizer\n",
    "\n",
    "Another vectorizer used sometimes for text classification is a HashingVectorizer. `HashingVectorizers` require less memory and are faster (because they are sparse and use hashes rather than tokens) but are more difficult to introspect. You can read a bit more about the pros and cons of using `HashingVectorizer` in the scikit-learn documentation if you are interested.\n",
    "\n",
    "You can give it a try and compare its results versus the other vectorizers. It performs fairly well, with better results than the TF-IDF vectorizer using `MultinomialNB` (this is somewhat expected due to the same reasons `CountVectorizers` perform better), but not as well as the TF-IDF vectorizer with Passive Aggressive linear algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "hash_vectorizer = HashingVectorizer(stop_words='english', alternate_sign=False)\n",
    "hash_train = hash_vectorizer.fit_transform(X_train)\n",
    "hash_test = hash_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MultinomialNB(alpha=.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:   0.902\n",
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd5wV1d3H8c8XEBQRAQELYEDFaEIsiL3EFqNGAzGa2EuImERN8fFJjBpLqik2otFojC2JYooNu6iJEtSIwfZYaBYQBUSxAFL8PX/MWbwssPfuMrt37u737Wtee+fMuTO/2bv+OPfMmTOKCMzMLD/tqh2AmVlr48RqZpYzJ1Yzs5w5sZqZ5cyJ1cwsZ06sZmY5c2K1lZK0hqTbJc2V9NdV2M8Rku7NM7ZqkbSrpBerHYcVmzyOtfZJOhw4BdgMeA+YAPwsIh5Zxf0eBZwM7BQRi1c50IKTFMDAiJhU7VistrnFWuMknQJcBPwcWBfYEPgdMDSH3X8CeKktJNVKSOpQ7RisRkSElxpdgLWB94FDGqjTiSzxvp6Wi4BOadvuwDTgf4CZwAzguLTtXGAhsCgdYzhwDvCnkn33BwLokNaPBaaQtZqnAkeUlD9S8r6dgP8Ac9PPnUq2PQT8BBib9nMv0HMl51YX//dL4h8G7A+8BMwBTi+pvx0wDngn1b0E6Ji2/SudywfpfL9asv8fAG8A19eVpfdsnI4xOK1vAMwGdq/234aX6i5usda2HYHVgZsbqHMGsAOwFbAlWXI5s2T7emQJug9Z8rxUUveIOJusFTwqIrpExFUNBSJpTWAksF9ErEWWPCesoF4P4I5Udx3gAuAOSeuUVDscOA7oDXQETm3g0OuR/Q76AGcBVwJHAtsAuwJnSdoo1V0CfA/oSfa72wv4FkBE7JbqbJnOd1TJ/nuQtd5HlB44IiaTJd0/S+oMXA1cExEPNRCvtQFOrLVtHWB2NPxV/QjgxxExMyJmkbVEjyrZvihtXxQRd5K11j7ZxHg+AgZJWiMiZkTEcyuo8wVgYkRcHxGLI+IG4AXgwJI6V0fESxExH7iJ7B+FlVlE1p+8CLiRLGleHBHvpeM/B2wBEBHjI+LRdNyXgd8Dn63gnM6OiA9TPMuIiCuBicBjwPpk/5BZG+fEWtveAnqW6fvbAHilZP2VVLZ0H/US8zygS2MDiYgPyL4+fwOYIekOSZtVEE9dTH1K1t9oRDxvRcSS9Lou8b1Zsn1+3fslbSpptKQ3JL1L1iLv2cC+AWZFxIIyda4EBgG/jYgPy9S1NsCJtbaNAxaQ9SuuzOtkX2PrbJjKmuIDoHPJ+nqlGyPinoj4HFnL7QWyhFMunrqYpjcxpsa4jCyugRHRFTgdUJn3NDhsRlIXsn7rq4BzUleHtXFOrDUsIuaS9SteKmmYpM6SVpO0n6RfpWo3AGdK6iWpZ6r/pyYecgKwm6QNJa0N/LBug6R1JX0x9bV+SNalsGQF+7gT2FTS4ZI6SPoq8ClgdBNjaoy1gHeB91Nr+pv1tr8JbLTcuxp2MTA+Ir5O1nd8+SpHaTXPibXGRcQFZGNYzwRmAa8BJwG3pCo/BZ4AngaeAZ5MZU051n3AqLSv8SybDNuRjS54nexK+WdJF4bq7eMt4IBU9y2yK/oHRMTspsTUSKeSXRh7j6w1Pare9nOAayW9I+kr5XYmaSiwL1n3B2Sfw2BJR+QWsdUk3yBgZpYzt1jNzHLmxGpmljMnVjNrtST9UdJMSc+WlPWQdJ+kieln91QuSSMlTZL0tKTBJe85JtWfKOmYcsd1YjWz1uwasguMpU4DxkTEQGBMWgfYDxiYlhFkw/Pq7hY8G9ie7M7Fs+uS8cq02kkl1GmtUOd1yle0wthigD+vWvPaq6/w1uzZ5cYCN0r7rp+IWLzcTW7Lifmz7omI+klz2ToR/5LUv17xULI5HwCuJZuf4gep/LrIrug/KqmbpPVT3fsiYg6ApPvIkvUNKztu602sndeh054/qnYY1ggPXFf2G5YVzJ67bp/7PmPxfDp9suxoNxZMuHQzSU+UFF0REVdUcIh1I2IGQETMkNQ7lfchG65YZ1oqW1n5SrXaxGpmtUqginopZ0fEkHwPvJxooHyl3MdqZsUioF378kvTvZm+4pN+zkzl04B+JfX6kt3wsrLylXJiNbPikcovTXcbUNfvdAxwa0n50Wl0wA7A3NRlcA+wj6Tu6aLVPqlspdwVYGYFU3FXQPk9STeQXXzqKWka2dX984CbJA0HXgUOSdXvJJskfRLZrGrHAUTEHEk/IZuUHbJpNuc0dFwnVjMrnlVrkS4VEYetZNNeK6gbwIkr2c8fgT9WelwnVjMrFpFbi7VanFjNrGBWuQ+16pxYzax4Vu2qf9U5sZpZweR38apanFjNrFiEuwLMzHLnFquZWZ7cFWBmlr927gowM8tP3VwBNcyJ1cwKxl0BZmb586gAM7OcucVqZpajVZ8WsOqcWM2seNxiNTPLkzwqwMwsd+4KMDPLkedjNTPLm8exmpnlz10BZmY588UrM7McyV0BZmb5c1eAmVm+5MRqZpaf7MksTqxmZvlRWmqYE6uZFYxo184Xr8zMcuWuADOznDmxmpnlyX2sZmb5EnKL1cwsb06sZmY586gAM7M8uY/VzCx/7gowM8uRL16ZmTUDJ1Yzs7zVdl51YjWzglHtjwqo7ejNrFWSVHapcD/fk/ScpGcl3SBpdUkDJD0maaKkUZI6prqd0vqktL1/U+N3YjWzQqm7eLWqiVVSH+DbwJCIGAS0Bw4FfglcGBEDgbeB4ektw4G3I2IT4MJUr0mcWM2seFTBUpkOwBqSOgCdgRnAnsDf0vZrgWHp9dC0Ttq+l5p4Fc19rAV08gGf5ti9P0kEPPfqHEZc8jA7btabnx+9He0kPliwiOMv+RdT3niPr++zGSfsuzlLPgo+WLCIEy8fywvT3qn2KbQpJ3/z69x715307NWbsf+ZAMDZZ/yAu++8g44dV6P/gI255PI/sHa3brz6ysvsuM1n2GTgpgAM2XZ7zh/5u2qGXzyqeFRAT0lPlKxfERFX1K1ExHRJvwFeBeYD9wLjgXciYnGqNg3ok173AV5L710saS6wDjC7safgFmvBbNCjM9/a/9Ps/P1bGfK9f9C+nThkl40YOWJnjrvoIXY49RZGPTyF0w7eCoBRD09m21NuZodTb+GCW57hl8duX+UzaHsOO+IYbrpl9DJlu++5N2P/M4GHH/svGw8cyIXnf/ytsv+AjfnnuPH8c9x4J9WVqLArYHZEDClZrqi3j+5krdABwAbAmsB+Kzhc1L2lgW2N4sRaQB3aizU6tqd9O7FGxw7MmDOPiKBr544AdO28GjPmzAPgvfmLlr5vzdU7ENGkvwNbBTvtsivdu/dYpmyPvT5Hhw7ZF8Ih227PjOnTqhFazVI7lV0qsDcwNSJmRcQi4B/ATkC31DUA0Bd4Pb2eBvQDSNvXBuY0JX53BRTM63PmcdFtz/LS5Ycyf+Fixjw1nTFPTedblz3CzWfsw4KFS3h33kI++8Pbl77nhH0359sHDqJjh3bse85dVYzeVuQv11/DsC8fsnT91VemsvtOQ1hrra6cftaP2XHnXaoYXTHldIPAq8AOkjqTdQXsBTwBPAgcDNwIHAPcmurfltbHpe0PRBNbKs3WYpW0RNKEkqV/ybaLJU2X1K6k7FhJl6TX7SRdK+mPyrws6ZmSfY1srrirrduaHTlg2w3Z/Fs3sdHxN7Dm6qtx6G4bc/IBg/jSz+5lkxE3cv2DE5f5yv/7u5/n0yf+lTOv/w+nfXmrKkZv9Z3/q1/Qvn0HDvnq4QCsu976PPX8FB769xP85LxfM+JrR/Huu+9WOcpiqaQboJLEGxGPkV2EehJ4hizfXQH8ADhF0iSyPtSr0luuAtZJ5acApzX1HJqzxTo/Ipb7vzwl0y+RdRLvBjxUb7uAy4HVgOMiItIvcY+IaHQncq3Zc4sNeHnme8x+dwEAtzz6Mjtuti6f6d+D/0ycBcDfxk7h1jM/v9x7bxo7hYtH7Nyi8drK3fDn67j37ju4efS9SxNBp06d6NSpEwBbbb0NAwZsxORJL7H14CHVDLVw8rqlNSLOBs6uVzwF2G4FdRcAh9Qvb4pq9LHuATwLXAYctoLtF5P9K3J0RHzUkoEVwWuzP2C7TXuzRsf2AOzxmQ144bV36Nq5I5us3xWAPbfsw4vTsyv/G6cygP226cekGXNbPmhbzpj77mHkBb/hz6NupnPnzkvLZ8+axZIlSwB4eeoUJk+eRP/+G1UrzMLK6waBamnOFusakiak11Mj4kvp9WHADWT9Gj+XtFrqWAY4HHge2L1kOESdByUtSa+vjYgL6x9Q0ghgRHb0HvU314T/TJzFzeOmMu43w1i8JHhq6ltcdd8LTH/rA2743734KIJ33l/ICb97GIBv7vcp9thiAxYt/oh3PviQ4y/5V5XPoO05/tgjGfvwP3nrrdkM2rQ/p51xFhed/ys+/PBDvvzFfYGPh1X9e+zDnPfTc+nQoT3t27fn/IsvpXuP2vxbbU4VXpwqLDXXVWRJ70dEl3plHYGXgU9GxHuS/gFcFRF3SDoWOBLYDPhqRIwted/LZHdPVNwV0K57/+i0549W/USsxUy/7phqh2CNtOeu2zPhyfG5ZsFO6w2MvkeUv4wy5YL9x0dEIftQWrorYF+yIQzPpGS5C8t2B7wAfAUYJenTLRybmRWAAKn8UmQtnVgPA74eEf0joj/ZwN190nAIACLi38A3gDskbdjC8ZlZ1eUzKqCaWmwca0qenwdOqCuLiA8kPQIcWFo3IkZL6gXcLWnXVFzax/p0RBzdEnGbWcsreN4sq9kSa/3+1YiYByzXSx8RB5WsXlNSfjVwdVrtn3+EZlZURW+RluM7r8ysUCRo396J1cwsVzXeYHViNbPicVeAmVmeamA4VTlOrGZWKNk41trOrE6sZlYwxR+nWo4Tq5kVTrsanyvAidXMisV9rGZm+XIfq5lZM6jxvOrEambF4xarmVnOajyvOrGaWbFIHhVgZpYzj2M1M8tdjedVJ1YzKx63WM3M8uQbBMzM8iWgXbuWfhxfvpxYzaxw3GI1M8uZ+1jNzPLkPlYzs3zJ41jNzPJX43nVidXMiqe9b2k1M8uP1IovXknq2tAbI+Ld/MMxM4Mab7A22GJ9Dgiy8bp16tYD2LAZ4zKzNqzVtlgjol9LBmJmVqfG8yoV3Tcm6VBJp6fXfSVt07xhmVlbJdKQqzL/FVnZxCrpEmAP4KhUNA+4vDmDMrM2TKJ9u/JLkVUyKmCniBgs6b8AETFHUsdmjsvM2rBa7wqoJLEuktSO7IIVktYBPmrWqMyszRLQrsYzayV9rJcCfwd6SToXeAT4ZbNGZWZtmlR+qWw/6ibpb5JekPS8pB0l9ZB0n6SJ6Wf3VFeSRkqaJOlpSYObGn/ZxBoR1wFnAr8B5gCHRMSNTT2gmVk5ksouFboYuDsiNgO2BJ4HTgPGRMRAYExaB9gPGJiWEcBlTY2/0tlk2wOLgIWNeI+ZWaNV0lqtJK+mm5x2A64CiIiFEfEOMBS4NlW7FhiWXg8FrovMo0A3Ses35RwqGRVwBnADsAHQF/iLpB825WBmZpVoL5VdgJ6SnihZRtTbzUbALOBqSf+V9AdJawLrRsQMgPSzd6rfB3it5P3TUlmjVXLx6khgm4iYByDpZ8B44BdNOaCZWTkVftWfHRFDGtjeARgMnBwRj0m6mI+/9q/wsCsoi0oCqa+Sr/WvsGwC7gBMacrBzMzKyUYFlF8qMA2YFhGPpfW/kSXaN+u+4qefM0vql95x2hd4vSnnsNLEKulCSReQ3RDwXGpGXwk8A7zTlIOZmZVVwYWrSlq0EfEG8JqkT6aivYD/A24DjkllxwC3pte3AUen0QE7AHPrugwaq6GugGfTz+eAO0rKH23KgczMKpXjMNaTgT+nm5qmAMeRNShvkjQceBU4JNW9E9gfmETWoDyuqQdtaBKWq5q6UzOzVZHX7FYRMQFYUT/sXiuoG8CJeRy37MUrSRsDPwM+BaxeEsSmeQRgZlZK1P4TBCq5eHUNcDXZ+e4H3AT4BgEzazaqYCmyShJr54i4ByAiJkfEmWSzXZmZ5U7K5gootxRZJeNYP1TW4TFZ0jeA6Xw8oNbMLHcFz5tlVZJYvwd0Ab5N1te6NvC15gzKzNq2Vvtoljolg2vf4+PJrs3MmoUo/kTW5TT0lNabaeB2rog4qFkiMrO2rRHTAhZVQy3WS1osimaw9UY9GTtqeLXDsEbovu1J1Q7BGunDF18rX6kJWm1XQESMaclAzMzq1PrcpJVcvDIzazGiFbdYzcyqpcavXVWeWCV1iogPmzMYMzOpDdzSKmk7Sc8AE9P6lpJ+2+yRmVmbldN8rFVTSR/xSOAA4C2AiHgK39JqZs0or6e0VkslXQHtIuKVep3JS5opHjNr47InCBQ8c5ZRSWJ9TdJ2QEhqTzZx7EvNG5aZtWVtYbjVN8m6AzYE3gTuT2VmZs2ixhusFc0VMBM4tAViMTNDasVzBdRJDxBcbs6AiKj/DG8zs1zUeF6tqCvg/pLXqwNfAprnBmEza/PaxMWriBhVui7peuC+ZovIzNq8Gs+rTbqldQDwibwDMTMDoAZuACinkj7Wt/m4j7UdMAc4rTmDMrO2TYV/XGDDGkys6VlXW5I95wrgo/TsbTOzZiGgQ40PZG0w/JREb46IJWlxUjWzZiep7FJklfy78Likwc0eiZkZdaMCansSloaeedUhIhYDuwDHS5oMfEB23hERTrZmlr8amGSlnIb6WB8HBgPDWigWMzOgdY9jFUBETG6hWMzMlnYF1LKGEmsvSaesbGNEXNAM8ZhZmyfat+IWa3ugC9T4gDIzqynZwwSrHcWqaSixzoiIH7dYJGZm0OrvvKrxUzOzWtWaL17t1WJRmJklrborICLmtGQgZmZ1Wv1E12ZmLUm0jWdemZm1HFH4uQDKcWI1s8Kp7bRa+y1uM2tl6h7NUm6peH9Se0n/lTQ6rQ+Q9JikiZJGSeqYyjul9Ulpe/+mnoMTq5kVjipYGuE7wPMl678ELoyIgcDbwPBUPhx4OyI2AS5M9ZrEidXMCka0a1d+qWhPUl/gC8Af0rqAPYG/pSrX8vFEU0PTOmn7XmpiZ68Tq5kVSt2ogHIL0FPSEyXLiBXs7iLg+8BHaX0d4J00JSrANKBPet2H9ATqtH1uqt9ovnhlZoVTYUNxdkQMaWAfBwAzI2K8pN3rildQNSrY1ihOrGZWODmNCtgZ+KKk/YHVga5kLdhuJRP59wVeT/WnAf2AaZI6AGuTPTy10dwVYGbFonyeeRURP4yIvhHRHzgUeCAijgAeBA5O1Y4Bbk2vb0vrpO0PNPU5f06sZlYojehjbaofAKdImkTWh3pVKr8KWCeVnwKc1tQDuCvAzAon79mtIuIh4KH0egqw3QrqLAAOyeN4TqxmVjg1fkerE6uZFUvWFVDbmdWJ1cwKxy1WM7NcCbnFamaWL7dYzcxyJNGqH39tZlYVNZ5XnVjNrHhqvY/Vd14V3MiLLmTwlp9mm60GcfSRh7FgwQJenjqVXXfankGbD+TIw7/KwoULqx1mm3P52Ufwyphf8MRfT19a1r1rZ0ZfdhLP3HoWoy87iW5rrbF02/nfP5hnbz2bx0f9kK026wvAhut3Z+yfv8+jN57G+L+dwdcP3qXFz6OIsomuyy9F5sRaYNOnT+d3l45k7KNPMH7CsyxZsoS/jrqRM07/ASd/53s8+/xEunfrzjV/vKr8zixX19/+KENPvHSZslOP+xwPPf4inxn6Yx56/EVOPW4fAD6/y6fYeMNeDBp6Lif99AZGnn4oADNmvcsex17ADoeex25H/ZpTj/sc6/dau8XPpYhUwX9F5sRacIsXL2b+/PnZz3nzWG/99fnngw9w0JezOSSOOOoYbr/tlipH2faMfXIyc+bOW6bsgN234E+3PwbAn25/jAP32CIr/+wW/GX04wA8/szLrL3WGqzXsyuLFi9h4aJsWtBOHVfL/TbOWiaVX4rMibXA+vTpw3e/dyqbbrQhA/qtT9eua7P14G1Yu1s3OnTIusf79O3L669Pr3KkBtB7nbV4Y/a7ALwx+1169VgLgA16d2PaG28vrTf9zXfYoHc3APqu243HR/2QiXf9hPOvuZ8Zs+a2fOAFI7JRAeWWImu2xCppiaQJkp6VdLukbqm8v6T5aVvdcnTJ+7aWFJI+X29/7zdXrEX19ttvM/r2W3l+4lSmvPo6H8z7gHvvvmu5ekX/WtTWrSgH1M1GN+3Nd9juq79g0NBzOfLA7eidknHbVklHQLH/5puzxTo/IraKiEFkk8WeWLJtctpWt1xXsu0w4JH0s017YMz99O8/gF69erHaaqsxbNhBPDru38x95x0WL86+Qk6fNo31N9igypEawMy33mO9nl0BWK9nV2bNeQ/IWqh91+u+tF6fdbst1zKdMWsu/zf5DXYevHHLBVxUFXQDFLzB2mJdAeP4+LkyK5Ue3HUwcCywj6TVmzmuQuvXb0Mef/xR5s2bR0Tw4ANj2GzzT7Hb7nvwj79nz0L78/XXcsCBQ6scqQHc8c9nOPLA7QE48sDtGf3Q00vLDz8gm6Vuu8/059335/PG7Hfp07sbq3daDYBua63BjlttxEsvz6xO8AWT81NaW1yzj2OV1B7Yi48nkwXYWNKEkvWTI+JhskcpTI2IyZIeAvYH/tGIY40ARgD023DDVQ296rbbfnu+dNDB7LjdYDp06MCWW27N8ONHsN/+X+CoIw7l3LPPZMuttubYrw0vvzPL1bW/OJZdtxlIz25dmHT3T/jJ5Xfym6vv40+//BrHDNuR12a8zRHfz/7k737kOT6/y6d57razmbdgESec8ycAPjlgPc475UsEgRAXXTeG5ya93tBh24RsuFXRU2fD1MQnD5TfsbQEeAboD4wH9omIJZL6A6NTF0H991wKTIiIKyV9ETgqIg5J296PiC6VHn+bbYbE2MeeWPUTsRbTfduTqh2CNdKHL97ER/Nm5poFN//M1nH1LQ+WrbfjJt3HN/QwwWpq9j5W4BNAR5btY11Oatl+GThL0svAb4H9JLk336yN8cWrMiJiLvBt4FRJqzVQdW/gqYjoFxH9I+ITwN+BYc0do5kViy9eVSAi/gs8RfakREh9rCXLt8lGAdxc761/Bw5PrztLmlaynNISsZtZy/PFq5Wo3x8aEQeWrK5BBSLiNrJH0hIRvpnBrK0oeuYsw7NbmVmhZC3S2s6sTqxmViw1MHtVOU6sZlY8TqxmZnkq/nCqcpxYzaxwij6cqhwnVjMrlFoYTlWOE6uZFU+NZ1YnVjMrnFqfhMWJ1cwKp7bTqhOrmRVNK+hkdWI1s8LxcCszsxwJD7cyM8tdjedVJ1YzKx7VeJPVidXMCqfG86oTq5kVT43nVSdWMyugGs+snpXfzAqlbqLrVX2YoKR+kh6U9Lyk5yR9J5X3kHSfpInpZ/dULkkjJU2S9LSkwU09BydWMyuWNNF1uaUCi4H/iYjNgR2AEyV9CjgNGBMRA4ExaR1gP2BgWkYAlzX1FJxYzax4cniaYETMiIgn0+v3gOeBPsBQ4NpU7Vo+fhL0UOC6yDwKdJO0flPCd2I1s4KppCNAAD0lPVGyjFjpHqX+wNbAY8C6ETEDsuQL9E7V+gCvlbxtWiprNF+8MrPCqXC41eyIGFJ+X+oC/B34bkS828AY2RVtiIoiqcctVjMrlEp6ASodNCBpNbKk+ueI+EcqfrPuK376OTOVTwP6lby9L/B6U87BidXMiieHzKqsaXoV8HxEXFCy6TbgmPT6GODWkvKj0+iAHYC5dV0GjeWuADMrnJwmut4ZOAp4RtKEVHY6cB5wk6ThwKvAIWnbncD+wCRgHnBcUw/sxGpmhZNHWo2IRxrY1V4rqB/AiTkc2onVzApGnivAzKwZ1HZmdWI1s0LxRNdmZs2gxvOqE6uZFY8ff21mlrfazqtOrGZWPDWeV51YzaxY5OFWZmb5q2Qi6yJzYjWz4qntvOrEambFU+ETAgrLidXMCqayZ1oVmROrmRVKa7jzyvOxmpnlzC1WMyucWm+xOrGaWeG4j9XMLEeSRwWYmeXPidXMLF/uCjAzy5kvXpmZ5azG86oTq5kVj2q8yerEamaF0hruvFL2KO3WR9Is4JVqx9EMegKzqx2ENUpr/sw+ERG98tyhpLvJfmflzI6IffM8dl5abWJtrSQ9ERFDqh2HVc6fWdvjuQLMzHLmxGpmljMn1tpzRbUDsEbzZ9bGuI/VzCxnbrGameXMidXMLGdOrDVO0jrVjsHMluXEWsMk7QNcJKm7av0ewDbCn1Pb4MRao1JS/TVwVUS8jW9PrhXrAEjy/3utmD/cGiRpX7KkekJEPCSpH3C6pEpuA7QqUKY38IqkL0bER06urZc/2Nq0PdA5Ih6V1Au4GZgZEa31fvSaF5mZwHHA1ZL2r0uuktpXOz7Ll78+1hBJOwOfjYhzJW0kaRzZP46/j4grS+r1i4jXqhaorVRE3CRpIXCjpMMi4o66lqukA7MqMbq6Udqqcou1BpR8ZdwHWBsgIo4B/gV0r5dUjwBGSlqrxQO15UjaV9KPJO1YVxYRt5C1XG+UdEBquZ4AXA68UK1YLT9usdaGtYG3gQXA0q+NEfEDSb0kPRgRe0j6MvA94OiIeK9KsdqyPgt8A9hX0nPAJcDUiPh7GiFwjaTRwHbA/hExqYqxWk7cYi04SQOAX0jaCHgTWCuVrwEQEV8DpkiaAZxOllT/r1rx2nJuA+4HvgzMAw4Frpe0UUT8DfgK8EXg8Ih4qnphWp7cYi2+1YGZwAlAL2BaKu8kaUG6KDJc0qnAnU6q1SdpM+DDiJgaEeMkdQK+GxHflXQ4cBrQRdI04GJgvYhYWM2YLV+ehKUGSBoE7AucBGxI1graGngdWAS8BwyLiEVVC9IAkLQ/8CPgqLqv9ZIGAscDL5J9q/g62We3E/BQREytUrjWTNxiLSBJu5N9Nv+KiIUR8aykRUBnYHPgGuAZYE2gK9lQKyfVKpP0ebKkek5ETJLUBQiyx7J8Ar6SF6oAAAXGSURBVDgR2C8i/pXqvxRu2bRKbrEWjKS1gTuAAcBFwJKIuCBt2xj4KrA+cH1EPF61QG0Zkj4DPAXsHREPpM/q98ApEfG0pC3I/kE8OCKmVDFUawG+eFUwETEXGA0sBCYC+0u6RtIwsr7WS8lGCHxF0uq+97y6Sn7/L5PdqPEVSf3JJre+JyXVdhHxNPAwsIdvCGj9nFgLQtJ6Jf+Tng/cBbwXEXsDHYELyMatfjb9/HlELPBXyarrCJCGtx0BdAEmA7dExK9TUv1I0lZkXQJ3R8SS6oVrLcGJtQAkfYHsglTPdDOAyFqnW6dhVjuQDSi/CDgI+G9EzKlWvJZJE+HcKOkcSQdFxAKy0Rt/AXYESEl1ODASuDIiplcvYmsp7mOtsjShyhnAzyLibkkdI2JhmlhlPFkL6Ct1tzlK6hwR86oYsrH0czsXuA7oDWwA/CoiJqa73n5HduHqXrIbBL4REc9WK15rWU6sVSSpB9nXw4Mi4pZ0weMs4H8jYqakEcAWEXFSXcKtasAGLPO5DY2I2yX1BX4GXBYRj6Y6HYFRZLchb+vxxW2LuwKqKH2dPxA4K101voLsa/7MVOUpYC9JmzqpFkfJ53aepK4RMY3s5o3zJF0k6X/IhsINBzZxUm17PI61ytLsRkuACcDpEXGRpPYRsSQiHpP0l2rHaMtLn9tHwHhJd5NdxLoU6EF2A8DmZEOt3BfeBrkroCAkfQ74LbB9RMyV1CkiPqx2XNYwSXuT9aOuHxFvprJ2QA/Pj9t2uSugICLiPrKZqR6X1MNJtTZExP3AF4AHJK2byj5yUm3b3BVQIBFxV7rocb+kIaSJ56sdlzWs5HO7S9KQiPio2jFZdbkroIAkdYmI96sdhzWOPzer48RqZpYz97GameXMidXMLGdOrGZmOXNiNTPLmRNrGyVpiaQJkp6V9FdJnVdhX7unJ40i6YuSTmugbjdJ32rCMc5Jz/WqqLxenWskHdyIY/WX5AlTrMmcWNuu+RGxVUQMIptU+xulG5Vp9N9HRNwWEec1UKUb0OjEalZLnFgNspntN0kttecl/Q54EugnaR9J4yQ9mVq2XSCbNk/SC5IeIZsjllR+rKRL0ut1Jd0s6am07AScB2ycWsu/TvX+V9J/JD0t6dySfZ0h6UVJ9wOfLHcSko5P+3lK0t/rtcL3lvSwpJckHZDqt5f065Jjn7Cqv0gzcGJt8yR1APYjezghZAnsuojYGvgAOJPsOU6DgSeAUyStDlxJNsPTrsB6K9n9SOCfEbElMBh4juzRz5NTa/l/02TRA4HtgK2AbSTtJmkb4FCyp9EeBGxbwen8IyK2Tcd7nmx2qTr9yZ6+8AXg8nQOw4G5EbFt2v/xkgZUcByzBvmW1rZrDUkT0uuHgavIJmt+pW5OUbInF3wKGJueGtMRGAdsBkyNiIkAkv4EjFjBMfYEjgZIjyOZK6l7vTr7pOW/ab0LWaJdC7i5blJvSbdVcE6DJP2UrLuhC3BPybab0q2mEyVNSeewD7BFSf/r2unYL1VwLLOVcmJtu+ZHxFalBSl5flBaBNwXEYfVq7cV2ez4eRDwi4j4fb1jfLcJx7gGGBYRT0k6Fti9ZFv9fUU69skRUZqAUfYwQLMmc1eANeRRYGdJm0D2WBhJmwIvAAPSEw8ADlvJ+8cA30zvbS+pK/AeWWu0zj3A10r6bvtI6k32wMQvSVojPerkwAriXQuYIWk1sgf7lTpEUrsU80bAi+nY30z1kbSppDUrOI5Zg9xitZWKiFmp5XeDpE6p+MyIeEnZY2PukDQbeAQYtIJdfAe4QtnD9JYA34yIcZLGpuFMd6V+1s2BcanF/D5wZEQ8KWkU2QTgr5B1V5TzI+CxVP8Zlk3gLwL/BNYle/7UAkl/IOt7fVLZwWcBwyr77ZitnCdhMTPLmbsCzMxy5sRqZpYzJ1Yzs5w5sZqZ5cyJ1cwsZ06sZmY5c2I1M8vZ/wOjEx8p42yxrgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "clf.fit(hash_train, y_train)\n",
    "pred = clf.predict(hash_test)\n",
    "score = accuracy_score(y_test, pred)\n",
    "print(\"accuracy:   %0.3f\" % score)\n",
    "cm = confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = PassiveAggressiveClassifier(max_iter=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:   0.923\n",
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEmCAYAAAA0k8gFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3debxVZb3H8c8XEBRxAEFUZHAANWcccsgp1BQHyJxJzSi1HFKrm6mZda9lE85ZGTlVAloqicp1vKmhBooDKYrigJKAIqnM8Lt/rOfgFuHsfRbnnL32Od93r/Vi77WevdZvs+PnM61nKSIwM7OGa1PtAMzMapUTqJlZTk6gZmY5OYGameXkBGpmlpMTqJlZTk6gtlKS1pD0N0lzJN26CucZIul/GzO2apG0l6TJ1Y7DikGeB1r7JB0PnAtsCXwATAQuiYhHV/G8JwBnAntExOJVDrTgJAXQNyKmVDsWqw2ugdY4SecClwM/AboDvYBfA4Ma4fS9gZdaQ/KshKR21Y7BCiYivNXoBqwDfAgcVU+ZDmQJ9u20XQ50SMf2BaYB3wZmANOBk9OxHwELgUXpGkOBi4E/lpy7DxBAu/T+K8CrZLXgqcCQkv2PlnxuD+CfwJz05x4lxx4G/ht4LJ3nf4GuK/ludfH/V0n8g4GBwEvAe8D5JeV3BcYB76eyVwPt07G/p+/yUfq+x5Sc/3vAv4Gb6/alz2yWrtE/vd8ImAXsW+3/b3hrns010Nq2O7A6cHs9ZS4AdgN2ALYnSyIXlhzfgCwR9yBLktdI6hwRPySr1Y6MiE4RMby+QCStCVwJHBwRa5ElyYkrKNcFGJPKrgcMA8ZIWq+k2PHAycD6QHvgO/VcegOyv4MewEXAdcCXgZ2AvYCLJG2ayi4BzgG6kv3dDQC+CRARe6cy26fvO7Lk/F3IauOnlF44Il4hS65/ktQRuB64ISIeridea0GcQGvbesCsqL+JPQT4cUTMiIiZZDXLE0qOL0rHF0XE3WS1ry1yxrMU2EbSGhExPSImraDMIcDLEXFzRCyOiFuAF4HDSspcHxEvRcQ8YBRZ8l+ZRWT9vYuAEWTJ8YqI+CBdfxKwHUBETIiIx9N1XwN+C+xTwXf6YUQsSPF8QkRcB7wMPAFsSPYfLGslnEBr27tA1zJ9cxsBr5e8fz3tW3aO5RLwXKBTQwOJiI/Imr2nAdMljZG0ZQXx1MXUo+T9vxsQz7sRsSS9rktw75Qcn1f3eUn9JN0l6d+S/kNWw+5az7kBZkbE/DJlrgO2Aa6KiAVlyloL4gRa28YB88n6/VbmbbLmZ51eaV8eHwEdS95vUHowIsZGxAFkNbEXyRJLuXjqYnorZ0wNcS1ZXH0jYm3gfEBlPlPvNBVJncj6lYcDF6cuCmslnEBrWETMIev3u0bSYEkdJa0m6WBJP0/FbgEulNRNUtdU/o85LzkR2FtSL0nrAN+vOyCpu6TDU1/oArKugCUrOMfdQD9Jx0tqJ+kY4DPAXTljaoi1gP8AH6ba8TeWO/4OsOmnPlW/K4AJEfE1sr7d36xylFYznEBrXEQMI5sDeiEwE3gTOAO4IxX5H2A88CzwHPBU2pfnWvcBI9O5JvDJpNeGbDT/bbKR6X1IAzTLneNd4NBU9l2yEfRDI2JWnpga6DtkA1QfkNWORy53/GLgRknvSzq63MkkDQIOIuu2gOx36C9pSKNFbIXmifRmZjm5BmpmlpMTqJlZTk6gZmY5OYGameXUYhdH0GprhlZft9phWAPs2G/DaodgDfT6668xa9ascnNpG6Tt2r0jFn/qpq9PiXkzx0bEQY157YZquQl09XXpsNNp5QtaYTx2/w+qHYI10J6f3bnRzxmL59Fhi7KzyJg/8Zpyd5E1uRabQM2sVglUG72LTqBmViwC2rStdhQVcQI1s+JRo3arNhknUDMrGDfhzczycw3UzCwH4RqomVk+cg3UzCw3j8KbmeXhQSQzs3yEm/BmZrm5Bmpmloeb8GZm+bVxE97MrOF8L7yZWV5uwpuZ5edReDOznFwDNTPLQb6V08wsP9dAzczykEfhzcxycxPezCwHrwdqZpaX54GameXnJryZWU4eRDIzy0FuwpuZ5ecmvJlZPnICNTNruOyJHk6gZmYNp7TVACdQMysY0aaNB5HMzHJxE97MLCcnUDOzPNwHamaWj5BroGZmeTmBmpnl5FF4M7M83AdqZpafm/BmZjl4EMnMbBXUSgKtjZ5aM2tdVMFWyWmkcyRNkvS8pFskrS5pE0lPSHpZ0khJ7VPZDun9lHS8T7nzO4GaWbEoG4Uvt5U9jdQDOAvYOSK2AdoCxwI/Ay6LiL7AbGBo+shQYHZEbA5clsrVywnUzApHUtmtQu2ANSS1AzoC04HPA7el4zcCg9PrQek96fgAlbmQE6iZFUrdIFIFCbSrpPEl2yml54mIt4BfAm+QJc45wATg/YhYnIpNA3qk1z2AN9NnF6fy69UXqweRzKx4KqtgzoqInVd6CqkzWa1yE+B94Fbg4BUUjXquGivYt4wTaMGc/qVdOfnQHRHi+jFPcfVtT3LRV/fl0D37sTSCmbM/4pRLRzP93Q8555jdOeaAbQBo17YNW/bqSs/Bv2L2B/Or+yVasZcmT+aE449Z9n7q1Ff5wQ9/zEY9enDJf1/Miy+8wCP/eJKddl7pv3tTo43C7w9MjYiZAJL+CuwBrCupXaplbgy8ncpPA3oC01KTfx3gvfou4ARaIJ/ZpBsnH7oje502nIWLlzD658dzz7gpXDbiH/z4Dw8D8M0jduH7J+3NWcPu5rKR47hs5DgABu7elzOP+qyTZ5X122ILnpgwEYAlS5awWe8eHD74i8ybO5cRo/7KGd88tcoR1oZGSqBvALtJ6gjMAwYA44GHgCOBEcBJwJ2p/Oj0flw6/mBEuAZaK7bs1ZUn//UW8xZk3TOPTHyDQXttwbAR45aV6bh6e1b0mx49YBtGPTCp2WK18h568AE22XQzevfuXe1Qao7arHoCjYgnJN0GPAUsBp4GfgeMAUZI+p+0b3j6yHDgZklTyGqex5a7hhNogUyaOpOLv7YfXdZeg3kLFnHQbpvz1OSsdXHx0P0Y8oVtmfPRAg46++ZPfG6NDu04YNfNOOeKe6oRtq3ErSNHcPQxx1U7jJrUWBPpI+KHwA+X2/0qsOsKys4HjmrI+ZtsFF7SEkkTS7Y+JceukPSWpDYl+74i6er0uo2kGyX9QZnXJD1Xcq4rmyruapr8xix+dcs/uOuXQxj98+N59pV3WLxkKQAXD3+IvkdfyYj7nue0L+7yic8dskc/xj3/ppvvBbJw4ULG3DWaI45s0L9Ho7IpTEW5U6kppzHNi4gdSrbXIEuOwBfJpgvsvfyH0ryr3wCrAV8r6YPYr+RcZzVh3FV1490T2eOU33PAt25i9n/mMWXaJ/uwRz3wPIP32fIT+476/Nbc6uZ7oYy99x522LE/3bt3r3YoNckJdOX2A54HrgVW1L65gmzu1YkRsbQ5AyuCbut2BKDn+mszaO8tGfXAJDbr0WXZ8UP26MdLb7y77P3aa3bgc9v35m+PTW72WG3lRo28xc33VVArCbQp+0DXkDQxvZ4aEV9Mr48DbiEb+fqJpNUiYlE6djzwArBvyUTXOg9JWpJe3xgRly1/wTSRNptM22GdxvsmzeiWHx9Fl7XXYNHipZx9+T28/+F8rv3uofTttR5LlwZvvDOHs4bdvaz84XttwQPjX2Xu/EX1nNWa09y5c3nw/vu4+te/Xbbvzjtu59yzz2TWzJkcMegQttt+B/5299gqRllsjTGI1BxUZpQ+/4mlDyOi03L72gOvAVtExAdpXtbwiBgj6SvAl4EtgWMi4rGSz71Gdj/rrEqv32atHtFhp9NW/YtYs5l9/w+qHYI10J6f3ZkJE8Y3arbrsEHf2HhI+WGOV4cNnFDfRPrm0NxN+IPIJqc+l5Li5/hkM/5F4GhgpKStmzk2MysAAVL5rQiaO4EeRzYw1Cci+pDdYnVgmugKQET8AzgNGCOpVzPHZ2ZVVzuj8M02DzQlyS8Ay27FiIiPJD0KHFZaNiLuktQNuFfSXml3aR/osxFxYnPEbWbNryD5sawmS6DL939GxFygywrKHVHy9oaS/dcD16e3fRo/QjMrqqLUMMvxnUhmVigStG3rBGpmlkuNVECdQM2seNyENzPLo0DTlMpxAjWzQsnmgdZGBnUCNbOCKc48z3KcQM2scNrUyL3wTqBmVizuAzUzy8d9oGZmq6BG8qcTqJkVj2ugZmY51Uj+dAI1s2KRPApvZpaT54GameVWI/nTCdTMisc1UDOzPDyR3swsHwFt2jT349rycQI1s8JxDdTMLCf3gZqZ5eE+UDOzfOR5oGZm+dVI/nQCNbPiaetbOc3MGk5qAYNIktau74MR8Z/GD8fMDGqkAlpvDXQSEGTzWuvUvQ+gVxPGZWatWM3XQCOiZ3MGYmZWp0byJxXdLyXpWEnnp9cbS9qpacMys9ZKpKlMZf5XBGUTqKSrgf2AE9KuucBvmjIoM2vFJNq2Kb9VdiqtK+k2SS9KekHS7pK6SLpP0svpz86prCRdKWmKpGcl9S93/kpqoHtExKnAfICIeA9oX1H0ZmY5SOW3Cl0B3BsRWwLbAy8A5wEPRERf4IH0HuBgoG/aTgGuLXfyShLoIkltyAaOkLQesLTi8M3MGkBAG6nsVvY82UyivYHhABGxMCLeBwYBN6ZiNwKD0+tBwE2ReRxYV9KG9V2jkgR6DfAXoJukHwGPAj+r4HNmZrlUWAPtKml8yXbKcqfZFJgJXC/paUm/l7Qm0D0ipgOkP9dP5XsAb5Z8flrat1JlJ9JHxE2SJgD7p11HRcTz5T5nZpZXhdOYZkXEzvUcbwf0B86MiCckXcHHzfUVXnYF+6K+ACpdtbQtsAhY2IDPmJk1WCW1zwr7QKcB0yLiifT+NrKE+k5d0zz9OaOkfOn0zY2Bt+u7QCWj8BcAtwAbpRP+WdL3KwrfzCyHtlLZrZyI+DfwpqQt0q4BwL+A0cBJad9JwJ3p9WjgxDQavxswp66pvzKV3Av/ZWCniJgLIOkSYALw0wo+a2bWYI14J9KZwJ8ktQdeBU4mqziOkjQUeAM4KpW9GxgITCGbrnlyuZNXkkBfX65cuxSImVmjy0bhG+dcETERWFE/6YAVlA3g9Iacv77FRC4j60CdC0ySNDa9P5BsJN7MrPGpZSyoXDfSPgkYU7L/8aYLx8ysdu6Fr28xkeHNGYiZWZ2WUAMFQNJmwCXAZ4DV6/ZHRL8mjMvMWilROyvSVzKn8wbgerLvdTAwChjRhDGZWSunCrYiqCSBdoyIsQAR8UpEXEi2OpOZWaOTGude+OZQyTSmBco6JF6RdBrwFh/fO2pm1ugKkh/LqiSBngN0As4i6wtdB/hqUwZlZq1bixlEKrmP9AM+XlTZzKxJiMoXTK62+ibS3049K5FExBFNEpGZtW4NWzC5quqrgV7dbFE0gR36bcjfx15Q7TCsATrvcka1Q7AGWjD5jSY5b8034SPigeYMxMysTq2smVnJIJKZWbMRLaAGamZWLTUyhlR5ApXUISIWNGUwZmZSC7qVU9Kukp4DXk7vt5d0VZNHZmatVhuV34qgkr7aK4FDgXcBIuIZfCunmTWhRnwufJOqpAnfJiJeX65Td0kTxWNmrVzdc+FrQSUJ9E1JuwIhqS3ZM0ZeatqwzKw1a0nTmL5B1ozvBbwD3J/2mZk1iRqpgFZ0L/wM4NhmiMXMDKkF3AtfR9J1rOCe+Ig4pUkiMrNWr0byZ0VN+PtLXq8OfBF4s2nCMbPWrkUNIkXEyNL3km4G7muyiMys1auR/JnrVs5NgN6NHYiZGQAFmihfTiV9oLP5uA+0DfAecF5TBmVmrZsK89i4+tWbQNOzkLYnew4SwNKIWOkiy2Zmq0pAuxqZCFpvmClZ3h4RS9Lm5GlmTU5S2a0IKsnzT0rq3+SRmJlRNwpfG4uJ1PdMpHYRsRj4HPB1Sa8AH5F9v4gIJ1Uza3wFWiyknPr6QJ8E+gODmykWMzOgZcwDFUBEvNJMsZiZLWvC14L6Emg3Seeu7GBEDGuCeMys1RNtW0ANtC3QCWpkQpaZtQjZQ+WqHUVl6kug0yPix80WiZkZtJg7kWrkK5hZS9MSBpEGNFsUZmZJi2jCR8R7zRmImVmdFrOgsplZcxK180ykWonTzFoLNe698JLaSnpa0l3p/SaSnpD0sqSRktqn/R3S+ynpeJ9y53YCNbPCUQVbA3wLeKHk/c+AyyKiLzAbGJr2DwVmR8TmwGWpXL2cQM2sUOoe6VFuq+hc0sbAIcDv03sBnwduS0Vu5OPb1Qel96TjA1SmqusEamaFU2ENtKuk8SXbih50eTnwX8DS9H494P20UBLANKBHet2D9Ly3dHxOKr9SHkQys4IRbSobhZ8VETuv9CzSocCMiJggad9lJ/+0qODYCjmBmlmhNOIo/J7A4ZIGkj1ReG2yGum6Jct1bgy8ncpPA3oC0yS1A9Yhe4TRSrkJb2aF0xij8BHx/YjYOCL6AMcCD0bEEOAh4MhU7CTgzvR6dHpPOv5guadwOIGaWeE08ij88r4HnCtpClkf5/C0fziwXtp/LhU8PNNNeDMrljQPtDFFxMPAw+n1q8CuKygzHziqIed1AjWzQqmlO5GcQM2scFrCakxmZlVRI/nTCdTMiiVrwtdGBnUCNbPCcQ3UzCwXIddAzczycQ3UzCwHiRbxWGMzs6qokfzpBGpmxVMrfaC1MuG/1fr11Veya//t2GXHbbnmqisAeO+99zh84IHssPUWHD7wQGbPnl3lKO304/Zl/K3nM+G2Czjj+H0B2LZfDx6+8dv8c9T53Hb5qay15uoA7Lx1bx4fcR6PjziPJ0aex+H7bVfFyIsnW1C5/FYETqAF9q9Jz3PDH37Pw48+zrh/Ps29d49hypSXGfbLn7HPfgOYOGky++w3gGG/LPvkAWtCn9lsQ04+Yg/2OuEX7HrMTzl4723YrFc3rr3oeC688k52OfonjH7oGc45KXtS+KRX3mbPIT9nt2MvZdDpv+aqC4+jbVv/UyylCv5XBP7VCmzyiy+wy66fpWPHjrRr147P7bU3f7vzDsb8bTRDvnwiAEO+fCJ3jb6zzJmsKW25yQY8+dxrzJu/iCVLlvLIhCkM2m97+vZen0cnTAHgwcdfZPCAHQCWlQPo0H41yqyY1ipJ5bcicAItsK223obHHn2Ed999l7lz5zJ27D28Ne1NZs54hw023BCADTbckFkzZ1Q50tZt0itv87n+m9NlnTVZY/XVOOhzW7PxBp351yvTOXTfbQE44oD+bNy987LP7LJNbybcdgHjbz2fsy4ZsSyhWtaEbyuV3YqgyQaRJC0BnkvXmAqcEBHvp0eFvgBMLik+LCJuSp/bEXgKOCgixpac78OI6NRU8RbRlltuxTnf/i6DDvkCa67ZiW233Y527TzuVzSTp77Dr264j7uuPYOP5i3g2ZfeYvHiJZx68Z/41X8dyfe/fjBj/u85Fi5asuwz/3z+dXY68hK22KQ7v//xCYx97F8sWLi4nqu0JsVpopfTlDXQeRGxQ0RsQ7Ys/uklx15Jx+q2m0qOHQc8mv5s9U46eSiPPj6esQ88TOfOXdhs8750W787/54+HYB/T59O127rVzlKu/GOcexx/M84YOjlzJ7zEVPemMlLr73DYd+8hj2H/JxR905g6rSZn/rc5Knv8NG8hWy9+UZViLqgKmi+F6QC2mxN+HF8/OS7lUqPED0S+ApwoKTVmziuwps5I2uev/nGG4y+83aOPPpYBh56GH/6Y/bfnD/98SYOOezwaoZoQLfOWeOo5wadGfT57Rl17/hl+yRx3te/wHW3PQpA743WWzZo1GvDzvTr053X3363OoEXVBOvSN9omrw9KKktMICPl80H2EzSxJL3Z0bEI2QPgZoaEa9IehgYCPy1Adc6BTgFoGfPXqsaeiEMOfYo3nvvXVZbbTWGXX4VnTt35tzvfI+ThhzLzTf8gY179uKmP4+sdpit3i2//Bpd1l2TRYuXcPalo3j/g3mcfty+nHrM3gDc+eBEbrrzcQD22HFTvnPygSxavISlS4Nv/WQk777/UTXDL5S658LXAjXVCGBJH2gfYAJwYEQsSX2gd6Wm/fKfuQaYGBHXSTqcrN/0qHSsQX2g/XfaOf7+jydX/YtYs+m221nVDsEaaMHkUSydO6NRs91W2+4Y19/xUNlyu2/eeUJ9jzVuDk3eBwr0BtrzyT7QT0k11S8BF0l6DbgKOFjSWk0Yo5kVkOeBJhExBzgL+I6k1eopuj/wTET0jIg+EdEb+AswuKljNLNi8SBSiYh4GniG7NnMkPpAS7azyEbdb1/uo38Bjk+vO0qaVrKd2xyxm1nza/WDSMv3V0bEYSVv16jwHKPJHnZPRHjSv1lrUZQMWYZnZZtZoWQ1zNrIoE6gZlYsBVptqRwnUDMrHidQM7M8ijNNqRwnUDMrnKJMUyrHCdTMCqVI05TKcQI1s+KpkQzqBGpmhVMri4k4gZpZ4dRG+nQCNbOiqaFOUCdQMyscT2MyM8tBeBqTmVluNZI/nUDNrHhUI1VQJ1AzK5wayZ/N9lROM7OKNcaCypJ6SnpI0guSJkn6VtrfRdJ9kl5Of3ZO+yXpSklTJD0rqX+5aziBmlnxNM6S9IuBb0fEVsBuwOmSPgOcBzwQEX2BB9J7gIOBvmk7Bbi23AWcQM2sUOoWVF7Vh8pFxPSIeCq9/gB4AegBDAJuTMVu5OPnrg0CborM48C6kjas7xruAzWzYql8QeWuksaXvP9dRPxuhafMHqe+I/AE0D0ipkOWZCWtn4r1AN4s+di0tG/6ygJwAjWz4qksgc6q5LnwkjqRPaDy7Ij4Tz0j/Cs6EPWd2014MyuYShrwlWXY9Cj1vwB/ioi/pt3v1DXN058z0v5pQM+Sj28MvF3f+Z1AzaxwGuO58MqqmsOBFyJiWMmh0cBJ6fVJwJ0l+09Mo/G7AXPqmvor4ya8mRVKI64lsidwAvCcpIlp3/nApcAoSUOBN4Cj0rG7gYHAFGAucHK5CziBmlnxNEIGjYhH6znTgBWUD+D0hlzDCdTMCscLKpuZ5VQb6dMJ1MyKpsJBoiJwAjWzAqqNDOoEamaF4gWVzcxWQY3kTydQMysej8KbmeVVG/nTCdTMiqdG8qcTqJkVS6X3uheBE6iZFY6fC29mlldt5E8nUDMrngpXpK86J1AzK5jKF0yuNidQMyuUWroTySvSm5nl5BqomRVOrdRAnUDNrHDcB2pmloMqfy581TmBmlnxOIGameXjJryZWU4eRDIzy6lG8qcTqJkVj2qkCuoEamaFUkt3Iikiqh1Dk5A0E3i92nE0ga7ArGoHYQ3Skn+z3hHRrTFPKOlesr+zcmZFxEGNee2GarEJtKWSND4idq52HFY5/2Ytl++FNzPLyQnUzCwnJ9Da87tqB2AN5t+shXIfqJlZTq6Bmpnl5ARqZpaTE2iNk7RetWMwa62cQGuYpAOByyV1Vq3c+9bK+XdqWZxAa1RKnr8AhkfEbHxbbq1YD0CS/+21AP4Ra5Ckg8iS56kR8bCknsD5kiq5/c2qQJn1gdclHR4RS51Ea59/wNr0WaBjRDwuqRtwOzAjIlrq/dY1LzIzgJOB6yUNrEuiktpWOz7Lx82+GiJpT2CfiPiRpE0ljSP7j+BvI+K6knI9I+LNqgVqKxURoyQtBEZIOi4ixtTVRCUdlhWJu6obpVXKNdAaUNLUOxBYByAiTgL+DnReLnkOAa6UtFazB2qfIukgST+QtHvdvoi4g6wmOkLSoakmeirwG+DFasVqDecaaG1YB5gNzAeWNfci4nuSukl6KCL2k/Ql4BzgxIj4oEqx2iftA5wGHCRpEnA1MDUi/pJG5G+QdBewKzAwIqZUMVZrINdAC07SJsBPJW0KvAOslfavARARXwVelTQdOJ8sef6rWvHap4wG7ge+BMwFjgVulrRpRNwGHA0cDhwfEc9UL0zLwzXQ4lsdmAGcCnQDpqX9HSTNT4MTQyV9B7jbybP6JG0JLIiIqRExTlIH4OyIOFvS8cB5QCdJ04ArgA0iYmE1Y7Z8vJhIDZC0DXAQcAbQi6xWsyPwNrAI+AAYHBGLqhakASBpIPAD4IS65rikvsDXgclkrYSvkf12ewAPR8TUKoVrq8g10AKStC/Zb/P3iFgYEc9LWgR0BLYCbgCeA9YE1iabwuTkWWWSvkCWPC+OiCmSOgFB9jiP3sDpwMER8fdU/qVwDaamuQZaMJLWAcYAmwCXA0siYlg6thlwDLAhcHNEPFm1QO0TJG0LPAPsHxEPpt/qt8C5EfGspO3I/sN3ZES8WsVQrRF5EKlgImIOcBewEHgZGCjpBkmDyfpCryEbkT9a0uq+t7q6Sv7+XyO7oeFoSX3IFlEem5Jnm4h4FngE2M8T51sOJ9CCkLRByT/GXwH3AB9ExP5Ae2AY2bzPfdKfP4mI+W4CVl17gDRtbAjQCXgFuCMifpGS51JJO5A15e+NiCXVC9cakxNoAUg6hGxgqGuaNC+y2uaOafrSbmQTry8HjgCejoj3qhWvZdKCLiMkXSzpiIiYTzZb4s/A7gApeQ4FrgSui4i3qhexNTb3gVZZWhjkAuCSiLhXUvuIWJgWCJlAVqM5uu72PkkdI2JuFUM2lv1uPwJuAtYHNgJ+HhEvp7vAfk02gPS/ZBPpT4uI56sVrzUNJ9AqktSFrFl3RETckQYeLgK+GxEzJJ0CbBcRZ9Ql1qoGbMAnfrdBEfE3SRsDlwDXRsTjqUx7YCTZ7be7eH5uy+QmfBWlZvhhwEVplPZ3ZM3zGanIM8AASf2cPIuj5He7VNLaETGN7CaHSyVdLunbZFPMhgKbO3m2XJ4HWmVpNZ4lwETg/Ii4XFLbiFgSEU9I+nO1Y7RPS7/bUmCCpHvJBpOuAbqQTZTfimwKk/uqWzA34QtC0gHAVcBnI2KOpA4RsaDacVn9JO1P1s+5YUS8k/a1Abp4fdaWz034goiI+8hWUnpSUhcnz9oQEfcDhwAPSuqe9i118mwd3IQvkIi4Jw0+3C9pZ6sSTvoAAANYSURBVNJC5tWOy+pX8rvdI2nniFha7ZisebgJX0CSOkXEh9WOwxrGv1vr4wRqZpaT+0DNzHJyAjUzy8kJ1MwsJydQM7OcnEBbKUlLJE2U9LykWyV1XIVz7ZueLImkwyWdV0/ZdSV9M8c1Lk7Pfapo/3JlbpB0ZAOu1UeSF/6wspxAW695EbFDRGxDtnjzaaUHlWnw/z8iYnREXFpPkXWBBidQsyJyAjXIVkrfPNW8XpD0a+ApoKekAyWNk/RUqql2gmw5N0kvSnqUbI1S0v6vSLo6ve4u6XZJz6RtD+BSYLNU+/1FKvddSf+U9KykH5Wc6wJJkyXdD2xR7ktI+no6zzOS/rJcrXp/SY9IeknSoal8W0m/KLn2qav6F2mtixNoKyepHXAw2UPqIEtUN0XEjsBHwIVkz/npD4wHzpW0OnAd2YpEewEbrOT0VwL/FxHbA/2BSWSP9H0l1X6/mxYl7gvsCuwA7CRpb0k7kT1DfUeyBL1LBV/nrxGxS7reC2SrIdXpQ7aa/yHAb9J3GArMiYhd0vm/LmmTCq5jBvhWztZsDUkT0+tHgOFkiwK/XremJdlK+J8BHktPG2kPjAO2BKZGxMsAkv4InLKCa3weOBEgPcZijqTOy5U5MG1Pp/edyBLqWsDtdYtHSxpdwXfaRtL/kHUTdALGlhwblW6xfFnSq+k7HAhsV9I/uk669ksVXMvMCbQVmxcRO5TuSEnyo9JdwH0Rcdxy5XYgW229MQj4aUT8drlrnJ3jGjcAgyPiGUlfAfYtObb8uSJd+8yIKE20KHsonFlZbsJbfR4H9pS0OWSPE5HUD3gR2CStoA9w3Eo+/wDwjfTZtpLWBj4gq13WGQt8taRvtYek9ckenPdFSWukR2QcVkG8awHTJa1G9oC3UkdJapNi3hSYnK79jVQeSf0krVnBdcwA10CtHhExM9XkbpHUIe2+MCJeUva4kTGSZgGPAtus4BTfAn6n7KFqS4BvRMQ4SY+laUL3pH7QrYBxqQb8IfDliHhK0kiyhaZfJ+tmKOcHwBOp/HN8MlFPBv4P6E72fKL5kn5P1jf6lLKLzwQGV/a3Y+bFRMzMcnMT3swsJydQM7OcnEDNzHJyAjUzy8kJ1MwsJydQM7OcnEDNzHL6f3JBd8Uq/9U0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "clf.fit(hash_train, y_train)\n",
    "pred = clf.predict(hash_test)\n",
    "score = accuracy_score(y_test, pred)\n",
    "print(\"accuracy:   %0.3f\" % score)\n",
    "cm = confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "So was your fake news classifier experiment a success? Definitely not.\n",
    "\n",
    "But you did get to play around with a new dataset, test out some NLP classification models and introspect how successful they were? Yes.\n",
    "\n",
    "As expected from the outset, defining fake news with simple bag-of-words or TF-IDF vectors is an oversimplified approach. Especially with a multilingual dataset full of noisy tokens. If you hadn't taken a look at what your model had actually learned, you might have thought the model learned something meaningful. So, remember: always introspect your models (as best you can!).\n",
    "\n",
    "I would be curious if you find other trends in the data I might have missed. I will be following up with a post on how different classifiers compare in terms of important features on my blog. If you spend some time researching and find anything interesting, feel free to share your findings and notes in the comments or you can always reach out on Twitter (I'm @kjam).\n",
    "\n",
    "I hope you had some fun exploring a new NLP dataset with me!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook Detecting_Fake_News.ipynb to html\n",
      "[NbConvertApp] Writing 445830 bytes to Detecting_Fake_News.html\n"
     ]
    }
   ],
   "source": [
    "! jupyter nbconvert --to html Detecting_Fake_News.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}