Naive_Bayes_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes Classifier \n",
    "## Text Classification  (Multinomial & Bernoulli Distributions)\n",
    "\n",
    "***Author: Rabindra Nepal***\n",
    "\n",
    "*Email: rnepal2@unl.edu*\n",
    "\n",
    "*Format prepared by: M. R. Hasan*\n",
    "\n",
    "In this notebook we will apply the Naive Bayes algorithms for text classification.\n",
    "\n",
    "\n",
    "### Dataset: The 20 Newsgroups data set\n",
    "\n",
    "\n",
    "The 20 newsgroups dataset comprises around 20,000 newsgroups posts on 20 topics split in two subsets: one for training (or development) and the other one for testing (or for performance evaluation). The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering. The split between the train and test set is based upon a messages posted before and after a specific date.\n",
    "\n",
    "Following is a list of the 20 newsgroups, partitioned (more or less) according to subject matter:\n",
    "\n",
    "- alt.atheism\n",
    "- comp.graphics\n",
    "- comp.os.ms-windows.misc\n",
    "- comp.sys.ibm.pc.hardware\n",
    "- comp.sys.mac.hardware\n",
    "- comp.windows.x\n",
    "- misc.forsale\n",
    "- rec.autos\n",
    "- rec.motorcycles\n",
    "- rec.sport.baseball\n",
    "- rec.sport.hockey\n",
    "- sci.crypt\n",
    "- sci.electronics\n",
    "- sci.med\n",
    "- sci.space\n",
    "- soc.religion.christian\n",
    "- talk.politics.guns\n",
    "- talk.politics.mideast\n",
    "- talk.politics.misc\n",
    "- talk.religion.misc\n",
    "\n",
    "\n",
    "We will normalize the documents, perform preprocessing and vectorize the features. Since the features are categorical, we will implement two different naive Bayes classifiers using Scikit-Learn. \n",
    "- Categorical features (binary valued) are modeled using the Multivariate Bernoulli distrubition \n",
    "- Categorical features (multi-valued) are modeled using the Multinomial distrubition \n",
    "\n",
    "\n",
    "## Steps for Classification:\n",
    "\n",
    "1. Exploratory Data Analysis\n",
    "2. Feature Extraction\n",
    "   - a. Text Normalization (Stemming & Lemmatization)\n",
    "   - b. Text Preprocessing (Tokenization, removing stop words, etc.)\n",
    "   - c. Vectorization of the features\n",
    "3. Model Selection by Hyperparameter Tuning\n",
    "4. Train the Optimal Model\n",
    "5. Analyzing Model Performance\n",
    "6. Evaluate the Model on Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\nepal\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "import seaborn as sns\n",
    "\n",
    "import nltk\n",
    "nltk.download('wordnet')\n",
    "\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.stem import PorterStemmer, LancasterStemmer\n",
    "from nltk import word_tokenize\n",
    "from wordcloud import WordCloud\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score, cross_val_predict\n",
    "from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, roc_curve\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score, cross_val_predict\n",
    "from sklearn.metrics import precision_recall_curve, classification_report\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer\n",
    "from sklearn.naive_bayes import MultinomialNB, BernoulliNB\n",
    "\n",
    "from sklearn.datasets import fetch_20newsgroups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "We will work on a partial dataset with only 4 categories out of the 20 available in the dataset:\n",
    "- alt.atheism\n",
    "- soc.religion.christian\n",
    "- comp.graphics\n",
    "- sci.med\n",
    "\n",
    "\n",
    "The samples are shuffled randomly. This is useful if you wish to select only a subset of samples to quickly train a model and get a first idea of the results before re-training on the complete dataset later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Train and Test Sets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']\n",
    "\n",
    "train_data = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)\n",
    "X_train = train_data.data\n",
    "y_train = train_data.target\n",
    "\n",
    "\n",
    "test_data = fetch_20newsgroups(subset='test', categories=categories, shuffle=True, random_state=42)\n",
    "X_test = test_data.data\n",
    "y_test = test_data.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quick Check of the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Names:  ['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']\n",
      "\n",
      "Number of Training Examples:  2257\n",
      "Number of Training Labels:  2257\n",
      "Number of Test Examples:  1502\n",
      "Number of Test Labels:  1502\n",
      "\n",
      "Print a Random Document:\n",
      "\n",
      "From: sd345@city.ac.uk (Michael Collier)\n",
      "Subject: Converting images to HP LaserJet III?\n",
      "Nntp-Posting-Host: hampton\n",
      "Organization: The City University\n",
      "Lines: 14\n",
      "\n",
      "Does anyone know of a good way (standard PC application/PD utility) to\n",
      "convert tif/img/tga files into LaserJet III format.  We would also like to\n",
      "do the same, converting to HPGL (HP plotter) files.\n",
      "\n",
      "Please email any response.\n",
      "\n",
      "Is this the correct group?\n",
      "\n",
      "Thanks in advance.  Michael.\n",
      "-- \n",
      "Michael Collier (Programmer)                 The Computer Unit,\n",
      "Email: M.P.Collier@uk.ac.city                The City University,\n",
      "Tel: 071 477-8000 x3769                      London,\n",
      "Fax: 071 477-8565                            EC1V 0HB.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Target Names: \", train_data.target_names)\n",
    "\n",
    "print(\"\\nNumber of Training Examples: \", len(X_train))\n",
    "print(\"Number of Training Labels: \", len(y_train))\n",
    "\n",
    "print(\"Number of Test Examples: \",len(X_test))\n",
    "print(\"Number of Test Labels: \", len(y_test))\n",
    "\n",
    "\n",
    "print(\"\\nPrint a Random Document:\\n\")\n",
    "print(X_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class Distribution\n",
    "\n",
    "#### Compute class distribution in the following block "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first creating two DataFrames for train and test dataset\n",
    "\n",
    "train_df = pd.DataFrame({'text': X_train, 'class': y_train})\n",
    "test_df = pd.DataFrame({'text': X_test, 'class': y_test})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    599\n",
       "2    594\n",
       "1    584\n",
       "0    480\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train data\n",
    "\n",
    "train_df['class'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    398\n",
       "2    396\n",
       "1    389\n",
       "0    319\n",
       "Name: class, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test data\n",
    "\n",
    "test_df['class'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization of the Class Distribution\n",
    "\n",
    "\n",
    "####  Generate visualization of the class distribution in the following block "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAF6CAYAAACp7HR5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHqhJREFUeJzt3XuUnVWZ5/FvVQpIuAjGEkwlCDhkGliIOEbE8YIGHAGVMDP6xBuiQ68MNtrKZTF4Z3rAibaIzIzNdJqAwXF1fMZbUNAGklbotYSxQZHhIiIiSSoDKRICGhOs1Jk/zltQqRSkTjiXTZ3vZ62z6rz73efUc8Lm5Je930tPrVZDkiRJ5entdAGSJEmamEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQfZ0uoIm8F5YkSXo+6dlZh6kU1BgcHOx0CVNGf38/Q0NDnS5DHeY4EDgOVOc4aK6BgYFJ9XPpU5IkqVAGNUmSpEIZ1CRJkgo1pY5RG69Wq7FlyxZGRkbo6dnp8XrPe7Vajd7eXqZPn94Vn1eSpKluSge1LVu2sNtuu9HXN6U/5naGh4fZsmULM2bM6HQpkiTpOZrSS58jIyNdFdIA+vr6GBkZ6XQZkiSpCaZ0UOvW5b9u/dySJE013TXd1GYbNmxg4cKFAKxfv55p06Yxc+ZMAK699lp23333nb7H2WefzVlnncWhhx7a0lolSVJ5uiqozZ49uYvLTdbatc9+gd2ZM2dyww03AHDJJZew1157ceaZZ27Xp1arPXUSwEQuvfTS5hQrSZKed9oW1CJiP+AK4Ejqt3v6D8CvgG8CBwMPApGZGyOiB7gMOBnYDHwwM29vV62t9tvf/pYzzjiDV7/61fz85z9n2bJlXHrppdx5551s2bKFU045hbPPPhuAU089lYsuuojDDjuMl7/85Zx22mmsWrWKGTNmcNVVV9Hf39/hTyNJklqlnceoXQb8KDMPA14B3ANcAKzMzLnAymob4CRgbvVYBFzexjrb4r777uM973kP119/PbNmzeITn/gEP/zhD7nhhhu46aabuO+++3Z4zeOPP86xxx7LjTfeyKte9SqWL1/egcolSVK7tCWoRcQLgDcCSwEy88nMfAxYACyrui0DTq2eLwCuzsxaZt4C7BcRs9pRa7scdNBBHH300U9tr1ixgre+9a2ceOKJ/PrXv54wqE2fPp358+cDcNRRR7F69eq21StJktqvXUufLwPWA1dFxCuA24CPAQdk5jqAzFwXEftX/WcDY1PImqptXZvqbbk999zzqecPPPAAV1xxBddeey377rsvH/3oR9m6desOrxl78sG0adPYtm1bW2qVJEmd0a6g1gf8K+CjmXlrRFzG08ucE5no+hK18Q0RsYj60iiZucPxWg8//HBLr6PWyHv39vbS29tLX18ffX199PT0PPX6P/7xj+yzzz688IUvZP369fzkJz/h+OOP367faN/Rn729vdu9x1h77LHHcz52ra+vz+PfnsUee+z8jN2po7kn4ZRq69YnO11Csfw+EDgOOqVdQW0NsCYzb622v0U9qD0cEbOq2bRZwCNj+h845vVzgB1OsczMJcCSarM2NDS03f6tW7cybdq05n2KcYaHhyfdd2RkhJGREYaHhxkeHqZWqz31+iOOOIJDDz2UN77xjbz0pS9l3rx5bNu2bbt+o31Hf46MjGz3HmNt3bqV8X8Wjerv73/O7zG1dUd46SaO92fm94HAcdBsAwOT+3ukp1bbYaKqJSLiZuDPM/NXEXEhsFe169HMXBwRFwAzM/P8iHgb8BHqZ32+BvhvmXnMTn5FbXBw+yy3efPm7ZYYu0UzPrf/Qz67Zl/qRZ23s8vtdDO/DwSOg2argtpOr1DfzuuofRT4RkTsDjwAfIj6yQwZEWcADwHvqvpeRz2k3U/98hwfamOdkiRNWnf9w607PmtJ/3BrW1DLzF8A8ybYdfwEfWvAWS0vSpIkqWBT+l6fkiRJz2cGNUmSpEIZ1CRJkgrVVTdll6Rm8iDyqaekg8glMKi11IYNG1i4cCEA69evZ9q0acycOROAa6+9drs7DTyb5cuXM3/+fPbff/+dd5YkSVNGVwW1gdmzm/p+g2vXPuv+mTNncsMNNwBwySWXsNdee3HmmWc2/HuWL1/OkUceaVCTJKnLdFVQK0lmsmzZMp588knmzZvHxRdfzMjICGeffTZ33303tVqN973vffT393PXXXfx4Q9/mOnTpzc0EydJkp7fDGodcO+99/KjH/2IFStW0NfXx/nnn8+KFSs46KCD2LhxIytXrgRg06ZN7Lvvvlx11VVcdNFFHHnkkR2uXJIktZNBrQNuvvlm7rjjDk466SQAtmzZwqxZszjuuOP4zW9+w2c/+1nmz5/Pcccd1+FKJUlSJxnUOqBWq7Fw4ULOP//8HfbdeOONrFq1iqVLl3LdddfxxS9+sQMVSpKkEngdtQ54wxvewPe//302bNgA1M8OXbt2LY8++ii1Wo13vOMdnHfeedx5550A7L333vzhD3/oZMmSJKkDnFHrgMMPP5xzzjmHhQsXUqvV6OvrY/HixUybNo1zzz2XWq1GT08Pn/rUpwCICM477zxPJpAkqcv01Gq1TtfQLLXBwe0vVLh582b23HPPDpXTOc343P39/QwNDTWpoqmnuy502h125UKnjoOpx3EgaM+FjwcGBgB6dtbPpU9JkqRCGdQkSZIKZVCTJEkq1JQOalPo+LuGdOvnliRpqpnSQa23t5fh4eFOl9FWw8PD9PZO6f+skiR1jSl9eY7p06ezZcsWtm7dSk/PTk+seN6r1Wr09vYyffr0TpciSZKaYEoHtZ6eHmbMmNHpMiRJknaJa2SSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklSovnb9ooh4EHgC2AYMZ+a8iJgJfBM4GHgQiMzcGBE9wGXAycBm4IOZeXu7apUkSSpBu2fU3pyZR2fmvGr7AmBlZs4FVlbbACcBc6vHIuDyNtcpSZLUcZ1e+lwALKueLwNOHdN+dWbWMvMWYL+ImNWJAiVJkjqlnUGtBlwfEbdFxKKq7YDMXAdQ/dy/ap8NrB7z2jVVmyRJUtdo2zFqwOsyczAi9gduiIh7n6VvzwRttfENVeBbBJCZ9Pf3N6dS0dfX55+nuorjXeA4UF1J46BtQS0zB6ufj0TEd4FjgIcjYlZmrquWNh+puq8BDhzz8jnA4ATvuQRYUm3WhoaGWlZ/t+nv78c/z2cz0OkC1GS7Nt4dB1ON40Cwq+OgMQMDkxs3bVn6jIi9ImKf0efAvwH+L3ANcHrV7XRgRfX8GuADEdETEccCm0aXSCVJkrpFu45ROwD4p4i4A/g/wLWZ+SNgMfCWiPg18JZqG+A64AHgfuDvgL9oU52SJEnF6KnVdjj06/mqNji4w+qodpFLn89u9myXOqaatWsb//5wHEw9jgPBro2DRlVLnxMdk7+dTl+eQ5IkSc/AoCZJklQog5okSVKhDGqSJEmFMqhJkiQVqp13Jnje674ze7rj87bj7B5JknaFM2qSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKh+tr5yyJiGvDPwNrMfHtEHAIsB2YCtwOnZeaTEbEHcDXwKuBRYGFmPtjOWiVJkjqt3TNqHwPuGbP9BeDSzJwLbATOqNrPADZm5qHApVU/SZKkrtK2oBYRc4C3AVdU2z3AfOBbVZdlwKnV8wXVNtX+46v+kiRJXaOdM2pfAc4HRqrtFwGPZeZwtb0GmF09nw2sBqj2b6r6S5IkdY22HKMWEW8HHsnM2yLiTVXzRDNktUnsG/u+i4BFAJlJf39/E6pVt3HcCBwHqnMcCMoaB+06meB1wCkRcTIwHXgB9Rm2/SKir5o1mwMMVv3XAAcCayKiD9gX2DD+TTNzCbCk2qwNDQ219lMw0OL3Vyfs2rhxLEw1uzIOak8tAmiqGBxauwuv8vtgqml9noCBgcmNm7YsfWbmJzJzTmYeDLwbWJWZ7wP+EXhn1e10YEX1/Jpqm2r/qszcYUat3Wr0+JiCD0mSStXp66j9J+CciLif+jFoS6v2pcCLqvZzgAs6VJ8kSVLH9NRqHZ+oapba4ODgzns9BwOzXeaYigbXNr7UMXu2Sx1Tzdq1jX9/+J0w9fh9INi174NGVUufO13W6fSMmiRJkp5BW+9MIEnSVOOxrlPPILtyUklrOKMmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklSoSQe1iPjLiOhvZTGSJEl6WiO3kDoB+HxE/Bj4OvC9zNzakqokSZI0+Rm1zDwFOAj4IfBx4P9FxBUR8cZWFSdJktTNemq12i69MCKOoj6zdiSwGvg74LLM/H3zymtIbXBwsKW/YGD27Ja+vzpjcG3jN9+dPXugBZWok9aubfz7w++EqWdXvg8cB1PProyDRg0MDAD07KxfI0ufAETE8cD7gQXAPwNfBB4CPkZ9tu0Njb6nJEmSdjTpoBYRXwLeDWwCrgY+nZlrx+y/BdjY9AolSZK6VCMzatOBf5uZP5toZ2b+KSLmNacsSZIkNRLU/iuweWxDRLwQmJGZgwCZeW8Ta5MkSepqjVzw9nvAnHFtc4DvNq8cSZIkjWokqP1ZZt45tqHaPqy5JUmSJAkaC2qPRMShYxuq7UebW5IkSZKgsWPUrgS+HRGfAh4A/gXwX4ArWlGYJElSt2skqC0G/gR8CTiQ+kVurwC+3IK6JEmSut6kg1pmjgB/XT0kSZLUYg3dmSAi/gx4BbD32PbMvLKZRUmSJKmxOxN8EvgscAfbX0+tRv34NUmSJDVRIzNqHweOycxftqoYSZIkPa2Ry3P8EfDOA5IkSW3SyIzaZ4D/HhEXAg+P3VGdaCBJkqQmaiSofa36+edj2nqoH6M2rVkFSZIkqa6RoHZIy6qQJEnSDhq5jtrvACKiFzggM9e1rCpJkiQ1dHmO/YC/Ad5J/Q4Fe0XEKdTPBP10i+qTJEnqWo2c9fk/gU3AQcCTVdtPgYXNLkqSJEmNBbXjgb+sljxrAJm5Hti/FYVJkiR1u0aC2iagf2xDRLwU8Fg1SZKkFmjkrM8rgG9HxKeA3oh4LfB56kuiUlep0dPpEtRkg6ztdAmStINGgtoXgC3AV4HdqN/f82+By1pQlyRJUtdr5PIcNeAr1UOSJEkt1sjlOeY/077MXNWcciRJkjSqkaXPpeO2XwzsDqwBXta0iiRJkgQ0tvS53S2kImIa8GngiWYXJUmSpMZm1LaTmdsi4mLqM2pffra+ETEduAnYo/qd38rMz0XEIcByYCZwO3BaZj4ZEXsAVwOvAh4FFmbmg7taqyRJ0vNRI9dRm8hbgJFJ9NsKzM/MVwBHAydGxLHUzyS9NDPnAhuBM6r+ZwAbM/NQ4NKqnyRJUldp5GSC1VR3JKjsCUwH/mJnr63OGP19tblb9agB84H3Vu3LgAuBy4EF1XOAbwH/IyJ6qveRJEnqCo0sfb5/3PYfgPsy8/HJvLg6pu024FDq12L7DfBYZg5XXdYAs6vns4HVAJk5HBGbgBcBQw3UK0mS9LzWyMkEP3kuvygztwFHR8R+wHeBwyfoNjpjNtFl33eYTYuIRcCi6v3p7+/f4UXSzjhuBI4D1TkOBGWNg0aWPr/OBGFpvMz8wE72PxYRPwaOBfaLiL5qVm0OMFh1WwMcCKyJiD5gX2DDBO+1BFhSbdaGhlo74TbQ0ndXp+zKuHEsTD2OA4HjQHWtzhMAAwOTGzmNnEzwGHAqMI16kOqlfizZY9SXMUcfO4iIF1czaUTEDOAE4B7gH4F3Vt1OB1ZUz6+ptqn2r/L4NEmS1G0aOUbtXwJvy8ybRxsi4vXAZzLzrTt57SxgWXWcWi+QmfmDiLgbWB4RFwE/5+mL6i4Fvh4R91OfSXt3A3VKkiRNCY0EtWOBW8a13Qq8dmcvzMxfAq+coP0B4JgJ2rcA72qgNkmSpCmnkaXPnwOfr5YuR5cwLwZ+0YrCJEmSul0jQe2DwOuATRHxMLAJeD1PH0smSZKkJmrk8hwPAv86Ig6kfpLLusx8qFWFSZIkdbuGbiEVES8C3gQcl5kPRcRARMxpSWWSJEldbtJBLSKOA34FvA/4TNU8l/otnyRJktRkjcyofQVYmJknAqO3fbqVCc7alCRJ0nPXSFA7ODNXVs9HLz77JI1d4kOSJEmT1EhQuzsixl/Y9gTgzibWI0mSpEojs2HnAj+IiGuBGRHxt8A7qN9GSpIkSU026Rm1zLwFOAq4C7gS+C1wTGb+rEW1SZIkdbVJzahV9+hcCbw1M7/Y2pIkSZIEk5xRy8xtwCGT7S9JkqTnrpFj1P4zcHlEfA5Yw9NnfpKZI80uTJIkqds1EtSuqH5+gKdDWk/1fFozi5IkSdIkljIj4iXV00PGPF5WPUafS5IkqckmM6N2H/CCzPwdQER8JzP/XWvLkiRJ0mRODugZt/2mFtQhSZKkcSYT1Go77yJJkqRmm8zSZ19EvJmnZ9bGb5OZq1pRnCRJUjebTFB7hPqdCEY9Om67hicUSJIkNd1Og1pmHtyGOiRJkjSOdxqQJEkqlEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQBjVJkqRCGdQkSZIKZVCTJEkqlEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQfe34JRFxIHA18BJgBFiSmZdFxEzgm8DBwINAZObGiOgBLgNOBjYDH8zM29tRqyRJUinaNaM2DJybmYcDxwJnRcQRwAXAysycC6ystgFOAuZWj0XA5W2qU5IkqRhtCWqZuW50RiwznwDuAWYDC4BlVbdlwKnV8wXA1ZlZy8xbgP0iYlY7apUkSSpF249Ri4iDgVcCtwIHZOY6qIc5YP+q22xg9ZiXranaJEmSukZbjlEbFRF7A98GPp6Zj0fEM3XtmaCtNsH7LaK+NEpm0t/f36xS1UUcNwLHgeocB4KyxkHbglpE7EY9pH0jM79TNT8cEbMyc121tPlI1b4GOHDMy+cAg+PfMzOXAEuqzdrQ0FBriq8MtPTd1Sm7Mm4cC1OP40DgOFBdq/MEwMDA5EZOu8767AGWAvdk5pfH7LoGOB1YXP1cMab9IxGxHHgNsGl0iVSSJKlbtGtG7XXAacCdEfGLqu2T1ANaRsQZwEPAu6p911G/NMf91C/P8aE21SlJklSMtgS1zPwnJj7uDOD4CfrXgLNaWpQkSVLhvDOBJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCZJklQog5okSVKhDGqSJEmFMqhJkiQVyqAmSZJUKIOaJElSoQxqkiRJhTKoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBWqrx2/JCKuBN4OPJKZR1ZtM4FvAgcDDwKRmRsjoge4DDgZ2Ax8MDNvb0edkiRJJWnXjNrXgBPHtV0ArMzMucDKahvgJGBu9VgEXN6mGiVJkorSlqCWmTcBG8Y1LwCWVc+XAaeOab86M2uZeQuwX0TMakedkiRJJenkMWoHZOY6gOrn/lX7bGD1mH5rqjZJkqSu0pZj1BrUM0FbbaKOEbGI+vIomUl/f38r69IU5bgROA5U5zgQlDUOOhnUHo6IWZm5rlrafKRqXwMcOKbfHGBwojfIzCXAkmqzNjQ01LJiAQZa+u7qlF0ZN46FqcdxIHAcqK7VeQJgYGByI6eTQe0a4HRgcfVzxZj2j0TEcuA1wKbRJVJJkqRu0q7Lc/w98CagPyLWAJ+jHtAyIs4AHgLeVXW/jvqlOe6nfnmOD7WjRkmSpNK0Jahl5nueYdfxE/StAWe1tiJJkqTyeWcCSZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQBjVJkqRCGdQkSZIKZVCTJEkqlEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQBjVJkqRCGdQkSZIKZVCTJEkqlEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQBjVJkqRCGdQkSZIKZVCTJEkqlEFNkiSpUAY1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSpUX6cLeCYRcSJwGTANuCIzF3e4JEmSpLYqckYtIqYBXwVOAo4A3hMRR3S2KkmSpPYqMqgBxwD3Z+YDmfkksBxY0OGaJEmS2qrUoDYbWD1me03VJkmS1DVKPUatZ4K22viGiFgELALITAYGBlpbVW2HEjQF7NKocSxMOY4DgeNAdS1OEw0pdUZtDXDgmO05wOD4Tpm5JDPnZeY86uHOR5MeEXFbp2vw0fmH48AHOA58OA5a+NipUmfUfgbMjYhDgLXAu4H3drYkSZKk9ipyRi0zh4GPAP8A3FNvyrs6W5UkSVJ7lTqjRmZeB1zX6Tq62JJOF6AiOA4EjgPVOQ46oKfmQZCSJElFKnLpU5IkSQUvfaozvHWXACLiSuDtwCOZeWSn61FnRMSBwNXAS4ARYElmXtbZqtRuETEduAnYg3pu+FZmfq6zVXUPZ9T0FG/dpTG+BpzY6SLUccPAuZl5OHAscJbfCV1pKzA/M18BHA2cGBHHdrimrmFQ01jeuksAZOZNwIZO16HOysx1mXl79fwJ6mfhe5eYLpOZtcz8fbW5W/XwAPc2celTY010667XdKgWSQWJiIOBVwK3drgUdUC14nIbcCjw1cx0HLSJM2oaa6KrJPuvJqnLRcTewLeBj2fm452uR+2Xmdsy82jqdwo6JiI8drVNDGoaa1K37pLUPSJiN+oh7RuZ+Z1O16POyszHgB/jMaxt49KnxvLWXZKeEhE9wFLgnsz8cqfrUWdExIuBP2XmYxExAzgB+EKHy+oaXvBW24mIk4GvUL88x5WZeXGHS1IHRMTfA28C+oGHgc9l5tKOFqW2i4jXAzcDd1K/PAfAJ6s7x6hLRMRRwDLqfy/0Ur+t4191tqruYVCTJEkqlMeoSZIkFcqgJkmSVCiDmiRJUqEMapIkSYUyqEmSJBXKoCap60XEhRHxvzpdhySN5wVvJXWNiHgvcA5wGPAE8AvAawVKKpZBTVJXiIhzgAuAM4F/AJ6kfhucBcAfOliaJD0jg5qkKS8i9gX+CvjQuPtVfh/4fkRcOK7//wbeAMwA7gA+nJl3VftOBr5E/b64jwOXZuaXIqIf+BrweupX8b8LOC4zR5CkXeQxapK6wWuB6cB3J9n/h8BcYH/gduAbY/YtBf5jZu4DHAmsqtrPBdYALwYOAD4JeOsXSc+JM2qSusGLgKHMHJ5M58y8cvR5Ndu2MSL2zcxNwJ+AIyLijszcCGysuv4JmAUclJn3U79HpiQ9JwY1Sd3gUaA/Ivp2FtYiYhr1EwzeRX12bHTpsh/YBPx74NPA4oj4JXBBZv4U+GvgQuD6iABYkpmLW/BZJHURlz4ldYOfAluAUyfR973UTzA4AdgXOLhq7wHIzJ9l5gLqy6LfA7JqfyIzz83MlwHvAM6JiOOb+SEkdR9n1CRNeZm5KSI+C3w1IoaB66kvVZ4AvBnYPKb7PsBW6rNwewKfH90REbtTn2n7QfWejwPbqn1vB+4FfkP9JINto/skaVc5oyapK2Tml6lfQ+3TwHpgNfAR6rNiY10N/A5YC9wN3DJu/2nAg1VIOxN4f9U+F7gR+D31Gby/ycwfN/2DSOoqPbWaJyVJkiSVyBk1SZKkQhnUJEmSCmVQkyRJKpRBTZIkqVAGNUmSpEIZ1CRJkgplUJMkSSqUQU2SJKlQBjVJkqRC/X8UwRPmXFuAcgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "plt.bar(x=train_df['class'].unique(), height=train_df['class'].value_counts(), width=0.90 , color='b')\n",
    "plt.bar(x=test_df['class'].unique(), height=test_df['class'].value_counts(), width=0.90, color='r');\n",
    "plt.xlabel('Class')\n",
    "plt.ylabel('Frequency')\n",
    "plt.xticks([0, 1, 2, 3]);\n",
    "plt.legend(('Train', 'Test'));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction \n",
    "\n",
    "#### Text Normalization by Lemmatization\n",
    "\n",
    "Stemming and Lemmatization are Text Normalization (or sometimes called Word Normalization) techniques in the field of Natural Language Processing that are used to prepare text, words, and documents for further processing.\n",
    "\n",
    "\n",
    "#### Lemmatize the training data,  stem  if it improves the classification accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training data\n",
    "train_df['lemmatized_text'] = train_df['text'].apply(lambda text: ' '.join([lemmatizer.lemmatize(word) \n",
    "                                                                      for word in word_tokenize(text)]))\n",
    "# testing data\n",
    "test_df['lemmatized_text'] = test_df['text'].apply(lambda text: ' '.join([lemmatizer.lemmatize(word) \n",
    "                                                                     for word in word_tokenize(text)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# original \n",
    "train_df['text'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'From : sd345 @ city.ac.uk ( Michael Collier ) Subject : Converting image to HP LaserJet III ? Nntp-Posting-Host : hampton Organization : The City University Lines : 14 Does anyone know of a good way ( standard PC application/PD utility ) to convert tif/img/tga file into LaserJet III format . We would also like to do the same , converting to HPGL ( HP plotter ) file . Please email any response . Is this the correct group ? Thanks in advance . Michael . -- Michael Collier ( Programmer ) The Computer Unit , Email : M.P.Collier @ uk.ac.city The City University , Tel : 071 477-8000 x3769 London , Fax : 071 477-8565 EC1V 0HB .'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lemmatized\n",
    "train_df['lemmatized_text'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stemming \n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "train_df['stemmed_text'] = train_df['text'].apply(lambda text: ' '.join([stemmer.stem(word) \n",
    "                                                                      for word in word_tokenize(text)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stemming on test data\n",
    "test_df['stemmed_text'] = test_df['text'].apply(lambda text: ' '.join([lemmatizer.lemmatize(word) \n",
    "                                                                     for word in word_tokenize(text)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction \n",
    "\n",
    "#### Text Preprocessing & Feature Vectorization\n",
    "\n",
    "We can combine text preprocessing, feature vectorization and model training using the sklearn Pipeline object. This Pipeline object can be used for model selection and for training the optimal model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Selection \n",
    "\n",
    "\n",
    "There are no hyperparameters in a NB model except the Laplace smoothing parameter alpha.\n",
    "\n",
    "However, there are multiple hyperparameters for the CountVectorizer() and TfidfTransformer(). We need to select the best model based on the optimal values of these hyperparameters. This process is called hyper-parameter tuning.\n",
    "\n",
    "For hyperparameter tuning, we will build a compund classifier using the sklearn Pipeline class. It will combine the CountVectorizer(), TfidfTransformer() and MultinomialNB() objects and will create a single object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Pipeline for Hyperparameter Tuning\n",
    "\n",
    "\n",
    "####  Build a Pipeline object by combining CountVectorizer() and MultinomialNB() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_clf_multinomialNB = Pipeline([\n",
    "        ('vect', CountVectorizer()),\n",
    "        ('clf', MultinomialNB()),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning\n",
    "\n",
    "#### hyperparamer tuning for the following hyperparameters\n",
    "- CountVectorizer()\n",
    "         -- ngram_range\n",
    "         -- stop_words\n",
    "- MultinomialNB()\n",
    "        -- alpha\n",
    "        \n",
    "### Important:\n",
    "\n",
    "The GridSearchCV takes an argument to define the scoring metric (performance measure). \n",
    "\n",
    "See the list of possible scoring functions:\n",
    "https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter\n",
    "\n",
    "For multiclass classification, we may use \"f1_micro\" scoring function. The f1_micro function is the average of the F1 score of each class with weighting depending on the average parameter.\n",
    "\n",
    "The macro-average (\"f1_macro\") will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average (\"f1_micro\") will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).\n",
    "\n",
    "In the binary classification, \"f1\" score function can be used. We may also use the precision_score, recall_score, roc_auc_score functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Best Score: 0.981391\n",
      "\n",
      "Optimal Hyperparameter Values: \n",
      "clf__alpha: 0.1\n",
      "vect__binary: False\n",
      "vect__ngram_range: (1, 2)\n",
      "vect__stop_words: 'english'\n",
      "\n",
      "Best Score:\n",
      "\n",
      "Optimal Hyperparameter Values: \n",
      "Wall time: 9min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "param_grid = {\n",
    "    'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "    'vect__stop_words': ['english', None],\n",
    "    'vect__binary': [True, False],\n",
    "    'clf__alpha': [0.1, 1.0, 1.5, 1.8]\n",
    "}\n",
    "\n",
    "clf_multinomial_CV = GridSearchCV(text_clf_multinomialNB, param_grid, scoring='f1_micro', cv=5)\n",
    "\n",
    "clf_multinomial_CV = clf_multinomial_CV.fit(train_df['lemmatized_text'].values, train_df['class'].values)\n",
    "\n",
    "\n",
    "print(\"\\nBest Score: %f\" % clf_multinomial_CV.best_score_)\n",
    "print(\"\\nOptimal Hyperparameter Values: \")\n",
    "\n",
    "for param_name in sorted(param_grid.keys()):\n",
    "    print(\"%s: %r\" % (param_name, clf_multinomial_CV.best_params_[param_name]))\n",
    "\n",
    "# best scores\n",
    "print(\"\\nBest Score:\")\n",
    "print(\"\\nOptimal Hyperparameter Values: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Train the Optimal Multinomial Model \n",
    "####  Using the optimal hyperparameter values, create the optimal model and then fit the model\n",
    "- Build a Pipeline object by combining CountVectorizer() and MultinomialNB()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=True, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words='english',\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)), ('clf', MultinomialNB(alpha=0.1, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# best model \n",
    "multinomialNB_clf = Pipeline([\n",
    "        ('vect', CountVectorizer(stop_words='english', ngram_range=(1, 2), binary=True)),\n",
    "        ('clf', MultinomialNB(alpha=0.1))])\n",
    "\n",
    "# training\n",
    "multinomialNB_clf.fit(train_df['lemmatized_text'].values, train_df['class'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=True, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words='english',\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)), ('clf', MultinomialNB(alpha=0.1, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# best model with stemming\n",
    "\n",
    "multinomialNB_clf_stemming = Pipeline([\n",
    "        ('vect', CountVectorizer(stop_words='english', ngram_range=(1, 2), binary=True)),\n",
    "        ('clf', MultinomialNB(alpha=0.1))])\n",
    "\n",
    "# training\n",
    "multinomialNB_clf_stemming.fit(train_df['stemmed_text'].values, train_df['class'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model on Test Data \n",
    "\n",
    "#### Evaluate the model on test data and generate\n",
    "- Confusion Matrix\n",
    "- Precision\n",
    "- Recall\n",
    "- F1 score\n",
    "- Classification Report\n",
    "\n",
    "\n",
    "#### Note: For multi-class classification, set the \"average\" attribute to \"micro\" for the following functions:\n",
    "- precision_score\n",
    "- recall_score\n",
    "- f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With Lemmatization\n",
      "\n",
      "Test Confusion Matrix:\n",
      " [[296   5   5  13]\n",
      " [  4 377   7   1]\n",
      " [  5  21 359  11]\n",
      " [  4   4   6 384]]\n",
      "\n",
      "Test Precision: 0.9427430093209055\n",
      "\n",
      "Test Recall: 0.9427430093209055\n",
      "\n",
      "Test F1 Score: 0.9427430093209055\n",
      "\n",
      "Classification Report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.93      0.94       319\n",
      "           1       0.93      0.97      0.95       389\n",
      "           2       0.95      0.91      0.93       396\n",
      "           3       0.94      0.96      0.95       398\n",
      "\n",
      "   micro avg       0.94      0.94      0.94      1502\n",
      "   macro avg       0.94      0.94      0.94      1502\n",
      "weighted avg       0.94      0.94      0.94      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# prediction on test data\n",
    "print('With Lemmatization')\n",
    "y_test_pred = multinomialNB_clf.predict(test_df['lemmatized_text'].values)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\\n\", confusion_matrix(test_df['class'].values, y_test_pred))\n",
    "print(\"\\nTest Precision:\", precision_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest Recall:\", recall_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest F1 Score:\", f1_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nClassification Report:\", classification_report(test_df['class'].values, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With stemming..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With Stemming\n",
      "\n",
      "Test Confusion Matrix:\n",
      " [[292   6   9  12]\n",
      " [  4 382   3   0]\n",
      " [  4  39 344   9]\n",
      " [  4   7   7 380]]\n",
      "\n",
      "Test Precision: 0.9307589880159787\n",
      "\n",
      "Test Recall: 0.9307589880159787\n",
      "\n",
      "Test F1 Score: 0.9307589880159787\n",
      "\n",
      "Classification Report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.92      0.94       319\n",
      "           1       0.88      0.98      0.93       389\n",
      "           2       0.95      0.87      0.91       396\n",
      "           3       0.95      0.95      0.95       398\n",
      "\n",
      "   micro avg       0.93      0.93      0.93      1502\n",
      "   macro avg       0.93      0.93      0.93      1502\n",
      "weighted avg       0.93      0.93      0.93      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# prediction on test data: with stemming\n",
    "print('With Stemming')\n",
    "y_test_stem = multinomialNB_clf_stemming.predict(test_df['stemmed_text'].values)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\\n\", confusion_matrix(test_df['class'].values, y_test_stem))\n",
    "print(\"\\nTest Precision:\", precision_score(test_df['class'].values, y_test_stem, average='micro'))\n",
    "print(\"\\nTest Recall:\", recall_score(test_df['class'].values, y_test_stem, average='micro'))\n",
    "print(\"\\nTest F1 Score:\", f1_score(test_df['class'].values, y_test_stem, average='micro'))\n",
    "print(\"\\nClassification Report:\", classification_report(test_df['class'].values, y_test_stem))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### No Lemmatization Results\n",
    "\n",
    "Test Confusion Matrix:\n",
    "\n",
    " [[297   3   6  13]\n",
    " [  4 375   9   1]\n",
    " [  5  19 360  12]\n",
    " [  4   2   7 385]]\n",
    "\n",
    "Test Precision: 0.9434087882822902\n",
    "\n",
    "Test Recall: 0.9434087882822902\n",
    "\n",
    "Test F1 Score: 0.9434087882822902\n",
    "\n",
    "Classification Report:               precision    recall  f1-score   support\n",
    "\n",
    "           0       0.96      0.93      0.94       319\n",
    "           1       0.94      0.96      0.95       389\n",
    "           2       0.94      0.91      0.93       396\n",
    "           3       0.94      0.97      0.95       398\n",
    "\n",
    "   micro avg       0.94      0.94      0.94      1502\n",
    "   \n",
    "   macro avg       0.94      0.94      0.94      1502\n",
    "   \n",
    "weighted avg       0.94      0.94      0.94      1502"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Multinomial NB: TF-IDF Model\n",
    "\n",
    "####  Implement the Multinomial model using the TF-IDF feature vectors \n",
    "- Build a Pipeline object by combining CountVectorizer(), TfidfTransformer() and MultinomialNB()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_clf_multinomialNB = Pipeline([\n",
    "        ('vect', CountVectorizer()),\n",
    "        ('tfidf', TfidfTransformer()),\n",
    "        ('clf', MultinomialNB()),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Best Score: 0.976961\n",
      "\n",
      "Optimal Hyperparameter Values: \n",
      "clf__alpha: 0.1\n",
      "tfidf__norm: 'l2'\n",
      "tfidf__use_idf: True\n",
      "vect__binary: True\n",
      "vect__ngram_range: (1, 2)\n",
      "vect__stop_words: 'english'\n",
      "\n",
      "Best Score:\n",
      "\n",
      "Optimal Hyperparameter Values: \n",
      "Wall time: 1min 34s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# use best parameters obtained for rest: search only for tfidf hyperparameters\n",
    "param_grid = {\n",
    "    'vect__ngram_range': [(1, 2)],\n",
    "    'vect__stop_words': ['english'],\n",
    "    'vect__binary': [True],\n",
    "    'clf__alpha': [0.1],\n",
    "    'tfidf__use_idf': (True, False),\n",
    "    'tfidf__norm': ('l1', 'l2'),\n",
    "}\n",
    "\n",
    "clf_multinomial_CV = GridSearchCV(text_clf_multinomialNB, param_grid, scoring='f1_micro', cv=5)\n",
    "\n",
    "clf_multinomial_CV = clf_multinomial_CV.fit(train_df['lemmatized_text'].values, train_df['class'].values)\n",
    "\n",
    "\n",
    "print(\"\\nBest Score: %f\" % clf_multinomial_CV.best_score_)\n",
    "print(\"\\nOptimal Hyperparameter Values: \")\n",
    "\n",
    "for param_name in sorted(param_grid.keys()):\n",
    "    print(\"%s: %r\" % (param_name, clf_multinomial_CV.best_params_[param_name]))\n",
    "\n",
    "# best scores\n",
    "print(\"\\nBest Score:\")\n",
    "print(\"\\nOptimal Hyperparameter Values: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=True, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words='english',\n",
       "        s...inear_tf=False, use_idf=True)), ('clf', MultinomialNB(alpha=0.1, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# best model \n",
    "\n",
    "multinomialNB_clf = Pipeline([\n",
    "        ('vect', CountVectorizer(stop_words='english', ngram_range=(1, 2), binary=True)),\n",
    "        ('tfidf', TfidfTransformer(norm='l2', use_idf=True)),\n",
    "        ('clf', MultinomialNB(alpha=0.1)),    \n",
    "])\n",
    "\n",
    "# training\n",
    "multinomialNB_clf.fit(train_df['lemmatized_text'].values, train_df['class'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model on Test Data \n",
    "\n",
    "#### Evaluating the model on test data and generate \n",
    "- Confusion Matrix\n",
    "- Precision\n",
    "- Recall\n",
    "- F1 score\n",
    "- Classification Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Confusion Matrix:\n",
      " [[276   3  11  29]\n",
      " [  1 373   8   7]\n",
      " [  5  21 353  17]\n",
      " [  3   3   6 386]]\n",
      "\n",
      "Test Precision: 0.9241011984021305\n",
      "\n",
      "Test Recall: 0.9241011984021305\n",
      "\n",
      "Test F1 Score: 0.9241011984021305\n",
      "\n",
      "Classification Report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.87      0.91       319\n",
      "           1       0.93      0.96      0.95       389\n",
      "           2       0.93      0.89      0.91       396\n",
      "           3       0.88      0.97      0.92       398\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      1502\n",
      "   macro avg       0.93      0.92      0.92      1502\n",
      "weighted avg       0.93      0.92      0.92      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# prediction on test data\n",
    "y_test_pred = multinomialNB_clf.predict(test_df['lemmatized_text'].values)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\\n\", confusion_matrix(test_df['class'].values, y_test_pred))\n",
    "print(\"\\nTest Precision:\", precision_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest Recall:\", recall_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest F1 Score:\", f1_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nClassification Report:\", classification_report(test_df['class'].values, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### No Lemmatization Results\n",
    "\n",
    "Test Confusion Matrix:\n",
    " [[276   3  11  29]\n",
    " [  1 372   8   8]\n",
    " [  4  22 353  17]\n",
    " [  4   3   6 385]]\n",
    "\n",
    "Test Precision: 0.9227696404793608\n",
    "\n",
    "Test Recall: 0.9227696404793608\n",
    "\n",
    "Test F1 Score: 0.9227696404793608\n",
    "\n",
    "Classification Report:               precision    recall  f1-score   support\n",
    "\n",
    "           0       0.97      0.87      0.91       319\n",
    "           1       0.93      0.96      0.94       389\n",
    "           2       0.93      0.89      0.91       396\n",
    "           3       0.88      0.97      0.92       398\n",
    "\n",
    "   micro avg       0.92      0.92      0.92      1502\n",
    "   \n",
    "   macro avg       0.93      0.92      0.92      1502\n",
    "   \n",
    "weighted avg       0.93      0.92      0.92      1502\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Observation on Multinomial Model With TF-IDF Feature Vectors\n",
    "We observe that both precision and recall decrease with TF-IDF feature vectors. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Multivariate Bernoulli NB\n",
    "\n",
    "#### Task 10: Implement the Multivariate Bernoulli Model (10 pts)\n",
    "- Build a Pipeline object by combining CountVectorizer() and BernoulliNB()\n",
    "\n",
    "#### Note\n",
    "The \"binary\" attribute of the CountVectorizer() object should be set to \"True\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=True, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words='english',\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)), ('clf', BernoulliNB(alpha=0.1, binarize=0.0, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pipeline\n",
    "bernoulliNB_clf = Pipeline([\n",
    "        ('vect', CountVectorizer(stop_words='english', ngram_range=(1, 2), binary=True)),\n",
    "        ('clf', BernoulliNB(alpha=0.1)),\n",
    "    ])\n",
    "\n",
    "bernoulliNB_clf.fit(train_df['lemmatized_text'].values, train_df['class'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model on Test Data \n",
    "\n",
    "\n",
    "#### Evaluate the model on test data and generate \n",
    "- Confusion Matrix\n",
    "- Precision\n",
    "- Recall\n",
    "- F1 score\n",
    "- Classification Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Confusion Matrix:\n",
      " [[285  19   6   9]\n",
      " [  2 384   2   1]\n",
      " [  3 106 286   1]\n",
      " [  3  36   1 358]]\n",
      "\n",
      "Test Precision: 0.874167776298269\n",
      "\n",
      "Test Recall: 0.874167776298269\n",
      "\n",
      "Test F1 Score: 0.874167776298269\n",
      "\n",
      "Classification Report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.89      0.93       319\n",
      "           1       0.70      0.99      0.82       389\n",
      "           2       0.97      0.72      0.83       396\n",
      "           3       0.97      0.90      0.93       398\n",
      "\n",
      "   micro avg       0.87      0.87      0.87      1502\n",
      "   macro avg       0.90      0.88      0.88      1502\n",
      "weighted avg       0.90      0.87      0.88      1502\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# prediction on test data\n",
    "y_test_pred = bernoulliNB_clf.predict(test_df['lemmatized_text'].values)\n",
    "\n",
    "print(\"\\nTest Confusion Matrix:\\n\", confusion_matrix(test_df['class'].values, y_test_pred))\n",
    "print(\"\\nTest Precision:\", precision_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest Recall:\", recall_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nTest F1 Score:\", f1_score(test_df['class'].values, y_test_pred, average='micro'))\n",
    "print(\"\\nClassification Report:\", classification_report(test_df['class'].values, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### No Lemmatization Results\n",
    "\n",
    "Test Confusion Matrix:\n",
    " [[288  18   5   8]\n",
    " [  2 385   2   0]\n",
    " [  3 107 284   2]\n",
    " [  3  36   1 358]]\n",
    "\n",
    "Test Precision: 0.8754993342210386\n",
    "\n",
    "Test Recall: 0.8754993342210386\n",
    "\n",
    "Test F1 Score: 0.8754993342210386\n",
    "\n",
    "Classification Report:               precision    recall  f1-score   support\n",
    "\n",
    "           0       0.97      0.90      0.94       319\n",
    "           1       0.71      0.99      0.82       389\n",
    "           2       0.97      0.72      0.83       396\n",
    "           3       0.97      0.90      0.93       398\n",
    "\n",
    "   micro avg       0.88      0.88      0.88      1502\n",
    "   \n",
    "   macro avg       0.91      0.88      0.88      1502\n",
    "   \n",
    "weighted avg       0.90      0.88      0.88      1502"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Observation on Multivariate Bernoulli Model \n",
    "\n",
    "#### Summerize:\n",
    "- Impact of data normalization technique.\n",
    "- Which classifier gave the best precision? Best recall? Best F1 Score? Explain their performance variance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above experimentations we conclude the following:\n",
    "\n",
    "- Data normalization:\n",
    "\n",
    "We checked the performance of the model both with lemmatization and without lemmatization. From the experiments, it is observed that the performance of the model is not affected that much by the lemmatization. As listed the results above, the performance of the models are changed by a very very small amount and the change is not consistent (increase or decrease) in the three different models we checked above. So, the conclusion is that eventhough the lemmatization affects the results, it is not affected significantly in this case. \n",
    "\n",
    "However, with stemming of the text, we find the performance of the model suffered a little bit. we find the consistent decrease in the performance scores of the models by about a percent.\n",
    "\n",
    "- Performance Variance\n",
    "\n",
    "The multinomial model without TfidfTransformer gives the best performances for all metrics: precision, recall and f1_score than multinoial model with TfidfTransformer and multinomial Bernoulli model. Also, one important to notice is that the precision, recall and f1_scores of each of the model are almost equal implying that each of the model is not biased. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Done!***"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}