notebooks/TextAnalytics - Amazon Book Reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## TextAnalytics - Amazon Book Reviews\n",
    "\n",
    "Again, try to predict Amazon book ratings greater than 3 out of 5, this time using\n",
    "the `TextFeaturizer` module which is a composition of several text analytics APIs that\n",
    "are native to Spark."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import os\r\n",
    "if os.environ.get(\"AZURE_SERVICE\", None) == \"Microsoft.ProjectArcadia\":\r\n",
    "    from pyspark.sql import SparkSession\r\n",
    "    spark = SparkSession.builder.getOrCreate()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import pandas as pd"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "data = spark.read.parquet(\"wasbs://publicwasb@mmlspark.blob.core.windows.net/BookReviewsFromAmazon10K.parquet\")\r\n",
    "data.limit(10).toPandas()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Use `TextFeaturizer` to generate our features column.  We remove stop words, and use TF-IDF\n",
    "to generate 2²⁰ sparse features."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from mmlspark.featurize.text import TextFeaturizer\r\n",
    "textFeaturizer = TextFeaturizer() \\\r\n",
    "  .setInputCol(\"text\").setOutputCol(\"features\") \\\r\n",
    "  .setUseStopWordsRemover(True).setUseIDF(True).setMinDocFreq(5).setNumFeatures(1 << 16).fit(data)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "processedData = textFeaturizer.transform(data)\r\n",
    "processedData.limit(5).toPandas()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Change the label so that we can predict whether the rating is greater than 3 using a binary\n",
    "classifier."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "processedData = processedData.withColumn(\"label\", processedData[\"rating\"] > 3) \\\r\n",
    "                             .select([\"features\", \"label\"])\r\n",
    "processedData.limit(5).toPandas()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Train several Logistic Regression models with different regularizations."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "train, test, validation = processedData.randomSplit([0.60, 0.20, 0.20])\r\n",
    "from pyspark.ml.classification import LogisticRegression\r\n",
    "\r\n",
    "lrHyperParams = [0.05, 0.1, 0.2, 0.4]\r\n",
    "logisticRegressions = [LogisticRegression(regParam = hyperParam) for hyperParam in lrHyperParams]\r\n",
    "\r\n",
    "from mmlspark.train import TrainClassifier\r\n",
    "lrmodels = [TrainClassifier(model=lrm, labelCol=\"label\").fit(train) for lrm in logisticRegressions]"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Find the model with the best AUC on the test set."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from mmlspark.automl import FindBestModel, BestModel\r\n",
    "bestModel = FindBestModel(evaluationMetric=\"AUC\", models=lrmodels).fit(test)\r\n",
    "bestModel.getRocCurve().show()\r\n",
    "bestModel.getBestModelMetrics().show()\r\n",
    "bestModel.getAllModelMetrics().show()\r\n"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Use the optimized `ComputeModelStatistics` API to find the model accuracy."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from mmlspark.train import ComputeModelStatistics\r\n",
    "predictions = bestModel.transform(validation)\r\n",
    "metrics = ComputeModelStatistics().transform(predictions)\r\n",
    "print(\"Best model's accuracy on validation set = \"\r\n",
    "      + \"{0:.2f}%\".format(metrics.first()[\"accuracy\"] * 100))"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}