3_ChoosingModel_MLPs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3-ChoosingModel-MLPs.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "jDm_GY9EMAbB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Deep Feedforward Network\n",
        "A Deep Feedforward Network, also called Feedforward Neural Network, or Multi-layer perceptrons (MLPs).\n",
        "<img src=\"https://www.learnopencv.com/wp-content/uploads/2017/10/mlp-diagram.jpg\" width=\"700px\">\n",
        "\n",
        "Among the layers, you can distinguish an input layer, hidden layers and an output layer. In general, there can be multiple hidden layers. \n",
        "\n",
        "Each node in the layer is a Neuron. An Artifical Neuron is the basic unit of a neural network.\n",
        "\n",
        "Multi-layer perceptrons are often fully connected. This means that there’s a connection from each perceptron in a certain layer to each perceptron in the next layer."
      ]
    },
    {
      "metadata": {
        "id": "CWMueMZZDehi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### NOTE:\n",
        "Basicaly this notebook prepared to use within **Google Colab**: https://colab.research.google.com/. \n",
        "\n",
        "The Google Colabatory has **free Tesla K80 GPU** and already prepared to develop deep learning applications.\n",
        "\n",
        "First time opens this notebook, do not forget to enable **Python 3** runtime and **GPU** accelerator in Google Colab **Notebook Settings**. \n"
      ]
    },
    {
      "metadata": {
        "id": "nzTTLgFtfq5o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Setup Project\n",
        "Create workspace and change directory.\n"
      ]
    },
    {
      "metadata": {
        "id": "-PGvIffmfq5u",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4fce247b-4211-4f3a-d8e2-06150bb43190",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523653966587,
          "user_tz": -180,
          "elapsed": 2141,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "PROJECT_HOME = '/content/keras-movie-reviews-classification'\n",
        "\n",
        "import os.path\n",
        "if not os.path.exists(PROJECT_HOME):\n",
        "  os.makedirs(PROJECT_HOME)\n",
        "os.chdir(PROJECT_HOME)\n",
        "\n",
        "!pwd"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/keras-movie-reviews-classification\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "e9fTIbvuGvMU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Import Project\n",
        "Import GitHub project to workspace."
      ]
    },
    {
      "metadata": {
        "id": "ZSe5_mzWl8ma",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 259
        },
        "outputId": "12121971-08db-4ed9-b201-9c571b4cf863",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523653981692,
          "user_tz": -180,
          "elapsed": 11797,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Import project and override existing data.\n",
        "!git init .\n",
        "!git remote add -t \\* -f origin https://github.com/alex-agency/keras-movie-reviews-classification.git\n",
        "!git reset --hard origin/master\n",
        "!git checkout\n",
        "\n",
        "!ls -la input"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initialized empty Git repository in /content/keras-movie-reviews-classification/.git/\n",
            "Updating origin\n",
            "remote: Counting objects: 22, done.\u001b[K\n",
            "remote: Compressing objects: 100% (17/17), done.\u001b[K\n",
            "remote: Total 22 (delta 4), reused 17 (delta 3), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (22/22), done.\n",
            "From https://github.com/alex-agency/keras-movie-reviews-classification\n",
            " * [new branch]      master     -> origin/master\n",
            "HEAD is now at 10b2912 modified dataset and updated explanations\n",
            "total 35692\n",
            "drwxr-xr-x 2 root root     4096 Apr 13 21:12 .\n",
            "drwxr-xr-x 4 root root     4096 Apr 13 21:12 ..\n",
            "-rw-r--r-- 1 root root 17163254 Apr 13 21:12 dataset.npz\n",
            "-rw-r--r-- 1 root root 19372046 Apr 13 21:12 reviews.tsv.bz2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "vtNNx3xGouq9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Load Dataset"
      ]
    },
    {
      "metadata": {
        "id": "rsIfxfvdoxY2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 138
        },
        "outputId": "0c73281c-6c4b-4b8f-b5bf-f25b848df975",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523653998958,
          "user_tz": -180,
          "elapsed": 1200,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "# load data from numpy array\n",
        "loaded = np.load('input/dataset.npz')\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = loaded['dataset']\n",
        "dictionary = loaded['dictionary'].item()\n",
        "label_encoder = loaded['label_encoder'].item()\n",
        "\n",
        "print('Vocabulary: {} words.'.format(len(dictionary)))\n",
        "print('Categorical classes: {}\\n'.format(list(label_encoder.classes_)))\n",
        "\n",
        "print('x_train shape:', x_train.shape)\n",
        "print('y_train shape:', y_train.shape)\n",
        "\n",
        "print('x_test shape:', x_test.shape)\n",
        "print('y_test shape:', y_test.shape)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary: 5000 words.\n",
            "Categorical classes: ['negative', 'positive']\n",
            "\n",
            "x_train shape: (40000, 400)\n",
            "y_train shape: (40000, 2)\n",
            "x_test shape: (10000, 400)\n",
            "y_test shape: (10000, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y-YXXsj-A_28",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Keras\n",
        "Keras is a high-level API, written in Python and capable of running on top of TensorFlow, Theano, or CNTK deep learning frameworks.\n",
        "Keras provides a simple and modular API to create and train Neural Networks, hiding most of the complicated details under the hood.\n",
        "By default, Keras is configured to use Tensorflow as the backend since it is the most popular choice.\n",
        "Keras is becoming super popular recently because of its simplicity.\n",
        "### Keras workflow\n",
        "<img src=\"https://www.learnopencv.com/wp-content/uploads/2017/09/keras-workflow.jpg\" width=\"700px\">"
      ]
    },
    {
      "metadata": {
        "id": "rcQa7qPNwxdp",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "95bea097-c537-466f-aa5f-1dca1eb7fd48",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523654011710,
          "user_tz": -180,
          "elapsed": 7727,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Load Keras libraries\n",
        "from keras.models import Sequential\n",
        "from keras.layers.embeddings import Embedding\n",
        "from keras.layers import Dense, Flatten, Dropout"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "7dIqL0Ro86Sc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Create Deep Feedforward Network model"
      ]
    },
    {
      "metadata": {
        "id": "UHPwK1ET06Wa",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 345
        },
        "outputId": "b59e9f5e-5973-4043-b09a-a0928629e4f7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523654014934,
          "user_tz": -180,
          "elapsed": 859,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Model parameters\n",
        "MAX_NB_WORDS = len(dictionary) # vocabulary size\n",
        "MAX_SEQUENCE_LENGTH = x_train.shape[1] # review length\n",
        "MAX_NB_LABELS = y_train.shape[1] # feedback legth\n",
        "EMBEDDING_SIZE = 1 # embedding layer dimension\n",
        "HIDDEN_SIZE = (MAX_SEQUENCE_LENGTH*EMBEDDING_SIZE//2) # hidden layer dimension\n",
        "DROPOUT_RATE = 0.2 # 20% dropout rate, use a small dropout value of 20%-50% of neurons.\n",
        "\n",
        "# Initialise neural network model.\n",
        "# The Sequential model is a linear stack of layers.\n",
        "# It allows to create models layer-by-layer.\n",
        "# Keras provides two ways to define a model: Sequential and Functional API.\n",
        "# Functional API model used for designing complex model architectures.\n",
        "model = Sequential()\n",
        "\n",
        "# This is the first layer. It is used to provide the input data or features to the network.\n",
        "# Embedding layer turns positive integers (indexes) into dense vectors of fixed size.\n",
        "model.add(Embedding(input_dim=MAX_NB_WORDS, # Number of features\n",
        "                    output_dim=EMBEDDING_SIZE, # Dimension of the dense embedding\n",
        "                    input_length=MAX_SEQUENCE_LENGTH)) # Length of input sequences\n",
        "\n",
        "# Flatten shape from 3 to 2 dimensions to fit with output layer.\n",
        "model.add(Flatten())\n",
        "\n",
        "# The Feedforward network in which all the neurons from one layer are connected to the neurons in the previous layer.\n",
        "\n",
        "# Hidden layer of the network.\n",
        "# Dense layer is a fully connected layer, since, each node in the input is connected to every node in the output.  \n",
        "model.add(Dense(HIDDEN_SIZE, # number of neurons in the hidden layer\n",
        "                activation='relu')) # the ReLU activation function which introduce non-linearity.\n",
        "\n",
        "# Dropout layer prevents network from overfitting.\n",
        "model.add(Dropout(DROPOUT_RATE))\n",
        "\n",
        "# Output layer, which gives out the predictions.\n",
        "# The activation function to be used in this layer is different for different problems.\n",
        "# For binary classification, we can use sigmoid activation.\n",
        "model.add(Dense(MAX_NB_LABELS, # Number of labels which is used in prediction\n",
        "                activation='sigmoid')) # Sigmoid activation function\n",
        "\n",
        "print(\"Model Summary\")\n",
        "print(model.summary())"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model Summary\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_1 (Embedding)      (None, 400, 1)            5000      \n",
            "_________________________________________________________________\n",
            "flatten_1 (Flatten)          (None, 400)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 200)               80200     \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 200)               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 2)                 402       \n",
            "=================================================================\n",
            "Total params: 85,602\n",
            "Trainable params: 85,602\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cKTKWb-FxIh_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Activation Functions\n",
        "\n",
        "The activation function is used as a decision making body at the output of a neuron. The neuron learns Linear or Non-linear decision boundaries based on the activation function. \n",
        "\n",
        "It also has a normalizing effect on the neuron output which prevents the output of neurons after several layers to become very large, due to the cascading effect. \n",
        "\n",
        "There are three most widely used activation functions: \n",
        "\n",
        "**Sigmoid** - It maps the input (x axis) to values between 0 and 1.\n",
        "\n",
        "**Tanh** - It is similar to the sigmoid function butmaps the input to values between -1 and 1.\n",
        "\n",
        "**Rectified Linear Unit (ReLU)** - It allows only positive values to pass through it. The negative values are mapped to zero.\n",
        "\n",
        "### Dropout\n",
        "During training, when dropout is applied to a layer, some percentage of its neurons (a hyperparameter, with common values being between 20 and 50%) are randomly deactivated or “dropped out,” along with their connections. Which neurons are dropped out are constantly shuffled randomly during training.  This forces the network to learn a more balanced representation, and helps with overfitting.\n",
        "\n",
        "<img src=\"https://ml4a.github.io/images/figures/dropout.png\" width=\"500px\">"
     ]
    },
    {
      "metadata": {
        "id": "zTjJ7YfKf4CF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Export model to file\n",
        "Saving whole Keras model into a single HDF5 file which will contain:\n",
        "* the architecture of the model, allowing to re-create the model\n",
        "* the weights of the model\n",
        "* the training configuration (loss, optimizer)\n",
        "* the state of the optimizer, allowing to resume training exactly where you left off.\n"
      ]
    },
    {
      "metadata": {
        "id": "UUWulpxTfyxV",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "98e91800-6666-4445-b52a-fb864270cd96",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1523654023791,
          "user_tz": -180,
          "elapsed": 2276,
          "user": {
            "displayName": "Alex Z",
            "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
            "userId": "110711877726297594950"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Model filename\n",
        "model_filename = 'mlps-model-definition.h5'\n",
        "\n",
        "model_file = os.path.join('input', model_filename)\n",
        "\n",
        "# Export model into HDF5 file.\n",
        "model.save(model_file)\n",
        "\n",
        "!ls -la input"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 36044\r\n",
            "drwxr-xr-x 2 root root     4096 Apr 13 21:13 .\r\n",
            "drwxr-xr-x 4 root root     4096 Apr 13 21:12 ..\r\n",
            "-rw-r--r-- 1 root root 17163254 Apr 13 21:12 dataset.npz\r\n",
            "-rw-r--r-- 1 root root   360416 Apr 13 21:13 mlps-model-definition.h5\r\n",
            "-rw-r--r-- 1 root root 19372046 Apr 13 21:12 reviews.tsv.bz2\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "R1qt-FhknGPK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Downloading file to your local file system\n",
        "\n",
        "It will invoke a browser download of the file to your local computer."
      ]
    },
    {
      "metadata": {
        "id": "vOWD80y7nGPM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "# Download file\n",
        "files.download(model_file)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}