Lab5_DocumentingYourResults.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab5_DocumentingYourResults.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L6qNPvACnxJj"
      },
      "source": [
        "#Documenting your results\n",
        "This Lab teaches you some of the basics of documenting machine learning research.\n",
        "\n",
        "Your task is to:\n",
        "\n",
        "1. Download and describe the CIFAR10 image classification dataset.\n",
        "2. Design and describe a small CNN that can solve the CIFAR10 problem.\n",
        "3. Train your model and explain how you trained it.\n",
        "4. Summarize your results using\n",
        " - Test set error (top-1 and top-5)\n",
        " - Test set accuracy\n",
        " - Confusion matrix\n",
        " - Precision/recall and Average Precision\n",
        "\n",
        "We will be using Keras and [scikit-learn](https://scikit-learn.org/stable/index.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v-5dmxzar30b"
      },
      "source": [
        "##Task 1: The dataset\n",
        "Describing your dataset is important. Here is a list of things that could be of interest to the reader:\n",
        "\n",
        "- What are you trying to predict? \n",
        "- Where did the dataset come from? (Remember to cite if its a public dataset)\n",
        "- How was it collected?\n",
        "- Why was it collected?\n",
        "- Why did you choose this dataset, and not that one over there?\n",
        "- What is the output: categorial {0, 1, ..., K}, continuous scalar in [0, 1], arbitrary real no., etc.\n",
        "- No. of classes, what are the classes?\n",
        "- No. of observations, no. of observations per class if unbalanced.\n",
        "- What is the size of the training set?\n",
        "- Is there a test set? What is its size?\n",
        "- Is there a baseline result that you can compare your results with?\n",
        "- What is the state-of-the-art performance on this data set?\n",
        "\n",
        "Finally, it might also be a good idea to show some actual observations/examples from the dataset.\n",
        "\n",
        "###1.1 Download"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KvE-RNCenk31"
      },
      "source": [
        "from __future__ import print_function\n",
        "import keras\n",
        "from keras.datasets import cifar10\n",
        "\n",
        "# The data, split between train and test sets:\n",
        "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
        "\n",
        "# Uncomment below to convert class vectors to binary class matrices.\n",
        "# num_classes = ??? # Number of classes\n",
        "#y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "#y_test = keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fD5YritnwSya"
      },
      "source": [
        "###1.2 Questions\n",
        "Try to answer as many of the above questions as possible (you don't need to write your answers down). You will be able to find many answers just by looking at the original source of the CIFAR 10 dataset:\n",
        "\n",
        "https://www.cs.toronto.edu/~kriz/cifar.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yVIxMin4xsp6"
      },
      "source": [
        "##Task 2: Network Architecture\n",
        "So you are faced with a machine lerning problem. Which model should you use? \n",
        "\n",
        "Well, that depends on the type of problem (classification, regression, clustering, etc.) and what the success criteria are (high accuracy, high speed, understanding structure in the data, etc).\n",
        "\n",
        "###2.1 Design a CNN\n",
        "Your task is design a CNN that solves the CIFAR10 problem. Motivate your choice of architecture and hyperparameters (number of layers, number of neurons/kernels in each layer, etc.) and regularization techniques.\n",
        "\n",
        "I left a template below that you could use - you just need to fill in the gaps (marked with `???`). Feel free to design your own network (look for inspiration in [Lab 2](https://github.com/aivclab/dlcourse/blob/master/Lab2_Solution.ipynb) or [Lab 3](https://github.com/aivclab/dlcourse/blob/master/Lab3_Solution.ipynb)) or search the internet for alternative models. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tiS6U1gfxrpW"
      },
      "source": [
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Activation, Flatten\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Conv2D(???, (3, 3), padding='same', input_shape=???))\n",
        "model.add(Activation('relu'))\n",
        "model.add(Conv2D(???, (3, 3)))\n",
        "model.add(Activation('relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Conv2D(???, (3, 3), padding='same'))\n",
        "model.add(Activation('relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "model.add(Flatten())\n",
        "model.add(Dense(???))\n",
        "model.add(Activation('relu'))\n",
        "model.add(Dropout(???))\n",
        "model.add(Dense(???))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nhcd2J921_SG"
      },
      "source": [
        "Always remember to summarize your model in your report. This is easily done in Keras:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rdb5usqJsX4j"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0TcXX0xo25tQ"
      },
      "source": [
        "##Task 3: Model Training\n",
        "Train your model and explain how you trained it. This includes:\n",
        "\n",
        "- Data preprocessing\n",
        "- Data augmentation?\n",
        "- Train/validation split\n",
        "- Choice of optimizer (not covered in lectures yet) and its hyperparameters, including\n",
        "\n",
        " - learning rate\n",
        " - number of training epochs.\n",
        " - batch size\n",
        "\n",
        "Here is a template that you can use (again, gaps are marked with ???).\n",
        "\n",
        "**Note:** The template code assumes that you already have a validation set. We will be using the test set provided with CIFAR10 as the validation set. Formally this is not the correct way to use a test set. Instead the validation set should be a random subset of training data (recall that the validation set serves as \"unseen data\" that you are allowed to use for optimizing your hyperparameters). Then, ONLY after you are done training your final model, you are allowed to evaluate it on your test set. If you wish to use a subset of the training data for validation, there is code for that in [Lab 2](https://github.com/aivclab/dlcourse/blob/master/Lab2_Solution.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kql5Si5E3wkU"
      },
      "source": [
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "# Initiate RMSprop optimizer\n",
        "# (Note: this choice is somewhat random - see option shere: https://keras.io/api/optimizers/)\n",
        "opt = keras.optimizers.RMSprop(learning_rate=???, decay=1e-6)\n",
        "\n",
        "# Let's train the model using RMSprop\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer=opt,\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Data preprocessing (normalization)\n",
        "# (Note: Consider zero-centering the data - how?)\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "data_augmentation = ???\n",
        "epochs = ???\n",
        "batch_size = ???\n",
        "\n",
        "if not data_augmentation:\n",
        "    print('Not using data augmentation.')\n",
        "    model.fit(x_train, y_train,\n",
        "              batch_size=batch_size,\n",
        "              epochs=epochs,\n",
        "              validation_data=(x_test, y_test),\n",
        "              shuffle=True)\n",
        "else:\n",
        "    print('Using real-time data augmentation.')\n",
        "    # This will do preprocessing and realtime data augmentation:\n",
        "    datagen = ImageDataGenerator(\n",
        "        featurewise_center=False,  # set input mean to 0 over the dataset\n",
        "        samplewise_center=False,  # set each sample mean to 0\n",
        "        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n",
        "        samplewise_std_normalization=False,  # divide each input by its std\n",
        "        zca_whitening=False,  # apply ZCA whitening\n",
        "        zca_epsilon=1e-06,  # epsilon for ZCA whitening\n",
        "        rotation_range=???,  # randomly rotate images in the range (degrees, 0 to 180)\n",
        "        # randomly shift images horizontally (fraction of total width)\n",
        "        width_shift_range=0.1,\n",
        "        # randomly shift images vertically (fraction of total height)\n",
        "        height_shift_range=0.1,\n",
        "        shear_range=0.,  # set range for random shear\n",
        "        zoom_range=0.,  # set range for random zoom\n",
        "        channel_shift_range=0.,  # set range for random channel shifts\n",
        "        # set mode for filling points outside the input boundaries\n",
        "        fill_mode='nearest',\n",
        "        cval=0.,  # value used for fill_mode = \"constant\"\n",
        "        horizontal_flip=???,  # randomly flip images\n",
        "        vertical_flip=False,  # randomly flip images\n",
        "        # set rescaling factor (applied before any other transformation)\n",
        "        rescale=None,\n",
        "        # set function that will be applied on each input\n",
        "        preprocessing_function=None,\n",
        "        # image data format, either \"channels_first\" or \"channels_last\"\n",
        "        data_format=None,\n",
        "        # fraction of images reserved for validation (strictly between 0 and 1)\n",
        "        validation_split=0)\n",
        "\n",
        "    # Compute quantities required for feature-wise normalization\n",
        "    # (std, mean, and principal components if ZCA whitening is applied).\n",
        "    datagen.fit(x_train)\n",
        "\n",
        "    # Fit the model on the batches generated by datagen.flow().\n",
        "    model.fit_generator(datagen.flow(x_train, y_train,\n",
        "                                     batch_size=batch_size),\n",
        "                        epochs=epochs,\n",
        "                        validation_data=(x_test, y_test),\n",
        "                        workers=4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pjfGbfnuu3yy"
      },
      "source": [
        "##Task 4: Summarizing your results\n",
        "Summarize your results using\n",
        " - Test set error (top-1 and top-5)\n",
        " - Test set accuracy\n",
        " - Confusion matrix\n",
        " - Precision/recall curve\n",
        " - Average Precision\n",
        "\n",
        "You might find the code below useful:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A86s9G5Xu8vb"
      },
      "source": [
        "# Evaluate the model on the test data using `evaluate`\n",
        "print('\\n# Evaluate on test data')\n",
        "results = model.evaluate(x_test, y_test, batch_size=???)\n",
        "print('test loss, test acc:', results)\n",
        "\n",
        "# Generate predictions (probabilities -- the output of the last layer)\n",
        "# on new data using `predict`\n",
        "N = ???\n",
        "print('\\n# Generate predictions for N samples')\n",
        "predictions = model.predict(x_test[:N])\n",
        "print('predictions shape:', predictions.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dxNr7Wk-odUH"
      },
      "source": [
        "Note that the whole test dataset might not fit into the memory of the GPU. In that case you have to perform some kind of looping to make predictions for all samples in the test set.\n",
        "\n",
        "Also note that if the last layer of your CNN model is softmax, then `predictions` contains predicted class probabilities (and not labels)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GsmVP7qxIMT8"
      },
      "source": [
        "###4.1 Test set error\n",
        "The **top-1 error** is simply the number of wrong predictions divided by the total numer of observations (in the test test set)\n",
        "\n",
        "To calculate the **top-5 error** you must consider predicitons of all labels/classes and rank them. The easiest way to rank the predictions is to sort them by the predicted class probabilities. Once ranked, the top-5 error is calculated the same way as the top-1 error, except that a prediction is considered wrong only when the true label is not among the top-5 five predicions.\n",
        "\n",
        "When sorting the predicted class probabilities, we are interested in the indices rather than the sorted values. You might want to take a look at numpy's [argsort](https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html).\n",
        "\n",
        "Your task is to calculate the top-1 error and the top-5 error *on the test set*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JwSVD7oCIOl5"
      },
      "source": [
        "# Your code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AnVYTwkIQWc"
      },
      "source": [
        "###4.2 Test set accuracy\n",
        "Is just 1 minus the test set error.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oDY64ib1IUo0"
      },
      "source": [
        "# Your code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ktcXzlx_IPbM"
      },
      "source": [
        "###4.3 Confusion matrix\n",
        "A confusion matrix is a table that summarizes the performance of a classification model on a set of test data for which the true labels are known. The number of correct and incorrect predictions is summarized and broken down for each label. The diagonal elements of the table (from top-left to bottom-right) represent the number of correct predictions for each label. The off-diagonal elements correspond to incorrect predictions; they show the ways in which the classification model is confused when it makes predictions.\n",
        "\n",
        "Example:\n",
        "\n",
        "![alt text](https://scikit-learn.org/stable/_images/sphx_glr_plot_confusion_matrix_001.png)\n",
        "\n",
        "You can normalize the entries of the table by dividing the numbers in each row with the sum of the numbers in that row. As a general rule of thumb, a score above 0.8 on the diagonal is desired.\n",
        "\n",
        "Example:\n",
        "\n",
        "![alt text](https://scikit-learn.org/stable/_images/sphx_glr_plot_confusion_matrix_002.png)\n",
        "\n",
        "Calculate and display the normalized confusion matrix. Use one of these sources as inspiration:\n",
        "\n",
        "- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html\n",
        "- https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "181Tn3s-JDzC"
      },
      "source": [
        "# Your code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0b3ehKfNig-q"
      },
      "source": [
        "###4.4 Precision-Recall and Average Precision\n",
        "Precision-Recall is a useful measure of success of prediction, especially when the classes are very imbalanced. In information retrieval, precision is a measure of result relevancy:\n",
        "\n",
        "```\n",
        "precision = #TP/(#TP + #FP)\n",
        "```\n",
        "\n",
        "while recall is a measure of how many truly relevant results are returned:\n",
        "\n",
        "```\n",
        "recall = #TP/(#TP + #FN)\n",
        "```\n",
        "\n",
        "The precision-recall curve shows the tradeoff between precision and recall for different thresholds. A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate. High scores for both show that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall).\n",
        "\n",
        "A system with high recall but low precision returns many results, but most of its predicted labels are incorrect when compared to the training labels. A system with high precision but low recall is just the opposite, returning very few results, but most of its predicted labels are correct when compared to the training labels. An ideal system with high precision and high recall will return many results, with all results labeled correctly.\n",
        "\n",
        "Your task is to modify the multi-class part of [this tutorial](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html) and make it work on your results.\n",
        "\n",
        "Note that `y_score` represents a *score* for *each* class. The score is predicted by your model and could for instance be the class probabilities."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9foJEGz6iufb"
      },
      "source": [
        "# Your code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0jrsdnKykv6j"
      },
      "source": [
        "See if you can make sense of the outputs and interpret the results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FpetyWBUkuQ1"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJXggp-HmasU"
      },
      "source": [
        "##5: Optional task\n",
        "Visualize the learned representation (i.e., the output of the encoder) using t-sne. See details in [Lab 2 solution](https://github.com/aivclab/dlcourse/blob/master/Lab2_Solution.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QnA0nlHRmn1Q"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}