Lab_6_(part_1)_Initialization_BatchNorm_SGD.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "Lab_6_(part_1)_Initialization_BatchNorm_SGD (1).ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Mihd9hpHKVh"
      },
      "source": [
        "#Lab 6: Training neural networks (part 1/2)\n",
        "**Like always, remember to set Runtime environment to GPU**\n",
        "\n",
        "In this lab we will explore different tools that will help you train your own neural networks.\n",
        "\n",
        "Today we will be using fully connected networks. Next time, in part 2, we will be using ConvNets.\n",
        "\n",
        "**Prerequisites:** You will most likely find it easier to solve this lab if you have already solved Lab 4 and read the slides for Lecture 6 up until Batch Normalization.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tGBcfgIumcH"
      },
      "source": [
        "##1. Download the CIFAR 10 dataset\n",
        "We will be using the CIFAR 10 dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u_awI9jy3V9P",
        "outputId": "1c352f73-be44-4673-87af-e08f4d4ccbee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "\n",
        "transform = transforms.Compose([transforms.ToTensor()])\n",
        "\n",
        "# Batch size\n",
        "bs = 256\n",
        "\n",
        "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
        "                                        download=True, transform=transform)\n",
        "trainloader = torch.utils.data.DataLoader(trainset, batch_size=bs,\n",
        "                                          shuffle=True, num_workers=2)\n",
        "\n",
        "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
        "                                       download=True, transform=transform)\n",
        "testloader = torch.utils.data.DataLoader(testset, batch_size=bs,\n",
        "                                         shuffle=False, num_workers=2)\n",
        "\n",
        "classes = ('plane', 'car', 'bird', 'cat',\n",
        "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iorLYSEz8Xbs"
      },
      "source": [
        "Display some stats"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U50pDb-_5uKq",
        "outputId": "ef7dcf47-1b0c-4edd-f76a-ed356eecbda1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 240
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "# Function to show an image\n",
        "def imshow(img):\n",
        "    npimg = img.numpy()\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "    plt.show()\n",
        "\n",
        "# Number of samples\n",
        "num_train = bs * len(trainloader)\n",
        "num_test = bs * len(testloader)\n",
        "print('num_train',num_train)\n",
        "print('num_test',num_test)\n",
        "\n",
        "# Get a batch of some random training images\n",
        "dataiter = iter(trainloader)\n",
        "images, labels = dataiter.next()\n",
        "print('images.shape',images.shape)\n",
        "print('images.min()',images.min())\n",
        "print('images.max()',images.max())\n",
        "\n",
        "# show 4 images\n",
        "imshow(torchvision.utils.make_grid(images[0:4]))\n",
        "\n",
        "# print 4 labels\n",
        "print(' '.join('%5s' % classes[labels[j]] for j in range(4)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_train 50176\n",
            "num_test 10240\n",
            "images.shape torch.Size([256, 3, 32, 32])\n",
            "images.min() tensor(0.)\n",
            "images.max() tensor(1.)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAB6CAYAAACvHqiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztvWmQXOd1Hvy89/bePfsKDAACIDZS\nXCWKpGxHkSXbkRUnSlKOyk7iKBWl9CNJZalUJXL8w8lXX1WcSipb1fc5YdmOlZTLsi1LEWPZcWSK\ntixbokRQJEiCAIgdAwxmn57eu2/fNz/Oee85PRs2CoOZvE8VahrvvX3vu93b55znLMZaCw8PDw+P\nnY9guzvg4eHh4fHewL/QPTw8PHYJ/Avdw8PDY5fAv9A9PDw8dgn8C93Dw8Njl8C/0D08PDx2CfwL\n3cPDw2OX4J5e6MaYjxtjzhpjzhtjPvdedcrDw8PD485h7jawyBgTAjgH4EcBTAP4LoCfttaefu+6\n5+Hh4eFxu0jdw3efBXDeWnsRAIwxXwDwSQCbvtCNMT4s1cPDw+POsWCtHbvVSfdicpkCcE39f5rb\nPDw8PDzeW1y5nZPuRUK/LRhjPgvgs9/v+3h4eHj83457eaFfB7Bf/X8ft/XAWvsCgBcAb3Lx8PDw\n+H7iXkwu3wVw1BhzyBiTAfBTAF58b7rl4eHh4XGnuGsJ3VobGWP+PoDfBxAC+BVr7dt3ep1vfOMk\nAKBWX03a2u0GAMBABHr3y+O8csIwTI6lUmkAQDeW6wYBfSOKIt1r+sOKQj5fSI4UckV3NXV+wNdS\nTcbdpMt9lIMWhv7G0hHX3y4fA4DImp5jvaDvLi1cTVrmZknxmb0xnbT9/L/8hZ5v/czf+ETyeWV5\nBQAwPDqZtN1YouvNl+UaR48cAADkU9SfM6/LMRtlAAD9Y7mkbbm5CAAYn5B+79vTDwA4/yb9LQ7K\nsezoDf6USdry2SwAIJ2X9culBgAAk8UnAACL8+Xk2MWrZ+h7mZGkLSzQmi41ZpI2E1M/x/rHAQCt\nZiM5tlSbo++lZA2KJdoz/+NXzmItri3Mr2tzsEauYUJee7fGat27Me2PKLaqjcYcxzJ2d7hj6fxY\n7ac44L0YyPmhobGHalO6feT2fKD6GASG2+R8d1Rv68CgB0Y1pPlZS+t7hnTPlHpGJzKyzgDwdz/z\nF5LPLX6mW6120ja/WAMAnL28mLRVazSHB/cMAwAG83J+zHOUKQ0mbblsH/XHqmeu2wIANBtVAECt\nVkuOtVsdAECnK+NbrdE9KpWWnNfhZzSkMQX8vANAKqB7BXo+Yjq/WJLnpVSgvd5p8PtMrUGr6+ZN\n2qKI2q7NXMbd4p5s6Nba3wXwu/dyDQ8PDw+P9wbfd1L0VkikCvVr5z7bWH4VDUsd8leuIRJKSp3f\n+z3+H7fRL2ysRPrYSf5rRRX1PfqUXJju3XMWj0GJPk5aNxtcw/Vb97HVbAIAGkqqCFiiD+zmFjIt\n+ZT6+nuuBQAj/STh5orSYydNpGK67tRecVK6dnme+y/3TKdIUisVtbRH90plaO7jjpq/iCSprqJO\nWk3ShKyRuS8UqC0VkETTjpVUll0GADTaIv0NxOS9VciIpNZudfn6/LempOUutQWhUW0i9a5Fh8/X\nWmCw0b5Yo2FpyThmqdroJWOpMFLfa/NeaLK0Hxu5pzW9ew0AMl06L9Xz6PLe5e8qBQBBis43684m\ntdohXDM8rQHEgbuGHgxdJVbP6FoslTtyDV4OqzpngjwAIJMSTTllKgCAfJa+OzlRTI5FbdrPNqok\nbY0aaXPtrpqPkPZ1sUiaX1pdf2VxCQDQaYk0ngpIW7NGafM85pDnL7ByLO1m0Or3E/+NRTOMOjSG\nfIH6k1UaTCpNn7PZfNKWzdHne5HQfei/h4eHxy6Bf6F7eHh47BJsu8ml1SI1qtUUFciRH7EiNG3Q\na3KxVpNCfL4iRoJg/W+VUw+d92S73VbnUz9SKVGHQ1bFQqWP2oQccfdUumrgVDG5Rsy6puLGEuLL\nmVxidbBaJVPLzMyNpK1RJXInMJsv16VL55PPJ46fAACUSn1J25WrlwEA+45PJG35Prrv1XdvAgAq\nFWW2YeKpWBB1FV1SIQcHlHmsS+eFYZ1OaSu1tTNEf1OytjYg9VOr78UMEWBhQGroal1Ispal72ZS\nVdUPMrmMFMREtByRiajRIBW82VH3ZPOONpA4M8xGiKw7X9Y2xeYMTS7GfA23xj1mw5Q7X9s/+PrK\nTNFmMq3tvqoI0JjvH2pCk0m6rtozlkdm2eQYW1nHDj9X+aKYLhKroZqQxALmjqmxOP6wrc+HI1s3\nN12974nHk88zN4nYn7s5m7Q1m9xP9bwEPP42E4SNpphtSmyaywfynA/yWJtq362UaQ9UF8vcf+X8\nUCrRLbUptsXzp8yLtTq9D4KQTT+j/cmxoSHa8xPjErg5Nk4mzYFBmeeQzWh9/fQc9PG9ASGhrXp/\nuM+/99LLuFt4Cd3Dw8Njl2DbJfQo6pVyAJGketriXgIxlVrvXtjjNuaIsA0kdedSZnsIHTovm1HS\nEH/W7kbNFpEezgXOhLofpuf6ut+aP3OSuZOsNDmbyxMxaLVWEDqSePPf3zAr55+98A4A4MnHnk7a\nsuwuuDi3LG1MDC2zW5812eSYc/cc6BfXxyJLgAP9QrauLpDEaAOSoNuRHMtgHwCgaeSeYIkuDORe\njrTqRCQN1VritthlCTZIi9RpAupbIRRtoxrQd+og0isORUJ3BJ9VJFa7LcfXgadS77+I910mlDVw\npGmynmqR3cd2VyTHepsJW6ulX54PS3MbKInNEfWBIpBn5shVc3JS1kXIU/qT0tua3ePSWkV0hJ+6\nl5OzXYvRLsBuPrQjgjuGzfG3/vbfST4vLpHWdf7cO0nbV778RQBAtSzaV9SinmR4bqOOdKSyTFpg\nbkIk47376PP4xJ6kra+PNNNCkSTjMCvSdf8gHbt86WLS9tKv/wYAoHypLv0o0p7M7qNrHB8UKX/i\n+CG657HDSdvYHpLQU+l00tZoU99DMOmq3gvNBmniiZYCIfbvBV5C9/Dw8Ngl8C90Dw8Pj12CbTe5\nOKSVquKIxq4mthxRxcd0BGjgyEhF0Dh12fYQLvT71WH/0KgjKk7XqaYpifTaiLRsNsiksFImFXJ4\nbDw5FrKfqfYrl370OM6jF5p0JQyNj8pRNi9Fzc1VshPvezT5fPIkRd++9Y5EQR47dhQAcG36ctLW\nXSUisZslVbOYF0JnuUqqYLUsazCxn9TPqCNJNiM2XeRLTO61REVu1cg8FQ+JGcbp7W0VqVep03mO\n/nKmFwCwbI7qhjJnUchEekd8fkM2B1neAqHe2oFbR2VCUfdYi6AndoFgnOlM2SIccejMf3qvdc1G\nUcb0RxPpjhxOu1gDZepITEXKbFPKc+Si6keXn4V0huMK2rJPcmxMSXU1icrjVF1LHqG1J0F86bXJ\nZQ2HuiH2TR1Tn+nv4YfETHH67CkAwNWrEvFbqZF5rm+AiMeHHx5Ojk0O0fg+8qOfStoc4Rh1ZDR1\nZm9Xy7Q/NLE/0E+mvr6HZE5zzz8FAHhp6U+TtotzCwCAE8PU3wMV2WsTZY4tqcg1Fq+fAwAU9kl/\nEdK9Ull6rqK8MrU5s6/aM+Y9EK+9hO7h4eGxS7DtEnoSKarIpnaLfvliLV27356EANK/do4wFckk\ncWHUZCRLe23OrbBalvwxJiQNoaTyREQuylORaZkc/epefP0SXaMuv9wHDx/hiylpyLktKslLOsX9\nVnJOl8fcPywSejZPRM7sdXH5WotiKC5RJ448AgA4fVok9JVlIguHBiUnyswiRdxlM64fInGELF3U\nakJYWY6urK6K1F6tk9SUK9CxVlNJsExMGiNz5FxSO5FEzTUaTHZlaQzZtJBYVXZDbLVEom6l6XrN\nWEgs59JpmFxMpZWskri6Ki0wkj6thRCT63MJxb0LCUC0S+0Kmbj1KWk8MOvX2xGagSNA1Z53Skmi\ngQIoZPi7HRl7jjWEtKU5alQkknJ5iSTeQ6yhAerZ2EAkdK6aW8dLQ2mZ6+fD4cy5t5LPTotZXrmZ\ntA2PEbH7gz/w/qRt7tTrAIB8hsjOpyePJ8cyMUny018Xt77VMmlr5UUZ841p2uvlJcppNDkylBwr\n8vMbrM4lbY8cpZxGH35a3Cz3nKJnZ4gfiT71fsrVae5TqxLNffaLX6MP/bKvcznax8OPkxvxxI/I\nOBOLhCLIw9Razf3O4SV0Dw8Pj10C/0L38PDw2CXYdpNLF+tTj7aZ5Onx3WZ9z0XNxdDEI6u6ytTh\niK1uV1R1R1ZWVkmNv3zh3eTY0Uceo/so1arJ343VNfLsJ+76GCmyCUzMacLNkaFaVTdOTeUh6MRP\nAUenplNidmg2SHVcWFL+3GuQDsWcMFCipETDQ0LQzM2Rinn4yJGkbXSQ6pN0LSc4aijTCPexUBR/\n8YAZx5ZKlNVmM0Oe1yWTly3lKOpApa3tMsHXVmGHrZhMM5bNCdm0jk7lOY3FHNThazQjMTsYVl0z\nARNQoRzLZFjNVnum3d6cYDZsmtMpZN1HvU+t24vOrLZBsreuIn9dzIWL9qT/8P4IrP4vn0/zcv7d\nM0nb1UsXAEgUIgDs30/reOQImVVmZ4VkXF0lk9lEa2/SlilREqhYR6Amn9cnvwvQaxaiE1zSuc3N\nBG+deT35PD9PJGO1tpK0xStkJvnoB38oaYs4QdXVU2Suqb8mJYrnlmgPr1TF1FHn2AzbUaYqXodB\nntPaFblnM2bTqkq21R2meInnn5Z6PQfZ172Ro71YuXAhOXbzTVqPUl36kWqQ+bZRVzEoAR2vTnFk\ns9rDOqpdvuBNLh4eHh4ejFtK6MaYXwHwEwDmrLWPcdswgN8AcBDAZQCfstZuLj5uASfddHWUnYug\n1L9YayIutWugI6N0hTvn6qd/85yL0AqTRi7lKwAMDJJE3NGSoItcXBXy1FrOXcLpL/fu25ccc66U\nXSWpbRRFmPRxTUpgbuXzpW82ps+ZdB6bodIWUsi5Hw72C8F75RqRuItLUrxhzzhJJLM3XB/VPQ2d\nlxLBHws3OF9LVpF/GZf3hE808gUXjBkrd8sUaxIdlXujw2Res0XXbylNwRGIsY6uZKmsq8jqbpOu\nl2INKgrkGmkmSF2aVKA3Qm8tunARfipPSZLqWO0xllKjyP2VMbW4rdlV+V3S7BLbk3OIm/j/sSJA\n5+eIBP+TP/5m0pZL03zU66KBNFhSHB0hwvvgwwfXdhutSIjsNFgD0mmesUZj6cm36xKPqMNuPrA5\ntMtmrUb7c0JpjSneK3uOCPGJCXqenDy80pRxLs/Td2dnJdfP+ZtEsjaVFt0/QE4ET7+ftO5XXnkt\nObZ4k7RRowjs8jBHHHdkTyyyNnV+ge514pCQyotZmr9l5RBRO0gE78CkELCNBmtpWXqussrvs+tc\nUtV6vwcC+m1J6L8K4ONr2j4H4CVr7VEAL/H/PTw8PDy2EbeU0K213zDGHFzT/EkAH+HPnwfwhwD+\n2d10wGVR7CqpxRUY0L5+Zo1k3ivVrs+X4uyVGk7ir3OWxdKg2KmjmO3lDZF0w8C54skv8aXLZI8b\nZLfCnArGaXFuFC2NOw3EbBSsskGbcxvrKukwz26LR48+su58h4WKZGfsFkjiGBgYSNqO5sh1anZB\n3MaWsyR97D/wEADg4mXJb5HiXBqZnPTj+k1yBysMizSUy/AcNdYn/CjlSGqpxGLDNDnWQGoqa2HK\nRa6QhDsSiI15op/GMhOJ+2TdufMp10QXZ2Osy3yotALWptI68CwQrWstuiw12dR6DTGlAnoSV0Zn\nB1d7uMX76eaK0py4PF42J+Nzpc1idrusqsImX/+DlwAAF94W91NnV9duvjHP+Sm2O0/+mNiCY85m\nGjVUQYc+l5VRjYXHFYiukBxLCrLozIpWjm6G1WWRpN3c7586lLS9vfwqAGB6TvbuiUcoyKfvfU8C\nABqrEsQ2yOlaLp8Uu/oQj+XsOSnZ6AqqVLo09jnlenuNNRujeK53qrRG4Q2Z+xwHab10muzlP/mf\n/kNy7M3f/AIAoF6/nrTd5DX9xA8/m7TV52mPzbxDYzjcUQU/NghatBvZ1e8Qd2tDn7DWOublJoCJ\nrU728PDw8Pj+4569XKy11mjD4hoYYz4L4LP3eh8PDw8Pj61xty/0WWPMHmvtjDFmD4C5zU601r4A\n4AUA2OjF32TmrK1UoOSzyrWSDnpTlW4EbcJw6ksvt8N5RFj1WZyVyMs9k6RkaC27wak7L54X90bD\nxNZTH/ggdVERYYn7mjIVbWVycVGyunalMxVF+kS+RkozlGsQ5oSTrrFLXl/hQNL28DHK9XKgIuaP\n06ffAAAslWkedD3VNKe3bbYkD0uD3QSLykXS2blidgOzqufVKpmqrKpBGnOykpZ28eM0o5NMKE1N\nPpwcK8+R2hoo1XumTsphqGtAOjNGez1ZF4Z0/azqdyazuXKapD/uqeGaVCRQbRzdyXszo2pG5ni9\n02pOm3VS/TNZMYW5ghkpvpbKgoy3X/8eAGDuyuWkzY0glZY9M80myn0jZAY8/c6bybERjpLMZ1SO\nIk4/HOuCKRvUyHUI3Th1JHZvxt4N4UhdAFhaIrNGrEjis+eJqL++KOavRc4d9Ikf/QsAgN//2m/J\n+VfJ9PTkCTE9FvuJoHz75H9J2spstkqxG3EhJ663xx6m+WiqaFrL86frzFb4uW7xO+D6tLiCzs+R\n2TKjck2t1OlzTUU0u9qxNX6GOoqINzlaSf1eCFRxk7vF3ZpcXgTwaf78aQBfueeeeHh4eHjcE27H\nbfHXQQToqDFmGsDPA/gFAL9pjPkMgCsAPrX5FbaGI4E6qjBCIiHp3BhbFLhwkms3Wp+BrucXkImk\n4QH6lZ5+V8iVhRtEzMSBXPcq/yqX+sUV6cRxcrEK+J6djiqTd5sSuvucuGzqcTpJSblPtrmoRqO6\nOZGHtEjoLdZ6WisikWZz5MK4f0rcLB8+TK5Y3/3OtwAA/X2SP6bRaHPfRGYbHRvhNilA4UqQmRRL\n3m1xM1tZoTntz6ixu+yJKoPgw/sOAgCOT5FknkqJe2aHtYxjVrJJ5mYpYGVmXsru1TjnSw5EUmcD\nyW3TaZNknE7LWNKBk5KXsBYuQMiEuugKu5n1JM102gD9DdWe7DL59vp3vpu0PfIE5fKoZ0US6++n\nfmaYlAyVK+2JQzT25YsyzpAlQNtWxSlaRO7PXKDzsqoYw+ISEerPvl/yiASgtdXBeYnLarBe5nZu\nxEYTwi4ozm4uo1eVu2+XXQIXF8Rt1gU9hXnRHr7y1S8BAMpLJAWnUkJUdptMMt4QLfMjH/0YAODp\nx59I2v7kle8AAC5cof2XUdd3DhfDY+LSe/36FQDA+x+TPbbMuX5cUYqTr8o6lnlcExml8XEBmfKq\nSP5tLp/XYEm+sih7LT1MaxarwDNNkN4tbsfL5ac3OfSxe767h4eHh8d7Bh8p6uHh4bFLsO25XBwB\n2lE+msEGeS3WpS1Vx5LiBtpt2GXPVb6dKf7SyCCZUNLq9+ydN0iNrzZUBCETSc8896GkqVAilb7F\n/e2hMVwV+A2IW7sBmebyZ+g+ulwhOo+IIxrDLVar05Zr1Dm6rqX8ZM0sqYTFgpgzHtpP/udzs3Te\n5UtC/BSKZApYWhT19uChgwCA1ZrMUcBEY5Bm4geyjmkmCa0im1zU6FRJamI++TD5HHfZwfzUG6eS\nY+EwVUbYu0+I0r19ZIqYW7yUtFUbZHJKpWmc+UCIxzqr6qEisTJGVO61SOZeL9n6jLpSAMWZaBTn\nn+X8MYcfEp/wZa4HqgJyEXGRjotvEJG5MCtxAte4/mYxo3fZ+sjqyiqp8lev0NynhsTkcuw4zVtn\nVdWS7ae5SascOy6tMVxuFBXm6T4Fep+6tL9bsKIdVbfVmSPHRyVSdIb3nauvCQBxTPPx1a/+TwDA\n1D5J9zxUJPPRhQvnkrbDR6iIxmf+zmeStpsLNNZKlfbkwLCstdvP2ZyYYRZmpwEAJcVIr7Rojw+w\n+arZEF/2gHPx1FS/2zGbKHXkp/vAprhqWcwxpT66rq6ZGnfv3eTiJXQPDw+PXYJtl9BdXgudqc46\nArQnJwr9ddKTdoBsseRl4/VSsJZ+IybiMizx7N0rlcLPv0UufC0lVew/QZKjUeRHs0u/xMZV8u6R\n2FxhDvWr6yLudIV141ww456/0utegiTJrJdTWQjXoFEVybvRYDcsVQCiXCfJb2FJJKRsivrx1BMU\nnTc/JySWK2zRVhkNl5dJ8jG64EKOJHmTY9dKKPe4OkkyXSWFHB4m6fqpqQ/KNQyNq8Fefx/8oR9L\njp3lXB2RIsdmy0RiLaxIhGHMRF+zTVJZoStEdpqJUl1VPbVFXhznOhrqUoLuY096F+eyyXu4o7Ns\nUtvRwweTtpMnyQ0RHZE6IyYar18hbaMvKy52+8cpS19JRYV2HBGrXF3rLhqUifSbV64kx9qcu+fC\nK68kbUPDFLH6lz8lvgzj7llw+W5UtG6Hn0Obknta9wBuQYoODIqWtLJEpPW1C+/I8TS9fhZjcYjI\nFmj8y8skqU/PiGaR2kf7pKXytrz4u18FAEzul2ySzzzzNADg9Jm3AQB7J+U5D8CZN9W7ZXyU1iOn\n1C/ncttXpHseOiwRrnPXyYXWllVkaZ0LsVySPemI6+FVula3Is9SirVzo1wVu7bHWfmu4CV0Dw8P\nj10C/0L38PDw2CXYdpNLl0lRq0nPNX7agKi3jgCNlXrkfJpD1bZWHaZ70F/nOz0wIMm5XJRkVhFQ\nk5Nj667R6ZB6mHU/hbqCe5LOVffbVfdWg1b+vGvhKsnbngIe/CHYPFI0HYgaP1CkMdQVkdPmZFGz\nS0J8url8aIqIO2d6AYA/+uYfAwBaKlq3zYTm5NiY3NeSySWV5YRnGTF1RFXq+IBK9fORY0QwpyEq\nfatJ/XRr2lFmG7do40pt/tYpSsQUZ4SwdVagao0SQuXqYp4a5ARfFZV4LdrCVHD2DCVkOnpC1bNM\nu/iAnsoP1Mbmhx6TH+/hpaWFpOnCWYp7CLMy9klOjObMNSdf+15yrJCl9a4of+7ViAlpZYZxEdCT\nU2R2sHUxG06/S4loayptcp2PDxakHx947gMAgCGu81kcENNcYZDiE+SqKlneFpHbQ8NyjWqF1tjV\njwWAIxkyhZUeGpcv9ZGZptql6GxdyKPFxVb6puT8GTZx/Ndf+uWkzfLz58xvb8zJ2B3xuH9S9nCO\nozZtKGNZYpNjgRPAvfnil5NjAxyfMqjsb3s5zW58UeqdxsbFx7A/+oSQs10cBABEWqZObcEw3ya8\nhO7h4eGxS7DtErqTwrVbX5IiV/smuhJgG7gGBhsUH9gor0rApE6Ho+3m5kV6chXQGy2Vn2Gefomn\nDkhOFJsi6dGy9KTLeFnxlZR7dl3qUdW3cPP8Li6vhFXX7cZO21h3eoJyWaLQUiERful0VrW5m8tF\nHPG5vEyEztioSNKPnKDiAN97QyLkOq2IzxOXw2UmetqcPjdU+UGO7yWJsdAWqf0mk1y1urg+jnDU\n3tAQnVdRZOfD+2jupyZlDcZZ0pmvyDViJy1znpLVumgipTxpYplQFR/oCBG3Fi7qT+8/J33r9C5J\n2TbHD2oNkdfWRdcCQJGjQE+p6NFSkSTS2ZuUDun6NXE1ReTSPPclTYPschsprdFFQmY5z1B+VDTP\nDGuIGUVkj7Acd0HlKOobIGm5OEPPxPAeiSh+7GmWtM1Gz+jmyGVF03L5ilpaom/R/tt3RdJTL+VI\ni/qBPi4kk5X9dJNzw5h+mY/UXtIeri+qSOkVjmSOuKBNVcjGPnbZ7FfrgkvktphT++k4u/cOM2Eb\nXpmW6y+yRllS42M3R+3gYLpO46S2mUXRmBuXiOx/6eVvJW2HDssev1t4Cd3Dw8Njl8C/0D08PDx2\nCbbd5JKYUDZIaBUoFTapYeiSV6mEVonZZgOTS4/5g9U+V/Oz1RIVq8J1LGuqVuPsNPnzHlQml2w/\nkSkB++l2VRrTiO9vFHEm0aCKxOXEYU4N7U37y+YV6LHw97A5ul0xFXXY77upqtT0u+hAVeWnxVGK\nC4tEGkUq2nQfm0vmZsW/t1IhtXZhWcjIEqvGl64RUbl/nyTFsmzVmJ2X7MqLfTS/QSjmoNII9W2A\n/aNzqrbjChOrb56S6MB8ngjPlDIjwJGWeWf/EDW+2iIzgmmLql5rCkG6Fvk8qdt6zVy90JQyMgRM\nijmzm1UkmauaE6j98WMf+ygA4F/8P/8qaVvgtLKLC9THTErmxfnDVyri79zhqNtQpeqd4jqchu1q\nKkMt8hwluf+AmMlGBsnc8MFnpbpOoUAml8jQdfMDYpKIea8b/TxumGi3F1nlUx8y2d+v2vrZxFZ9\nUyoy3eBUwQcmye+7WZW9ll8i4jFflCphg0yaZlV9zzo/MG2OWu6XqULKckRnXfm3t2l+8xUxW47M\nkK/5YJafR0XmNnmvBSm93pyaW5lKne9DwPbAG9MSBVzupzFcuiBpoesN8a+/W3gJ3cPDw2OX4IGU\n0B3MBm6ISYIXLXnz3x4XxTUSvYZzW9zHuUwA4I3XqL5heVVSw+Y4B8O1cyIdFvpJkiqNkKQ+9NBU\ncixOu+hR/TvpfsVVxe+oNyJsoz52lVtdkrdji8QZSmFBvMGc1tg1MF8SCQmcArjSojEHSmjNchTf\ns0+LK+PXv/F1AMCptyTt8P4DHEHXIMk8XRMpuMnzEWfEhXBkfJL7KHOwyCTWSp1JV1XgIsqxFjEt\nazC3QJpTJq1SKLOLmKuvmU7JPTuuHmlH3D7DLUSZkKWsb3/rT5O25z5Abn1GPTIhk12BS7KjNIa4\nwxK6iugcHCcCb6QgfXv320SK5Zlkb6jCGy2+bKqhtFd2RTXK5XB1liXXApOiU5IGeSVkkjgt5xvW\nBr73re8kbROHaM6rrNU9/ewPJscSV02j01nzkLeQCdNKO3G5U2pKK567QhLrlUWRXDv8nTM5Inbz\naZmrDM99VT2jQYs2bWFFJO4RXr/BtHNFljVLu8I3Z2Q/lbgARrEs/Rhvk6toOqR+L2RlnGMfpgIb\n7YtSwGNhhkl4FUlseH90eLMglyvZAAAgAElEQVRNl8UJY+Am3XNqQqJpC6XtK3Dh4eHh4fGA4XYK\nXOwH8N9AhaAtgBestf/RGDMM4DcAHARwGcCnrLXLm11n0w6wFKylSZd5UduRE6+7jdwcg80DkXQh\nDPcVJ8iHKlDHlYtqKlF3tUq2tYUZ+eXOcgDBjcvk8rV3WXI8jOwhaV3bHy0Xa+jqiulY39+1iDcK\nLNq8dCsiJaG74hixCnCqN6ltSUySGBzk8mdsI213xQ7ZbNLYR1VBjOefew6AFBAAgBs3yAb41DGS\n5I/sF76hyiW3rs9K9fcbM+SWV6uKlDU4SrZzy9JYkBMpf6SPJP8Lc9+U69bJhSyjSvI1Wfo2lvPk\nxHLMubiGSn4ZKEnQy1qU58mW+s2vS5DIM4+TG6dVxSkiLn6Q4v0XaK2x4/KfiEbUZtfHfmX/HuK8\nJPkyzX1K2ZiX2e20lZe8M012Nc2rwgiGXQ7LbS5csSKBSAN7aG7LNVG/3j1FeYuiiqx33xi5rOY4\n6Glyr6z75BRroTogLuQxY/M93O2KNO6Cfa4siDvpyiIXt1GBU32W9sAfzVHfSkrT6jekeVzMiQ19\nMmTXxxXZTy3WJFIs5eeU2FpibWpPKHOa5vMbyqV3qETHoxaXiVRFMqr8sHVWVZbIJJBR7tXg7I1N\n3ndzi3J+vUjS+kifjCWv9sXd4nYk9AjAP7HWPgrgeQB/zxjzKIDPAXjJWnsUwEv8fw8PDw+PbcIt\nX+jW2hlr7Wv8uQLgHQBTAD4J4PN82ucB/KXvVyc9PDw8PG6NOyJFjTEHATwN4BUAE9Zapz/dBFTC\njjuAK4IQd9c75cXKfS0xT7D7U6BrdPLfjdJKaLOGq4noIgC1WSNMsRui0pkaHFFaUXUCAyZYupyu\ndemc9DG6yYTpmLiI9U+Q219qSFR8y2lwk2hWbT6Cq74uv7XuuI02d1wMUvq3ma/XM6d0r3pdJepn\nM1CpwDkm1FyFbLoIlVljnE0jT7P5AQC++W1Ky7qySq6JuZLUZWxbmrd2W6XlrdOcFhQxePQEfSeT\nJ/Wz2VRpRjnyc2bxLTVYukag6r92XURuzG2hcgXlj822qLzFnERTrsUpzqcyWhLC6sLblN/l8WdV\nbc50r0tqz15zaZ5VmyOaB/qL6jwmVnnfHVqS8/vYKnBB0o7AlGhNjeK2R5+gIg9LNXZTVe5x4QyZ\nIh5ShSUGP0Sms0sqUrRyk6yljQaNaX5BXE2TDK+q6EWcpKfGpogiVbSGzYVpZVbIHybTZKEqA6ye\nIfdXMGm+oOZvmd1qV43MX4ZrET8aiSmsyDlZFtkkdqMkrrQuD/L0fjGLTgY0byMVFX3LtXHrq/Q3\nUoVhIr5/qyi5WaKQ5lyn5W0V6NmpcZOpy2T1FZkEb4vZa3mLksG3i9t+oRtjSgB+G8A/staurvGd\ntsZsbOA1xnwWwGfvtaMeHh4eHlvjtl7oxpg06GX+a9baL3HzrDFmj7V2xhizB8DcRt+11r4A4AW+\nzrqXfiJ0qp/6DBOZRrlJOQksZpJJy6OBk3QV8eiKTWj3tBT/cnfZpaxvRCSD8T2UcXD2hjj6h1yt\nPlLkbJNJLBef02lJTpA6B0E0lIQ5N02FCwampBTZ5CNcci1LkkMM3W9H+qocMYnX4uZuTQVFnHW5\nHFa7I66BkgNHgo0qnJnQZYPLGHUNJ10pqSzNUucT75MK6+fPk0R19t2LAIC945IV8amnaJzXbkp+\nkoEBkmom9sh5eS5qUKkucR+Vm15ESmAzuqbOZxJXhBtY3isu2CNQC99u0zzU1Lpk0/J5LRYXaV76\nlRax5DL2qX3qipC4jIAdNd8Z5yaoiXqWGJeXxX0t5uryZc74V8+o54Cv//CSXLdToLb5gtK+mMTd\nO0lEZnxYtKrgJu1JW5V9mhshCfOpZ5+TNi69WGBSdO+UKNxtzlrYUhpzgcnqrST0ONYFRZw7qfTN\n5WTJqqynF09TkFGGif2VQOavwSR7qqgcHVLsnhzI+KwLsmvwvKmCGE3eC3MFmdOioesOQ8j41gJp\nly5/UXtIXEFnOLDpvMpmeoi1r1ysrltlRwveA+miPF/9Jdpb5y5Kjpjyyh37lKzDLW3ohkTxXwbw\njrX236lDLwL4NH/+NICv3HNvPDw8PDzuGrcjof8ggJ8B8KYx5nVu++cAfgHAbxpjPgPgCoBPbfJ9\nDw8PD4/7gFu+0K2138TmWTI/dq8dsC6NqVJVnLqcUn6vIas0MatsmhQNnDlBmSQiTp2pc5c43tD5\nC4dZUYFOPELE3MV33pC+cVGIrhXVsc0fI1bd2qoARItV5ExGGKsKR7XNrogDeHqIyMXhh45Sv3v8\n511V9/UFP6zdnBTNZJVPfeRqimqzzXp/YWeWaraIjekEYo6JYvdZtsgQp3rtqhTD46OUj2NmnlLe\nnrsmRBvYZLWkIvvGuFBFqPrbZZW30yJTR6sj7FA5OgUACJSJpFAks4DO5xO4KE2eq46KRK1zHo4w\nL+vSiFTq5DVIsTqcLaoarkyGr1ZEzS4NkIq+ESnq6temlBLcZRNEJxbTRY7PW+Fjs0VFnPF+nqyp\n6NQOtR2fFjPC6tcoorXvY88DAIafkMIc3SFas5qqzdnh+ajU1J7k6vYlNn+U+hWRyJbSWk3WIF8q\n8tixKTqqxqrzYW+pIhJvvfkmAOCQ2qc5zsmScjVq8+r5dXy3Ih7DLjV2srJWk49SJGd/RHuhsiRx\nEGXOi5NpSj8GmfXtpFR+KE5dvOzMaVOS0+jV16nfFZX3aZxz4QR12etuL7aYlS8UxZe9y33rGxAH\nilyB3oVzqlbpncJHinp4eHjsEmx7LpcMk0KhksaDris3p9ykmJBzkk+g8ppk+LPOMudc1XQJKyfV\nZziCK1JVto9wdfb3f+CZpO3175DkU1eZ1gy7XaWZKK1U5dc04DwY6bRIn7UaHY9rQtoMz5A0O7iP\nokx1wKiLqOvqbHbu4xakaE9xD7s+l4uLitUCVSK1c2Okioy5fDPZphBFWY5cXFESz9695A63UKO/\nV29Ixfkqj72QlWsc6NA9qg2RZDpMCkuBCFmzCkvSRrUFLOVpV03DEYDdlsukKYypm5tcSWlCrc19\nxBqsmRVVSbAVzky5rKMaB4ngjdbk5gGADs99uEHE78iEkPEt3pN5zteSa8leW81TP64pTWE4Jsn5\nYEXlVVkkSfvit/+A7qlyguxlzTNUxRjyCzSWlQXRUly20VHWMssVcdWtnHkHAHD1qpDbT32A3Dcn\nR5VP5Ro02iKhR11HXsqaLXJ5vmORPC8jvP/LPORKSrn05lmS1uGYMWeYVMU0Bp+lZzjkTKClBSlB\nV3mVioukQ3GfrDZon4Zt0ZzAGlOFNZHSIYmAXj1NLrR1RYLXmEAPhTvFjSV65m9YmoesyqO0skLz\nHUfK9XarCja3CS+he3h4eOwS+Be6h4eHxy7Btptc8jGpmDrQMUgKXIjq2LaujaCjJlOcojRQKkvi\nh65UXmfVsexjnVbEapY78OgjJ5K286fJqaeqEhulDJFGkSM81PVbTBZ2ayppj/NV1gUu2J83ZlW9\nR9FyRTJ6Ugfz3y2Sc7WVuthsNvh7KlmUC/dT/u0xz5Gbq0gR0y7KbzyvfH45avPanKjedU7idXA/\n+dnrhGcXL5D5Zc+4kM+dDq33wk0xubRjIhr7h2huKytinupnX+VcXvzWV+0lHpTsD1fLtMtkVqR8\nj/Ps/xuklD93Z/OkUtPXyDd4ZUUKHhTZZJFOi2qfzdB1XUGMflXrMmKS1pFfdH9SuR99Qvz4l176\nI+p/RHtnSJ2/yIT7nFrbFI95ekjmucTPRpajgKe/9g3pR5naRj/4ZNKWGSeidKJf1R7lOreDvN4n\nT74mfeQkVG2V+rbOJq0f/jN/FpuhWZc1WF7iIhI5IVs/9NyHAABjlyRhV7pK5rypx8lcslCSiM4L\nV2jPXFtSzgFlautA7tVaoec1WyDCflntyWqa+p0pyDynFvl6Ovqc57k/T/29uSx7Icemx3pD5qM6\nQudFKnXxye+Rg0DEUayPq2jTWpnMZIEV008ude/ytZfQPTw8PHYJtl9C73JJMiVxJyk5O1q64fwn\nifudSFgplkS7XU0ksqSrruvcxsI2SUMZRXp1OcfD+LBIWU8+8T4AwBuvibTiqq23kvSo8uu/wClh\n0yplryOB0oo4y7hfYpaIbawlaf6reZ/YlbbbnBRtNHVUKOe7URIszPocOC69bofJq25Tft8PTBwG\nAOyZEMl4ZomiQht10UAKOZKgjj/8QQDA4cNC6sXx79L3rosEdvUaSTJTEweTtlaXyVMmjeKukEeX\n2Q1yco+44o0M0F5YTl+W4YW9Wkk2MyR9LHJJsrZoWq3W5i6gK9cpL8dMpMr6cekyo3izXIak9YMH\naSx15cbWYXfZoKFSsnI+lT0HJeVymiNbGym6cFqRhqO8J0eripw1RPAtpGWO5kdI8jvcJgKvT7qN\nlVNUjCRSBJ6ZJG2qb0Rc8Upcqm5sktbzyrSs2cI851dRWuMylyFcUe64a7E4L1Jt1Kbv/tiP/HjS\nNsFpf0//3v9O2q69Tnl0JtiF8PhTki/lyRP0eWZeBnj5LK1t9LK4y17/2h8CAIb2UeRspFxk06vs\njliWhTxQ5udgRBXCYK0/w2T7qtLSG8H61NwRn1etr49uT4fsaq1KU86V6XqttkoRHmz+fN8uvITu\n4eHhsUvgX+geHh4euwTbbnJBg6MUO0KEOUtIRvlzBwlhRyqNVQReisnCSJkunM9vNxb1zHKEWcht\npq3UUCYLszlJmXri2BEAwOm3JXWri0A8epSi0bIDohI2T1FU48yNG0lbqUQmnFhFzS0tkv9tzMRd\nT5ZbR4qqsbhK88EWFY4MhFwJWf2zkAu7z4Gy5bigPVfl58j+R5JjTx2jGprnZ0SVvblEJqXBopgz\nwLU8FxYpN9uBh6TSzUf+7A8AAP7w5T9O2s6eIUKz1ZKt99iTFDk7wH7DUSCE6UDAEZQzUsd0uE6J\nowbGpL8r2csAgFyOa0GqGpqZLJljmg3ZY9EWpGi8QiYga2R/DAyTqWh0WEi6Ba5eFUV0rbNnz8g9\nOUIz3VaJ17heZkkVb3UR0C1OCdxQWneGVfaCiqXo40ciVZe+Nbg6U5vTG1eKcv4wP0u5i2JCaVbp\nJteX5NmosTnIBWYODco4u/zcVCqyLnPztN5XrlzGZhgekmfp2WeJAH333Nmk7c1XibzVibX2fvhZ\n6iObEOfnJTJ3ZIDW9NH9sncybSJ2r31bfPXT7N/eWKKIzk5PxSIyVQ0oU2lfmn3fYzG3OutIk9Nk\nl1cl0nbPOO2/xUDiMSybTuYX5TzDz1omTxerdsVUmc7SGDKq3mmzoSJr7xJeQvfw8PDYJdh2Cb3L\nZENLkQ4xVz5P5cRFLM2EpnOx60SqXqEj+qyQDq5ye1ed51wHQ45ONVbnmqA/qbRIsFlO3Tk+LBJp\nuUrE15FjVCX9oYePJceGR0mq+a3f/lLSNj45yfcWafn6DXKLe7JJ0kdKpxRN6l+qfPMu6nCLSFEn\n6dEX6E9XRcK6CFSriK0Up8s9uv8hAMCxQ1K44socEWFnLopEFbsiElbdi4WaeSZMJ8bFbWtsmKT1\n9z8tLnNfvU4FMa5cP5+09bMLXsgXM0Mildk0rd/QPlUflSWw8lVxuxsbpHUYzHNhgkgqste5WIH2\n+gy32Pr7HyayUKUYweGjpK0d2D+VtCUcPP+d2isEsmXJS2+xVuy0JHFfa7CGZ1niNVZHnXLuF6Up\npJnw7ley2BBrubV5knS7XZGM2/0keWfzssdSHZK0i/PyzKFMEvzsChHCZkD2vIsQrimtp8ka5xtv\niea0Fp2OaAA3Z2jPnzsnWkyXNeThYdFy97BWHDVpvXVkbq3JrsJNlSqX1zF/UNXx5cjPHBPBBV33\nlLWkYRUV7dyZo27PBqExcHGMTJ9I73sC0gYaKpq2yWMNVeGRg1PUp0HOPVQoqLxI7MKoNcUKv1uu\n6XW5Q3gJ3cPDw2OXYNsl9Da7wHVUME7UcnZkZVxmm3LK2YKVuOVsmPr0Lv9WGRVU4IIyohr/6gcq\nuxp/zqREpHKZ7Q5PSUa02WWykRkuFGGbIk0e20fS2yNHxC3tyg2SeJ58Qkm/F8kuXWYpeGJSru/s\nlUbZTZ0rnisLtxF0LpfA5W1RE+KEDx34c+QASbUHJkkqujJzNTl26jwFVenyeI7SqLWllklY48yO\nHIh0c1Yk46MlCtKaGJM8GMePkyvbpemLSdvli/SdNJf2Gk1LvzvMlQRZCUgZf5ikpYUrkq9lcY6u\nO9BPUtFQUXKMhCmSfFpFlRkzKTe2PuviX/lrf5XHK49HnnN1pDJiqw15QkySDVMFj7E2FSnX245b\no1jcGxde47wgHFwTdlXGy26DvydSXJrL7qUUF5IN6TtFcPbMVbnnAue2WVHa654JmqPJomg4zpN2\niAttdFXZtkfeR3s3Vm6+2QJpz6OqtOJavPbdP00+X714DgCwd49oOIOTBwEAExNSTGNpiZ6v2ZvE\nQ9VUycSxCeIIxsdkbaf66Luaq7i2SN8pc46bfpULZyhHknE8J3u4wlJyQ2VxtP2kNZxky8Hiu9KP\n0THqRz4rGgvYnTmXk35kOe9TLp0UyUyOOc4rLMr5JS4agtPCwd0pvITu4eHhsUvgX+geHh4euwS3\nNLkYY3IAvgEgy+d/0Vr788aYQwC+AGAEwEkAP2OtbW9+pY1RqZJK06wKEZDOUbfaistocsGKwRyx\nDlZFPHbY769RUYnh2XXPVZIHgAwnwV/iGokNVRMwy/fMqcjSNkf59Qs3i7if7p8PycRgOqrfAalR\nP/whScH7hS9/GQAQqcT3Jw6R2nnjIhFED00KAeXIlUAxcnFictl8uVJG1L90ijrcaQt55Ewy/UrN\nPjBFpOX0zQsAgHPX3kmOcaAc0qHc07k8hkrdb4MJx4j6O7soJpehPhqnJmz376cI1KYqknHlMhGk\nC5wLpTCqcqJk6bphRplhON9J/6SQf9UWmVzOXSKz0eOcNpb6QSTnkgpqTGfXp7x1yHHul4yqUO+I\n644yY8VMUG5kcrGBG7OwZEFI5xmV+jnbTyTyTbjCLSoCmtcgVC66OSbXFXePHN8rw/JZQbn75tgs\n0GzLfN+Yplw8lSF5Xlx93QZHO+85dDg59sRxipjO5YTMdTaa1BYyYT4lY5m9fhkAUF0SU0cfE7YD\nQ1InZ89+cjaY58UaV4U29h8g06SOgB4YoGsU1Zj7CvSd2vwy90PW0TDxGM9KSl2Xjlmbfc/coOPn\nOD12qyrzF7NpZt+UIsHZBVkHe6bYbJrlQi9d9W5xDhxGvcdS6vPd4nYk9BaAj1prnwTwFICPG2Oe\nB/CvAfx7a+0RAMsAPnPPvfHw8PDwuGvcTgk6C8CJsmn+ZwF8FMBf4/bPA/gXAH7xTjuwyqRHT3k1\nlkRrKhCkxJnqChwA1FFBQZVlkiqaKvAhmydpvKAynPX3U9tqjSTYIBYC1P3qlhflGpaDiLJaGmJ3\npwJL6LlAu0/SNY7sF9e95x6nHCRNVSTjyF6SGM+eJaIororomA3Wu2B2WQMJu5uTojlF0ETsqhmo\niumuQMSqKjv2J9/7Jp2fFA2R6yXl+lRbmidCe0hmWep00VFLFSEZr94gyXugX0ipcpkkn6y6yOAg\nSVlXrpHkOL5PMl725WiuurHO9MfufE2RqFzxitG9NA9Xpq8lx/awJBW31HxsUUygxe5utifvDQd3\nqRwqAefycPlxNIEcsYZl1TpGvMd0kJnla1SYlA91aUX+bkoR5G3ryHtpqxvnMEDX71OuuuB8NNkB\nkXRzWRpDM5T9VOXnMGItaVQFvFQ4e2cnkvnrcj/Tm1anlAyVgBQ70c9Bs0n9fPkPXkraHn3sKQBA\nq0lzdOSYOBO4wiDVRZGuV+fpsy5HOPUY7Z/FWQr8mRiU5zGZNfXsz9RIu3vzm99O2l49ycGEliT/\nVFE0gH4uPRhod0h+bgM1H07DyvKHttVz1avdAfKc3wtuy4ZujAm5QPQcgK8BuABgxdrEaXYawNQm\n3/2sMeZVY8yr99xbDw8PD49NcVsvdGtt11r7FIB9AJ4FcOIWX9HffcFa+4y19plbn+3h4eHhcbe4\nIz90a+2KMeZlAB8CMGiMSbGUvg/A9a2/vTFa7HebVeSlyyipa4QWWM1ps7+zLjeecxGlyoe3ydFi\n7YZEmjUqNNyQ1dtiXswUXc4dUWvKNUL2NU+lRTXty7Eq3SYVdWVJcmQEXKcwjsQ/ei/nAFktK5/w\nOvVprI9MEVVFJBbY3znSFeRdfU/t97oG2Ywm01zom0xgi61X7Y4qhMHEZMj91pGUTv0zaocYNuFk\nsnLdIuepSDExVy3L2JcrNDfFwkG5CBN8XRVFODVFyt0pjjqcvnozOfbcXpID6rGYwhZXZ/gaMpaA\n16o0SOaEVRWhOX2FiLhcTu0xI/1ch4SwUmmQjTOhqKGw73EnWq8qu9TERuXTcTVeazVJK1uvk4nq\n4UPkG57SWjmbBULVFjrzijLDrE27WsgLqZzuI+I4KIjZK8tpf01W5sMy4dk/TOavSBG3hk0zusav\n86neouYKBoaE7E9zXdmFOTHJudw6N66LeezSRcr1k+X+nL80nRx7+hnKBzMxJlGh44NkThsfkWjT\nSpWezUaRrr+4LHtnlXOyZAtiOsuP0Dyk1OL2cx6aFBOqowMyp/3sg69TUQfJ3KhcSXzYPTd674Q8\np5og36qAze3ilhK6MWbMGDPIn/MAfhTAOwBeBvCTfNqnAXzlnnvj4eHh4XHXuB0JfQ+AzxtjQtAP\nwG9aa3/HGHMawBeMMf8vgO8B+OW76UDfILnRFYfkV/cGk2OlknI5zPAvGkt4bSVJZ5h0KAxL1Fql\nTFKQVUUKGqtEkri8MVFbZSNkEiudlt+4LLshxoqwNXyvOkebNspCMqbSJFWUVVttlT4bRXg0ayRB\n5Phe5UXJ2lZeos9d9Uvvfv37VPTeWqQyqso951zJqhwxUYukvVgl1I85pZzL7yLFQ4CY+xsr/7h0\n1kXfKpKO3TabnEmurfLNZHl7jY2Je5crTLJalqx0faypHDxCbmlXLl+Ra5ykMe95SNa2FdCaRrGQ\nohnnVsiqSFslUak32H3MqDwzQ+K+uRYud4l2yXNupKqCH2ImT10JxB6CyxFmVu1Tdve8viRSZ9yi\n/TGUZSJbrbtbA+3NZhMXPOX6yBpqhv+mVNm2oEjzZxUJ3WW31iAUh4EwQ2M2aZd3RJ69Uo7WJ6ek\nyYDdBMMthMqu4mbd81XoE6l9doHWud6QOcqza3GXtdKFGZmrb738+wCARx6X3EATH/4hAMCli2Ig\nqJTp2Yw4q+rMtBy7dI00gCNHjkjnrtLzePX8haRpbJgk9BI/cymjFp6fE6u0aK5h0eu2yP+xHMFr\nA+UGy1MZKHLbhHdkMNkQt+PlcgrA0xu0XwTZ0z08PDw8HgD4SFEPDw+PXYJtT87VP8iFAJR5pcUk\nk6lJJOf1a5cBADlWreO2SqLFBNH4hCTt6SuR6pYNtV8vqT59rBbFWdGPAiaKoq6ooXGbzBgtReBZ\nVnXbrBKGyqzhapbOz0s0XL1G381nxSe3zeNzyZ/qqkZnl9W4rmJQkvS6W6i3w2PS79VlVvFU3UzL\nJgitJsIyOcbkn/ardfe3KmI1AI9BJQlLAiLZLz8Vyrok06v8b4cGKLHRjayo0i4ic+IAjWGpKvc8\nx776uaJEfuZGuOp6U0xVjkCPeG5bLTFBZTjpUbMt85wPZL+tRcqNTwWThhvEB4TOD92ZoNT6OD9x\nrYLXeB+1VUSiYZI/5Oxp2pyVdtGP2uaSoTmy6sLOoSDM0jFTEIIwYJ9zKNU+neGUsGkh+gp5WtvB\nIfpuX1ZI1IzzPzea/I35spu/QmrK5FKtcfK2eSFFZ5eYvFTxJn1FmocBNnVourfMZtTv/umfJG3n\nT1MRi66KPE7xvI2OcsR0S+a7yXVz6y3Zp2e46MqqqtmbZgeExDdcRfA6933FSyfJ9JRrPwoFWreI\n7XRBT3Cyq4Os5vQ9MLl4Cd3Dw8Njl8Do/BPf95uZ98Avx8PDw+P/Ppy8nVgeL6F7eHh47BL4F7qH\nh4fHLoF/oXt4eHjsEvgXuoeHh8cuwf12W1wAUMNGhRx3Fkaxs8ew0/sP7Pwx7PT+Azt/DDup/w/d\nzkn31csFAIwxr+70zIs7fQw7vf/Azh/DTu8/sPPHsNP7vxG8ycXDw8Njl8C/0D08PDx2Cbbjhf7C\nNtzzvcZOH8NO7z+w88ew0/sP7Pwx7PT+r8N9t6F7eHh4eHx/4E0uHh4eHrsE9/WFboz5uDHmrDHm\nvDHmc/fz3ncDY8x+Y8zLxpjTxpi3jTH/kNuHjTFfM8a8y3+HbnWt7QQX+f6eMeZ3+P+HjDGv8Dr8\nhjFm89p2DwCMMYPGmC8aY84YY94xxnxoB67BP+Y99JYx5teNMbkHeR2MMb9ijJkzxryl2jacc0P4\nTzyOU8aY929fzwWbjOHf8D46ZYz5sqvGxsd+lsdw1hjz57an1/eG+/ZC54pH/x+AHwfwKICfNsY8\nuvW3th0RgH9irX0UwPMA/h73+XMAXrLWHgXwEv//QcY/BJUNdPjXAP69tfYIgGUAn9mWXt0+/iOA\n/2WtPQHgSdBYdswaGGOmAPwDAM9Yax8DZYX9KTzY6/CrAD6+pm2zOf9xAEf532cB/OJ96uOt8KtY\nP4avAXjMWvsEgHMAfhYA+Ln+KQDv4+/8//zO2lG4nxL6swDOW2svWmvbAL4A4JP38f53DGvtjLX2\nNf5cAb1IpkD9/jyf9nkAf2l7enhrGGP2AfjzAH6J/28AfBTAF/mUB73/AwA+DC5xaK1tW2tXsIPW\ngJECkDeUfL4AYAYP8DpYa78BYGlN82Zz/kkA/80Svg0qIL8H24yNxmCt/d9c2B4Avg0qcA/QGL5g\nrW1Zay8BOI8dWJHtfj3+lwQAAAKtSURBVL7QpwBcU/+f5rYdAWPMQVApvlcATFhrZ/jQTQAT29St\n28F/APBPgaR6xQiAFbWpH/R1OARgHsB/ZbPRLxljithBa2CtvQ7g3wK4CnqRlwGcxM5aB2DzOd+p\nz/bfBvB7/HmnjqEHnhS9DRhjSgB+G8A/stau6mOW3IQeSFchY8xPAJiz1p7c7r7cA1IA3g/gF621\nT4NSR/SYVx7kNQAAtjV/EvTjtBdAEetNATsKD/qc3wrGmJ8DmVR/bbv78l7ifr7QrwPYr/6/j9se\naBhj0qCX+a9Za7/EzbNOpeS/c5t9f5vxgwD+ojHmMsjE9VGQPXrQuLpzD/46TAOYtta+wv//IugF\nv1PWAAB+BMAla+28pVqAXwKtzU5aB2DzOd9Rz7Yx5m8B+AkAf92K3/aOGsNmuJ8v9O8COMrMfgZE\nQLx4H+9/x2B78y8DeMda++/UoRcBfJo/fxrAV+53324H1tqftdbus9YeBM331621fx3AywB+kk97\nYPsPANbamwCuGWOOc9PHAJzGDlkDxlUAzxtjCryn3Bh2zDowNpvzFwH8TfZ2eR5AWZlmHigYYz4O\nMkH+RWttXR16EcBPGWOyxphDIIL3O9vRx3uCtfa+/QPwCRCzfAHAz93Pe99lf38IpFaeAvA6//sE\nyA79EoB3AfwBgOHt7uttjOUjAH6HPx8GbdbzAH4LQHa7+3eLvj8F4FVeh/8BYGinrQGAfwngDIC3\nAPx3ANkHeR0A/DrI3t8BaUmf2WzOARiQB9sFAG+CvHke1DGcB9nK3fP8n9X5P8djOAvgx7e7/3fz\nz0eKenh4eOwSeFLUw8PDY5fAv9A9PDw8dgn8C93Dw8Njl8C/0D08PDx2CfwL3cPDw2OXwL/QPTw8\nPHYJ/Avdw8PDY5fAv9A9PDw8dgn+D4Chk13WD7gaAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            " deer  bird  ship truck\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2PEUHshm8jNO"
      },
      "source": [
        "###Questions\n",
        "1. What is the batch size?\n",
        "2. What is the size of the images?\n",
        "3. What is the range of the pixel intensities?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2AdROOxx9ldc"
      },
      "source": [
        "##2. Task 0: Preprocessing\n",
        "Recall that we always prefer to have our data zero-centered. For this purpose, we can extend the transform function used above:\n",
        "\n",
        "```\n",
        "transform = transforms.Compose([transforms.ToTensor()])\n",
        "```\n",
        "\n",
        "This transformer converts a loaded image to a PyTorch tensor, as required by PyTorch models. As you saw in [Lab 4 (part 3)](https://github.com/aivclab/dlcourse/blob/master/Lab_4_(part3)_PyTorch_Transfer_Learning.ipynb) you can chain together multiple transforms to perform a sequence of operations for data augmentation and/or preprocessing(see complete list [here](https://pytorch.org/docs/stable/torchvision/transforms.html)). \n",
        "\n",
        "Your task is to add a second transformation that normalizes the pixel intensities to range -1 to 1. Use [`transforms.Normalize()`](https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Normalize) with suitable input parameters. The normalization is based on the equation\n",
        "\n",
        "```\n",
        "y = (x - mean) / std\n",
        "```\n",
        "\n",
        "where mean and std are user-defined."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cqiXairclI6D"
      },
      "source": [
        "transform = transforms.Compose(\n",
        "    [transforms.ToTensor(),\n",
        "     transforms.Normalize(???)]) # Your code goes here\n",
        "\n",
        "# Re-initialize trainloader and testloader, this time with the transform included\n",
        "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
        "                                        download=True, transform=transform)\n",
        "trainloader = torch.utils.data.DataLoader(trainset, batch_size=bs,\n",
        "                                          shuffle=True, num_workers=2)\n",
        "\n",
        "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
        "                                       download=True, transform=transform)\n",
        "testloader = torch.utils.data.DataLoader(testset, batch_size=bs,\n",
        "                                         shuffle=False, num_workers=2)\n",
        "\n",
        "# Verify that intensities are in range -1 to 1\n",
        "dataiter = iter(trainloader)\n",
        "images, labels = dataiter.next()\n",
        "print('images.min()',images.min())\n",
        "print('images.max()',images.max())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7I4AH1C1LRl"
      },
      "source": [
        "###Questions\n",
        "Below you are going to initialize and normalize the network weights based. To prepare for this task, answer the following questions using appropriate PyTorch functions (use Google to figure out which):\n",
        "1. What is the mean and standard deviation (std) of tensor `x` below?\n",
        "2. What is the mean and std `y`?\n",
        "3. What is the mean and std `z`?\n",
        "\n",
        "Bonus task: See of you can figure out a formula to calculate the mean and std of `x` and `y`, given the mean and std and `x`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3aY6pX-q1G-C"
      },
      "source": [
        "x = torch.randn(512, 512)\n",
        "y = x * 10 + 2\n",
        "z = (y-2) / 10"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y945T69aHKVy"
      },
      "source": [
        "##3. Base model\n",
        "We'll first create a base model that we can use for our initial experiments. The base model is a fully connected neural network. By default it has\n",
        "\n",
        "- `L = 10` layers\n",
        "- `N = 16` units in each hidden layer.\n",
        "\n",
        "When initializing the model you can specify \n",
        "\n",
        "- a `normalizer` function, which is applied when initializing the weight matrix `W` of each layer. The input to the normalizer function is `W` and the output is some normalized version of `W`, like Xavier or Kaiming.\n",
        "\n",
        "- an `activation_function` which could be sigmoid, tanh, ReLu, etc.\n",
        "\n",
        "- a `preprocess_function` which is applied in each layer after calculating `Wx+b` but before applying the activation function. It can be used to implement batch normalization.\n",
        "\n",
        "Initially we set all these functions to the identity function. This makes the base model is a purely linear model without any activation functions, normalization or anything like that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vdOBKw9G0YPO"
      },
      "source": [
        "from torch import nn\n",
        "\n",
        "def base_normalizer(x): return x\n",
        "def base_activation(x): return x\n",
        "def base_preprocess(x): return x\n",
        "\n",
        "class BaseModel(nn.Module):\n",
        "    def __init__(self,\n",
        "                 normalizer = base_normalizer,\n",
        "                 activation_function = base_activation,\n",
        "                 preprocess_function = base_preprocess,\n",
        "                 L = 10,\n",
        "                 N = 16):\n",
        "      \n",
        "        super().__init__()\n",
        "        \n",
        "        self.W = nn.ParameterList() # List of weights\n",
        "        self.b = nn.ParameterList() # List of biases\n",
        "        self.L = L # Number of layers\n",
        "        self.N = N # Number of units in each hidden layer\n",
        "        self.activation_function = activation_function\n",
        "        self.preprocess_function = preprocess_function\n",
        "        self.layer_activations = [] # Store layer activations here\n",
        "\n",
        "        # Initialize weights and biases\n",
        "        for layer in range(L):\n",
        "          dims = [self.N, self.N] # Size of hidden layer\n",
        "          if layer == 0: dims = [32*32*3,N] # Size of first layer\n",
        "          if layer == L-1: dims = [N,10] # Size of last layer (10 classes)\n",
        "          W = nn.Parameter(normalizer(torch.randn(dims[0], dims[1]))) # Call normalizer here\n",
        "          b = nn.Parameter(torch.zeros(dims[1]))            \n",
        "          self.W.append(W)\n",
        "          self.b.append(b)\n",
        "\n",
        "    # Forward propagation\n",
        "    def forward(self, x):\n",
        "        self.layer_activations = []\n",
        "        x = x.view(-1, 32*32*3) # Vectorize image to a 32*32*3 dimensional vector\n",
        "        for layer in range(self.L):\n",
        "          x = x @ self.W[layer] + self.b[layer]\n",
        "          x = self.preprocess_function(x) # Call preprocess function before activation\n",
        "          x = self.activation_function(x)\n",
        "          self.layer_activations.append(x) # Store activations\n",
        "        return x\n",
        "\n",
        "    # Return stored layer activations\n",
        "    def activations(self):\n",
        "        return self.layer_activations"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83a4sLSObAyE"
      },
      "source": [
        "###3.1 Test the model\n",
        "Let's test the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p68n0tD_bEFO",
        "outputId": "c6fdcba6-aef3-441e-9ccd-d413a5ee4f0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Move data to GPU\n",
        "images = images.cuda()\n",
        "labels = labels.cuda()\n",
        "\n",
        "# Calculate scores\n",
        "model = BaseModel().cuda()\n",
        "scores = model(images)  # predictions\n",
        "\n",
        "print(scores.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([256, 10])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HP2f2yzibZXg"
      },
      "source": [
        "###3.2 Calculating the accuracy\n",
        "These numbers are scores (logits), which don't have any meaningful interpretation. We can convert them into class probabilities using softmax. Since we are only going to be interested in the model's accuracy, we will wrap the softmax inside the function `accuracy` that calculates the accuracy on a batch:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f_hUiKnIcBZU",
        "outputId": "6215d837-0165-4a5d-9cf2-09c48d5615fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "def accuracy(scores, yb):\n",
        "    score2prob = nn.Softmax(dim=1)\n",
        "    preds = torch.argmax(score2prob(scores), dim=1)\n",
        "    return (preds == yb).float().mean()\n",
        "\n",
        "print('Accuracy', accuracy(scores,labels))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy tensor(0.1328, device='cuda:0')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Sclk5hbtRgK"
      },
      "source": [
        "### Question\n",
        "1. What does `torch.argmax` do?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4gCGFOqycidx"
      },
      "source": [
        "###3.3 Calculating the loss\n",
        "I order to train your model, we also need a loss function. We will use the cross entropy loss [already provided in PyTorch](https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.cross_entropy). Note that `cross_entropy` does the softmax for you, so the input is just the scores."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tpdvLkhNxaTc",
        "outputId": "87ea941b-7c7f-4043-c787-feba371497f7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import torch.nn.functional as F\n",
        "loss_func = F.cross_entropy\n",
        "loss = loss_func(scores, labels)\n",
        "print('Loss', loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loss tensor(10602066., device='cuda:0', grad_fn=<NllLossBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SILeyqxcc7ed"
      },
      "source": [
        "###3.4 Getting layer activations and gradients\n",
        "One special ting about our model is that it stores the activations after each layer (in variable `self.layer_activations`).\n",
        "\n",
        "Our goal is develop tools that we can use to inspect our model both before and during training. For this purpose it will be useful to be able to grab the layer activations as well as the gradients at each layer. The activations have alreay been stored during our forward propagation and can be extracted as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "diJkCQWMd3xH",
        "outputId": "d4175381-d293-4612-a8b1-6a67a563d87f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "activations = model.activations()\n",
        "print('Activation list length:',len(activations))\n",
        "print('Activations shape layer 1:',activations[0].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Activation list length: 10\n",
            "Activations shape layer 1: torch.Size([256, 16])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojGryp4yeLDX"
      },
      "source": [
        "To get the gradients w.r.t. the loss, first call `loss.backward()` and then access the `grad` property of the relevant model parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7chAT_iNf9vf"
      },
      "source": [
        "# Calculate gradients\n",
        "loss.backward()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IcypHRQ1gls5"
      },
      "source": [
        "*Side-note:* If you are interested in understanding in detail how `backward()` works, I recommend [this blog post](https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95).\n",
        "\n",
        "Since we are going to need both the layer activations and layer-wise gradients, lets wrap that up in a single function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "btQ9Lid5glGN",
        "outputId": "e4554ca6-90e7-4b5d-d091-9a93c68d3019",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 101
        }
      },
      "source": [
        "def get_layer_data(model):\n",
        "  gradients = []\n",
        "  layer_names = []\n",
        "\n",
        "  with torch.no_grad():\n",
        "    for name, param in model.named_parameters():\n",
        "      if param.requires_grad and name.startswith('W'):\n",
        "          layer_names.append(name)\n",
        "          gradients.append(param.grad)\n",
        "\n",
        "  activations = model.activations()\n",
        "\n",
        "  return layer_names, activations, gradients\n",
        "\n",
        "layer_names, activations, gradients = get_layer_data(model)\n",
        "print('layer_names',layer_names)\n",
        "print('Activation list length:',len(activations))\n",
        "print('Activations shape layer 1:',activations[0].shape)\n",
        "print('Gradient list length:',len(gradients))\n",
        "print('Gradient shape layer 1:',gradients[0].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "layer_names ['W.0', 'W.1', 'W.2', 'W.3', 'W.4', 'W.5', 'W.6', 'W.7', 'W.8', 'W.9']\n",
            "Activation list length: 10\n",
            "Activations shape layer 1: torch.Size([256, 16])\n",
            "Gradient list length: 10\n",
            "Gradient shape layer 1: torch.Size([3072, 16])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sAz2dtpSiA3m"
      },
      "source": [
        "Note that we are not considering the biases in this Lab."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VALKE9BohZIs"
      },
      "source": [
        "###Questions\n",
        "1. Why is the shape of the activations of the first layer [256,16]?\n",
        "2. Why is the shape of the gradients of the first layer [3072,16]?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oaV8gN64ihO0"
      },
      "source": [
        "###3.5 Getting layer stats\n",
        "We can create a simple function that calculates the mean and the variance of the layer activations and gradients. For the gradients, we are interested in the *gradient flow*, so we will take the mean of the absolute gradient values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IczlmBWRGkte",
        "outputId": "61c0d591-b088-40e7-d838-8354ef9e7181",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "def get_layer_stats(x,absolute=False):\n",
        "  avg = []\n",
        "  std = []\n",
        "  for layer in range(len(x)):\n",
        "    if absolute: # for gradient flow\n",
        "      avg.append(x[layer].abs().mean().detach().cpu().numpy())\n",
        "    else: # for activations\n",
        "      avg.append(x[layer].mean().detach().cpu().numpy())\n",
        "    \n",
        "    std.append(x[layer].std().detach().cpu().numpy())\n",
        "\n",
        "  return avg, std\n",
        "\n",
        "activation_mean, activation_std = get_layer_stats(activations)\n",
        "gradient_mean, gradient_std = get_layer_stats(gradients,absolute=True)\n",
        "\n",
        "print('activation_mean',activation_mean)\n",
        "print('activation_std',activation_std)\n",
        "print('gradient_mean',gradient_mean)\n",
        "print('gradient_std',gradient_std)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "activation_mean [array(1.4172416, dtype=float32), array(-1.7468169, dtype=float32), array(-22.715889, dtype=float32), array(32.521477, dtype=float32), array(-203.4187, dtype=float32), array(-239.7403, dtype=float32), array(4276.4805, dtype=float32), array(642.9778, dtype=float32), array(25003.129, dtype=float32), array(82938.1, dtype=float32)]\n",
            "activation_std [array(28.163591, dtype=float32), array(112.06292, dtype=float32), array(435.47925, dtype=float32), array(1890.085, dtype=float32), array(8096.546, dtype=float32), array(33412.457, dtype=float32), array(150908.6, dtype=float32), array(447697.62, dtype=float32), array(1855403.6, dtype=float32), array(7517420., dtype=float32)]\n",
            "gradient_mean [array(7619.8037, dtype=float32), array(100021.23, dtype=float32), array(96390., dtype=float32), array(128607.49, dtype=float32), array(127211.95, dtype=float32), array(179272.6, dtype=float32), array(228736.19, dtype=float32), array(234113.31, dtype=float32), array(142066.61, dtype=float32), array(120784.15, dtype=float32)]\n",
            "gradient_std [array(9543.602, dtype=float32), array(148811.34, dtype=float32), array(156618.58, dtype=float32), array(182660.06, dtype=float32), array(185742.14, dtype=float32), array(300866.2, dtype=float32), array(347462.53, dtype=float32), array(293546., dtype=float32), array(212261.95, dtype=float32), array(235263.36, dtype=float32)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NiiKrxrGkH8w"
      },
      "source": [
        "###3.6 Visualizing layer stats\n",
        "Let's also make a function that allows us to plot histograms (and display mean and standard deviation at the same time).\n",
        "\n",
        "Note that you can specifiy a fixed range of x values (like -1 to 1). If set to `None` the x range is adapted to each individual plot. So notice the values on the x-axis, when you plot the histograms!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2il8nJOukZfk"
      },
      "source": [
        "def plot_hist(hs,xrange=(-1,1),avg=None,sd=None):\n",
        "  plt.figure(figsize=(20,3))\n",
        "  for layer in range(len(hs)):\n",
        "    plt.subplot(1,len(hs),layer+1)\n",
        "    activations = hs[layer].detach().cpu().numpy().flatten()\n",
        "    plt.hist(activations, bins=20, range=xrange)\n",
        "\n",
        "    title = 'Layer ' + str(layer+1)\n",
        "    if avg:\n",
        "      title += '\\n' + \"mean {0:.2f}\".format(avg[layer])\n",
        "    if sd:\n",
        "      title += '\\n' + \"std {0:.4f}\".format(sd[layer])\n",
        "\n",
        "    plt.title(title)\n",
        "\n",
        "print('Gradients:\\n')\n",
        "plot_hist(gradients,xrange=None,avg=gradient_mean,sd=gradient_std)\n",
        "plt.show()\n",
        "\n",
        "print('Activations:\\n')\n",
        "plot_hist(activations,xrange=None,avg=activation_mean,sd=activation_std)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Infpb-u-l4Z2"
      },
      "source": [
        "### Questions\n",
        "Carefully inspect the histograms above, and remember that the x-axes can be scaled differently.\n",
        "1. What happens to the gradients across layers? How are they distributed? Do they get smaller or larger towards the end of the network?\n",
        "2. What about the activations?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W9GH7xMKndnB"
      },
      "source": [
        "###3.7 Wrapping up\n",
        "Finally, let's combine all of the above into just one function that we can call. To do this we first define a helper function `get_stats` which takes a model as input, runs a batch through the model, and calculates all the stats that we need."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9IpKGmjVoJ_6"
      },
      "source": [
        "def get_stats(model,dataloader=trainloader):\n",
        "\n",
        "  dataiter = iter(dataloader)\n",
        "  images, labels = dataiter.next()\n",
        "  images = images.cuda()\n",
        "  labels = labels.cuda()\n",
        "\n",
        "  scores = model(images)  # predictions\n",
        "  loss = loss_func(scores, labels)\n",
        "  acc = accuracy(scores,labels)\n",
        "  \n",
        "  # Calculate gradients\n",
        "  loss.backward()\n",
        "\n",
        "  layer_names, activations, gradients = get_layer_data(model)\n",
        "\n",
        "  activation_mean, activation_std = get_layer_stats(activations)\n",
        "  gradient_mean, gradient_std = get_layer_stats(gradients,absolute=True)\n",
        "\n",
        "  stats = {'loss': loss,\n",
        "           'accuracy': acc,\n",
        "           'names': layer_names,\n",
        "           'grads': gradients,\n",
        "           'activations': activations,\n",
        "           'activation_mean': activation_mean,\n",
        "           'activation_std': activation_std,\n",
        "           'gradient_mean': gradient_mean,\n",
        "           'gradient_std': gradient_std\n",
        "           }\n",
        "\n",
        "  return stats"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBT8SJstoTpG"
      },
      "source": [
        "Here is the function to display the stats:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pcy9gvEsx8bv"
      },
      "source": [
        "def show_stats(stats):\n",
        "  print('loss',stats['loss'].detach().cpu().numpy())\n",
        "  print('accuracy',stats['accuracy'].detach().cpu().numpy(),'\\n')\n",
        "  \n",
        "  print('Gradients:\\n')\n",
        "  print(' (note that we use the mean of the absolute gradient values to quantify gradient flow\\n')\n",
        "  #[print(name, avg, std) for name, avg, std in iter(zip(stats['names'],stats['gradient_mean'],stats['gradient_std']))]\n",
        "\n",
        "  plot_hist(stats['grads'],xrange=None,avg=stats['gradient_mean'],sd=stats['gradient_std'])\n",
        "  plt.show()\n",
        "\n",
        "  print('Activations:\\n')\n",
        "  #[print(name, avg, std) for name, avg, std in iter(zip(stats['names'],stats['activation_mean'],stats['activation_std']))]\n",
        "    \n",
        "  plot_hist(stats['activations'],xrange=None,avg=stats['activation_mean'],sd=stats['activation_std'])\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XN8DoAeVoh5E"
      },
      "source": [
        "So from now on we just make these two function calls:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9YmjWw8Erer"
      },
      "source": [
        "# Calculate stats\n",
        "stats = get_stats(model)\n",
        "\n",
        "# Show stats\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zksMi4QUpZnP"
      },
      "source": [
        "## 4. Task 1: tanh activation\n",
        "Your task is very simple: Add tanh activation function to our base model and explain what you see (use Google to search PyTorch's documentation and tanh)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VYTdu6g4wyQ-"
      },
      "source": [
        "def tanh(x):\n",
        "  # Your code goes here\n",
        "\n",
        "model_tanh = BaseModel(activation_function=tanh).cuda()\n",
        "stats = get_stats(model_tanh)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dRUIID0JqZ2t"
      },
      "source": [
        "### Questions:\n",
        "1. How are the gradients and activations distributed across layers?\n",
        "2. Is there gradient flow through the entire network?\n",
        "3. Do the activations have similar variance across layers?\n",
        "4. Are we within the linear range of tanh, within the saturated range, or a little of both?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V_JhLhvprXIT"
      },
      "source": [
        "##5. Task 2: Increase values of initial weights\n",
        "In our base model the weights are initialized with random values from a standard normal distribution (mean 0 and standard deviation 1). Let's increase the std to 10 and see what happens. Note that we keep using tanh activation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P5i3_DabFXbN"
      },
      "source": [
        "def upscale(x): return x * 10\n",
        "\n",
        "model_saturated = BaseModel(normalizer=upscale, activation_function=tanh).cuda()\n",
        "stats = get_stats(model_saturated)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZcQmnrlr7WQ"
      },
      "source": [
        "### Questions:\n",
        "1. How are the gradients and activations distributed across layers?\n",
        "2. Is there gradient flow through the entire network?\n",
        "3. Do the activations have similar variance across layers?\n",
        "4. Are we within the linear range of tanh, within the saturated range, or a little of both?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUiAszp6sK0S"
      },
      "source": [
        "##6. Task 3: Implement Xavier initialization\n",
        "Your task is to implement Xavier initialization by providing a new normalizer function. Remember that the normalizer takes a weight matrix (`x`) as input, which contains random values with zero mean and standard deviation 1. To change the standard deviation of `x` to some value `sd`, simply multiply `x` with `sd`.\n",
        "\n",
        "Note that there are different ways to implement Xavier. In the lecture slides, we normalize based on the number of columns of the weight matrix (often referred to as the *fan-in*). Others normalize based on the number of rows (*fan-out*). Others again normalize based on the sum of the fan-in and fan-out. For instance, [this is how PyTorch does it](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_normal_).\n",
        "\n",
        "**Do not use PyTorch's Xavier implementation - make your own :-)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WOTVJkerzNEs"
      },
      "source": [
        "def xavier(x):\n",
        "  # Your code goes here\n",
        "\n",
        "model_xavier = BaseModel(normalizer=xavier, activation_function=tanh).cuda()\n",
        "stats = get_stats(model_xavier)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeB6m1H_tMIU"
      },
      "source": [
        "### Questions:\n",
        "1. How are the gradients and activations distributed across layers?\n",
        "2. Is there gradient flow through the entire network?\n",
        "3. Do the activations have similar variance across layers?\n",
        "4. Are we within the linear range of tanh, within the saturated range, or a little of both?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C3bOqMwBta9S"
      },
      "source": [
        "##7. Task 4: ReLu and Kaiming initialization\n",
        "Your task is to change the activation function to ReLU. In this case, we know that Xavier initialization will not work, so you also need to implement Kaiming initialization.\n",
        "\n",
        "You can check PyTorch's Kaiming formula [here](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_) (again, recall that fan-in is the number of columns of the weight matrix).\n",
        "\n",
        "**Again, make our own implementation - do not PyTorch's**\n",
        "\n",
        "**Hint:** For ReLU you could use PyTorch's [`clamp`](https://pytorch.org/docs/stable/generated/torch.clamp.html#torch.clamp) function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jh4udDWJEXop"
      },
      "source": [
        "def relu(x):\n",
        "  # Your code goes here\n",
        "\n",
        "def kaiming(x):\n",
        "  # Your code goes here\n",
        "\n",
        "model_kaiming = BaseModel(normalizer=kaiming, activation_function=relu).cuda()\n",
        "stats = get_stats(model_kaiming)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqhJnpJIugjr"
      },
      "source": [
        "### Questions:\n",
        "1. How are the gradients and activations distributed across layers?\n",
        "2. Is there gradient flow through the entire network?\n",
        "3. Do the activations have similar variance across layers?\n",
        "4. Do enough of the ReLUs get activated in all layers?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cExn9vfUuy_P"
      },
      "source": [
        "##8. Task 5: Batch normalization\n",
        "Finally, see if you can implement simple batch normalization (without the learnable parameters).\n",
        "\n",
        "Recall that batch norm is applied **after** the linear operation (say `Wx + b`) and **before** the non-linear activation (say `tanh`). Defining `y = Wx+b`, batch normalization should output `z = (y - mean(y)) / std(y)`. If implemented correctly, `z` will have mean 0 and standard deviation 1.\n",
        "\n",
        "In our base model, batch normalization is passed on as a preprocessing function. So your task is to implement this function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XOkgJsggBdzd"
      },
      "source": [
        "def batch_norm(y):\n",
        "  # Your code goes here\n",
        "  return ???"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GQxifTwawa8I"
      },
      "source": [
        "# Check your results\n",
        "y = torch.randn(512,512)*10 + 2\n",
        "print('before', y.mean(),y.std())\n",
        "z = batch_norm(y)\n",
        "print('after', z.mean(),z.std())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qSGhVl2225S6"
      },
      "source": [
        "If your implementation is correct, z should have mean 0 and standard deviation 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNymTnEoDP-4"
      },
      "source": [
        "model_batchnorm = BaseModel(normalizer=kaiming, activation_function=relu,preprocess_function=batch_norm).cuda()\n",
        "stats = get_stats(model_batchnorm)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SrsmRwouwO4a"
      },
      "source": [
        "### Questions:\n",
        "1. How are the gradients and activations distributed across layers?\n",
        "2. Is there gradient flow through the entire network?\n",
        "3. Do the activations have similar variance across layers?\n",
        "4. Do enough of the ReLUs get activated in all layers?\n",
        "\n",
        "Compare to the results with ReLu and Kaiming (without batch norm).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "praCQ52q_LwQ"
      },
      "source": [
        "##9. Model training\n",
        "The code below can be used to train a model and monitor important stats as training progresses. The training is carried out by calling the `fit` function, which takes a model as input, as well as a function handle returning an optimizer. The base optimizer is just SGD with zero momentum. You can change the optimizer by replacing `base_optimizer`with your own function handle.\n",
        "\n",
        "`lr` is the learning rate, `bs` the batch size, and `epochs` the number of epochs.\n",
        "\n",
        "(Note: the cpu/gpu copying and torch/numpy conversions could have been made nicer - sorry...)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "56_hDxQWHKWo"
      },
      "source": [
        "from torch import optim\n",
        "\n",
        "# Function handle that returns an optimizer - in this case just a simple SGD without momentum\n",
        "def base_optimizer(model,lr=0.1):\n",
        "    return optim.SGD(model.parameters(), lr=lr,momentum=0.)\n",
        "\n",
        "# Used to print gradients/activations as a function of time\n",
        "def print_history(history,title=''):\n",
        "  plt.figure()\n",
        "  history = np.asarray(history)\n",
        "  lines = []\n",
        "  labels = []\n",
        "  for i in range(history.shape[1]):\n",
        "    l, = plt.plot(history[:,i])\n",
        "    lines.append(l)\n",
        "    labels.append('Layer ' + str(i+1))\n",
        "  plt.legend(lines, labels, loc=(1, 0), prop=dict(size=14))\n",
        "  plt.title(title)\n",
        "    \n",
        "# Function to fit a model\n",
        "def fit(model,\n",
        "        opt_func=base_optimizer,\n",
        "        lr=0.1,\n",
        "        bs=256,\n",
        "        epochs=2):\n",
        "\n",
        "  train_dl = torch.utils.data.DataLoader(trainset, batch_size=bs,\n",
        "                                          shuffle=True, num_workers=2)\n",
        "\n",
        "  valid_dl = torch.utils.data.DataLoader(testset, batch_size=bs,\n",
        "                                         shuffle=False, num_workers=2)\n",
        "\n",
        "  opt = opt_func(model,lr) # Initialize optimizer\n",
        "\n",
        "  train_loss_history = []\n",
        "  valid_loss_history = []\n",
        "  plot_time_train = []\n",
        "  plot_time_valid = []\n",
        "  activation_mean_history = []\n",
        "  gradient_mean_history = []\n",
        "\n",
        "  t = 1\n",
        "\n",
        "  # Get initial validation loss and accuracy\n",
        "  model.eval()\n",
        "  with torch.no_grad():\n",
        "    valid_acc = sum(accuracy(model(xb.cuda()), yb.cuda()) for xb, yb in valid_dl) / len(valid_dl)\n",
        "    valid_loss = sum(loss_func(model(xb.cuda()), yb.cuda()) for xb, yb in valid_dl) / len(valid_dl)\n",
        "    valid_loss_history.append(valid_loss)\n",
        "    plot_time_valid.append(t)\n",
        "\n",
        "  # Train\n",
        "  for epoch in range(epochs):\n",
        "    model.train()\n",
        "    for xb, yb in train_dl:\n",
        "      pred = model(xb.cuda())\n",
        "      loss = loss_func(pred, yb.cuda())\n",
        "\n",
        "      train_loss_history.append(loss.detach().cpu().numpy())\n",
        "      plot_time_train.append(t)\n",
        "      t += 1\n",
        "\n",
        "      loss.backward()\n",
        "\n",
        "      layer_names, activations, gradients = get_layer_data(model)\n",
        "      activation_mean, activation_std = get_layer_stats(activations)\n",
        "      gradient_mean, gradient_std = get_layer_stats(gradients,absolute=True)\n",
        "      activation_mean_history.append(activation_mean)\n",
        "      gradient_mean_history.append(gradient_mean)\n",
        "\n",
        "      opt.step()\n",
        "      opt.zero_grad() # By default, PyTorch accumulates the gradient - force to zero before next iteration\n",
        "\n",
        "    # Validation loss and accuracy\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        valid_acc = sum(accuracy(model(xb.cuda()), yb.cuda()) for xb, yb in valid_dl) / len(valid_dl)\n",
        "        valid_loss = sum(loss_func(model(xb.cuda()), yb.cuda()) for xb, yb in valid_dl) / len(valid_dl)\n",
        "        valid_loss_history.append(valid_loss)\n",
        "        plot_time_valid.append(t-1)\n",
        "        if epoch == epochs-1:\n",
        "          print('validation loss',valid_loss.detach().cpu().numpy())\n",
        "          print('validation accuracy', valid_acc.detach().cpu().numpy())\n",
        "\n",
        "  # Summary\n",
        "  plt.figure()\n",
        "  plt.plot(plot_time_train,train_loss_history)\n",
        "  plt.plot(plot_time_valid,valid_loss_history)\n",
        "  plt.title('Loss')  \n",
        "  print_history(activation_mean_history,'Layer activations (mean)')\n",
        "  print_history(gradient_mean_history,'Layer gradients (mean)')\n",
        "  print('train loss',loss_func(model(xb.cuda()), yb.cuda()).detach().cpu().numpy())\n",
        "  print('train accuracy', accuracy(model(xb.cuda()), yb.cuda()).detach().cpu().numpy())\n",
        "  print('\\n')\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3V0LMYxaN85v"
      },
      "source": [
        "###9.1 Training the base model\n",
        "Let's train the base model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9VMx9bxyEPrs"
      },
      "source": [
        "model = BaseModel().cuda() # Re-initialize weights\n",
        "fit(model,lr=1e-8)\n",
        "stats = get_stats(model)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXYnX37XO8bG"
      },
      "source": [
        "###Questions\n",
        "1. What is the validation accuracy?\n",
        "2. What do the first three plots show?\n",
        "3. What happened to the gradient flow over time? Did it increase, decrease, or stay constant?\n",
        "4. What happens if you increase the learning rate to 0.1?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ufjD_62jPQSy"
      },
      "source": [
        "###9.2 Training the tanh model\n",
        "Recall that the base model is entirely linear (i.e., no activation functions used). Let's see what happens if we add tanh activations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mk0H5D2LCT3t"
      },
      "source": [
        "model_tanh = BaseModel(activation_function=tanh).cuda() # Re-initialize weights\n",
        "fit(model_tanh,lr=0.1)\n",
        "stats = get_stats(model_tanh)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fkBQ4ZDMPhRv"
      },
      "source": [
        "###Questions\n",
        "1. Did results improve significantly?\n",
        "2. Are the activations within the linear range of tanh, within the saturated range, or a mix of both?\n",
        "3. What happens to the gradients over time?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPwnhZQhQNqd"
      },
      "source": [
        "###9.3 Could we possible do any worse?\n",
        "Yes! Let's increase the std of the weights (from 1 to 10) by calling the `upscale` function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FPeBEzGEF2aN"
      },
      "source": [
        "model_saturated = BaseModel(normalizer=upscale, activation_function=tanh).cuda() # Re-initialize weights\n",
        "fit(model_saturated,lr=0.1)\n",
        "stats = get_stats(model_saturated)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r6ESARbbQjO6"
      },
      "source": [
        "###Questions\n",
        "1. What happens to the loss over time? Does it decrease, as it should?\n",
        "2. It seems that there is almost no gradient signal. Why? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCApI-TCHKWq"
      },
      "source": [
        "###9.4 Xavier initialization\n",
        "Now, let's see what happens with Xavier initialization:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oufKaXqtHKWr"
      },
      "source": [
        "model_xavier = BaseModel(normalizer=xavier, activation_function=tanh).cuda()\n",
        "fit(model_xavier,lr=0.1)\n",
        "stats = get_stats(model_xavier)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhKplpxGR15s"
      },
      "source": [
        "###Questions\n",
        "1. What happened to the validation accuracy?\n",
        "2. What happens to the gradient signal over time? Does it vanish? Or is it maintained?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50C_gv8HSOl7"
      },
      "source": [
        "###9.5 ReLU and Kaiming initialization\n",
        "Does ReLU with Kaiming initialization perform better than tanh with Xavier initialization?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lUzWSVU2EwrD"
      },
      "source": [
        "model_kaiming = BaseModel(normalizer=kaiming, activation_function=relu).cuda()\n",
        "fit(model_kaiming,lr=0.1)\n",
        "stats = get_stats(model_kaiming)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0hDbVWHgBLlL"
      },
      "source": [
        "###Questions\n",
        "1. We do not end up at the same level of accuracy as with tanh and Xavier initialization. Why do you think that is?\n",
        "2. Could we fix it?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQJdvWcsSjZK"
      },
      "source": [
        "###9.6 Batch normalization\n",
        "On average, results should become better if we add batch normalization:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Ut46Hb6DrPW"
      },
      "source": [
        "model_batchnorm = BaseModel(normalizer=kaiming, activation_function=relu,preprocess_function=batch_norm).cuda()\n",
        "fit(model_batchnorm,lr=0.1)\n",
        "stats = get_stats(model_batchnorm)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c6Iiz96aS7iI"
      },
      "source": [
        "###Questions\n",
        "1. Compare the curves with and without batch normalization (i.e., section 9.5 and 9.6). Can you explain the differences?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GzPSPmyrTeSw"
      },
      "source": [
        "###9.7 Decreasing the batch size\n",
        "Finally, let's see what happens if we decrease the batch size from 256 to 32:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXAtDU-ANFW6"
      },
      "source": [
        "bs=32\n",
        "model_batchsize = BaseModel(normalizer=kaiming, activation_function=relu,preprocess_function=batch_norm).cuda()\n",
        "fit(model_batchsize,lr=0.1,bs=bs)\n",
        "stats = get_stats(model_batchsize)\n",
        "show_stats(stats)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TuK5QM3AT5VY"
      },
      "source": [
        "###Questions\n",
        "1. What just happened to the three top-most curves? Why?\n",
        "2. Did your results get better, worse, or same?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4NOgnYtEUXlL"
      },
      "source": [
        "## 9. Optimizers\n",
        "Below I have defined a few other optimizers (read more here: https://pytorch.org/docs/stable/optim.html). Your task is to experiment with these.\n",
        "\n",
        "Suggestions:\n",
        "\n",
        "1. Start with `momentum_optimizer` and train the model with momenum 0.0, 0.9, 0.99 and 1.0. See if you can explain what happens.\n",
        "\n",
        "2. Move on to experiment with Adagrad or RMSprop. Notice any different behavior?\n",
        "\n",
        "3. Finally, try Adam with different learning rates. Theory says it shouldn't make a huge difference. Do your experiments confirm this?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "46WEppfOLVQj"
      },
      "source": [
        "#SGD + momentum\n",
        "def momentum_optimizer(model,lr=0.1):\n",
        "    return optim.SGD(model.parameters(), lr=lr, momentum=0.9)\n",
        "\n",
        "def adagrad_optimizer(model,lr=0.1):\n",
        "  return optim.Adagrad(model.parameters(), lr=lr, lr_decay=0.0, weight_decay=0.0, initial_accumulator_value=0)\n",
        "\n",
        "def rmsprop_optimizer(model,lr=0.1):\n",
        "  return optim.RMSprop(model.parameters(), lr=lr, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False)\n",
        "\n",
        "def adam_optimizer(model,lr=0.001):\n",
        "  return optim.Adam(model.parameters(), lr=lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)\n",
        "\n",
        "model = BaseModel(normalizer=kaiming, activation_function=relu,preprocess_function=batch_norm).cuda()\n",
        "fit(model,opt_func=momentum_optimizer,lr=0.01,bs=64)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZL_9mIyLB3_N"
      },
      "source": [
        "##10. Competition\n",
        "How high validation accuracy can you get in 5 epochs?\n",
        "\n",
        "Rules - you are allowed to:\n",
        "- modify the number of layers of the base model (parameter `L`).\n",
        "- modify the unit in the hidden layers of the base model (parameter `N`).\n",
        "- choose between different activation functions and normalizers\n",
        "- use batch norm\n",
        "- change the batch size\n",
        "- use any optimizer and any optimization hyperparameters (e.g., learning rate)\n",
        "- and **no more than 5 epochs!**\n",
        "\n",
        "Think about your hyperparameter search strategy. What quick pre-experiments could you do before training the final model?\n",
        "\n",
        "You are not done, until you have reached at least 50% accuracy! ;-)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPOCvs9eRdnb"
      },
      "source": [
        "##11. Optional\n",
        "If you want a dataset, where you can achieve higher accuracy, try MNIST. Below are all the pieces you need to get started."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tr6zwwYSZgDR"
      },
      "source": [
        "from pathlib import Path\n",
        "import requests\n",
        "\n",
        "DATA_PATH = Path(\"data\")\n",
        "PATH = DATA_PATH / \"mnist\"\n",
        "\n",
        "PATH.mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "URL = \"http://deeplearning.net/data/mnist/\"\n",
        "FILENAME = \"mnist.pkl.gz\"\n",
        "\n",
        "if not (PATH / FILENAME).exists():\n",
        "        content = requests.get(URL + FILENAME).content\n",
        "        (PATH / FILENAME).open(\"wb\").write(content)\n",
        "\n",
        "import pickle\n",
        "import gzip\n",
        "\n",
        "with gzip.open((PATH / FILENAME).as_posix(), \"rb\") as f:\n",
        "  ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding=\"latin-1\")\n",
        "\n",
        "x_train, y_train, x_valid, y_valid = map(\n",
        "    torch.tensor, (x_train, y_train, x_valid, y_valid)\n",
        ")\n",
        "\n",
        "train_ds = TensorDataset(x_train, y_train)\n",
        "train_dl = DataLoader(train_ds, batch_size=bs, shuffle=True)\n",
        "\n",
        "valid_ds = TensorDataset(x_valid, y_valid)\n",
        "valid_dl = DataLoader(valid_ds, batch_size=bs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vnl8gRJqP8nl"
      },
      "source": [
        ""
      ]
    }
  ]
}