t81_558_class_04_3_dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_3_dropout.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "\n",
    "**Module 4: Training for Tabular Data**\n",
    "\n",
    "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "- Part 4.1: Using K-Fold Cross-validation with PyTorch [[Video]](https://www.youtube.com/watch?v=Q8ZQNvZwsNE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_1_kfold.ipynb)\n",
    "- Part 4.2: Training Schedules for PyTorch  [[Video]](https://www.youtube.com/watch?v=lMMlbmfvKDQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_2_schedule.ipynb)\n",
    "- **Part 4.3: Dropout Regularization** [[Video]](https://www.youtube.com/watch?v=4ixjgw6Q42U&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_3_dropout.ipynb)\n",
    "- Part 4.4: Batch Normalization [[Video]](https://www.youtube.com/watch?v=1U5nOKh9OLQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_4_batch_norm.ipynb)\n",
    "- Part 4.5: RAPIDS for Tabular Data [[Video]](https://www.youtube.com/watch?v=KgoXuhG_kfs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_5_rapids.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed. We also initialize the PyTorch device to either GPU/MPS (if available) or CPU.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'\n",
    "import copy\n",
    "import torch\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")\n",
    "\n",
    "# Early stopping (see module 3.4)\n",
    "class EarlyStopping:\n",
    "    def __init__(self, patience=5, min_delta=0, restore_best_weights=True):\n",
    "        self.patience = patience\n",
    "        self.min_delta = min_delta\n",
    "        self.restore_best_weights = restore_best_weights\n",
    "        self.best_model = None\n",
    "        self.best_loss = None\n",
    "        self.counter = 0\n",
    "        self.status = \"\"\n",
    "\n",
    "    def __call__(self, model, val_loss):\n",
    "        if self.best_loss is None:\n",
    "            self.best_loss = val_loss\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "        elif self.best_loss - val_loss >= self.min_delta:\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "            self.best_loss = val_loss\n",
    "            self.counter = 0\n",
    "            self.status = f\"Improvement found, counter reset to {self.counter}\"\n",
    "        else:\n",
    "            self.counter += 1\n",
    "            self.status = f\"No improvement in the last {self.counter} epochs\"\n",
    "            if self.counter >= self.patience:\n",
    "                self.status = f\"Early stopping triggered after {self.counter} epochs.\"\n",
    "                if self.restore_best_weights:\n",
    "                    model.load_state_dict(self.best_model)\n",
    "                return True\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.3: Dropout Regularization\n",
    "\n",
    "When building effective deep learning models, we frequently encounter the challenge of overfitting, where a model performs exceptionally well on training data but fails to generalize well to unseen data. Regularization techniques are thus critical to ensure our model doesn't suffer from this common pitfall, striking a balance between bias (underfitting) and variance (overfitting). Regularization methods work by adding a penalty on the complexity of the model, effectively preventing the model from learning too much noise from the training data and thus enhancing its ability to generalize.\n",
    "\n",
    "One such powerful regularization technique is 'Dropout', a concept that metaphorically 'drops out' or temporarily 'turns off' a fraction of neurons in the model during training, thereby reducing the interdependencies of neurons. The randomness introduced by dropping out neurons compels the network to learn more robust features, leading to a more generalized and less overfit model.\n",
    "\n",
    "Dropout operates differently from most other regularization techniques. Instead of adding a penalty to the loss function, it randomly disables a fraction of neurons (defined by a probability parameter, typically ranging from 0.2 to 0.5), effectively creating a different architecture of the network for each training instance. This can be seen as training an ensemble of networks, which results in a more robust and generalizable final model.\n",
    "\n",
    "In this section, we will delve into the nuances of dropout, explore its theoretical underpinnings, and illustrate how to apply it using various deep learning frameworks. Understanding and correctly implementing dropout is a valuable tool in the deep learning practitioner's toolkit, aiding the creation of robust and generalized models.\n",
    "\n",
    "As we go through the chapter, you will understand how dropout fits perfectly into the grand schema of regularization methods, learn about its benefits and limitations, and grasp how to effectively use dropout in your deep learning models.\n",
    "\n",
    "Dropout is a regularization technique that was introduced by Geoffrey Hinton, a pioneer in the field of deep learning, and his students Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov in a paper titled \"Dropout: A Simple Way to Prevent Neural Networks from Overfitting,\" which was published in 2014.\n",
    "\n",
    "The development of dropout came from the recognition of the challenges with overfitting in large neural networks. Large neural networks with millions of parameters are prone to overfitting because of their capacity to memorize training data. This is especially true when the amount of training data available is small relative to the complexity of the network.\n",
    "\n",
    "Geoffrey Hinton, who is often referred to as the \"godfather of deep learning,\" has made several seminal contributions to the field of artificial intelligence, with dropout being just one of them. He and his team were looking for simple and effective ways to make neural networks more robust and to improve their generalization capabilities. Inspired by biological systems, they proposed the idea of dropout, which involves randomly \"dropping out\" or deactivating a subset of neurons during the training process to prevent them from co-adapting too much to the data.\n",
    "\n",
    "This simple yet effective technique has since been widely adopted in the deep learning community and has formed the basis of numerous subsequent research and developments. Dropout has proven to be a powerful tool in the training of neural networks, helping to mitigate overfitting and improve model generalization, particularly in scenarios with limited training data.\n",
    "\n",
    "We will begin by modifying the previous example to use dropout. We will preprocess the data in the same way as before.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\",dtype=int)],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\",dtype=int)],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\",dtype=int)],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add dropout to the existing code, I'll use the **nn.Dropout** module from PyTorch. It randomly zeroes some of the elements of the input tensor with probability p (given in the argument) during training, which can help prevent overfitting. The modified code is shown below with the dropout layers added.\n",
    "\n",
    "```\n",
    "# Create the model with dropout\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(x.shape[1], 20),\n",
    "    nn.Dropout(0.5),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(20, 10),\n",
    "    nn.Dropout(0.5),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(10, 1)\n",
    ")\n",
    "```\n",
    "\n",
    "The complete changes can be seen here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/function.py:539: UserWarning: The operator 'aten::native_dropout' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/aten/src/ATen/mps/MPSFallback.mm:13.)\n",
      "  return super().apply(*args, **kwargs)  # type: ignore[misc]\n",
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:09:59,805] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:09:59,805] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:09:59,805] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:09:59,805] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n",
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:09:59,846] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:09:59,846] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:09:59,846] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:09:59,846] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 95/500, Validation Loss: 4.972429275512695, Early stopping triggered after 5 epochs.\n",
      "Fold #2\n",
      "Epoch 51/500, Validation Loss: 19.33069610595703, Early stopping triggered after 5 epochs.\n",
      "Fold #3\n",
      "Epoch 50/500, Validation Loss: 13.100104331970215, Early stopping triggered after 5 epochs.\n",
      "Fold #4\n",
      "Epoch 51/500, Validation Loss: 13.250371932983398, Early stopping triggered after 5 epochs.\n",
      "Fold #5\n",
      "Epoch 42/500, Validation Loss: 15.019149780273438, Early stopping triggered after 5 epochs.\n",
      "Fold score (RMSE): 3.725794792175293\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Convert to PyTorch Tensors\n",
    "x_columns = df.columns.drop([\"age\", \"id\"])\n",
    "x = torch.tensor(df[x_columns].values, dtype=torch.float32, device=device)\n",
    "y = torch.tensor(df[\"age\"].values, dtype=torch.float32, device=device).view(-1, 1)\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# Cross-Validate\n",
    "kf = KFold(n_splits=5, shuffle=True, random_state=42)\n",
    "\n",
    "# Early stopping parameters\n",
    "patience = 10\n",
    "\n",
    "fold = 0\n",
    "for train_idx, test_idx in kf.split(x):\n",
    "    fold += 1\n",
    "    print(f\"Fold #{fold}\")\n",
    "\n",
    "    x_train, x_test = x[train_idx], x[test_idx]\n",
    "    y_train, y_test = y[train_idx], y[test_idx]\n",
    "\n",
    "    # PyTorch DataLoader\n",
    "    train_dataset = TensorDataset(x_train, y_train)\n",
    "    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "    # Create the model with dropout\n",
    "    model = nn.Sequential(\n",
    "        nn.Linear(x.shape[1], 20),\n",
    "        nn.Dropout(0.1),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(20, 10),\n",
    "        nn.Dropout(0.1),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(10, 1),\n",
    "    )\n",
    "    model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "\n",
    "    # Create the optimizer\n",
    "    optimizer = optim.Adam(model.parameters())\n",
    "    loss_fn = nn.MSELoss()\n",
    "\n",
    "    # Early Stopping variables\n",
    "    best_loss = float(\"inf\")\n",
    "    early_stopping_counter = 0\n",
    "\n",
    "    # Training loop\n",
    "    EPOCHS = 500\n",
    "    epoch = 0\n",
    "    done = False\n",
    "    es = EarlyStopping()\n",
    "\n",
    "    while not done and epoch < EPOCHS:\n",
    "        epoch += 1\n",
    "        model.train()\n",
    "        for x_batch, y_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            output = model(x_batch)\n",
    "            loss = loss_fn(output, y_batch)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        # Validation\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            val_output = model(x_test)\n",
    "            val_loss = loss_fn(val_output, y_test)\n",
    "\n",
    "        if es(model, val_loss):\n",
    "            done = True\n",
    "\n",
    "    print(\n",
    "        f\"Epoch {epoch}/{EPOCHS}, Validation Loss: \" f\"{val_loss.item()}, {es.status}\"\n",
    "    )\n",
    "\n",
    "# Final evaluation\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    oos_pred = model(x_test)\n",
    "score = torch.sqrt(loss_fn(oos_pred, y_test)).item()\n",
    "print(f\"Fold score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The changes to the code are straightforward, we simply add Dropout layers in our model's architecture. Here, we've added two Dropout layers, one between the first linear layer and the second linear layer, and another between the second linear layer and the third. Each dropout layer will randomly set 50% of input elements to zero during training. Dropout layers are typically added after non-linear activation functions, such as ReLU in this case.\n",
    "\n",
    "Dropout is a regularization technique that prevents overfitting by reducing the interdependent learning amongst the neurons, encouraging individual neuron to be independently capable. During the training, dropout will randomly disable some neurons which forces the data to find new paths to propagate through the network. Consequently, this results in a network that is capable of better generalization and is less likely to overfit the training data."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}