IPU_gpt2_pretraining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-training Tiny-GPT-2 on BabyLM data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This notebook is based on Tutorials provided by Optimum Graphcore**\n",
    "\n",
    "Pre-training GPT-2 on BabyLM requires:\n",
    "\n",
    "* BabyLM data that can be loaded via:  \n",
    "```\n",
    "git clone https://github.com/upunaprosk/small-language-models.git\n",
    "cd small-language-models\n",
    "bash download_data.sh\n",
    "```\n",
    "\n",
    "* This notebooks assumes you have trained a tokenizer on the corpus you are using, see the [How to train a tokenizer](https://github.com/huggingface/notebooks/blob/master/examples/tokenizer_training.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "First of all, make sure your environment has installed the latest version of [🤗 Optimum Graphcore](https://github.com/huggingface/optimum-graphcore)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In order to improve usability and support for future users, Graphcore would like to collect information about the\n",
    "applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
    "\n",
    "- User progression through the notebook\n",
    "- Notebook details: number of cells, code being run and the output of the cells\n",
    "- Environment details\n",
    "\n",
    "You can disable logging at any time by running `%unload_ext gc_logger` from any cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "Collecting examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable\n",
      "  Cloning https://github.com/graphcore/examples-utils (to revision latest_stable) to /tmp/pip-install-zqkpf2l7/examples-utils_d6ebb1f115d14b40a97c6fb86c9555cc\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/graphcore/examples-utils /tmp/pip-install-zqkpf2l7/examples-utils_d6ebb1f115d14b40a97c6fb86c9555cc\n",
      "  Running command git checkout -q 40c62e6646db8f9d60d1707a61204c95a15c7ccb\n",
      "  Resolved https://github.com/graphcore/examples-utils to commit 40c62e6646db8f9d60d1707a61204c95a15c7ccb\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: awscli>=1.24.10 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.27.143)\n",
      "Requirement already satisfied: boto3>=1.26 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.26.143)\n",
      "Requirement already satisfied: cppimport>=22.07.17 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (22.8.2)\n",
      "Requirement already satisfied: filelock>=3.9.0 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.12.2)\n",
      "Requirement already satisfied: gitpython>=3.1 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.1.32)\n",
      "Requirement already satisfied: ipynbname>=2021.3.2 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2023.1.0.0)\n",
      "Requirement already satisfied: nbformat>=5.7.3 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.0)\n",
      "Requirement already satisfied: psutil>=5.7.0 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.5)\n",
      "Requirement already satisfied: pyyaml>=5.4.1 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.4.1)\n",
      "Requirement already satisfied: simple-parsing==0.0.19.post1 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.0.19.post1)\n",
      "Requirement already satisfied: typing-inspect in /usr/local/lib/python3.8/dist-packages (from simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.9.0)\n",
      "Requirement already satisfied: botocore==1.29.143 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.29.143)\n",
      "Requirement already satisfied: docutils<0.17,>=0.10 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.16)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.6.1)\n",
      "Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.4.4)\n",
      "Requirement already satisfied: rsa<4.8,>=3.1.2 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.7.2)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.0.1)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.8/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.8.2)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.8/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.26.16)\n",
      "Requirement already satisfied: mako in /usr/local/lib/python3.8/dist-packages (from cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.2.4)\n",
      "Requirement already satisfied: pybind11 in /usr/local/lib/python3.8/dist-packages (from cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.11.1)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.8/dist-packages (from gitpython>=3.1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.0.10)\n",
      "Requirement already satisfied: ipykernel in /usr/local/lib/python3.8/dist-packages (from ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.5.6)\n",
      "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.17.1)\n",
      "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.17.3)\n",
      "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.3.0)\n",
      "Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.0)\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.8/dist-packages (from gitdb<5,>=4.0.1->gitpython>=3.1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.0.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (23.1.0)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.12.0)\n",
      "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.3.10)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.19.3)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.8/dist-packages (from rsa<4.8,>=3.1.2->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.5.0)\n",
      "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.0)\n",
      "Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (7.16.3)\n",
      "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (8.2.0)\n",
      "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (6.3.2)\n",
      "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.8/dist-packages (from jupyter-core->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.5.1)\n",
      "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.8/dist-packages (from mako->cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.1.2)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from typing-inspect->simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.0.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.8/dist-packages (from typing-inspect->simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.6.2)\n",
      "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.15.0)\n",
      "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (67.8.0)\n",
      "Requirement already satisfied: jedi<=0.17.2,>=0.10 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.17.2)\n",
      "Requirement already satisfied: decorator in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.1.1)\n",
      "Requirement already satisfied: pickleshare in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.0.38)\n",
      "Requirement already satisfied: pygments in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.15.1)\n",
      "Requirement already satisfied: backcall in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.0)\n",
      "Requirement already satisfied: pexpect in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.8.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.14.0)\n",
      "Requirement already satisfied: importlib-metadata>=4.8.3 in /usr/local/lib/python3.8/dist-packages (from jupyter-client->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (6.6.0)\n",
      "Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (25.1.0)\n",
      "Requirement already satisfied: parso<0.8.0,>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from jedi<=0.17.2,>=0.10->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.1)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.6)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.8/dist-packages (from pexpect->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "Loading extensions from ~/.ipython/extensions is deprecated. We recommend managing extensions like any other Python packages, in site-packages.\n",
      "In order to improve usability and support for future users, Graphcore would like to collect information about the applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore: \n",
      "\t- User progression through the notebook \n",
      "\t- Notebook details: number of cells, code being run and the output of the cells \n",
      "\t- Environment details \n",
      "\n",
      "You can disable logging at any time by running `%unload_ext gc_logger` from any cell. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%pip install -q \"optimum-graphcore==0.6.1\"\n",
    "%pip install examples-utils[common]@git+https://github.com/graphcore/examples-utils@latest_stable\n",
    "from examples_utils import notebook_logging\n",
    "%load_ext gc_logger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow.\n",
    "\n",
    "First you have to store your authentication token from the Hugging Face website (sign up [here](https://huggingface.co/join) if you haven't already!) then execute the following cell and input your username and password:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from huggingface_hub import notebook_login\n",
    "\n",
    "# notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then you need to install Git-LFS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !apt install git-lfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print out the versions of Transformers and Optimum Graphcore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.25.1\n",
      "0.6.1\n"
     ]
    }
   ],
   "source": [
    "import transformers\n",
    "import optimum.graphcore\n",
    "from datasets import ClassLabel\n",
    "import random\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "\n",
    "print(transformers.__version__)\n",
    "print(optimum.graphcore.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Values for machine size and cache directories can be configured through environment variables or directly in the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "pod_type = os.getenv(\"GRAPHCORE_POD_TYPE\", \"pod4\")\n",
    "executable_cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\") + \"/language_modelling_from_scratch\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a3KD3WXU3l-O"
   },
   "source": [
    "# Train a language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAscNNUD3l-P"
   },
   "source": [
    "In this notebook, we'll see how to train a [🤗 Transformers](https://github.com/huggingface/transformers) model on a language modeling task. We will cover two types of language modeling tasks which are:\n",
    "\n",
    "- Causal language modeling: the model has to predict the next token in the sentence (so the labels are the same as the inputs shifted to the right). To make sure the model does not cheat, it gets an attention mask that will prevent it to access the tokens after token i when trying to predict the token i+1 in the sentence.\n",
    "\n",
    "![Widget inference representing the causal language modeling task](images/causal_language_modeling.png)\n",
    "\n",
    "- Masked language modeling: the model has to predict some tokens that are masked in the input. It still has access to the whole sentence, so it can use the tokens before and after the tokens masked to predict their value.\n",
    "\n",
    "![Widget inference representing the masked language modeling task](images/masked_language_modeling.png)\n",
    "\n",
    "We will see how to easily load and preprocess the dataset for each one of those tasks, and how to use the `IPUTrainer` API to train a model on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1r_n9OWV3l-Q"
   },
   "source": [
    "## Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kswRMhPc3l-Q"
   },
   "source": [
    "For each of those tasks, we will use the [Wikitext 2]() dataset as an example. You can load it very easily with the 🤗 Datasets library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset text (/tmp/huggingface_caches/datasets/text/default-d8fd0c9a344ad3d7/0.0.0/cb1e9bd71a82ad27976be3b12b407850fe2837d80c22c5e03a28949843a8ace2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['babylm_data/babylm_dev/open_subtitles.dev', 'babylm_data/babylm_dev/simple_wikipedia.dev', 'babylm_data/babylm_dev/cbt.dev', 'babylm_data/babylm_dev/gutenberg.dev', 'babylm_data/babylm_dev/switchboard.dev', 'babylm_data/babylm_dev/aochildes.dev', 'babylm_data/babylm_dev/qed.dev', 'babylm_data/babylm_dev/wikipedia.dev', 'babylm_data/babylm_dev/children_stories.dev', 'babylm_data/babylm_dev/bnc_spoken.dev']\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cf0e5f2e44946409d0630a75e5ca1b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "from pathlib import Path\n",
    "files=[p.as_posix() for p in Path('./babylm_data/babylm_10M/').glob('*.train')]\n",
    "files_dev=[p.as_posix() for p in Path('./babylm_data/babylm_dev/').glob('*.dev')]\n",
    "print(files_dev)\n",
    "d = load_dataset('text', data_files={'train': list(files), 'eval': list(files_dev)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f1-9jepM3l-W"
   },
   "source": [
    "You can replace the dataset above with any dataset hosted on [the hub](https://huggingface.co/datasets) or use your own files. Just uncomment the following cell and replace the paths with values that will lead to your files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "uxSaGa_l3l-W"
   },
   "outputs": [],
   "source": [
    "# datasets = load_dataset(\"text\", data_files={\"train\": path_to_train.txt, \"validation\": path_to_validation.txt}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jY1SwIrY3l-a"
   },
   "source": [
    "You can also load datasets from a csv or a JSON file, see the [full documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHUmphG3IrI3"
   },
   "source": [
    "To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "ur5sNUcZ3l-g"
   },
   "outputs": [],
   "source": [
    "def show_random_elements(dataset, num_examples=10):\n",
    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
    "    picks = []\n",
    "    for _ in range(num_examples):\n",
    "        pick = random.randint(0, len(dataset)-1)\n",
    "        while pick in picks:\n",
    "            pick = random.randint(0, len(dataset)-1)\n",
    "        picks.append(pick)\n",
    "    \n",
    "    df = pd.DataFrame(dataset[picks])\n",
    "    for column, typ in dataset.features.items():\n",
    "        if isinstance(typ, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
    "    display(HTML(df.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "1Uk8NROQ3l-k",
    "outputId": "a822dcec-51e3-4dba-c73c-dba9e0301726"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Please, please!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>For the three points, let's see if you are correct.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Emacs versus vi?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Er</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good evening, everyone!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Yeah.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Not for drinking, but to trap a gangster by name Tiger Bhai.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>That's what that sound is.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>go over to him and show him .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>That was my grandfather's name.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(d[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CKerdF353l-o"
   },
   "source": [
    "As we can see, some of the texts are a full paragraph of a Wikipedia article while others are just titles or empty lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JEA1ju653l-p"
   },
   "source": [
    "## Causal Language modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v5GTGKZS3l-q"
   },
   "source": [
    "For causal language modeling (CLM) we are going to take all the texts in our dataset and concatenate them after they are tokenized. Then we will split them in examples of a certain sequence length. This way the model will receive chunks of contiguous text that may look like:\n",
    "```\n",
    "part of text 1\n",
    "```\n",
    "or \n",
    "```\n",
    "end of text 1 [BOS_TOKEN] beginning of text 2\n",
    "```\n",
    "depending on whether they span over several of the original texts in the dataset or not. The labels will be the same as the inputs, shifted to the left.\n",
    "\n",
    "We will use the [`gpt2`](https://huggingface.co/gpt2) architecture for this example. You can pick any of the checkpoints listed [here](https://huggingface.co/models?filter=causal-lm) as long as that model is supported by Optimum Graphcore. The IPU config files of the supported models are available in Graphcore's [Hugging Face account](https://huggingface.co/Graphcore). You can also create your own IPU config file locally. For the tokenizer, you can replace the checkpoint by the one you trained yourself.\n",
    "\n",
    "In this notebook, we are using both data parallelism and pipeline parallelism (see this [tutorial](https://github.com/graphcore/tutorials/tree/master/tutorials/pytorch/efficient_data_loading) for more). Therefore the global batch size, which is the actual number of samples used for the weight update, is determined with three factors:\n",
    "- global batch size = micro_batch_size * gradient accumulation steps * replication factor\n",
    "\n",
    "and replication factor is determined by `pod_type`, which will be used as a key to select the replication factor from a dictionary defined in the IPU config file. For example, the dictionary in the IPU config file [Graphcore/gpt2-small-ipu](https://huggingface.co/Graphcore/gpt2-small-ipu/blob/main/ipu_config.json) looks like this:\n",
    "- \"replication_factor\": {\"pod4\": 1, \"pod8\": 2, \"pod16\": 4, \"pod32\": 8, \"pod64\": 16, \"default\": 1}\n",
    "\n",
    "Depending on you model and the pod machine you are using, you might need to adjust these three batch-size-related arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2TokenizerFast\n",
    "path_tokenizer_config=\"custom_tokenizer_gpt_2.json\"\n",
    "tokenizer = GPT2TokenizerFast(vocab_file=None,\n",
    "                                     merges_file=None,\n",
    "                                     tokenizer_file=path_tokenizer_config,\n",
    "                                     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "-WGBCO343l-q"
   },
   "outputs": [],
   "source": [
    "model_checkpoint = \"gpt2\"\n",
    "ipu_config_name = \"Graphcore/gpt2-small-ipu\"\n",
    "micro_batch_size = 1\n",
    "gradient_accumulation_steps = 64\n",
    "dataloader_workers = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5io6fY_d3l-u"
   },
   "source": [
    "To tokenize all our texts with the same vocabulary that was used when training the model, we have to download a pretrained tokenizer. This is all done by the `AutoTokenizer` class:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rpOiBrJ13l-y"
   },
   "source": [
    "We can now call the tokenizer on all our texts. This is very simple, using the [`map`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map) method from the Datasets library. First we define a function that call the tokenizer on our texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /tmp/huggingface_caches/datasets/text/default-d8fd0c9a344ad3d7/0.0.0/cb1e9bd71a82ad27976be3b12b407850fe2837d80c22c5e03a28949843a8ace2/cache-81de5b7289ea3c29_*_of_00004.arrow\n",
      "Loading cached processed dataset at /tmp/huggingface_caches/datasets/text/default-d8fd0c9a344ad3d7/0.0.0/cb1e9bd71a82ad27976be3b12b407850fe2837d80c22c5e03a28949843a8ace2/cache-227037c1b68f519b_*_of_00004.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of train data=1058740\n"
     ]
    }
   ],
   "source": [
    "def tokenize_function(examples):\n",
    "    return tokenizer(examples[\"text\"])\n",
    "\n",
    "tokenized_datasets = d.map(tokenize_function, batched=True, num_proc=4, remove_columns=[\"text\"])\n",
    "print(f'Length of train data={len(tokenized_datasets[\"train\"])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M9xVAa3s3l-2"
   },
   "source": [
    "Then we apply it to all the splits in our `datasets` object, using `batched=True` and 4 processes to speed up the preprocessing. We won't need the `text` column afterward, so we discard it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8qik3J_C3l-7"
   },
   "source": [
    "If we now look at an element of our datasets, we will see the text have been replaced by the `input_ids` the model will need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "nYv_mcKk3l-7",
    "outputId": "8334734c-0f86-4e18-ec17-4216a2d5dd18"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'input_ids': [2391,\n",
       "   11146,\n",
       "   2109,\n",
       "   29820,\n",
       "   6648,\n",
       "   18456,\n",
       "   325,\n",
       "   1287,\n",
       "   16655,\n",
       "   1412,\n",
       "   36,\n",
       "   20090,\n",
       "   7751,\n",
       "   3095,\n",
       "   27251,\n",
       "   34,\n",
       "   4407,\n",
       "   3075,\n",
       "   37,\n",
       "   354,\n",
       "   167],\n",
       "  'attention_mask': [1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1,\n",
       "   1]},\n",
       " {'input_ids': [7201, 12, 391, 471, 444, 524, 31],\n",
       "  'attention_mask': [1, 1, 1, 1, 1, 1, 1]})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_datasets[\"train\"][1],tokenized_datasets[\"eval\"][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "obvgcXda3l--"
   },
   "source": [
    "Now for the harder part: we need to concatenate all our texts together then split the result in small chunks of a certain `block_size`. To do this, we will use the `map` method again, with the option `batched=True`. This option actually lets us change the number of examples in the datasets by returning a different number of examples than we got. This way, we can create our new samples from a batch of examples.\n",
    "\n",
    "First, we grab the maximum length our model was pretrained with. This might be too big to fit on your IPU RAM, so here we take a bit less at just 128."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "DVHs5aCA3l-_"
   },
   "outputs": [],
   "source": [
    "# block_size = tokenizer.model_max_length\n",
    "block_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RpNfGiMw3l_A"
   },
   "source": [
    "Then we write the preprocessing function that will group our texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "iaAJy5Hu3l_B"
   },
   "outputs": [],
   "source": [
    "def group_texts(examples):\n",
    "    # Concatenate all texts.\n",
    "    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n",
    "    total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
    "    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n",
    "        # customize this part to your needs.\n",
    "    total_length = (total_length // block_size) * block_size\n",
    "    # Split by chunks of max_len.\n",
    "    result = {\n",
    "        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n",
    "        for k, t in concatenated_examples.items()\n",
    "    }\n",
    "    result[\"labels\"] = result[\"input_ids\"].copy()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LGJWXtNv3l_C"
   },
   "source": [
    "First note that we duplicate the inputs for our labels. This is because the model of the 🤗 Transformers library apply the shifting to the right, so we don't need to do it manually.\n",
    "\n",
    "Also note that by default, the `map` method will send a batch of 1,000 examples to be treated by the preprocessing function. So here, we will drop the remainder to make the concatenated tokenized texts a multiple of `block_size` every 1,000 examples. You can adjust this behavior by passing a higher batch size (which will also be processed slower). You can also speed-up the preprocessing by using multiprocessing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "gXUSfBrq3l_C",
    "outputId": "34e55885-3d8f-4f05-cbdb-706ce56a25f8",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /tmp/huggingface_caches/datasets/text/default-d8fd0c9a344ad3d7/0.0.0/cb1e9bd71a82ad27976be3b12b407850fe2837d80c22c5e03a28949843a8ace2/cache-0825af2c0192db43_*_of_00004.arrow\n",
      "Loading cached processed dataset at /tmp/huggingface_caches/datasets/text/default-d8fd0c9a344ad3d7/0.0.0/cb1e9bd71a82ad27976be3b12b407850fe2837d80c22c5e03a28949843a8ace2/cache-ad033bdbebe5df8b_*_of_00004.arrow\n"
     ]
    }
   ],
   "source": [
    "lm_datasets = tokenized_datasets.map(\n",
    "    group_texts,\n",
    "    batched=True,\n",
    "    batch_size=64,\n",
    "    num_proc=4,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6n84V8Gc3l_G"
   },
   "source": [
    "And we can check our datasets have changed: now the samples contain chunks of `block_size` contiguous tokens, potentially spanning over several of our original texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PreTrainedTokenizerFast(name_or_path='', vocab_size=30000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>'})"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "hTeGCLl_3l_G",
    "outputId": "ab381a07-f92e-4b14-f7b6-e4af5513d7c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"TH A PRlCE. JUST ASK ClNDERELLA, WHO GAVE UP HER FlRSTBORN TO BECOME A PRlNCESS. DlD YOU GET EVERYTHlNG  YOU DESlRED? BUT NO ONE KNOWS MORE ABOUT THE PRlCE OF MAGlC THAN THE DARK WlZARD HlMSELF-- RUMPLESTlLTSKlN. ALL MAGlC COMES WlTH A PRlCE. (Granny) IT'S ALL HERE. WHY, YES, OF COURSE IT\""
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(lm_datasets[\"train\"][1][\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iEmeQ7Xm3l_H"
   },
   "source": [
    "To instantiate an `IPUTrainer`, we will need to define three more things. The first thing we need to define is the `IPUConfig`, which is a class that specifies attributes and configuration parameters to compile and put the model on the device. We initialize it with one config name or path, which we set earlier. We also get the model configuration from the model name set earlier and initialize our model using that config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2Config {\n",
       "  \"activation_function\": \"gelu_new\",\n",
       "  \"attn_pdrop\": 0.1,\n",
       "  \"bos_token_id\": 50256,\n",
       "  \"embd_pdrop\": 0.1,\n",
       "  \"eos_token_id\": 50256,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"layer_norm_epsilon\": 1e-05,\n",
       "  \"model_type\": \"gpt2\",\n",
       "  \"n_embd\": 768,\n",
       "  \"n_head\": 12,\n",
       "  \"n_inner\": null,\n",
       "  \"n_layer\": 12,\n",
       "  \"n_positions\": 1024,\n",
       "  \"reorder_and_upcast_attn\": false,\n",
       "  \"resid_pdrop\": 0.1,\n",
       "  \"scale_attn_by_inverse_layer_idx\": false,\n",
       "  \"scale_attn_weights\": true,\n",
       "  \"summary_activation\": null,\n",
       "  \"summary_first_dropout\": 0.1,\n",
       "  \"summary_proj_to_labels\": true,\n",
       "  \"summary_type\": \"cls_index\",\n",
       "  \"summary_use_proj\": true,\n",
       "  \"transformers_version\": \"4.25.1\",\n",
       "  \"use_cache\": true,\n",
       "  \"vocab_size\": 50257\n",
       "}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import GPT2Config\n",
    "model_checkpoint\n",
    "GPT2Config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "sPqQA3TT3l_I"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`replicated_tensor_sharding` is not used when `replication_factor=1`\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoConfig, AutoModelForCausalLM\n",
    "from optimum.graphcore import IPUConfig, IPUTrainer, IPUTrainingArguments\n",
    "\n",
    "# ipu_config = IPUConfig.from_pretrained(ipu_config_name, executable_cache_dir=executable_cache_dir)\n",
    "\n",
    "ipu_config = IPUConfig.from_dict({\n",
    "  \"_mode\": \"training\",\n",
    "  \"auto_loss_scaling\": False,\n",
    "  \"device_iterations\": 2,\n",
    "  \"embedding_serialization_factor\": 4,\n",
    "  \"enable_half_partials\": True,\n",
    "  \"executable_cache_dir\": \"/tmp/exe_cache/3.2.1/language_modelling_from_scratch\",\n",
    "  \"execute_encoder_on_cpu_for_generation\": False,\n",
    "  \"gradient_accumulation_steps\": 16,\n",
    "  \"inference_device_iterations\": 5,\n",
    "  \"inference_embedding_serialization_factor\": 1,\n",
    "  \"inference_ipus_per_replica\": 4,\n",
    "  \"inference_layers_per_ipu\": [\n",
    "    2,\n",
    "    2,\n",
    "    2,\n",
    "    2\n",
    "  ],\n",
    "  \"inference_matmul_proportion\": 0.25,\n",
    "  \"inference_projection_serialization_factor\": 1,\n",
    "  \"inference_replication_factor\": 1,\n",
    "  \"inference_serialized_embedding_splits_per_ipu\": None,\n",
    "  \"inference_serialized_projection_splits_per_ipu\": None,\n",
    "  \"ipus_per_replica\": 4,\n",
    "  \"layers_per_ipu\": [\n",
    "    0,\n",
    "    2,\n",
    "    2,\n",
    "    2\n",
    "  ],\n",
    "  \"matmul_proportion\": 0.25,\n",
    "  \"optimizer_state_offchip\": True,\n",
    "  \"optimum_version\": \"1.6.1\",\n",
    "  \"output_mode\": \"final\",\n",
    "  \"projection_serialization_factor\": 1,\n",
    "  \"recompute_checkpoint_every_layer\": True,\n",
    "  \"replicated_tensor_sharding\": True,\n",
    "  \"replication_factor\": 1,\n",
    "  \"seed\": None,\n",
    "  \"serialized_embedding_splits_per_ipu\": None,\n",
    "  \"serialized_projection_splits_per_ipu\": None,\n",
    "  \"transformers_version\": \"4.25.1\"\n",
    "})\n",
    "\n",
    "# ipu_config = IPUConfig.from_pretrained(\n",
    "#     ipu_config_name,\n",
    "#     executable_cache_dir=executable_cache_dir,\n",
    "#     ipus_per_replica=2,\n",
    "#     layers_per_ipu= [\n",
    "#     0,\n",
    "#     2,\n",
    "#     2,\n",
    "#     2\n",
    "#   ],\n",
    "#     inference_layers_per_ipu=[-1]\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2Config {\n",
       "  \"activation_function\": \"gelu\",\n",
       "  \"attn_pdrop\": 0.2,\n",
       "  \"bos_token_id\": 50256,\n",
       "  \"embd_pdrop\": 0.2,\n",
       "  \"eos_token_id\": 50256,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"layer_norm_epsilon\": 1e-05,\n",
       "  \"model_type\": \"gpt2\",\n",
       "  \"n_embd\": 768,\n",
       "  \"n_head\": 12,\n",
       "  \"n_inner\": null,\n",
       "  \"n_layer\": 6,\n",
       "  \"n_positions\": 1024,\n",
       "  \"reorder_and_upcast_attn\": false,\n",
       "  \"resid_pdrop\": 0.1,\n",
       "  \"scale_attn_by_inverse_layer_idx\": false,\n",
       "  \"scale_attn_weights\": true,\n",
       "  \"summary_activation\": null,\n",
       "  \"summary_first_dropout\": 0.1,\n",
       "  \"summary_proj_to_labels\": true,\n",
       "  \"summary_type\": \"cls_index\",\n",
       "  \"summary_use_proj\": true,\n",
       "  \"transformers_version\": \"4.25.1\",\n",
       "  \"use_cache\": true,\n",
       "  \"vocab_size\": 30000\n",
       "}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OPT_CONFIG={\n",
    "  \"activation_function\": \"gelu_new\",\n",
    "  \"attn_pdrop\": 0.2,\n",
    "  \"embd_pdrop\": 0.2,\n",
    "  \"initializer_range\": 0.02,\n",
    "  \"layer_norm_epsilon\": 1e-05,\n",
    "  \"model_type\": \"gpt2\",\n",
    "  \"n_embd\": 768,\n",
    "  \"n_head\": 12,\n",
    "  \"n_inner\": None,\n",
    "  \"n_layer\": 6,\n",
    "  \"n_positions\": 1024,\n",
    "  \"reorder_and_upcast_attn\": False,\n",
    "  \"resid_pdrop\": 0.1,\n",
    "  \"scale_attn_by_inverse_layer_idx\": False,\n",
    "  \"scale_attn_weights\": True,\n",
    "  \"summary_activation\": None,\n",
    "  \"summary_first_dropout\": 0.1,\n",
    "  \"summary_proj_to_labels\": True,\n",
    "  \"summary_type\": \"cls_index\",\n",
    "  \"summary_use_proj\": True,\n",
    "  \"transformers_version\": \"4.25.1\",\n",
    "  \"use_cache\": True,\n",
    "  \"vocab_size\": tokenizer.vocab_size\n",
    "}\n",
    "config = GPT2Config(**OPT_CONFIG)\n",
    "config.update({'activation_function':'gelu'})\n",
    "config\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = AutoModelForCausalLM.from_config(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other thing we need to define is the `IPUTrainingArguments`, which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "YbSwEhQ63l_L"
   },
   "outputs": [],
   "source": [
    "training_args = IPUTrainingArguments(\n",
    "    f\"{model_checkpoint}-baby2\",\n",
    "    overwrite_output_dir=True,\n",
    "    auto_loss_scaling=True,\n",
    "    learning_rate=2e-5,\n",
    "    weight_decay=0.01,\n",
    "    per_device_train_batch_size=micro_batch_size,\n",
    "    per_device_eval_batch_size=micro_batch_size,\n",
    "    gradient_accumulation_steps=gradient_accumulation_steps,\n",
    "    num_train_epochs=15,\n",
    "    loss_scaling=16384,\n",
    "    n_ipu=4,\n",
    "    resume_from_checkpoint=\"./gpt2-baby/checkpoint-4500/\",\n",
    "    warmup_ratio=0.1,\n",
    "    dataloader_drop_last=True,\n",
    "    dataloader_num_workers=64,\n",
    "    logging_steps=10,\n",
    "    push_to_hub=False,\n",
    "    save_total_limit=2\n",
    "    # hub_model_id=f\"username-or-organization/{model_checkpoint}-wikitext2\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last argument to setup everything so we can push the model to the [Hub](https://huggingface.co/models) regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/gpt-finetuned-wikitext2\"` or `\"huggingface/gpt-finetuned-wikitext2\"`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sZRbT9ui3l_N"
   },
   "source": [
    "Finally, we pass along all of those to the `IPUTrainer` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = AutoModelForCausalLM.from_pretrained(\"baby-gpt-2\") # Alternatively, model can be continued to be pre-trained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "OEuqwIra3l_N",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Overriding IPU config: gradient_accumulation_steps=64,auto_loss_scaling=True\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from transformers import default_data_collator\n",
    "\n",
    "trainer = IPUTrainer(\n",
    "    model=model,\n",
    "    ipu_config=ipu_config,\n",
    "    args=training_args,\n",
    "    train_dataset=lm_datasets[\"train\"],\n",
    "    eval_dataset=lm_datasets[\"eval\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=default_data_collator,\n",
    "    # resume_from_checkpoint = \"./gpt2-baby/checkpoint-4500/\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Vvz34Td3l_O"
   },
   "source": [
    "And we can train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.25.1\n"
     ]
    }
   ],
   "source": [
    "print(transformers.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer,GPT2Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PreTrainedTokenizer(name_or_path='baby-gpt-2', vocab_size=30000, model_max_len=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>'})"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = AutoModelForCausalLM.from_pretrained(\"baby-gpt-2\")\n",
    "tokenizer=GPT2Tokenizer.from_pretrained(\"baby-gpt-2\")\n",
    "tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "58e0389c30084d33af8086144e0520fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/272M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/iproskurina/gpt-2/commit/ce33a702d4056f8fe8c176443f092b58172c8285', commit_message='Upload tokenizer', commit_description='', oid='ce33a702d4056f8fe8c176443f092b58172c8285', pr_url=None, pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.push_to_hub(\"gpt-2\")\n",
    "tokenizer.push_to_hub(\"gpt-2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenizer.save_pretrained(\"baby-gpt-2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 64 worker processes in total. Our suggested max number of worker in current system is 52, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "  warnings.warn(_create_warning_msg(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Compiling Model...\n",
      "Graph compilation: 100%|██████████| 100/100 [03:24<00:00]\n",
      "Compiled/Loaded model in 229.32772605400532 secs\n",
      "***** Running training *****\n",
      "  Num examples = 96527\n",
      "  Num epochs = 15\n",
      "  Instantaneous batch size per device = 1\n",
      "  Total training batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient accumulation steps = 64\n",
      "  Total optimization steps = 11310\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3c551049a6f4956801caa4b09f6db2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11310 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 9.7867, 'learning_rate': 1.7683465959328031e-07, 'epoch': 0.01}\n",
      "{'loss': 9.8477, 'learning_rate': 3.5366931918656063e-07, 'epoch': 0.03}\n",
      "{'loss': 9.7594, 'learning_rate': 5.305039787798409e-07, 'epoch': 0.04}\n",
      "{'loss': 9.6477, 'learning_rate': 7.073386383731213e-07, 'epoch': 0.05}\n",
      "{'loss': 9.4656, 'learning_rate': 8.841732979664015e-07, 'epoch': 0.07}\n",
      "{'loss': 9.343, 'learning_rate': 1.0610079575596817e-06, 'epoch': 0.08}\n",
      "{'loss': 9.3031, 'learning_rate': 1.2378426171529622e-06, 'epoch': 0.09}\n",
      "{'loss': 9.1258, 'learning_rate': 1.4146772767462425e-06, 'epoch': 0.11}\n",
      "{'loss': 9.1359, 'learning_rate': 1.5915119363395226e-06, 'epoch': 0.12}\n",
      "{'loss': 9.1211, 'learning_rate': 1.768346595932803e-06, 'epoch': 0.13}\n",
      "{'loss': 9.0523, 'learning_rate': 1.945181255526083e-06, 'epoch': 0.15}\n",
      "{'loss': 9.0109, 'learning_rate': 2.1220159151193635e-06, 'epoch': 0.16}\n",
      "{'loss': 8.9391, 'learning_rate': 2.2988505747126437e-06, 'epoch': 0.17}\n",
      "{'loss': 8.9156, 'learning_rate': 2.4756852343059245e-06, 'epoch': 0.19}\n",
      "{'loss': 9.0273, 'learning_rate': 2.6525198938992047e-06, 'epoch': 0.2}\n",
      "{'loss': 8.8242, 'learning_rate': 2.829354553492485e-06, 'epoch': 0.21}\n",
      "{'loss': 8.6734, 'learning_rate': 3.006189213085765e-06, 'epoch': 0.23}\n",
      "{'loss': 8.7469, 'learning_rate': 3.183023872679045e-06, 'epoch': 0.24}\n",
      "{'loss': 8.6539, 'learning_rate': 3.3598585322723255e-06, 'epoch': 0.25}\n",
      "{'loss': 8.6977, 'learning_rate': 3.536693191865606e-06, 'epoch': 0.27}\n",
      "{'loss': 8.9539, 'learning_rate': 3.7135278514588865e-06, 'epoch': 0.28}\n",
      "{'loss': 8.8461, 'learning_rate': 3.890362511052166e-06, 'epoch': 0.29}\n",
      "{'loss': 8.4246, 'learning_rate': 4.067197170645446e-06, 'epoch': 0.31}\n",
      "{'loss': 8.4848, 'learning_rate': 4.244031830238727e-06, 'epoch': 0.32}\n",
      "{'loss': 8.284, 'learning_rate': 4.420866489832008e-06, 'epoch': 0.33}\n",
      "{'loss': 8.3328, 'learning_rate': 4.5977011494252875e-06, 'epoch': 0.34}\n",
      "{'loss': 8.284, 'learning_rate': 4.774535809018568e-06, 'epoch': 0.36}\n",
      "{'loss': 8.1469, 'learning_rate': 4.951370468611849e-06, 'epoch': 0.37}\n",
      "{'loss': 8.0004, 'learning_rate': 5.128205128205128e-06, 'epoch': 0.38}\n",
      "{'loss': 7.9664, 'learning_rate': 5.3050397877984095e-06, 'epoch': 0.4}\n",
      "{'loss': 7.9781, 'learning_rate': 5.481874447391689e-06, 'epoch': 0.41}\n",
      "{'loss': 7.7645, 'learning_rate': 5.65870910698497e-06, 'epoch': 0.42}\n",
      "{'loss': 7.7129, 'learning_rate': 5.83554376657825e-06, 'epoch': 0.44}\n",
      "{'loss': 8.0629, 'learning_rate': 6.01237842617153e-06, 'epoch': 0.45}\n",
      "{'loss': 7.7305, 'learning_rate': 6.1892130857648105e-06, 'epoch': 0.46}\n",
      "{'loss': 7.8641, 'learning_rate': 6.36604774535809e-06, 'epoch': 0.48}\n",
      "{'loss': 7.8156, 'learning_rate': 6.54288240495137e-06, 'epoch': 0.49}\n",
      "{'loss': 7.5512, 'learning_rate': 6.719717064544651e-06, 'epoch': 0.5}\n",
      "{'loss': 7.4094, 'learning_rate': 6.896551724137932e-06, 'epoch': 0.52}\n",
      "{'loss': 7.25, 'learning_rate': 7.073386383731212e-06, 'epoch': 0.53}\n",
      "{'loss': 7.2352, 'learning_rate': 7.250221043324492e-06, 'epoch': 0.54}\n",
      "{'loss': 7.4352, 'learning_rate': 7.427055702917773e-06, 'epoch': 0.56}\n",
      "{'loss': 7.3074, 'learning_rate': 7.603890362511053e-06, 'epoch': 0.57}\n",
      "{'loss': 7.0617, 'learning_rate': 7.780725022104333e-06, 'epoch': 0.58}\n",
      "{'loss': 7.4629, 'learning_rate': 7.957559681697613e-06, 'epoch': 0.6}\n",
      "{'loss': 7.1188, 'learning_rate': 8.134394341290892e-06, 'epoch': 0.61}\n",
      "{'loss': 7.2164, 'learning_rate': 8.311229000884174e-06, 'epoch': 0.62}\n",
      "{'loss': 7.1059, 'learning_rate': 8.488063660477454e-06, 'epoch': 0.64}\n",
      "{'loss': 7.1125, 'learning_rate': 8.664898320070735e-06, 'epoch': 0.65}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 7.0914, 'learning_rate': 8.841732979664015e-06, 'epoch': 0.66}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-3000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 7.1285, 'learning_rate': 9.018567639257295e-06, 'epoch': 0.68}\n",
      "{'loss': 6.8723, 'learning_rate': 9.195402298850575e-06, 'epoch': 0.69}\n",
      "{'loss': 6.9945, 'learning_rate': 9.372236958443855e-06, 'epoch': 0.7}\n",
      "{'loss': 6.873, 'learning_rate': 9.549071618037136e-06, 'epoch': 0.72}\n",
      "{'loss': 6.859, 'learning_rate': 9.725906277630416e-06, 'epoch': 0.73}\n",
      "{'loss': 7.1285, 'learning_rate': 9.902740937223698e-06, 'epoch': 0.74}\n",
      "{'loss': 6.7961, 'learning_rate': 1.0079575596816978e-05, 'epoch': 0.76}\n",
      "{'loss': 6.5598, 'learning_rate': 1.0256410256410256e-05, 'epoch': 0.77}\n",
      "{'loss': 6.4555, 'learning_rate': 1.0433244916003537e-05, 'epoch': 0.78}\n",
      "{'loss': 7.0137, 'learning_rate': 1.0610079575596819e-05, 'epoch': 0.8}\n",
      "{'loss': 6.6117, 'learning_rate': 1.0786914235190097e-05, 'epoch': 0.81}\n",
      "{'loss': 7.0465, 'learning_rate': 1.0963748894783379e-05, 'epoch': 0.82}\n",
      "{'loss': 6.5336, 'learning_rate': 1.1140583554376659e-05, 'epoch': 0.84}\n",
      "{'loss': 6.7133, 'learning_rate': 1.131741821396994e-05, 'epoch': 0.85}\n",
      "{'loss': 6.7473, 'learning_rate': 1.1494252873563218e-05, 'epoch': 0.86}\n",
      "{'loss': 6.6039, 'learning_rate': 1.16710875331565e-05, 'epoch': 0.88}\n",
      "{'loss': 6.4176, 'learning_rate': 1.184792219274978e-05, 'epoch': 0.89}\n",
      "{'loss': 6.632, 'learning_rate': 1.202475685234306e-05, 'epoch': 0.9}\n",
      "{'loss': 6.4844, 'learning_rate': 1.2201591511936341e-05, 'epoch': 0.92}\n",
      "{'loss': 6.6516, 'learning_rate': 1.2378426171529621e-05, 'epoch': 0.93}\n",
      "{'loss': 6.4207, 'learning_rate': 1.2555260831122903e-05, 'epoch': 0.94}\n",
      "{'loss': 6.8707, 'learning_rate': 1.273209549071618e-05, 'epoch': 0.95}\n",
      "{'loss': 6.0891, 'learning_rate': 1.2908930150309462e-05, 'epoch': 0.97}\n",
      "{'loss': 6.5746, 'learning_rate': 1.308576480990274e-05, 'epoch': 0.98}\n",
      "{'loss': 6.6598, 'learning_rate': 1.3262599469496022e-05, 'epoch': 0.99}\n",
      "{'loss': 5.9969, 'learning_rate': 1.3439434129089302e-05, 'epoch': 1.01}\n",
      "{'loss': 6.9109, 'learning_rate': 1.3616268788682583e-05, 'epoch': 1.02}\n",
      "{'loss': 6.4707, 'learning_rate': 1.3793103448275863e-05, 'epoch': 1.03}\n",
      "{'loss': 5.9137, 'learning_rate': 1.3969938107869143e-05, 'epoch': 1.05}\n",
      "{'loss': 6.1602, 'learning_rate': 1.4146772767462425e-05, 'epoch': 1.06}\n",
      "{'loss': 6.566, 'learning_rate': 1.4323607427055703e-05, 'epoch': 1.07}\n",
      "{'loss': 6.4184, 'learning_rate': 1.4500442086648984e-05, 'epoch': 1.09}\n",
      "{'loss': 6.4918, 'learning_rate': 1.4677276746242264e-05, 'epoch': 1.1}\n",
      "{'loss': 6.034, 'learning_rate': 1.4854111405835546e-05, 'epoch': 1.11}\n",
      "{'loss': 6.1937, 'learning_rate': 1.5030946065428824e-05, 'epoch': 1.13}\n",
      "{'loss': 6.3563, 'learning_rate': 1.5207780725022106e-05, 'epoch': 1.14}\n",
      "{'loss': 6.3555, 'learning_rate': 1.5384615384615387e-05, 'epoch': 1.15}\n",
      "{'loss': 6.1172, 'learning_rate': 1.5561450044208665e-05, 'epoch': 1.17}\n",
      "{'loss': 6.8289, 'learning_rate': 1.5738284703801947e-05, 'epoch': 1.18}\n",
      "{'loss': 6.3129, 'learning_rate': 1.5915119363395225e-05, 'epoch': 1.19}\n",
      "{'loss': 5.9848, 'learning_rate': 1.6091954022988507e-05, 'epoch': 1.21}\n",
      "{'loss': 6.3602, 'learning_rate': 1.6268788682581785e-05, 'epoch': 1.22}\n",
      "{'loss': 6.0812, 'learning_rate': 1.6445623342175066e-05, 'epoch': 1.23}\n",
      "{'loss': 6.5379, 'learning_rate': 1.6622458001768348e-05, 'epoch': 1.25}\n",
      "{'loss': 6.4391, 'learning_rate': 1.679929266136163e-05, 'epoch': 1.26}\n",
      "{'loss': 6.1359, 'learning_rate': 1.6976127320954908e-05, 'epoch': 1.27}\n",
      "{'loss': 6.5789, 'learning_rate': 1.715296198054819e-05, 'epoch': 1.29}\n",
      "{'loss': 6.1699, 'learning_rate': 1.732979664014147e-05, 'epoch': 1.3}\n",
      "{'loss': 5.6445, 'learning_rate': 1.750663129973475e-05, 'epoch': 1.31}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.8621, 'learning_rate': 1.768346595932803e-05, 'epoch': 1.33}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-1000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-3500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.3398, 'learning_rate': 1.786030061892131e-05, 'epoch': 1.34}\n",
      "{'loss': 6.0242, 'learning_rate': 1.803713527851459e-05, 'epoch': 1.35}\n",
      "{'loss': 6.2926, 'learning_rate': 1.8213969938107872e-05, 'epoch': 1.37}\n",
      "{'loss': 6.0695, 'learning_rate': 1.839080459770115e-05, 'epoch': 1.38}\n",
      "{'loss': 5.5141, 'learning_rate': 1.856763925729443e-05, 'epoch': 1.39}\n",
      "{'loss': 5.9891, 'learning_rate': 1.874447391688771e-05, 'epoch': 1.41}\n",
      "{'loss': 5.9512, 'learning_rate': 1.892130857648099e-05, 'epoch': 1.42}\n",
      "{'loss': 6.1879, 'learning_rate': 1.9098143236074273e-05, 'epoch': 1.43}\n",
      "{'loss': 6.3832, 'learning_rate': 1.927497789566755e-05, 'epoch': 1.45}\n",
      "{'loss': 5.7137, 'learning_rate': 1.9451812555260833e-05, 'epoch': 1.46}\n",
      "{'loss': 5.7715, 'learning_rate': 1.9628647214854114e-05, 'epoch': 1.47}\n",
      "{'loss': 6.2711, 'learning_rate': 1.9805481874447396e-05, 'epoch': 1.49}\n",
      "{'loss': 5.8688, 'learning_rate': 1.9982316534040674e-05, 'epoch': 1.5}\n",
      "{'loss': 6.0582, 'learning_rate': 1.9982316534040674e-05, 'epoch': 1.51}\n",
      "{'loss': 5.8469, 'learning_rate': 1.996266823853031e-05, 'epoch': 1.53}\n",
      "{'loss': 6.1059, 'learning_rate': 1.9943019943019945e-05, 'epoch': 1.54}\n",
      "{'loss': 6.2125, 'learning_rate': 1.992337164750958e-05, 'epoch': 1.55}\n",
      "{'loss': 6.0508, 'learning_rate': 1.9903723351999215e-05, 'epoch': 1.56}\n",
      "{'loss': 6.0633, 'learning_rate': 1.988407505648885e-05, 'epoch': 1.58}\n",
      "{'loss': 5.7211, 'learning_rate': 1.9864426760978486e-05, 'epoch': 1.59}\n",
      "{'loss': 6.5621, 'learning_rate': 1.984477846546812e-05, 'epoch': 1.6}\n",
      "{'loss': 5.7301, 'learning_rate': 1.9825130169957757e-05, 'epoch': 1.62}\n",
      "{'loss': 6.1754, 'learning_rate': 1.9805481874447396e-05, 'epoch': 1.63}\n",
      "{'loss': 5.6582, 'learning_rate': 1.9785833578937028e-05, 'epoch': 1.64}\n",
      "{'loss': 6.2027, 'learning_rate': 1.9766185283426663e-05, 'epoch': 1.66}\n",
      "{'loss': 6.1012, 'learning_rate': 1.9746536987916302e-05, 'epoch': 1.67}\n",
      "{'loss': 5.6879, 'learning_rate': 1.9726888692405934e-05, 'epoch': 1.68}\n",
      "{'loss': 5.5508, 'learning_rate': 1.970724039689557e-05, 'epoch': 1.7}\n",
      "{'loss': 5.7922, 'learning_rate': 1.9687592101385208e-05, 'epoch': 1.71}\n",
      "{'loss': 5.9813, 'learning_rate': 1.9667943805874843e-05, 'epoch': 1.72}\n",
      "{'loss': 5.7988, 'learning_rate': 1.9648295510364475e-05, 'epoch': 1.74}\n",
      "{'loss': 6.5488, 'learning_rate': 1.9628647214854114e-05, 'epoch': 1.75}\n",
      "{'loss': 5.9027, 'learning_rate': 1.960899891934375e-05, 'epoch': 1.76}\n",
      "{'loss': 5.784, 'learning_rate': 1.958935062383338e-05, 'epoch': 1.78}\n",
      "{'loss': 5.6496, 'learning_rate': 1.956970232832302e-05, 'epoch': 1.79}\n",
      "{'loss': 5.6516, 'learning_rate': 1.9550054032812656e-05, 'epoch': 1.8}\n",
      "{'loss': 5.8523, 'learning_rate': 1.953040573730229e-05, 'epoch': 1.82}\n",
      "{'loss': 6.1145, 'learning_rate': 1.9510757441791926e-05, 'epoch': 1.83}\n",
      "{'loss': 5.7102, 'learning_rate': 1.9491109146281562e-05, 'epoch': 1.84}\n",
      "{'loss': 5.6758, 'learning_rate': 1.9471460850771197e-05, 'epoch': 1.86}\n",
      "{'loss': 5.3547, 'learning_rate': 1.9451812555260833e-05, 'epoch': 1.87}\n",
      "{'loss': 5.3738, 'learning_rate': 1.9432164259750468e-05, 'epoch': 1.88}\n",
      "{'loss': 5.8637, 'learning_rate': 1.9412515964240103e-05, 'epoch': 1.9}\n",
      "{'loss': 5.4504, 'learning_rate': 1.939286766872974e-05, 'epoch': 1.91}\n",
      "{'loss': 5.7379, 'learning_rate': 1.9373219373219374e-05, 'epoch': 1.92}\n",
      "{'loss': 5.5238, 'learning_rate': 1.935357107770901e-05, 'epoch': 1.94}\n",
      "{'loss': 5.9816, 'learning_rate': 1.9333922782198645e-05, 'epoch': 1.95}\n",
      "{'loss': 5.6422, 'learning_rate': 1.931427448668828e-05, 'epoch': 1.96}\n",
      "{'loss': 6.0086, 'learning_rate': 1.9294626191177916e-05, 'epoch': 1.98}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-1500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.6949, 'learning_rate': 1.927497789566755e-05, 'epoch': 1.99}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-1500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.05, 'learning_rate': 1.925532960015719e-05, 'epoch': 2.0}\n",
      "{'loss': 5.9723, 'learning_rate': 1.9235681304646822e-05, 'epoch': 2.02}\n",
      "{'loss': 5.798, 'learning_rate': 1.921603300913646e-05, 'epoch': 2.03}\n",
      "{'loss': 6.1957, 'learning_rate': 1.9196384713626096e-05, 'epoch': 2.04}\n",
      "{'loss': 5.6531, 'learning_rate': 1.9176736418115728e-05, 'epoch': 2.06}\n",
      "{'loss': 5.8344, 'learning_rate': 1.9157088122605367e-05, 'epoch': 2.07}\n",
      "{'loss': 5.9811, 'learning_rate': 1.9137439827095002e-05, 'epoch': 2.08}\n",
      "{'loss': 5.8945, 'learning_rate': 1.9117791531584637e-05, 'epoch': 2.1}\n",
      "{'loss': 6.1191, 'learning_rate': 1.9098143236074273e-05, 'epoch': 2.11}\n",
      "{'loss': 5.5098, 'learning_rate': 1.9078494940563908e-05, 'epoch': 2.12}\n",
      "{'loss': 5.5949, 'learning_rate': 1.9058846645053544e-05, 'epoch': 2.14}\n",
      "{'loss': 5.5258, 'learning_rate': 1.903919834954318e-05, 'epoch': 2.15}\n",
      "{'loss': 5.4055, 'learning_rate': 1.9019550054032814e-05, 'epoch': 2.16}\n",
      "{'loss': 5.7687, 'learning_rate': 1.899990175852245e-05, 'epoch': 2.18}\n",
      "{'loss': 5.4062, 'learning_rate': 1.8980253463012085e-05, 'epoch': 2.19}\n",
      "{'loss': 5.2957, 'learning_rate': 1.896060516750172e-05, 'epoch': 2.2}\n",
      "{'loss': 5.8867, 'learning_rate': 1.8940956871991356e-05, 'epoch': 2.21}\n",
      "{'loss': 5.8289, 'learning_rate': 1.892130857648099e-05, 'epoch': 2.23}\n",
      "{'loss': 5.7316, 'learning_rate': 1.890166028097063e-05, 'epoch': 2.24}\n",
      "{'loss': 5.8187, 'learning_rate': 1.8882011985460262e-05, 'epoch': 2.25}\n",
      "{'loss': 5.6117, 'learning_rate': 1.8862363689949897e-05, 'epoch': 2.27}\n",
      "{'loss': 5.5164, 'learning_rate': 1.8842715394439536e-05, 'epoch': 2.28}\n",
      "{'loss': 5.4539, 'learning_rate': 1.8823067098929168e-05, 'epoch': 2.29}\n",
      "{'loss': 5.6695, 'learning_rate': 1.8803418803418804e-05, 'epoch': 2.31}\n",
      "{'loss': 5.8723, 'learning_rate': 1.8783770507908442e-05, 'epoch': 2.32}\n",
      "{'loss': 5.3676, 'learning_rate': 1.8764122212398078e-05, 'epoch': 2.33}\n",
      "{'loss': 5.7695, 'learning_rate': 1.874447391688771e-05, 'epoch': 2.35}\n",
      "{'loss': 6.2945, 'learning_rate': 1.872482562137735e-05, 'epoch': 2.36}\n",
      "{'loss': 5.3812, 'learning_rate': 1.8705177325866984e-05, 'epoch': 2.37}\n",
      "{'loss': 5.3539, 'learning_rate': 1.8685529030356616e-05, 'epoch': 2.39}\n",
      "{'loss': 6.1883, 'learning_rate': 1.8665880734846255e-05, 'epoch': 2.4}\n",
      "{'loss': 5.307, 'learning_rate': 1.864623243933589e-05, 'epoch': 2.41}\n",
      "{'loss': 5.7539, 'learning_rate': 1.8626584143825525e-05, 'epoch': 2.43}\n",
      "{'loss': 5.4414, 'learning_rate': 1.860693584831516e-05, 'epoch': 2.44}\n",
      "{'loss': 5.5961, 'learning_rate': 1.8587287552804796e-05, 'epoch': 2.45}\n",
      "{'loss': 5.4707, 'learning_rate': 1.856763925729443e-05, 'epoch': 2.47}\n",
      "{'loss': 5.5734, 'learning_rate': 1.8547990961784067e-05, 'epoch': 2.48}\n",
      "{'loss': 6.1012, 'learning_rate': 1.8528342666273702e-05, 'epoch': 2.49}\n",
      "{'loss': 5.4934, 'learning_rate': 1.8508694370763338e-05, 'epoch': 2.51}\n",
      "{'loss': 5.7797, 'learning_rate': 1.8489046075252973e-05, 'epoch': 2.52}\n",
      "{'loss': 5.4805, 'learning_rate': 1.846939777974261e-05, 'epoch': 2.53}\n",
      "{'loss': 5.2391, 'learning_rate': 1.8449749484232244e-05, 'epoch': 2.55}\n",
      "{'loss': 5.5742, 'learning_rate': 1.843010118872188e-05, 'epoch': 2.56}\n",
      "{'loss': 5.4574, 'learning_rate': 1.8410452893211515e-05, 'epoch': 2.57}\n",
      "{'loss': 5.7641, 'learning_rate': 1.839080459770115e-05, 'epoch': 2.59}\n",
      "{'loss': 6.1051, 'learning_rate': 1.8371156302190785e-05, 'epoch': 2.6}\n",
      "{'loss': 5.9301, 'learning_rate': 1.8351508006680424e-05, 'epoch': 2.61}\n",
      "{'loss': 5.5586, 'learning_rate': 1.8331859711170056e-05, 'epoch': 2.63}\n",
      "{'loss': 5.6801, 'learning_rate': 1.831221141565969e-05, 'epoch': 2.64}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-2000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.5859, 'learning_rate': 1.829256312014933e-05, 'epoch': 2.65}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-2000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-1000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.8504, 'learning_rate': 1.8272914824638962e-05, 'epoch': 2.67}\n",
      "{'loss': 5.3621, 'learning_rate': 1.8253266529128598e-05, 'epoch': 2.68}\n",
      "{'loss': 5.5867, 'learning_rate': 1.8233618233618236e-05, 'epoch': 2.69}\n",
      "{'loss': 5.6281, 'learning_rate': 1.8213969938107872e-05, 'epoch': 2.71}\n",
      "{'loss': 5.5922, 'learning_rate': 1.8194321642597507e-05, 'epoch': 2.72}\n",
      "{'loss': 5.7059, 'learning_rate': 1.8174673347087143e-05, 'epoch': 2.73}\n",
      "{'loss': 5.2332, 'learning_rate': 1.8155025051576778e-05, 'epoch': 2.75}\n",
      "{'loss': 5.0934, 'learning_rate': 1.8135376756066413e-05, 'epoch': 2.76}\n",
      "{'loss': 5.5301, 'learning_rate': 1.811572846055605e-05, 'epoch': 2.77}\n",
      "{'loss': 5.3617, 'learning_rate': 1.8096080165045684e-05, 'epoch': 2.79}\n",
      "{'loss': 5.3016, 'learning_rate': 1.807643186953532e-05, 'epoch': 2.8}\n",
      "{'loss': 5.3484, 'learning_rate': 1.8056783574024955e-05, 'epoch': 2.81}\n",
      "{'loss': 5.2383, 'learning_rate': 1.803713527851459e-05, 'epoch': 2.82}\n",
      "{'loss': 5.0801, 'learning_rate': 1.8017486983004226e-05, 'epoch': 2.84}\n",
      "{'loss': 5.4355, 'learning_rate': 1.799783868749386e-05, 'epoch': 2.85}\n",
      "{'loss': 5.5988, 'learning_rate': 1.7978190391983496e-05, 'epoch': 2.86}\n",
      "{'loss': 5.0535, 'learning_rate': 1.7958542096473132e-05, 'epoch': 2.88}\n",
      "{'loss': 5.1127, 'learning_rate': 1.793889380096277e-05, 'epoch': 2.89}\n",
      "{'loss': 5.2131, 'learning_rate': 1.7919245505452403e-05, 'epoch': 2.9}\n",
      "{'loss': 5.716, 'learning_rate': 1.7899597209942038e-05, 'epoch': 2.92}\n",
      "{'loss': 5.5148, 'learning_rate': 1.7879948914431677e-05, 'epoch': 2.93}\n",
      "{'loss': 5.2957, 'learning_rate': 1.786030061892131e-05, 'epoch': 2.94}\n",
      "{'loss': 5.5012, 'learning_rate': 1.7840652323410944e-05, 'epoch': 2.96}\n",
      "{'loss': 5.3836, 'learning_rate': 1.7821004027900583e-05, 'epoch': 2.97}\n",
      "{'loss': 5.6324, 'learning_rate': 1.7801355732390218e-05, 'epoch': 2.98}\n",
      "{'loss': 5.1029, 'learning_rate': 1.778170743687985e-05, 'epoch': 3.0}\n",
      "{'loss': 5.1195, 'learning_rate': 1.776205914136949e-05, 'epoch': 3.01}\n",
      "{'loss': 5.7203, 'learning_rate': 1.7742410845859124e-05, 'epoch': 3.02}\n",
      "{'loss': 5.676, 'learning_rate': 1.7722762550348756e-05, 'epoch': 3.04}\n",
      "{'loss': 5.0756, 'learning_rate': 1.7703114254838395e-05, 'epoch': 3.05}\n",
      "{'loss': 5.3359, 'learning_rate': 1.768346595932803e-05, 'epoch': 3.06}\n",
      "{'loss': 5.6512, 'learning_rate': 1.7663817663817666e-05, 'epoch': 3.08}\n",
      "{'loss': 5.4227, 'learning_rate': 1.76441693683073e-05, 'epoch': 3.09}\n",
      "{'loss': 4.8746, 'learning_rate': 1.7624521072796937e-05, 'epoch': 3.1}\n",
      "{'loss': 5.7059, 'learning_rate': 1.7604872777286572e-05, 'epoch': 3.12}\n",
      "{'loss': 5.2578, 'learning_rate': 1.7585224481776207e-05, 'epoch': 3.13}\n",
      "{'loss': 5.2996, 'learning_rate': 1.7565576186265843e-05, 'epoch': 3.14}\n",
      "{'loss': 5.3268, 'learning_rate': 1.7545927890755478e-05, 'epoch': 3.16}\n",
      "{'loss': 5.4633, 'learning_rate': 1.7526279595245114e-05, 'epoch': 3.17}\n",
      "{'loss': 5.3084, 'learning_rate': 1.750663129973475e-05, 'epoch': 3.18}\n",
      "{'loss': 5.2074, 'learning_rate': 1.7486983004224384e-05, 'epoch': 3.2}\n",
      "{'loss': 5.8125, 'learning_rate': 1.746733470871402e-05, 'epoch': 3.21}\n",
      "{'loss': 5.3625, 'learning_rate': 1.7447686413203655e-05, 'epoch': 3.22}\n",
      "{'loss': 5.5494, 'learning_rate': 1.742803811769329e-05, 'epoch': 3.24}\n",
      "{'loss': 5.2598, 'learning_rate': 1.7408389822182926e-05, 'epoch': 3.25}\n",
      "{'loss': 5.3229, 'learning_rate': 1.7388741526672565e-05, 'epoch': 3.26}\n",
      "{'loss': 5.1395, 'learning_rate': 1.7369093231162197e-05, 'epoch': 3.28}\n",
      "{'loss': 5.1139, 'learning_rate': 1.7349444935651832e-05, 'epoch': 3.29}\n",
      "{'loss': 5.1371, 'learning_rate': 1.732979664014147e-05, 'epoch': 3.3}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-2500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.327, 'learning_rate': 1.7310148344631103e-05, 'epoch': 3.32}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-2500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-1500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.3504, 'learning_rate': 1.7290500049120738e-05, 'epoch': 3.33}\n",
      "{'loss': 5.7926, 'learning_rate': 1.7270851753610377e-05, 'epoch': 3.34}\n",
      "{'loss': 5.5867, 'learning_rate': 1.7251203458100012e-05, 'epoch': 3.36}\n",
      "{'loss': 4.6754, 'learning_rate': 1.7231555162589648e-05, 'epoch': 3.37}\n",
      "{'loss': 5.0941, 'learning_rate': 1.7211906867079283e-05, 'epoch': 3.38}\n",
      "{'loss': 5.4141, 'learning_rate': 1.719225857156892e-05, 'epoch': 3.4}\n",
      "{'loss': 5.04, 'learning_rate': 1.7172610276058554e-05, 'epoch': 3.41}\n",
      "{'loss': 5.784, 'learning_rate': 1.715296198054819e-05, 'epoch': 3.42}\n",
      "{'loss': 4.9451, 'learning_rate': 1.7133313685037825e-05, 'epoch': 3.44}\n",
      "{'loss': 5.9281, 'learning_rate': 1.711366538952746e-05, 'epoch': 3.45}\n",
      "{'loss': 5.1631, 'learning_rate': 1.7094017094017095e-05, 'epoch': 3.46}\n",
      "{'loss': 4.9932, 'learning_rate': 1.707436879850673e-05, 'epoch': 3.47}\n",
      "{'loss': 5.6676, 'learning_rate': 1.7054720502996366e-05, 'epoch': 3.49}\n",
      "{'loss': 5.2141, 'learning_rate': 1.7035072207486e-05, 'epoch': 3.5}\n",
      "{'loss': 5.5031, 'learning_rate': 1.7015423911975637e-05, 'epoch': 3.51}\n",
      "{'loss': 5.1871, 'learning_rate': 1.6995775616465272e-05, 'epoch': 3.53}\n",
      "{'loss': 5.066, 'learning_rate': 1.6976127320954908e-05, 'epoch': 3.54}\n",
      "{'loss': 5.6059, 'learning_rate': 1.6956479025444543e-05, 'epoch': 3.55}\n",
      "{'loss': 5.0109, 'learning_rate': 1.693683072993418e-05, 'epoch': 3.57}\n",
      "{'loss': 5.34, 'learning_rate': 1.6917182434423817e-05, 'epoch': 3.58}\n",
      "{'loss': 5.2225, 'learning_rate': 1.6897534138913453e-05, 'epoch': 3.59}\n",
      "{'loss': 5.7094, 'learning_rate': 1.6877885843403085e-05, 'epoch': 3.61}\n",
      "{'loss': 4.9941, 'learning_rate': 1.6858237547892723e-05, 'epoch': 3.62}\n",
      "{'loss': 5.1281, 'learning_rate': 1.683858925238236e-05, 'epoch': 3.63}\n",
      "{'loss': 4.984, 'learning_rate': 1.681894095687199e-05, 'epoch': 3.65}\n",
      "{'loss': 5.1266, 'learning_rate': 1.679929266136163e-05, 'epoch': 3.66}\n",
      "{'loss': 5.332, 'learning_rate': 1.6779644365851265e-05, 'epoch': 3.67}\n",
      "{'loss': 5.132, 'learning_rate': 1.67599960703409e-05, 'epoch': 3.69}\n",
      "{'loss': 5.4527, 'learning_rate': 1.6740347774830536e-05, 'epoch': 3.7}\n",
      "{'loss': 5.5172, 'learning_rate': 1.672069947932017e-05, 'epoch': 3.71}\n",
      "{'loss': 5.5754, 'learning_rate': 1.6701051183809806e-05, 'epoch': 3.73}\n",
      "{'loss': 5.3664, 'learning_rate': 1.6681402888299442e-05, 'epoch': 3.74}\n",
      "{'loss': 4.9346, 'learning_rate': 1.6661754592789077e-05, 'epoch': 3.75}\n",
      "{'loss': 4.9902, 'learning_rate': 1.6642106297278713e-05, 'epoch': 3.77}\n",
      "{'loss': 5.4152, 'learning_rate': 1.6622458001768348e-05, 'epoch': 3.78}\n",
      "{'loss': 5.5691, 'learning_rate': 1.6602809706257983e-05, 'epoch': 3.79}\n",
      "{'loss': 5.5563, 'learning_rate': 1.658316141074762e-05, 'epoch': 3.81}\n",
      "{'loss': 5.1488, 'learning_rate': 1.6563513115237254e-05, 'epoch': 3.82}\n",
      "{'loss': 5.407, 'learning_rate': 1.654386481972689e-05, 'epoch': 3.83}\n",
      "{'loss': 5.6918, 'learning_rate': 1.6524216524216525e-05, 'epoch': 3.85}\n",
      "{'loss': 5.7379, 'learning_rate': 1.650456822870616e-05, 'epoch': 3.86}\n",
      "{'loss': 5.2543, 'learning_rate': 1.64849199331958e-05, 'epoch': 3.87}\n",
      "{'loss': 4.8494, 'learning_rate': 1.646527163768543e-05, 'epoch': 3.89}\n",
      "{'loss': 5.4043, 'learning_rate': 1.6445623342175066e-05, 'epoch': 3.9}\n",
      "{'loss': 5.7082, 'learning_rate': 1.6425975046664705e-05, 'epoch': 3.91}\n",
      "{'loss': 4.959, 'learning_rate': 1.6406326751154337e-05, 'epoch': 3.93}\n",
      "{'loss': 5.4215, 'learning_rate': 1.6386678455643973e-05, 'epoch': 3.94}\n",
      "{'loss': 5.3215, 'learning_rate': 1.636703016013361e-05, 'epoch': 3.95}\n",
      "{'loss': 5.4541, 'learning_rate': 1.6347381864623247e-05, 'epoch': 3.97}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-3000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9645, 'learning_rate': 1.632773356911288e-05, 'epoch': 3.98}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-3000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-2000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.5418, 'learning_rate': 1.6308085273602517e-05, 'epoch': 3.99}\n",
      "{'loss': 5.4422, 'learning_rate': 1.6288436978092153e-05, 'epoch': 4.01}\n",
      "{'loss': 5.0492, 'learning_rate': 1.6268788682581785e-05, 'epoch': 4.02}\n",
      "{'loss': 5.3301, 'learning_rate': 1.6249140387071424e-05, 'epoch': 4.03}\n",
      "{'loss': 5.2953, 'learning_rate': 1.622949209156106e-05, 'epoch': 4.05}\n",
      "{'loss': 5.3105, 'learning_rate': 1.6209843796050694e-05, 'epoch': 4.06}\n",
      "{'loss': 5.032, 'learning_rate': 1.619019550054033e-05, 'epoch': 4.07}\n",
      "{'loss': 5.5426, 'learning_rate': 1.6170547205029965e-05, 'epoch': 4.08}\n",
      "{'loss': 5.1285, 'learning_rate': 1.61508989095196e-05, 'epoch': 4.1}\n",
      "{'loss': 5.3355, 'learning_rate': 1.6131250614009236e-05, 'epoch': 4.11}\n",
      "{'loss': 5.2383, 'learning_rate': 1.611160231849887e-05, 'epoch': 4.12}\n",
      "{'loss': 5.2797, 'learning_rate': 1.6091954022988507e-05, 'epoch': 4.14}\n",
      "{'loss': 4.6182, 'learning_rate': 1.6072305727478142e-05, 'epoch': 4.15}\n",
      "{'loss': 4.9969, 'learning_rate': 1.6052657431967777e-05, 'epoch': 4.16}\n",
      "{'loss': 5.7941, 'learning_rate': 1.6033009136457413e-05, 'epoch': 4.18}\n",
      "{'loss': 5.1154, 'learning_rate': 1.6013360840947048e-05, 'epoch': 4.19}\n",
      "{'loss': 5.2176, 'learning_rate': 1.5993712545436684e-05, 'epoch': 4.2}\n",
      "{'loss': 5.423, 'learning_rate': 1.597406424992632e-05, 'epoch': 4.22}\n",
      "{'loss': 5.4598, 'learning_rate': 1.5954415954415958e-05, 'epoch': 4.23}\n",
      "{'loss': 4.8797, 'learning_rate': 1.5934767658905593e-05, 'epoch': 4.24}\n",
      "{'loss': 5.4508, 'learning_rate': 1.5915119363395225e-05, 'epoch': 4.26}\n",
      "{'loss': 5.1238, 'learning_rate': 1.5895471067884864e-05, 'epoch': 4.27}\n",
      "{'loss': 4.9797, 'learning_rate': 1.58758227723745e-05, 'epoch': 4.28}\n",
      "{'loss': 5.5141, 'learning_rate': 1.585617447686413e-05, 'epoch': 4.3}\n",
      "{'loss': 5.2666, 'learning_rate': 1.583652618135377e-05, 'epoch': 4.31}\n",
      "{'loss': 5.2227, 'learning_rate': 1.5816877885843405e-05, 'epoch': 4.32}\n",
      "{'loss': 5.0959, 'learning_rate': 1.579722959033304e-05, 'epoch': 4.34}\n",
      "{'loss': 5.5012, 'learning_rate': 1.5777581294822676e-05, 'epoch': 4.35}\n",
      "{'loss': 5.1922, 'learning_rate': 1.575793299931231e-05, 'epoch': 4.36}\n",
      "{'loss': 4.7879, 'learning_rate': 1.5738284703801947e-05, 'epoch': 4.38}\n",
      "{'loss': 4.8422, 'learning_rate': 1.5718636408291582e-05, 'epoch': 4.39}\n",
      "{'loss': 5.1318, 'learning_rate': 1.5698988112781218e-05, 'epoch': 4.4}\n",
      "{'loss': 5.152, 'learning_rate': 1.5679339817270853e-05, 'epoch': 4.42}\n",
      "{'loss': 5.5082, 'learning_rate': 1.565969152176049e-05, 'epoch': 4.43}\n",
      "{'loss': 4.9916, 'learning_rate': 1.5640043226250124e-05, 'epoch': 4.44}\n",
      "{'loss': 5.3336, 'learning_rate': 1.562039493073976e-05, 'epoch': 4.46}\n",
      "{'loss': 5.1391, 'learning_rate': 1.5600746635229395e-05, 'epoch': 4.47}\n",
      "{'loss': 5.4738, 'learning_rate': 1.558109833971903e-05, 'epoch': 4.48}\n",
      "{'loss': 4.9043, 'learning_rate': 1.5561450044208665e-05, 'epoch': 4.5}\n",
      "{'loss': 5.002, 'learning_rate': 1.55418017486983e-05, 'epoch': 4.51}\n",
      "{'loss': 5.2572, 'learning_rate': 1.552215345318794e-05, 'epoch': 4.52}\n",
      "{'loss': 4.9271, 'learning_rate': 1.550250515767757e-05, 'epoch': 4.54}\n",
      "{'loss': 5.2613, 'learning_rate': 1.5482856862167207e-05, 'epoch': 4.55}\n",
      "{'loss': 5.0676, 'learning_rate': 1.5463208566656846e-05, 'epoch': 4.56}\n",
      "{'loss': 5.2184, 'learning_rate': 1.5443560271146478e-05, 'epoch': 4.58}\n",
      "{'loss': 5.0879, 'learning_rate': 1.5423911975636113e-05, 'epoch': 4.59}\n",
      "{'loss': 5.1682, 'learning_rate': 1.5404263680125752e-05, 'epoch': 4.6}\n",
      "{'loss': 5.2258, 'learning_rate': 1.5384615384615387e-05, 'epoch': 4.62}\n",
      "{'loss': 5.2012, 'learning_rate': 1.536496708910502e-05, 'epoch': 4.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-3500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.125, 'learning_rate': 1.5345318793594658e-05, 'epoch': 4.64}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-3500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-2500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.1145, 'learning_rate': 1.5325670498084293e-05, 'epoch': 4.66}\n",
      "{'loss': 5.1658, 'learning_rate': 1.5306022202573925e-05, 'epoch': 4.67}\n",
      "{'loss': 4.5479, 'learning_rate': 1.5286373907063564e-05, 'epoch': 4.68}\n",
      "{'loss': 4.7945, 'learning_rate': 1.52667256115532e-05, 'epoch': 4.69}\n",
      "{'loss': 5.4285, 'learning_rate': 1.5247077316042835e-05, 'epoch': 4.71}\n",
      "{'loss': 4.9301, 'learning_rate': 1.5227429020532469e-05, 'epoch': 4.72}\n",
      "{'loss': 5.3055, 'learning_rate': 1.5207780725022106e-05, 'epoch': 4.73}\n",
      "{'loss': 4.6533, 'learning_rate': 1.5188132429511741e-05, 'epoch': 4.75}\n",
      "{'loss': 5.3129, 'learning_rate': 1.5168484134001376e-05, 'epoch': 4.76}\n",
      "{'loss': 5.5691, 'learning_rate': 1.5148835838491012e-05, 'epoch': 4.77}\n",
      "{'loss': 5.4941, 'learning_rate': 1.5129187542980647e-05, 'epoch': 4.79}\n",
      "{'loss': 5.7043, 'learning_rate': 1.5109539247470284e-05, 'epoch': 4.8}\n",
      "{'loss': 5.2754, 'learning_rate': 1.5089890951959918e-05, 'epoch': 4.81}\n",
      "{'loss': 5.059, 'learning_rate': 1.5070242656449553e-05, 'epoch': 4.83}\n",
      "{'loss': 5.3035, 'learning_rate': 1.505059436093919e-05, 'epoch': 4.84}\n",
      "{'loss': 5.5816, 'learning_rate': 1.5030946065428824e-05, 'epoch': 4.85}\n",
      "{'loss': 5.2609, 'learning_rate': 1.5011297769918461e-05, 'epoch': 4.87}\n",
      "{'loss': 5.2221, 'learning_rate': 1.4991649474408097e-05, 'epoch': 4.88}\n",
      "{'loss': 5.1668, 'learning_rate': 1.4972001178897732e-05, 'epoch': 4.89}\n",
      "{'loss': 4.9807, 'learning_rate': 1.4952352883387367e-05, 'epoch': 4.91}\n",
      "{'loss': 4.7068, 'learning_rate': 1.4932704587877003e-05, 'epoch': 4.92}\n",
      "{'loss': 5.1027, 'learning_rate': 1.4913056292366638e-05, 'epoch': 4.93}\n",
      "{'loss': 4.8145, 'learning_rate': 1.4893407996856275e-05, 'epoch': 4.95}\n",
      "{'loss': 5.2467, 'learning_rate': 1.4873759701345909e-05, 'epoch': 4.96}\n",
      "{'loss': 5.0148, 'learning_rate': 1.4854111405835546e-05, 'epoch': 4.97}\n",
      "{'loss': 5.1441, 'learning_rate': 1.4834463110325181e-05, 'epoch': 4.99}\n",
      "{'loss': 5.3027, 'learning_rate': 1.4814814814814815e-05, 'epoch': 5.0}\n",
      "{'loss': 4.7889, 'learning_rate': 1.4795166519304452e-05, 'epoch': 5.01}\n",
      "{'loss': 4.9031, 'learning_rate': 1.4775518223794087e-05, 'epoch': 5.03}\n",
      "{'loss': 5.4941, 'learning_rate': 1.4755869928283724e-05, 'epoch': 5.04}\n",
      "{'loss': 4.7967, 'learning_rate': 1.4736221632773358e-05, 'epoch': 5.05}\n",
      "{'loss': 4.7547, 'learning_rate': 1.4716573337262994e-05, 'epoch': 5.07}\n",
      "{'loss': 5.4473, 'learning_rate': 1.469692504175263e-05, 'epoch': 5.08}\n",
      "{'loss': 5.2166, 'learning_rate': 1.4677276746242264e-05, 'epoch': 5.09}\n",
      "{'loss': 4.8498, 'learning_rate': 1.46576284507319e-05, 'epoch': 5.11}\n",
      "{'loss': 5.2305, 'learning_rate': 1.4637980155221537e-05, 'epoch': 5.12}\n",
      "{'loss': 4.9551, 'learning_rate': 1.4618331859711172e-05, 'epoch': 5.13}\n",
      "{'loss': 5.124, 'learning_rate': 1.4598683564200806e-05, 'epoch': 5.15}\n",
      "{'loss': 5.2102, 'learning_rate': 1.4579035268690443e-05, 'epoch': 5.16}\n",
      "{'loss': 4.9828, 'learning_rate': 1.4559386973180078e-05, 'epoch': 5.17}\n",
      "{'loss': 5.5402, 'learning_rate': 1.4539738677669712e-05, 'epoch': 5.19}\n",
      "{'loss': 5.2119, 'learning_rate': 1.4520090382159349e-05, 'epoch': 5.2}\n",
      "{'loss': 4.7875, 'learning_rate': 1.4500442086648984e-05, 'epoch': 5.21}\n",
      "{'loss': 5.0361, 'learning_rate': 1.4480793791138622e-05, 'epoch': 5.23}\n",
      "{'loss': 5.457, 'learning_rate': 1.4461145495628255e-05, 'epoch': 5.24}\n",
      "{'loss': 5.2934, 'learning_rate': 1.444149720011789e-05, 'epoch': 5.25}\n",
      "{'loss': 5.2418, 'learning_rate': 1.4421848904607528e-05, 'epoch': 5.27}\n",
      "{'loss': 5.5668, 'learning_rate': 1.4402200609097161e-05, 'epoch': 5.28}\n",
      "{'loss': 4.7223, 'learning_rate': 1.4382552313586797e-05, 'epoch': 5.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.5477, 'learning_rate': 1.4362904018076434e-05, 'epoch': 5.31}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-4000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-3000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9875, 'learning_rate': 1.434325572256607e-05, 'epoch': 5.32}\n",
      "{'loss': 5.3879, 'learning_rate': 1.4323607427055703e-05, 'epoch': 5.33}\n",
      "{'loss': 5.3109, 'learning_rate': 1.430395913154534e-05, 'epoch': 5.34}\n",
      "{'loss': 5.1684, 'learning_rate': 1.4284310836034975e-05, 'epoch': 5.36}\n",
      "{'loss': 5.0809, 'learning_rate': 1.4264662540524609e-05, 'epoch': 5.37}\n",
      "{'loss': 4.8801, 'learning_rate': 1.4245014245014246e-05, 'epoch': 5.38}\n",
      "{'loss': 5.4352, 'learning_rate': 1.4225365949503882e-05, 'epoch': 5.4}\n",
      "{'loss': 5.1777, 'learning_rate': 1.4205717653993519e-05, 'epoch': 5.41}\n",
      "{'loss': 4.7789, 'learning_rate': 1.4186069358483152e-05, 'epoch': 5.42}\n",
      "{'loss': 4.7395, 'learning_rate': 1.4166421062972788e-05, 'epoch': 5.44}\n",
      "{'loss': 5.2344, 'learning_rate': 1.4146772767462425e-05, 'epoch': 5.45}\n",
      "{'loss': 5.1406, 'learning_rate': 1.4127124471952058e-05, 'epoch': 5.46}\n",
      "{'loss': 5.3781, 'learning_rate': 1.4107476176441694e-05, 'epoch': 5.48}\n",
      "{'loss': 5.0145, 'learning_rate': 1.4087827880931331e-05, 'epoch': 5.49}\n",
      "{'loss': 5.4359, 'learning_rate': 1.4068179585420966e-05, 'epoch': 5.5}\n",
      "{'loss': 5.0895, 'learning_rate': 1.40485312899106e-05, 'epoch': 5.52}\n",
      "{'loss': 5.3803, 'learning_rate': 1.4028882994400237e-05, 'epoch': 5.53}\n",
      "{'loss': 5.1887, 'learning_rate': 1.4009234698889872e-05, 'epoch': 5.54}\n",
      "{'loss': 5.2129, 'learning_rate': 1.3989586403379508e-05, 'epoch': 5.56}\n",
      "{'loss': 4.6635, 'learning_rate': 1.3969938107869143e-05, 'epoch': 5.57}\n",
      "{'loss': 5.2812, 'learning_rate': 1.3950289812358779e-05, 'epoch': 5.58}\n",
      "{'loss': 5.0365, 'learning_rate': 1.3930641516848416e-05, 'epoch': 5.6}\n",
      "{'loss': 5.1213, 'learning_rate': 1.391099322133805e-05, 'epoch': 5.61}\n",
      "{'loss': 4.7998, 'learning_rate': 1.3891344925827686e-05, 'epoch': 5.62}\n",
      "{'loss': 5.0715, 'learning_rate': 1.3871696630317322e-05, 'epoch': 5.64}\n",
      "{'loss': 5.057, 'learning_rate': 1.3852048334806955e-05, 'epoch': 5.65}\n",
      "{'loss': 5.0186, 'learning_rate': 1.3832400039296593e-05, 'epoch': 5.66}\n",
      "{'loss': 5.2957, 'learning_rate': 1.3812751743786228e-05, 'epoch': 5.68}\n",
      "{'loss': 4.9938, 'learning_rate': 1.3793103448275863e-05, 'epoch': 5.69}\n",
      "{'loss': 4.818, 'learning_rate': 1.3773455152765499e-05, 'epoch': 5.7}\n",
      "{'loss': 4.8252, 'learning_rate': 1.3753806857255134e-05, 'epoch': 5.72}\n",
      "{'loss': 4.785, 'learning_rate': 1.3734158561744771e-05, 'epoch': 5.73}\n",
      "{'loss': 5.0139, 'learning_rate': 1.3714510266234405e-05, 'epoch': 5.74}\n",
      "{'loss': 5.0637, 'learning_rate': 1.369486197072404e-05, 'epoch': 5.76}\n",
      "{'loss': 4.8949, 'learning_rate': 1.3675213675213677e-05, 'epoch': 5.77}\n",
      "{'loss': 5.698, 'learning_rate': 1.3655565379703313e-05, 'epoch': 5.78}\n",
      "{'loss': 4.9437, 'learning_rate': 1.3635917084192946e-05, 'epoch': 5.8}\n",
      "{'loss': 5.1152, 'learning_rate': 1.3616268788682583e-05, 'epoch': 5.81}\n",
      "{'loss': 5.2488, 'learning_rate': 1.3596620493172219e-05, 'epoch': 5.82}\n",
      "{'loss': 5.057, 'learning_rate': 1.3576972197661853e-05, 'epoch': 5.84}\n",
      "{'loss': 5.0379, 'learning_rate': 1.355732390215149e-05, 'epoch': 5.85}\n",
      "{'loss': 4.8375, 'learning_rate': 1.3537675606641125e-05, 'epoch': 5.86}\n",
      "{'loss': 5.0477, 'learning_rate': 1.3518027311130762e-05, 'epoch': 5.88}\n",
      "{'loss': 4.8477, 'learning_rate': 1.3498379015620396e-05, 'epoch': 5.89}\n",
      "{'loss': 5.2262, 'learning_rate': 1.3478730720110031e-05, 'epoch': 5.9}\n",
      "{'loss': 4.9793, 'learning_rate': 1.3459082424599668e-05, 'epoch': 5.92}\n",
      "{'loss': 4.8223, 'learning_rate': 1.3439434129089302e-05, 'epoch': 5.93}\n",
      "{'loss': 5.1437, 'learning_rate': 1.3419785833578937e-05, 'epoch': 5.94}\n",
      "{'loss': 5.0711, 'learning_rate': 1.3400137538068574e-05, 'epoch': 5.95}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-4500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.701, 'learning_rate': 1.338048924255821e-05, 'epoch': 5.97}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-4500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-3500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8125, 'learning_rate': 1.3360840947047843e-05, 'epoch': 5.98}\n",
      "{'loss': 5.1516, 'learning_rate': 1.334119265153748e-05, 'epoch': 5.99}\n",
      "{'loss': 4.7371, 'learning_rate': 1.3321544356027116e-05, 'epoch': 6.01}\n",
      "{'loss': 4.934, 'learning_rate': 1.330189606051675e-05, 'epoch': 6.02}\n",
      "{'loss': 5.2363, 'learning_rate': 1.3282247765006387e-05, 'epoch': 6.03}\n",
      "{'loss': 5.0172, 'learning_rate': 1.3262599469496022e-05, 'epoch': 6.05}\n",
      "{'loss': 4.8961, 'learning_rate': 1.3242951173985659e-05, 'epoch': 6.06}\n",
      "{'loss': 4.8738, 'learning_rate': 1.3223302878475293e-05, 'epoch': 6.07}\n",
      "{'loss': 5.0334, 'learning_rate': 1.3203654582964928e-05, 'epoch': 6.09}\n",
      "{'loss': 5.2609, 'learning_rate': 1.3184006287454565e-05, 'epoch': 6.1}\n",
      "{'loss': 5.2771, 'learning_rate': 1.3164357991944199e-05, 'epoch': 6.11}\n",
      "{'loss': 4.7986, 'learning_rate': 1.3144709696433834e-05, 'epoch': 6.13}\n",
      "{'loss': 4.7371, 'learning_rate': 1.3125061400923471e-05, 'epoch': 6.14}\n",
      "{'loss': 5.2352, 'learning_rate': 1.3105413105413107e-05, 'epoch': 6.15}\n",
      "{'loss': 4.759, 'learning_rate': 1.308576480990274e-05, 'epoch': 6.17}\n",
      "{'loss': 4.9629, 'learning_rate': 1.3066116514392378e-05, 'epoch': 6.18}\n",
      "{'loss': 5.2859, 'learning_rate': 1.3046468218882013e-05, 'epoch': 6.19}\n",
      "{'loss': 4.8926, 'learning_rate': 1.3026819923371648e-05, 'epoch': 6.21}\n",
      "{'loss': 5.4332, 'learning_rate': 1.3007171627861284e-05, 'epoch': 6.22}\n",
      "{'loss': 5.1449, 'learning_rate': 1.2987523332350919e-05, 'epoch': 6.23}\n",
      "{'loss': 5.0719, 'learning_rate': 1.2967875036840556e-05, 'epoch': 6.25}\n",
      "{'loss': 5.3098, 'learning_rate': 1.294822674133019e-05, 'epoch': 6.26}\n",
      "{'loss': 5.2918, 'learning_rate': 1.2928578445819825e-05, 'epoch': 6.27}\n",
      "{'loss': 4.7787, 'learning_rate': 1.2908930150309462e-05, 'epoch': 6.29}\n",
      "{'loss': 4.4391, 'learning_rate': 1.2889281854799098e-05, 'epoch': 6.3}\n",
      "{'loss': 5.0445, 'learning_rate': 1.2869633559288733e-05, 'epoch': 6.31}\n",
      "{'loss': 5.1143, 'learning_rate': 1.2849985263778368e-05, 'epoch': 6.33}\n",
      "{'loss': 5.0723, 'learning_rate': 1.2830336968268004e-05, 'epoch': 6.34}\n",
      "{'loss': 5.1875, 'learning_rate': 1.281068867275764e-05, 'epoch': 6.35}\n",
      "{'loss': 4.9982, 'learning_rate': 1.2791040377247275e-05, 'epoch': 6.37}\n",
      "{'loss': 5.1541, 'learning_rate': 1.277139208173691e-05, 'epoch': 6.38}\n",
      "{'loss': 4.6867, 'learning_rate': 1.2751743786226547e-05, 'epoch': 6.39}\n",
      "{'loss': 5.1453, 'learning_rate': 1.273209549071618e-05, 'epoch': 6.41}\n",
      "{'loss': 4.6861, 'learning_rate': 1.2712447195205818e-05, 'epoch': 6.42}\n",
      "{'loss': 4.7668, 'learning_rate': 1.2692798899695453e-05, 'epoch': 6.43}\n",
      "{'loss': 5.1086, 'learning_rate': 1.2673150604185087e-05, 'epoch': 6.45}\n",
      "{'loss': 4.8367, 'learning_rate': 1.2653502308674724e-05, 'epoch': 6.46}\n",
      "{'loss': 5.1039, 'learning_rate': 1.263385401316436e-05, 'epoch': 6.47}\n",
      "{'loss': 4.7408, 'learning_rate': 1.2614205717653996e-05, 'epoch': 6.49}\n",
      "{'loss': 4.9277, 'learning_rate': 1.259455742214363e-05, 'epoch': 6.5}\n",
      "{'loss': 5.0309, 'learning_rate': 1.2574909126633265e-05, 'epoch': 6.51}\n",
      "{'loss': 4.8678, 'learning_rate': 1.2555260831122903e-05, 'epoch': 6.53}\n",
      "{'loss': 4.9514, 'learning_rate': 1.2535612535612536e-05, 'epoch': 6.54}\n",
      "{'loss': 4.9387, 'learning_rate': 1.2515964240102172e-05, 'epoch': 6.55}\n",
      "{'loss': 5.0568, 'learning_rate': 1.2496315944591809e-05, 'epoch': 6.56}\n",
      "{'loss': 4.9633, 'learning_rate': 1.2476667649081444e-05, 'epoch': 6.58}\n",
      "{'loss': 4.8965, 'learning_rate': 1.2457019353571078e-05, 'epoch': 6.59}\n",
      "{'loss': 4.8758, 'learning_rate': 1.2437371058060715e-05, 'epoch': 6.6}\n",
      "{'loss': 5.1559, 'learning_rate': 1.241772276255035e-05, 'epoch': 6.62}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-5000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0238, 'learning_rate': 1.2398074467039984e-05, 'epoch': 6.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-5000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-4000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.4713, 'learning_rate': 1.2378426171529621e-05, 'epoch': 6.64}\n",
      "{'loss': 4.6389, 'learning_rate': 1.2358777876019256e-05, 'epoch': 6.66}\n",
      "{'loss': 4.875, 'learning_rate': 1.2339129580508893e-05, 'epoch': 6.67}\n",
      "{'loss': 5.1289, 'learning_rate': 1.2319481284998527e-05, 'epoch': 6.68}\n",
      "{'loss': 4.8781, 'learning_rate': 1.2299832989488163e-05, 'epoch': 6.7}\n",
      "{'loss': 4.8652, 'learning_rate': 1.22801846939778e-05, 'epoch': 6.71}\n",
      "{'loss': 4.3197, 'learning_rate': 1.2260536398467433e-05, 'epoch': 6.72}\n",
      "{'loss': 5.257, 'learning_rate': 1.2240888102957069e-05, 'epoch': 6.74}\n",
      "{'loss': 5.3227, 'learning_rate': 1.2221239807446706e-05, 'epoch': 6.75}\n",
      "{'loss': 4.924, 'learning_rate': 1.2201591511936341e-05, 'epoch': 6.76}\n",
      "{'loss': 4.8963, 'learning_rate': 1.2181943216425975e-05, 'epoch': 6.78}\n",
      "{'loss': 4.6277, 'learning_rate': 1.2162294920915612e-05, 'epoch': 6.79}\n",
      "{'loss': 4.4197, 'learning_rate': 1.2142646625405247e-05, 'epoch': 6.8}\n",
      "{'loss': 5.1719, 'learning_rate': 1.2122998329894881e-05, 'epoch': 6.82}\n",
      "{'loss': 4.7242, 'learning_rate': 1.2103350034384518e-05, 'epoch': 6.83}\n",
      "{'loss': 4.8193, 'learning_rate': 1.2083701738874153e-05, 'epoch': 6.84}\n",
      "{'loss': 4.8066, 'learning_rate': 1.206405344336379e-05, 'epoch': 6.86}\n",
      "{'loss': 4.9832, 'learning_rate': 1.2044405147853424e-05, 'epoch': 6.87}\n",
      "{'loss': 5.057, 'learning_rate': 1.202475685234306e-05, 'epoch': 6.88}\n",
      "{'loss': 4.9332, 'learning_rate': 1.2005108556832697e-05, 'epoch': 6.9}\n",
      "{'loss': 4.6441, 'learning_rate': 1.198546026132233e-05, 'epoch': 6.91}\n",
      "{'loss': 4.6324, 'learning_rate': 1.1965811965811966e-05, 'epoch': 6.92}\n",
      "{'loss': 5.1152, 'learning_rate': 1.1946163670301603e-05, 'epoch': 6.94}\n",
      "{'loss': 5.2703, 'learning_rate': 1.1926515374791238e-05, 'epoch': 6.95}\n",
      "{'loss': 4.7059, 'learning_rate': 1.1906867079280872e-05, 'epoch': 6.96}\n",
      "{'loss': 5.3805, 'learning_rate': 1.1887218783770509e-05, 'epoch': 6.98}\n",
      "{'loss': 4.9168, 'learning_rate': 1.1867570488260144e-05, 'epoch': 6.99}\n",
      "{'loss': 5.325, 'learning_rate': 1.184792219274978e-05, 'epoch': 7.0}\n",
      "{'loss': 5.2604, 'learning_rate': 1.1828273897239415e-05, 'epoch': 7.02}\n",
      "{'loss': 5.1215, 'learning_rate': 1.180862560172905e-05, 'epoch': 7.03}\n",
      "{'loss': 4.8656, 'learning_rate': 1.1788977306218688e-05, 'epoch': 7.04}\n",
      "{'loss': 5.1914, 'learning_rate': 1.1769329010708321e-05, 'epoch': 7.06}\n",
      "{'loss': 4.8961, 'learning_rate': 1.1749680715197958e-05, 'epoch': 7.07}\n",
      "{'loss': 4.5658, 'learning_rate': 1.1730032419687594e-05, 'epoch': 7.08}\n",
      "{'loss': 4.9527, 'learning_rate': 1.1710384124177227e-05, 'epoch': 7.1}\n",
      "{'loss': 4.5836, 'learning_rate': 1.1690735828666864e-05, 'epoch': 7.11}\n",
      "{'loss': 5.0641, 'learning_rate': 1.16710875331565e-05, 'epoch': 7.12}\n",
      "{'loss': 5.0371, 'learning_rate': 1.1651439237646135e-05, 'epoch': 7.14}\n",
      "{'loss': 5.1566, 'learning_rate': 1.163179094213577e-05, 'epoch': 7.15}\n",
      "{'loss': 4.8605, 'learning_rate': 1.1612142646625406e-05, 'epoch': 7.16}\n",
      "{'loss': 4.4451, 'learning_rate': 1.1592494351115043e-05, 'epoch': 7.18}\n",
      "{'loss': 5.3996, 'learning_rate': 1.1572846055604677e-05, 'epoch': 7.19}\n",
      "{'loss': 4.9984, 'learning_rate': 1.1553197760094312e-05, 'epoch': 7.2}\n",
      "{'loss': 4.708, 'learning_rate': 1.153354946458395e-05, 'epoch': 7.21}\n",
      "{'loss': 5.2916, 'learning_rate': 1.1513901169073585e-05, 'epoch': 7.23}\n",
      "{'loss': 4.7906, 'learning_rate': 1.1494252873563218e-05, 'epoch': 7.24}\n",
      "{'loss': 4.8713, 'learning_rate': 1.1474604578052855e-05, 'epoch': 7.25}\n",
      "{'loss': 5.0375, 'learning_rate': 1.145495628254249e-05, 'epoch': 7.27}\n",
      "{'loss': 4.6963, 'learning_rate': 1.1435307987032124e-05, 'epoch': 7.28}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-5500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8291, 'learning_rate': 1.1415659691521762e-05, 'epoch': 7.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-5500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-4500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9039, 'learning_rate': 1.1396011396011397e-05, 'epoch': 7.31}\n",
      "{'loss': 5.3684, 'learning_rate': 1.1376363100501034e-05, 'epoch': 7.32}\n",
      "{'loss': 4.6115, 'learning_rate': 1.1356714804990668e-05, 'epoch': 7.33}\n",
      "{'loss': 4.4752, 'learning_rate': 1.1337066509480303e-05, 'epoch': 7.35}\n",
      "{'loss': 4.6789, 'learning_rate': 1.131741821396994e-05, 'epoch': 7.36}\n",
      "{'loss': 4.9105, 'learning_rate': 1.1297769918459574e-05, 'epoch': 7.37}\n",
      "{'loss': 4.91, 'learning_rate': 1.127812162294921e-05, 'epoch': 7.39}\n",
      "{'loss': 4.593, 'learning_rate': 1.1258473327438846e-05, 'epoch': 7.4}\n",
      "{'loss': 4.8967, 'learning_rate': 1.1238825031928482e-05, 'epoch': 7.41}\n",
      "{'loss': 4.884, 'learning_rate': 1.1219176736418115e-05, 'epoch': 7.43}\n",
      "{'loss': 4.5545, 'learning_rate': 1.1199528440907752e-05, 'epoch': 7.44}\n",
      "{'loss': 4.7791, 'learning_rate': 1.1179880145397388e-05, 'epoch': 7.45}\n",
      "{'loss': 5.1178, 'learning_rate': 1.1160231849887021e-05, 'epoch': 7.47}\n",
      "{'loss': 5.3754, 'learning_rate': 1.1140583554376659e-05, 'epoch': 7.48}\n",
      "{'loss': 4.2174, 'learning_rate': 1.1120935258866294e-05, 'epoch': 7.49}\n",
      "{'loss': 4.7301, 'learning_rate': 1.1101286963355931e-05, 'epoch': 7.51}\n",
      "{'loss': 4.9637, 'learning_rate': 1.1081638667845565e-05, 'epoch': 7.52}\n",
      "{'loss': 4.8215, 'learning_rate': 1.10619903723352e-05, 'epoch': 7.53}\n",
      "{'loss': 5.166, 'learning_rate': 1.1042342076824837e-05, 'epoch': 7.55}\n",
      "{'loss': 4.8322, 'learning_rate': 1.1022693781314471e-05, 'epoch': 7.56}\n",
      "{'loss': 5.1816, 'learning_rate': 1.1003045485804106e-05, 'epoch': 7.57}\n",
      "{'loss': 4.5234, 'learning_rate': 1.0983397190293743e-05, 'epoch': 7.59}\n",
      "{'loss': 4.4788, 'learning_rate': 1.0963748894783379e-05, 'epoch': 7.6}\n",
      "{'loss': 5.1729, 'learning_rate': 1.0944100599273012e-05, 'epoch': 7.61}\n",
      "{'loss': 4.7047, 'learning_rate': 1.092445230376265e-05, 'epoch': 7.63}\n",
      "{'loss': 4.5217, 'learning_rate': 1.0904804008252285e-05, 'epoch': 7.64}\n",
      "{'loss': 4.7529, 'learning_rate': 1.0885155712741922e-05, 'epoch': 7.65}\n",
      "{'loss': 4.6648, 'learning_rate': 1.0865507417231556e-05, 'epoch': 7.67}\n",
      "{'loss': 5.0508, 'learning_rate': 1.0845859121721191e-05, 'epoch': 7.68}\n",
      "{'loss': 5.0162, 'learning_rate': 1.0826210826210828e-05, 'epoch': 7.69}\n",
      "{'loss': 4.6977, 'learning_rate': 1.0806562530700462e-05, 'epoch': 7.71}\n",
      "{'loss': 5.2047, 'learning_rate': 1.0786914235190097e-05, 'epoch': 7.72}\n",
      "{'loss': 4.9418, 'learning_rate': 1.0767265939679734e-05, 'epoch': 7.73}\n",
      "{'loss': 4.9258, 'learning_rate': 1.074761764416937e-05, 'epoch': 7.75}\n",
      "{'loss': 4.941, 'learning_rate': 1.0727969348659005e-05, 'epoch': 7.76}\n",
      "{'loss': 4.8387, 'learning_rate': 1.070832105314864e-05, 'epoch': 7.77}\n",
      "{'loss': 4.7695, 'learning_rate': 1.0688672757638276e-05, 'epoch': 7.79}\n",
      "{'loss': 4.6906, 'learning_rate': 1.0669024462127911e-05, 'epoch': 7.8}\n",
      "{'loss': 5.1281, 'learning_rate': 1.0649376166617547e-05, 'epoch': 7.81}\n",
      "{'loss': 4.9545, 'learning_rate': 1.0629727871107182e-05, 'epoch': 7.82}\n",
      "{'loss': 4.8227, 'learning_rate': 1.0610079575596819e-05, 'epoch': 7.84}\n",
      "{'loss': 4.718, 'learning_rate': 1.0590431280086453e-05, 'epoch': 7.85}\n",
      "{'loss': 4.7066, 'learning_rate': 1.057078298457609e-05, 'epoch': 7.86}\n",
      "{'loss': 4.8779, 'learning_rate': 1.0551134689065725e-05, 'epoch': 7.88}\n",
      "{'loss': 4.7861, 'learning_rate': 1.0531486393555359e-05, 'epoch': 7.89}\n",
      "{'loss': 4.9404, 'learning_rate': 1.0511838098044996e-05, 'epoch': 7.9}\n",
      "{'loss': 4.5205, 'learning_rate': 1.0492189802534631e-05, 'epoch': 7.92}\n",
      "{'loss': 4.9867, 'learning_rate': 1.0472541507024268e-05, 'epoch': 7.93}\n",
      "{'loss': 4.7412, 'learning_rate': 1.0452893211513902e-05, 'epoch': 7.94}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-6000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.1219, 'learning_rate': 1.0433244916003537e-05, 'epoch': 7.96}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-6000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-5000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.7074, 'learning_rate': 1.0413596620493174e-05, 'epoch': 7.97}\n",
      "{'loss': 4.6156, 'learning_rate': 1.0393948324982808e-05, 'epoch': 7.98}\n",
      "{'loss': 4.7965, 'learning_rate': 1.0374300029472444e-05, 'epoch': 8.0}\n",
      "{'loss': 4.7666, 'learning_rate': 1.035465173396208e-05, 'epoch': 8.01}\n",
      "{'loss': 5.1873, 'learning_rate': 1.0335003438451716e-05, 'epoch': 8.02}\n",
      "{'loss': 4.765, 'learning_rate': 1.031535514294135e-05, 'epoch': 8.04}\n",
      "{'loss': 5.4934, 'learning_rate': 1.0295706847430987e-05, 'epoch': 8.05}\n",
      "{'loss': 4.9908, 'learning_rate': 1.0276058551920622e-05, 'epoch': 8.06}\n",
      "{'loss': 4.9559, 'learning_rate': 1.0256410256410256e-05, 'epoch': 8.08}\n",
      "{'loss': 4.6074, 'learning_rate': 1.0236761960899893e-05, 'epoch': 8.09}\n",
      "{'loss': 4.8414, 'learning_rate': 1.0217113665389528e-05, 'epoch': 8.1}\n",
      "{'loss': 4.776, 'learning_rate': 1.0197465369879165e-05, 'epoch': 8.12}\n",
      "{'loss': 5.2559, 'learning_rate': 1.0177817074368799e-05, 'epoch': 8.13}\n",
      "{'loss': 5.2598, 'learning_rate': 1.0158168778858434e-05, 'epoch': 8.14}\n",
      "{'loss': 4.8832, 'learning_rate': 1.0138520483348072e-05, 'epoch': 8.16}\n",
      "{'loss': 4.9152, 'learning_rate': 1.0118872187837705e-05, 'epoch': 8.17}\n",
      "{'loss': 4.7141, 'learning_rate': 1.009922389232734e-05, 'epoch': 8.18}\n",
      "{'loss': 4.8773, 'learning_rate': 1.0079575596816978e-05, 'epoch': 8.2}\n",
      "{'loss': 4.5227, 'learning_rate': 1.0059927301306613e-05, 'epoch': 8.21}\n",
      "{'loss': 4.7607, 'learning_rate': 1.0040279005796247e-05, 'epoch': 8.22}\n",
      "{'loss': 4.6762, 'learning_rate': 1.0020630710285884e-05, 'epoch': 8.24}\n",
      "{'loss': 4.6072, 'learning_rate': 1.000098241477552e-05, 'epoch': 8.25}\n",
      "{'loss': 4.8795, 'learning_rate': 9.981334119265155e-06, 'epoch': 8.26}\n",
      "{'loss': 4.6105, 'learning_rate': 9.96168582375479e-06, 'epoch': 8.28}\n",
      "{'loss': 4.5887, 'learning_rate': 9.942037528244425e-06, 'epoch': 8.29}\n",
      "{'loss': 4.7391, 'learning_rate': 9.92238923273406e-06, 'epoch': 8.3}\n",
      "{'loss': 4.9262, 'learning_rate': 9.902740937223698e-06, 'epoch': 8.32}\n",
      "{'loss': 4.884, 'learning_rate': 9.883092641713332e-06, 'epoch': 8.33}\n",
      "{'loss': 4.4682, 'learning_rate': 9.863444346202967e-06, 'epoch': 8.34}\n",
      "{'loss': 4.8447, 'learning_rate': 9.843796050692604e-06, 'epoch': 8.36}\n",
      "{'loss': 4.9486, 'learning_rate': 9.824147755182238e-06, 'epoch': 8.37}\n",
      "{'loss': 4.773, 'learning_rate': 9.804499459671875e-06, 'epoch': 8.38}\n",
      "{'loss': 4.8137, 'learning_rate': 9.78485116416151e-06, 'epoch': 8.4}\n",
      "{'loss': 5.241, 'learning_rate': 9.765202868651146e-06, 'epoch': 8.41}\n",
      "{'loss': 4.8951, 'learning_rate': 9.745554573140781e-06, 'epoch': 8.42}\n",
      "{'loss': 4.5584, 'learning_rate': 9.725906277630416e-06, 'epoch': 8.44}\n",
      "{'loss': 4.7596, 'learning_rate': 9.706257982120052e-06, 'epoch': 8.45}\n",
      "{'loss': 4.8461, 'learning_rate': 9.686609686609687e-06, 'epoch': 8.46}\n",
      "{'loss': 4.6916, 'learning_rate': 9.666961391099322e-06, 'epoch': 8.47}\n",
      "{'loss': 4.8607, 'learning_rate': 9.647313095588958e-06, 'epoch': 8.49}\n",
      "{'loss': 4.7297, 'learning_rate': 9.627664800078595e-06, 'epoch': 8.5}\n",
      "{'loss': 4.7654, 'learning_rate': 9.60801650456823e-06, 'epoch': 8.51}\n",
      "{'loss': 4.6748, 'learning_rate': 9.588368209057864e-06, 'epoch': 8.53}\n",
      "{'loss': 4.9672, 'learning_rate': 9.568719913547501e-06, 'epoch': 8.54}\n",
      "{'loss': 5.0262, 'learning_rate': 9.549071618037136e-06, 'epoch': 8.55}\n",
      "{'loss': 4.7332, 'learning_rate': 9.529423322526772e-06, 'epoch': 8.57}\n",
      "{'loss': 4.991, 'learning_rate': 9.509775027016407e-06, 'epoch': 8.58}\n",
      "{'loss': 4.8531, 'learning_rate': 9.490126731506043e-06, 'epoch': 8.59}\n",
      "{'loss': 5.1719, 'learning_rate': 9.470478435995678e-06, 'epoch': 8.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-6500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8875, 'learning_rate': 9.450830140485315e-06, 'epoch': 8.62}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-6500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-5500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9395, 'learning_rate': 9.431181844974949e-06, 'epoch': 8.63}\n",
      "{'loss': 4.9434, 'learning_rate': 9.411533549464584e-06, 'epoch': 8.65}\n",
      "{'loss': 4.7836, 'learning_rate': 9.391885253954221e-06, 'epoch': 8.66}\n",
      "{'loss': 4.7729, 'learning_rate': 9.372236958443855e-06, 'epoch': 8.67}\n",
      "{'loss': 4.6176, 'learning_rate': 9.352588662933492e-06, 'epoch': 8.69}\n",
      "{'loss': 5.0152, 'learning_rate': 9.332940367423127e-06, 'epoch': 8.7}\n",
      "{'loss': 4.6039, 'learning_rate': 9.313292071912763e-06, 'epoch': 8.71}\n",
      "{'loss': 4.7744, 'learning_rate': 9.293643776402398e-06, 'epoch': 8.73}\n",
      "{'loss': 5.1273, 'learning_rate': 9.273995480892033e-06, 'epoch': 8.74}\n",
      "{'loss': 4.8971, 'learning_rate': 9.254347185381669e-06, 'epoch': 8.75}\n",
      "{'loss': 4.8797, 'learning_rate': 9.234698889871304e-06, 'epoch': 8.77}\n",
      "{'loss': 4.8184, 'learning_rate': 9.21505059436094e-06, 'epoch': 8.78}\n",
      "{'loss': 4.8047, 'learning_rate': 9.195402298850575e-06, 'epoch': 8.79}\n",
      "{'loss': 4.7918, 'learning_rate': 9.175754003340212e-06, 'epoch': 8.81}\n",
      "{'loss': 5.0863, 'learning_rate': 9.156105707829846e-06, 'epoch': 8.82}\n",
      "{'loss': 4.7109, 'learning_rate': 9.136457412319481e-06, 'epoch': 8.83}\n",
      "{'loss': 4.7434, 'learning_rate': 9.116809116809118e-06, 'epoch': 8.85}\n",
      "{'loss': 4.5631, 'learning_rate': 9.097160821298754e-06, 'epoch': 8.86}\n",
      "{'loss': 4.9133, 'learning_rate': 9.077512525788389e-06, 'epoch': 8.87}\n",
      "{'loss': 4.9689, 'learning_rate': 9.057864230278024e-06, 'epoch': 8.89}\n",
      "{'loss': 4.76, 'learning_rate': 9.03821593476766e-06, 'epoch': 8.9}\n",
      "{'loss': 5.2438, 'learning_rate': 9.018567639257295e-06, 'epoch': 8.91}\n",
      "{'loss': 4.9061, 'learning_rate': 8.99891934374693e-06, 'epoch': 8.93}\n",
      "{'loss': 4.9156, 'learning_rate': 8.979271048236566e-06, 'epoch': 8.94}\n",
      "{'loss': 4.7266, 'learning_rate': 8.959622752726201e-06, 'epoch': 8.95}\n",
      "{'loss': 4.9248, 'learning_rate': 8.939974457215838e-06, 'epoch': 8.97}\n",
      "{'loss': 4.6756, 'learning_rate': 8.920326161705472e-06, 'epoch': 8.98}\n",
      "{'loss': 4.5049, 'learning_rate': 8.900677866195109e-06, 'epoch': 8.99}\n",
      "{'loss': 5.0359, 'learning_rate': 8.881029570684744e-06, 'epoch': 9.01}\n",
      "{'loss': 4.6527, 'learning_rate': 8.861381275174378e-06, 'epoch': 9.02}\n",
      "{'loss': 4.7391, 'learning_rate': 8.841732979664015e-06, 'epoch': 9.03}\n",
      "{'loss': 4.8441, 'learning_rate': 8.82208468415365e-06, 'epoch': 9.05}\n",
      "{'loss': 4.8605, 'learning_rate': 8.802436388643286e-06, 'epoch': 9.06}\n",
      "{'loss': 5.0805, 'learning_rate': 8.782788093132921e-06, 'epoch': 9.07}\n",
      "{'loss': 5.1369, 'learning_rate': 8.763139797622557e-06, 'epoch': 9.08}\n",
      "{'loss': 4.9273, 'learning_rate': 8.743491502112192e-06, 'epoch': 9.1}\n",
      "{'loss': 4.7605, 'learning_rate': 8.723843206601828e-06, 'epoch': 9.11}\n",
      "{'loss': 4.9662, 'learning_rate': 8.704194911091463e-06, 'epoch': 9.12}\n",
      "{'loss': 4.6869, 'learning_rate': 8.684546615581098e-06, 'epoch': 9.14}\n",
      "{'loss': 4.8012, 'learning_rate': 8.664898320070735e-06, 'epoch': 9.15}\n",
      "{'loss': 4.8711, 'learning_rate': 8.645250024560369e-06, 'epoch': 9.16}\n",
      "{'loss': 5.1115, 'learning_rate': 8.625601729050006e-06, 'epoch': 9.18}\n",
      "{'loss': 4.6832, 'learning_rate': 8.605953433539642e-06, 'epoch': 9.19}\n",
      "{'loss': 5.0086, 'learning_rate': 8.586305138029277e-06, 'epoch': 9.2}\n",
      "{'loss': 4.877, 'learning_rate': 8.566656842518912e-06, 'epoch': 9.22}\n",
      "{'loss': 4.6076, 'learning_rate': 8.547008547008548e-06, 'epoch': 9.23}\n",
      "{'loss': 5.0148, 'learning_rate': 8.527360251498183e-06, 'epoch': 9.24}\n",
      "{'loss': 4.8289, 'learning_rate': 8.507711955987818e-06, 'epoch': 9.26}\n",
      "{'loss': 4.6723, 'learning_rate': 8.488063660477454e-06, 'epoch': 9.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-7000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6928, 'learning_rate': 8.46841536496709e-06, 'epoch': 9.28}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-7000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-6000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9227, 'learning_rate': 8.448767069456726e-06, 'epoch': 9.3}\n",
      "{'loss': 4.9646, 'learning_rate': 8.429118773946362e-06, 'epoch': 9.31}\n",
      "{'loss': 4.4879, 'learning_rate': 8.409470478435995e-06, 'epoch': 9.32}\n",
      "{'loss': 4.8428, 'learning_rate': 8.389822182925632e-06, 'epoch': 9.34}\n",
      "{'loss': 4.784, 'learning_rate': 8.370173887415268e-06, 'epoch': 9.35}\n",
      "{'loss': 4.902, 'learning_rate': 8.350525591904903e-06, 'epoch': 9.36}\n",
      "{'loss': 4.6236, 'learning_rate': 8.330877296394539e-06, 'epoch': 9.38}\n",
      "{'loss': 4.999, 'learning_rate': 8.311229000884174e-06, 'epoch': 9.39}\n",
      "{'loss': 4.5885, 'learning_rate': 8.29158070537381e-06, 'epoch': 9.4}\n",
      "{'loss': 4.6002, 'learning_rate': 8.271932409863445e-06, 'epoch': 9.42}\n",
      "{'loss': 4.6053, 'learning_rate': 8.25228411435308e-06, 'epoch': 9.43}\n",
      "{'loss': 4.5727, 'learning_rate': 8.232635818842715e-06, 'epoch': 9.44}\n",
      "{'loss': 4.9293, 'learning_rate': 8.212987523332353e-06, 'epoch': 9.46}\n",
      "{'loss': 4.5693, 'learning_rate': 8.193339227821986e-06, 'epoch': 9.47}\n",
      "{'loss': 4.624, 'learning_rate': 8.173690932311623e-06, 'epoch': 9.48}\n",
      "{'loss': 4.6943, 'learning_rate': 8.154042636801259e-06, 'epoch': 9.5}\n",
      "{'loss': 4.4867, 'learning_rate': 8.134394341290892e-06, 'epoch': 9.51}\n",
      "{'loss': 4.7822, 'learning_rate': 8.11474604578053e-06, 'epoch': 9.52}\n",
      "{'loss': 4.5568, 'learning_rate': 8.095097750270165e-06, 'epoch': 9.54}\n",
      "{'loss': 4.7457, 'learning_rate': 8.0754494547598e-06, 'epoch': 9.55}\n",
      "{'loss': 4.4926, 'learning_rate': 8.055801159249436e-06, 'epoch': 9.56}\n",
      "{'loss': 4.4754, 'learning_rate': 8.036152863739071e-06, 'epoch': 9.58}\n",
      "{'loss': 4.4889, 'learning_rate': 8.016504568228706e-06, 'epoch': 9.59}\n",
      "{'loss': 4.2084, 'learning_rate': 7.996856272718342e-06, 'epoch': 9.6}\n",
      "{'loss': 4.7736, 'learning_rate': 7.977207977207979e-06, 'epoch': 9.62}\n",
      "{'loss': 4.6717, 'learning_rate': 7.957559681697613e-06, 'epoch': 9.63}\n",
      "{'loss': 5.026, 'learning_rate': 7.93791138618725e-06, 'epoch': 9.64}\n",
      "{'loss': 4.6307, 'learning_rate': 7.918263090676885e-06, 'epoch': 9.66}\n",
      "{'loss': 4.7088, 'learning_rate': 7.89861479516652e-06, 'epoch': 9.67}\n",
      "{'loss': 5.0715, 'learning_rate': 7.878966499656156e-06, 'epoch': 9.68}\n",
      "{'loss': 4.734, 'learning_rate': 7.859318204145791e-06, 'epoch': 9.69}\n",
      "{'loss': 4.8459, 'learning_rate': 7.839669908635427e-06, 'epoch': 9.71}\n",
      "{'loss': 4.8715, 'learning_rate': 7.820021613125062e-06, 'epoch': 9.72}\n",
      "{'loss': 4.9838, 'learning_rate': 7.800373317614697e-06, 'epoch': 9.73}\n",
      "{'loss': 4.5037, 'learning_rate': 7.780725022104333e-06, 'epoch': 9.75}\n",
      "{'loss': 4.632, 'learning_rate': 7.76107672659397e-06, 'epoch': 9.76}\n",
      "{'loss': 4.7041, 'learning_rate': 7.741428431083603e-06, 'epoch': 9.77}\n",
      "{'loss': 5.0463, 'learning_rate': 7.721780135573239e-06, 'epoch': 9.79}\n",
      "{'loss': 4.9781, 'learning_rate': 7.702131840062876e-06, 'epoch': 9.8}\n",
      "{'loss': 4.7082, 'learning_rate': 7.68248354455251e-06, 'epoch': 9.81}\n",
      "{'loss': 4.8297, 'learning_rate': 7.662835249042147e-06, 'epoch': 9.83}\n",
      "{'loss': 4.8545, 'learning_rate': 7.643186953531782e-06, 'epoch': 9.84}\n",
      "{'loss': 4.5486, 'learning_rate': 7.6235386580214174e-06, 'epoch': 9.85}\n",
      "{'loss': 5.1656, 'learning_rate': 7.603890362511053e-06, 'epoch': 9.87}\n",
      "{'loss': 5.0557, 'learning_rate': 7.584242067000688e-06, 'epoch': 9.88}\n",
      "{'loss': 4.1846, 'learning_rate': 7.5645937714903236e-06, 'epoch': 9.89}\n",
      "{'loss': 4.5045, 'learning_rate': 7.544945475979959e-06, 'epoch': 9.91}\n",
      "{'loss': 4.9338, 'learning_rate': 7.525297180469595e-06, 'epoch': 9.92}\n",
      "{'loss': 4.5783, 'learning_rate': 7.5056488849592306e-06, 'epoch': 9.93}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-7500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.467, 'learning_rate': 7.486000589448866e-06, 'epoch': 9.95}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-7500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-6500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.907, 'learning_rate': 7.466352293938501e-06, 'epoch': 9.96}\n",
      "{'loss': 4.8244, 'learning_rate': 7.4467039984281376e-06, 'epoch': 9.97}\n",
      "{'loss': 4.6631, 'learning_rate': 7.427055702917773e-06, 'epoch': 9.99}\n",
      "{'loss': 4.3504, 'learning_rate': 7.4074074074074075e-06, 'epoch': 10.0}\n",
      "{'loss': 4.6555, 'learning_rate': 7.387759111897044e-06, 'epoch': 10.01}\n",
      "{'loss': 4.391, 'learning_rate': 7.368110816386679e-06, 'epoch': 10.03}\n",
      "{'loss': 4.8383, 'learning_rate': 7.348462520876315e-06, 'epoch': 10.04}\n",
      "{'loss': 4.658, 'learning_rate': 7.32881422536595e-06, 'epoch': 10.05}\n",
      "{'loss': 4.4602, 'learning_rate': 7.309165929855586e-06, 'epoch': 10.07}\n",
      "{'loss': 4.6344, 'learning_rate': 7.2895176343452215e-06, 'epoch': 10.08}\n",
      "{'loss': 4.6898, 'learning_rate': 7.269869338834856e-06, 'epoch': 10.09}\n",
      "{'loss': 4.9758, 'learning_rate': 7.250221043324492e-06, 'epoch': 10.11}\n",
      "{'loss': 5.1133, 'learning_rate': 7.230572747814128e-06, 'epoch': 10.12}\n",
      "{'loss': 4.6113, 'learning_rate': 7.210924452303764e-06, 'epoch': 10.13}\n",
      "{'loss': 4.998, 'learning_rate': 7.191276156793398e-06, 'epoch': 10.15}\n",
      "{'loss': 4.5092, 'learning_rate': 7.171627861283035e-06, 'epoch': 10.16}\n",
      "{'loss': 4.4771, 'learning_rate': 7.15197956577267e-06, 'epoch': 10.17}\n",
      "{'loss': 4.6934, 'learning_rate': 7.1323312702623045e-06, 'epoch': 10.19}\n",
      "{'loss': 4.6152, 'learning_rate': 7.112682974751941e-06, 'epoch': 10.2}\n",
      "{'loss': 5.2562, 'learning_rate': 7.093034679241576e-06, 'epoch': 10.21}\n",
      "{'loss': 4.818, 'learning_rate': 7.073386383731212e-06, 'epoch': 10.23}\n",
      "{'loss': 5.0082, 'learning_rate': 7.053738088220847e-06, 'epoch': 10.24}\n",
      "{'loss': 4.7738, 'learning_rate': 7.034089792710483e-06, 'epoch': 10.25}\n",
      "{'loss': 4.7949, 'learning_rate': 7.0144414972001185e-06, 'epoch': 10.27}\n",
      "{'loss': 4.6703, 'learning_rate': 6.994793201689754e-06, 'epoch': 10.28}\n",
      "{'loss': 4.9145, 'learning_rate': 6.975144906179389e-06, 'epoch': 10.29}\n",
      "{'loss': 4.849, 'learning_rate': 6.955496610669025e-06, 'epoch': 10.31}\n",
      "{'loss': 4.616, 'learning_rate': 6.935848315158661e-06, 'epoch': 10.32}\n",
      "{'loss': 4.5076, 'learning_rate': 6.916200019648296e-06, 'epoch': 10.33}\n",
      "{'loss': 4.6602, 'learning_rate': 6.896551724137932e-06, 'epoch': 10.34}\n",
      "{'loss': 4.9125, 'learning_rate': 6.876903428627567e-06, 'epoch': 10.36}\n",
      "{'loss': 4.476, 'learning_rate': 6.857255133117202e-06, 'epoch': 10.37}\n",
      "{'loss': 4.8488, 'learning_rate': 6.837606837606839e-06, 'epoch': 10.38}\n",
      "{'loss': 4.9008, 'learning_rate': 6.817958542096473e-06, 'epoch': 10.4}\n",
      "{'loss': 4.9648, 'learning_rate': 6.798310246586109e-06, 'epoch': 10.41}\n",
      "{'loss': 4.8875, 'learning_rate': 6.778661951075745e-06, 'epoch': 10.42}\n",
      "{'loss': 5.0723, 'learning_rate': 6.759013655565381e-06, 'epoch': 10.44}\n",
      "{'loss': 4.5879, 'learning_rate': 6.7393653600550156e-06, 'epoch': 10.45}\n",
      "{'loss': 4.6875, 'learning_rate': 6.719717064544651e-06, 'epoch': 10.46}\n",
      "{'loss': 4.8041, 'learning_rate': 6.700068769034287e-06, 'epoch': 10.48}\n",
      "{'loss': 4.75, 'learning_rate': 6.680420473523922e-06, 'epoch': 10.49}\n",
      "{'loss': 4.7367, 'learning_rate': 6.660772178013558e-06, 'epoch': 10.5}\n",
      "{'loss': 4.7869, 'learning_rate': 6.641123882503193e-06, 'epoch': 10.52}\n",
      "{'loss': 4.1906, 'learning_rate': 6.6214755869928296e-06, 'epoch': 10.53}\n",
      "{'loss': 4.8158, 'learning_rate': 6.601827291482464e-06, 'epoch': 10.54}\n",
      "{'loss': 4.6117, 'learning_rate': 6.5821789959720995e-06, 'epoch': 10.56}\n",
      "{'loss': 4.5014, 'learning_rate': 6.562530700461736e-06, 'epoch': 10.57}\n",
      "{'loss': 4.6863, 'learning_rate': 6.54288240495137e-06, 'epoch': 10.58}\n",
      "{'loss': 4.8105, 'learning_rate': 6.5232341094410065e-06, 'epoch': 10.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-8000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.5818, 'learning_rate': 6.503585813930642e-06, 'epoch': 10.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-8000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-7000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6629, 'learning_rate': 6.483937518420278e-06, 'epoch': 10.62}\n",
      "{'loss': 4.6846, 'learning_rate': 6.464289222909913e-06, 'epoch': 10.64}\n",
      "{'loss': 4.6904, 'learning_rate': 6.444640927399549e-06, 'epoch': 10.65}\n",
      "{'loss': 4.9, 'learning_rate': 6.424992631889184e-06, 'epoch': 10.66}\n",
      "{'loss': 4.7863, 'learning_rate': 6.40534433637882e-06, 'epoch': 10.68}\n",
      "{'loss': 4.9039, 'learning_rate': 6.385696040868455e-06, 'epoch': 10.69}\n",
      "{'loss': 4.8822, 'learning_rate': 6.36604774535809e-06, 'epoch': 10.7}\n",
      "{'loss': 4.8254, 'learning_rate': 6.346399449847727e-06, 'epoch': 10.72}\n",
      "{'loss': 4.4928, 'learning_rate': 6.326751154337362e-06, 'epoch': 10.73}\n",
      "{'loss': 4.5971, 'learning_rate': 6.307102858826998e-06, 'epoch': 10.74}\n",
      "{'loss': 4.666, 'learning_rate': 6.287454563316633e-06, 'epoch': 10.76}\n",
      "{'loss': 4.6531, 'learning_rate': 6.267806267806268e-06, 'epoch': 10.77}\n",
      "{'loss': 4.5289, 'learning_rate': 6.248157972295904e-06, 'epoch': 10.78}\n",
      "{'loss': 4.773, 'learning_rate': 6.228509676785539e-06, 'epoch': 10.8}\n",
      "{'loss': 4.7736, 'learning_rate': 6.208861381275175e-06, 'epoch': 10.81}\n",
      "{'loss': 4.498, 'learning_rate': 6.1892130857648105e-06, 'epoch': 10.82}\n",
      "{'loss': 4.958, 'learning_rate': 6.169564790254447e-06, 'epoch': 10.84}\n",
      "{'loss': 5.1008, 'learning_rate': 6.149916494744081e-06, 'epoch': 10.85}\n",
      "{'loss': 5.0246, 'learning_rate': 6.130268199233717e-06, 'epoch': 10.86}\n",
      "{'loss': 4.8525, 'learning_rate': 6.110619903723353e-06, 'epoch': 10.88}\n",
      "{'loss': 5.1098, 'learning_rate': 6.090971608212987e-06, 'epoch': 10.89}\n",
      "{'loss': 4.6508, 'learning_rate': 6.071323312702624e-06, 'epoch': 10.9}\n",
      "{'loss': 4.3114, 'learning_rate': 6.051675017192259e-06, 'epoch': 10.92}\n",
      "{'loss': 4.7098, 'learning_rate': 6.032026721681895e-06, 'epoch': 10.93}\n",
      "{'loss': 5.1096, 'learning_rate': 6.01237842617153e-06, 'epoch': 10.94}\n",
      "{'loss': 5.0016, 'learning_rate': 5.992730130661165e-06, 'epoch': 10.95}\n",
      "{'loss': 4.5707, 'learning_rate': 5.973081835150801e-06, 'epoch': 10.97}\n",
      "{'loss': 4.8252, 'learning_rate': 5.953433539640436e-06, 'epoch': 10.98}\n",
      "{'loss': 4.9414, 'learning_rate': 5.933785244130072e-06, 'epoch': 10.99}\n",
      "{'loss': 4.3871, 'learning_rate': 5.9141369486197076e-06, 'epoch': 11.01}\n",
      "{'loss': 4.4271, 'learning_rate': 5.894488653109344e-06, 'epoch': 11.02}\n",
      "{'loss': 4.6041, 'learning_rate': 5.874840357598979e-06, 'epoch': 11.03}\n",
      "{'loss': 4.2844, 'learning_rate': 5.855192062088614e-06, 'epoch': 11.05}\n",
      "{'loss': 4.6711, 'learning_rate': 5.83554376657825e-06, 'epoch': 11.06}\n",
      "{'loss': 5.1063, 'learning_rate': 5.815895471067885e-06, 'epoch': 11.07}\n",
      "{'loss': 4.8918, 'learning_rate': 5.7962471755575215e-06, 'epoch': 11.09}\n",
      "{'loss': 4.957, 'learning_rate': 5.776598880047156e-06, 'epoch': 11.1}\n",
      "{'loss': 4.9047, 'learning_rate': 5.756950584536792e-06, 'epoch': 11.11}\n",
      "{'loss': 4.7879, 'learning_rate': 5.737302289026428e-06, 'epoch': 11.13}\n",
      "{'loss': 4.6703, 'learning_rate': 5.717653993516062e-06, 'epoch': 11.14}\n",
      "{'loss': 4.6182, 'learning_rate': 5.6980056980056985e-06, 'epoch': 11.15}\n",
      "{'loss': 4.7396, 'learning_rate': 5.678357402495334e-06, 'epoch': 11.17}\n",
      "{'loss': 4.8791, 'learning_rate': 5.65870910698497e-06, 'epoch': 11.18}\n",
      "{'loss': 4.8406, 'learning_rate': 5.639060811474605e-06, 'epoch': 11.19}\n",
      "{'loss': 4.9336, 'learning_rate': 5.619412515964241e-06, 'epoch': 11.21}\n",
      "{'loss': 4.5523, 'learning_rate': 5.599764220453876e-06, 'epoch': 11.22}\n",
      "{'loss': 4.6852, 'learning_rate': 5.580115924943511e-06, 'epoch': 11.23}\n",
      "{'loss': 4.5814, 'learning_rate': 5.560467629433147e-06, 'epoch': 11.25}\n",
      "{'loss': 4.5627, 'learning_rate': 5.540819333922782e-06, 'epoch': 11.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-8500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8518, 'learning_rate': 5.521171038412419e-06, 'epoch': 11.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-8500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-7500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.3998, 'learning_rate': 5.501522742902053e-06, 'epoch': 11.29}\n",
      "{'loss': 4.9838, 'learning_rate': 5.481874447391689e-06, 'epoch': 11.3}\n",
      "{'loss': 5.1437, 'learning_rate': 5.462226151881325e-06, 'epoch': 11.31}\n",
      "{'loss': 5.1496, 'learning_rate': 5.442577856370961e-06, 'epoch': 11.33}\n",
      "{'loss': 4.7799, 'learning_rate': 5.4229295608605955e-06, 'epoch': 11.34}\n",
      "{'loss': 4.5578, 'learning_rate': 5.403281265350231e-06, 'epoch': 11.35}\n",
      "{'loss': 5.025, 'learning_rate': 5.383632969839867e-06, 'epoch': 11.37}\n",
      "{'loss': 4.5498, 'learning_rate': 5.3639846743295025e-06, 'epoch': 11.38}\n",
      "{'loss': 4.5252, 'learning_rate': 5.344336378819138e-06, 'epoch': 11.39}\n",
      "{'loss': 4.8492, 'learning_rate': 5.324688083308773e-06, 'epoch': 11.41}\n",
      "{'loss': 4.801, 'learning_rate': 5.3050397877984095e-06, 'epoch': 11.42}\n",
      "{'loss': 4.7518, 'learning_rate': 5.285391492288045e-06, 'epoch': 11.43}\n",
      "{'loss': 4.3689, 'learning_rate': 5.265743196777679e-06, 'epoch': 11.45}\n",
      "{'loss': 4.6994, 'learning_rate': 5.246094901267316e-06, 'epoch': 11.46}\n",
      "{'loss': 4.7781, 'learning_rate': 5.226446605756951e-06, 'epoch': 11.47}\n",
      "{'loss': 4.64, 'learning_rate': 5.206798310246587e-06, 'epoch': 11.49}\n",
      "{'loss': 4.4799, 'learning_rate': 5.187150014736222e-06, 'epoch': 11.5}\n",
      "{'loss': 4.6969, 'learning_rate': 5.167501719225858e-06, 'epoch': 11.51}\n",
      "{'loss': 4.5654, 'learning_rate': 5.147853423715493e-06, 'epoch': 11.53}\n",
      "{'loss': 5.0508, 'learning_rate': 5.128205128205128e-06, 'epoch': 11.54}\n",
      "{'loss': 4.6922, 'learning_rate': 5.108556832694764e-06, 'epoch': 11.55}\n",
      "{'loss': 4.3748, 'learning_rate': 5.0889085371843995e-06, 'epoch': 11.56}\n",
      "{'loss': 4.8033, 'learning_rate': 5.069260241674036e-06, 'epoch': 11.58}\n",
      "{'loss': 4.5359, 'learning_rate': 5.04961194616367e-06, 'epoch': 11.59}\n",
      "{'loss': 4.9285, 'learning_rate': 5.0299636506533065e-06, 'epoch': 11.6}\n",
      "{'loss': 4.5506, 'learning_rate': 5.010315355142942e-06, 'epoch': 11.62}\n",
      "{'loss': 5.0516, 'learning_rate': 4.990667059632577e-06, 'epoch': 11.63}\n",
      "{'loss': 4.541, 'learning_rate': 4.971018764122213e-06, 'epoch': 11.64}\n",
      "{'loss': 4.6443, 'learning_rate': 4.951370468611849e-06, 'epoch': 11.66}\n",
      "{'loss': 4.7375, 'learning_rate': 4.9317221731014834e-06, 'epoch': 11.67}\n",
      "{'loss': 5.0246, 'learning_rate': 4.912073877591119e-06, 'epoch': 11.68}\n",
      "{'loss': 4.2678, 'learning_rate': 4.892425582080755e-06, 'epoch': 11.7}\n",
      "{'loss': 4.9277, 'learning_rate': 4.8727772865703904e-06, 'epoch': 11.71}\n",
      "{'loss': 4.7699, 'learning_rate': 4.853128991060026e-06, 'epoch': 11.72}\n",
      "{'loss': 4.3246, 'learning_rate': 4.833480695549661e-06, 'epoch': 11.74}\n",
      "{'loss': 4.2695, 'learning_rate': 4.8138324000392974e-06, 'epoch': 11.75}\n",
      "{'loss': 4.5889, 'learning_rate': 4.794184104528932e-06, 'epoch': 11.76}\n",
      "{'loss': 4.817, 'learning_rate': 4.774535809018568e-06, 'epoch': 11.78}\n",
      "{'loss': 4.8758, 'learning_rate': 4.754887513508204e-06, 'epoch': 11.79}\n",
      "{'loss': 4.3109, 'learning_rate': 4.735239217997839e-06, 'epoch': 11.8}\n",
      "{'loss': 4.777, 'learning_rate': 4.715590922487474e-06, 'epoch': 11.82}\n",
      "{'loss': 4.76, 'learning_rate': 4.695942626977111e-06, 'epoch': 11.83}\n",
      "{'loss': 4.8996, 'learning_rate': 4.676294331466746e-06, 'epoch': 11.84}\n",
      "{'loss': 5.0512, 'learning_rate': 4.656646035956381e-06, 'epoch': 11.86}\n",
      "{'loss': 4.8088, 'learning_rate': 4.636997740446017e-06, 'epoch': 11.87}\n",
      "{'loss': 4.4793, 'learning_rate': 4.617349444935652e-06, 'epoch': 11.88}\n",
      "{'loss': 4.3666, 'learning_rate': 4.5977011494252875e-06, 'epoch': 11.9}\n",
      "{'loss': 4.5127, 'learning_rate': 4.578052853914923e-06, 'epoch': 11.91}\n",
      "{'loss': 4.5906, 'learning_rate': 4.558404558404559e-06, 'epoch': 11.92}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-9000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.7156, 'learning_rate': 4.5387562628941945e-06, 'epoch': 11.94}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-9000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-8000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.3924, 'learning_rate': 4.51910796738383e-06, 'epoch': 11.95}\n",
      "{'loss': 4.4813, 'learning_rate': 4.499459671873465e-06, 'epoch': 11.96}\n",
      "{'loss': 4.7814, 'learning_rate': 4.479811376363101e-06, 'epoch': 11.98}\n",
      "{'loss': 4.0447, 'learning_rate': 4.460163080852736e-06, 'epoch': 11.99}\n",
      "{'loss': 4.9271, 'learning_rate': 4.440514785342372e-06, 'epoch': 12.0}\n",
      "{'loss': 4.9385, 'learning_rate': 4.420866489832008e-06, 'epoch': 12.02}\n",
      "{'loss': 4.6615, 'learning_rate': 4.401218194321643e-06, 'epoch': 12.03}\n",
      "{'loss': 4.6324, 'learning_rate': 4.381569898811278e-06, 'epoch': 12.04}\n",
      "{'loss': 4.8871, 'learning_rate': 4.361921603300914e-06, 'epoch': 12.06}\n",
      "{'loss': 4.4109, 'learning_rate': 4.342273307790549e-06, 'epoch': 12.07}\n",
      "{'loss': 4.7875, 'learning_rate': 4.3226250122801845e-06, 'epoch': 12.08}\n",
      "{'loss': 4.4365, 'learning_rate': 4.302976716769821e-06, 'epoch': 12.1}\n",
      "{'loss': 5.0094, 'learning_rate': 4.283328421259456e-06, 'epoch': 12.11}\n",
      "{'loss': 4.4807, 'learning_rate': 4.2636801257490915e-06, 'epoch': 12.12}\n",
      "{'loss': 4.9082, 'learning_rate': 4.244031830238727e-06, 'epoch': 12.14}\n",
      "{'loss': 4.7365, 'learning_rate': 4.224383534728363e-06, 'epoch': 12.15}\n",
      "{'loss': 4.6984, 'learning_rate': 4.204735239217998e-06, 'epoch': 12.16}\n",
      "{'loss': 4.6811, 'learning_rate': 4.185086943707634e-06, 'epoch': 12.18}\n",
      "{'loss': 4.6877, 'learning_rate': 4.165438648197269e-06, 'epoch': 12.19}\n",
      "{'loss': 4.4494, 'learning_rate': 4.145790352686905e-06, 'epoch': 12.2}\n",
      "{'loss': 4.5098, 'learning_rate': 4.12614205717654e-06, 'epoch': 12.21}\n",
      "{'loss': 4.9203, 'learning_rate': 4.106493761666176e-06, 'epoch': 12.23}\n",
      "{'loss': 4.8471, 'learning_rate': 4.086845466155812e-06, 'epoch': 12.24}\n",
      "{'loss': 4.6658, 'learning_rate': 4.067197170645446e-06, 'epoch': 12.25}\n",
      "{'loss': 4.9691, 'learning_rate': 4.0475488751350824e-06, 'epoch': 12.27}\n",
      "{'loss': 4.8871, 'learning_rate': 4.027900579624718e-06, 'epoch': 12.28}\n",
      "{'loss': 4.8412, 'learning_rate': 4.008252284114353e-06, 'epoch': 12.29}\n",
      "{'loss': 4.7738, 'learning_rate': 3.9886039886039894e-06, 'epoch': 12.31}\n",
      "{'loss': 4.6104, 'learning_rate': 3.968955693093625e-06, 'epoch': 12.32}\n",
      "{'loss': 4.5047, 'learning_rate': 3.94930739758326e-06, 'epoch': 12.33}\n",
      "{'loss': 4.8262, 'learning_rate': 3.9296591020728956e-06, 'epoch': 12.35}\n",
      "{'loss': 4.6668, 'learning_rate': 3.910010806562531e-06, 'epoch': 12.36}\n",
      "{'loss': 4.5795, 'learning_rate': 3.890362511052166e-06, 'epoch': 12.37}\n",
      "{'loss': 4.5744, 'learning_rate': 3.870714215541802e-06, 'epoch': 12.39}\n",
      "{'loss': 4.4951, 'learning_rate': 3.851065920031438e-06, 'epoch': 12.4}\n",
      "{'loss': 4.8598, 'learning_rate': 3.831417624521073e-06, 'epoch': 12.41}\n",
      "{'loss': 5.018, 'learning_rate': 3.8117693290107087e-06, 'epoch': 12.43}\n",
      "{'loss': 4.7203, 'learning_rate': 3.792121033500344e-06, 'epoch': 12.44}\n",
      "{'loss': 4.5434, 'learning_rate': 3.7724727379899795e-06, 'epoch': 12.45}\n",
      "{'loss': 4.5559, 'learning_rate': 3.7528244424796153e-06, 'epoch': 12.47}\n",
      "{'loss': 4.8156, 'learning_rate': 3.7331761469692507e-06, 'epoch': 12.48}\n",
      "{'loss': 4.3469, 'learning_rate': 3.7135278514588865e-06, 'epoch': 12.49}\n",
      "{'loss': 4.5955, 'learning_rate': 3.693879555948522e-06, 'epoch': 12.51}\n",
      "{'loss': 4.95, 'learning_rate': 3.6742312604381577e-06, 'epoch': 12.52}\n",
      "{'loss': 4.7221, 'learning_rate': 3.654582964927793e-06, 'epoch': 12.53}\n",
      "{'loss': 4.8312, 'learning_rate': 3.634934669417428e-06, 'epoch': 12.55}\n",
      "{'loss': 4.7588, 'learning_rate': 3.615286373907064e-06, 'epoch': 12.56}\n",
      "{'loss': 4.7328, 'learning_rate': 3.595638078396699e-06, 'epoch': 12.57}\n",
      "{'loss': 5.1285, 'learning_rate': 3.575989782886335e-06, 'epoch': 12.59}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-9500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.7701, 'learning_rate': 3.5563414873759704e-06, 'epoch': 12.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-9500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-8500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.5652, 'learning_rate': 3.536693191865606e-06, 'epoch': 12.61}\n",
      "{'loss': 4.1226, 'learning_rate': 3.5170448963552416e-06, 'epoch': 12.63}\n",
      "{'loss': 5.2922, 'learning_rate': 3.497396600844877e-06, 'epoch': 12.64}\n",
      "{'loss': 4.8582, 'learning_rate': 3.4777483053345123e-06, 'epoch': 12.65}\n",
      "{'loss': 4.5152, 'learning_rate': 3.458100009824148e-06, 'epoch': 12.67}\n",
      "{'loss': 4.5451, 'learning_rate': 3.4384517143137835e-06, 'epoch': 12.68}\n",
      "{'loss': 4.5436, 'learning_rate': 3.4188034188034193e-06, 'epoch': 12.69}\n",
      "{'loss': 4.49, 'learning_rate': 3.3991551232930547e-06, 'epoch': 12.71}\n",
      "{'loss': 4.6365, 'learning_rate': 3.3795068277826905e-06, 'epoch': 12.72}\n",
      "{'loss': 4.2344, 'learning_rate': 3.3598585322723255e-06, 'epoch': 12.73}\n",
      "{'loss': 4.8107, 'learning_rate': 3.340210236761961e-06, 'epoch': 12.75}\n",
      "{'loss': 4.7383, 'learning_rate': 3.3205619412515967e-06, 'epoch': 12.76}\n",
      "{'loss': 4.8027, 'learning_rate': 3.300913645741232e-06, 'epoch': 12.77}\n",
      "{'loss': 4.7828, 'learning_rate': 3.281265350230868e-06, 'epoch': 12.79}\n",
      "{'loss': 5.0539, 'learning_rate': 3.2616170547205032e-06, 'epoch': 12.8}\n",
      "{'loss': 4.7893, 'learning_rate': 3.241968759210139e-06, 'epoch': 12.81}\n",
      "{'loss': 4.9172, 'learning_rate': 3.2223204636997744e-06, 'epoch': 12.82}\n",
      "{'loss': 4.6916, 'learning_rate': 3.20267216818941e-06, 'epoch': 12.84}\n",
      "{'loss': 4.1947, 'learning_rate': 3.183023872679045e-06, 'epoch': 12.85}\n",
      "{'loss': 4.0974, 'learning_rate': 3.163375577168681e-06, 'epoch': 12.86}\n",
      "{'loss': 4.7102, 'learning_rate': 3.1437272816583164e-06, 'epoch': 12.88}\n",
      "{'loss': 4.9531, 'learning_rate': 3.124078986147952e-06, 'epoch': 12.89}\n",
      "{'loss': 4.5422, 'learning_rate': 3.1044306906375876e-06, 'epoch': 12.9}\n",
      "{'loss': 4.5023, 'learning_rate': 3.0847823951272234e-06, 'epoch': 12.92}\n",
      "{'loss': 4.1824, 'learning_rate': 3.0651340996168583e-06, 'epoch': 12.93}\n",
      "{'loss': 4.798, 'learning_rate': 3.0454858041064937e-06, 'epoch': 12.94}\n",
      "{'loss': 4.7871, 'learning_rate': 3.0258375085961295e-06, 'epoch': 12.96}\n",
      "{'loss': 4.9781, 'learning_rate': 3.006189213085765e-06, 'epoch': 12.97}\n",
      "{'loss': 4.8766, 'learning_rate': 2.9865409175754007e-06, 'epoch': 12.98}\n",
      "{'loss': 4.0732, 'learning_rate': 2.966892622065036e-06, 'epoch': 13.0}\n",
      "{'loss': 4.7973, 'learning_rate': 2.947244326554672e-06, 'epoch': 13.01}\n",
      "{'loss': 4.5652, 'learning_rate': 2.927596031044307e-06, 'epoch': 13.02}\n",
      "{'loss': 4.3363, 'learning_rate': 2.9079477355339427e-06, 'epoch': 13.04}\n",
      "{'loss': 4.3941, 'learning_rate': 2.888299440023578e-06, 'epoch': 13.05}\n",
      "{'loss': 4.6119, 'learning_rate': 2.868651144513214e-06, 'epoch': 13.06}\n",
      "{'loss': 4.6354, 'learning_rate': 2.8490028490028492e-06, 'epoch': 13.08}\n",
      "{'loss': 4.7514, 'learning_rate': 2.829354553492485e-06, 'epoch': 13.09}\n",
      "{'loss': 4.6852, 'learning_rate': 2.8097062579821204e-06, 'epoch': 13.1}\n",
      "{'loss': 4.8594, 'learning_rate': 2.7900579624717554e-06, 'epoch': 13.12}\n",
      "{'loss': 4.627, 'learning_rate': 2.770409666961391e-06, 'epoch': 13.13}\n",
      "{'loss': 4.5396, 'learning_rate': 2.7507613714510266e-06, 'epoch': 13.14}\n",
      "{'loss': 4.9391, 'learning_rate': 2.7311130759406624e-06, 'epoch': 13.16}\n",
      "{'loss': 4.7959, 'learning_rate': 2.7114647804302977e-06, 'epoch': 13.17}\n",
      "{'loss': 4.5439, 'learning_rate': 2.6918164849199336e-06, 'epoch': 13.18}\n",
      "{'loss': 4.81, 'learning_rate': 2.672168189409569e-06, 'epoch': 13.2}\n",
      "{'loss': 4.5945, 'learning_rate': 2.6525198938992047e-06, 'epoch': 13.21}\n",
      "{'loss': 4.5592, 'learning_rate': 2.6328715983888397e-06, 'epoch': 13.22}\n",
      "{'loss': 4.6145, 'learning_rate': 2.6132233028784755e-06, 'epoch': 13.24}\n",
      "{'loss': 4.7641, 'learning_rate': 2.593575007368111e-06, 'epoch': 13.25}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-10000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.2078, 'learning_rate': 2.5739267118577467e-06, 'epoch': 13.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-10000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-9000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6094, 'learning_rate': 2.554278416347382e-06, 'epoch': 13.28}\n",
      "{'loss': 4.459, 'learning_rate': 2.534630120837018e-06, 'epoch': 13.29}\n",
      "{'loss': 4.6016, 'learning_rate': 2.5149818253266533e-06, 'epoch': 13.3}\n",
      "{'loss': 4.5135, 'learning_rate': 2.4953335298162887e-06, 'epoch': 13.32}\n",
      "{'loss': 4.934, 'learning_rate': 2.4756852343059245e-06, 'epoch': 13.33}\n",
      "{'loss': 4.8123, 'learning_rate': 2.4560369387955594e-06, 'epoch': 13.34}\n",
      "{'loss': 4.6449, 'learning_rate': 2.4363886432851952e-06, 'epoch': 13.36}\n",
      "{'loss': 4.8328, 'learning_rate': 2.4167403477748306e-06, 'epoch': 13.37}\n",
      "{'loss': 5.2373, 'learning_rate': 2.397092052264466e-06, 'epoch': 13.38}\n",
      "{'loss': 4.4238, 'learning_rate': 2.377443756754102e-06, 'epoch': 13.4}\n",
      "{'loss': 4.3738, 'learning_rate': 2.357795461243737e-06, 'epoch': 13.41}\n",
      "{'loss': 4.9051, 'learning_rate': 2.338147165733373e-06, 'epoch': 13.42}\n",
      "{'loss': 4.5969, 'learning_rate': 2.3184988702230084e-06, 'epoch': 13.44}\n",
      "{'loss': 4.9535, 'learning_rate': 2.2988505747126437e-06, 'epoch': 13.45}\n",
      "{'loss': 4.7777, 'learning_rate': 2.2792022792022796e-06, 'epoch': 13.46}\n",
      "{'loss': 4.7096, 'learning_rate': 2.259553983691915e-06, 'epoch': 13.47}\n",
      "{'loss': 4.6912, 'learning_rate': 2.2399056881815503e-06, 'epoch': 13.49}\n",
      "{'loss': 4.9309, 'learning_rate': 2.220257392671186e-06, 'epoch': 13.5}\n",
      "{'loss': 4.3656, 'learning_rate': 2.2006090971608215e-06, 'epoch': 13.51}\n",
      "{'loss': 4.717, 'learning_rate': 2.180960801650457e-06, 'epoch': 13.53}\n",
      "{'loss': 4.7494, 'learning_rate': 2.1613125061400923e-06, 'epoch': 13.54}\n",
      "{'loss': 4.5676, 'learning_rate': 2.141664210629728e-06, 'epoch': 13.55}\n",
      "{'loss': 4.8887, 'learning_rate': 2.1220159151193635e-06, 'epoch': 13.57}\n",
      "{'loss': 4.6846, 'learning_rate': 2.102367619608999e-06, 'epoch': 13.58}\n",
      "{'loss': 4.5936, 'learning_rate': 2.0827193240986346e-06, 'epoch': 13.59}\n",
      "{'loss': 4.2211, 'learning_rate': 2.06307102858827e-06, 'epoch': 13.61}\n",
      "{'loss': 4.9566, 'learning_rate': 2.043422733077906e-06, 'epoch': 13.62}\n",
      "{'loss': 5.1395, 'learning_rate': 2.0237744375675412e-06, 'epoch': 13.63}\n",
      "{'loss': 4.7539, 'learning_rate': 2.0041261420571766e-06, 'epoch': 13.65}\n",
      "{'loss': 4.6375, 'learning_rate': 1.9844778465468124e-06, 'epoch': 13.66}\n",
      "{'loss': 4.407, 'learning_rate': 1.9648295510364478e-06, 'epoch': 13.67}\n",
      "{'loss': 4.9535, 'learning_rate': 1.945181255526083e-06, 'epoch': 13.69}\n",
      "{'loss': 4.7832, 'learning_rate': 1.925532960015719e-06, 'epoch': 13.7}\n",
      "{'loss': 4.6961, 'learning_rate': 1.9058846645053544e-06, 'epoch': 13.71}\n",
      "{'loss': 4.71, 'learning_rate': 1.8862363689949897e-06, 'epoch': 13.73}\n",
      "{'loss': 4.4842, 'learning_rate': 1.8665880734846253e-06, 'epoch': 13.74}\n",
      "{'loss': 4.6199, 'learning_rate': 1.846939777974261e-06, 'epoch': 13.75}\n",
      "{'loss': 4.7094, 'learning_rate': 1.8272914824638965e-06, 'epoch': 13.77}\n",
      "{'loss': 4.8965, 'learning_rate': 1.807643186953532e-06, 'epoch': 13.78}\n",
      "{'loss': 4.5262, 'learning_rate': 1.7879948914431675e-06, 'epoch': 13.79}\n",
      "{'loss': 4.9291, 'learning_rate': 1.768346595932803e-06, 'epoch': 13.81}\n",
      "{'loss': 4.2689, 'learning_rate': 1.7486983004224385e-06, 'epoch': 13.82}\n",
      "{'loss': 4.4342, 'learning_rate': 1.729050004912074e-06, 'epoch': 13.83}\n",
      "{'loss': 4.5057, 'learning_rate': 1.7094017094017097e-06, 'epoch': 13.85}\n",
      "{'loss': 4.8416, 'learning_rate': 1.6897534138913453e-06, 'epoch': 13.86}\n",
      "{'loss': 4.4277, 'learning_rate': 1.6701051183809804e-06, 'epoch': 13.87}\n",
      "{'loss': 4.6297, 'learning_rate': 1.650456822870616e-06, 'epoch': 13.89}\n",
      "{'loss': 4.6414, 'learning_rate': 1.6308085273602516e-06, 'epoch': 13.9}\n",
      "{'loss': 4.8504, 'learning_rate': 1.6111602318498872e-06, 'epoch': 13.91}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-10500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8109, 'learning_rate': 1.5915119363395226e-06, 'epoch': 13.93}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-10500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-9500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9775, 'learning_rate': 1.5718636408291582e-06, 'epoch': 13.94}\n",
      "{'loss': 4.5158, 'learning_rate': 1.5522153453187938e-06, 'epoch': 13.95}\n",
      "{'loss': 4.8225, 'learning_rate': 1.5325670498084292e-06, 'epoch': 13.97}\n",
      "{'loss': 5.0773, 'learning_rate': 1.5129187542980648e-06, 'epoch': 13.98}\n",
      "{'loss': 4.3811, 'learning_rate': 1.4932704587877004e-06, 'epoch': 13.99}\n",
      "{'loss': 4.7947, 'learning_rate': 1.473622163277336e-06, 'epoch': 14.01}\n",
      "{'loss': 4.8762, 'learning_rate': 1.4539738677669713e-06, 'epoch': 14.02}\n",
      "{'loss': 4.607, 'learning_rate': 1.434325572256607e-06, 'epoch': 14.03}\n",
      "{'loss': 4.6545, 'learning_rate': 1.4146772767462425e-06, 'epoch': 14.05}\n",
      "{'loss': 4.2311, 'learning_rate': 1.3950289812358777e-06, 'epoch': 14.06}\n",
      "{'loss': 4.7617, 'learning_rate': 1.3753806857255133e-06, 'epoch': 14.07}\n",
      "{'loss': 4.4838, 'learning_rate': 1.3557323902151489e-06, 'epoch': 14.08}\n",
      "{'loss': 5.0535, 'learning_rate': 1.3360840947047845e-06, 'epoch': 14.1}\n",
      "{'loss': 4.5631, 'learning_rate': 1.3164357991944199e-06, 'epoch': 14.11}\n",
      "{'loss': 5.1043, 'learning_rate': 1.2967875036840554e-06, 'epoch': 14.12}\n",
      "{'loss': 4.8609, 'learning_rate': 1.277139208173691e-06, 'epoch': 14.14}\n",
      "{'loss': 4.6559, 'learning_rate': 1.2574909126633266e-06, 'epoch': 14.15}\n",
      "{'loss': 4.4375, 'learning_rate': 1.2378426171529622e-06, 'epoch': 14.16}\n",
      "{'loss': 4.9238, 'learning_rate': 1.2181943216425976e-06, 'epoch': 14.18}\n",
      "{'loss': 4.6576, 'learning_rate': 1.198546026132233e-06, 'epoch': 14.19}\n",
      "{'loss': 4.799, 'learning_rate': 1.1788977306218686e-06, 'epoch': 14.2}\n",
      "{'loss': 4.9176, 'learning_rate': 1.1592494351115042e-06, 'epoch': 14.22}\n",
      "{'loss': 4.9281, 'learning_rate': 1.1396011396011398e-06, 'epoch': 14.23}\n",
      "{'loss': 4.5934, 'learning_rate': 1.1199528440907752e-06, 'epoch': 14.24}\n",
      "{'loss': 4.6754, 'learning_rate': 1.1003045485804108e-06, 'epoch': 14.26}\n",
      "{'loss': 4.8906, 'learning_rate': 1.0806562530700461e-06, 'epoch': 14.27}\n",
      "{'loss': 4.4975, 'learning_rate': 1.0610079575596817e-06, 'epoch': 14.28}\n",
      "{'loss': 4.7395, 'learning_rate': 1.0413596620493173e-06, 'epoch': 14.3}\n",
      "{'loss': 4.5082, 'learning_rate': 1.021711366538953e-06, 'epoch': 14.31}\n",
      "{'loss': 4.5492, 'learning_rate': 1.0020630710285883e-06, 'epoch': 14.32}\n",
      "{'loss': 4.4283, 'learning_rate': 9.824147755182239e-07, 'epoch': 14.34}\n",
      "{'loss': 4.1471, 'learning_rate': 9.627664800078595e-07, 'epoch': 14.35}\n",
      "{'loss': 4.9324, 'learning_rate': 9.431181844974949e-07, 'epoch': 14.36}\n",
      "{'loss': 4.8359, 'learning_rate': 9.234698889871305e-07, 'epoch': 14.38}\n",
      "{'loss': 4.615, 'learning_rate': 9.03821593476766e-07, 'epoch': 14.39}\n",
      "{'loss': 4.4355, 'learning_rate': 8.841732979664015e-07, 'epoch': 14.4}\n",
      "{'loss': 4.8904, 'learning_rate': 8.64525002456037e-07, 'epoch': 14.42}\n",
      "{'loss': 5.1262, 'learning_rate': 8.448767069456726e-07, 'epoch': 14.43}\n",
      "{'loss': 4.6012, 'learning_rate': 8.25228411435308e-07, 'epoch': 14.44}\n",
      "{'loss': 4.634, 'learning_rate': 8.055801159249436e-07, 'epoch': 14.46}\n",
      "{'loss': 4.8293, 'learning_rate': 7.859318204145791e-07, 'epoch': 14.47}\n",
      "{'loss': 5.0354, 'learning_rate': 7.662835249042146e-07, 'epoch': 14.48}\n",
      "{'loss': 3.908, 'learning_rate': 7.466352293938502e-07, 'epoch': 14.5}\n",
      "{'loss': 4.3689, 'learning_rate': 7.269869338834857e-07, 'epoch': 14.51}\n",
      "{'loss': 4.4734, 'learning_rate': 7.073386383731213e-07, 'epoch': 14.52}\n",
      "{'loss': 4.598, 'learning_rate': 6.876903428627566e-07, 'epoch': 14.54}\n",
      "{'loss': 4.65, 'learning_rate': 6.680420473523922e-07, 'epoch': 14.55}\n",
      "{'loss': 4.442, 'learning_rate': 6.483937518420277e-07, 'epoch': 14.56}\n",
      "{'loss': 4.4799, 'learning_rate': 6.287454563316633e-07, 'epoch': 14.58}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby2/checkpoint-11000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6174, 'learning_rate': 6.090971608212988e-07, 'epoch': 14.59}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby2/checkpoint-11000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby2/checkpoint-10000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.2498, 'learning_rate': 5.894488653109343e-07, 'epoch': 14.6}\n",
      "{'loss': 4.518, 'learning_rate': 5.698005698005699e-07, 'epoch': 14.62}\n",
      "{'loss': 4.6605, 'learning_rate': 5.501522742902054e-07, 'epoch': 14.63}\n",
      "{'loss': 4.6223, 'learning_rate': 5.305039787798409e-07, 'epoch': 14.64}\n",
      "{'loss': 4.7449, 'learning_rate': 5.108556832694765e-07, 'epoch': 14.66}\n",
      "{'loss': 4.7162, 'learning_rate': 4.912073877591119e-07, 'epoch': 14.67}\n",
      "{'loss': 4.823, 'learning_rate': 4.7155909224874743e-07, 'epoch': 14.68}\n",
      "{'loss': 4.9621, 'learning_rate': 4.51910796738383e-07, 'epoch': 14.69}\n",
      "{'loss': 4.8561, 'learning_rate': 4.322625012280185e-07, 'epoch': 14.71}\n",
      "{'loss': 4.4756, 'learning_rate': 4.12614205717654e-07, 'epoch': 14.72}\n",
      "{'loss': 4.758, 'learning_rate': 3.9296591020728955e-07, 'epoch': 14.73}\n",
      "{'loss': 4.5178, 'learning_rate': 3.733176146969251e-07, 'epoch': 14.75}\n",
      "{'loss': 4.4219, 'learning_rate': 3.5366931918656063e-07, 'epoch': 14.76}\n",
      "{'loss': 4.4131, 'learning_rate': 3.340210236761961e-07, 'epoch': 14.77}\n",
      "{'loss': 4.5861, 'learning_rate': 3.1437272816583166e-07, 'epoch': 14.79}\n",
      "{'loss': 4.3521, 'learning_rate': 2.9472443265546715e-07, 'epoch': 14.8}\n",
      "{'loss': 4.3643, 'learning_rate': 2.750761371451027e-07, 'epoch': 14.81}\n",
      "{'loss': 4.6727, 'learning_rate': 2.5542784163473823e-07, 'epoch': 14.83}\n",
      "{'loss': 4.3957, 'learning_rate': 2.3577954612437372e-07, 'epoch': 14.84}\n",
      "{'loss': 4.6762, 'learning_rate': 2.1613125061400926e-07, 'epoch': 14.85}\n",
      "{'loss': 4.6287, 'learning_rate': 1.9648295510364477e-07, 'epoch': 14.87}\n",
      "{'loss': 4.9199, 'learning_rate': 1.7683465959328031e-07, 'epoch': 14.88}\n",
      "{'loss': 4.4195, 'learning_rate': 1.5718636408291583e-07, 'epoch': 14.89}\n",
      "{'loss': 4.2844, 'learning_rate': 1.3753806857255134e-07, 'epoch': 14.91}\n",
      "{'loss': 4.5658, 'learning_rate': 1.1788977306218686e-07, 'epoch': 14.92}\n",
      "{'loss': 4.8076, 'learning_rate': 9.824147755182239e-08, 'epoch': 14.93}\n",
      "{'loss': 4.427, 'learning_rate': 7.859318204145791e-08, 'epoch': 14.95}\n",
      "{'loss': 4.8945, 'learning_rate': 5.894488653109343e-08, 'epoch': 14.96}\n",
      "{'loss': 4.8529, 'learning_rate': 3.929659102072896e-08, 'epoch': 14.97}\n",
      "{'loss': 4.7188, 'learning_rate': 1.964829551036448e-08, 'epoch': 14.99}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.4004, 'learning_rate': 0.0, 'epoch': 15.0}\n",
      "{'train_runtime': 5038.1857, 'train_samples_per_second': 287.342, 'train_steps_per_second': 2.245, 'train_loss': 5.1905201771109635, 'epoch': 15.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=11310, training_loss=5.1905201771109635, metrics={'train_runtime': 5038.1857, 'train_samples_per_second': 287.342, 'train_steps_per_second': 2.245, 'train_loss': 5.1905201771109635, 'epoch': 15.0})"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 64 worker processes in total. Our suggested max number of worker in current system is 52, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "  warnings.warn(_create_warning_msg(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Compiling Model...\n",
      "Graph compilation: 100%|██████████| 100/100 [00:10<00:00]\n",
      "Compiled/Loaded model in 19.17071347218007 secs\n",
      "***** Running training *****\n",
      "  Num examples = 104211\n",
      "  Num epochs = 10\n",
      "  Instantaneous batch size per device = 1\n",
      "  Total training batch size (w. parallel, distributed & accumulation) = 128\n",
      "  Gradient accumulation steps = 64\n",
      "  Total optimization steps = 8140\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18d5b0371ab545a1a696466b348a3871",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/8140 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 9.9641, 'learning_rate': 2.457002457002457e-07, 'epoch': 0.01}\n",
      "{'loss': 9.7258, 'learning_rate': 4.914004914004914e-07, 'epoch': 0.02}\n",
      "{'loss': 9.7422, 'learning_rate': 7.371007371007371e-07, 'epoch': 0.04}\n",
      "{'loss': 9.5406, 'learning_rate': 9.828009828009828e-07, 'epoch': 0.05}\n",
      "{'loss': 9.4789, 'learning_rate': 1.2285012285012285e-06, 'epoch': 0.06}\n",
      "{'loss': 9.2594, 'learning_rate': 1.4742014742014743e-06, 'epoch': 0.07}\n",
      "{'loss': 9.2539, 'learning_rate': 1.7199017199017202e-06, 'epoch': 0.09}\n",
      "{'loss': 9.3633, 'learning_rate': 1.9656019656019657e-06, 'epoch': 0.1}\n",
      "{'loss': 9.1586, 'learning_rate': 2.2113022113022116e-06, 'epoch': 0.11}\n",
      "{'loss': 9.0164, 'learning_rate': 2.457002457002457e-06, 'epoch': 0.12}\n",
      "{'loss': 9.0344, 'learning_rate': 2.702702702702703e-06, 'epoch': 0.14}\n",
      "{'loss': 8.9391, 'learning_rate': 2.9484029484029485e-06, 'epoch': 0.15}\n",
      "{'loss': 8.875, 'learning_rate': 3.194103194103194e-06, 'epoch': 0.16}\n",
      "{'loss': 8.6836, 'learning_rate': 3.4398034398034404e-06, 'epoch': 0.17}\n",
      "{'loss': 8.6703, 'learning_rate': 3.685503685503686e-06, 'epoch': 0.18}\n",
      "{'loss': 8.7414, 'learning_rate': 3.931203931203931e-06, 'epoch': 0.2}\n",
      "{'loss': 8.5852, 'learning_rate': 4.176904176904178e-06, 'epoch': 0.21}\n",
      "{'loss': 8.4492, 'learning_rate': 4.422604422604423e-06, 'epoch': 0.22}\n",
      "{'loss': 8.5051, 'learning_rate': 4.668304668304669e-06, 'epoch': 0.23}\n",
      "{'loss': 8.2438, 'learning_rate': 4.914004914004914e-06, 'epoch': 0.25}\n",
      "{'loss': 8.3824, 'learning_rate': 5.1597051597051605e-06, 'epoch': 0.26}\n",
      "{'loss': 8.173, 'learning_rate': 5.405405405405406e-06, 'epoch': 0.27}\n",
      "{'loss': 8.2125, 'learning_rate': 5.6511056511056515e-06, 'epoch': 0.28}\n",
      "{'loss': 8.0938, 'learning_rate': 5.896805896805897e-06, 'epoch': 0.29}\n",
      "{'loss': 8.1598, 'learning_rate': 6.1425061425061425e-06, 'epoch': 0.31}\n",
      "{'loss': 7.6512, 'learning_rate': 6.388206388206388e-06, 'epoch': 0.32}\n",
      "{'loss': 7.9699, 'learning_rate': 6.633906633906635e-06, 'epoch': 0.33}\n",
      "{'loss': 8.0391, 'learning_rate': 6.879606879606881e-06, 'epoch': 0.34}\n",
      "{'loss': 7.7176, 'learning_rate': 7.125307125307126e-06, 'epoch': 0.36}\n",
      "{'loss': 7.7125, 'learning_rate': 7.371007371007372e-06, 'epoch': 0.37}\n",
      "{'loss': 7.7754, 'learning_rate': 7.616707616707617e-06, 'epoch': 0.38}\n",
      "{'loss': 7.5223, 'learning_rate': 7.862407862407863e-06, 'epoch': 0.39}\n",
      "{'loss': 7.5328, 'learning_rate': 8.108108108108109e-06, 'epoch': 0.41}\n",
      "{'loss': 7.4449, 'learning_rate': 8.353808353808355e-06, 'epoch': 0.42}\n",
      "{'loss': 6.9613, 'learning_rate': 8.5995085995086e-06, 'epoch': 0.43}\n",
      "{'loss': 7.0566, 'learning_rate': 8.845208845208846e-06, 'epoch': 0.44}\n",
      "{'loss': 7.1414, 'learning_rate': 9.090909090909091e-06, 'epoch': 0.45}\n",
      "{'loss': 6.9488, 'learning_rate': 9.336609336609337e-06, 'epoch': 0.47}\n",
      "{'loss': 6.7941, 'learning_rate': 9.582309582309584e-06, 'epoch': 0.48}\n",
      "{'loss': 7.3902, 'learning_rate': 9.828009828009828e-06, 'epoch': 0.49}\n",
      "{'loss': 7.2227, 'learning_rate': 1.0073710073710075e-05, 'epoch': 0.5}\n",
      "{'loss': 7.3613, 'learning_rate': 1.0319410319410321e-05, 'epoch': 0.52}\n",
      "{'loss': 6.7609, 'learning_rate': 1.0565110565110566e-05, 'epoch': 0.53}\n",
      "{'loss': 7.0387, 'learning_rate': 1.0810810810810812e-05, 'epoch': 0.54}\n",
      "{'loss': 6.9688, 'learning_rate': 1.1056511056511057e-05, 'epoch': 0.55}\n",
      "{'loss': 6.782, 'learning_rate': 1.1302211302211303e-05, 'epoch': 0.57}\n",
      "{'loss': 7.0516, 'learning_rate': 1.1547911547911548e-05, 'epoch': 0.58}\n",
      "{'loss': 6.7914, 'learning_rate': 1.1793611793611794e-05, 'epoch': 0.59}\n",
      "{'loss': 6.577, 'learning_rate': 1.203931203931204e-05, 'epoch': 0.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.9078, 'learning_rate': 1.2285012285012285e-05, 'epoch': 0.61}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-500/ipu_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.9375, 'learning_rate': 1.2530712530712531e-05, 'epoch': 0.63}\n",
      "{'loss': 6.9258, 'learning_rate': 1.2776412776412776e-05, 'epoch': 0.64}\n",
      "{'loss': 6.7703, 'learning_rate': 1.3022113022113022e-05, 'epoch': 0.65}\n",
      "{'loss': 6.0734, 'learning_rate': 1.326781326781327e-05, 'epoch': 0.66}\n",
      "{'loss': 6.5992, 'learning_rate': 1.3513513513513515e-05, 'epoch': 0.68}\n",
      "{'loss': 6.9984, 'learning_rate': 1.3759213759213761e-05, 'epoch': 0.69}\n",
      "{'loss': 6.325, 'learning_rate': 1.4004914004914006e-05, 'epoch': 0.7}\n",
      "{'loss': 6.5742, 'learning_rate': 1.4250614250614252e-05, 'epoch': 0.71}\n",
      "{'loss': 6.4613, 'learning_rate': 1.4496314496314499e-05, 'epoch': 0.72}\n",
      "{'loss': 6.848, 'learning_rate': 1.4742014742014743e-05, 'epoch': 0.74}\n",
      "{'loss': 6.4121, 'learning_rate': 1.498771498771499e-05, 'epoch': 0.75}\n",
      "{'loss': 6.2102, 'learning_rate': 1.5233415233415234e-05, 'epoch': 0.76}\n",
      "{'loss': 6.527, 'learning_rate': 1.547911547911548e-05, 'epoch': 0.77}\n",
      "{'loss': 6.5516, 'learning_rate': 1.5724815724815725e-05, 'epoch': 0.79}\n",
      "{'loss': 6.152, 'learning_rate': 1.5970515970515972e-05, 'epoch': 0.8}\n",
      "{'loss': 6.6215, 'learning_rate': 1.6216216216216218e-05, 'epoch': 0.81}\n",
      "{'loss': 6.4184, 'learning_rate': 1.6461916461916464e-05, 'epoch': 0.82}\n",
      "{'loss': 6.2594, 'learning_rate': 1.670761670761671e-05, 'epoch': 0.84}\n",
      "{'loss': 5.9465, 'learning_rate': 1.6953316953316954e-05, 'epoch': 0.85}\n",
      "{'loss': 6.2328, 'learning_rate': 1.71990171990172e-05, 'epoch': 0.86}\n",
      "{'loss': 6.5891, 'learning_rate': 1.7444717444717446e-05, 'epoch': 0.87}\n",
      "{'loss': 6.3289, 'learning_rate': 1.7690417690417693e-05, 'epoch': 0.88}\n",
      "{'loss': 6.2328, 'learning_rate': 1.793611793611794e-05, 'epoch': 0.9}\n",
      "{'loss': 6.3551, 'learning_rate': 1.8181818181818182e-05, 'epoch': 0.91}\n",
      "{'loss': 6.143, 'learning_rate': 1.842751842751843e-05, 'epoch': 0.92}\n",
      "{'loss': 5.891, 'learning_rate': 1.8673218673218675e-05, 'epoch': 0.93}\n",
      "{'loss': 6.4555, 'learning_rate': 1.891891891891892e-05, 'epoch': 0.95}\n",
      "{'loss': 6.2023, 'learning_rate': 1.9164619164619167e-05, 'epoch': 0.96}\n",
      "{'loss': 5.9539, 'learning_rate': 1.941031941031941e-05, 'epoch': 0.97}\n",
      "{'loss': 6.4691, 'learning_rate': 1.9656019656019657e-05, 'epoch': 0.98}\n",
      "{'loss': 6.348, 'learning_rate': 1.9901719901719903e-05, 'epoch': 1.0}\n",
      "{'loss': 6.0254, 'learning_rate': 1.9983619983619986e-05, 'epoch': 1.01}\n",
      "{'loss': 6.0281, 'learning_rate': 1.995631995631996e-05, 'epoch': 1.02}\n",
      "{'loss': 6.2887, 'learning_rate': 1.992901992901993e-05, 'epoch': 1.03}\n",
      "{'loss': 6.1086, 'learning_rate': 1.9901719901719903e-05, 'epoch': 1.04}\n",
      "{'loss': 5.7215, 'learning_rate': 1.9874419874419876e-05, 'epoch': 1.06}\n",
      "{'loss': 6.0516, 'learning_rate': 1.984711984711985e-05, 'epoch': 1.07}\n",
      "{'loss': 6.1664, 'learning_rate': 1.981981981981982e-05, 'epoch': 1.08}\n",
      "{'loss': 6.1797, 'learning_rate': 1.9792519792519793e-05, 'epoch': 1.09}\n",
      "{'loss': 6.2625, 'learning_rate': 1.9765219765219767e-05, 'epoch': 1.11}\n",
      "{'loss': 6.0938, 'learning_rate': 1.973791973791974e-05, 'epoch': 1.12}\n",
      "{'loss': 6.1109, 'learning_rate': 1.9710619710619713e-05, 'epoch': 1.13}\n",
      "{'loss': 5.8883, 'learning_rate': 1.9683319683319687e-05, 'epoch': 1.14}\n",
      "{'loss': 6.3484, 'learning_rate': 1.9656019656019657e-05, 'epoch': 1.15}\n",
      "{'loss': 5.677, 'learning_rate': 1.962871962871963e-05, 'epoch': 1.17}\n",
      "{'loss': 5.8875, 'learning_rate': 1.9601419601419604e-05, 'epoch': 1.18}\n",
      "{'loss': 6.1277, 'learning_rate': 1.9574119574119577e-05, 'epoch': 1.19}\n",
      "{'loss': 6.0441, 'learning_rate': 1.954681954681955e-05, 'epoch': 1.2}\n",
      "{'loss': 5.8266, 'learning_rate': 1.951951951951952e-05, 'epoch': 1.22}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.3648, 'learning_rate': 1.9492219492219494e-05, 'epoch': 1.23}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-1000/ipu_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.2207, 'learning_rate': 1.9464919464919467e-05, 'epoch': 1.24}\n",
      "{'loss': 6.4652, 'learning_rate': 1.943761943761944e-05, 'epoch': 1.25}\n",
      "{'loss': 6.0277, 'learning_rate': 1.941031941031941e-05, 'epoch': 1.27}\n",
      "{'loss': 5.8172, 'learning_rate': 1.9383019383019384e-05, 'epoch': 1.28}\n",
      "{'loss': 5.5039, 'learning_rate': 1.9355719355719357e-05, 'epoch': 1.29}\n",
      "{'loss': 6.2402, 'learning_rate': 1.932841932841933e-05, 'epoch': 1.3}\n",
      "{'loss': 5.5945, 'learning_rate': 1.93011193011193e-05, 'epoch': 1.31}\n",
      "{'loss': 6.2457, 'learning_rate': 1.9273819273819277e-05, 'epoch': 1.33}\n",
      "{'loss': 6.2223, 'learning_rate': 1.9246519246519247e-05, 'epoch': 1.34}\n",
      "{'loss': 5.6219, 'learning_rate': 1.921921921921922e-05, 'epoch': 1.35}\n",
      "{'loss': 6.0824, 'learning_rate': 1.9191919191919194e-05, 'epoch': 1.36}\n",
      "{'loss': 6.1848, 'learning_rate': 1.9164619164619167e-05, 'epoch': 1.38}\n",
      "{'loss': 6.0227, 'learning_rate': 1.913731913731914e-05, 'epoch': 1.39}\n",
      "{'loss': 6.3336, 'learning_rate': 1.911001911001911e-05, 'epoch': 1.4}\n",
      "{'loss': 6.2766, 'learning_rate': 1.9082719082719084e-05, 'epoch': 1.41}\n",
      "{'loss': 6.0453, 'learning_rate': 1.9055419055419058e-05, 'epoch': 1.43}\n",
      "{'loss': 5.9766, 'learning_rate': 1.902811902811903e-05, 'epoch': 1.44}\n",
      "{'loss': 6.3234, 'learning_rate': 1.9000819000819e-05, 'epoch': 1.45}\n",
      "{'loss': 5.6547, 'learning_rate': 1.8973518973518974e-05, 'epoch': 1.46}\n",
      "{'loss': 5.5395, 'learning_rate': 1.8946218946218948e-05, 'epoch': 1.47}\n",
      "{'loss': 6.1742, 'learning_rate': 1.891891891891892e-05, 'epoch': 1.49}\n",
      "{'loss': 5.6488, 'learning_rate': 1.889161889161889e-05, 'epoch': 1.5}\n",
      "{'loss': 5.6047, 'learning_rate': 1.8864318864318868e-05, 'epoch': 1.51}\n",
      "{'loss': 5.5172, 'learning_rate': 1.8837018837018838e-05, 'epoch': 1.52}\n",
      "{'loss': 6.1066, 'learning_rate': 1.880971880971881e-05, 'epoch': 1.54}\n",
      "{'loss': 5.7629, 'learning_rate': 1.8782418782418785e-05, 'epoch': 1.55}\n",
      "{'loss': 5.5211, 'learning_rate': 1.8755118755118758e-05, 'epoch': 1.56}\n",
      "{'loss': 5.9801, 'learning_rate': 1.8727818727818728e-05, 'epoch': 1.57}\n",
      "{'loss': 6.477, 'learning_rate': 1.87005187005187e-05, 'epoch': 1.58}\n",
      "{'loss': 5.6359, 'learning_rate': 1.8673218673218675e-05, 'epoch': 1.6}\n",
      "{'loss': 5.5742, 'learning_rate': 1.8645918645918648e-05, 'epoch': 1.61}\n",
      "{'loss': 5.5516, 'learning_rate': 1.861861861861862e-05, 'epoch': 1.62}\n",
      "{'loss': 5.8172, 'learning_rate': 1.859131859131859e-05, 'epoch': 1.63}\n",
      "{'loss': 5.7965, 'learning_rate': 1.8564018564018568e-05, 'epoch': 1.65}\n",
      "{'loss': 6.4184, 'learning_rate': 1.8536718536718538e-05, 'epoch': 1.66}\n",
      "{'loss': 5.8262, 'learning_rate': 1.850941850941851e-05, 'epoch': 1.67}\n",
      "{'loss': 5.4992, 'learning_rate': 1.848211848211848e-05, 'epoch': 1.68}\n",
      "{'loss': 5.1414, 'learning_rate': 1.845481845481846e-05, 'epoch': 1.7}\n",
      "{'loss': 5.3992, 'learning_rate': 1.842751842751843e-05, 'epoch': 1.71}\n",
      "{'loss': 5.4703, 'learning_rate': 1.8400218400218402e-05, 'epoch': 1.72}\n",
      "{'loss': 6.2008, 'learning_rate': 1.8372918372918375e-05, 'epoch': 1.73}\n",
      "{'loss': 5.8332, 'learning_rate': 1.834561834561835e-05, 'epoch': 1.74}\n",
      "{'loss': 6.3414, 'learning_rate': 1.831831831831832e-05, 'epoch': 1.76}\n",
      "{'loss': 6.1227, 'learning_rate': 1.8291018291018292e-05, 'epoch': 1.77}\n",
      "{'loss': 5.7324, 'learning_rate': 1.8263718263718265e-05, 'epoch': 1.78}\n",
      "{'loss': 6.1121, 'learning_rate': 1.823641823641824e-05, 'epoch': 1.79}\n",
      "{'loss': 5.6352, 'learning_rate': 1.8209118209118212e-05, 'epoch': 1.81}\n",
      "{'loss': 5.5531, 'learning_rate': 1.8181818181818182e-05, 'epoch': 1.82}\n",
      "{'loss': 5.3445, 'learning_rate': 1.8154518154518155e-05, 'epoch': 1.83}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-1500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.7414, 'learning_rate': 1.812721812721813e-05, 'epoch': 1.84}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-1500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 6.2109, 'learning_rate': 1.8099918099918102e-05, 'epoch': 1.86}\n",
      "{'loss': 5.7977, 'learning_rate': 1.8072618072618072e-05, 'epoch': 1.87}\n",
      "{'loss': 5.7398, 'learning_rate': 1.804531804531805e-05, 'epoch': 1.88}\n",
      "{'loss': 5.5156, 'learning_rate': 1.801801801801802e-05, 'epoch': 1.89}\n",
      "{'loss': 5.9043, 'learning_rate': 1.7990717990717992e-05, 'epoch': 1.9}\n",
      "{'loss': 5.6461, 'learning_rate': 1.7963417963417962e-05, 'epoch': 1.92}\n",
      "{'loss': 5.6109, 'learning_rate': 1.793611793611794e-05, 'epoch': 1.93}\n",
      "{'loss': 5.7582, 'learning_rate': 1.790881790881791e-05, 'epoch': 1.94}\n",
      "{'loss': 5.5078, 'learning_rate': 1.7881517881517882e-05, 'epoch': 1.95}\n",
      "{'loss': 5.5406, 'learning_rate': 1.7854217854217856e-05, 'epoch': 1.97}\n",
      "{'loss': 5.6883, 'learning_rate': 1.782691782691783e-05, 'epoch': 1.98}\n",
      "{'loss': 5.2387, 'learning_rate': 1.7799617799617803e-05, 'epoch': 1.99}\n",
      "{'loss': 5.3887, 'learning_rate': 1.7772317772317773e-05, 'epoch': 2.0}\n",
      "{'loss': 5.6902, 'learning_rate': 1.7745017745017746e-05, 'epoch': 2.01}\n",
      "{'loss': 5.9711, 'learning_rate': 1.771771771771772e-05, 'epoch': 2.03}\n",
      "{'loss': 5.2328, 'learning_rate': 1.7690417690417693e-05, 'epoch': 2.04}\n",
      "{'loss': 5.5062, 'learning_rate': 1.7663117663117663e-05, 'epoch': 2.05}\n",
      "{'loss': 5.6471, 'learning_rate': 1.763581763581764e-05, 'epoch': 2.06}\n",
      "{'loss': 5.5734, 'learning_rate': 1.760851760851761e-05, 'epoch': 2.08}\n",
      "{'loss': 5.7383, 'learning_rate': 1.7581217581217583e-05, 'epoch': 2.09}\n",
      "{'loss': 5.7105, 'learning_rate': 1.7553917553917553e-05, 'epoch': 2.1}\n",
      "{'loss': 5.4297, 'learning_rate': 1.752661752661753e-05, 'epoch': 2.11}\n",
      "{'loss': 4.951, 'learning_rate': 1.74993174993175e-05, 'epoch': 2.13}\n",
      "{'loss': 5.493, 'learning_rate': 1.7472017472017473e-05, 'epoch': 2.14}\n",
      "{'loss': 5.6312, 'learning_rate': 1.7444717444717446e-05, 'epoch': 2.15}\n",
      "{'loss': 5.6109, 'learning_rate': 1.741741741741742e-05, 'epoch': 2.16}\n",
      "{'loss': 5.4859, 'learning_rate': 1.739011739011739e-05, 'epoch': 2.17}\n",
      "{'loss': 6.2238, 'learning_rate': 1.7362817362817363e-05, 'epoch': 2.19}\n",
      "{'loss': 5.4773, 'learning_rate': 1.7335517335517337e-05, 'epoch': 2.2}\n",
      "{'loss': 5.8113, 'learning_rate': 1.730821730821731e-05, 'epoch': 2.21}\n",
      "{'loss': 5.6145, 'learning_rate': 1.7280917280917283e-05, 'epoch': 2.22}\n",
      "{'loss': 5.8414, 'learning_rate': 1.7253617253617253e-05, 'epoch': 2.24}\n",
      "{'loss': 5.5992, 'learning_rate': 1.722631722631723e-05, 'epoch': 2.25}\n",
      "{'loss': 5.682, 'learning_rate': 1.71990171990172e-05, 'epoch': 2.26}\n",
      "{'loss': 5.491, 'learning_rate': 1.7171717171717173e-05, 'epoch': 2.27}\n",
      "{'loss': 5.7687, 'learning_rate': 1.7144417144417143e-05, 'epoch': 2.29}\n",
      "{'loss': 5.8053, 'learning_rate': 1.711711711711712e-05, 'epoch': 2.3}\n",
      "{'loss': 5.4777, 'learning_rate': 1.708981708981709e-05, 'epoch': 2.31}\n",
      "{'loss': 5.3836, 'learning_rate': 1.7062517062517064e-05, 'epoch': 2.32}\n",
      "{'loss': 5.4313, 'learning_rate': 1.7035217035217037e-05, 'epoch': 2.33}\n",
      "{'loss': 5.5777, 'learning_rate': 1.700791700791701e-05, 'epoch': 2.35}\n",
      "{'loss': 5.5242, 'learning_rate': 1.698061698061698e-05, 'epoch': 2.36}\n",
      "{'loss': 5.5059, 'learning_rate': 1.6953316953316954e-05, 'epoch': 2.37}\n",
      "{'loss': 5.6602, 'learning_rate': 1.6926016926016927e-05, 'epoch': 2.38}\n",
      "{'loss': 5.3289, 'learning_rate': 1.68987168987169e-05, 'epoch': 2.4}\n",
      "{'loss': 5.6797, 'learning_rate': 1.6871416871416874e-05, 'epoch': 2.41}\n",
      "{'loss': 5.4207, 'learning_rate': 1.6844116844116844e-05, 'epoch': 2.42}\n",
      "{'loss': 5.7922, 'learning_rate': 1.6816816816816817e-05, 'epoch': 2.43}\n",
      "{'loss': 5.6082, 'learning_rate': 1.678951678951679e-05, 'epoch': 2.44}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-2000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.6793, 'learning_rate': 1.6762216762216764e-05, 'epoch': 2.46}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-2000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-1000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.4363, 'learning_rate': 1.6734916734916734e-05, 'epoch': 2.47}\n",
      "{'loss': 5.6934, 'learning_rate': 1.670761670761671e-05, 'epoch': 2.48}\n",
      "{'loss': 5.3977, 'learning_rate': 1.668031668031668e-05, 'epoch': 2.49}\n",
      "{'loss': 5.1281, 'learning_rate': 1.6653016653016654e-05, 'epoch': 2.51}\n",
      "{'loss': 5.2936, 'learning_rate': 1.6625716625716628e-05, 'epoch': 2.52}\n",
      "{'loss': 5.7656, 'learning_rate': 1.65984165984166e-05, 'epoch': 2.53}\n",
      "{'loss': 5.5334, 'learning_rate': 1.657111657111657e-05, 'epoch': 2.54}\n",
      "{'loss': 5.707, 'learning_rate': 1.6543816543816544e-05, 'epoch': 2.56}\n",
      "{'loss': 5.7602, 'learning_rate': 1.6516516516516518e-05, 'epoch': 2.57}\n",
      "{'loss': 5.5594, 'learning_rate': 1.648921648921649e-05, 'epoch': 2.58}\n",
      "{'loss': 5.1334, 'learning_rate': 1.6461916461916464e-05, 'epoch': 2.59}\n",
      "{'loss': 5.1512, 'learning_rate': 1.6434616434616434e-05, 'epoch': 2.6}\n",
      "{'loss': 5.4928, 'learning_rate': 1.6407316407316408e-05, 'epoch': 2.62}\n",
      "{'loss': 5.8844, 'learning_rate': 1.638001638001638e-05, 'epoch': 2.63}\n",
      "{'loss': 4.9787, 'learning_rate': 1.6352716352716355e-05, 'epoch': 2.64}\n",
      "{'loss': 5.2687, 'learning_rate': 1.6325416325416328e-05, 'epoch': 2.65}\n",
      "{'loss': 4.9996, 'learning_rate': 1.62981162981163e-05, 'epoch': 2.67}\n",
      "{'loss': 5.7141, 'learning_rate': 1.627081627081627e-05, 'epoch': 2.68}\n",
      "{'loss': 5.4758, 'learning_rate': 1.6243516243516245e-05, 'epoch': 2.69}\n",
      "{'loss': 5.3107, 'learning_rate': 1.6216216216216218e-05, 'epoch': 2.7}\n",
      "{'loss': 5.3242, 'learning_rate': 1.618891618891619e-05, 'epoch': 2.71}\n",
      "{'loss': 5.4492, 'learning_rate': 1.616161616161616e-05, 'epoch': 2.73}\n",
      "{'loss': 5.3641, 'learning_rate': 1.6134316134316135e-05, 'epoch': 2.74}\n",
      "{'loss': 5.2252, 'learning_rate': 1.6107016107016108e-05, 'epoch': 2.75}\n",
      "{'loss': 5.5023, 'learning_rate': 1.607971607971608e-05, 'epoch': 2.76}\n",
      "{'loss': 5.6928, 'learning_rate': 1.605241605241605e-05, 'epoch': 2.78}\n",
      "{'loss': 5.6016, 'learning_rate': 1.6025116025116025e-05, 'epoch': 2.79}\n",
      "{'loss': 5.5551, 'learning_rate': 1.5997815997816e-05, 'epoch': 2.8}\n",
      "{'loss': 5.6703, 'learning_rate': 1.5970515970515972e-05, 'epoch': 2.81}\n",
      "{'loss': 5.4363, 'learning_rate': 1.5943215943215945e-05, 'epoch': 2.83}\n",
      "{'loss': 5.7133, 'learning_rate': 1.591591591591592e-05, 'epoch': 2.84}\n",
      "{'loss': 5.2559, 'learning_rate': 1.5888615888615892e-05, 'epoch': 2.85}\n",
      "{'loss': 5.2984, 'learning_rate': 1.5861315861315862e-05, 'epoch': 2.86}\n",
      "{'loss': 5.6805, 'learning_rate': 1.5834015834015835e-05, 'epoch': 2.87}\n",
      "{'loss': 5.2168, 'learning_rate': 1.580671580671581e-05, 'epoch': 2.89}\n",
      "{'loss': 5.2676, 'learning_rate': 1.5779415779415782e-05, 'epoch': 2.9}\n",
      "{'loss': 5.2402, 'learning_rate': 1.5752115752115752e-05, 'epoch': 2.91}\n",
      "{'loss': 5.6484, 'learning_rate': 1.5724815724815725e-05, 'epoch': 2.92}\n",
      "{'loss': 5.6223, 'learning_rate': 1.56975156975157e-05, 'epoch': 2.94}\n",
      "{'loss': 5.3264, 'learning_rate': 1.5670215670215672e-05, 'epoch': 2.95}\n",
      "{'loss': 5.3844, 'learning_rate': 1.5642915642915642e-05, 'epoch': 2.96}\n",
      "{'loss': 4.8615, 'learning_rate': 1.5615615615615616e-05, 'epoch': 2.97}\n",
      "{'loss': 5.3215, 'learning_rate': 1.558831558831559e-05, 'epoch': 2.99}\n",
      "{'loss': 4.8551, 'learning_rate': 1.5561015561015562e-05, 'epoch': 3.0}\n",
      "{'loss': 5.2664, 'learning_rate': 1.5533715533715536e-05, 'epoch': 3.01}\n",
      "{'loss': 5.2059, 'learning_rate': 1.550641550641551e-05, 'epoch': 3.02}\n",
      "{'loss': 4.9039, 'learning_rate': 1.547911547911548e-05, 'epoch': 3.03}\n",
      "{'loss': 5.3186, 'learning_rate': 1.5451815451815452e-05, 'epoch': 3.05}\n",
      "{'loss': 5.2436, 'learning_rate': 1.5424515424515426e-05, 'epoch': 3.06}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-2500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.3844, 'learning_rate': 1.53972153972154e-05, 'epoch': 3.07}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-2500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-1500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.5363, 'learning_rate': 1.5369915369915373e-05, 'epoch': 3.08}\n",
      "{'loss': 5.1562, 'learning_rate': 1.5342615342615343e-05, 'epoch': 3.1}\n",
      "{'loss': 5.2203, 'learning_rate': 1.5315315315315316e-05, 'epoch': 3.11}\n",
      "{'loss': 5.5367, 'learning_rate': 1.528801528801529e-05, 'epoch': 3.12}\n",
      "{'loss': 4.9617, 'learning_rate': 1.5260715260715263e-05, 'epoch': 3.13}\n",
      "{'loss': 5.0309, 'learning_rate': 1.5233415233415234e-05, 'epoch': 3.14}\n",
      "{'loss': 5.3258, 'learning_rate': 1.5206115206115208e-05, 'epoch': 3.16}\n",
      "{'loss': 5.3164, 'learning_rate': 1.5178815178815181e-05, 'epoch': 3.17}\n",
      "{'loss': 5.1254, 'learning_rate': 1.5151515151515153e-05, 'epoch': 3.18}\n",
      "{'loss': 5.1707, 'learning_rate': 1.5124215124215125e-05, 'epoch': 3.19}\n",
      "{'loss': 5.3867, 'learning_rate': 1.50969150969151e-05, 'epoch': 3.21}\n",
      "{'loss': 5.4793, 'learning_rate': 1.5069615069615071e-05, 'epoch': 3.22}\n",
      "{'loss': 4.874, 'learning_rate': 1.5042315042315043e-05, 'epoch': 3.23}\n",
      "{'loss': 5.2789, 'learning_rate': 1.5015015015015015e-05, 'epoch': 3.24}\n",
      "{'loss': 4.8371, 'learning_rate': 1.498771498771499e-05, 'epoch': 3.26}\n",
      "{'loss': 5.2352, 'learning_rate': 1.4960414960414961e-05, 'epoch': 3.27}\n",
      "{'loss': 5.1635, 'learning_rate': 1.4933114933114933e-05, 'epoch': 3.28}\n",
      "{'loss': 5.1496, 'learning_rate': 1.4905814905814907e-05, 'epoch': 3.29}\n",
      "{'loss': 5.4469, 'learning_rate': 1.487851487851488e-05, 'epoch': 3.3}\n",
      "{'loss': 5.1078, 'learning_rate': 1.4851214851214852e-05, 'epoch': 3.32}\n",
      "{'loss': 5.5004, 'learning_rate': 1.4823914823914825e-05, 'epoch': 3.33}\n",
      "{'loss': 5.3281, 'learning_rate': 1.4796614796614798e-05, 'epoch': 3.34}\n",
      "{'loss': 5.1221, 'learning_rate': 1.4769314769314772e-05, 'epoch': 3.35}\n",
      "{'loss': 5.3066, 'learning_rate': 1.4742014742014743e-05, 'epoch': 3.37}\n",
      "{'loss': 5.5266, 'learning_rate': 1.4714714714714715e-05, 'epoch': 3.38}\n",
      "{'loss': 5.4391, 'learning_rate': 1.468741468741469e-05, 'epoch': 3.39}\n",
      "{'loss': 5.6023, 'learning_rate': 1.4660114660114662e-05, 'epoch': 3.4}\n",
      "{'loss': 4.8803, 'learning_rate': 1.4632814632814634e-05, 'epoch': 3.42}\n",
      "{'loss': 5.0441, 'learning_rate': 1.4605514605514605e-05, 'epoch': 3.43}\n",
      "{'loss': 5.4801, 'learning_rate': 1.457821457821458e-05, 'epoch': 3.44}\n",
      "{'loss': 5.55, 'learning_rate': 1.4550914550914552e-05, 'epoch': 3.45}\n",
      "{'loss': 5.4, 'learning_rate': 1.4523614523614524e-05, 'epoch': 3.46}\n",
      "{'loss': 5.0629, 'learning_rate': 1.4496314496314499e-05, 'epoch': 3.48}\n",
      "{'loss': 5.3518, 'learning_rate': 1.446901446901447e-05, 'epoch': 3.49}\n",
      "{'loss': 5.4432, 'learning_rate': 1.4441714441714442e-05, 'epoch': 3.5}\n",
      "{'loss': 5.3387, 'learning_rate': 1.4414414414414416e-05, 'epoch': 3.51}\n",
      "{'loss': 5.7102, 'learning_rate': 1.4387114387114389e-05, 'epoch': 3.53}\n",
      "{'loss': 5.3262, 'learning_rate': 1.435981435981436e-05, 'epoch': 3.54}\n",
      "{'loss': 5.4928, 'learning_rate': 1.4332514332514334e-05, 'epoch': 3.55}\n",
      "{'loss': 5.1555, 'learning_rate': 1.4305214305214306e-05, 'epoch': 3.56}\n",
      "{'loss': 5.5352, 'learning_rate': 1.4277914277914279e-05, 'epoch': 3.57}\n",
      "{'loss': 5.0535, 'learning_rate': 1.4250614250614252e-05, 'epoch': 3.59}\n",
      "{'loss': 5.6523, 'learning_rate': 1.4223314223314224e-05, 'epoch': 3.6}\n",
      "{'loss': 5.4742, 'learning_rate': 1.4196014196014196e-05, 'epoch': 3.61}\n",
      "{'loss': 5.5961, 'learning_rate': 1.4168714168714171e-05, 'epoch': 3.62}\n",
      "{'loss': 5.4545, 'learning_rate': 1.4141414141414143e-05, 'epoch': 3.64}\n",
      "{'loss': 5.3469, 'learning_rate': 1.4114114114114114e-05, 'epoch': 3.65}\n",
      "{'loss': 5.3877, 'learning_rate': 1.408681408681409e-05, 'epoch': 3.66}\n",
      "{'loss': 5.4582, 'learning_rate': 1.4059514059514061e-05, 'epoch': 3.67}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-3000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0285, 'learning_rate': 1.4032214032214033e-05, 'epoch': 3.69}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-3000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-2000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.7695, 'learning_rate': 1.4004914004914006e-05, 'epoch': 3.7}\n",
      "{'loss': 5.2102, 'learning_rate': 1.397761397761398e-05, 'epoch': 3.71}\n",
      "{'loss': 5.0146, 'learning_rate': 1.3950313950313951e-05, 'epoch': 3.72}\n",
      "{'loss': 5.0525, 'learning_rate': 1.3923013923013925e-05, 'epoch': 3.73}\n",
      "{'loss': 4.9785, 'learning_rate': 1.3895713895713896e-05, 'epoch': 3.75}\n",
      "{'loss': 5.6598, 'learning_rate': 1.386841386841387e-05, 'epoch': 3.76}\n",
      "{'loss': 5.4984, 'learning_rate': 1.3841113841113843e-05, 'epoch': 3.77}\n",
      "{'loss': 5.4832, 'learning_rate': 1.3813813813813815e-05, 'epoch': 3.78}\n",
      "{'loss': 5.2936, 'learning_rate': 1.3786513786513786e-05, 'epoch': 3.8}\n",
      "{'loss': 4.6398, 'learning_rate': 1.3759213759213761e-05, 'epoch': 3.81}\n",
      "{'loss': 5.4941, 'learning_rate': 1.3731913731913733e-05, 'epoch': 3.82}\n",
      "{'loss': 5.4863, 'learning_rate': 1.3704613704613705e-05, 'epoch': 3.83}\n",
      "{'loss': 5.3971, 'learning_rate': 1.367731367731368e-05, 'epoch': 3.85}\n",
      "{'loss': 5.1391, 'learning_rate': 1.3650013650013652e-05, 'epoch': 3.86}\n",
      "{'loss': 5.3225, 'learning_rate': 1.3622713622713623e-05, 'epoch': 3.87}\n",
      "{'loss': 5.3973, 'learning_rate': 1.3595413595413595e-05, 'epoch': 3.88}\n",
      "{'loss': 5.2977, 'learning_rate': 1.356811356811357e-05, 'epoch': 3.89}\n",
      "{'loss': 4.7176, 'learning_rate': 1.3540813540813542e-05, 'epoch': 3.91}\n",
      "{'loss': 5.3629, 'learning_rate': 1.3513513513513515e-05, 'epoch': 3.92}\n",
      "{'loss': 5.0293, 'learning_rate': 1.3486213486213487e-05, 'epoch': 3.93}\n",
      "{'loss': 5.1508, 'learning_rate': 1.345891345891346e-05, 'epoch': 3.94}\n",
      "{'loss': 5.1605, 'learning_rate': 1.3431613431613434e-05, 'epoch': 3.96}\n",
      "{'loss': 4.8324, 'learning_rate': 1.3404313404313405e-05, 'epoch': 3.97}\n",
      "{'loss': 5.4102, 'learning_rate': 1.3377013377013379e-05, 'epoch': 3.98}\n",
      "{'loss': 5.1002, 'learning_rate': 1.3349713349713352e-05, 'epoch': 3.99}\n",
      "{'loss': 5.0896, 'learning_rate': 1.3322413322413324e-05, 'epoch': 4.0}\n",
      "{'loss': 5.5184, 'learning_rate': 1.3295113295113295e-05, 'epoch': 4.02}\n",
      "{'loss': 5.051, 'learning_rate': 1.326781326781327e-05, 'epoch': 4.03}\n",
      "{'loss': 4.9969, 'learning_rate': 1.3240513240513242e-05, 'epoch': 4.04}\n",
      "{'loss': 5.4039, 'learning_rate': 1.3213213213213214e-05, 'epoch': 4.05}\n",
      "{'loss': 5.1852, 'learning_rate': 1.3185913185913185e-05, 'epoch': 4.07}\n",
      "{'loss': 5.0723, 'learning_rate': 1.315861315861316e-05, 'epoch': 4.08}\n",
      "{'loss': 5.5543, 'learning_rate': 1.3131313131313132e-05, 'epoch': 4.09}\n",
      "{'loss': 5.2707, 'learning_rate': 1.3104013104013104e-05, 'epoch': 4.1}\n",
      "{'loss': 5.5137, 'learning_rate': 1.3076713076713077e-05, 'epoch': 4.12}\n",
      "{'loss': 5.0172, 'learning_rate': 1.304941304941305e-05, 'epoch': 4.13}\n",
      "{'loss': 5.1777, 'learning_rate': 1.3022113022113022e-05, 'epoch': 4.14}\n",
      "{'loss': 5.4977, 'learning_rate': 1.2994812994812996e-05, 'epoch': 4.15}\n",
      "{'loss': 5.1068, 'learning_rate': 1.2967512967512969e-05, 'epoch': 4.16}\n",
      "{'loss': 5.1271, 'learning_rate': 1.2940212940212943e-05, 'epoch': 4.18}\n",
      "{'loss': 5.3637, 'learning_rate': 1.2912912912912914e-05, 'epoch': 4.19}\n",
      "{'loss': 5.2861, 'learning_rate': 1.2885612885612886e-05, 'epoch': 4.2}\n",
      "{'loss': 5.3559, 'learning_rate': 1.2858312858312861e-05, 'epoch': 4.21}\n",
      "{'loss': 5.3566, 'learning_rate': 1.2831012831012833e-05, 'epoch': 4.23}\n",
      "{'loss': 5.1723, 'learning_rate': 1.2803712803712804e-05, 'epoch': 4.24}\n",
      "{'loss': 5.1902, 'learning_rate': 1.2776412776412776e-05, 'epoch': 4.25}\n",
      "{'loss': 5.2562, 'learning_rate': 1.2749112749112751e-05, 'epoch': 4.26}\n",
      "{'loss': 5.0445, 'learning_rate': 1.2721812721812723e-05, 'epoch': 4.28}\n",
      "{'loss': 4.8184, 'learning_rate': 1.2694512694512694e-05, 'epoch': 4.29}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-3500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0545, 'learning_rate': 1.2667212667212668e-05, 'epoch': 4.3}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-3500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-2500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.1865, 'learning_rate': 1.2639912639912641e-05, 'epoch': 4.31}\n",
      "{'loss': 5.0086, 'learning_rate': 1.2612612612612613e-05, 'epoch': 4.32}\n",
      "{'loss': 5.0129, 'learning_rate': 1.2585312585312586e-05, 'epoch': 4.34}\n",
      "{'loss': 4.9273, 'learning_rate': 1.255801255801256e-05, 'epoch': 4.35}\n",
      "{'loss': 5.1789, 'learning_rate': 1.2530712530712531e-05, 'epoch': 4.36}\n",
      "{'loss': 5.0949, 'learning_rate': 1.2503412503412505e-05, 'epoch': 4.37}\n",
      "{'loss': 4.9094, 'learning_rate': 1.2476112476112476e-05, 'epoch': 4.39}\n",
      "{'loss': 4.8402, 'learning_rate': 1.244881244881245e-05, 'epoch': 4.4}\n",
      "{'loss': 4.8744, 'learning_rate': 1.2421512421512423e-05, 'epoch': 4.41}\n",
      "{'loss': 4.8107, 'learning_rate': 1.2394212394212395e-05, 'epoch': 4.42}\n",
      "{'loss': 5.0334, 'learning_rate': 1.2366912366912367e-05, 'epoch': 4.43}\n",
      "{'loss': 5.0383, 'learning_rate': 1.2339612339612342e-05, 'epoch': 4.45}\n",
      "{'loss': 5.0402, 'learning_rate': 1.2312312312312313e-05, 'epoch': 4.46}\n",
      "{'loss': 5.2195, 'learning_rate': 1.2285012285012285e-05, 'epoch': 4.47}\n",
      "{'loss': 5.2348, 'learning_rate': 1.225771225771226e-05, 'epoch': 4.48}\n",
      "{'loss': 5.3207, 'learning_rate': 1.2230412230412232e-05, 'epoch': 4.5}\n",
      "{'loss': 5.232, 'learning_rate': 1.2203112203112203e-05, 'epoch': 4.51}\n",
      "{'loss': 5.2402, 'learning_rate': 1.2175812175812177e-05, 'epoch': 4.52}\n",
      "{'loss': 5.1479, 'learning_rate': 1.214851214851215e-05, 'epoch': 4.53}\n",
      "{'loss': 5.3797, 'learning_rate': 1.2121212121212122e-05, 'epoch': 4.55}\n",
      "{'loss': 5.0059, 'learning_rate': 1.2093912093912095e-05, 'epoch': 4.56}\n",
      "{'loss': 5.2977, 'learning_rate': 1.2066612066612067e-05, 'epoch': 4.57}\n",
      "{'loss': 5.3732, 'learning_rate': 1.203931203931204e-05, 'epoch': 4.58}\n",
      "{'loss': 5.1559, 'learning_rate': 1.2012012012012014e-05, 'epoch': 4.59}\n",
      "{'loss': 4.8701, 'learning_rate': 1.1984711984711985e-05, 'epoch': 4.61}\n",
      "{'loss': 5.0156, 'learning_rate': 1.1957411957411957e-05, 'epoch': 4.62}\n",
      "{'loss': 5.0293, 'learning_rate': 1.1930111930111932e-05, 'epoch': 4.63}\n",
      "{'loss': 5.1688, 'learning_rate': 1.1902811902811904e-05, 'epoch': 4.64}\n",
      "{'loss': 5.0424, 'learning_rate': 1.1875511875511876e-05, 'epoch': 4.66}\n",
      "{'loss': 5.5793, 'learning_rate': 1.184821184821185e-05, 'epoch': 4.67}\n",
      "{'loss': 5.1256, 'learning_rate': 1.1820911820911822e-05, 'epoch': 4.68}\n",
      "{'loss': 4.8664, 'learning_rate': 1.1793611793611794e-05, 'epoch': 4.69}\n",
      "{'loss': 5.334, 'learning_rate': 1.1766311766311766e-05, 'epoch': 4.71}\n",
      "{'loss': 5.3129, 'learning_rate': 1.173901173901174e-05, 'epoch': 4.72}\n",
      "{'loss': 5.1203, 'learning_rate': 1.1711711711711713e-05, 'epoch': 4.73}\n",
      "{'loss': 4.7949, 'learning_rate': 1.1684411684411686e-05, 'epoch': 4.74}\n",
      "{'loss': 5.2133, 'learning_rate': 1.1657111657111658e-05, 'epoch': 4.75}\n",
      "{'loss': 4.8461, 'learning_rate': 1.1629811629811631e-05, 'epoch': 4.77}\n",
      "{'loss': 5.2418, 'learning_rate': 1.1602511602511604e-05, 'epoch': 4.78}\n",
      "{'loss': 5.0883, 'learning_rate': 1.1575211575211576e-05, 'epoch': 4.79}\n",
      "{'loss': 5.093, 'learning_rate': 1.1547911547911548e-05, 'epoch': 4.8}\n",
      "{'loss': 4.8539, 'learning_rate': 1.1520611520611523e-05, 'epoch': 4.82}\n",
      "{'loss': 4.9262, 'learning_rate': 1.1493311493311494e-05, 'epoch': 4.83}\n",
      "{'loss': 5.3445, 'learning_rate': 1.1466011466011466e-05, 'epoch': 4.84}\n",
      "{'loss': 5.6285, 'learning_rate': 1.1438711438711441e-05, 'epoch': 4.85}\n",
      "{'loss': 5.3695, 'learning_rate': 1.1411411411411413e-05, 'epoch': 4.86}\n",
      "{'loss': 4.9492, 'learning_rate': 1.1384111384111385e-05, 'epoch': 4.88}\n",
      "{'loss': 5.3273, 'learning_rate': 1.1356811356811356e-05, 'epoch': 4.89}\n",
      "{'loss': 4.9297, 'learning_rate': 1.1329511329511331e-05, 'epoch': 4.9}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8096, 'learning_rate': 1.1302211302211303e-05, 'epoch': 4.91}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-4000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-3000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9254, 'learning_rate': 1.1274911274911275e-05, 'epoch': 4.93}\n",
      "{'loss': 5.4734, 'learning_rate': 1.1247611247611248e-05, 'epoch': 4.94}\n",
      "{'loss': 5.1844, 'learning_rate': 1.1220311220311222e-05, 'epoch': 4.95}\n",
      "{'loss': 5.0664, 'learning_rate': 1.1193011193011193e-05, 'epoch': 4.96}\n",
      "{'loss': 5.1027, 'learning_rate': 1.1165711165711167e-05, 'epoch': 4.98}\n",
      "{'loss': 5.3, 'learning_rate': 1.113841113841114e-05, 'epoch': 4.99}\n",
      "{'loss': 4.5398, 'learning_rate': 1.1111111111111113e-05, 'epoch': 5.0}\n",
      "{'loss': 4.6867, 'learning_rate': 1.1083811083811085e-05, 'epoch': 5.01}\n",
      "{'loss': 5.1727, 'learning_rate': 1.1056511056511057e-05, 'epoch': 5.02}\n",
      "{'loss': 5.0848, 'learning_rate': 1.1029211029211032e-05, 'epoch': 5.04}\n",
      "{'loss': 4.6723, 'learning_rate': 1.1001911001911003e-05, 'epoch': 5.05}\n",
      "{'loss': 4.9207, 'learning_rate': 1.0974610974610975e-05, 'epoch': 5.06}\n",
      "{'loss': 5.0801, 'learning_rate': 1.0947310947310947e-05, 'epoch': 5.07}\n",
      "{'loss': 4.9523, 'learning_rate': 1.0920010920010922e-05, 'epoch': 5.09}\n",
      "{'loss': 5.1326, 'learning_rate': 1.0892710892710894e-05, 'epoch': 5.1}\n",
      "{'loss': 5.4437, 'learning_rate': 1.0865410865410865e-05, 'epoch': 5.11}\n",
      "{'loss': 4.8119, 'learning_rate': 1.0838110838110839e-05, 'epoch': 5.12}\n",
      "{'loss': 4.4744, 'learning_rate': 1.0810810810810812e-05, 'epoch': 5.14}\n",
      "{'loss': 5.3004, 'learning_rate': 1.0783510783510784e-05, 'epoch': 5.15}\n",
      "{'loss': 5.0332, 'learning_rate': 1.0756210756210757e-05, 'epoch': 5.16}\n",
      "{'loss': 5.4246, 'learning_rate': 1.072891072891073e-05, 'epoch': 5.17}\n",
      "{'loss': 5.0914, 'learning_rate': 1.0701610701610702e-05, 'epoch': 5.18}\n",
      "{'loss': 5.8906, 'learning_rate': 1.0674310674310676e-05, 'epoch': 5.2}\n",
      "{'loss': 4.8498, 'learning_rate': 1.0647010647010647e-05, 'epoch': 5.21}\n",
      "{'loss': 5.0344, 'learning_rate': 1.061971061971062e-05, 'epoch': 5.22}\n",
      "{'loss': 4.9168, 'learning_rate': 1.0592410592410594e-05, 'epoch': 5.23}\n",
      "{'loss': 4.7699, 'learning_rate': 1.0565110565110566e-05, 'epoch': 5.25}\n",
      "{'loss': 5.227, 'learning_rate': 1.0537810537810537e-05, 'epoch': 5.26}\n",
      "{'loss': 5.0328, 'learning_rate': 1.0510510510510512e-05, 'epoch': 5.27}\n",
      "{'loss': 5.1236, 'learning_rate': 1.0483210483210484e-05, 'epoch': 5.28}\n",
      "{'loss': 4.9328, 'learning_rate': 1.0455910455910456e-05, 'epoch': 5.29}\n",
      "{'loss': 5.1309, 'learning_rate': 1.0428610428610428e-05, 'epoch': 5.31}\n",
      "{'loss': 4.8006, 'learning_rate': 1.0401310401310403e-05, 'epoch': 5.32}\n",
      "{'loss': 5.1529, 'learning_rate': 1.0374010374010374e-05, 'epoch': 5.33}\n",
      "{'loss': 5.5078, 'learning_rate': 1.0346710346710348e-05, 'epoch': 5.34}\n",
      "{'loss': 4.7971, 'learning_rate': 1.0319410319410321e-05, 'epoch': 5.36}\n",
      "{'loss': 5.284, 'learning_rate': 1.0292110292110293e-05, 'epoch': 5.37}\n",
      "{'loss': 5.0701, 'learning_rate': 1.0264810264810266e-05, 'epoch': 5.38}\n",
      "{'loss': 5.6098, 'learning_rate': 1.0237510237510238e-05, 'epoch': 5.39}\n",
      "{'loss': 5.1416, 'learning_rate': 1.0210210210210211e-05, 'epoch': 5.41}\n",
      "{'loss': 4.542, 'learning_rate': 1.0182910182910185e-05, 'epoch': 5.42}\n",
      "{'loss': 4.8666, 'learning_rate': 1.0155610155610156e-05, 'epoch': 5.43}\n",
      "{'loss': 5.0695, 'learning_rate': 1.0128310128310128e-05, 'epoch': 5.44}\n",
      "{'loss': 4.9705, 'learning_rate': 1.0101010101010103e-05, 'epoch': 5.45}\n",
      "{'loss': 4.7152, 'learning_rate': 1.0073710073710075e-05, 'epoch': 5.47}\n",
      "{'loss': 4.9111, 'learning_rate': 1.0046410046410046e-05, 'epoch': 5.48}\n",
      "{'loss': 5.084, 'learning_rate': 1.0019110019110021e-05, 'epoch': 5.49}\n",
      "{'loss': 4.9873, 'learning_rate': 9.991809991809993e-06, 'epoch': 5.5}\n",
      "{'loss': 4.9777, 'learning_rate': 9.964509964509965e-06, 'epoch': 5.52}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-4500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9736, 'learning_rate': 9.937209937209938e-06, 'epoch': 5.53}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-4500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-3500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0348, 'learning_rate': 9.90990990990991e-06, 'epoch': 5.54}\n",
      "{'loss': 5.2871, 'learning_rate': 9.882609882609883e-06, 'epoch': 5.55}\n",
      "{'loss': 5.1373, 'learning_rate': 9.855309855309857e-06, 'epoch': 5.57}\n",
      "{'loss': 5.2066, 'learning_rate': 9.828009828009828e-06, 'epoch': 5.58}\n",
      "{'loss': 5.2141, 'learning_rate': 9.800709800709802e-06, 'epoch': 5.59}\n",
      "{'loss': 5.2354, 'learning_rate': 9.773409773409775e-06, 'epoch': 5.6}\n",
      "{'loss': 4.9994, 'learning_rate': 9.746109746109747e-06, 'epoch': 5.61}\n",
      "{'loss': 4.918, 'learning_rate': 9.71880971880972e-06, 'epoch': 5.63}\n",
      "{'loss': 4.6516, 'learning_rate': 9.691509691509692e-06, 'epoch': 5.64}\n",
      "{'loss': 5.316, 'learning_rate': 9.664209664209665e-06, 'epoch': 5.65}\n",
      "{'loss': 5.0049, 'learning_rate': 9.636909636909639e-06, 'epoch': 5.66}\n",
      "{'loss': 4.9471, 'learning_rate': 9.60960960960961e-06, 'epoch': 5.68}\n",
      "{'loss': 5.0221, 'learning_rate': 9.582309582309584e-06, 'epoch': 5.69}\n",
      "{'loss': 5.2291, 'learning_rate': 9.555009555009555e-06, 'epoch': 5.7}\n",
      "{'loss': 5.0426, 'learning_rate': 9.527709527709529e-06, 'epoch': 5.71}\n",
      "{'loss': 4.8613, 'learning_rate': 9.5004095004095e-06, 'epoch': 5.72}\n",
      "{'loss': 4.6117, 'learning_rate': 9.473109473109474e-06, 'epoch': 5.74}\n",
      "{'loss': 4.9367, 'learning_rate': 9.445809445809446e-06, 'epoch': 5.75}\n",
      "{'loss': 5.0168, 'learning_rate': 9.418509418509419e-06, 'epoch': 5.76}\n",
      "{'loss': 4.7402, 'learning_rate': 9.391209391209392e-06, 'epoch': 5.77}\n",
      "{'loss': 4.799, 'learning_rate': 9.363909363909364e-06, 'epoch': 5.79}\n",
      "{'loss': 4.9699, 'learning_rate': 9.336609336609337e-06, 'epoch': 5.8}\n",
      "{'loss': 5.375, 'learning_rate': 9.30930930930931e-06, 'epoch': 5.81}\n",
      "{'loss': 5.2607, 'learning_rate': 9.282009282009284e-06, 'epoch': 5.82}\n",
      "{'loss': 5.1777, 'learning_rate': 9.254709254709256e-06, 'epoch': 5.84}\n",
      "{'loss': 4.6961, 'learning_rate': 9.22740922740923e-06, 'epoch': 5.85}\n",
      "{'loss': 5.1678, 'learning_rate': 9.200109200109201e-06, 'epoch': 5.86}\n",
      "{'loss': 5.1684, 'learning_rate': 9.172809172809174e-06, 'epoch': 5.87}\n",
      "{'loss': 5.2824, 'learning_rate': 9.145509145509146e-06, 'epoch': 5.88}\n",
      "{'loss': 4.8012, 'learning_rate': 9.11820911820912e-06, 'epoch': 5.9}\n",
      "{'loss': 4.876, 'learning_rate': 9.090909090909091e-06, 'epoch': 5.91}\n",
      "{'loss': 4.474, 'learning_rate': 9.063609063609064e-06, 'epoch': 5.92}\n",
      "{'loss': 4.6105, 'learning_rate': 9.036309036309036e-06, 'epoch': 5.93}\n",
      "{'loss': 4.9973, 'learning_rate': 9.00900900900901e-06, 'epoch': 5.95}\n",
      "{'loss': 5.2996, 'learning_rate': 8.981708981708981e-06, 'epoch': 5.96}\n",
      "{'loss': 4.9434, 'learning_rate': 8.954408954408955e-06, 'epoch': 5.97}\n",
      "{'loss': 4.8027, 'learning_rate': 8.927108927108928e-06, 'epoch': 5.98}\n",
      "{'loss': 5.0891, 'learning_rate': 8.899808899808901e-06, 'epoch': 6.0}\n",
      "{'loss': 5.0414, 'learning_rate': 8.872508872508873e-06, 'epoch': 6.01}\n",
      "{'loss': 4.9859, 'learning_rate': 8.845208845208846e-06, 'epoch': 6.02}\n",
      "{'loss': 4.7482, 'learning_rate': 8.81790881790882e-06, 'epoch': 6.03}\n",
      "{'loss': 4.8908, 'learning_rate': 8.790608790608791e-06, 'epoch': 6.04}\n",
      "{'loss': 4.85, 'learning_rate': 8.763308763308765e-06, 'epoch': 6.06}\n",
      "{'loss': 4.8844, 'learning_rate': 8.736008736008737e-06, 'epoch': 6.07}\n",
      "{'loss': 5.5848, 'learning_rate': 8.70870870870871e-06, 'epoch': 6.08}\n",
      "{'loss': 4.5551, 'learning_rate': 8.681408681408682e-06, 'epoch': 6.09}\n",
      "{'loss': 4.9611, 'learning_rate': 8.654108654108655e-06, 'epoch': 6.11}\n",
      "{'loss': 4.5844, 'learning_rate': 8.626808626808627e-06, 'epoch': 6.12}\n",
      "{'loss': 4.7758, 'learning_rate': 8.5995085995086e-06, 'epoch': 6.13}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-5000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.7141, 'learning_rate': 8.572208572208572e-06, 'epoch': 6.14}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-5000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-4000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.201, 'learning_rate': 8.544908544908545e-06, 'epoch': 6.15}\n",
      "{'loss': 4.4113, 'learning_rate': 8.517608517608518e-06, 'epoch': 6.17}\n",
      "{'loss': 5.1268, 'learning_rate': 8.49030849030849e-06, 'epoch': 6.18}\n",
      "{'loss': 4.8824, 'learning_rate': 8.463008463008464e-06, 'epoch': 6.19}\n",
      "{'loss': 4.9555, 'learning_rate': 8.435708435708437e-06, 'epoch': 6.2}\n",
      "{'loss': 4.9744, 'learning_rate': 8.408408408408409e-06, 'epoch': 6.22}\n",
      "{'loss': 4.9805, 'learning_rate': 8.381108381108382e-06, 'epoch': 6.23}\n",
      "{'loss': 5.0613, 'learning_rate': 8.353808353808355e-06, 'epoch': 6.24}\n",
      "{'loss': 5.0078, 'learning_rate': 8.326508326508327e-06, 'epoch': 6.25}\n",
      "{'loss': 4.5564, 'learning_rate': 8.2992082992083e-06, 'epoch': 6.27}\n",
      "{'loss': 5.009, 'learning_rate': 8.271908271908272e-06, 'epoch': 6.28}\n",
      "{'loss': 4.8566, 'learning_rate': 8.244608244608246e-06, 'epoch': 6.29}\n",
      "{'loss': 4.7586, 'learning_rate': 8.217308217308217e-06, 'epoch': 6.3}\n",
      "{'loss': 4.6697, 'learning_rate': 8.19000819000819e-06, 'epoch': 6.31}\n",
      "{'loss': 5.4508, 'learning_rate': 8.162708162708164e-06, 'epoch': 6.33}\n",
      "{'loss': 4.6902, 'learning_rate': 8.135408135408136e-06, 'epoch': 6.34}\n",
      "{'loss': 4.8615, 'learning_rate': 8.108108108108109e-06, 'epoch': 6.35}\n",
      "{'loss': 4.6887, 'learning_rate': 8.08080808080808e-06, 'epoch': 6.36}\n",
      "{'loss': 5.1424, 'learning_rate': 8.053508053508054e-06, 'epoch': 6.38}\n",
      "{'loss': 5.0051, 'learning_rate': 8.026208026208026e-06, 'epoch': 6.39}\n",
      "{'loss': 5.192, 'learning_rate': 7.998907998908e-06, 'epoch': 6.4}\n",
      "{'loss': 4.8096, 'learning_rate': 7.971607971607973e-06, 'epoch': 6.41}\n",
      "{'loss': 5.1219, 'learning_rate': 7.944307944307946e-06, 'epoch': 6.43}\n",
      "{'loss': 5.1801, 'learning_rate': 7.917007917007918e-06, 'epoch': 6.44}\n",
      "{'loss': 4.8207, 'learning_rate': 7.889707889707891e-06, 'epoch': 6.45}\n",
      "{'loss': 4.9918, 'learning_rate': 7.862407862407863e-06, 'epoch': 6.46}\n",
      "{'loss': 4.5451, 'learning_rate': 7.835107835107836e-06, 'epoch': 6.47}\n",
      "{'loss': 5.3527, 'learning_rate': 7.807807807807808e-06, 'epoch': 6.49}\n",
      "{'loss': 5.2016, 'learning_rate': 7.780507780507781e-06, 'epoch': 6.5}\n",
      "{'loss': 5.2723, 'learning_rate': 7.753207753207755e-06, 'epoch': 6.51}\n",
      "{'loss': 5.341, 'learning_rate': 7.725907725907726e-06, 'epoch': 6.52}\n",
      "{'loss': 5.2896, 'learning_rate': 7.6986076986077e-06, 'epoch': 6.54}\n",
      "{'loss': 4.5252, 'learning_rate': 7.671307671307671e-06, 'epoch': 6.55}\n",
      "{'loss': 4.9016, 'learning_rate': 7.644007644007645e-06, 'epoch': 6.56}\n",
      "{'loss': 4.9779, 'learning_rate': 7.616707616707617e-06, 'epoch': 6.57}\n",
      "{'loss': 4.7473, 'learning_rate': 7.589407589407591e-06, 'epoch': 6.58}\n",
      "{'loss': 4.8951, 'learning_rate': 7.562107562107562e-06, 'epoch': 6.6}\n",
      "{'loss': 4.9789, 'learning_rate': 7.534807534807536e-06, 'epoch': 6.61}\n",
      "{'loss': 4.7695, 'learning_rate': 7.507507507507507e-06, 'epoch': 6.62}\n",
      "{'loss': 4.8178, 'learning_rate': 7.480207480207481e-06, 'epoch': 6.63}\n",
      "{'loss': 4.8543, 'learning_rate': 7.452907452907453e-06, 'epoch': 6.65}\n",
      "{'loss': 5.0059, 'learning_rate': 7.425607425607426e-06, 'epoch': 6.66}\n",
      "{'loss': 5.2938, 'learning_rate': 7.398307398307399e-06, 'epoch': 6.67}\n",
      "{'loss': 4.4334, 'learning_rate': 7.371007371007372e-06, 'epoch': 6.68}\n",
      "{'loss': 5.107, 'learning_rate': 7.343707343707345e-06, 'epoch': 6.7}\n",
      "{'loss': 4.5643, 'learning_rate': 7.316407316407317e-06, 'epoch': 6.71}\n",
      "{'loss': 5.1219, 'learning_rate': 7.28910728910729e-06, 'epoch': 6.72}\n",
      "{'loss': 5.0656, 'learning_rate': 7.261807261807262e-06, 'epoch': 6.73}\n",
      "{'loss': 4.8705, 'learning_rate': 7.234507234507235e-06, 'epoch': 6.74}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-5500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.032, 'learning_rate': 7.207207207207208e-06, 'epoch': 6.76}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-5500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-4500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.2883, 'learning_rate': 7.17990717990718e-06, 'epoch': 6.77}\n",
      "{'loss': 5.1148, 'learning_rate': 7.152607152607153e-06, 'epoch': 6.78}\n",
      "{'loss': 4.7234, 'learning_rate': 7.125307125307126e-06, 'epoch': 6.79}\n",
      "{'loss': 5.0787, 'learning_rate': 7.098007098007098e-06, 'epoch': 6.81}\n",
      "{'loss': 4.4553, 'learning_rate': 7.070707070707071e-06, 'epoch': 6.82}\n",
      "{'loss': 5.3098, 'learning_rate': 7.043407043407045e-06, 'epoch': 6.83}\n",
      "{'loss': 5.1744, 'learning_rate': 7.016107016107016e-06, 'epoch': 6.84}\n",
      "{'loss': 4.985, 'learning_rate': 6.98880698880699e-06, 'epoch': 6.86}\n",
      "{'loss': 4.883, 'learning_rate': 6.961506961506962e-06, 'epoch': 6.87}\n",
      "{'loss': 4.7674, 'learning_rate': 6.934206934206935e-06, 'epoch': 6.88}\n",
      "{'loss': 4.8146, 'learning_rate': 6.906906906906907e-06, 'epoch': 6.89}\n",
      "{'loss': 5.0488, 'learning_rate': 6.879606879606881e-06, 'epoch': 6.9}\n",
      "{'loss': 5.1303, 'learning_rate': 6.852306852306852e-06, 'epoch': 6.92}\n",
      "{'loss': 4.666, 'learning_rate': 6.825006825006826e-06, 'epoch': 6.93}\n",
      "{'loss': 4.9811, 'learning_rate': 6.7977067977067975e-06, 'epoch': 6.94}\n",
      "{'loss': 4.8488, 'learning_rate': 6.770406770406771e-06, 'epoch': 6.95}\n",
      "{'loss': 4.8426, 'learning_rate': 6.743106743106743e-06, 'epoch': 6.97}\n",
      "{'loss': 4.9877, 'learning_rate': 6.715806715806717e-06, 'epoch': 6.98}\n",
      "{'loss': 4.7979, 'learning_rate': 6.688506688506689e-06, 'epoch': 6.99}\n",
      "{'loss': 4.4604, 'learning_rate': 6.661206661206662e-06, 'epoch': 7.0}\n",
      "{'loss': 5.068, 'learning_rate': 6.633906633906635e-06, 'epoch': 7.01}\n",
      "{'loss': 4.7633, 'learning_rate': 6.606606606606607e-06, 'epoch': 7.03}\n",
      "{'loss': 5.3465, 'learning_rate': 6.57930657930658e-06, 'epoch': 7.04}\n",
      "{'loss': 5.2145, 'learning_rate': 6.552006552006552e-06, 'epoch': 7.05}\n",
      "{'loss': 5.2023, 'learning_rate': 6.524706524706525e-06, 'epoch': 7.06}\n",
      "{'loss': 5.1332, 'learning_rate': 6.497406497406498e-06, 'epoch': 7.08}\n",
      "{'loss': 4.9379, 'learning_rate': 6.470106470106471e-06, 'epoch': 7.09}\n",
      "{'loss': 4.8268, 'learning_rate': 6.442806442806443e-06, 'epoch': 7.1}\n",
      "{'loss': 4.6387, 'learning_rate': 6.415506415506416e-06, 'epoch': 7.11}\n",
      "{'loss': 4.9021, 'learning_rate': 6.388206388206388e-06, 'epoch': 7.13}\n",
      "{'loss': 5.1732, 'learning_rate': 6.360906360906361e-06, 'epoch': 7.14}\n",
      "{'loss': 5.0504, 'learning_rate': 6.333606333606334e-06, 'epoch': 7.15}\n",
      "{'loss': 4.5955, 'learning_rate': 6.3063063063063065e-06, 'epoch': 7.16}\n",
      "{'loss': 4.66, 'learning_rate': 6.27900627900628e-06, 'epoch': 7.17}\n",
      "{'loss': 5.423, 'learning_rate': 6.251706251706252e-06, 'epoch': 7.19}\n",
      "{'loss': 4.8105, 'learning_rate': 6.224406224406225e-06, 'epoch': 7.2}\n",
      "{'loss': 4.9316, 'learning_rate': 6.1971061971061975e-06, 'epoch': 7.21}\n",
      "{'loss': 4.4627, 'learning_rate': 6.169806169806171e-06, 'epoch': 7.22}\n",
      "{'loss': 4.7596, 'learning_rate': 6.1425061425061425e-06, 'epoch': 7.24}\n",
      "{'loss': 4.9824, 'learning_rate': 6.115206115206116e-06, 'epoch': 7.25}\n",
      "{'loss': 5.1781, 'learning_rate': 6.0879060879060884e-06, 'epoch': 7.26}\n",
      "{'loss': 5.0789, 'learning_rate': 6.060606060606061e-06, 'epoch': 7.27}\n",
      "{'loss': 4.5285, 'learning_rate': 6.0333060333060335e-06, 'epoch': 7.29}\n",
      "{'loss': 5.0682, 'learning_rate': 6.006006006006007e-06, 'epoch': 7.3}\n",
      "{'loss': 4.7605, 'learning_rate': 5.9787059787059786e-06, 'epoch': 7.31}\n",
      "{'loss': 5.0738, 'learning_rate': 5.951405951405952e-06, 'epoch': 7.32}\n",
      "{'loss': 4.5973, 'learning_rate': 5.924105924105925e-06, 'epoch': 7.33}\n",
      "{'loss': 4.9303, 'learning_rate': 5.896805896805897e-06, 'epoch': 7.35}\n",
      "{'loss': 4.8545, 'learning_rate': 5.86950586950587e-06, 'epoch': 7.36}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-6000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.1469, 'learning_rate': 5.842205842205843e-06, 'epoch': 7.37}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-6000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-5000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.241, 'learning_rate': 5.8149058149058155e-06, 'epoch': 7.38}\n",
      "{'loss': 4.3453, 'learning_rate': 5.787605787605788e-06, 'epoch': 7.4}\n",
      "{'loss': 4.943, 'learning_rate': 5.760305760305761e-06, 'epoch': 7.41}\n",
      "{'loss': 5.0551, 'learning_rate': 5.733005733005733e-06, 'epoch': 7.42}\n",
      "{'loss': 4.9742, 'learning_rate': 5.7057057057057065e-06, 'epoch': 7.43}\n",
      "{'loss': 4.9807, 'learning_rate': 5.678405678405678e-06, 'epoch': 7.44}\n",
      "{'loss': 4.9857, 'learning_rate': 5.6511056511056515e-06, 'epoch': 7.46}\n",
      "{'loss': 4.8121, 'learning_rate': 5.623805623805624e-06, 'epoch': 7.47}\n",
      "{'loss': 4.6758, 'learning_rate': 5.596505596505597e-06, 'epoch': 7.48}\n",
      "{'loss': 4.7604, 'learning_rate': 5.56920556920557e-06, 'epoch': 7.49}\n",
      "{'loss': 4.9824, 'learning_rate': 5.5419055419055425e-06, 'epoch': 7.51}\n",
      "{'loss': 4.6535, 'learning_rate': 5.514605514605516e-06, 'epoch': 7.52}\n",
      "{'loss': 4.9008, 'learning_rate': 5.487305487305488e-06, 'epoch': 7.53}\n",
      "{'loss': 4.583, 'learning_rate': 5.460005460005461e-06, 'epoch': 7.54}\n",
      "{'loss': 4.8709, 'learning_rate': 5.432705432705433e-06, 'epoch': 7.56}\n",
      "{'loss': 4.9369, 'learning_rate': 5.405405405405406e-06, 'epoch': 7.57}\n",
      "{'loss': 5.0678, 'learning_rate': 5.3781053781053786e-06, 'epoch': 7.58}\n",
      "{'loss': 4.9029, 'learning_rate': 5.350805350805351e-06, 'epoch': 7.59}\n",
      "{'loss': 5.2035, 'learning_rate': 5.323505323505324e-06, 'epoch': 7.6}\n",
      "{'loss': 4.8887, 'learning_rate': 5.296205296205297e-06, 'epoch': 7.62}\n",
      "{'loss': 5.4266, 'learning_rate': 5.268905268905269e-06, 'epoch': 7.63}\n",
      "{'loss': 4.7553, 'learning_rate': 5.241605241605242e-06, 'epoch': 7.64}\n",
      "{'loss': 4.9195, 'learning_rate': 5.214305214305214e-06, 'epoch': 7.65}\n",
      "{'loss': 5.1297, 'learning_rate': 5.187005187005187e-06, 'epoch': 7.67}\n",
      "{'loss': 4.5967, 'learning_rate': 5.1597051597051605e-06, 'epoch': 7.68}\n",
      "{'loss': 4.7258, 'learning_rate': 5.132405132405133e-06, 'epoch': 7.69}\n",
      "{'loss': 4.5148, 'learning_rate': 5.105105105105106e-06, 'epoch': 7.7}\n",
      "{'loss': 4.8246, 'learning_rate': 5.077805077805078e-06, 'epoch': 7.71}\n",
      "{'loss': 4.7977, 'learning_rate': 5.0505050505050515e-06, 'epoch': 7.73}\n",
      "{'loss': 4.8496, 'learning_rate': 5.023205023205023e-06, 'epoch': 7.74}\n",
      "{'loss': 4.8148, 'learning_rate': 4.995904995904997e-06, 'epoch': 7.75}\n",
      "{'loss': 4.8434, 'learning_rate': 4.968604968604969e-06, 'epoch': 7.76}\n",
      "{'loss': 4.8516, 'learning_rate': 4.941304941304942e-06, 'epoch': 7.78}\n",
      "{'loss': 4.8051, 'learning_rate': 4.914004914004914e-06, 'epoch': 7.79}\n",
      "{'loss': 4.6676, 'learning_rate': 4.8867048867048876e-06, 'epoch': 7.8}\n",
      "{'loss': 4.6301, 'learning_rate': 4.85940485940486e-06, 'epoch': 7.81}\n",
      "{'loss': 4.4205, 'learning_rate': 4.832104832104833e-06, 'epoch': 7.83}\n",
      "{'loss': 4.7785, 'learning_rate': 4.804804804804805e-06, 'epoch': 7.84}\n",
      "{'loss': 5.1973, 'learning_rate': 4.777504777504778e-06, 'epoch': 7.85}\n",
      "{'loss': 4.8061, 'learning_rate': 4.75020475020475e-06, 'epoch': 7.86}\n",
      "{'loss': 4.5193, 'learning_rate': 4.722904722904723e-06, 'epoch': 7.87}\n",
      "{'loss': 4.7832, 'learning_rate': 4.695604695604696e-06, 'epoch': 7.89}\n",
      "{'loss': 5.1795, 'learning_rate': 4.668304668304669e-06, 'epoch': 7.9}\n",
      "{'loss': 4.6137, 'learning_rate': 4.641004641004642e-06, 'epoch': 7.91}\n",
      "{'loss': 4.6986, 'learning_rate': 4.613704613704615e-06, 'epoch': 7.92}\n",
      "{'loss': 4.8713, 'learning_rate': 4.586404586404587e-06, 'epoch': 7.94}\n",
      "{'loss': 4.7195, 'learning_rate': 4.55910455910456e-06, 'epoch': 7.95}\n",
      "{'loss': 4.7359, 'learning_rate': 4.531804531804532e-06, 'epoch': 7.96}\n",
      "{'loss': 5.0566, 'learning_rate': 4.504504504504505e-06, 'epoch': 7.97}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-6500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8801, 'learning_rate': 4.477204477204477e-06, 'epoch': 7.99}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-6500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-5500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.8295, 'learning_rate': 4.449904449904451e-06, 'epoch': 8.0}\n",
      "{'loss': 5.093, 'learning_rate': 4.422604422604423e-06, 'epoch': 8.01}\n",
      "{'loss': 4.9922, 'learning_rate': 4.395304395304396e-06, 'epoch': 8.02}\n",
      "{'loss': 4.7334, 'learning_rate': 4.368004368004368e-06, 'epoch': 8.03}\n",
      "{'loss': 5.1297, 'learning_rate': 4.340704340704341e-06, 'epoch': 8.05}\n",
      "{'loss': 5.0297, 'learning_rate': 4.313404313404313e-06, 'epoch': 8.06}\n",
      "{'loss': 5.0107, 'learning_rate': 4.286104286104286e-06, 'epoch': 8.07}\n",
      "{'loss': 4.8109, 'learning_rate': 4.258804258804259e-06, 'epoch': 8.08}\n",
      "{'loss': 4.4176, 'learning_rate': 4.231504231504232e-06, 'epoch': 8.1}\n",
      "{'loss': 4.5625, 'learning_rate': 4.204204204204204e-06, 'epoch': 8.11}\n",
      "{'loss': 4.8842, 'learning_rate': 4.176904176904178e-06, 'epoch': 8.12}\n",
      "{'loss': 5.073, 'learning_rate': 4.14960414960415e-06, 'epoch': 8.13}\n",
      "{'loss': 5.0264, 'learning_rate': 4.122304122304123e-06, 'epoch': 8.14}\n",
      "{'loss': 4.8195, 'learning_rate': 4.095004095004095e-06, 'epoch': 8.16}\n",
      "{'loss': 4.9039, 'learning_rate': 4.067704067704068e-06, 'epoch': 8.17}\n",
      "{'loss': 4.6666, 'learning_rate': 4.04040404040404e-06, 'epoch': 8.18}\n",
      "{'loss': 4.7295, 'learning_rate': 4.013104013104013e-06, 'epoch': 8.19}\n",
      "{'loss': 4.7385, 'learning_rate': 3.985803985803986e-06, 'epoch': 8.21}\n",
      "{'loss': 5.0586, 'learning_rate': 3.958503958503959e-06, 'epoch': 8.22}\n",
      "{'loss': 4.893, 'learning_rate': 3.931203931203931e-06, 'epoch': 8.23}\n",
      "{'loss': 5.1238, 'learning_rate': 3.903903903903904e-06, 'epoch': 8.24}\n",
      "{'loss': 5.0424, 'learning_rate': 3.876603876603877e-06, 'epoch': 8.26}\n",
      "{'loss': 4.5318, 'learning_rate': 3.84930384930385e-06, 'epoch': 8.27}\n",
      "{'loss': 5.1594, 'learning_rate': 3.822003822003822e-06, 'epoch': 8.28}\n",
      "{'loss': 4.9785, 'learning_rate': 3.7947037947037953e-06, 'epoch': 8.29}\n",
      "{'loss': 4.8891, 'learning_rate': 3.767403767403768e-06, 'epoch': 8.3}\n",
      "{'loss': 4.6676, 'learning_rate': 3.7401037401037404e-06, 'epoch': 8.32}\n",
      "{'loss': 4.7348, 'learning_rate': 3.712803712803713e-06, 'epoch': 8.33}\n",
      "{'loss': 4.7209, 'learning_rate': 3.685503685503686e-06, 'epoch': 8.34}\n",
      "{'loss': 5.0699, 'learning_rate': 3.6582036582036584e-06, 'epoch': 8.35}\n",
      "{'loss': 4.6525, 'learning_rate': 3.630903630903631e-06, 'epoch': 8.37}\n",
      "{'loss': 5.0693, 'learning_rate': 3.603603603603604e-06, 'epoch': 8.38}\n",
      "{'loss': 4.9908, 'learning_rate': 3.5763035763035764e-06, 'epoch': 8.39}\n",
      "{'loss': 4.707, 'learning_rate': 3.549003549003549e-06, 'epoch': 8.4}\n",
      "{'loss': 4.3576, 'learning_rate': 3.5217035217035223e-06, 'epoch': 8.42}\n",
      "{'loss': 4.7861, 'learning_rate': 3.494403494403495e-06, 'epoch': 8.43}\n",
      "{'loss': 4.9727, 'learning_rate': 3.4671034671034674e-06, 'epoch': 8.44}\n",
      "{'loss': 4.9557, 'learning_rate': 3.4398034398034404e-06, 'epoch': 8.45}\n",
      "{'loss': 5.2691, 'learning_rate': 3.412503412503413e-06, 'epoch': 8.46}\n",
      "{'loss': 4.9617, 'learning_rate': 3.3852033852033854e-06, 'epoch': 8.48}\n",
      "{'loss': 5.5578, 'learning_rate': 3.3579033579033584e-06, 'epoch': 8.49}\n",
      "{'loss': 4.735, 'learning_rate': 3.330603330603331e-06, 'epoch': 8.5}\n",
      "{'loss': 4.8287, 'learning_rate': 3.3033033033033035e-06, 'epoch': 8.51}\n",
      "{'loss': 4.8141, 'learning_rate': 3.276003276003276e-06, 'epoch': 8.53}\n",
      "{'loss': 5.209, 'learning_rate': 3.248703248703249e-06, 'epoch': 8.54}\n",
      "{'loss': 5.0535, 'learning_rate': 3.2214032214032215e-06, 'epoch': 8.55}\n",
      "{'loss': 4.9217, 'learning_rate': 3.194103194103194e-06, 'epoch': 8.56}\n",
      "{'loss': 4.5668, 'learning_rate': 3.166803166803167e-06, 'epoch': 8.57}\n",
      "{'loss': 4.8693, 'learning_rate': 3.13950313950314e-06, 'epoch': 8.59}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-7000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6361, 'learning_rate': 3.1122031122031125e-06, 'epoch': 8.6}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-7000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-6000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.7561, 'learning_rate': 3.0849030849030854e-06, 'epoch': 8.61}\n",
      "{'loss': 4.9262, 'learning_rate': 3.057603057603058e-06, 'epoch': 8.62}\n",
      "{'loss': 4.7322, 'learning_rate': 3.0303030303030305e-06, 'epoch': 8.64}\n",
      "{'loss': 4.8152, 'learning_rate': 3.0030030030030034e-06, 'epoch': 8.65}\n",
      "{'loss': 4.4463, 'learning_rate': 2.975702975702976e-06, 'epoch': 8.66}\n",
      "{'loss': 4.5986, 'learning_rate': 2.9484029484029485e-06, 'epoch': 8.67}\n",
      "{'loss': 5.1449, 'learning_rate': 2.9211029211029215e-06, 'epoch': 8.69}\n",
      "{'loss': 4.7652, 'learning_rate': 2.893802893802894e-06, 'epoch': 8.7}\n",
      "{'loss': 5.2344, 'learning_rate': 2.8665028665028665e-06, 'epoch': 8.71}\n",
      "{'loss': 4.6113, 'learning_rate': 2.839202839202839e-06, 'epoch': 8.72}\n",
      "{'loss': 4.9936, 'learning_rate': 2.811902811902812e-06, 'epoch': 8.73}\n",
      "{'loss': 4.6564, 'learning_rate': 2.784602784602785e-06, 'epoch': 8.75}\n",
      "{'loss': 5.159, 'learning_rate': 2.757302757302758e-06, 'epoch': 8.76}\n",
      "{'loss': 4.6332, 'learning_rate': 2.7300027300027305e-06, 'epoch': 8.77}\n",
      "{'loss': 5.0305, 'learning_rate': 2.702702702702703e-06, 'epoch': 8.78}\n",
      "{'loss': 5.1547, 'learning_rate': 2.6754026754026755e-06, 'epoch': 8.8}\n",
      "{'loss': 5.0965, 'learning_rate': 2.6481026481026485e-06, 'epoch': 8.81}\n",
      "{'loss': 4.9689, 'learning_rate': 2.620802620802621e-06, 'epoch': 8.82}\n",
      "{'loss': 4.5961, 'learning_rate': 2.5935025935025936e-06, 'epoch': 8.83}\n",
      "{'loss': 4.4832, 'learning_rate': 2.5662025662025665e-06, 'epoch': 8.85}\n",
      "{'loss': 4.9615, 'learning_rate': 2.538902538902539e-06, 'epoch': 8.86}\n",
      "{'loss': 4.7867, 'learning_rate': 2.5116025116025116e-06, 'epoch': 8.87}\n",
      "{'loss': 4.6652, 'learning_rate': 2.4843024843024846e-06, 'epoch': 8.88}\n",
      "{'loss': 4.2301, 'learning_rate': 2.457002457002457e-06, 'epoch': 8.89}\n",
      "{'loss': 4.5227, 'learning_rate': 2.42970242970243e-06, 'epoch': 8.91}\n",
      "{'loss': 4.9047, 'learning_rate': 2.4024024024024026e-06, 'epoch': 8.92}\n",
      "{'loss': 4.8438, 'learning_rate': 2.375102375102375e-06, 'epoch': 8.93}\n",
      "{'loss': 5.0635, 'learning_rate': 2.347802347802348e-06, 'epoch': 8.94}\n",
      "{'loss': 4.9254, 'learning_rate': 2.320502320502321e-06, 'epoch': 8.96}\n",
      "{'loss': 4.6564, 'learning_rate': 2.2932022932022936e-06, 'epoch': 8.97}\n",
      "{'loss': 4.8246, 'learning_rate': 2.265902265902266e-06, 'epoch': 8.98}\n",
      "{'loss': 4.9313, 'learning_rate': 2.2386022386022386e-06, 'epoch': 8.99}\n",
      "{'loss': 4.9246, 'learning_rate': 2.2113022113022116e-06, 'epoch': 9.0}\n",
      "{'loss': 5.0094, 'learning_rate': 2.184002184002184e-06, 'epoch': 9.02}\n",
      "{'loss': 4.8486, 'learning_rate': 2.1567021567021567e-06, 'epoch': 9.03}\n",
      "{'loss': 4.7016, 'learning_rate': 2.1294021294021296e-06, 'epoch': 9.04}\n",
      "{'loss': 4.716, 'learning_rate': 2.102102102102102e-06, 'epoch': 9.05}\n",
      "{'loss': 5.0312, 'learning_rate': 2.074802074802075e-06, 'epoch': 9.07}\n",
      "{'loss': 4.8469, 'learning_rate': 2.0475020475020476e-06, 'epoch': 9.08}\n",
      "{'loss': 4.9547, 'learning_rate': 2.02020202020202e-06, 'epoch': 9.09}\n",
      "{'loss': 4.5742, 'learning_rate': 1.992901992901993e-06, 'epoch': 9.1}\n",
      "{'loss': 5.3242, 'learning_rate': 1.9656019656019657e-06, 'epoch': 9.12}\n",
      "{'loss': 4.9207, 'learning_rate': 1.9383019383019386e-06, 'epoch': 9.13}\n",
      "{'loss': 4.9887, 'learning_rate': 1.911001911001911e-06, 'epoch': 9.14}\n",
      "{'loss': 4.8301, 'learning_rate': 1.883701883701884e-06, 'epoch': 9.15}\n",
      "{'loss': 4.882, 'learning_rate': 1.8564018564018564e-06, 'epoch': 9.16}\n",
      "{'loss': 4.6732, 'learning_rate': 1.8291018291018292e-06, 'epoch': 9.18}\n",
      "{'loss': 4.7062, 'learning_rate': 1.801801801801802e-06, 'epoch': 9.19}\n",
      "{'loss': 4.9549, 'learning_rate': 1.7745017745017745e-06, 'epoch': 9.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-7500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0066, 'learning_rate': 1.7472017472017474e-06, 'epoch': 9.21}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-7500/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-6500] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 5.0762, 'learning_rate': 1.7199017199017202e-06, 'epoch': 9.23}\n",
      "{'loss': 4.8764, 'learning_rate': 1.6926016926016927e-06, 'epoch': 9.24}\n",
      "{'loss': 4.8078, 'learning_rate': 1.6653016653016655e-06, 'epoch': 9.25}\n",
      "{'loss': 4.7547, 'learning_rate': 1.638001638001638e-06, 'epoch': 9.26}\n",
      "{'loss': 4.7928, 'learning_rate': 1.6107016107016107e-06, 'epoch': 9.28}\n",
      "{'loss': 4.9145, 'learning_rate': 1.5834015834015835e-06, 'epoch': 9.29}\n",
      "{'loss': 4.8777, 'learning_rate': 1.5561015561015562e-06, 'epoch': 9.3}\n",
      "{'loss': 4.8738, 'learning_rate': 1.528801528801529e-06, 'epoch': 9.31}\n",
      "{'loss': 4.9338, 'learning_rate': 1.5015015015015017e-06, 'epoch': 9.32}\n",
      "{'loss': 4.9049, 'learning_rate': 1.4742014742014743e-06, 'epoch': 9.34}\n",
      "{'loss': 4.8363, 'learning_rate': 1.446901446901447e-06, 'epoch': 9.35}\n",
      "{'loss': 4.7764, 'learning_rate': 1.4196014196014195e-06, 'epoch': 9.36}\n",
      "{'loss': 4.8486, 'learning_rate': 1.3923013923013925e-06, 'epoch': 9.37}\n",
      "{'loss': 5.1439, 'learning_rate': 1.3650013650013652e-06, 'epoch': 9.39}\n",
      "{'loss': 5.3152, 'learning_rate': 1.3377013377013378e-06, 'epoch': 9.4}\n",
      "{'loss': 4.6623, 'learning_rate': 1.3104013104013105e-06, 'epoch': 9.41}\n",
      "{'loss': 4.7475, 'learning_rate': 1.2831012831012833e-06, 'epoch': 9.42}\n",
      "{'loss': 4.5514, 'learning_rate': 1.2558012558012558e-06, 'epoch': 9.43}\n",
      "{'loss': 4.576, 'learning_rate': 1.2285012285012285e-06, 'epoch': 9.45}\n",
      "{'loss': 5.0068, 'learning_rate': 1.2012012012012013e-06, 'epoch': 9.46}\n",
      "{'loss': 4.8625, 'learning_rate': 1.173901173901174e-06, 'epoch': 9.47}\n",
      "{'loss': 4.7279, 'learning_rate': 1.1466011466011468e-06, 'epoch': 9.48}\n",
      "{'loss': 4.9502, 'learning_rate': 1.1193011193011193e-06, 'epoch': 9.5}\n",
      "{'loss': 4.9891, 'learning_rate': 1.092001092001092e-06, 'epoch': 9.51}\n",
      "{'loss': 4.7619, 'learning_rate': 1.0647010647010648e-06, 'epoch': 9.52}\n",
      "{'loss': 4.6689, 'learning_rate': 1.0374010374010376e-06, 'epoch': 9.53}\n",
      "{'loss': 4.7619, 'learning_rate': 1.01010101010101e-06, 'epoch': 9.55}\n",
      "{'loss': 4.8859, 'learning_rate': 9.828009828009828e-07, 'epoch': 9.56}\n",
      "{'loss': 4.8975, 'learning_rate': 9.555009555009556e-07, 'epoch': 9.57}\n",
      "{'loss': 4.7039, 'learning_rate': 9.282009282009282e-07, 'epoch': 9.58}\n",
      "{'loss': 4.5563, 'learning_rate': 9.00900900900901e-07, 'epoch': 9.59}\n",
      "{'loss': 4.8404, 'learning_rate': 8.736008736008737e-07, 'epoch': 9.61}\n",
      "{'loss': 4.8098, 'learning_rate': 8.463008463008464e-07, 'epoch': 9.62}\n",
      "{'loss': 4.3885, 'learning_rate': 8.19000819000819e-07, 'epoch': 9.63}\n",
      "{'loss': 4.7883, 'learning_rate': 7.917007917007917e-07, 'epoch': 9.64}\n",
      "{'loss': 4.3908, 'learning_rate': 7.644007644007645e-07, 'epoch': 9.66}\n",
      "{'loss': 4.5395, 'learning_rate': 7.371007371007371e-07, 'epoch': 9.67}\n",
      "{'loss': 5.0711, 'learning_rate': 7.098007098007098e-07, 'epoch': 9.68}\n",
      "{'loss': 4.8957, 'learning_rate': 6.825006825006826e-07, 'epoch': 9.69}\n",
      "{'loss': 4.5594, 'learning_rate': 6.552006552006553e-07, 'epoch': 9.71}\n",
      "{'loss': 5.0281, 'learning_rate': 6.279006279006279e-07, 'epoch': 9.72}\n",
      "{'loss': 4.8883, 'learning_rate': 6.006006006006006e-07, 'epoch': 9.73}\n",
      "{'loss': 4.8324, 'learning_rate': 5.733005733005734e-07, 'epoch': 9.74}\n",
      "{'loss': 5.1766, 'learning_rate': 5.46000546000546e-07, 'epoch': 9.75}\n",
      "{'loss': 4.4811, 'learning_rate': 5.187005187005188e-07, 'epoch': 9.77}\n",
      "{'loss': 4.6939, 'learning_rate': 4.914004914004914e-07, 'epoch': 9.78}\n",
      "{'loss': 4.6891, 'learning_rate': 4.641004641004641e-07, 'epoch': 9.79}\n",
      "{'loss': 4.7156, 'learning_rate': 4.3680043680043686e-07, 'epoch': 9.8}\n",
      "{'loss': 4.8869, 'learning_rate': 4.095004095004095e-07, 'epoch': 9.82}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to gpt2-baby/checkpoint-8000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9395, 'learning_rate': 3.8220038220038224e-07, 'epoch': 9.83}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n",
      "  warnings.warn(\n",
      "-------------------- Device Allocation --------------------\n",
      "Token Embedding     --> IPU 0\n",
      "Position Embedding  --> IPU 0\n",
      "Layer 0             --> IPU 1\n",
      "Layer 1             --> IPU 1\n",
      "Layer 2             --> IPU 2\n",
      "Layer 3             --> IPU 2\n",
      "Layer 4             --> IPU 3\n",
      "Layer 5             --> IPU 3\n",
      "Head                --> IPU 0\n",
      "-----------------------------------------------------------\n",
      "Configuration saved in gpt2-baby/checkpoint-8000/ipu_config.json\n",
      "Deleting older checkpoint [gpt2-baby/checkpoint-7000] due to args.save_total_limit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.9828, 'learning_rate': 3.549003549003549e-07, 'epoch': 9.84}\n",
      "{'loss': 4.802, 'learning_rate': 3.2760032760032763e-07, 'epoch': 9.85}\n",
      "{'loss': 5.0852, 'learning_rate': 3.003003003003003e-07, 'epoch': 9.86}\n",
      "{'loss': 4.7496, 'learning_rate': 2.73000273000273e-07, 'epoch': 9.88}\n",
      "{'loss': 4.5805, 'learning_rate': 2.457002457002457e-07, 'epoch': 9.89}\n",
      "{'loss': 5.0316, 'learning_rate': 2.1840021840021843e-07, 'epoch': 9.9}\n",
      "{'loss': 5.1301, 'learning_rate': 1.9110019110019112e-07, 'epoch': 9.91}\n",
      "{'loss': 5.1613, 'learning_rate': 1.6380016380016382e-07, 'epoch': 9.93}\n",
      "{'loss': 5.2223, 'learning_rate': 1.365001365001365e-07, 'epoch': 9.94}\n",
      "{'loss': 5.1188, 'learning_rate': 1.0920010920010921e-07, 'epoch': 9.95}\n",
      "{'loss': 4.6152, 'learning_rate': 8.190008190008191e-08, 'epoch': 9.96}\n",
      "{'loss': 4.3162, 'learning_rate': 5.460005460005461e-08, 'epoch': 9.98}\n",
      "{'loss': 4.7705, 'learning_rate': 2.7300027300027304e-08, 'epoch': 9.99}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 4.6656, 'learning_rate': 0.0, 'epoch': 10.0}\n",
      "{'train_runtime': 2876.2842, 'train_samples_per_second': 362.245, 'train_steps_per_second': 2.83, 'train_loss': 5.375703988789926, 'epoch': 10.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=8140, training_loss=5.375703988789926, metrics={'train_runtime': 2876.2842, 'train_samples_per_second': 362.245, 'train_steps_per_second': 2.83, 'train_loss': 5.375703988789926, 'epoch': 10.0})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3APq-vUc3l_R"
   },
   "source": [
    "Once the training is completed, we can evaluate our model and get its perplexity on the validation set like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import math\n",
    "# eval_results = dtrainer.evaluate()\n",
    "# print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The perplexity is still quite high since for this demo we trained on a small dataset for a small number of epochs. For a real LM training, you  would need a larger dataset and more epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wY82caEX3l_i"
   },
   "source": [
    "You can now upload the result of the training to the Hub, just execute this instruction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainer.push_to_hub()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier `\"your-username/the-name-you-picked\"` so for instance:\n",
    "\n",
    "```python\n",
    "from transformers import AutoModelForCausalLM\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(\"sgugger/my-awesome-model\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Train a language model",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}