notebooks/optimal-stopping-point/model_inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3f5d1031-4a3c-4bd7-9089-4766917156f2",
   "metadata": {},
   "source": [
    "# Optimal Stopping Point Inference Service\n",
    "\n",
    "In the previous notebook, we explored various distribution models to find the best model for predicting the optimal stopping point for a given test. We then deployed the model as a service using Seldon. The purpose of this notebook is to check whether this service is running as intended, and more specifically to ensure that the model performance is what we expect it to be. So here, we will use the test set from the aforementioned notebook as the query payload for the service, and then verify that the return values are the same as those obtained during training/testing locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "091c5463",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import json\n",
    "import requests\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "import datetime\n",
    "\n",
    "metric_template_path = \"../data-sources/TestGrid/metrics\"\n",
    "if metric_template_path not in sys.path:\n",
    "    sys.path.insert(1, metric_template_path)\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e4672acb",
   "metadata": {},
   "outputs": [],
   "source": [
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs,\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = \"osp\"\n",
    "REMOTE = os.getenv(\"REMOTE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7ab0777c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# endpoint from the seldon deployment\n",
    "base_url = \"http://optimal-stopping-point-ds-ml-workflows-ws.apps.smaug.na.operate-first.cloud/predict\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3b680f9-ef4e-4065-8f9a-74f46a0b5126",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Send request by adding a testname and timestamp.\n",
    "data = {\n",
    "    \"jsonData\": {\n",
    "        \"test_name\": \"operator.Run multi-stage test e2e-aws-upgrade - \"\n",
    "        \"e2e-aws-upgrade-openshift-e2e-test container test\",\n",
    "        \"timestamp\": datetime.datetime(2021, 8, 24).timestamp(),\n",
    "    }\n",
    "}\n",
    "\n",
    "# create the query payload\n",
    "json_data = json.dumps(data)\n",
    "headers = {\"content-Type\": \"application/json\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "587cd909-aeab-4318-8d70-e3c5d50c96cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# query our inference service\n",
    "response = requests.post(base_url, data=json_data, headers=headers)\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0e2f83db-db20-4d40-ac95-e51e488dea1b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104.44102946500577"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b3d3e60-469e-4dd9-aed9-853c05b76ba8",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "This notebook shows how test name and timestamp can be sent to the deployed Seldon service to get optimal-stopping-point predictions. So, great, looks like our inference service and model are working as expected, and are ready to predict some stopping times for the failing tests! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}