notebooks/evaluation_tap_vit_b_v1_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Guide for TAP-B\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "We provide three simple scripts to evaluate **TAP** on ***Instance Segmentation***, ***Instance Classification*** and ***Region Caption*** tasks respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Necessary datasets, imports and models for evaluation.\n",
    "\n",
    "```\n",
    "datasets\n",
    "|_ coco\n",
    "|  |_ train2017\n",
    "|  |  |_ 000000000009.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ val2017\n",
    "|  |  |_ 000000000139.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ annotations\n",
    "|  |  |_ coco_instances_val2017.json\n",
    "|  |_ results\n",
    "|  |  |_ coco_instances_val2017_vitdet_h_cascade.json  # Run detectron2 to generate this file.\n",
    "|- lvis\n",
    "|  |_ annotations\n",
    "|  |  |_ lvis_val_v1.json\n",
    "|  |_ results\n",
    "|  |  |_ lvis_val_v1_vitdet_h_cascade.json  # Run detectron2 to generate this file.\n",
    "|_ vg\n",
    "|  |_ images\n",
    "|  |  |_ 1.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ annotations\n",
    "|  |  |_ test.json  # https://datarelease.blob.core.windows.net/grit/VG_preprocessed_annotations/test.json\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "\n",
    "class AttrDict(dict):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(AttrDict, self).__init__(*args, **kwargs)\n",
    "        self.__dict__ = self\n",
    "\n",
    "# global arguments.\n",
    "args = AttrDict(read_every=100, prompt_size=256)\n",
    "args.model_type = \"tap_vit_b\"\n",
    "args.checkpoint = \"../models/tap_vit_b_v1_1.pkl\"\n",
    "args.device = [0, 1, 2, 3, 4, 5, 6, 7]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Segmentation on COCO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "92857 instances in 5000 images.\n",
      "im_process: 5000/5000 [0.071s + 0.014s] (eta: 0:00:00)\n",
      "Writing segmentations to /data/workspace/models/tokenize-anything/scripts/../outputs/coco_segmentations.json\n",
      "\n",
      "Evaluating COCO segmentations...\n",
      "loading annotations into memory...\n",
      "Done (t=0.47s)\n",
      "creating index...\n",
      "index created!\n",
      "Loading and preparing results...\n",
      "DONE (t=1.55s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=24.56s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=3.62s).\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.451\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.712\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.481\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.287\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.501\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.606\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.340\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.540\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.562\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.423\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.620\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.686\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_seg import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/coco/annotations/coco_instances_val2017.json\"\n",
    "args.det_json_file = \"../datasets/coco/results/coco_instances_val2017_vitdet_h_cascade.json\"\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Segmentation on LVIS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3293288 instances in 19809 images.\n",
      "im_process: 19809/19809 [0.131s + 0.127s] (eta: 0:00:00)\n",
      "Writing segmentations to /data/workspace/models/tokenize-anything/scripts/../outputs/lvis_segmentations.json\n",
      "\n",
      "Evaluating LVIS segmentations...\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.422\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.609\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.450\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.295\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.550\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.640\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.336\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.431\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.450\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.510\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.368\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.638\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.700\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_seg import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/lvis/annotations/lvis_val_v1.json\"\n",
    "args.det_json_file = \"../datasets/lvis/results/lvis_val_v1_vitdet_h_cascade.json\"\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Classification on LVIS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/workspace/programs/conda3/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "244707 instances in 19809 images.\n",
      "im_process: 19809/19809 [0.057s + 0.020s] (eta: 0:00:00)\n",
      "Writing detections to /data/workspace/models/tokenize-anything/scripts/../outputs/lvis_detections.json\n",
      "\n",
      "Evaluating LVIS detections...\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.574\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.579\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.574\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.431\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.701\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.805\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.586\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.568\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.575\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.655\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.497\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.766\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.876\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_cls import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/lvis/annotations/lvis_val_v1.json\"\n",
    "args.concept = \"../concepts/lvis_1203.pkl\"\n",
    "args.max_dets = 300\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Region Caption on Visual Genome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "232935 instances in 5000 images.\n",
      "im_process: 5000/5000 [0.185s] (eta: 0:00:00)\n",
      "Evaluating captions...\n",
      "Bleu [0.3737476039273303, 0.23526039725915165, 0.16513572981883956, 0.12226500630696996]\n",
      "METEOR 0.17705993786192956\n",
      "Rouge 0.359520834691781\n",
      "CIDEr 1.5234709370648996\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_cap import main\n",
    "\n",
    "args.images_dir = \"../datasets/vg/images\"\n",
    "args.gt_json_file = \"../datasets/vg/annotations/test.json\"\n",
    "main(args)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}