Skip to content

Instantly share code, notes, and snippets.

@botcs

botcs/unit_test_cocoeval.ipynb

Created March 1, 2019 22:20
Show Gist options
  • Save botcs/904d96ebb50708f7690f52c831e76018 to your computer and use it in GitHub Desktop.
Save botcs/904d96ebb50708f7690f52c831e76018 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
unit_test_cocoeval.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Unit test for COCOWrapper and COCOeval\n",
"\n",
"### 1. Introduce `debugdataset` : \n",
"A demo implementation with the bare minimum functions required to train a new dataset using the `maskrcnn_benchmark` lib, featuring the core fields and methods which must be implemented.\n",
"The dataset generates non-overlapping uniform sized white boxes with annotations.\n",
"The size and number of instances can be controlled.\n",
"\n",
"_Maybe later it could be used for expanding unit tests with training networks on this dataset._\n",
"\n",
"### 2. Introduce `COCOWrapper`: \n",
"An additional layer on the top of the generic `maskrcnn-dataset`, only requiring basic fields and methods to be implemented.\n",
"Only serves the sole purpose to replace the `dataset.coco` dependency, more specifically this wrapper implements functions that the `COCOeval` requires.\n",
" \n",
"### 3. Test on `COCOeval`:\n",
"Using the original implementation from [MS-COCO](https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools) test if perfect score can be achieved using the annotations as predictions with 1.0 objectness-score."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import numpy as np\n",
"\n",
"from pycocotools.cocoeval import COCOeval\n",
"from maskrcnn_benchmark.data.datasets.evaluation.coco.abstract_cocostyle_eval import COCOWrapper\n",
"import maskrcnn_benchmark.data.datasets.debugdataset as debugdataset\n",
"\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# initialize database\n",
"dataset = debugdataset.DebugDataset(\n",
" dataset_size=100,\n",
" width=640,\n",
" height=480,\n",
" side_length=50, # medium sized instances\n",
" min_num_instances=30, \n",
" max_num_instances=30,\n",
" allow_overlap=False, # the most straightforward case\n",
" transforms=None\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Mimic predictions\n",
"\n",
"Use the same format that `do_coco_evaluation` recieves in the original `maskrcnn_benchmark` implementation [here](https://github.com/facebookresearch/maskrcnn-benchmark/blob/13b4f82efd953276b24ce01f0fd1cd08f94fbaf8/maskrcnn_benchmark/data/datasets/evaluation/coco/coco_eval.py#L13)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"predictions = []\n",
"for img, target, idx in dataset:\n",
" target.add_field(\"scores\", torch.ones(len(target))) \n",
" predictions.append(target)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Wrap dataset and predictions\n",
"Notice: compared to the current implementation, here `cocoDt` and `cocoGt` is formatted to RLE representation before feeding them to `COCOeval`, and their instances are reused"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO GT annots\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Parsing images: 100%|██████████| 100/100 [00:04<00:00, 24.13it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO Predictions\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"Parsing images: 100it [00:01, 80.09it/s]\n"
]
}
],
"source": [
"cocoGt = COCOWrapper(dataset)\n",
"cocoDt = COCOWrapper(dataset, predictions)\n",
"coco_eval_bbox = COCOeval(cocoGt, cocoDt, 'bbox')\n",
"coco_eval_segm = COCOeval(cocoGt, cocoDt, 'segm')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualize one input from the dataset\n",
"`dataset[idx]` returns an `img, target, idx` tuple"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAAIhklEQVR4nO3dS3LbMBQAQTGl+18ZWcRLlaP/AGD3BYyFrfEDReByAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA+66gXAHB2Y4x6CTcch0B81p96AQBwRgIMAAEBBoCAAANAQIABIHCtFwDww5eBORUTMAAEBBgAAragadhsBE7OBAwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgMC1XsBljFEv4YbjOOolALAzEzAABPoJGIBF2cJ8hQkYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgICzoAFiq5xdzHuZgAEgIMAAEBBgAAgIMAAEBBgAAr4FDczCl4E5FRMwAAQEGAACtqBp2GwETs4EDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAi4jpCvGmPUS7jB3YjA95mAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAj0Z0E7hheAE+oDDLAiN4vwIgEG4El6/wrPgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANA4FovgP8bY9RLuOE4jnoJAAszAQNAQIABIGALmq+ycQ3wjwkYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAgSWPonQ7EPzO3wjMzwQMAIElJ2CAnIGeF5mAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0DAQRzwG2c6Ah9iAgaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQdx8GZOrgC4hwkYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAi4D3gB7rIF2I8JGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABILDkbUj33w40xvjoSp77WW434tP8jsH8TMAAEBBgAAgsuQUNAJfvPme8353PgEzAABAQYAAICDAABAQYAAICDAABAQaAgAADQMB7wPAbZzrykKVfS+XLTMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABIOAgDt7MK//bc9YEvIUJGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABBwHSFMwR1/cDYmYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0Bg8+sIXaYGwJxMwAAQEGAACGy+BQ3AxpZ+zmgCBoCAAANAwBY0AN8zxqiXcEOylW0CBoDApBOwf5EA2JsJGAACAgwAgUm3oAFW5EEV9zMBA0DABMzUfB0P2JUJGAACAgwAAVvQS7IxC7A6EzAABAQYAAICDAABz4CBx3jYD29hAgaAgAADQECAASAgwAAQEGAACAgwAAQEGAAC3gMGTsdp6szABAwAAQEGgIAAA0BAgAEgIMAAEBBgAAh4DQmm4BUUOBsTMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgMC1XgAAJ3IcR72EWZiAASAw6QTsXyQA9mYCBoDApBPwc8YY9RJuMM0Ds/FpOQMTMAAEtpqAge8zS8FzBJip+RgFdmULGgACAgwAAVvQS7IxC7A6EzAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAWdBA6fjNHVmYAIGgIAJGIB5jTHqJdzwlk0UEzAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIDLGGANGx9JD+dkAgaAgAADQECAASAgwAAQEGAACAgwAAQEGAAC3gPmDbyiCvAoEzAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACGx1EIeDFwDu4dNyBiZgAAhsNQED32eWgueYgAEgIMAAENhhC9pVPAAsxwQMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACOxwHSEAu9r4alcTMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgMsYYA0bH0kP52QCBoCAAANAQIABICDAABAQYAAICDAABAQYAALeA+YNvKIK8CgTMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAKOogRgc2OMegk3mIABICDAABBwiQ0Am7MFDQD8EGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQYAAAAAAADYx1/pOWQs7WGWqAAAAABJRU5ErkJggg==\n",
"text/plain": [
"<PIL.Image.Image image mode=RGB size=640x480 at 0x7F66476B9A90>"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[0][0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### evaluate the results as it is done originally\n",
"\n",
"1. `coco_eval_<type>.evaluate()` sets field `coco_eval_<type>.evalImgs`\n",
"1. `coco_eval_<type>.accumulate()` sets field `coco_eval_<type>.eval`\n",
"1. `coco_eval_<type>.summarize()` computes some specific scores from `coco_eval_<type>.eval` and prints it"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=2.30s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.02s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.300\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.967\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.967\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=2.15s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.02s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.928\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.300\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.967\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.967\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# PROBLEM! The scores are not perfect (< 1.0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Let's inspect the bug using an even simpler dataset\n",
"\n",
"reduce the dataset size to a single image and the number of instances to 2"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO GT annots\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Parsing images: 100%|██████████| 1/1 [00:00<00:00, 314.65it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO Predictions\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"Parsing images: 1it [00:00, 676.61it/s]\n"
]
}
],
"source": [
"num_of_inst = 2\n",
"dataset = debugdataset.DebugDataset(\n",
" dataset_size=1, \n",
" min_num_instances=num_of_inst, \n",
" max_num_instances=num_of_inst\n",
")\n",
"\n",
"predictions = []\n",
"for img, target, idx in dataset:\n",
" target.add_field(\"scores\", torch.ones(len(target))) \n",
" predictions.append(target)\n",
"\n",
"cocoGt = COCOWrapper(dataset)\n",
"cocoDt = COCOWrapper(dataset, predictions)\n",
"coco_eval_bbox = COCOeval(cocoGt, cocoDt, 'bbox')\n",
"coco_eval_segm = COCOeval(cocoGt, cocoDt, 'segm')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAAEgklEQVR4nO3VsQ2DQBAAwX/67/kIHJrAskAbMFPBZrsWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPA2uw4AgD/NTJ1wYe+f3no83QEAfDNgAAgYMAAEDBgAAgYMAAEDBoCAAQNAwIABIGDAABAwYAAIGDAABAwYAAIGDAABAwaAgAEDQMCAASBgwAAQMGAACBgwAAQMGAACBgwAAQMGgIABA0DAgAEgYMAAEDBgAAgYMAAEDBgAAgYMAAEDBoCAAQNAwIABIGDAABAwYAAIGDAABAwYAAIGDAABAwaAgAEDQMCAASBgwAAQMGAACBgwAAQMGAACBgwAAQMGgIABA0DAgAEgYMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB87DoAgKfMTJ1wYW/rWWutow4AgDcyYAAIGDAABAwYAAIGDAABAwaAgAEDQMCAASBgwAAQMGAACBgwAAQMGAACBgwAAQMGgIABA0DAgAEgYMAAEDBgAAgYMAAEDBgAAgYMAAEDBoCAAQNAwIABIGDAABAwYAAIGDAABAwYAAIGDAABAwaAgAEDQMCAASBgwAAQMGAACBgwAAQMGAACBgwAAQMGgIABA0DAgAEgYMAAEDBgAAgYMAAEDBgAAgYMAAEDBoCAAQNAwIABIGDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA3O0EqJQGx+8S9bQAAAAASUVORK5CYII=\n",
"text/plain": [
"<PIL.Image.Image image mode=RGB size=640x480 at 0x7F65AC93DA90>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"img, target, idx = dataset[0]\n",
"img"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualize the masks as well for safety"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 0\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbpJREFUeJzt3W2MpXV5x/Hvr+wDPrKAhGx3N12IpIYXDZANQjTGQGyVGuEFWoypG0OzSUsTDU3s0iZtTPpC+8KnpMFuxHZtrEDRFkIwFAHT+MLVVR7kochoNbtbcKsCao0U9OqL8196ul2dudiZOWfo95NM5r7/9z1zrrM5+91znzMDqSokSUvzK7MeQJLWEqMpSQ1GU5IajKYkNRhNSWowmpLUsCLRTPLGJI8kWUiyeyVuQ5JmIcv9c5pJTgC+AbwBOAh8BXh7VT20rDckSTOwEs80zwcWqupbVfVfwPXApStwO5K06tatwPfcAhyY2j8IvPqXfcGGbKwTeckKjCJJS/MjnvheVZ222HkrEc0lSbIL2AVwIi/m1bl4VqNIEp+vm76zlPNW4vL8ELBtan/rWPtfqmpPVe2oqh3r2bgCY0jS8luJaH4FOCvJGUk2AFcAt6zA7UjSqlv2y/OqejbJHwK3AycAn6iqB5f7diRpFlbkNc2qug24bSW+tyTNkr8RJEkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqWDSaST6R5HCSB6bWTklyR5JHx+eTx3qSfDTJQpL7k5y3ksNL0mpbyjPNvwXeeNTabuDOqjoLuHPsA7wJOGt87AKuXZ4xJWk+LBrNqvoX4AdHLV8K7B3be4HLptY/WRNfAjYl2bxcw0rSrD3f1zRPr6rHxvbjwOljewtwYOq8g2NNkl4QjvuNoKoqoLpfl2RXkv1J9j/D08c7hiStiucbze8euewenw+P9UPAtqnzto61/6Oq9lTVjqrasZ6Nz3MMSVpdzzeatwA7x/ZO4Oap9XeOd9EvAJ6auoyXpDVv3WInJPk08HrgFUkOAn8OvB+4McmVwHeAt43TbwMuARaAnwDvWoGZJWlmFo1mVb39Fxy6+BjnFnDV8Q4lSfPK3wiSpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqWDSaSbYluTvJQ0keTPLusX5KkjuSPDo+nzzWk+SjSRaS3J/kvJW+E5K0WpbyTPNZ4I+q6mzgAuCqJGcDu4E7q+os4M6xD/Am4KzxsQu4dtmnlqQZWTSaVfVYVX1tbP8IeBjYAlwK7B2n7QUuG9uXAp+siS8Bm5JsXvbJJWkGWq9pJtkOnAvsA06vqsfGoceB08f2FuDA1JcdHGtHf69dSfYn2f8MTzfHlqTZWHI0k7wU+Azwnqr64fSxqiqgOjdcVXuqakdV7VjPxs6XStLMLCmaSdYzCeanquqzY/m7Ry67x+fDY/0QsG3qy7eONUla85by7nmA64CHq+qDU4duAXaO7Z3AzVPr7xzvol8APDV1GS9Ja9q6JZzzGuB3ga8nuXes/QnwfuDGJFcC3wHeNo7dBlwCLAA/Ad61rBNL0gwtGs2q+iKQX3D44mOcX8BVxzmXJM0lfyNIkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktSwaDSTnJjky0nuS/JgkveN9TOS7EuykOSGJBvG+saxvzCOb1/ZuyBJq2fdEs55Grioqn6cZD3wxSSfA64GPlRV1yf5GHAlcO34/ERVvTLJFcAHgN9ZofklvQDd/u/3rtj3/q1fPee4vn7RZ5o18eOxu358FHARcNNY3wtcNrYvHfuM4xcnyXFNKUlzYkmvaSY5Icm9wGHgDuCbwJNV9ew45SCwZWxvAQ4AjONPAace43vuSrI/yf5nePr47oUkrZIlRbOqflZV5wBbgfOBVx3vDVfVnqraUVU71rPxeL+dJK2K1rvnVfUkcDdwIbApyZHXRLcCh8b2IWAbwDh+EvD9ZZlWkmZsKe+en5Zk09h+EfAG4GEm8bx8nLYTuHls3zL2GcfvqqpazqElaVaW8u75ZmBvkhOYRPbGqro1yUPA9Un+ArgHuG6cfx3wd0kWgB8AV6zA3JI0E4tGs6ruB849xvq3mLy+efT6T4G3Lst0kjRn/I0gSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IalvIfIZakVXW8/5vdleQzTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSw5KjmeSEJPckuXXsn5FkX5KFJDck2TDWN479hXF8+8qMLkmrr/NM893Aw1P7HwA+VFWvBJ4ArhzrVwJPjPUPjfMk6QVhSdFMshX4beDjYz/ARcBN45S9wGVj+9Kxzzh+8Thfkta8pT7T/DDwXuDnY/9U4MmqenbsHwS2jO0twAGAcfypcb4krXmLRjPJm4HDVfXV5bzhJLuS7E+y/xmeXs5vLUkrZt0SznkN8JYklwAnAi8HPgJsSrJuPJvcChwa5x8CtgEHk6wDTgK+f/Q3rao9wB6Al+eUOt47IkmrYdFnmlV1TVVtrartwBXAXVX1DuBu4PJx2k7g5rF9y9hnHL+rqoyipBeE4/k5zT8Grk6ywOQ1y+vG+nXAqWP9amD38Y0oSfNjKZfnz6mqLwBfGNvfAs4/xjk/Bd66DLNJ0tzxN4IkqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWpIVc16BpL8CHhk1nM8D68AvjfrIZqcefWsxbn/P8/8a1V12mInrVuGG1oOj1TVjlkP0ZVk/1qb25lXz1qc25kX5+W5JDUYTUlqmJdo7pn1AM/TWpzbmVfPWpzbmRcxF28ESdJaMS/PNCVpTZh5NJO8MckjSRaS7J71PEck+USSw0kemFo7JckdSR4dn08e60ny0XEf7k9y3oxm3pbk7iQPJXkwybvXyNwnJvlykvvG3O8b62ck2TfmuyHJhrG+cewvjOPbZzH3mOWEJPckuXUtzJzk20m+nuTeJPvH2rw/PjYluSnJvyZ5OMmFM525qmb2AZwAfBM4E9gA3AecPcuZpmZ7HXAe8MDU2l8Cu8f2buADY/sS4HNAgAuAfTOaeTNw3th+GfAN4Ow1MHeAl47t9cC+Mc+NwBVj/WPA74/tPwA+NravAG6Y4ePkauDvgVvH/lzPDHwbeMVRa/P++NgL/N7Y3gBsmuXMM3mgTf1hXAjcPrV/DXDNLGc6ar7tR0XzEWDz2N7M5OdLAf4aePuxzpvx/DcDb1hLcwMvBr4GvJrJDyyvO/qxAtwOXDi2143zMoNZtwJ3AhcBt46/qPM+87GiObePD+Ak4N+O/rOa5cyzvjzfAhyY2j841ubV6VX12Nh+HDh9bM/d/RiXf+cyedY293OPy9x7gcPAHUyuQJ6sqmePMdtzc4/jTwGnru7EAHwYeC/w87F/KvM/cwH/nOSrSXaNtXl+fJwB/AfwN+NlkI8neQkznHnW0VyzavLP2Fz+6EGSlwKfAd5TVT+cPjavc1fVz6rqHCbP3s4HXjXjkX6pJG8GDlfVV2c9S9Nrq+o84E3AVUleN31wDh8f65i8THZtVZ0L/CeTy/HnrPbMs47mIWDb1P7WsTavvptkM8D4fHisz839SLKeSTA/VVWfHctzP/cRVfUkcDeTS9tNSY78qu/0bM/NPY6fBHx/lUd9DfCWJN8Grmdyif4R5ntmqurQ+HwY+Ecm/0DN8+PjIHCwqvaN/ZuYRHRmM886ml8BzhrvOG5g8gL5LTOe6Ze5Bdg5tncyec3wyPo7xzt3FwBPTV06rJokAa4DHq6qD04dmve5T0uyaWy/iMnrsA8ziefl47Sj5z5yfy4H7hrPNlZNVV1TVVurajuTx+1dVfUO5njmJC9J8rIj28BvAg8wx4+PqnocOJDk18fSxcBDM515NV/U/QUv9F7C5F3ebwJ/Out5pub6NPAY8AyTf+2uZPIa1J3Ao8DngVPGuQH+atyHrwM7ZjTza5lcptwP3Ds+LlkDc/8GcM+Y+wHgz8b6mcCXgQXgH4CNY/3Esb8wjp8548fK6/mfd8/nduYx233j48Ejf9/WwOPjHGD/eHz8E3DyLGf2N4IkqWHWl+eStKYYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGv4btY2Otks13b4AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 1\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbRJREFUeJzt3W+MpXV5h/Hr2/2Hf1lAQra7mwKR1PCiBbJBiMYYiC1SI7xAgzF1Y7bZpKWJxiZ2aZM2Jn2hfSFq0miJ2K6NVSjaQggNRcA0feHqKn8EtshoNbtbcKsC2hop6N0X57f0dLs6c7Mzc87g9Ukm8zy/55lz7iFnrz3POTNsqgpJ0tL80qwHkKS1xGhKUoPRlKQGoylJDUZTkhqMpiQ1rEg0k1yW5JEkC0n2rMR9SNIsZLl/TjPJOuDrwBuAQ8CXgbdV1cPLekeSNAMr8UzzQmChqr5ZVf8NfAa4YgXuR5JW3foVuM2twMGp/UPAq3/eF2zMpjqJl6zAKJK0ND/kie9W1emLnbcS0VySJLuB3QAn8WJenUtnNYok8fm6+dtLOW8lLs8PA9un9reNtf+jqq6vqh1VtWMDm1ZgDElafisRzS8D5yQ5K8lG4Grg1hW4H0ladct+eV5Vzyb5feAOYB3wiap6aLnvR5JmYUVe06yq24HbV+K2JWmW/I0gSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlLDotFM8okkR5I8OLV2apI7kzw6Pp8y1pPkI0kWkjyQ5IKVHF6SVttSnmn+NXDZMWt7gLuq6hzgrrEP8EbgnPGxG/jo8owpSfNh0WhW1T8D3z9m+Qpg79jeC1w5tf7JmvgisDnJluUaVpJm7fm+pnlGVT02th8HzhjbW4GDU+cdGmuS9IJwwm8EVVUB1f26JLuT7E+y/xmePtExJGlVPN9ofufoZff4fGSsHwa2T523baz9P1V1fVXtqKodG9j0PMeQpNX1fKN5K7BzbO8Ebplaf8d4F/0i4Kmpy3hJWvPWL3ZCkk8DrwdekeQQ8KfA+4GbkuwCvg28dZx+O3A5sAD8CHjnCswsSTOzaDSr6m0/49Clxzm3gGtOdChJmlf+RpAkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlLDotFMsj3JPUkeTvJQkneN9VOT3Jnk0fH5lLGeJB9JspDkgSQXrPQ3IUmrZSnPNJ8F/qCqzgUuAq5Jci6wB7irqs4B7hr7AG8Ezhkfu4GPLvvUkjQji0azqh6rqq+O7R8CB4CtwBXA3nHaXuDKsX0F8Mma+CKwOcmWZZ9ckmag9ZpmkjOB84F9wBlV9dg49DhwxtjeChyc+rJDY+3Y29qdZH+S/c/wdHNsSZqNJUczyUuBzwLvrqofTB+rqgKqc8dVdX1V7aiqHRvY1PlSSZqZJUUzyQYmwfxUVX1uLH/n6GX3+HxkrB8Gtk99+baxJklr3lLePQ9wA3Cgqj44dehWYOfY3gncMrX+jvEu+kXAU1OX8ZK0pq1fwjmvAX4b+FqS+8baHwHvB25Ksgv4NvDWcex24HJgAfgR8M5lnViSZmjRaFbVvwD5GYcvPc75BVxzgnNJ0lzyN4IkqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ2LRjPJSUm+lOT+JA8led9YPyvJviQLSW5MsnGsbxr7C+P4mSv7LUjS6lnKM82ngUuq6teB84DLklwEfAC4rqpeCTwB7Brn7wKeGOvXjfMk6QVh0WjWxH+O3Q3jo4BLgJvH+l7gyrF9xdhnHL80SZZtYkmaoSW9pplkXZL7gCPAncA3gCer6tlxyiFg69jeChwEGMefAk47zm3uTrI/yf5nePrEvgtJWiVLimZV/aSqzgO2ARcCrzrRO66q66tqR1Xt2MCmE705SVoVrXfPq+pJ4B7gYmBzkvXj0Dbg8Ng+DGwHGMdPBr63LNNK0owt5d3z05NsHtsvAt4AHGASz6vGaTuBW8b2rWOfcfzuqqrlHFqSZmX94qewBdibZB2TyN5UVbcleRj4TJI/A+4Fbhjn3wD8TZIF4PvA1SswtyTNxKLRrKoHgPOPs/5NJq9vHrv+Y+AtyzKdJM0ZfyNIkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKSGRf/dc0m/OO749/tW7LZ/85fPW7HbXk0+05SkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNSw5mknWJbk3yW1j/6wk+5IsJLkxycaxvmnsL4zjZ67M6JK0+jrPNN8FHJja/wBwXVW9EngC2DXWdwFPjPXrxnmS9IKwpGgm2Qb8FvDxsR/gEuDmccpe4MqxfcXYZxy/dJwvSWveUp9pfgh4L/DTsX8a8GRVPTv2DwFbx/ZW4CDAOP7UOF+S1rxFo5nkTcCRqvrKct5xkt1J9ifZ/wxPL+dNS9KKWcr/uf01wJuTXA6cBLwc+DCwOcn68WxyG3B4nH8Y2A4cSrIeOBn43rE3WlXXA9cDvDyn1ol+I5K0GhZ9pllV11bVtqo6E7gauLuq3g7cA1w1TtsJ3DK2bx37jON3V5VRlPSCcCI/p/mHwHuSLDB5zfKGsX4DcNpYfw+w58RGlKT50fqH1arqC8AXxvY3gQuPc86Pgbcsw2ySNHf8jSBJavCf8JX0nBfKP7O7knymKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1JCqmvUMJPkh8Mis53geXgF8d9ZDNDnz6lmLc/8iz/wrVXX6YietX4Y7Wg6PVNWOWQ/RlWT/WpvbmVfPWpzbmRfn5bkkNRhNSWqYl2heP+sBnqe1OLczr561OLczL2Iu3giSpLViXp5pStKaMPNoJrksySNJFpLsmfU8RyX5RJIjSR6cWjs1yZ1JHh2fTxnrSfKR8T08kOSCGc28Pck9SR5O8lCSd62RuU9K8qUk94+53zfWz0qyb8x3Y5KNY33T2F8Yx8+cxdxjlnVJ7k1y21qYOcm3knwtyX1J9o+1eX98bE5yc5J/TXIgycUznbmqZvYBrAO+AZwNbATuB86d5UxTs70OuAB4cGrtz4E9Y3sP8IGxfTnwj0CAi4B9M5p5C3DB2H4Z8HXg3DUwd4CXju0NwL4xz03A1WP9Y8Dvju3fAz42tq8Gbpzh4+Q9wN8Ct439uZ4Z+BbwimPW5v3xsRf4nbG9Edg8y5ln8kCb+o9xMXDH1P61wLWznOmY+c48JpqPAFvG9hYmP18K8JfA24533oznvwV4w1qaG3gx8FXg1Ux+YHn9sY8V4A7g4rG9fpyXGcy6DbgLuAS4bfxBnfeZjxfNuX18ACcD/3bsf6tZzjzry/OtwMGp/UNjbV6dUVWPje3HgTPG9tx9H+Py73wmz9rmfu5xmXsfcAS4k8kVyJNV9exxZntu7nH8KeC01Z0YgA8B7wV+OvZPY/5nLuCfknwlye6xNs+Pj7OA/wD+arwM8vEkL2GGM886mmtWTf4am8sfPUjyUuCzwLur6gfTx+Z17qr6SVWdx+TZ24XAq2Y80s+V5E3Akar6yqxnaXptVV0AvBG4Jsnrpg/O4eNjPZOXyT5aVecD/8Xkcvw5qz3zrKN5GNg+tb9trM2r7yTZAjA+Hxnrc/N9JNnAJJifqqrPjeW5n/uoqnoSuIfJpe3mJEd/1Xd6tufmHsdPBr63yqO+Bnhzkm8Bn2Fyif5h5ntmqurw+HwE+Hsmf0HN8+PjEHCoqvaN/ZuZRHRmM886ml8GzhnvOG5k8gL5rTOe6ee5Fdg5tncyec3w6Po7xjt3FwFPTV06rJokAW4ADlTVB6cOzfvcpyfZPLZfxOR12ANM4nnVOO3YuY9+P1cBd49nG6umqq6tqm1VdSaTx+3dVfV25njmJC9J8rKj28BvAA8yx4+PqnocOJjkV8fSpcDDM515NV/U/Rkv9F7O5F3ebwB/POt5pub6NPAY8AyTv+12MXkN6i7gUeDzwKnj3AB/Mb6HrwE7ZjTza5lcpjwA3Dc+Ll8Dc/8acO+Y+0HgT8b62cCXgAXg74BNY/2ksb8wjp8948fK6/nfd8/nduYx2/3j46Gjf97WwOPjPGD/eHz8A3DKLGf2N4IkqWHWl+eStKYYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGv4HxMCMS66qcWcAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"for inst_idx, mask in enumerate(target.get_field('masks')):\n",
" print(\"instance:\", inst_idx)\n",
" plt.imshow(mask.get_mask_tensor())\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now compute the results"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scores are even worse\n",
"\n",
"\n",
"### Let's iterate over these fields to find the source:\n",
"1. `coco_eval_<type>.evaluate()` sets field `coco_eval_<type>.evalImgs`\n",
"1. `coco_eval_<type>.accumulate()` sets field `coco_eval_<type>.eval`\n",
"1. `coco_eval_<type>.summarize()` computes some specific scores from `coco_eval_<type>.eval` and prints it"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'image_id': 0,\n",
" 'category_id': 99,\n",
" 'aRng': [0, 10000000000.0],\n",
" 'maxDet': 100,\n",
" 'dtIds': [0, 1],\n",
" 'gtIds': [0, 1],\n",
" 'dtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'gtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'dtScores': [1.0, 1.0],\n",
" 'gtIgnore': array([0, 0]),\n",
" 'dtIgnore': array([[False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False]])},\n",
" {'image_id': 0,\n",
" 'category_id': 99,\n",
" 'aRng': [0, 1024],\n",
" 'maxDet': 100,\n",
" 'dtIds': [0, 1],\n",
" 'gtIds': [0, 1],\n",
" 'dtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'gtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'dtScores': [1.0, 1.0],\n",
" 'gtIgnore': array([1, 1]),\n",
" 'dtIgnore': array([[ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True]])},\n",
" {'image_id': 0,\n",
" 'category_id': 99,\n",
" 'aRng': [1024, 9216],\n",
" 'maxDet': 100,\n",
" 'dtIds': [0, 1],\n",
" 'gtIds': [0, 1],\n",
" 'dtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'gtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'dtScores': [1.0, 1.0],\n",
" 'gtIgnore': array([0, 0]),\n",
" 'dtIgnore': array([[False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False],\n",
" [False, False]])},\n",
" {'image_id': 0,\n",
" 'category_id': 99,\n",
" 'aRng': [9216, 10000000000.0],\n",
" 'maxDet': 100,\n",
" 'dtIds': [0, 1],\n",
" 'gtIds': [0, 1],\n",
" 'dtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'gtMatches': array([[0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.],\n",
" [0., 1.]]),\n",
" 'dtScores': [1.0, 1.0],\n",
" 'gtIgnore': array([1, 1]),\n",
" 'dtIgnore': array([[ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True],\n",
" [ True, True]])}]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"evalImgs = coco_eval_segm.evalImgs\n",
"\n",
"evalImgs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Mostly seems like that the scores and the matches look OK\n",
"\n",
"but for safety, just check it out"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[1.0, 1.0], [1.0, 1.0], [1.0, 1.0], [1.0, 1.0]]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dtScores_all = [x['dtScores'] for x in evalImgs]\n",
"dtScores_all"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### test self.eval, which is used to compute scores in self.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/plain": [
"(10, 101, 1, 4, 3)"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coco_eval_segm.eval['precision'].shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## what about the shape?\n",
"\n",
"A quick tutorial extracted from the [source code](https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py)\n",
"```\n",
" # The evaluation parameters are as follows (defaults in brackets):\n",
" # imgIds - [all] N img ids to use for evaluation\n",
" # catIds - [all] K cat ids to use for evaluation\n",
" # iouThrs - [.5:.05:.95] T=10 IoU thresholds for evaluation\n",
" # recThrs - [0:.01:1] R=101 recall thresholds for evaluation\n",
" # areaRng - [...] A=4 object area ranges for evaluation\n",
" # maxDets - [1 10 100] M=3 thresholds on max detections per image\n",
"\n",
" T = len(p.iouThrs)\n",
" R = len(p.recThrs)\n",
" K = len(p.catIds) if p.useCats else 1\n",
" A = len(p.areaRng)\n",
" M = len(p.maxDets)\n",
" precision = -np.ones((T,R,K,A,M)) # -1 for the precision of absent categories\n",
" recall = -np.ones((T,K,A,M))\n",
" scores = -np.ones((T,R,K,A,M))\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"T: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]\n",
"R: [0. 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 0.12 0.13\n",
" 0.14 0.15 0.16 0.17 0.18 0.19 0.2 0.21 0.22 0.23 0.24 0.25 0.26 0.27\n",
" 0.28 0.29 0.3 0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4 0.41\n",
" 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5 0.51 0.52 0.53 0.54 0.55\n",
" 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69\n",
" 0.7 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.81 0.82 0.83\n",
" 0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97\n",
" 0.98 0.99 1. ]\n",
"K: [99]\n",
"A: [[0, 10000000000.0], [0, 1024], [1024, 9216], [9216, 10000000000.0]]\n",
"M: [1, 10, 100]\n"
]
}
],
"source": [
"print('T: ', coco_eval_segm.params.iouThrs)\n",
"print('R: ', coco_eval_segm.params.recThrs)\n",
"print('K: ', coco_eval_segm.params.catIds)\n",
"print('A: ', coco_eval_segm.params.areaRng)\n",
"print('M: ', coco_eval_segm.params.maxDets)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## try reducing thresholds to much smaller tests"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.5]"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.500\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.500\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A bit easier to start with 0.5 having 2 instances than the previous 0.252 mAP score\n",
"\n",
"let's inspect the fields that are used in `coco_eval_<type>.summarize()`, most importantly `coco_eval_<type>.eval['precision']` and `coco_eval_<type>.eval['recall']`"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1, 1, 1, 4, 3)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coco_eval_segm.eval['precision'].shape"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[[[[ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ],\n",
" [ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ]]]]])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coco_eval_segm.eval['precision']"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[[[ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ],\n",
" [ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ]]]])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coco_eval_segm.eval['recall']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What can we see?\n",
"\n",
"1. The first dim is reduced to length one, because we are now using only a single `iouThrs=[1.]`\n",
"1. The second dim is the same, since the recall threshold is also reduced to `recThrs=[0.5]`\n",
"1. The third dim is one because there is only a single class, the 'box'\n",
"1. The fourth dim is 4 because there are 4 area ranges that are being tested separately: `all`, `small`, `medium`, `large`. Our boxes have side of 50, therefore they fall into the `medium` category - this is why the second and the third row is filled with `-1`\n",
"1. The fifth dim is 3 because there are three cases tested: when only the top `1`, `10` and `100` detections are taken into consideration only.\n",
"\n",
"Ideally the `coco_eval_segm.eval['precision']` should look like the following:\n",
"```\n",
"array([[[[ 1. , 1. , 1. ],\n",
" [-1. , -1. , -1. ],\n",
" [ 1. , 1. , 1. ],\n",
" [-1. , -1. , -1. ]]]])\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Strange behaviour\n",
"\n",
"if the `recThrs` is increased over `1/num_of_inst` even by the smallest measure, then the precision score falls to zero:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.50000001]\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[[[[ 0., 0., 0.],\n",
" [-1., -1., -1.],\n",
" [ 0., 0., 0.],\n",
" [-1., -1., -1.]]]]])"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coco_eval_segm.eval['precision']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Nothing remains now, let's debug `.accumulate`\n",
"\n",
"copy the whole function from the repo, and hack a `pdb` tracer inside (line 80)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.500\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.500\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.500\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.5]\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"import pdb\n",
"import time\n",
"def accumulate(self, p = None):\n",
" '''\n",
" Accumulate per image evaluation results and store the result in self.eval\n",
" :param p: input params for evaluation\n",
" :return: None\n",
" '''\n",
" print('Accumulating evaluation results...')\n",
" tic = time.time()\n",
" if not self.evalImgs:\n",
" print('Please run evaluate() first')\n",
" # allows input customized parameters\n",
" if p is None:\n",
" p = self.params\n",
" p.catIds = p.catIds if p.useCats == 1 else [-1]\n",
" T = len(p.iouThrs)\n",
" R = len(p.recThrs)\n",
" K = len(p.catIds) if p.useCats else 1\n",
" A = len(p.areaRng)\n",
" M = len(p.maxDets)\n",
" precision = -np.ones((T,R,K,A,M)) # -1 for the precision of absent categories\n",
" recall = -np.ones((T,K,A,M))\n",
" scores = -np.ones((T,R,K,A,M))\n",
"\n",
" # create dictionary for future indexing\n",
" _pe = self._paramsEval\n",
" catIds = _pe.catIds if _pe.useCats else [-1]\n",
" setK = set(catIds)\n",
" setA = set(map(tuple, _pe.areaRng))\n",
" setM = set(_pe.maxDets)\n",
" setI = set(_pe.imgIds)\n",
" # get inds to evaluate\n",
" k_list = [n for n, k in enumerate(p.catIds) if k in setK]\n",
" m_list = [m for n, m in enumerate(p.maxDets) if m in setM]\n",
" a_list = [n for n, a in enumerate(map(lambda x: tuple(x), p.areaRng)) if a in setA]\n",
" i_list = [n for n, i in enumerate(p.imgIds) if i in setI]\n",
" I0 = len(_pe.imgIds)\n",
" A0 = len(_pe.areaRng)\n",
" # retrieve E at each category, area range, and max number of detections\n",
" for k, k0 in enumerate(k_list):\n",
" Nk = k0*A0*I0\n",
" for a, a0 in enumerate(a_list):\n",
" Na = a0*I0\n",
" for m, maxDet in enumerate(m_list):\n",
" E = [self.evalImgs[Nk + Na + i] for i in i_list]\n",
" E = [e for e in E if not e is None]\n",
" if len(E) == 0:\n",
" continue\n",
" dtScores = np.concatenate([e['dtScores'][0:maxDet] for e in E])\n",
"\n",
" # different sorting method generates slightly different results.\n",
" # mergesort is used to be consistent as Matlab implementation.\n",
" inds = np.argsort(-dtScores, kind='mergesort')\n",
" dtScoresSorted = dtScores[inds]\n",
"\n",
" dtm = np.concatenate([e['dtMatches'][:,0:maxDet] for e in E], axis=1)[:,inds]\n",
" dtIg = np.concatenate([e['dtIgnore'][:,0:maxDet] for e in E], axis=1)[:,inds]\n",
" gtIg = np.concatenate([e['gtIgnore'] for e in E])\n",
" npig = np.count_nonzero(gtIg==0 )\n",
" if npig == 0:\n",
" continue\n",
" tps = np.logical_and( dtm, np.logical_not(dtIg) )\n",
" fps = np.logical_and(np.logical_not(dtm), np.logical_not(dtIg) )\n",
"\n",
" tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float)\n",
" fp_sum = np.cumsum(fps, axis=1).astype(dtype=np.float)\n",
" for t, (tp, fp) in enumerate(zip(tp_sum, fp_sum)):\n",
" tp = np.array(tp)\n",
" fp = np.array(fp)\n",
" nd = len(tp)\n",
" rc = tp / npig\n",
" pr = tp / (fp+tp+np.spacing(1))\n",
" q = np.zeros((R,))\n",
" ss = np.zeros((R,))\n",
" \n",
"#################################################################################################### \n",
"######################## PDB ################################################################ \n",
"#################################################################################################### \n",
" pdb.set_trace()\n",
"####################################################################################################\n",
"#################################################################################################### \n",
"#################################################################################################### \n",
" \n",
" if nd:\n",
" recall[t,k,a,m] = rc[-1]\n",
" else:\n",
" recall[t,k,a,m] = 0\n",
"\n",
" # numpy is slow without cython optimization for accessing elements\n",
" # use python array gets significant speed improvement\n",
" pr = pr.tolist(); q = q.tolist()\n",
"\n",
" for i in range(nd-1, 0, -1):\n",
" if pr[i] > pr[i-1]:\n",
" pr[i-1] = pr[i]\n",
"\n",
" inds = np.searchsorted(rc, p.recThrs, side='left')\n",
" try:\n",
" for ri, pi in enumerate(inds):\n",
" q[ri] = pr[pi]\n",
" ss[ri] = dtScoresSorted[pi]\n",
" except:\n",
" pass\n",
" precision[t,:,k,a,m] = np.array(q)\n",
" scores[t,:,k,a,m] = np.array(ss)\n",
" self.eval = {\n",
" 'params': p,\n",
" 'counts': [T, R, K, A, M],\n",
" 'date': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),\n",
" 'precision': precision,\n",
" 'recall': recall,\n",
" 'scores': scores,\n",
" }\n",
" toc = time.time()\n",
" print('DONE (t={:0.2f}s).'.format( toc-tic))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Debug:\n",
"\n",
"the debugger will stop at the computation of each entry (row-major) of this array:\n",
"```\n",
"array([[[[ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ],\n",
" [ 0. , 0.5, 0.5],\n",
" [-1. , -1. , -1. ]]]])\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accumulating evaluation results...\n",
"> <ipython-input-79-2b2fdef2e0f1>(85)accumulate()\n",
"-> if nd:\n",
"(Pdb) tps\n",
"array([[False]])\n",
"(Pdb) fps\n",
"array([[ True]])\n",
"(Pdb) print('Only a single detection is allowed, and it is listed as a false-positive detection')\n",
"Only a single detection is allowed, and it is listed as a false-positive detection\n",
"(Pdb) c\n",
"> <ipython-input-79-2b2fdef2e0f1>(80)accumulate()\n",
"-> pdb.set_trace()\n",
"(Pdb) tps\n",
"array([[False, True]])\n",
"(Pdb) fps\n",
"array([[ True, False]])\n",
"(Pdb) tp\n",
"array([0., 1.])\n",
"(Pdb) fp\n",
"array([1., 1.])\n",
"(Pdb) print('something is fishy again...')\n",
"something is fishy again...\n",
"(Pdb) tps_sum\n",
"*** NameError: name 'tps_sum' is not defined\n",
"(Pdb) tp_sum\n",
"array([[0., 1.]])\n",
"(Pdb) fp_sum\n",
"array([[1., 1.]])\n",
"(Pdb) print('this is possibly the result of the np.cumsum')\n",
"this is possibly the result of the np.cumsum\n",
"(Pdb) tp\n",
"array([0., 1.])\n",
"(Pdb) tp / npig #this is assigned to the recall score\n",
"array([0. , 0.5])\n",
"(Pdb) print('more specifically the last entry of this')\n",
"more specifically the last entry of this\n",
"(Pdb) (tp / npig)[-1]\n",
"0.5\n",
"(Pdb) tp / (fp+tp+np.spacing(1)) #this is assigned to the precision score\n",
"array([0. , 0.5])\n",
"(Pdb) print('I have completely no intuition about how this will be exactly assigned to the precision score')\n",
"I have completely no intuition about how this will be exactly assigned to the precision score\n",
"(Pdb) exit\n"
]
},
{
"ename": "BdbQuit",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mBdbQuit\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-80-4467590f2895>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0maccumulate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcoco_eval_segm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-79-2b2fdef2e0f1>\u001b[0m in \u001b[0;36maccumulate\u001b[0;34m(self, p)\u001b[0m\n\u001b[1;32m 78\u001b[0m \u001b[0;31m######################## PDB ################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 79\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m \u001b[0mpdb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 81\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 82\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-79-2b2fdef2e0f1>\u001b[0m in \u001b[0;36maccumulate\u001b[0;34m(self, p)\u001b[0m\n\u001b[1;32m 78\u001b[0m \u001b[0;31m######################## PDB ################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 79\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m \u001b[0mpdb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 81\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 82\u001b[0m \u001b[0;31m####################################################################################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/envs/debugmask/lib/python3.7/bdb.py\u001b[0m in \u001b[0;36mtrace_dispatch\u001b[0;34m(self, frame, event, arg)\u001b[0m\n\u001b[1;32m 86\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;31m# None\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 87\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'line'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_line\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 89\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'call'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 90\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/anaconda3/envs/debugmask/lib/python3.7/bdb.py\u001b[0m in \u001b[0;36mdispatch_line\u001b[0;34m(self, frame)\u001b[0m\n\u001b[1;32m 111\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstop_here\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbreak_here\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 112\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muser_line\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 113\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquitting\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mBdbQuit\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 114\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrace_dispatch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 115\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mBdbQuit\u001b[0m: "
]
}
],
"source": [
"accumulate(coco_eval_segm)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion:\n",
"The debugging suggests that the ordering of the annotations could be misaligned, leading to false positive entries which will reduce the score"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<pre>\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"</pre>\n",
"### A few more cases with increasing number of instances"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO GT annots\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Parsing images: 100%|██████████| 1/1 [00:00<00:00, 213.83it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO Predictions\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"Parsing images: 1it [00:00, 521.03it/s]\n"
]
}
],
"source": [
"num_of_inst = 3\n",
"dataset = debugdataset.DebugDataset(\n",
" dataset_size=1, \n",
" min_num_instances=num_of_inst, \n",
" max_num_instances=num_of_inst\n",
")\n",
"\n",
"predictions = []\n",
"for img, target, idx in dataset:\n",
" target.add_field(\"scores\", torch.ones(len(target))) \n",
" predictions.append(target)\n",
"\n",
"cocoGt = COCOWrapper(dataset)\n",
"cocoDt = COCOWrapper(dataset, predictions)\n",
"coco_eval_bbox = COCOeval(cocoGt, cocoDt, 'bbox')\n",
"coco_eval_segm = COCOeval(cocoGt, cocoDt, 'segm')"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAAEuElEQVR4nO3duwoCQRQFQa/4/798TQ1E8UUrVqWbnGChmWCYwwEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPhnUw/gvt2tJ1wx4+cBeN6xHgAA/0iAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABA41QMA4FW7W0+4YmZufHUCBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAInOoB3Dcz9QQA3swJGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANA4OHHGHb3Ezte5LkCAH6LEzAABAQYAAICDAABAQaAgAADQECAASDw8DUkAPg2v3gZ1QkYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXDoDJEQJ8LDesNcAAAAASUVORK5CYII=\n",
"text/plain": [
"<PIL.Image.Image image mode=RGB size=640x480 at 0x7F65AC9EDA90>"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"img, target, idx = dataset[0]\n",
"img"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualize the masks as well for safety"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 0\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbJJREFUeJzt3X+s3XV9x/Hna/QH/qSAhHRts2IkM/yxAWkQojEG4obMCH+gwZjZmC5NNpZoXOJgS7aY7A/dH6Imi64Rt7o4haEbDWFhCJhlf1it8kOgQ65O03ZgpwK6GRnoe3+cT91ZV733Te+951x8PpKb+/1+vt97zvuQ02fPj3toqgpJ0tL80qwHkKS1xGhKUoPRlKQGoylJDUZTkhqMpiQ1rEg0k1ye5JEkC0muW4nrkKRZyHL/nmaSU4CvAa8HDgNfAt5aVQ8v6xVJ0gysxCPNi4CFqvpGVf038GngyhW4HkladetW4DK3AIem9g8Dr/p5P7AhG+tUXrQCo0jS0vyAJ75TVWctdt5KRHNJkuwGdgOcygt5VS6b1SiSxOfqlm8t5byVeHp+BNg2tb91rP0fVbWnqnZU1Y71bFyBMSRp+a1ENL8EnJvknCQbgGuAfStwPZK06pb96XlVPZvk94E7gFOAj1fVQ8t9PZI0CyvymmZV3Q7cvhKXLUmz5CeCJKnBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNi0YzyceTHE3y4NTaGUnuTPLo+H76WE+SDydZSPJAkgtXcnhJWm1LeaT518Dlx61dB9xVVecCd419gDcA546v3cBHlmdMSZoPi0azqv4Z+N5xy1cCe8f2XuCqqfVP1MQXgE1JNi/XsJI0a8/1Nc2zq+qxsf04cPbY3gIcmjrv8FiTpOeFk34jqKoKqO7PJdmd5ECSA8/w9MmOIUmr4rlG89vHnnaP70fH+hFg29R5W8fa/1NVe6pqR1XtWM/G5ziGJK2u5xrNfcDOsb0TuHVq/e3jXfSLgaemnsZL0pq3brETknwKeB3wsiSHgT8F3gfcnGQX8C3gLeP024ErgAXgh8A7VmBmSZqZRaNZVW/9GYcuO8G5BVx7skNJ0rzyE0GS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNi0YzybYk9yR5OMlDSd451s9IcmeSR8f308d6knw4yUKSB5JcuNI3QpJWy1IeaT4L/EFVnQdcDFyb5DzgOuCuqjoXuGvsA7wBOHd87QY+suxTS9KMLBrNqnqsqr4ytn8AHAS2AFcCe8dpe4GrxvaVwCdq4gvApiSbl31ySZqB1muaSbYDFwD7gbOr6rFx6HHg7LG9BTg09WOHx9rxl7U7yYEkB57h6ebYkjQbS45mkhcDnwHeVVXfnz5WVQVU54qrak9V7aiqHevZ2PlRSZqZJUUzyXomwfxkVX12LH/72NPu8f3oWD8CbJv68a1jTZLWvKW8ex7gRuBgVX1g6tA+YOfY3gncOrX+9vEu+sXAU1NP4yVpTVu3hHNeDfw28NUk9421PwLeB9ycZBfwLeAt49jtwBXAAvBD4B3LOrEkzdCi0ayqfwHyMw5fdoLzC7j2JOeSpLnkJ4IkqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ2LRjPJqUm+mOT+JA8lee9YPyfJ/iQLSW5KsmGsbxz7C+P49pW9CZK0epbySPNp4NKq+nXgfODyJBcD7wduqKpXAE8Au8b5u4AnxvoN4zxJel5YNJo18Z9jd/34KuBS4Jaxvhe4amxfOfYZxy9LkmWbWJJmaEmvaSY5Jcl9wFHgTuDrwJNV9ew45TCwZWxvAQ4BjONPAWee4DJ3JzmQ5MAzPH1yt0KSVsmSollVP66q84GtwEXAK0/2iqtqT1XtqKod69l4shcnSaui9e55VT0J3ANcAmxKsm4c2gocGdtHgG0A4/hpwHeXZVpJmrGlvHt+VpJNY/sFwOuBg0ziefU4bSdw69jeN/YZx++uqlrOoSVpVtYtfgqbgb1JTmES2Zur6rYkDwOfTvJnwL3AjeP8G4G/SbIAfA+4ZgXmlqSZWDSaVfUAcMEJ1r/B5PXN49d/BLx5WaaTpDnjJ4IkqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqWEpH6PUCrrj3+9bscv+zV8+f8UuW/pF5SNNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBv81yhnzX4yU1hYfaUpSg9GUpAajKUkNRlOSGpYczSSnJLk3yW1j/5wk+5MsJLkpyYaxvnHsL4zj21dmdElafZ1Hmu8EDk7tvx+4oapeATwB7Brru4AnxvoN4zxJel5YUjSTbAV+C/jY2A9wKXDLOGUvcNXYvnLsM45fNs6XpDVvqY80Pwi8B/jJ2D8TeLKqnh37h4EtY3sLcAhgHH9qnC9Ja96i0UzyRuBoVX15Oa84ye4kB5IceIanl/OiJWnFLOUTQa8G3pTkCuBU4KXAh4BNSdaNR5NbgSPj/CPANuBwknXAacB3j7/QqtoD7AF4ac6ok70hkrQaFn2kWVXXV9XWqtoOXAPcXVVvA+4Brh6n7QRuHdv7xj7j+N1VZRQlPS+czO9p/iHw7iQLTF6zvHGs3wicOdbfDVx3ciNK0vxo/Q87qurzwOfH9jeAi05wzo+ANy/DbJI0d/xEkCQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDamqWc9Akh8Aj8x6jufgZcB3Zj1EkzOvnrU49y/yzL9SVWctdtK6Zbii5fBIVe2Y9RBdSQ6stbmdefWsxbmdeXE+PZekBqMpSQ3zEs09sx7gOVqLczvz6lmLczvzIubijSBJWivm5ZGmJK0JM49mksuTPJJkIcl1s57nmCQfT3I0yYNTa2ckuTPJo+P76WM9ST48bsMDSS6c0czbktyT5OEkDyV55xqZ+9QkX0xy/5j7vWP9nCT7x3w3Jdkw1jeO/YVxfPss5h6znJLk3iS3rYWZk3wzyVeT3JfkwFib9/vHpiS3JPnXJAeTXDLTmatqZl/AKcDXgZcDG4D7gfNmOdPUbK8FLgQenFr7c+C6sX0d8P6xfQXwj0CAi4H9M5p5M3Dh2H4J8DXgvDUwd4AXj+31wP4xz83ANWP9o8Dvju3fAz46tq8Bbprh/eTdwN8Ct439uZ4Z+CbwsuPW5v3+sRf4nbG9Adg0y5lnckeb+o9xCXDH1P71wPWznOm4+bYfF81HgM1jezOT3y8F+EvgrSc6b8bz3wq8fi3NDbwQ+ArwKia/sLzu+PsKcAdwydheN87LDGbdCtwFXArcNv6gzvvMJ4rm3N4/gNOAfzv+v9UsZ5710/MtwKGp/cNjbV6dXVWPje3HgbPH9tzdjvH07wImj9rmfu7xNPc+4ChwJ5NnIE9W1bMnmO2nc4/jTwFnru7EAHwQeA/wk7F/JvM/cwH/lOTLSXaPtXm+f5wD/AfwV+NlkI8leREznHnW0VyzavLX2Fz+6kGSFwOfAd5VVd+fPjavc1fVj6vqfCaP3i4CXjnjkX6uJG8EjlbVl2c9S9NrqupC4A3AtUleO31wDu8f65i8TPaRqroA+C8mT8d/arVnnnU0jwDbpva3jrV59e0kmwHG96NjfW5uR5L1TIL5yar67Fie+7mPqaongXuYPLXdlOTYR32nZ/vp3OP4acB3V3nUVwNvSvJN4NNMnqJ/iPmemao6Mr4fBf6eyV9Q83z/OAwcrqr9Y/8WJhGd2cyzjuaXgHPHO44bmLxAvm/GM/08+4CdY3snk9cMj62/fbxzdzHw1NRTh1WTJMCNwMGq+sDUoXmf+6wkm8b2C5i8DnuQSTyvHqcdP/ex23M1cPd4tLFqqur6qtpaVduZ3G/vrqq3McczJ3lRkpcc2wZ+A3iQOb5/VNXjwKEkvzqWLgMenunMq/mi7s94ofcKJu/yfh3441nPMzXXp4DHgGeY/G23i8lrUHcBjwKfA84Y5wb4i3EbvgrsmNHMr2HyNOUB4L7xdcUamPvXgHvH3A8CfzLWXw58EVgA/g7YONZPHfsL4/jLZ3xfeR3/++753M48Zrt/fD107M/bGrh/nA8cGPePfwBOn+XMfiJIkhpm/fRcktYUoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUsP/ABWrjEvNx3EDAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 1\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbhJREFUeJzt3X+s3XV9x/Hna/2FPykgIV3brBDJDH9sQBqEaIyBuCEzwh/oMGY0hqXJxhKNS1zZki0m+0P3h7+SBdeIW12cwNCNhrAwBMyyP6xU+SHQIVenaTuwUwHdjAz0vT/Op+6uq977bu+951z2fCQ39/v9fL/33Pchp8+e7zn3llQVkqTF+YVpDyBJq4nRlKQGoylJDUZTkhqMpiQ1GE1JaliWaCa5PMnjSeaS7FqO7yFJ05Cl/jnNJGuArwFvAg4B9wPvqKrHlvQbSdIULMczzYuAuar6RlX9F3AzcOUyfB9JWnFrl+E2NwMH5+0fAl77875gfTbUKbxsGUaRpMX5AU9/p6rOXOi85YjmoiTZCewEOIWX8tpcNq1RJInP123fWsx5y3F5fhjYOm9/y1j7X6pqd1Vtr6rt69iwDGNI0tJbjmjeD5yb5Owk64FrgL3L8H0kacUt+eV5Vb2Q5PeAu4A1wCer6tGl/j6SNA3L8ppmVd0J3Lkcty1J0+RvBElSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGhaMZpJPJjmS5JF5a6cnuTvJE+PzaWM9ST6WZC7Jw0kuXM7hJWmlLeaZ5l8Blx+ztgu4p6rOBe4Z+wBvBs4dHzuBG5dmTEmaDQtGs6r+CfjeMctXAnvG9h7gqnnrn6qJLwIbk2xaqmEladpO9DXNs6rqybH9FHDW2N4MHJx33qGxJkkvCif9RlBVFVDdr0uyM8n+JPuf57mTHUOSVsSJRvPbRy+7x+cjY/0wsHXeeVvG2v9RVburantVbV/HhhMcQ5JW1olGcy+wY2zvAG6ft37teBf9YuDZeZfxkrTqrV3ohCSfAd4IvCrJIeBPgA8Atya5DvgW8PZx+p3AFcAc8EPgXcswsyRNzYLRrKp3/IxDlx3n3AKuP9mhJGlW+RtBktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDQtGM8nWJPcleSzJo0nePdZPT3J3kifG59PGepJ8LMlckoeTXLjcd0KSVspinmm+APx+VZ0HXAxcn+Q8YBdwT1WdC9wz9gHeDJw7PnYCNy751JI0JQtGs6qerKqvjO0fAAeAzcCVwJ5x2h7gqrF9JfCpmvgisDHJpiWfXJKmoPWaZpJtwAXAPuCsqnpyHHoKOGtsbwYOzvuyQ2Pt2NvamWR/kv3P81xzbEmajkVHM8nLgc8C76mq788/VlUFVOcbV9XuqtpeVdvXsaHzpZI0NYuKZpJ1TIL56ar63Fj+9tHL7vH5yFg/DGyd9+VbxpokrXqLefc8wE3Agar60LxDe4EdY3sHcPu89WvHu+gXA8/Ou4yXpFVt7SLOeR3wW8BXkzw41v4Q+ABwa5LrgG8Bbx/H7gSuAOaAHwLvWtKJJWmKFoxmVf0zkJ9x+LLjnF/A9Sc5lyTNJH8jSJIajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUsGA0k5yS5EtJHkryaJL3j/Wzk+xLMpfkliTrx/qGsT83jm9b3rsgSStnMc80nwMurapfBc4HLk9yMfBB4MNV9WrgaeC6cf51wNNj/cPjPEl6UVgwmjXxH2N33fgo4FLgtrG+B7hqbF859hnHL0uSJZtYkqZoUa9pJlmT5EHgCHA38HXgmap6YZxyCNg8tjcDBwHG8WeBM45zmzuT7E+y/3meO7l7IUkrZFHRrKofV9X5wBbgIuA1J/uNq2p3VW2vqu3r2HCyNydJK6L17nlVPQPcB1wCbEyydhzaAhwe24eBrQDj+KnAd5dkWkmassW8e35mko1j+yXAm4ADTOJ59ThtB3D72N479hnH762qWsqhJWla1i58CpuAPUnWMInsrVV1R5LHgJuT/CnwAHDTOP8m4K+TzAHfA65ZhrklaSoWjGZVPQxccJz1bzB5ffPY9R8Bb1uS6SRpxvgbQZLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkhkVHM8maJA8kuWPsn51kX5K5JLckWT/WN4z9uXF82/KMLkkrb23j3HcDB4BXjv0PAh+uqpuTfBy4DrhxfH66ql6d5Jpx3m+e6IB3/duDJ/qlC/r1Xzx/2W5b0ovTop5pJtkC/AbwibEf4FLgtnHKHuCqsX3l2Gccv2ycL0mr3mIvzz8CvA/4ydg/A3imql4Y+4eAzWN7M3AQYBx/dpwvSavegtFM8hbgSFV9eSm/cZKdSfYn2f88zy3lTUvSslnMa5qvA96a5ArgFCavaX4U2Jhk7Xg2uQU4PM4/DGwFDiVZC5wKfPfYG62q3cBugFfm9DrZOyJJK2HBZ5pVdUNVbamqbcA1wL1V9U7gPuDqcdoO4PaxvXfsM47fW1VGUdKLwsn8nOYfAO9NMsfkNcubxvpNwBlj/b3ArpMbUZJmR+dHjqiqLwBfGNvfAC46zjk/At62BLNJ0szxN4IkqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWpo/Xua0+D/ZlfSLPGZpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDamqac9Akh8Aj097jhPwKuA70x6iyZlXzmqc+//zzL9UVWcudNKs/CPEj1fV9mkP0ZVk/2qb25lXzmqc25kX5uW5JDUYTUlqmJVo7p72ACdoNc7tzCtnNc7tzAuYiTeCJGm1mJVnmpK0Kkw9mkkuT/J4krkku6Y9z1FJPpnkSJJH5q2dnuTuJE+Mz6eN9ST52LgPDye5cEozb01yX5LHkjya5N2rZO5TknwpyUNj7veP9bOT7Bvz3ZJk/VjfMPbnxvFt05h7zLImyQNJ7lgNMyf5ZpKvJnkwyf6xNuuPj41JbkvyL0kOJLlkqjNX1dQ+gDXA14FzgPXAQ8B505xp3mxvAC4EHpm39mfArrG9C/jg2L4C+AcgwMXAvinNvAm4cGy/AvgacN4qmDvAy8f2OmDfmOdW4Jqx/nHgd8b27wIfH9vXALdM8XHyXuBvgDvG/kzPDHwTeNUxa7P++NgD/PbYXg9snObMU3mgzfuPcQlw17z9G4AbpjnTMfNtOyaajwObxvYmJj9fCvAXwDuOd96U578deNNqmht4KfAV4LVMfmB57bGPFeAu4JKxvXaclynMugW4B7gUuGP8QZ31mY8XzZl9fACnAv967H+rac487cvzzcDBefuHxtqsOquqnhzbTwFnje2Zux/j8u8CJs/aZn7ucZn7IHAEuJvJFcgzVfXCcWb76dzj+LPAGSs7MQAfAd4H/GTsn8Hsz1zAPyb5cpKdY22WHx9nA/8O/OV4GeQTSV7GFGeedjRXrZr8NTaTP3qQ5OXAZ4H3VNX35x+b1bmr6sdVdT6TZ28XAa+Z8kg/V5K3AEeq6svTnqXp9VV1IfBm4Pokb5h/cAYfH2uZvEx2Y1VdAPwnk8vxn1rpmacdzcPA1nn7W8barPp2kk0A4/ORsT4z9yPJOibB/HRVfW4sz/zcR1XVM8B9TC5tNyY5+qu+82f76dzj+KnAd1d41NcBb03yTeBmJpfoH2W2Z6aqDo/PR4C/Y/IX1Cw/Pg4Bh6pq39i/jUlEpzbztKN5P3DueMdxPZMXyPdOeaafZy+wY2zvYPKa4dH1a8c7dxcDz867dFgxSQLcBByoqg/NOzTrc5+ZZOPYfgmT12EPMInn1eO0Y+c+en+uBu4dzzZWTFXdUFVbqmobk8ftvVX1TmZ45iQvS/KKo9vArwGPMMOPj6p6CjiY5JfH0mXAY1OdeSVf1P0ZL/ReweRd3q8DfzTteebN9RngSeB5Jn/bXcfkNah7gCeAzwOnj3MD/Pm4D18Ftk9p5tczuUx5GHhwfFyxCub+FeCBMfcjwB+P9XOALwFzwN8CG8b6KWN/bhw/Z8qPlTfyP++ez+zMY7aHxsejR/+8rYLHx/nA/vH4+HvgtGnO7G8ESVLDtC/PJWlVMZqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDX8N2jBjUPXyIkvAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 2\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbJJREFUeJzt3X+o3fV9x/Hna/llq63xFyFLwqI0rPjHphI0YilFcVNXGv+wRSkzlIzA5sDioIsbbBT2R90ftRWGXahu6ehane1qEIezahn7o9FY48/MenVKkqlZraZupU7b9/44n7izLHrvJ7n3fs91zwdc7vf7+X7vPe8TTp4533PuJakqJEkz80tDDyBJC4nRlKQORlOSOhhNSepgNCWpg9GUpA5zEs0klyR5OslUkq1zcRuSNITM9s9pJlkE/BC4GNgHPARcVVVPzeoNSdIA5uKZ5rnAVFU9V1X/BXwT2DgHtyNJ827xHHzPVcDesf19wHnv9gVLs6yO4/g5GEWSZuZ1Xv1RVZ023XlzEc0ZSbIF2AJwHO/nvFw01CiSxHfrjhdmct5cXJ7vB9aM7a9ua/9LVW2rqvVVtX4Jy+ZgDEmafXMRzYeAdUlOT7IUuBLYMQe3I0nzbtYvz6vqrSS/D9wDLAJuraonZ/t2JGkIc/KaZlXdDdw9F99bkobkbwRJUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUYdpoJrk1yYEkT4ytnZzk3iTPtM8ntfUkuSnJVJLHkpwzl8NL0nybyTPNvwYuOWxtK3BfVa0D7mv7AJcC69rHFuDm2RlTkibDtNGsqn8CfnzY8kZge9veDlw+tv61Gvk+sDzJytkaVpKGdrSvaa6oqhfb9kvAira9Ctg7dt6+tiZJ7wnH/EZQVRVQvV+XZEuSXUl2vckbxzqGJM2Lo43my4cuu9vnA219P7Bm7LzVbe3/qKptVbW+qtYvYdlRjiFJ8+too7kD2NS2NwF3jq1f3d5F3wAcHLuMl6QFb/F0JyT5BvAx4NQk+4A/Bb4A3J5kM/AC8Kl2+t3AZcAU8FPgM3MwsyQNZtpoVtVV73DooiOcW8A1xzqUJE0qfyNIkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjpMG80ka5I8kOSpJE8mubatn5zk3iTPtM8ntfUkuSnJVJLHkpwz13dCkubLTJ5pvgX8QVWdCWwArklyJrAVuK+q1gH3tX2AS4F17WMLcPOsTy1JA5k2mlX1YlX9oG2/DuwBVgEbge3ttO3A5W17I/C1Gvk+sDzJylmfXJIG0PWaZpK1wNnATmBFVb3YDr0ErGjbq4C9Y1+2r60d/r22JNmVZNebvNE5tiQNY8bRTHIC8C3gs1X1k/FjVVVA9dxwVW2rqvVVtX4Jy3q+VJIGM6NoJlnCKJhfr6pvt+WXD112t88H2vp+YM3Yl69ua5K04M3k3fMAtwB7quqLY4d2AJva9ibgzrH1q9u76BuAg2OX8ZK0oC2ewTkXAL8NPJ5kd1v7I+ALwO1JNgMvAJ9qx+4GLgOmgJ8Cn5nViSVpQNNGs6r+Gcg7HL7oCOcXcM0xziVJE8nfCJKkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JanDtNFMclySB5M8muTJJJ9v66cn2ZlkKsltSZa29WVtf6odXzu3d0GS5s9Mnmm+AVxYVb8OnAVckmQDcANwY1V9CHgV2NzO3wy82tZvbOdJ0nvCtNGskf9ou0vaRwEXAne09e3A5W17Y9unHb8oSWZtYkka0Ixe00yyKMlu4ABwL/As8FpVvdVO2QesaturgL0A7fhB4JQjfM8tSXYl2fUmbxzbvZCkeTKjaFbVz6vqLGA1cC7w4WO94araVlXrq2r9EpYd67eTpHnR9e55Vb0GPACcDyxPsrgdWg3sb9v7gTUA7fiJwCuzMq0kDWwm756flmR5234fcDGwh1E8r2inbQLubNs72j7t+P1VVbM5tCQNZfH0p7AS2J5kEaPI3l5VdyV5Cvhmkj8DHgFuaeffAvxNkingx8CVczC3JA1i2mhW1WPA2UdYf47R65uHr/8M+OSsTCdJE8bfCJKkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOszk/wiSpEHc82+75+T7/uYvn3XUX+szTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSeow42gmWZTkkSR3tf3Tk+xMMpXktiRL2/qytj/Vjq+dm9Elaf71PNO8Ftgztn8DcGNVfQh4Fdjc1jcDr7b1G9t5kvSeMKNoJlkN/Bbw1bYf4ELgjnbKduDytr2x7dOOX9TOl6QFb6bPNL8EfA74Rds/BXitqt5q+/uAVW17FbAXoB0/2M6XpAVv2mgm+ThwoKoens0bTrIlya4ku97kjdn81pI0Z2byX/heAHwiyWXAccAHgS8Dy5Msbs8mVwP72/n7gTXAviSLgROBVw7/plW1DdgG8MGcXMd6RyS99xzLf7U7V6Z9pllV11fV6qpaC1wJ3F9VnwYeAK5op20C7mzbO9o+7fj9VWUUJb0nHMvPaf4hcF2SKUavWd7S1m8BTmnr1wFbj21ESZocM7k8f1tVfQ/4Xtt+Djj3COf8DPjkLMwmSRPH3wiSpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqkKoaegaSvA48PfQcR+FU4EdDD9HJmefPQpz7//PMv1JVp0130uJZuKHZ8HRVrR96iF5Jdi20uZ15/izEuZ15el6eS1IHoylJHSYlmtuGHuAoLcS5nXn+LMS5nXkaE/FGkCQtFJPyTFOSFoTBo5nkkiRPJ5lKsnXoeQ5JcmuSA0meGFs7Ocm9SZ5pn09q60lyU7sPjyU5Z6CZ1yR5IMlTSZ5Mcu0Cmfu4JA8mebTN/fm2fnqSnW2+25IsbevL2v5UO752iLnbLIuSPJLkroUwc5LnkzyeZHeSXW1t0h8fy5PckeRfkuxJcv6gM1fVYB/AIuBZ4AxgKfAocOaQM43N9lHgHOCJsbU/B7a27a3ADW37MuAfgAAbgJ0DzbwSOKdtfwD4IXDmApg7wAltewmws81zO3BlW/8K8Ltt+/eAr7TtK4HbBnycXAf8LXBX25/omYHngVMPW5v0x8d24Hfa9lJg+ZAzD/JAG/vDOB+4Z2z/euD6IWc6bL61h0XzaWBl217J6OdLAf4SuOpI5w08/53AxQtpbuD9wA+A8xj9wPLiwx8rwD3A+W17cTsvA8y6GrgPuBC4q/1FnfSZjxTNiX18ACcC/3r4n9WQMw99eb4K2Du2v6+tTaoVVfVi234JWNG2J+5+tMu/sxk9a5v4udtl7m7gAHAvoyuQ16rqrSPM9vbc7fhB4JT5nRiALwGfA37R9k9h8mcu4B+TPJxkS1ub5MfH6cC/A3/VXgb5apLjGXDmoaO5YNXon7GJ/NGDJCcA3wI+W1U/GT82qXNX1c+r6ixGz97OBT488EjvKsnHgQNV9fDQs3T6SFWdA1wKXJPko+MHJ/DxsZjRy2Q3V9XZwH8yuhx/23zPPHQ09wNrxvZXt7VJ9XKSlQDt84G2PjH3I8kSRsH8elV9uy1P/NyHVNVrwAOMLm2XJzn0q77js709dzt+IvDKPI96AfCJJM8D32R0if5lJntmqmp/+3wA+HtG/0BN8uNjH7Cvqna2/TsYRXSwmYeO5kPAuvaO41JGL5DvGHimd7MD2NS2NzF6zfDQ+tXtnbsNwMGxS4d5kyTALcCeqvri2KFJn/u0JMvb9vsYvQ67h1E8r2inHT73oftzBXB/e7Yxb6rq+qpaXVVrGT1u76+qTzPBMyc5PskHDm0DvwE8wQQ/PqrqJWBvkl9tSxcBTw0683y+qPsOL/Rexuhd3meBPx56nrG5vgG8CLzJ6F+7zYxeg7oPeAb4LnByOzfAX7T78DiwfqCZP8LoMuUxYHf7uGwBzP1rwCNt7ieAP2nrZwAPAlPA3wHL2vpxbX+qHT9j4MfKx/ifd88nduY226Pt48lDf98WwOPjLGBXe3x8BzhpyJn9jSBJ6jD05bkkLShGU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqcN/A6Z6jEkVKvDHAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"for inst_idx, mask in enumerate(target.get_field('masks')):\n",
" print(\"instance:\", inst_idx)\n",
" plt.imshow(mask.get_mask_tensor())\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now compute the results"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.442\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_bbox.params.iouThrs = [1.]\n",
"coco_eval_bbox.params.recThrs = [.66666]\n",
"\n",
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.66666]\n",
"\n",
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision:\n",
"[[[[[ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]]]]]\n",
"\n",
"\n",
"Recall:\n",
"[[[[ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]]]]\n"
]
}
],
"source": [
"print('Precision:')\n",
"print(coco_eval_segm.eval['precision'])\n",
"print('\\n\\nRecall:')\n",
"print(coco_eval_segm.eval['recall'])"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.667\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_bbox.params.iouThrs = [1.]\n",
"coco_eval_bbox.params.recThrs = [.7]\n",
"\n",
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.7]\n",
"\n",
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision:\n",
"[[[[[ 0. 0. 0.]\n",
" [-1. -1. -1.]\n",
" [ 0. 0. 0.]\n",
" [-1. -1. -1.]]]]]\n",
"\n",
"\n",
"Recall:\n",
"[[[[ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.66666667 0.66666667]\n",
" [-1. -1. -1. ]]]]\n"
]
}
],
"source": [
"print('Precision:')\n",
"print(coco_eval_segm.eval['precision'])\n",
"print('\\n\\nRecall:')\n",
"print(coco_eval_segm.eval['recall'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4 instances should systematically end up in 0.75"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO GT annots\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Parsing images: 100%|██████████| 1/1 [00:00<00:00, 156.83it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building COCO Predictions\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"Parsing images: 1it [00:00, 365.07it/s]\n"
]
}
],
"source": [
"num_of_inst = 4\n",
"dataset = debugdataset.DebugDataset(\n",
" dataset_size=1, \n",
" min_num_instances=num_of_inst, \n",
" max_num_instances=num_of_inst\n",
")\n",
"\n",
"predictions = []\n",
"for img, target, idx in dataset:\n",
" target.add_field(\"scores\", torch.ones(len(target))) \n",
" predictions.append(target)\n",
"\n",
"cocoGt = COCOWrapper(dataset)\n",
"cocoDt = COCOWrapper(dataset, predictions)\n",
"coco_eval_bbox = COCOeval(cocoGt, cocoDt, 'bbox')\n",
"coco_eval_segm = COCOeval(cocoGt, cocoDt, 'segm')"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAAFaElEQVR4nO3Vi2kDQRAFQc3ln/M4gTNYH7utU1UAy2Ng6dsNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgXUw/4FbtbTzgxc81rA/CAox4AAJ9IgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADxi6gEA8DK7W084MXNS2+PvdwAAAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA4KemHgDXsbv1hBMzvjn8R0c9AAA+kQADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAtzb1AAD41u7WE07MvKCex/NPAAD3EmAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAQEGAACAgwAAQEGAACAgwAAQEGgIAAA0BAgAEgIMAAEBBgAAgIMAAEBBgAAgIMAAEBBoCAAANAQIABICDAABAQYAAICDAABAQYAAICDAABAQaAgAADQECAASAgwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMBlfQGQ5w2NLu3OxwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<PIL.Image.Image image mode=RGB size=640x480 at 0x7F66280B87B8>"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"img, target, idx = dataset[0]\n",
"img"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualize the masks as well for safety"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 0\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbBJREFUeJzt3X+s3XV9x/Hna/2FPykgIV3bDIxkhj82IA1CNMZA3JAZ8Q80GDMb06XJxhKNS7RsyRaT/aH7Q9Rk0TXiVhenMHRCCAtDwCz7Q7SOHwIdcnWatgM7FdDNyEDf++N8ys666r3v9t57zmXPR3Jzv9/P93vPeZ/m9Nnz4x5IVSFJWppfmvUAkrSWGE1JajCaktRgNCWpwWhKUoPRlKSGFYlmksuTPJJkIcmelbgOSZqFLPfvaSZZB3wDeD1wCPgq8LaqenhZr0iSZmAlHmleBCxU1beq6r+AzwJXrsD1SNKqW78Cl7kVODi1fwh41S/6gY3ZVKfwohUYRZKW5kc88b2qOnOx81YimkuSZDewG+AUXsirctmsRpEkvlg3fWcp563E0/PDwPap/W1j7X+pqr1VtaOqdmxg0wqMIUnLbyWi+VXg3CTnJNkIXA3csgLXI0mrbtmfnlfVs0l+H7gdWAd8sqoeWu7rkaRZWJHXNKvqNuC2lbhsSZolPxEkSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWpYNJpJPpnkSJIHp9ZOT3JHkkfH99PGepJ8NMlCkgeSXLiSw0vSalvKI82/Ai4/Zm0PcGdVnQvcOfYB3gCcO752Ax9bnjElaT4sGs2q+kfgB8csXwnsG9v7gDdPrX+qJr4MbE6yZbmGlaRZO9HXNM+qqsfG9uPAWWN7K3Bw6rxDY02SnhdO+o2gqiqguj+XZHeS/Un2P8PTJzuGJK2KE43md48+7R7fj4z1w8D2qfO2jbX/o6r2VtWOqtqxgU0nOIYkra4TjeYtwM6xvRO4eWr9HeNd9IuBp6aexkvSmrd+sROSfAZ4HfCyJIeAPwE+ANyYZBfwHeCt4/TbgCuABeDHwDtXYGZJmplFo1lVb/s5hy47zrkFXHOyQ0nSvPITQZLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlKD0ZSkBqMpSQ2LRjPJ9iR3J3k4yUNJ3jXWT09yR5JHx/fTxnqSfDTJQpIHkly40jdCklbLUh5pPgv8QVWdB1wMXJPkPGAPcGdVnQvcOfYB3gCcO752Ax9b9qklaUbWL3ZCVT0GPDa2f5TkALAVuBJ43ThtH/Al4H1j/VNVVcCXk2xOsmVcjiS13P5v963YZf/mL5/f/pnWa5pJzgYuAO4BzpoK4ePAWWN7K3Bw6scOjbVjL2t3kv1J9j/D082xJWk2lhzNJC8GPge8u6p+OH1sPKqszhVX1d6q2lFVOzawqfOjkjQzS4pmkg1Mgvnpqvr8WP5uki3j+BbgyFg/DGyf+vFtY02S1rylvHse4HrgQFV9aOrQLcDOsb0TuHlq/R3jXfSLgad8PVPS88WibwQBrwZ+G/h6kqOvyP4h8AHgxiS7gO8Abx3HbgOuABaAHwPvXNaJJWmGlvLu+T8B+TmHLzvO+QVcc5JzSdJc8hNBktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JaljKf7ldkmbmRP43uyvJR5qS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSw6LRTHJKkq8kuT/JQ0neP9bPSXJPkoUkNyTZONY3jf2Fcfzslb0JkrR6lvJI82ng0qr6deB84PIkFwMfBK6rqlcATwC7xvm7gCfG+nXjPEl6Xlg0mjXxH2N3w/gq4FLgprG+D3jz2L5y7DOOX5YkyzaxJM3Qkl7TTLIuyX3AEeAO4JvAk1X17DjlELB1bG8FDgKM408BZxznMncn2Z9k/zM8fXK3QpJWyZKiWVU/rarzgW3ARcArT/aKq2pvVe2oqh0b2HSyFydJq6L17nlVPQncDVwCbE6yfhzaBhwe24eB7QDj+KnA95dlWkmasaW8e35mks1j+wXA64EDTOJ51ThtJ3Dz2L5l7DOO31VVtZxDS9KsrF/8FLYA+5KsYxLZG6vq1iQPA59N8qfAvcD14/zrgb9OsgD8ALh6BeaWpJlYNJpV9QBwwXHWv8Xk9c1j138CvGVZppOkOeMngiSpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUZTkhqMpiQ1GE1JajCaktRgNCWpwWhKUoPRlKQGoylJDUuOZpJ1Se5NcuvYPyfJPUkWktyQZONY3zT2F8bxs1dmdElafZ1Hmu8CDkztfxC4rqpeATwB7Brru4Anxvp14zxJel5YUjSTbAN+C/jE2A9wKXDTOGUf8OaxfeXYZxy/bJwvSWveUh9pfhh4L/CzsX8G8GRVPTv2DwFbx/ZW4CDAOP7UOF+S1rxFo5nkjcCRqvracl5xkt1J9ifZ/wxPL+dFS9KKWb+Ec14NvCnJFcApwEuBjwCbk6wfjya3AYfH+YeB7cChJOuBU4HvH3uhVbUX2Avw0pxeJ3tDJGk1LPpIs6quraptVXU2cDVwV1W9HbgbuGqcthO4eWzfMvYZx++qKqMo6XnhZH5P833Ae5IsMHnN8vqxfj1wxlh/D7Dn5EaUpPmxlKfnz6mqLwFfGtvfAi46zjk/Ad6yDLNJ0tzxE0GS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDUYTUlqMJqS1GA0JanBaEpSg9GUpAajKUkNRlOSGoymJDWkqmY9A0l+BDwy6zlOwMuA7816iCZnXj1rce7/zzP/SlWdudhJ65fhipbDI1W1Y9ZDdCXZv9bmdubVsxbndubF+fRckhqMpiQ1zEs09856gBO0Fud25tWzFud25kXMxRtBkrRWzMsjTUlaE2YezSSXJ3kkyUKSPbOe56gkn0xyJMmDU2unJ7kjyaPj+2ljPUk+Om7DA0kunNHM25PcneThJA8ledcamfuUJF9Jcv+Y+/1j/Zwk94z5bkiycaxvGvsL4/jZs5h7zLIuyb1Jbl0LMyf5dpKvJ7kvyf6xNu/3j81JbkryL0kOJLlkpjNX1cy+gHXAN4GXAxuB+4HzZjnT1GyvBS4EHpxa+zNgz9jeA3xwbF8B/D0Q4GLgnhnNvAW4cGy/BPgGcN4amDvAi8f2BuCeMc+NwNVj/ePA747t3wM+PravBm6Y4f3kPcDfALeO/bmeGfg28LJj1ub9/rEP+J2xvRHYPMuZZ3JHm/rDuAS4fWr/WuDaWc50zHxnHxPNR4AtY3sLk98vBfgL4G3HO2/G898MvH4tzQ28EPhn4FVMfmF5/bH3FeB24JKxvX6clxnMug24E7gUuHX8RZ33mY8Xzbm9fwCnAv967J/VLGee9dPzrcDBqf1DY21enVVVj43tx4Gzxvbc3Y7x9O8CJo/a5n7u8TT3PuAIcAeTZyBPVtWzx5ntubnH8aeAM1Z3YgA+DLwX+NnYP4P5n7mAf0jytSS7x9o83z/OAf4d+MvxMsgnkryIGc4862iuWTX5Z2wuf/UgyYuBzwHvrqofTh+b17mr6qdVdT6TR28XAa+c8Ui/UJI3Akeq6muznqXpNVV1IfAG4Jokr50+OIf3j/VMXib7WFVdAPwnk6fjz1ntmWcdzcPA9qn9bWNtXn03yRaA8f3IWJ+b25FkA5NgfrqqPj+W537uo6rqSeBuJk9tNyc5+lHf6dmem3scPxX4/iqP+mrgTUm+DXyWyVP0jzDfM1NVh8f3I8DfMfkHap7vH4eAQ1V1z9i/iUlEZzbzrKP5VeDc8Y7jRiYvkN8y45l+kVuAnWN7J5PXDI+uv2O8c3cx8NTUU4dVkyTA9cCBqvrQ1KF5n/vMJJvH9guYvA57gEk8rxqnHTv30dtzFXDXeLSxaqrq2qraVlVnM7nf3lVVb2eOZ07yoiQvOboN/AbwIHN8/6iqx4GDSX51LF0GPDzTmVfzRd2f80LvFUze5f0m8Eeznmdqrs8AjwHPMPnXbheT16DuBB4FvgicPs4N8OfjNnwd2DGjmV/D5GnKA8B94+uKNTD3rwH3jrkfBP54rL8c+AqwAPwtsGmsnzL2F8bxl8/4vvI6/ufd87mdecx2//h66OjftzVw/zgf2D/uH18ATpvlzH4iSJIaZv30XJLWFKMpSQ1GU5IajKYkNRhNSWowmpLUYDQlqcFoSlLDfwOJ8YrafwOo3QAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 1\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbFJREFUeJzt3X+o3fV9x/Hna/llq63xFyFLwqI0rPjHpiFoxFKK4qZZafzDFqXMUDICmwOLgy5usFHYH3V/1FYYdqG6paNrdbabQRzORsvYH43G+tvMenVKkqlZrVq3Uqfte3+cT9xZFr33Y+6933Pt8wGX+/1+vt97zvuEk2fOj3tIqgpJ0sz80tADSNJCYjQlqYPRlKQORlOSOhhNSepgNCWpw5xEM8nFSZ5MMpVk+1xchyQNIbP9e5pJFgE/AC4CDgD3A1dU1ROzekWSNIC5eKR5DjBVVc9U1X8D3wQ2z8H1SNK8WzwHl7kK2D+2fwA4951+YGmW1XEcPwejSNLMvMbLP6yq06Y7by6iOSNJtgHbAI7j/ZybC4caRZL4Tt323EzOm4un5weBNWP7q9va/1FVO6pqQ1VtWMKyORhDkmbfXETzfmBdktOTLAUuB3bNwfVI0ryb9afnVfVmkt8H7gIWATdX1eOzfT2SNIQ5eU2zqu4E7pyLy5akIfmJIEnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjpMG80kNyc5lOSxsbWTk9yd5Kn2/aS2niQ3JJlK8kiS9XM5vCTNt5k80vxr4OIj1rYDu6tqHbC77QNcAqxrX9uAG2dnTEmaDNNGs6r+GfjREcubgZ1teydw6dj612rke8DyJCtna1hJGtq7fU1zRVU937ZfAFa07VXA/rHzDrQ1SXpPOOY3gqqqgOr9uSTbkuxNsvcNXj/WMSRpXrzbaL54+Gl3+36orR8E1oydt7qt/T9VtaOqNlTVhiUse5djSNL8erfR3AVsadtbgNvH1q9s76JvBF4dexovSQve4ulOSPIN4GPAqUkOAH8KfAG4NclW4DngU+30O4FNwBTwE+AzczCzJA1m2mhW1RVvc+jCo5xbwFXHOpQkTSo/ESRJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHaaNZpI1Se5N8kSSx5Nc3dZPTnJ3kqfa95PaepLckGQqySNJ1s/1jZCk+TKTR5pvAn9QVWcCG4GrkpwJbAd2V9U6YHfbB7gEWNe+tgE3zvrUkjSQaaNZVc9X1ffb9mvAPmAVsBnY2U7bCVzatjcDX6uR7wHLk6yc9cklaQBdr2kmWQucDewBVlTV8+3QC8CKtr0K2D/2Ywfa2pGXtS3J3iR73+D1zrElaRgzjmaSE4BvAZ+tqh+PH6uqAqrniqtqR1VtqKoNS1jW86OSNJgZRTPJEkbB/HpVfbstv3j4aXf7fqitHwTWjP346rYmSQveTN49D3ATsK+qvjh2aBewpW1vAW4fW7+yvYu+EXh17Gm8JC1oi2dwzvnAbwOPJnmorf0R8AXg1iRbgeeAT7VjdwKbgCngJ8BnZnViSRrQtNGsqn8B8jaHLzzK+QVcdYxzSdJE8hNBktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHRYPPYB0178/NCeX+5u/fNacXK5+sflIU5I6GE1J6mA0JamD0ZSkDtNGM8lxSe5L8nCSx5N8vq2fnmRPkqkktyRZ2taXtf2pdnzt3N4ESZo/M3mk+TpwQVX9OnAWcHGSjcB1wPVV9SHgZWBrO38r8HJbv76dJ0nvCdNGs0b+s+0uaV8FXADc1tZ3Ape27c1tn3b8wiSZtYklaUAzek0zyaIkDwGHgLuBp4FXqurNdsoBYFXbXgXsB2jHXwVOOcplbkuyN8neN3j92G6FJM2TGUWzqn5WVWcBq4FzgA8f6xVX1Y6q2lBVG5aw7FgvTpLmRde751X1CnAvcB6wPMnhTxStBg627YPAGoB2/ETgpVmZVpIGNpN3z09Lsrxtvw+4CNjHKJ6XtdO2ALe37V1tn3b8nqqq2RxakoYyk8+erwR2JlnEKLK3VtUdSZ4Avpnkz4AHgZva+TcBf5NkCvgRcPkczC1Jg5g2mlX1CHD2UdafYfT65pHrPwU+OSvTSdKE8RNBktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwf/CV4Pzv9rVQuIjTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqcOMo5lkUZIHk9zR9k9PsifJVJJbkixt68va/lQ7vnZuRpek+dfzSPNqYN/Y/nXA9VX1IeBlYGtb3wq83Navb+dJ0nvCjKKZZDXwW8BX236AC4Db2ik7gUvb9ua2Tzt+YTtfkha8mT7S/BLwOeDnbf8U4JWqerPtHwBWte1VwH6AdvzVdr4kLXjTRjPJx4FDVfXAbF5xkm1J9ibZ+wavz+ZFS9KcWTyDc84HPpFkE3Ac8EHgy8DyJIvbo8nVwMF2/kFgDXAgyWLgROClIy+0qnYAOwA+mJPrWG+IJM2HaR9pVtW1VbW6qtYClwP3VNWngXuBy9ppW4Db2/autk87fk9VGUVJ7wnH8nuafwhck2SK0WuWN7X1m4BT2vo1wPZjG1GSJsdMnp6/paq+C3y3bT8DnHOUc34KfHIWZpOkieMngiSpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjqkqoaegSSvAU8OPce7cCrww6GH6OTM82chzv2LPPOvVNVp0520eBauaDY8WVUbhh6iV5K9C21uZ54/C3FuZ56eT88lqYPRlKQOkxLNHUMP8C4txLmdef4sxLmdeRoT8UaQJC0Uk/JIU5IWhMGjmeTiJE8mmUqyfeh5Dktyc5JDSR4bWzs5yd1JnmrfT2rrSXJDuw2PJFk/0Mxrktyb5Ikkjye5eoHMfVyS+5I83Ob+fFs/PcmeNt8tSZa29WVtf6odXzvE3G2WRUkeTHLHQpg5ybNJHk3yUJK9bW3S7x/Lk9yW5F+T7Ety3qAzV9VgX8Ai4GngDGAp8DBw5pAzjc32UWA98NjY2p8D29v2duC6tr0J+EcgwEZgz0AzrwTWt+0PAD8AzlwAcwc4oW0vAfa0eW4FLm/rXwF+t23/HvCVtn05cMuA95NrgL8F7mj7Ez0z8Cxw6hFrk37/2An8TtteCiwfcuZB7mhjfxjnAXeN7V8LXDvkTEfMt/aIaD4JrGzbKxn9finAXwJXHO28gee/HbhoIc0NvB/4PnAuo19YXnzkfQW4CzivbS9u52WAWVcDu4ELgDvaX9RJn/lo0ZzY+wdwIvBvR/5ZDTnz0E/PVwH7x/YPtLVJtaKqnm/bLwAr2vbE3Y729O9sRo/aJn7u9jT3IeAQcDejZyCvVNWbR5ntrbnb8VeBU+Z3YgC+BHwO+HnbP4XJn7mAf0ryQJJtbW2S7x+nA/8B/FV7GeSrSY5nwJmHjuaCVaN/xibyVw+SnAB8C/hsVf14/Nikzl1VP6uqsxg9ejsH+PDAI72jJB8HDlXVA0PP0ukjVbUeuAS4KslHxw9O4P1jMaOXyW6sqrOB/2L0dPwt8z3z0NE8CKwZ21/d1ibVi0lWArTvh9r6xNyOJEsYBfPrVfXttjzxcx9WVa8A9zJ6ars8yeGP+o7P9tbc7fiJwEvzPOr5wCeSPAt8k9FT9C8z2TNTVQfb90PA3zP6B2qS7x8HgANVtaft38YoooPNPHQ07wfWtXcclzJ6gXzXwDO9k13Alra9hdFrhofXr2zv3G0EXh176jBvkgS4CdhXVV8cOzTpc5+WZHnbfh+j12H3MYrnZe20I+c+fHsuA+5pjzbmTVVdW1Wrq2oto/vtPVX1aSZ45iTHJ/nA4W3gN4DHmOD7R1W9AOxP8qtt6ULgiUFnns8Xdd/mhd5NjN7lfRr446HnGZvrG8DzwBuM/rXbyug1qN3AU8B3gJPbuQH+ot2GR4ENA838EUZPUx4BHmpfmxbA3L8GPNjmfgz4k7Z+BnAfMAX8HbCsrR/X9qfa8TMGvq98jP9993xiZ26zPdy+Hj/8920B3D/OAva2+8c/ACcNObOfCJKkDkM/PZekBcVoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdfgfkXWMSx7NJD8AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 2\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADbJJREFUeJzt3X+o3fV9x/Hna/nZamv8hWRJWJTKin9sGoJGLKUobpqVxj9sUcoMJSOwObA46OIGG4X9UfdHbYVhF6pbOrpWZ7saxOFstIz90Wis8WdmvTolydSsVq1bqdP2vT/OJ+4si977Mffe77nu+YDL/X4/3+89533l5JnzPedek6pCkjQzvzT0AJK0kBhNSepgNCWpg9GUpA5GU5I6GE1J6jAn0UxySZInk0wl2TYX9yFJQ8hs/5xmkkXAD4GLgQPAA8CVVfXErN6RJA1gLp5pngtMVdUzVfVfwDeBTXNwP5I07xbPwW2uAvaP7R8AznunL1iaZbWc4+ZgFEmamdd4+UdVdep0581FNGckyVZgK8By3s95uWioUSSJ79btz83kvLm4PD8IrBnbX93W/peq2l5V66tq/RKWzcEYkjT75iKaDwBnJjk9yVLgCmDnHNyPJM27Wb88r6o3k/w+cDewCLilqh6f7fuRpCHMyWuaVXUXcNdc3LYkDcnfCJKkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOiweeoBJcPe/7Z2z2/7NXz57zm5b0vzzmaYkdTCaktTBaEpSB6MpSR2MpiR1mDaaSW5JcijJY2NrJyW5J8lT7fOJbT1JbkwyleSRJOvmcnhJmm8zeab518AlR6xtA3ZV1ZnArrYPcClwZvvYCtw0O2NK0mSYNppV9U/Aj49Y3gTsaNs7gMvG1r9WI98HViRZOVvDStLQ3u1rmqdV1fNt+wXgtLa9Ctg/dt6BtiZJ7wnH/EZQVRVQvV+XZGuSPUn2vMHrxzqGJM2LdxvNFw9fdrfPh9r6QWDN2Hmr29r/UVXbq2p9Va1fwrJ3OYYkza93G82dwOa2vRm4Y2z9qvYu+gbg1bHLeEla8Kb9H3Yk+QbwMeCUJAeAPwW+ANyWZAvwHPCpdvpdwEZgCvgp8Jk5mFmSBjNtNKvqyrc5dNFRzi3g6mMdSpImlb8RJEkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIH/wlf/Gd2Jc2czzQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSeowbTSTrElyX5Inkjye5Jq2flKSe5I81T6f2NaT5MYkU0keSbJurr8JSZovM3mm+SbwB1V1FrABuDrJWcA2YFdVnQnsavsAlwJnto+twE2zPrUkDWTaaFbV81X1g7b9GrAPWAVsAna003YAl7XtTcDXauT7wIokK2d9ckkaQNdrmknWAucAu4HTqur5dugF4LS2vQrYP/ZlB9rakbe1NcmeJHve4PXOsSVpGDOOZpLjgW8Bn62qn4wfq6oCqueOq2p7Va2vqvVLWNbzpZI0mBlFM8kSRsH8elV9uy2/ePiyu30+1NYPAmvGvnx1W5OkBW8m754HuBnYV1VfHDu0E9jctjcDd4ytX9XeRd8AvDp2GS9JC9riGZxzAfDbwKNJ9ra1PwK+ANyWZAvwHPCpduwuYCMwBfwU+MysTixJA5o2mlX1z0De5vBFRzm/gKuPcS5Jmkj+RpAkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdpo1mkuVJ7k/ycJLHk3y+rZ+eZHeSqSS3Jlna1pe1/al2fO3cfguSNH9m8kzzdeDCqvp14GzgkiQbgOuBG6rqQ8DLwJZ2/hbg5bZ+QztPkt4Tpo1mjfxH213SPgq4ELi9re8ALmvbm9o+7fhFSTJrE0vSgGb0mmaSRUn2AoeAe4CngVeq6s12ygFgVdteBewHaMdfBU4+ym1uTbInyZ43eP3YvgtJmiczimZV/byqzgZWA+cCHz7WO66q7VW1vqrWL2HZsd6cJM2LrnfPq+oV4D7gfGBFksXt0GrgYNs+CKwBaMdPAF6alWklaWAzeff81CQr2vb7gIuBfYzieXk7bTNwR9ve2fZpx++tqprNoSVpKIunP4WVwI4kixhF9raqujPJE8A3k/wZ8BBwczv/ZuBvkkwBPwaumIO5JWkQ00azqh4BzjnK+jOMXt88cv1nwCdnZTpJmjD+RpAkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdZhzNJIuSPJTkzrZ/epLdSaaS3JpkaVtf1van2vG1czO6JM2/nmea1wD7xvavB26oqg8BLwNb2voW4OW2fkM7T5LeE2YUzSSrgd8Cvtr2A1wI3N5O2QFc1rY3tX3a8Yva+ZK04M30meaXgM8Bv2j7JwOvVNWbbf8AsKptrwL2A7Tjr7bzJWnBmzaaST4OHKqqB2fzjpNsTbInyZ43eH02b1qS5sziGZxzAfCJJBuB5cAHgS8DK5Isbs8mVwMH2/kHgTXAgSSLgROAl4680araDmwH+GBOqmP9RiRpPkz7TLOqrquq1VW1FrgCuLeqPg3cB1zeTtsM3NG2d7Z92vF7q8ooSnpPOJaf0/xD4NokU4xes7y5rd8MnNzWrwW2HduIkjQ5ZnJ5/paq+h7wvbb9DHDuUc75GfDJWZhNkiaOvxEkSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUIVU19AwkeQ14cug53oVTgB8NPUQnZ54/C3Hu/88z/0pVnTrdSYtn4Y5mw5NVtX7oIXol2bPQ5nbm+bMQ53bm6Xl5LkkdjKYkdZiUaG4feoB3aSHO7czzZyHO7czTmIg3giRpoZiUZ5qStCAMHs0klyR5MslUkm1Dz3NYkluSHEry2NjaSUnuSfJU+3xiW0+SG9v38EiSdQPNvCbJfUmeSPJ4kmsWyNzLk9yf5OE29+fb+ulJdrf5bk2ytK0va/tT7fjaIeZusyxK8lCSOxfCzEmeTfJokr1J9rS1SX98rEhye5J/SbIvyfmDzlxVg30Ai4CngTOApcDDwFlDzjQ220eBdcBjY2t/Dmxr29uA69v2RuAfgAAbgN0DzbwSWNe2PwD8EDhrAcwd4Pi2vQTY3ea5DbiirX8F+N22/XvAV9r2FcCtAz5OrgX+Friz7U/0zMCzwClHrE3642MH8DtteymwYsiZB3mgjf3HOB+4e2z/OuC6IWc6Yr61R0TzSWBl217J6OdLAf4SuPJo5w08/x3AxQtpbuD9wA+A8xj9wPLiIx8rwN3A+W17cTsvA8y6GtgFXAjc2f6gTvrMR4vmxD4+gBOAfz3yv9WQMw99eb4K2D+2f6CtTarTqur5tv0CcFrbnrjvo13+ncPoWdvEz90uc/cCh4B7GF2BvFJVbx5ltrfmbsdfBU6e34kB+BLwOeAXbf9kJn/mAv4xyYNJtra1SX58nA78O/BX7WWQryY5jgFnHjqaC1aN/hqbyB89SHI88C3gs1X1k/Fjkzp3Vf28qs5m9OztXODDA4/0jpJ8HDhUVQ8OPUunj1TVOuBS4OokHx0/OIGPj8WMXia7qarOAf6T0eX4W+Z75qGjeRBYM7a/uq1NqheTrARonw+19Yn5PpIsYRTMr1fVt9vyxM99WFW9AtzH6NJ2RZLDv+o7Pttbc7fjJwAvzfOoFwCfSPIs8E1Gl+hfZrJnpqoOts+HgL9n9BfUJD8+DgAHqmp327+dUUQHm3noaD4AnNnecVzK6AXynQPP9E52Apvb9mZGrxkeXr+qvXO3AXh17NJh3iQJcDOwr6q+OHZo0uc+NcmKtv0+Rq/D7mMUz8vbaUfOffj7uRy4tz3bmDdVdV1Vra6qtYwet/dW1aeZ4JmTHJfkA4e3gd8AHmOCHx9V9QKwP8mvtqWLgCcGnXk+X9R9mxd6NzJ6l/dp4I+Hnmdsrm8AzwNvMPrbbguj16B2AU8B3wVOaucG+Iv2PTwKrB9o5o8wukx5BNjbPjYugLl/DXiozf0Y8Cdt/QzgfmAK+DtgWVtf3van2vEzBn6sfIz/efd8Ymdusz3cPh4//OdtATw+zgb2tMfHd4ATh5zZ3wiSpA5DX55L0oJiNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjr8N0OTjEv/2sduAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"instance: 3\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADa9JREFUeJzt3V2MXOV9x/Hvr35NIMG8Cbm2VYNiNeKiBWTxIqIoAtGCG8VckAgUFStyZamlEhGVUtNKrSL1IvQiJEgVqRXTOlWaQEkaLERFiSGqehGDCebVJSwUZLuAGwKENgqF5N+LeUynrmH3wbt7Zun3I632nOecnf0PGn89Z2YXp6qQJM3MLw09gCQtJEZTkjoYTUnqYDQlqYPRlKQORlOSOsxJNJNcmuTJJFNJts7F95CkIWS2f04zySLgh8AlwAHgAeCqqnpiVr+RJA1gLp5pngtMVdUzVfVfwDeBjXPwfSRp3i2eg9tcBewf2z8AnPdOX7A0y2o5x83BKJI0M6/x8o+q6tTpzpuLaM5Iki3AFoDlvJ/zcvFQo0gS363bn5vJeXNxeX4QWDO2v7qt/S9Vta2q1lfV+iUsm4MxJGn2zUU0HwDWJTk9yVLgSmDnHHwfSZp3s355XlVvJvl94G5gEXBLVT0+299HkoYwJ69pVtVdwF1zcduSNCR/I0iSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQO00YzyS1JDiV5bGztpCT3JHmqfT6xrSfJTUmmkjyS5Jy5HF6S5ttMnmn+NXDpEWtbgV1VtQ7Y1fYBLgPWtY8twM2zM6YkTYZpo1lV/wT8+IjljcCOtr0DuHxs/Ws18n1gRZKVszWsJA3t3b6meVpVPd+2XwBOa9urgP1j5x1oa5L0nnDMbwRVVQHV+3VJtiTZk2TPG7x+rGNI0rx4t9F88fBld/t8qK0fBNaMnbe6rf0fVbWtqtZX1folLHuXY0jS/Hq30dwJbGrbm4A7xtavbu+inw+8OnYZL0kL3uLpTkjyDeBjwClJDgB/CnwBuC3JZuA54FPt9LuADcAU8FPgM3MwsyQNZtpoVtVVb3Po4qOcW8A1xzqUJE0qfyNIkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjpMG80ka5Lcl+SJJI8nubatn5TkniRPtc8ntvUkuSnJVJJHkpwz13dCkubLTJ5pvgn8QVWdCZwPXJPkTGArsKuq1gG72j7AZcC69rEFuHnWp5akgUwbzap6vqp+0LZfA/YBq4CNwI522g7g8ra9EfhajXwfWJFk5axPLkkD6HpNM8la4GxgN3BaVT3fDr0AnNa2VwH7x77sQFs78ra2JNmTZM8bvN45tiQNY8bRTHI88C3gs1X1k/FjVVVA9XzjqtpWVeurav0SlvV8qSQNZkbRTLKEUTC/XlXfbssvHr7sbp8PtfWDwJqxL1/d1iRpwZvJu+cBtgP7quqLY4d2Apva9ibgjrH1q9u76OcDr45dxkvSgrZ4BudcCPw28GiSvW3tj4AvALcl2Qw8B3yqHbsL2ABMAT8FPjOrE0vSgKaNZlX9M5C3OXzxUc4v4JpjnEuSJpK/ESRJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlKHaaOZZHmS+5M8nOTxJJ9v66cn2Z1kKsmtSZa29WVtf6odXzu3d0GS5s9Mnmm+DlxUVb8OnAVcmuR84Abgxqr6EPAysLmdvxl4ua3f2M6TpPeEaaNZI//Rdpe0jwIuAm5v6zuAy9v2xrZPO35xkszaxJI0oBm9pplkUZK9wCHgHuBp4JWqerOdcgBY1bZXAfsB2vFXgZOPcptbkuxJsucNXj+2eyFJ82RG0ayqn1fVWcBq4Fzgw8f6jatqW1Wtr6r1S1h2rDcnSfOi693zqnoFuA+4AFiRZHE7tBo42LYPAmsA2vETgJdmZVpJGthM3j0/NcmKtv0+4BJgH6N4XtFO2wTc0bZ3tn3a8XurqmZzaEkayuLpT2ElsCPJIkaRva2q7kzyBPDNJH8GPARsb+dvB/4myRTwY+DKOZhbkgYxbTSr6hHg7KOsP8Po9c0j138GfHJWppOkCeNvBElSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHUwmpLUwWhKUgejKUkdjKYkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHVYPPQAknT3v+2ds9v+zV8+a1Zvz2eaktTBaEpSB6MpSR2MpiR1mHE0kyxK8lCSO9v+6Ul2J5lKcmuSpW19WdufasfXzs3okjT/ep5pXgvsG9u/Abixqj4EvAxsbuubgZfb+o3tPEl6T5hRNJOsBn4L+GrbD3ARcHs7ZQdwedve2PZpxy9u50vSgjfTZ5pfAj4H/KLtnwy8UlVvtv0DwKq2vQrYD9COv9rOl6QFb9poJvk4cKiqHpzNb5xkS5I9Sfa8weuzedOSNGdm8htBFwKfSLIBWA58EPgysCLJ4vZscjVwsJ1/EFgDHEiyGDgBeOnIG62qbcA2gA/mpDrWOyJJ82HaZ5pVdX1Vra6qtcCVwL1V9WngPuCKdtom4I62vbPt047fW1VGUdJ7wrH8nOYfAtclmWL0muX2tr4dOLmtXwdsPbYRJWlydP0PO6rqe8D32vYzwLlHOednwCdnYTZJmjj+RpAkdTCaktTBaEpSB6MpSR2MpiR1MJqS1MFoSlIHoylJHYymJHXwn/CVNLjZ/md255LPNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqYPRlKQORlOSOhhNSepgNCWpg9GUpA5GU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqUOqaugZSPIa8OTQc7wLpwA/GnqITs48fxbi3P+fZ/6Vqjp1upMm5Z/wfbKq1g89RK8kexba3M48fxbi3M48PS/PJamD0ZSkDpMSzW1DD/AuLcS5nXn+LMS5nXkaE/FGkCQtFJPyTFOSFoTBo5nk0iRPJplKsnXoeQ5LckuSQ0keG1s7Kck9SZ5qn09s60lyU7sPjyQ5Z6CZ1yS5L8kTSR5Pcu0CmXt5kvuTPNzm/nxbPz3J7jbfrUmWtvVlbX+qHV87xNxtlkVJHkpy50KYOcmzSR5NsjfJnrY26Y+PFUluT/IvSfYluWDQmatqsA9gEfA0cAawFHgYOHPImcZm+yhwDvDY2NqfA1vb9lbghra9AfgHIMD5wO6BZl4JnNO2PwD8EDhzAcwd4Pi2vQTY3ea5DbiyrX8F+N22/XvAV9r2lcCtAz5OrgP+Friz7U/0zMCzwClHrE3642MH8DtteymwYsiZB3mgjf3HuAC4e2z/euD6IWc6Yr61R0TzSWBl217J6OdLAf4SuOpo5w08/x3AJQtpbuD9wA+A8xj9wPLiIx8rwN3ABW17cTsvA8y6GtgFXATc2f6gTvrMR4vmxD4+gBOAfz3yv9WQMw99eb4K2D+2f6CtTarTqur5tv0CcFrbnrj70S7/zmb0rG3i526XuXuBQ8A9jK5AXqmqN48y21tzt+OvAifP78QAfAn4HPCLtn8ykz9zAf+Y5MEkW9raJD8+Tgf+Hfir9jLIV5Mcx4AzDx3NBatGf41N5I8eJDke+Bbw2ar6yfixSZ27qn5eVWcxevZ2LvDhgUd6R0k+DhyqqgeHnqXTR6rqHOAy4JokHx0/OIGPj8WMXia7uarOBv6T0eX4W+Z75qGjeRBYM7a/uq1NqheTrARonw+19Ym5H0mWMArm16vq22154uc+rKpeAe5jdGm7IsnhX/Udn+2tudvxE4CX5nnUC4FPJHkW+CajS/QvM9kzU1UH2+dDwN8z+gtqkh8fB4ADVbW77d/OKKKDzTx0NB8A1rV3HJcyeoF858AzvZOdwKa2vYnRa4aH169u79ydD7w6dukwb5IE2A7sq6ovjh2a9LlPTbKibb+P0euw+xjF84p22pFzH74/VwD3tmcb86aqrq+q1VW1ltHj9t6q+jQTPHOS45J84PA28BvAY0zw46OqXgD2J/nVtnQx8MSgM8/ni7pv80LvBkbv8j4N/PHQ84zN9Q3geeANRn/bbWb0GtQu4Cngu8BJ7dwAf9Huw6PA+oFm/gijy5RHgL3tY8MCmPvXgIfa3I8Bf9LWzwDuB6aAvwOWtfXlbX+qHT9j4MfKx/ifd88nduY228Pt4/HDf94WwOPjLGBPe3x8BzhxyJn9jSBJ6jD05bkkLShGU5I6GE1J6mA0JamD0ZSkDkZTkjoYTUnqYDQlqcN/A9knjEu3katkAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"for inst_idx, mask in enumerate(target.get_field('masks')):\n",
" print(\"instance:\", inst_idx)\n",
" plt.imshow(mask.get_mask_tensor())\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now compute the results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.564\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_bbox.params.iouThrs = [1.]\n",
"coco_eval_bbox.params.recThrs = [.75]\n",
"\n",
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.75]\n",
"\n",
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision:\n",
"[[[[[ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]]]]]\n",
"\n",
"\n",
"Recall:\n",
"[[[[ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]]]]\n"
]
}
],
"source": [
"print('Precision:')\n",
"print(coco_eval_segm.eval['precision'])\n",
"print('\\n\\nRecall:')\n",
"print(coco_eval_segm.eval['recall'])"
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bbox\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
"\n",
"\n",
"segm\n",
"Running per image evaluation...\n",
"Evaluate annotation type *segm*\n",
"DONE (t=0.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.00s).\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 1 ] = 0.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets= 10 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= all | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= small | maxDets=100 ] = -1.000\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area=medium | maxDets=100 ] = 0.750\n",
" Average Recall (AR) @[ IoU=1.00:1.00 | area= large | maxDets=100 ] = -1.000\n"
]
}
],
"source": [
"coco_eval_bbox.params.iouThrs = [1.]\n",
"coco_eval_bbox.params.recThrs = [.76]\n",
"\n",
"coco_eval_segm.params.iouThrs = [1.]\n",
"coco_eval_segm.params.recThrs = [.76]\n",
"\n",
"print('bbox')\n",
"coco_eval_bbox.evaluate()\n",
"coco_eval_bbox.accumulate()\n",
"coco_eval_bbox.summarize()\n",
"print('\\n\\nsegm')\n",
"coco_eval_segm.evaluate()\n",
"coco_eval_segm.accumulate()\n",
"coco_eval_segm.summarize()"
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision:\n",
"[[[[[ 0. 0. 0.]\n",
" [-1. -1. -1.]\n",
" [ 0. 0. 0.]\n",
" [-1. -1. -1.]]]]]\n",
"\n",
"\n",
"Recall:\n",
"[[[[ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]\n",
" [ 0. 0.75 0.75]\n",
" [-1. -1. -1. ]]]]\n"
]
}
],
"source": [
"print('Precision:')\n",
"print(coco_eval_segm.eval['precision'])\n",
"print('\\n\\nRecall:')\n",
"print(coco_eval_segm.eval['recall'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"please contact: [email protected]"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment