agitter · August 16, 2017 18:28 · agitter · Aug 16, 2017
diff --git a/sklearn-auprc-v18.ipynb b/sklearn-auprc-v18.ipynb
 {
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Precision recall curves in scikit-learn with ties\n",
    "\n",
    "August 16, 2017\n",
    "\n",
    "Anthony Gitter\n",
    "\n",
    "An exaggerated example of what my group saw on a recent dataset, which is the same problem Anshul describes here https://twitter.com/anshul/status/761117086915497984\n",
    "\n",
    "Testing with scikit-learn version 18.1 to establish a baseline before the average precision function was updated.  The naive auPRC computes the area with `metrics.auc` on the precision and recall points so it does not have the opportunity to do advanced interpolation.\n",
    "\n",
    "In all three cases tested here, the `average_precision_score` auPRC matches this naive calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "scikit-learn version: 0.18.1\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn import metrics\n",
    "\n",
    "print 'scikit-learn version: {}'.format(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper function to compute auPRC and plot the precision recall curve\n",
    "scikit-learn uses linear interpolation when ties are present"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pr(y_true, y_scores):\n",
    "    precision, recall, thresholds = metrics.precision_recall_curve(y_true, y_scores)\n",
    "    auc = metrics.auc(recall, precision)\n",
    "    \n",
    "    print 'precision: %s' % precision\n",
    "    print 'recall: %s' % recall\n",
    "    print 'thresholds: %s' % thresholds\n",
    "    print 'Naive auPRC: %s' % auc\n",
    "    print 'average_precision_score auPRC: %s' % metrics.average_precision_score(y_true, y_scores)\n",
    "    \n",
    "    plot(recall, precision);\n",
    "    xlabel('recall');\n",
    "    ylabel('precision');\n",
    "    xlim(0,1);\n",
    "    ylim(0,1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier A predicts identical scores for all instances\n",
    "The naive auPRC treats the precision recall curve as a linear interpoloation from 1 to the baseline pos/(pos+neg) = 1/6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.16666667  1.        ]\n",
      "recall: [ 1.  0.]\n",
      "thresholds: [ 0.5]\n",
      "Naive auPRC: 0.583333333333\n",
      "average_precision_score auPRC: 0.583333333333\n"
     ]
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAFThJREFUeJzt3X+s5HV97/Hni12h4g1Vwq29d9kWi2jBFqsgF7Wtx2ra\n",
       "lVwksXoJVRtqTTe93V5/tJFtm9ZNrmhI2qa1WGKpcqV/dCnVprQlEm5xK1cEivLLLou7WJpdaIiK\n",
       "bbxKvCDv+8fMYWfOnh/fM+d8Z74z83wkJGdmvmfOm2/Ozut8v6/5fCdVhSRJi46b9ACSpG4xGCRJ\n",
       "QwwGSdIQg0GSNMRgkCQNMRgkSUNaDYYkH0/yWJL7V9nmw0kOJrk3ycvanEeStLa2jxiuAXas9GCS\n",
       "C4AXVtUZwC8BV7U8jyRpDa0GQ1XdCnxjlU3eCHyiv+0dwHOTPL/NmSRJq5t0x7ANODxw+whw6oRm\n",
       "kSQx+WAAyJLbXqNDkiZo64R//iPA9oHbp/bvG5LEsJCkEVTV0j++1zTpYLgB2AXsTXI+8G9V9djy\n",
       "m9b3A3uANwMfAj5SxXfGM2Z3JNlTVXsmPUcXuC+Ocl8c5b44atQ/qtt+u+qfA7cBL05yOMk7kuxM\n",
       "shOgqm4EvpLkEPBR4L+v9FxVPFbFLwOvAV4LPJDw35JjTkVJkjag1SOGqrqkwTa71vec7AcuTPgp\n",
       "4HeB9yT8WhW3jTimJGlAF8rnkVRxC3Au8MfA3oTrE06f8FjjsG/SA3TIvkkP0CH7Jj1Ah+yb9ADT\n",
       "LtPwQT1JarUCJeFE4N3Ae4FrgQ9U8fi45pOkLlrrtXMlU3vEMKiKb1fxQeAlwLOBBxPem3DChEeT\n",
       "pKkzE8GwyIJakjZuJk4lrfx9zxTU3wELaknzZa5PJa1kjgtqSRrZTAcDQBVPV/FnwA8DdwN3JPx+\n",
       "wskTHk2SOmnmg2GRBbUkNTM3wbDIglqSVjfT5XOz57agljSbLJ9HZEEtScPmPhjAglqSBhkMAyyo\n",
       "JclgWJYFtaR5Nvflc7Ofb0EtafpYPrfIglrSPDEYGrKgljQvDIZ1sqCWNOsMhhFZUEuaVZbPm8SC\n",
       "WlLXWD5PmAW1pFlhMGwiC2pJs8BgaIEFtaRpZjC0yIJa0jSyfB4jC2pJ42T5PAUsqCVNA4NhzCyo\n",
       "JXWdwTAhFtSSuspgmDALakldY/ncMRbUkjaL5fOMsKCWNGkGQwdZUEuaJIOhwyyoJU2CwTAFLKgl\n",
       "jZPl8xSyoJbUhOXzHLGgltSmVoMhyY4kB5IcTHLZMo+fkuTTSe5J8qUkl7Y5zyyxoJbUltaCIckW\n",
       "4EpgB3AWcEmSM5dstgu4u6p+DFgAfi/J1rZmmkUW1JI2W5tHDOcBh6rq4ap6EtgLXLRkm38FTup/\n",
       "fRLw9ap6qsWZZpYFtaTN0mYwbAMOD9w+0r9v0NXAS5I8CtwLvKvFeeZCFfuruBB4J7AbuC3hVRMe\n",
       "S9IUafO0TZO3O/0mcE9VLSQ5Hbg5yUur6ptLN0yyZ+DmvqratzljzqYqbkk4F3grvYL6DmB3FQ9N\n",
       "eDRJLUmyQO+0/Ia0GQyPANsHbm+nd9Qw6FXA5QBV9VCSfwZeDNy19Mmqak87Y86uKp4G/izhk8C7\n",
       "6RXU1wIfqOLxyU4nabP1/2Det3g7yftHeZ42TyXdBZyR5LQkxwMXAzcs2eYA8HqAJM+nFwpfaXGm\n",
       "uWRBLWk9WguGfom8C7gJ2A9cV1UPJNmZZGd/sw8C5ya5F/jfwPuqyr9kW2JBLakJVz7PMVdQS7PN\n",
       "lc9aN1dQS1qOwTDnXEEtaSmDQYAFtaSjDAYNsaCWZPmsVVlQS9PL8lmtsKCW5o/BoDVZUEvzxWBQ\n",
       "YxbU0nwwGLRuFtTSbLN81oZZUEvdZPmsibGglmaLwaBNYUEtzQ6DQZvKglqafgaDWmFBLU0vy2eN\n",
       "hQW1NH6Wz+q0gYL6KuA6C2qpuwwGjU2/oL6W3ke4WlBLHWUwaOwsqKVuMxg0MRbUUjdZPqszLKil\n",
       "zWX5rKlnQS11g8GgTrGglibPYFAnWVBLk2MwqNMsqKXxs3zWVLGglpqzfNZcsKCW2mcwaOpYUEvt\n",
       "Mhg0tSyopXYYDJp6FtTS5rJ81syxoJZ6LJ+lPgtqaWMMBs0kC2ppdAaDZpoFtbR+BoPmggW11Jzl\n",
       "s+aSBbXmQSfL5yQ7khxIcjDJZStss5Dk7iRfSrKvzXmkRRbU0spaO2JIsgV4EHg98Ajwj8AlVfXA\n",
       "wDbPBT4H/ExVHUlySlV9bZnn8ohBrUk4EXg38F7gWuADVTw+2amkjeviEcN5wKGqeriqngT2Ahct\n",
       "2ebngE9W1RGA5UJBapsFtTSszWDYBhweuH2kf9+gM4CTk3wmyV1J3t7iPNKqLKilnjaDock5qmcB\n",
       "LwcuAH4G+O0kZ7Q4k7SmKvZXcSHwTmA3cFvCqyY8ljQ2W1t87keA7QO3t9M7ahh0GPhaVT0BPJHk\n",
       "s8BLgYNLnyzJnoGb+6pq36ZOKy1RxS0J5wJvo1dQ3w7sruKhCY8mLSvJArCw4edpUj4n2QacBmwB\n",
       "AlRVfXaN79lKr3x+HfAocCfHls8/DFxJ72jhBOAO4OKq2r/kuSyfNVEW1JpGo752rnnEkOQK4GJg\n",
       "P/DdgYdWDYaqeirJLuAmeoHysap6IMnO/uMfraoDST4N3Ac8DVy9NBSkLqji28AHEz4G7KFXUH8I\n",
       "+EgV35nocNImW/OIIcmXgR+tqon98nvEoK5JOAu4gt47mXYD11c16tWksWnz7aoPAcevfyRpdllQ\n",
       "a5Y1KZ+fAO5J8vfwzCFzVdX/aG8saTpYUGsWNTmVdGn/y8UNF8vnT7Q419IZPJWkzrOgVteM+trZ\n",
       "9F1JJwAv6t880F/JPDYGg6ZJwvPpFdRvBgtqTU5rHUP/fbFfBj7S/+9gktese0JpTriCWtOuyamk\n",
       "L9Jbf/Bg//aLgL1V9fIxzLc4g0cMmlpe4luT0ua7krYuhgJAVX2ZdldMSzPFS3xr2jQJhi8k+dP+\n",
       "5ya8NsmfAne1PZg0S/wMak2TJqeSvgf4FeDV/btuBf54nAvePJWkWWNBrXFo9V1Jk2YwaFa5glpt\n",
       "2vRgSHJ9Vb0lyf3LPFxVdfZ6f9ioDAbNOgtqtaGNYPjPVfVoktOWe7yqHl7vDxuVwaB5kHAcvRXU\n",
       "l4MrqLVxm/6upKp6tP/lV4HD/SA4ATib3mctSNpEFtTqiibvSroVOKH/mQw3AW8H/lebQ0nzzM+g\n",
       "1qQ1CYZU1beBN9F7N9JbgB9pdyxJrqDWpDT6zOckrwTeCvzder5P0sZ5iW+NW5MX+HcDvwH8VVX9\n",
       "U5LTgc+0O5akpVxBrXFxHYM0hbzEt5po4+2qf1hV70ryN8s8XFX1xvX+sFEZDNLyXEGt1bQRDOdU\n",
       "1Rf6l91eqqrqH9b7w0ZlMEircwW1ltPaJTGS/Afgiar6bv/2FuB7qupbI006AoNBasYV1BrU5mW3\n",
       "/57ee6kXnQjcvN4fJKl9FtTaDE2C4YSq+r+LN6rqm/TCQVIHuYJaG9UkGL6V5JzFG0nOBZ5obyRJ\n",
       "m8EV1BpVk47hFcB1wOK1k/4TcHFVje3DeuwYpI2zoJ4/rX4eQ5Jn0TssDXCgqp5c/4ijMxikzWNB\n",
       "PT9aK5+TPIfeXxfvqqr7gdOS/NcRZpTUARbUWkuTjuEa4P/BM9dmeZTe9eIlTSkLaq2mSTCcXlVX\n",
       "0AsHxrl+QVK7LKi1nCbB8J0kz6xj6F9EzyX30gzxEt8a1ORdST8N/BZwFr2Fba8GLq2qsV1h1fJZ\n",
       "Gi8L6tnQyruSkhwHvIXe6ufz+3ffUVVfHWnKERkM0vj5GdTTr81rJX2hqs5ZdaOWGQzS5HiJ7+nV\n",
       "5rWSbk7y60m2Jzl58b8RZpQ0hSyo50+TI4aH4djVkVX1gpZmWm4GjxikjnAF9fRo81TSs4FfAX4c\n",
       "eBr4P8BVVTW26yUZDFL3WFB3X5unkq4FzgT+ELiS3ruTrm041I4kB5IcTHLZKtu9IslTSd7U5Hkl\n",
       "TZ4rqGdXkyOG/VV11lr3LfN9W4AHgdcDjwD/CFxSVQ8ss93NwLeBa6rqk8s8l0cMUodZUHdTm0cM\n",
       "X0zyyoEfdD7whQbfdx5wqKoe7l90by9w0TLb/Srwl8BY3wIrafNYUM+WJsFwLvC5JP/SL6JvA85N\n",
       "cn+S+1b5vm3A4YHbR/r3PSPJNnphcVX/LgssaYq5gno2bG2wzY4Rn7vJi/wfALurqpIE/OWRZkEV\n",
       "+4ELBwrq9yQW1NNizWCoqodHfO5HgO0Dt7fTO2oYdA6wt5cJnAK8IcmTVXXD0idLsmfg5r6q2jfi\n",
       "XJLGpIpbEs6lt4L6usQV1G1KsgAsbPh5mnxQz0hPnGylVz6/jt6luu9kmfJ5YPtrgL+pqk8t85jl\n",
       "szTlLKjHr83yeSRV9RSwC7gJ2A9cV1UPJNmZZGdbP1dSN1lQT4/Wjhg2k0cM0uxxBXX7Wv3M50kz\n",
       "GKTZ5Qrq9nTuVJIkNeEK6u4xGCRNnJ9B3S0Gg6TOsKDuBoNBUue4gnqyLJ8ldZ4F9WgsnyXNLAvq\n",
       "8TIYJE0FC+rxMRgkTRUL6vYZDJKmkgV1eyyfJc0EC+pjWT5LmmsW1JvHYJA0MyyoN4fBIGnmWFBv\n",
       "jMEgaWZZUI/G8lnS3Ji3gtryWZLWYEHdjMEgaa5YUK/NYJA0lyyoV2YwSJprFtTHsnyWpAGzVFBb\n",
       "PkvSJrCgNhgk6RjzXlAbDJK0gnktqA0GSVrDvBXUls+StE7TUlBbPkvSmMx6QW0wSNIIZrmgNhgk\n",
       "aQNmsaA2GCRpE8xSQW35LEkt6EJBbfksSR0yzQW1wSBJLZnWgtpgkKSWTVtBbTBI0phMS0Ft+SxJ\n",
       "E9J2Qd3Z8jnJjiQHkhxMctkyj781yb1J7kvyuSRntz2TJHVBVwvqVoMhyRbgSmAHcBZwSZIzl2z2\n",
       "FeAnq+ps4H8Cf9LmTJLUJV0sqNs+YjgPOFRVD1fVk8Be4KLBDarq81X17/2bdwCntjyTJHVOlwrq\n",
       "toNhG3B44PaR/n0r+UXgxlYnkqQO60JBvbXl52/cbCd5LfAO4NUrPL5n4Oa+qtq3ockkqcOq2A9c\n",
       "OFBQvydZvaBOsgAsbPRnt/qupCTnA3uqakf/9m8AT1fVFUu2Oxv4FLCjqg4t8zy+K0nS3Eo4Dngb\n",
       "cDlwO7C7iofW/r5uvivpLuCMJKclOR64GLhhcIMkP0AvFN62XChI0rwbd0HdajBU1VPALuAmYD9w\n",
       "XVU9kGRnkp39zX4HeB5wVZK7k9zZ5kySNK3GVVC7wE2SplTCWcAV9IJiN3B91dFud9TXToNBkqbc\n",
       "Siuou9oxSJJattkrqA0GSZoByxXUoz6Xp5IkaQYlfB/kMTsGSdIz7BgkSZvCYJAkDTEYJElDDAZJ\n",
       "0hCDQZI0xGCQJA0xGCRJQwwGSdIQg0GSNMRgkCQNMRgkSUMMBknSEINBkjTEYJAkDTEYJElDDAZJ\n",
       "0hCDQZI0xGCQJA0xGCRJQwwGSdIQg0GSNMRgkCQNMRgkSUMMBknSEINBkjTEYJAkDTEYJElDDAZJ\n",
       "0hCDQZI0xGCQJA1pNRiS7EhyIMnBJJetsM2H+4/fm+Rlbc4jSVpba8GQZAtwJbADOAu4JMmZS7a5\n",
       "AHhhVZ0B/BJwVVvzzIokC5OeoSvcF0e5L45yX2xcm0cM5wGHqurhqnoS2AtctGSbNwKfAKiqO4Dn\n",
       "Jnl+izPNgoVJD9AhC5MeoEMWJj1AhyxMeoBp12YwbAMOD9w+0r9vrW1ObXEmSdIa2gyGarhdRvw+\n",
       "SVILtrb43I8A2wdub6d3RLDaNqf27ztGEgOjL8n7Jz1DV7gvjnJfHOW+2Jg2g+Eu4IwkpwGPAhcD\n",
       "lyzZ5gZgF7A3yfnAv1XVY0ufqKqWHlVIklrSWjBU1VNJdgE3AVuAj1XVA0l29h//aFXdmOSCJIeA\n",
       "bwG/0NY8kqRmUuUZGknSUZ1a+eyCuKPW2hdJ3trfB/cl+VySsycx5zg0+b3ob/eKJE8ledM45xuX\n",
       "hv8+FpLcneRLSfaNecSxafDv45Qkn05yT39fXDqBMcciyceTPJbk/lW2Wd/rZlV14j96p5sOAacB\n",
       "zwLuAc5css0FwI39r/8LcPuk557gvngl8L39r3fM874Y2O4W4G+Bn5303BP6nXgu8E/Aqf3bp0x6\n",
       "7gnuiz3Ahxb3A/B1YOukZ29pf/wE8DLg/hUeX/frZpeOGFwQd9Sa+6KqPl9V/96/eQezu/6jye8F\n",
       "wK8Cfwl8dZzDjVGT/fBzwCer6ghAVX1tzDOOS5N98a/ASf2vTwK+XlVPjXHGsamqW4FvrLLJul83\n",
       "uxQMLog7qsm+GPSLwI2tTjQ5a+6LJNvovTAsXlJlFouzJr8TZwAnJ/lMkruSvH1s041Xk31xNfCS\n",
       "JI8C9wLvGtNsXbTu18023666Xi6IO6rx/1OS1wLvAF7d3jgT1WRf/AGwu6oqSTj2d2QWNNkPzwJe\n",
       "DrwOOBH4fJLbq+pgq5ONX5N98ZvAPVW1kOR04OYkL62qb7Y8W1et63WzS8GwqQviplyTfUG/cL4a\n",
       "2FFVqx1KTrMm++IcemthoHc++Q1JnqyqG8Yz4lg02Q+Hga9V1RPAE0k+C7wUmLVgaLIvXgVcDlBV\n",
       "DyX5Z+DF9NZXzZt1v2526VTSMwvikhxPb0Hc0n/YNwA/D7DagrgZsOa+SPIDwKeAt1XVoQnMOC5r\n",
       "7ouq+qGqekFVvYBez/DLMxYK0Ozfx18DP55kS5IT6RWN+8c85zg02RcHgNcD9M+nvxj4ylin7I51\n",
       "v2525oihXBD3jCb7Avgd4HnAVf2/lJ+sqvMmNXNbGu6Lmdfw38eBJJ8G7gOeBq6uqpkLhoa/Ex8E\n",
       "rklyL70/gN9XVY9PbOgWJflz4DXAKUkOA++nd1px5NdNF7hJkoZ06VSSJKkDDAZJ0hCDQZI0xGCQ\n",
       "JA0xGCRJQwwGSdIQg0FqWZJLk/xR/+s9SX5t0jNJqzEYpBWkbxOeqlb4Wuokg0Ea0L/MwoNJPgHc\n",
       "D/x2kjv7H3CyZ2C7n+/fd09/W5JcmOT2JF9McnOS75vQ/4a0IZ25JIbUIS8E3g58L/DmqjovyXHA\n",
       "Xyf5CeBx4LeAV1bV40me1/++W6vqfIAk7wTeB/w6s3m1V80wg0E61r9U1Z1Jfhf46SR39+9/Dr3Q\n",
       "eA7wF4vX3hm4su32JH8BfD9wPPN70TZNOU8lScf61sDXH6qql/X/e1FVXdO/f7mjgD8CPlxVZwM7\n",
       "gWe3PajUBoNBWtlNwDuSPAd6nxSX5D/S+2zptyQ5uX//4qmkk4BH+19fusJzelpJneepJOlYBVBV\n",
       "Nyc5k94noQF8k97nX+xPcjnwD0m+C3yR3qfo7QGuT/INeuHxgwPPV8t8LXWSl92WJA3xVJIkaYjB\n",
       "IEkaYjBIkoYYDJKkIQaDJGmIwSBJGmIwSJKGGAySpCH/H4V3cjcI6OAQAAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7e7de10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.5,.5,.5,.5,.5,.5])\n",
    "pr(y_true, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier B predicts identical scores for all instances except the first\n",
    "Classifier B is nearly identical to classifier A but the scikit-learn auPRC is much worse.  One of the predicted scores is slightly larger, breaking the tie.  The point (0,0) is introduced into the precision recall curve so the linear interpolation is from 0 to 1/6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.16666667  0.          1.        ]\n",
      "recall: [ 1.  0.  0.]\n",
      "thresholds: [ 0.5   0.51]\n",
      "Naive auPRC: 0.0833333333333\n",
      "average_precision_score auPRC: 0.0833333333333\n"
     ]
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAEqVJREFUeJzt3XuMpXV9x/H3x11AoUU0W9EuS7GKVG0hiHLxEqeV1JW0\n",
       "kmgtQcWiNiWNazVpI2pT2X/UmrSJIikhiFTT1K23xrUlEKKOUqJc5Krs4q66Zi+GiKChSOyufvvH\n",
       "OcOeZ5jLc2bnOefM7PuVTPY85/zmOd/9Zeb7med6UlVIkjTjSeMuQJI0WQwGSVKDwSBJajAYJEkN\n",
       "BoMkqcFgkCQ1dBoMST6Z5IEk9y4w5vIkO5LcneT0LuuRJC2u6y2Ga4GN872Y5DzguVV1MvBXwJUd\n",
       "1yNJWkSnwVBVNwEPLzDktcCn+mNvAY5LcnyXNUmSFjbuYwzrgd0Dy3uAE8ZUiySJ8QcDQGYte48O\n",
       "SRqjtWN+/73AhoHlE/rPNSQxLCRpCapq9h/fixp3MGwFNgFbkpwN/KyqHphr4FL+c6tRks1VtXnc\n",
       "dUwC5+Ig5+Ig5+Kgpf5R3WkwJPkM8EpgXZLdwGXAEQBVdVVVXZfkvCQ7gUeBt3ZZjyRpcZ0GQ1Vd\n",
       "2GLMpi5rkCQNZxIOPms40+MuYIJMj7uACTI97gImyPS4C1jpshI+qCdJeYxBkoaz1N7pFoMkqcFg\n",
       "kCQ1GAySpAaDQZLUYDBIkhoMBklSg8EgSWowGCRJDQaDJKnBYJAkNRgMkqQGg0GS1GAwSJIaDAZJ\n",
       "UoPBIElqMBgkSQ0GgySpwWCQJDUYDJKkBoNBktRgMEiSGgwGSVKDwSBJajAYJEkNBoMkqcFgkCQ1\n",
       "GAySpAaDQZLUYDBIkhoMBklSg8EgSWowGCRJDQaDJKnBYJAkNXQaDEk2JtmeZEeSS+d4fV2S65Pc\n",
       "leQ7SS7ush5J0uJSVd2sOFkD3A+cC+wFbgMurKptA2M2A0dV1fuSrOuPP76qDsxaV1VVOilUklap\n",
       "pfbOLrcYzgR2VtWuqtoPbAHOnzXmx8Cx/cfHAj+dHQqSpNFa2+G61wO7B5b3AGfNGnM18NUk+4Df\n",
       "BP68w3okSS10GQxt9lG9H7irqqaSPAe4MclpVfXI7IH93U4zpqtqennKlKTVIckUMHWo6+kyGPYC\n",
       "GwaWN9Dbahj0UuCDAFX1/SQ/BE4Bbp+9sqra3E2ZkrQ69P9gnp5ZTnLZUtbT5TGG24GTk5yU5Ejg\n",
       "AmDrrDHb6R2cJsnx9ELhBx3WJElaRGdbDFV1IMkm4AZgDXBNVW1Lckn/9auADwHXJrmbXki9p6oe\n",
       "6qomSdLiOjtddTl5uqokDW8ST1eVJK1ABoMkqcFgkCQ1GAySpAaDQZLUYDBIkhoMBklSg8EgSWow\n",
       "GCRJDQaDJKnBYJAkNRgMkqQGg0GS1GAwSJIaDAZJUoPBIElqMBgkSQ0GgySpwWCQJDUYDJKkBoNB\n",
       "ktRgMEiSGgwGSVKDwSBJajAYJEkNBoMkqcFgkCQ1GAySpAaDQZLUYDBIkhoMBklSg8EgSWowGCRJ\n",
       "DQaDJKnBYJAkNXQaDEk2JtmeZEeSS+cZM5XkziTfSTLdZT2SpMWlqrpZcbIGuB84F9gL3AZcWFXb\n",
       "BsYcB9wMvLqq9iRZV1UPzrGuqqp0UqgkrVJL7Z1dbjGcCeysql1VtR/YApw/a8wbgS9U1R6AuUJB\n",
       "kjRaXQbDemD3wPKe/nODTgaenuRrSW5PclGH9UiSWljb4brb7KM6AngR8CrgaOCbSb5VVTs6rEuS\n",
       "tIAug2EvsGFgeQO9rYZBu4EHq+ox4LEk3wBOA54QDEk2DyxOV9X0slYrSStckilg6pDX0+bgc5L1\n",
       "wEnAGiBAVdU3FvmetfQOPr8K2AfcyhMPPv8ecAXwauAo4Bbggqq6b9a6PPgsSUNaau9cdIshyUeA\n",
       "C4D7gF8NvLRgMFTVgSSbgBvoBco1VbUtySX916+qqu1JrgfuAX4NXD07FCRJo7XoFkOS7wF/UFW/\n",
       "HE1Jc9bgFoMkDanL01W/Dxw5fEmSpJWozcHnx4C7knwFmNlqqKr6m+7KkiSNS5tg2Nr/mtnnFNqd\n",
       "iipJWoHanpV0FPC8/uL2/pXMI+MxBkkaXpdnJU0BnwJ+1H/qxCR/UVVfH/bNJEmTr81ZSXfQu/7g\n",
       "/v7y84AtVfWiEdQ3U4NbDJI0pC7PSlo7EwoAVfU9ur1iWpI0Rm0a/LeTfAL4N3oHnt8E3N5pVZKk\n",
       "sWmzK+nJwDuAl/Wfugn4l1Fe8OauJEka3lJ7Z2cf1LOcDAZJGt6yn5WU5HNV9YYk987xclXVqcO+\n",
       "mSRp8s27xZDkt6tqX5KT5nq9qnZ1V9YTanGLQZKGtOxnJVXVvv7DnwC7+0FwFHAqvc9akCStQm1O\n",
       "V70JOKr/mQw3ABcB/9plUZKk8WkTDKmqXwCvo3c20huA3++2LEnSuLQJBpKcQ+/6hf8e5vskSStP\n",
       "mwb/buB9wH9W1XeTPAf4WrdlSZLGxesYJGmV6uI6ho9V1buSfHmOl6uqXjvsm0mSJt9C90r6dP/f\n",
       "f57jtcnfzJAkLUmbeyX9BvBYVf2qv7wGeHJVPTqC+mZqcFeSJA2py9tufwV4ysDy0cCNw76RJGll\n",
       "aBMMR1XV/84sVNUj9MJBkrQKtQmGR5OcMbOQ5MXAY92VJEkapzYf1PNu4HNJZu6d9Czggu5KkiSN\n",
       "U6vrGJIcAZxC7xPctlfV/q4Lm/X+HnyWpCF1dvA5yTHAe4F3VdW9wElJ/mQJNUqSVoA2xxiuBf4P\n",
       "eGl/eR/wwc4qkiSNVZtgeE5VfYReODDK6xckSaPXJhh+meTx6xj6N9H7ZXclSZLGqc1ZSZuB64ET\n",
       "kvw78DLg4g5rkiSN0YLBkORJwNOA1wNn959+V1X9pOvCJEnj0eZeSd+uqjMWHNQxT1eVpOEttXe2\n",
       "CYZ/BB4E/gN4/MBzVT007JstlcEgScPrMhh2Mcdttqvq2cO+2VIZDJI0vGX/oJ4BzwfeAbwc+DXw\n",
       "P8CVw76RJGllaHO66qfphcPHgCuAF3DwQ3wWlGRjku1JdiS5dIFxL0lyIMnr2qxXktSdNlsML6yq\n",
       "FwwsfzXJfYt9U/8Dfa4AzgX2Arcl2VpV2+YY9xF6p8S6u0iSxqzNFsMdSc6ZWUhyNvDtFt93JrCz\n",
       "qnb1b7q3BTh/jnHvBD4PeAqsJE2ANlsMLwZuTrKb3kHoE4H7k9wLVFWdOs/3rQd2DyzvAc4aHJBk\n",
       "Pb2w+CPgJfhZ0pI0dm2CYeMS192myX8UeG9VVZLgriRJGrtFg6Gqdi1x3XuBDQPLG+htNQw6A9jS\n",
       "ywTWAa9Jsr+qts5eWZLNA4vTVTW9xLokaVVKMgVMHfJ62nxQz5JWnKwF7gdeRe9W3bcCF84++Dww\n",
       "/lrgy1X1xTle8zoGSRpSl9cxLElVHUiyCbgBWANcU1XbklzSf/2qrt5bkrR0nW0xLCe3GCRpeJ19\n",
       "tKck6fBiMEiSGgwGSVKDwSBJajAYJEkNBoMkqcFgkCQ1GAySpAaDQZLUYDBIkhoMBklSg8EgSWow\n",
       "GCRJDQaDJKnBYJAkNRgMkqQGg0GS1GAwSJIaDAZJUoPBIElqMBgkSQ0GgySpwWCQJDUYDJKkBoNB\n",
       "ktRgMEiSGgwGSVKDwSBJajAYJEkNBoMkqcFgkCQ1GAySpAaDQZLUYDBIkhoMBklSg8EgSWroPBiS\n",
       "bEyyPcmOJJfO8fqbktyd5J4kNyc5teuaJEnzS1V1t/JkDXA/cC6wF7gNuLCqtg2MOQe4r6p+nmQj\n",
       "sLmqzp61nqqqdFaoJK1CS+2dXW8xnAnsrKpdVbUf2AKcPzigqr5ZVT/vL94CnNBxTZKkBXQdDOuB\n",
       "3QPLe/rPzeftwHWdViRJWtDajtffej9Vkj8E3ga8bJ7XNw8sTlfV9CFVJkmrTJIpYOpQ19N1MOwF\n",
       "Ngwsb6C31dDQP+B8NbCxqh6ea0VVtbmLAiVptej/wTw9s5zksqWsp+tdSbcDJyc5KcmRwAXA1sEB\n",
       "SU4Evgi8uap2dlyPJGkRnW4xVNWBJJuAG4A1wDVVtS3JJf3XrwI+ADwNuDIJwP6qOrPLuiRJ8+v0\n",
       "dNXl4umqkjS8ST1dVZK0whgMkqSGrs9KkiR1IGEN8Ax6Z3ueMMe/S75Y2GMMkjRhWjT9DcAzgYfp\n",
       "XQKwe+5/84Ol9E6DQZJGaPmaPvuq+OXC77W03mkwSNIyGWXTb1ePwSBJnZm0pt+uZoNBkpZkJTb9\n",
       "NgwGSZrDam36bRgMkg47h3PTb8NgkLSq2PQPncEgacWw6Y+GwSBpItj0J4fBIKlzNv2VxWCQdEhs\n",
       "+quPwSBpXjb9w5PBIB2mbPqaj8EgrUI2fR0Kg0FaYWz66prBIE0Qm74mgcEgjYhNXyuFwSAtg5ZN\n",
       "/1nAQ9j0NeEMBmkRy9D0Zx7b9LUiGAw6rNn0pScyGLRq2fSlpTEYtCLZ9KXuGAyaODZ9abwMBo2U\n",
       "TV+afAaDlo1NX1odDAa1YtOXDh8Gg2z6khoMhlXOpi9pWAbDCmbTl9QFg2FC2fQljYvBMAY2fUmT\n",
       "bCKDIclG4KPAGuATVfWROcZcDrwG+AVwcVXdOceYkQeDTV/SSrfU3rm2i2IAkqwBrgDOBfYCtyXZ\n",
       "WlXbBsacBzy3qk5OchZwJXB2VzUdfN9Davp3Msamn2SqqqZH+Z6Tyrk4yLk4yLk4dJ0FA3AmsLOq\n",
       "dgEk2QKcD2wbGPNa4FMAVXVLkuOSHF9VDyz1TVdy029pCpgecw2TYgrnYsYUzsWMKZyLQ9JlMKyn\n",
       "12Bn7AHOajHmBGDOYDgMmr4kjV2XwdD24MXs/V9zfl/Cj7DpS1LnugyGvfT+gp+xgV7zXmjMCf3n\n",
       "5pAT+w+O73+dMd8bZ+LOX1peSS4bdw2Twrk4yLk4yLk4NF0Gw+3AyUlOAvYBFwAXzhqzFdgEbEly\n",
       "NvCzuY4vTOKpqpK0WnUWDFV1IMkm4AZ6p6teU1XbklzSf/2qqrouyXlJdgKPAm/tqh5JUjsr4gI3\n",
       "SdLoPGncBQxKsjHJ9iQ7klw6z5jL+6/fneT0Udc4KovNRZI39efgniQ3Jzl1HHWOQpufi/64lyQ5\n",
       "kOR1o6xvVFr+fkwluTPJd5JMj7jEkWnx+7EuyfVJ7urPxcVjKHMkknwyyQNJ7l1gzHB9s6om4ove\n",
       "7qadwEnAEcBdwPNnjTkPuK7/+CzgW+Oue4xzcQ7w1P7jjYfzXAyM+yrwX8Drx133mH4mjgO+C5zQ\n",
       "X1437rrHOBebgQ/PzAPwU2DtuGvvaD5eAZwO3DvP60P3zUnaYnj8griq2g/MXBA3qHFBHHBckuNH\n",
       "W+ZILDoXVfXNqvp5f/EWemd0rUZtfi4A3gl8HvjJKIsboTbz8EbgC1W1B6CqHhxxjaPSZi5+DBzb\n",
       "f3ws8NOqOjDCGkemqm4CHl5gyNB9c5KCYa6L3da3GLMaG2KbuRj0duC6Tisan0XnIsl6eo3hyv5T\n",
       "q/HAWZufiZOBpyf5WpLbk1w0supGq81cXA28MMk+4G7gXSOqbRIN3Te7PF11WMt6QdwK1/r/lOQP\n",
       "gbcBL+uunLFqMxcfBd5bVZUkPPFnZDVoMw9HAC8CXgUcDXwzybeqakenlY1em7l4P3BXVU0leQ5w\n",
       "Y5LTquqRjmubVEP1zUkKhmW+IG5FazMX9A84Xw1srKqFNiVXsjZzcQa9a2Ggtz/5NUn2V9XW0ZQ4\n",
       "Em3mYTfwYFU9BjyW5BvAacBqC4Y2c/FS4IMAVfX9JD8ETqF3fdXhZui+OUm7kh6/IC7JkfQuiJv9\n",
       "i70VeAvAQhfErQKLzkWSE4EvAm+uqp1jqHFUFp2Lqvrdqnp2VT2b3nGGv15loQDtfj++BLw8yZok\n",
       "R9M70HjfiOschTZzsZ3enZ3p708/BfjBSKucHEP3zYnZYigviHtcm7kAPgA8Dbiy/5fy/qo6c1w1\n",
       "d6XlXKx6LX8/tie5HrgH+DVwdVWtumBo+TPxIeDaJHfT+wP4PVX10NiK7lCSzwCvBNYl2Q1cRm+3\n",
       "4pL7phe4SZIaJmlXkiRpAhgMkqQGg0GS1GAwSJIaDAZJUoPBIElqMBikjiW5OMnH+483J/nbcdck\n",
       "LcRgkOaRvmVYVc3zWJpIBoM0oH+bhfuTfAq4F/iHJLf2P+Bk88C4t/Sfu6s/liR/muRbSe5IcmOS\n",
       "Z4zpvyEdkom5JYY0QZ4LXAQ8FfizqjozyZOALyV5BfAQ8PfAOVX1UJKn9b/vpqo6GyDJXwLvAf6O\n",
       "1Xm3V61iBoP0RD+qqluT/BPwx0nu7D9/DL3QOAb47My9dwbubLshyWeBZwJHcvjetE0rnLuSpCd6\n",
       "dODxh6vq9P7X86rq2v7zc20FfBy4vKpOBS4BntJ1oVIXDAZpfjcAb0tyDPQ+KS7Jb9H7bOk3JHl6\n",
       "//mZXUnHAvv6jy+eZ53uVtLEc1eS9EQFUFU3Jnk+vU9CA3iE3udf3Jfkg8DXk/wKuIPep+htBj6X\n",
       "5GF64fE7A+urOR5LE8nbbkuSGtyVJElqMBgkSQ0GgySpwWCQJDUYDJKkBoNBktRgMEiSGgwGSVLD\n",
       "/wNbVNZXgI3/9QAAAABJRU5ErkJggg==\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x3975080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.51,.5,.5,.5,.5,.5])\n",
    "pr(y_true, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier C flips the predictions from classifier B\n",
    "Classifier B performed worse than a random classifier by the naive auPRC, whose precision recall curve would be a horizontal line at 1/6, but the same as random when using `average_precision_score`.  We flip the predictions of classifier B to obtain classifer C.  Classifier C attains perfect recall after making 5 predictions (the tied predictions) so the linear interpolation is from 1 to 1/5.\n",
    "\n",
    "Classifier C has naive auPRC comparable to classifier A (slighly better, as expected), but arguably both A and C have inflated auPRC due to the linear interpolation in the naive calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.2  1. ]\n",
      "recall: [ 1.  0.]\n",
      "thresholds: [-0.5]\n",
      "Naive auPRC: 0.6\n",
      "average_precision_score auPRC: 0.6\n"
     ]
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAFjpJREFUeJzt3X+wpQV93/H3x10gSkLEobUVNsUfKwIRqig/tGk21dEN\n",
       "U6UxOkiMGUI6bmyxdpoGhBjdGSvE/HACkjIWhOh0AhLWjqRloHR1I0MUA/LLwMJuLJldNmNFMGMN\n",
       "Y3fl2z/Ouew5d+/ee+7d+5znOee8XzM7c59znvvc7z5z7/3c5zzP5zypKiRJmvO8tgeQJHWLwSBJ\n",
       "GmIwSJKGGAySpCEGgyRpiMEgSRrSaDAkuS7Jt5M8tMg6VybZkeSBJK9pch5J0tKaPmK4Hth4sCeT\n",
       "nA28oqrWA+8Drm54HknSEhoNhqq6E3h6kVXeDny2v+7dwAuTvLjJmSRJi2v7HMOxwK6B5d3AcS3N\n",
       "Ikmi/WAAyLxl36NDklq0tuWv/wSwbmD5uP5jQ5IYFpK0AlU1/4/vJbUdDLcAFwI3JjkT+F5VfXvh\n",
       "Veto4CJgE3AdcHkVT41pzs5IsrmqNrc9Rxe4L/ZzX+znvthvpX9UN3256g3AXwAnJNmV5IIkm5Js\n",
       "AqiqW4FvJdkJfBr4NwfbVhXfq+JS4NXAjwOPJlyacGST/wdJmjWNHjFU1XkjrHPh8rbJHuD9CZ8E\n",
       "PgbsSPgYcG0Ve1c2qSRpThdOPq9IFTuqeDfwNuBfAY8kvDuZ3P/TiLa1PUCHbGt7gA7Z1vYAHbKt\n",
       "7QEmXSbhRj1JaqkTKAn/AvgdekdBlwD/s8ornCTNrlF+dy74edMSDL31CPAO4OPAHuCSKu5uej5J\n",
       "6qKVBsNUvexSRVWxBfhp4E+AmxO2JLyq5dEkaWJMVTDMqWJfFdcCrwS+Bnwl4drEVrUkLWUqg2FO\n",
       "Fc9U8Xv0AuL/AA8k/F7Ci1oeTZI6a6qDYY4dCEka3UwEw5wq9lTxfuANwCn0OhDvTzis5dEkqTNm\n",
       "KhjmzHAHQpKWNFWXq658+3YgJE0fewyH/DXsQEiaLvYYDpEdCEnqMRjmsQMhadYZDAdhB0LSrDIY\n",
       "lmAHQtKsMRhGZAdC0qwwGJbJDoSkaeflqofIDoSkrrLH0CI7EJK6yB5Di+xASJomBsMqsgMhaRoY\n",
       "DA2wAyFpkhkMDbIDIWkSGQxjYAdC0iQxGMbIDoSkSeDlqi2yAyGpSfYYJpQdCElNsccwoexASOoa\n",
       "g6Ej7EBI6gqDoWPsQEhqm8HQUXYgJLXFYOg4OxCSxs1gmBB2ICSNi5erTig7EJKWYo9hBtmBkLQY\n",
       "ewwzyA6EpCY0GgxJNibZnmRHkosXeP6YJLcluT/JN5Oc3+Q808oOhKTV1FgwJFkDXAVsBE4Czkty\n",
       "4rzVLgTuq6p/CmwA/iDJ2qZmmnZ2ICSthiaPGE4HdlbV41W1F7gROGfeOn8LHNX/+Cjgu1W1r8GZ\n",
       "ZoIdCEmHoslgOBbYNbC8u//YoGuAk5PsAR4APtjgPDPHDoSklWjyZZtRLne6FLi/qjYkeTlwR5JT\n",
       "q+r781dMsnlgcVtVbVudMadfFTuAdyecBlwG/EbCh4Gbqni23ekkrZYkG+i9LH9o22nqctUkZwKb\n",
       "q2pjf/kS4Nmq+sTAOrcCH6+qu/rLW4GLq+qeedvyctVVZAdCmg1dvFz1HmB9kuOTHA6cC9wyb53t\n",
       "wJsBkrwYOAH4VoMzCajiS8AZ9PoPVwBbE85odypJXdFYMPRPIl8I3A48DHy+qh5JsinJpv5qlwGv\n",
       "S/IA8L+Ai6rqqaZm0n52ICQdjM1nAZDwfHpB/pv0juw2V7G73akkHYouvpSkCWIHQtIcg0FD7EBI\n",
       "Mhi0IDsQ0uwyGLQo7wMhzR5PPmtZ7EBIk8P7MWhsvA+ENBm8KkljYwdCmm4Gg1bM+0BI08lg0CGz\n",
       "AyFNF4NBq8YOhDQdDAatunkdiFOxAyFNFINBjel3IM6l14H4BexASBPBy1U1NglvAi7HDoQ0FvYY\n",
       "NBH6HYhfpNeBeAI7EFJj7DFoIvQ7EDcDJwM3AFvsQEjdYjCoFf0OxDXAeuBu4E47EFI3GAxqVb8D\n",
       "8bv0OhDfwQ6E1DqDQZ1QxdNVXEKvA/ETwGN2IKR2GAzqlH4H4teBs7ADIbXCYFAn2YGQ2uPlqpoI\n",
       "diCk5bPHoKlnB0JaHnsMmnp2IKTxMBg0cexASM0yGDSx7EBIzTAYNPHsQEiry2DQ1LADIa0Og0FT\n",
       "xw6EdGi8XFVTzw6EZpU9BmkRdiA0i+wxSIuwAyGNzmDQTLEDIS3NYNBMsgMhHZzBoJlmB0I6kMEg\n",
       "YQdCGtRoMCTZmGR7kh1JLj7IOhuS3Jfkm0m2NTmPtBQ7EFKDl6smWQM8CryZ3uWBfwmcV1WPDKzz\n",
       "QuAu4K1VtTvJMVX15ALb8nJVtcIOhCZZFy9XPR3YWVWPV9Ve4EbgnHnr/BKwpap2AywUClKbqtgK\n",
       "nAFcBlwJbE04o92ppGY1GQzHArsGlnf3Hxu0HnhRki8nuSfJexucR1oROxCaNU0GwyiH24cBrwXO\n",
       "Bt4K/HaS9Q3OJK2YHQjNirUNbvsJYN3A8jp6Rw2DdgFPVtUzwDNJvkL/ipD5G0uyeWBxW1VtW9Vp\n",
       "pRFV8QzwuwnXABfR60BcB1xexVPtTqdZlmQDsOGQtzPKyeckxwLHA2uAAFVVX1nic9bSO/n8JmAP\n",
       "8HUOPPn8KuAqekcLR9D7K+zcqnp43rY8+azOSngJ8BHgncAngSuq+EG7U0kr/9255BFDkk8A5wIP\n",
       "Az8aeGrRYKiqfUkuBG6nFyifqapHkmzqP//pqtqe5DbgQeBZ4Jr5oSB1XRV7gF9P+APgP9HrQHwM\n",
       "uLaKve1OJy3fkkcMSR4DXl1VPxzPSAvO4BGDJkbCafQucX0Z8GHgpiqebXcqzaImL1f9a+Dw5Y8k\n",
       "zaYq7q3iLcAm4D8A9yS8tf/W31LnjXLE8AV6J4S3AnNHDVVV/67h2QZn8IhBE8n7QKhNjd2oJ8n5\n",
       "/Q/nVpw7+fzZ5X6xlTIYNOkS1gK/CnyU3kUWv1XF9nan0rRr9A5uSY6g9/bEANv7TeaxMRg0LRKe\n",
       "D3wA+E3gi8DmqgMu45ZWRWPnGPrXxT4G/FH/344kP7vsCSV5HwhNhFFeSvoGvf7Bo/3lVwI3VtVr\n",
       "xzDf3AweMWgq2YFQk5q8KmntXCgAVNVjNNuYlmaG94FQF41yxHA9vWLbf6V34vk9wPOq6oLmx3tu\n",
       "Bo8YNBPsQGg1NXlV0o8B/xZ4Y/+hO4H/PM7Cm8GgWeN9ILQaGr0qqW0Gg2aRHQgdqlUPhiR/WlXv\n",
       "SvLQAk9XVZ2y3C+2UgaDZpkdCK1UE8Hwkqrak+T4hZ6vqseX+8VWymCQ7EBo+Vb9qqSq2tP/8DvA\n",
       "rn4QHAGcQu+wVtIY2YHQuIxyueqdwBH9ezLcDrwX+OMmh5J0cFU8XcUlwKuBnwAeS7g04ciWR9OU\n",
       "GCUYUlV/D7yD3tVI7wJ+utmxJC3FDoSaMtI9n5OcRa+/8D+W83mSmlfFjirOBd4G/ALwSMK7E39O\n",
       "tTKjfOP8e3rXUf+3qvqrJC8HvtzsWJKWy/tAaLXYY5CmkB0IQTOXq15RVR9M8mcLPF1V9fblfrGV\n",
       "MhiklbEDMduaCIbTqure/ttuz1dV9efL/WIrZTBIh8YOxGxq8r2Sfhx4pqp+1F9eA/xYVY3trYEN\n",
       "Bml1JBwNXAS8D7gOuLyKp9qdSk1p8m23twLPH1h+AXDHcr+QpPbZgdAoRgmGI6rq/84tVNX36YWD\n",
       "pAllB0KLGSUYfpDktLmFJK8DnmluJEnjYgdCCxnlHMPrgc8Dc++d9I+Bc6vqnoZnG5zBcwzSGHgf\n",
       "iOnS6P0YkhwGnEDvDm7bq2rv8kdcOYNBGh87ENOjsZPPSY4EPgR8sKoeAo5P8i9XMKOkCVBFVXEz\n",
       "cDJwA7AlYUvCq1oeTWMyyuuI1wP/D3hDf3kPvb8kJE2xKvZVcQ2wnl457s6EaxOOa3k0NWyUYHh5\n",
       "VX2CXjgwzv6CpPZ5H4jZM0ow/DDJcz2G/pvo/bC5kSR1kR2I2TFKMGwGbgOOS/InwJeAi5scSlJ3\n",
       "2YGYfotelZTkecC76LWfz+w/fHdVfWcMsw3O4VVJUkclnEbvEteXAR8Gbqri2XanEjT7Xkn3VtVp\n",
       "i67UMINB6j47EN3TZDD8DvAkvZLbcyeeq2psb7xlMEiTwQ5EtzQZDI/DgalfVS9d7hdbKYNBmize\n",
       "B6Ibmnx31ROBPwIeAO4DPgWctNwvJGl22IGYbKMEw+fohcMVwFX0QuFzo2w8ycYk25PsSHLQK5mS\n",
       "vD7JviTvGGW7kiaDHYjJNMpLSQ9X1UlLPbbA560BHgXeTO+1xr8EzquqRxZY7w7g74Hrq2rLAtvy\n",
       "pSRpCiS8BPgI8E7gk8AVVViabUiTLyV9I8lZA1/oTODeET7vdGBnVT3ef9O9G4FzFljvA8DN9P6a\n",
       "kDTF7EBMhlGC4XXAXUn+pn8i+i+A1yV5KMmDi3zescCugeXd/ceek+RYemFxdf8hL22TZoD3gei2\n",
       "tSOss3GF2x7ll/wfAh+qqkoSem/rLWlGVHEv8JaBDsRFiR2Iti0ZDFX1+Aq3/QSwbmB5Hb2jhkGn\n",
       "ATf2MoFjgJ9Psreqbpm/sSSbBxa3VdW2Fc4lqWOq2JpwBr0OxJXAE4kdiOVKsgHYcMjbGeVGPSva\n",
       "cLKW3snnN9F7q+6vs8DJ54H1rwf+rKq+sMBznnyWZoQdiNXT5MnnFamqfcCFwO3Aw8Dnq+qRJJuS\n",
       "bGrq60qabHYg2tfYEcNq8ohBml0JRwMXAe8DrgMur2Jsb8kzyTp3xCBJq8H7QIyfwSBpItiBGB+D\n",
       "QdJEsQPRPM8xSJpo3gfi4Bp72+0uMBgkLcb7QCzMk8+SZlYVVcXNwMnADcCWhC0Jr2p5tIlkMEia\n",
       "GnYgVofBIGnqeB+IQ2MwSJpadiBWxmCQNPXsQCyPwSBpZtiBGI2Xq0qaWdPegbDHIEkrMM0dCHsM\n",
       "krQCdiAOZDBIEnYgBhkMkjTADoTBIEkLmuUOhMEgSYuYxQ6EwSBJI5ilDoSXq0rSCkxCB8IegySN\n",
       "Wdc7EPYYJGnMprUDYTBI0iGatg6EwSBJq2RaOhAGgyStsknvQBgMktSQSe1AGAyS1LBJ60B4uaok\n",
       "jdm4OhD2GCRpgoyjA2GPQZImSJc7EAaDJLWoix0Ig0GSOqBLHQiDQZI6pAsdCINBkjqozQ6EwSBJ\n",
       "HdZGB8LLVSVpgiynA9HZy1WTbEyyPcmOJBcv8Px7kjyQ5MEkdyU5pemZJGlSVbEVOAO4DLgS2Jpw\n",
       "xmp+jUaDIcka4CpgI3AScF6SE+et9i3gn1fVKcDHgP/S5EySNOma7kA0fcRwOrCzqh6vqr3AjcA5\n",
       "gytU1Ver6u/6i3fDZL5/uSSNW1MdiKaD4Vhg18Dy7v5jB/NrwK2NTiRJU+ZgHYiVbm/tqk22sJHP\n",
       "bCf5OeAC4I0HeX7zwOK2qtp2SJNJ0tTJqcAP4cV/DG/dsNKtNB0MTwDrBpbX0TtqGNI/4XwNsLGq\n",
       "nl5oQ1W1uYkBJWla9P9g3ja3nHxuRZedNv1S0j3A+iTHJzkcOBe4ZXCFJD8FfAH45ara2fA8kqQl\n",
       "NHrEUFX7klwI3A6sAT5TVY8k2dR//tPAR4CjgauTAOytqtObnEuSdHAW3CRpSnW24CZJmiwGgyRp\n",
       "iMEgSRpiMEiShhgMkqQhBoMkaYjBIEkaYjBIkoYYDJKkIQaDJGmIwSBJGmIwSJKGGAySpCEGgyRp\n",
       "iMEgSRpiMEiShhgMkqQhBoMkaYjBIEkaYjBIkoYYDJKkIQaDJGmIwSBJGmIwSJKGGAySpCEGgyRp\n",
       "iMEgSRpiMEiShhgMkqQhBoMkaYjBIEkaYjBIkoYYDJKkIQaDJGmIwSBJGtJoMCTZmGR7kh1JLj7I\n",
       "Olf2n38gyWuanEeStLTGgiHJGuAqYCNwEnBekhPnrXM28IqqWg+8D7i6qXmmRZINbc/QFe6L/dwX\n",
       "+7kvDl2TRwynAzur6vGq2gvcCJwzb523A58FqKq7gRcmeXGDM02DDW0P0CEb2h6gQza0PUCHbGh7\n",
       "gEnXZDAcC+waWN7df2ypdY5rcCZJ0hKaDIYacb2s8PMkSQ1Y2+C2nwDWDSyvo3dEsNg6x/UfO0AS\n",
       "A6MvyUfbnqEr3Bf7uS/2c18cmiaD4R5gfZLjgT3AucB589a5BbgQuDHJmcD3qurb8zdUVfOPKiRJ\n",
       "DWksGKpqX5ILgduBNcBnquqRJJv6z3+6qm5NcnaSncAPgF9tah5J0mhS5Ss0kqT9OtV8thC331L7\n",
       "Isl7+vvgwSR3JTmljTnHYZTvi/56r0+yL8k7xjnfuIz487EhyX1Jvplk25hHHJsRfj6OSXJbkvv7\n",
       "++L8FsYciyTXJfl2kocWWWd5vzerqhP/6L3ctBM4HjgMuB84cd46ZwO39j8+A/ha23O3uC/OAn6y\n",
       "//HGWd4XA+t9CfjvwC+2PXdL3xMvBP4KOK6/fEzbc7e4LzYDl8/tB+C7wNq2Z29of/wM8BrgoYM8\n",
       "v+zfm106YrAQt9+S+6KqvlpVf9dfvJvp7X+M8n0B8AHgZuA74xxujEbZD78EbKmq3QBV9eSYZxyX\n",
       "UfbF3wJH9T8+CvhuVe0b44xjU1V3Ak8vssqyf292KRgsxO03yr4Y9GvArY1O1J4l90WSY+n9Yph7\n",
       "S5VpPHE2yvfEeuBFSb6c5J4k7x3bdOM1yr64Bjg5yR7gAeCDY5qti5b9e7PJy1WXy0LcfiP/n5L8\n",
       "HHAB8MbmxmnVKPviD4EPVVUlCQd+j0yDUfbDYcBrgTcBLwC+muRrVbWj0cnGb5R9cSlwf1VtSPJy\n",
       "4I4kp1bV9xuerauW9XuzS8GwqoW4CTfKvqB/wvkaYGNVLXYoOclG2Ren0evCQO/15J9PsreqbhnP\n",
       "iGMxyn7YBTxZVc8AzyT5CnAqMG3BMMq+eAPwcYCq+usk/xs4gV6/atYs+/dml15Keq4Ql+RweoW4\n",
       "+T/YtwC/ArBYIW4KLLkvkvwU8AXgl6tqZwszjsuS+6KqXlZVL62ql9I7z/D+KQsFGO3n44vAP0uy\n",
       "JskL6J1ofHjMc47DKPtiO/BmgP7r6ScA3xrrlN2x7N+bnTliKAtxzxllXwAfAY4Gru7/pby3qk5v\n",
       "a+amjLgvpt6IPx/bk9wGPAg8C1xTVVMXDCN+T1wGXJ/kAXp/AF9UVU+1NnSDktwA/CxwTJJdwEfp\n",
       "vay44t+bFtwkSUO69FKSJKkDDAZJ0hCDQZI0xGCQJA0xGCRJQwwGSdIQg0FqWJLzk3yq//HmJL/R\n",
       "9kzSYgwG6SDStwqbqoN8LHWSwSAN6L/NwqNJPgs8BPx2kq/3b3CyeWC9X+k/dn9/XZK8LcnXknwj\n",
       "yR1J/mFL/w3pkHTmLTGkDnkF8F7gJ4F3VtXpSZ4HfDHJzwBPAb8FnFVVTyU5uv95d1bVmQBJ/jVw\n",
       "EfAfmc53e9UUMxikA/1NVX09ye8Db0lyX//xI+mFxpHATXPvvTPwzrbrktwE/CPgcGb3Tds04Xwp\n",
       "STrQDwY+vryqXtP/98qqur7/+EJHAZ8CrqyqU4BNwPObHlRqgsEgHdztwAVJjoTeneKS/AN695Z+\n",
       "V5IX9R+feynpKGBP/+PzD7JNX1ZS5/lSknSgAqiqO5KcSO9OaADfp3f/i4eTfBz48yQ/Ar5B7y56\n",
       "m4E/TfI0vfD4JwPbqwU+ljrJt92WJA3xpSRJ0hCDQZI0xGCQJA0xGCRJQwwGSdIQg0GSNMRgkCQN\n",
       "MRgkSUP+P+PSpkasw261AAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x357a710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.51,.5,.5,.5,.5,.5])\n",
    "# flip the predicted class labels\n",
    "y_scores = -y_scores\n",
    "pr(y_true, y_scores)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
 }