Precision recall curves in scikit-learn version 0.19.0 with ties

sklearn-auprc-v19.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Precision recall curves in scikit-learn with ties\n",
    "\n",
    "August 16, 2017\n",
    "\n",
    "Anthony Gitter\n",
    "\n",
    "An exaggerated example of what my group saw on a recent dataset, which is the same problem Anshul describes here https://twitter.com/anshul/status/761117086915497984\n",
    "\n",
    "Testing with scikit-learn version 19.0 to see if the updated average precision function changes the behavior.  The previous example manually computed the auPRC with `metrics.auc` on the precision and recall points so it did not have the opportunity to do advanced interpolation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "scikit-learn version: 0.19.0\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn import metrics\n",
    "\n",
    "print 'scikit-learn version: {}'.format(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper function to compute auPRC and plot the precision recall curve\n",
    "scikit-learn uses linear interpolation when ties are present"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pr(y_true, y_scores):\n",
    "    precision, recall, thresholds = metrics.precision_recall_curve(y_true, y_scores)\n",
    "    auc = metrics.auc(recall, precision)\n",
    "    \n",
    "    print 'precision: %s' % precision\n",
    "    print 'recall: %s' % recall\n",
    "    print 'thresholds: %s' % thresholds\n",
    "    print 'Naive auPRC: %s' % auc\n",
    "    print 'average_precision_score auPRC: %s' % metrics.average_precision_score(y_true, y_scores)\n",
    "    \n",
    "    plot(recall, precision);\n",
    "    xlabel('recall');\n",
    "    ylabel('precision');\n",
    "    xlim(0,1);\n",
    "    ylim(0,1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier A predicts identical scores for all instances\n",
    "The naive auPRC treats the precision recall curve as a linear interpoloation from 1 to the baseline pos/(pos+neg) = 1/6.  The `average_precision_score` instead returns 1/6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.16666667  1.        ]\n",
      "recall: [ 1.  0.]\n",
      "thresholds: [ 0.5]\n",
      "Naive auPRC: 0.583333333333\n",
      "average_precision_score auPRC: 0.166666666667\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VGX+9/H3N91ACAKhh56ACIL0XkQU1F0sKFJEUUFE\nEHGtv2f32f393N2fi12qiMCKfZW1U0Qh9N47gdC7KCUhJIH7+SPjQxZlGAKTmUk+r+vKBWfmzJxv\n7gvyyZn7Pt9jzjlEREQuJCzQBYiISHBTUIiIiFcKChER8UpBISIiXikoRETEKwWFiIh45begMLMJ\nZnbIzNZd4HkzszfNLNXM1phZI3/VIiIi+efPM4pJQBcvz3cFkjxfA4AxfqxFRETyyW9B4ZybAxz1\nsks34F2XaxFQ0swq+KseERHJn4gAHrsSsDvP9h7PY/vP39HMBpB71kHYVSUax5WpQIX4GIpFB7J8\nEZHQsXz58iPOuYT8vDYkftI658YB4wBq1r3Olb3vNfYfy6R57QSe7VKHayqUCHCFIiLBzcx25ve1\ngVz1tBdIzLNd2fOYV1fHRjHrqQ4837UOK3b+xC1vzuUPn6xm78+n/FaoiEhRFsig+BLo61n91AI4\n5pz71cdOvyUmMpxH2tdkzjMdGdC2Bl+t2UfHl2fz92838nNGln+rFhEpYsxf3WPN7EOgA1AGOAj8\nGYgEcM6NNTMDRpK7MioD6OecW3ax923SpIlbtuw/d9v78yle+24Ln63YQ1x0BIM61uKBVtWIiQy/\not+TiEioMrPlzrkm+XptqLUZ/62g+MWmA8f5x9RNzNp8mArxMQzrnMxdjSoTHmYFXKWISHC5nKAo\nVFdm1ylfgon9mvFh/xaULRHDM5+uoesbc/h+40FCLRBFRIJFoQqKX7SsWZrPB7VidO9GZJ9xPPTP\nZfQYt4gVu34KdGkiIiGnUAYFgJlxS/0KzBjWjhdur8f2w+ncOXoBj763nG2HTwa6PBGRkFGo5ii8\nST+dw/i5aYybs43MnLPc2zSRoZ2SKFsixg9ViogEF01mX4LDJ04z4oetfLB4F5HhYfRvW53+7WoQ\nFxN5BasUEQkuCop82HEknZdmbOabNfspXSyKITfUolfzqkRFFNpP40SkCNOqp3yoVqYYo3o14ovH\nWpNcLo6/fLWBG19N4cvV+zh7NrTCU0TEn4psUPyiQWJJPujfnEn9mhIbFc7jH66k26j5zE89EujS\nRESCQpEPCshdIdWhdlm+ebwtr97TgKPpWfQev5i+E5awft+xQJcnIhJQRXaOwpvM7DNMXriTkbNS\nOZ6Zze0NK/Fk52QSS8X69bgiIv6iyWw/OXYqmzGztzFxfhrOwX0tqzK4Yy2uLhZVIMcXEblSFBR+\ntv9YbtPBT5fvoVhUBAM71OTB1tW5KkpNB0UkNCgoCsiWgycYPm0TMzceolyJaIbdmEz3xpWJCNdU\nj4gENy2PLSDJ5eIYf39TPnmkJRVLXsVzU9bS5Y25zFh/QE0HRaTQUlDkQ7PqpZjyaCvG9mnE2bOO\nAZOXc/fYhSzfeTTQpYmIXHEKinwyM7rUy206+Lc76rHzaAZ3jVnIgHeXkXpITQdFpPDQHMUVkpGV\nwztz03hrznYysnLo0TSRJ25MppyaDopIENBkdhD58eRpRvyQyvuLdxIeZjzUpjqPtK9JCTUdFJEA\nUlAEoV0/ZvDyjM18uXofV8dGMviGJPq0qEJ0hJbUikjB06qnIFSldCxv9ryer4e04dqK8bzw9QY6\nvZLC5yv3qumgiIQUBYWf1asUz3sPN+fdB5tRIiaSJz5exW0j5jFny+FAlyYi4hMFRQFpl5zA10Pa\n8HqPhhzPzKbvhCX0Gb+YdXvVdFBEgpuCogCFhRm3X1+J7//Qnj/dVpf1+45x24h5PP7hSnb9mBHo\n8kREfpMmswPoeGY2b6Vs4515aZw56+jdvCpDbqhF6eLRgS5NRAoZrXoKcQePZ/L6zC18vHQ3sVER\nPNKuBg+1rU5sVESgSxORQkJBUUikHjrB8GmbmbHhIAlx0TxxYxI9miSq6aCIXDYtjy0kapWNY1zf\nJnw6sCVVSsXyf/69jpten8O0dWo6KCKBo6AIQk2qleLTgS0Zd19jDBj43nLuGrOApTvUdFBECp6C\nIkiZGTddW57pT7TjxTvrs/fnU9w9diEP/3MpWw+eCHR5IlKEaI4iRJzKOsOE+WmMnb2N9Kwcujeu\nzLDOyVSIvyrQpYlICNBkdhFyND2LUbNSmbxwJ2bQr3V1Hu1Qk/ir1HRQRC5MQVEE7T6awavfbeHz\nVXuJvyqSwR1r0adFVWIi1XRQRH5Nq56KoMRSsbzWoyFfD2nDdZVL8tdvNtLplRSmrNjDGTUdFJEr\nSEER4q6tGM+7Dzbj/Yebc3WxSJ78ZDW3vjmX2ZsPaUmtiFwRCopConWtMnz5WBve7Hk9GVlneGDi\nUnq9vZg1e34OdGkiEuL8GhRm1sXMNptZqpk99xvPx5vZV2a22szWm1k/f9ZT2IWFGb9vUJGZT7bn\nL7+ry+aDJ/j9yPk89sEKdhxJD3R5IhKi/DaZbWbhwBagM7AHWAr0dM5tyLPPfwHxzrlnzSwB2AyU\nd85lXeh9NZntuxOZ2bw9Zztvz00j+8xZejevwpBOSZRR00GRIidYJ7ObAanOue2eH/wfAd3O28cB\ncWZmQHHgKJDjx5qKlLiYSJ68qTYpT3egR9NE3lu8i/bDZ/HGzK2kn9Ywi4hv/BkUlYDdebb3eB7L\nayRwDbAPWAsMdc6dPf+NzGyAmS0zs2WHD+vOcJeqbIkY/nZHfWYMa0e75ARem7mF9i/NZvKinWSf\n+dVwi4j8h0BPZt8MrAIqAg2BkWZW4vydnHPjnHNNnHNNEhISCrrGQqNmQnHG9GnMlEGtqFGmGH/6\nfB03vTaHb9fu1wopEbkgfwbFXiAxz3Zlz2N59QOmuFypQBpQx481CdCoytV8/EgL3rm/CZHhxqD3\nV3D76AUs2v5joEsTkSDkz6BYCiSZWXUziwLuBb48b59dQCcAMysH1Aa2+7Em8TAzOl1TjqlD2zG8\n+3UcOp7JveMW8eCkpWw6cDzQ5YlIEPFrCw8zuwV4HQgHJjjn/mZmAwGcc2PNrCIwCagAGPCic+49\nb++pVU/+kZl9hkkLdjBqVionT+dwV6PcpoOVSqrpoEhhoF5PcsX8nJHF6NnbmLRgBwD9WlVjUIda\nxMeq6aBIKFNQyBW356fcpoP/XrmXuOgIHutYi/tbVVPTQZEQpaAQv9m4/zj/mLaJ2ZsPUzE+hmGd\nk7mzUWXCwyzQpYnIJQjWC+6kELimQgkm9WvGB/2bkxAXzdOfruGWN+byw6aDWlIrUkQoKMQnrWqW\n4fPHWjOqVyNO55zhwUnLuHfcIlbu+inQpYmInykoxGdmxq3XVeC7J9vzQrdr2Xb4JHeMXsCg95ez\n/fDJQJcnIn6iOQrJt5OnczxNB7dzOucsPZsl8ninJMrGxQS6NBE5jyazJaAOnzjNm99v5cMlu4iK\nCOPhtjUY0K4GxaMjAl2aiHgoKCQopB1J5+Xpm/lm7X5KF4vi8U5J9GxWhagIfcIpEmha9SRBoXqZ\nYozq3YjPH2tNUrni/PnL9XR+LYWvVu/jrO7jLRKyFBRyxTVMLMmH/VswsV9TrooMZ8iHK7l99HwW\npB4JdGkikg8KCvELM6Nj7bJ883hbXrm7AUdOnKbX+MXcP2EJG/ap6aBIKNEchRSIzOwzvLtwB6Nm\nbeN4ZjZ3NKzEkzclU/nq2ECXJlIkaDJbQsaxjGxGp6Qycf4OcNC3ZVUe61iLq4tFBbo0kUJNQSEh\nZ9/Pp3jtuy18tmIPxaIjeLRDTR5sXV1NB0X8REEhIWvzgRMMn7aJ7zcdonyJGIZ1TuKuRpWJCNf0\nmciVpOWxErJql4/jnQea8vGAFpSPj+HZz9bS9Y25fLdBTQdFgoWCQoJC8xql+fegVozp3YgzZx39\n313GPW8tZPlONR0UCTQFhQQNM6Nr/QpMH9aOv95ej7QjGdw1ZgGPTF7GNjUdFAkYzVFI0Eo/ncM7\n89J4K2UbmTlnuadJIsNuTKJsCTUdFLlUmsyWQu3IydOM/CGV9xfvJCIsjIfaVOeR9jWIi9F9vEV8\npaCQImHnj+m8PGMLX63eR6liUQzuWIveLaoQHaEltSIXo1VPUiRULV2MET2v58vBralTPo7/+XoD\nN76awher9qrpoIgfKSgk5FxXuSTvP9ycfz7YjOLRkQz9aBW/HzWPeVvVdFDEHxQUEpLMjPbJCXwz\npA2v9WjAT+nZ9HlnMfe9s5h1e48FujyRQkVBISEtLMy44/rK/PBUe/546zWs3XuM20bMY+hHK9l9\nNCPQ5YkUCprMlkLleGY2Y2dvY8L8NM6cdfRpUZUhNyRRSk0HpYjTqieR8xw4lsnrM7fwybLdFIuK\n4JH2NXiwTXVio3QfbymaFBQiF7D14AmGT9/MdxsOUjYummGdk7m7sZoOStGj5bEiF5BULo63+zbh\nXwNbklgqluenrOXm1+cwff0BNR0U8ZGCQoqEptVK8enAlrx1X2MAHpm8nO5jF7Jsx9EAVyYS/BQU\nUmSYGTdfW57pT7Tjf++sz+6jGXQfu5CH/7mMrQdPBLo8kaClOQopsjKycpg4fwdjZ28jPSuHuxsn\nMqxzMuXj1XRQCh9NZotchqPpWYz8IZXJi3YQHmY82Lo6j7SvSfxVajoohYeCQuQK2H00g1dmbObz\nVfsoGRvJ4I61uK9lVTUdlEIhaFc9mVkXM9tsZqlm9twF9ulgZqvMbL2ZpfizHhFvEkvF8vq91/P1\nkDbUrxTPX7/ZyA0vp/DvlXvUdFCKNL+dUZhZOLAF6AzsAZYCPZ1zG/LsUxJYAHRxzu0ys7LOuUPe\n3ldnFFJQ5m09wv9O3cj6fce5pkIJnutah3ZJZTCzQJcmcsmC9YyiGZDqnNvunMsCPgK6nbdPL2CK\nc24XwMVCQqQgtUkqw1eD2/DGvQ05eTqb+ycsoff4xazdo6aDUrT4MygqAbvzbO/xPJZXMnC1mc02\ns+Vm1ve33sjMBpjZMjNbdvjwYT+VK/JrYWFGt4aVmPlke/78u7psOnCC342cx5APV7Lzx/RAlydS\nIAJ9HUUE0Bi4FbgZ+JOZJZ+/k3NunHOuiXOuSUJCQkHXKEJ0RDj9Wlcn5ekODLmhFjM3HOTGV1P4\ny5frOXLydKDLE/ErfwbFXiAxz3Zlz2N57QGmO+fSnXNHgDlAAz/WJHJZ4mIi+cNNtUl5ugN3N0lk\n8qKdtB8+ize/30r66ZxAlyfiFz5NZptZNHAXUI3cswAAnHP/4+U1EeROZnciNyCWAr2cc+vz7HMN\nMJLcs4koYAlwr3Nu3YXeV5PZEkxSD53kpembmL7+IGWKR/PEjUn0aJpIpJoOSpApiMnsL8idiM4B\n0vN8XZBzLgcYDEwHNgKfOOfWm9lAMxvo2WcjMA1YQ25IjPcWEiLBplbZ4rx1XxM+e7QV1cvE8sfP\n13Hza3OYuna/mg5KoeHrGcU651y9AqjnonRGIcHKOcf3Gw/xj2mb2HroJA0TS/J81zo0r1E60KWJ\nFMgZxQIzq5+fA4gUFWbGjXXLMXVoW4bfdR0HjmXSY9wiHpq0lM0H1HRQQpevZxQbgFpAGnAaMMA5\n567zb3m/pjMKCRWnss4wacEORs9OJf10Dnc1qsywzslULHlVoEuTIsjvvZ7MrOpvPe6c25mfg14O\nBYWEmp/Ssxg9O5V/LtiJGTzQuhqD2tciPlZNB6XgFEhTQDNrALT1bM51zq3OzwEvl4JCQtWenzJ4\ndcYW/r1qLyViInmsY036tqxGTKSaDor/+X2OwsyGAu8DZT1f75nZkPwcUKSoqnx1LK/2aMg3Q9rS\nMLEkf/92Eze8PJtPl+/hjJoOShDz9aOnNUBL51y6Z7sYsFBzFCL5tyD1CC9O28SaPceoUz6OZ7vU\noUPtBDUdFL8oiFVPBpzJs33G85iI5FOrWmX4fFBrRva6nlPZZ+g3aSk9317Eqt0/B7o0kf8QcfFd\nAJgILDazf3u2bwfe8U9JIkVHWJhx23UVualueT5auos3Zm7l9lHzubV+BZ66uTbVyxQLdIkilzSZ\n3Qho49mc65xb6beqvNBHT1KYnTydw7g52xk/dztZOWfp2awKj3dKIiEuOtClSYjz26onMyvhnDtu\nZqV+63nn3NH8HPRyKCikKDh0IpM3v9/Kh0t2Ex0RRv+2NejfrgbFo339EEDkP/kzKL52zt1mZmlA\n3h1/ueCuRn4OejkUFFKUbD98kpdnbObbtQcoUzyKxzsl0bNZFTUdlEtWINdRBAsFhRRFK3f9xItT\nN7E47SjVSsfy1M21ubV+Ba2QEp8VxHUUrT1LYjGzPmb2qplVyc8BReTSXV/laj4a0IKJDzQlOiKc\nwR+s5PZR81mw7UigS5MiwNfz1zFAhufq7D8A24DJfqtKRH7FzOhYpyzfDm3LS92v49CJ0/R6ezEP\nTFzCxv3HA12eFGK+BkWOy/2Mqhsw0jk3CojzX1kiciHhYcbdTRKZ9VQHnu9ahxU7f+KWN+fy5Cer\n2PvzqUCXJ4WQr0FxwsyeB/oA35hZGKCOZiIBFBMZziPtazL3mRsY0LYGX6/ZT8eXZ/O3bzbwc0ZW\noMuTQsTXoOhBbnvxh5xzB8i9//VLfqtKRHwWHxvJ87dcw+ynOvD7BhUZPy+NtsNnMWb2NjKzz1z8\nDUQuQqueRAqZTQeOM3zaZn7YdIjyJWJ4snMydzWuTHiYVkgVZX5b9WRm8zx/njCz43m+TpiZZs9E\nglCd8iWY8EBTPhrQgnLxMTzz2Rq6vjGHmRsO6j7eki86oxApxJxzTF13gJembybtSDrNqpXiuVvq\n0KjK1YEuTQpYQVxH0cLM4vJsx5lZ8/wcUEQKjplxS/0KzBjWjhdur8f2I+ncOXoBAycvZ9vhk4Eu\nT0KEr/ejWAk08iyRxbPqaZlzrpGf6/sVnVGI5F/66RzGz01j3JxtZOacpUfTRJ7olETZEjGBLk38\nrEDuR+HyJIpz7iy+tygXkSBRLDqCoTcmkfJMR/o0r8InS3fT/qXZvDJjMycyswNdngQpX4Niu5k9\nbmaRnq+hwHZ/FiYi/lOmeDT/3a0eM59sT6dryjLih1TavzSbifPTyMo5G+jyJMj4GhQDgVbAXmAP\n0BwY4K+iRKRgVCtTjJG9GvHFY62pXS6O//5qA51enc0Xq/ZyVvfxFg+tehIRIHeFVMqWw7w4dROb\nDpygXqUSPNflGtoklQl0aXIFFMSqp2Qz+97M1nm2rzOzP+bngCISnMyMDrXL8u3jbXn1ngb8lJ5N\nn3cWc987i1m391igy5MA8vWjp7eB54FsAOfcGuBefxUlIoETFmbc2agy3/+hPX+89RrW7j3GbSPm\n8cRHK9l9NCPQ5UkA+BoUsc65Jec9lnOlixGR4BETGc7DbWuQ8nRHHu1Qk6nrDtDplRT+56sNHE1X\n08GixNegOGJmNfHcDtXMugP7/VaViASN+KsiebZLHWY/3YE7rq/EpAVptB8+i1GzUjmVpaaDRYGv\nF9zVAMaRu/LpJyAN6O2c2+nf8n5Nk9kigbXl4AmGT9vMzI0HKRsXzbDOydzduDIRuo93UPPrPbM9\nV2F3d8594rkdaphz7kR+DnYlKChEgsOStKO8OHUjK3b9TM2EYjzTpQ431S2n+3gHKb+uevJchf2M\n5+/pgQwJEQkezaqX4rNHWzG2T2Mc8Mjk5XQfu5BlO44GujS5wnw9V5xpZk+ZWaKZlfrly6+ViUjQ\nMzO61CvPjCfa8fc76rPraAbdxy6k/7vLSD2k3ykLC1/nKNLwTGTn5Zyr4Y+ivNFHTyLBKyMrhwnz\n0hibsp2MrBzuaZLIEzcmUz5eTQcDrSCaAtYFRgGrgVXACOBaHwrrYmabzSzVzJ7zsl9TM8vxrKYS\nkRAVGxXB4BuSSHm6A/e3qsZnK/bQ4eVZDJ+2ieNqOhiyfD2j+AQ4DrzveagXEO+cu8fLa8KBLUBn\ncvtDLQV6Ouc2/MZ+3wGZwATn3KfeatEZhUjo2PVjBq98t5kvVu2jZGwkgzvW4r6WVYmOCA90aUVO\nQZxR1HPOPeycm+X56g/Uu8hrmgGpzrntzrks4COg22/sNwT4DDjkc9UiEhKqlI7ljXuv5+shbahX\nMZ6/frORG15O4d8r96jpYAjxNShWmFmLXzY8d7e72K/1lYDdebb3eB77/8ysEnAHMMbbG5nZADNb\nZmbLDh8+7GPJIhIs6lWK572HmzP5oWaUjI1k2MeruXXEPFK2HNZ9vEOAr0HRGFhgZjvMbAewEGhq\nZmvNbM1lHP914FnPEtwLcs6Nc841cc41SUhIuIzDiUggtU1K4KvBbXjj3oacyMzm/glL6PPOYtbu\nUdPBYObrXeq65OO99wKJebYrex7LqwnwkecCnTLALWaW45z7PB/HE5EQEBZmdGtYiS71yvP+ol2M\n+GErvxs5j981qMjTN9WmSunYQJco5/Hb/SjMLILcyexO5AbEUqCXc279BfafBHytyWyRouV4Zjbj\nUrYzft52zpx19G5elSE31KJ08ehAl1aoFMRk9iVzzuUAg4HpwEbgE+fcejMbaGYD/XVcEQktJWIi\neerm2qQ83ZHujROZvGgn7V+azZvfbyUjS02qg4HucCciQSX10EmGT9vEjA0HSYiLZminJHo0TSRS\nTQcvS1CeUYiI5EetssUZ17cJnz3akqqlYvnj5+u4+bU5TF27XyukAkRBISJBqXHVUvxrYEve7tuE\nsDDj0fdXcOeYBSxJU9PBgqagEJGgZWZ0rluOaUPb8o+76rPv51Pc89ZCHpq0lC0H1XSwoGiOQkRC\nxqmsM0xckMaYWdtIz8rhrkaVefKmZCrEXxXo0oKeX29cFGwUFCLyU3oWI2elMnnhTszggdbVGNS+\nFvGxkYEuLWgpKESkSNp9NINXv9vC56v2UiImksc61qRvy2rERKrp4Pm06klEiqTEUrG81qMhXw9p\nQ4PEkvz92010eiWFz5bv4YyaDl4xCgoRCXnXVozn3Qeb8f7DzSlVLIo//Gs1t745l1mbD2lJ7RWg\noBCRQqN1rTJ88VhrRvS8noysM/SbuJSeby9i9e6fA11aSFNQiEihEhZm/K5BRWY+2Z6//K4uWw6e\npNuo+Tz2/gp2HEkPdHkhSZPZIlKoncjM5u0523l7bhrZZ87Ss1kVHu+UREJc0Wo6qFVPIiIXcehE\nJm/M3MpHS3cTExFG/3Y1eLhtDYpH+3q3hdCmoBAR8dH2wyd5afpmpq47QJniUQztlMS9zaoU+qaD\nWh4rIuKjGgnFGdOnMVMGtaJGQnH+9MV6Or+awjdr1HTwQhQUIlIkNapyNR8PaME79zchKiKMxz5Y\nwe2j5rNw24+BLi3oKChEpMgyMzpdU46pQ9sxvPt1HDpxmp5vL+KBiUvYuP94oMsLGpqjEBHxyMw+\nw6QFOxg9K5UTp3O48/rcpoOVSoZ+00FNZouIXEE/Z2QxevY2Ji3YAcADraoxqENNSsZGBbawy6Cg\nEBHxg70/n+LVGVuYsnIPcdERDOpYiwdahWbTQQWFiIgfbTpwnH9M3cSszYepEB/DsM7J3NWoMuFh\nFujSfKblsSIiflSnfAkm9mvGh/1bULZEDM98uoaub8zh+40Hi8SSWgWFiIiPWtYszeeDWjG6dyOy\nzzge+ucyeoxbxIpdPwW6NL9SUIiIXAIz45b6FZgxrB0v3F6P7YfTuXP0Ah59bznbDp8MdHl+oTkK\nEZHLkH46h/Fz0xg3ZxuZOWe5t2kiQzslUbZETKBL+w+azBYRCbDDJ04z4oetfLB4F5HhYfRvW53+\n7WoQFxMc9/FWUIiIBIkdR9J5acZmvlmzn9LFohhyQy16Na9KVERgP+nXqicRkSBRrUwxRvVqxBeP\ntSa5XBx/+WoDN76awper93E2RO/jraAQEfGDBokl+aB/cyb1a0psVDiPf7iSbqPmMz/1SKBLu2QK\nChERPzEzOtQuyzePt+XVexpwND2L3uMX03fCEtbvOxbo8nymOQoRkQKSmX2GyQt3MnJWKsczs7m9\nYSWe7JxMYqlYvx9bk9kiIiHk2KlsxszexsT5aTgH97WsyuCOtbi6mP+aDiooRERC0P5jp3jtuy18\nunwPxaIiGNihJg+2rs5VUVe+6aCCQkQkhG05eILh0zYxc+MhypWIZtiNyXRvXJmIK3gfby2PFREJ\nYcnl4hh/f1M+eaQlFUtexXNT1tLljbnMWH8gKJoO+jUozKyLmW02s1Qze+43nu9tZmvMbK2ZLTCz\nBv6sR0QkmDWrXoopj7ZibJ9GnD3rGDB5OXePXcjynUcDWpffgsLMwoFRQFegLtDTzOqet1sa0N45\nVx94ARjnr3pEREKBmdGlXm7Twb/dUY+dRzO4a8xCBry7jNRDgWk66M8zimZAqnNuu3MuC/gI6JZ3\nB+fcAufcL/15FwGV/ViPiEjIiAgPo3fzqqQ83YE/dE5mwbYfuem1FJ6fsoaDxzMLtBZ/BkUlYHee\n7T2exy7kIWDqbz1hZgPMbJmZLTt8+PAVLFFEJLjFRkUwpFMSKU93oG/Lany6fA/tX5rFS9M3cTwz\nu0BqCIrJbDPrSG5QPPtbzzvnxjnnmjjnmiQkJBRscSIiQaB08Wj+8vtr+f7JDtxUtzyjZm2j/fBZ\nvDMvjdM5Z/x6bH8GxV4gMc92Zc9j/8HMrgPGA92ccz/6sR4RkZBXpXQsb/a8nq+HtOHaivG88PUG\nOr2Swucr9/qt6aA/g2IpkGRm1c0sCrgX+DLvDmZWBZgC3Oec2+LHWkRECpV6leJ57+HmvPtgM0rE\nRPLEx6u4bcQ85my58h/P+y0onHM5wGBgOrAR+MQ5t97MBprZQM9u/xcoDYw2s1VmpivpREQuQbvk\nBL4e0obXezTkeGY2fScsoc/4xazbe+WaDurKbBGRQuJ0zhneW7SLkT9s5aeMbH7foCJP3VSbKqVj\nL+vK7IgrXaiIiARGdEQ4D7Wpzt1NKvNWyjbemZfG1HX76d286mW9r4JCRKSQKRETydM316Fvy2q8\nPnML7y6xNmoIAAAGP0lEQVTccVnvFxTLY0VE5MorVyKG/73zOmYMa3dZ76OgEBEp5GqVjbus1yso\nRETEKwWFiIh4paAQERGvFBQiIuKVgkJERLxSUIiIiFcKChER8UpBISIiXikoRETEKwWFiIh4paAQ\nERGvFBQiIuKVgkJERLxSUIiIiFcKChER8UpBISIiXikoRETEKwWFiIh4paAQERGvFBQiIuKVgkJE\nRLxSUIiIiFcKChER8UpBISIiXikoRETEKwWFiIh4paAQERGvFBQiIuKVgkJERLxSUIiIiFcKChER\n8cqvQWFmXcxss5mlmtlzv/G8mdmbnufXmFkjf9YjIiKXzm9BYWbhwCigK1AX6Glmdc/brSuQ5Pka\nAIzxVz0iIpI//jyjaAakOue2O+eygI+Abuft0w141+VaBJQ0swp+rElERC5RhB/fuxKwO8/2HqC5\nD/tUAvbn3cnMBpB7xgFw2szWXdlSQ1YZ4EigiwgSGotzNBbnaCzOqZ3fF/ozKK4Y59w4YByAmS1z\nzjUJcElBQWNxjsbiHI3FORqLc8xsWX5f68+PnvYCiXm2K3seu9R9REQkgPwZFEuBJDOrbmZRwL3A\nl+ft8yXQ17P6qQVwzDm3//w3EhGRwPHbR0/OuRwzGwxMB8KBCc659WY20PP8WOBb4BYgFcgA+vnw\n1uP8VHIo0lico7E4R2NxjsbinHyPhTnnrmQhIiJSyOjKbBER8UpBISIiXgVtUKj9xzk+jEVvzxis\nNbMFZtYgEHUWhIuNRZ79mppZjpl1L8j6CpIvY2FmHcxslZmtN7OUgq6xoPjwfyTezL4ys9WesfBl\nPjTkmNkEMzt0oWvN8v1z0zkXdF/kTn5vA2oAUcBqoO55+9wCTAUMaAEsDnTdARyLVsDVnr93Lcpj\nkWe/H8hdLNE90HUH8N9FSWADUMWzXTbQdQdwLP4L+Ifn7wnAUSAq0LX7YSzaAY2AdRd4Pl8/N4P1\njELtP8656Fg45xY4537ybC4i93qUwsiXfxcAQ4DPgEMFWVwB82UsegFTnHO7AJxzhXU8fBkLB8SZ\nmQHFyQ2KnIIt0/+cc3PI/d4uJF8/N4M1KC7U2uNS9ykMLvX7fIjc3xgKo4uOhZlVAu6g8DeY9OXf\nRTJwtZnNNrPlZta3wKorWL6MxUjgGmAfsBYY6pw7WzDlBZV8/dwMiRYe4hsz60huULQJdC0B9Drw\nrHPubO4vj0VaBNAY6ARcBSw0s0XOuS2BLSsgbgZWATcANYHvzGyuc+54YMsKDcEaFGr/cY5P36eZ\nXQeMB7o6534soNoKmi9j0QT4yBMSZYBbzCzHOfd5wZRYYHwZiz3Aj865dCDdzOYADYDCFhS+jEU/\n4EWX+0F9qpmlAXWAJQVTYtDI18/NYP3oSe0/zrnoWJhZFWAKcF8h/23xomPhnKvunKvmnKsGfAoM\nKoQhAb79H/kCaGNmEWYWS2735o0FXGdB8GUsdpF7ZoWZlSO3k+r2Aq0yOOTr52ZQnlE4/7X/CDk+\njsX/BUoDoz2/See4Qtgx08exKBJ8GQvn3EYzmwasAc4C451zha5Fv4//Ll4AJpnZWnJX/DzrnCt0\n7cfN7EOgA1DGzPYAfwYi4fJ+bqqFh4iIeBWsHz2JiEiQUFCIiIhXCgoREfFKQSEiIl4pKERExCsF\nhUgBMrNqv3T29HR2/TrQNYlcjIJCxAeeC5T0/0WKJP3DF7kAz2//m83sXWAdcJ+ZLTSzFWb2LzMr\n7tmvqec+IKvNbImZxXleO9ez7wozaxXY70Yk/4LyymyRIJIE3E/ulaxTgBudc+lm9izwpJm9CHwM\n9HDOLTWzEsApclucd3bOZZpZEvAhuX2oREKOgkLEu53OuUVmdhtQF5jvaZMSBSwkt2fQfufcUoBf\nupGaWTFgpJk1BM6Q2/JbJCQpKES8S/f8acB3zrmeeZ80s/oXeN0w4CC53VrDgEy/VSjiZ5qjEPHN\nIqC1mdWC3DMGM0sGNgMVzKyp5/E4M4sA4sk90zgL3EduszqRkKSgEPGBc+4w8ADwoZmtIfdjpzqe\nW2/2AEaY2WrgOyAGGA3c73msDufOTERCjrrHioiIVzqjEBERrxQUIiLilYJCRES8UlCIiIhXCgoR\nEfFKQSEiIl4pKERExKv/B25slrOyY5ekAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x43bfe80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.5,.5,.5,.5,.5,.5])\n",
    "pr(y_true, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier B predicts identical scores for all instances except the first\n",
    "Classifier B is nearly identical to classifier A but the scikit-learn auPRC is much worse.  One of the predicted scores is slightly larger, breaking the tie.  The point (0,0) is introduced into the precision recall curve so the linear interpolation is from 0 to 1/6.  The `average_precision_score` again returns 1/6 despite the different classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.16666667  0.          1.        ]\n",
      "recall: [ 1.  0.  0.]\n",
      "thresholds: [ 0.5   0.51]\n",
      "Naive auPRC: 0.0833333333333\n",
      "average_precision_score auPRC: 0.166666666667\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFe5JREFUeJzt3X2MXXed3/H3x48zSTzjBMcPsZNNgJjghYSC8yCIKJRF\nmyCqdFUkAiwsEVIULVDUBzVsVe1KRapAVbdbSCByaUrpA6naTZfsancp3ZZl25AlDiVPoETeUBLn\nyYRQ23E8dsb+9o977Lkzto+vJz4zd8bvlzTynHPPufc7R+Pzmd/5nfP7paqQJOlElsx3AZKk4WZQ\nSJJaGRSSpFYGhSSplUEhSWplUEiSWnUWFEnuTLIrySMneD1JvphkR5KHkry1q1okSbPXZYvia8B1\nLa9fD1zafN0MfKXDWiRJs9RZUFTVd4EXWza5Afh69dwHrE6yoat6JEmzs2weP3sj8FTf8s5m3bMz\nN0xyM71WB0tGV73tr23ZPCcFStJi8cADD7xQVefPZt/5DIqBVdU2YBvA6AWba/v27fNckSQtLEl+\nOtt95/Oup6eBC/uWNzXrJElDZD6D4h7gY83dT9cAu6vqmMtOkqT51dmlpyTfAN4FrEmyE/gdYDlA\nVd0B/DHwPmAH8DJwU1e1SJJmr7OgqKoPneT1Aj7Z1edLkk4Pn8yWJLUyKCRJrQwKSVIrg0KS1Mqg\nkCS1MigkSa0MCklSK4NCktTKoJAktTIoJEmtDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1Mqg\nkCS1MigkSa0MCklSK4NCktTKoJAktTIoJEmtDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1Mqg\nkCS1MigkSa0MCklSK4NCktTKoJAktTIoJEmtDApJUqtOgyLJdUkeS7IjyWeP8/p4kj9M8mCSR5Pc\n1GU9kqRT11lQJFkK3A5cD2wBPpRky4zNPgn8qKquAN4F/PMkK7qqSZJ06rpsUVwF7KiqJ6rqIHAX\ncMOMbQpYlSTAOcCLwGSHNUmSTlGXQbEReKpveWezrt9twBuBZ4CHgc9U1eGZb5Tk5iTbk2yvqq7q\nlSQdx3x3Zv8q8EPgAuAtwG1JxmZuVFXbqmprVW3tNT4kSXOly6B4Griwb3lTs67fTcDd1bMD+Alw\nWYc1SZJOUZdBcT9waZJLmg7qG4F7ZmzzJPAegCTrgDcAT3RYkyTpFC3r6o2rajLJp4BvAUuBO6vq\n0SS3NK/fAXwO+FqSh4EAt1bVC13VJEk6dVloncOjF2yu/c88Pt9lSNKCkuSBqto6m33nuzNbkjTk\nDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1MqgkCS1MigkSa0MCklSK4NCktTKoJAktTIoJEmt\nDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1MqgkCS1MigkSa0MCklSK4NCktTKoJAktTIoJEmt\nDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1MqgkCS1MigkSa0MCklSq06DIsl1SR5LsiPJZ0+w\nzbuS/DDJo0n+vMt6JEmnbllXb5xkKXA78F5gJ3B/knuq6kd926wGvgxcV1VPJlnbVT2SpNnpskVx\nFbCjqp6oqoPAXcANM7b5MHB3VT0JUFW7OqxHkjQLXQbFRuCpvuWdzbp+m4Fzk3wnyQNJPna8N0py\nc5LtSbZXVUflSpKOp7NLT6fw+W8D3gOMAt9Lcl9VPd6/UVVtA7YBjF6w2aSQpDnUZVA8DVzYt7yp\nWddvJ/DzqtoH7EvyXeAK4HEkSUNhoKBIshL428DF/ftU1T9p2e1+4NIkl9ALiBvp9Un0+yZwW5Jl\nwArgauBfDFq8JKl7g7YovgnsBh4ADgyyQ1VNJvkU8C1gKXBnVT2a5Jbm9Tuq6sdJ/hR4CDgMfLWq\nHjnVH0KS1J0M0jmc5JGqetMc1HNSoxdsrv3PeGVKkk5Fkgeqauts9h30rqd7k7x5Nh8gSVrYBr30\ndC3w8SQ/oXfpKUBV1eWdVSZJGgqDBsX1nVYhSRpaA116qqqfAquBv9l8rW7WSZIWuYGCIslngP8A\nrG2+/n2ST3dZmCRpOAx66ekTwNXNg3Ek+QLwPeBLXRUmSRoOg971FOBQ3/KhZp0kaZEbtEXxb4C/\nTPJfm+W/BfzrbkqSJA2TgYKiqn43yXfo3SYLcFNV/Z/OqpIkDY3WoEgyVlV7kpwH/N/m68hr51XV\ni92WJ0mabydrUfxH4P30xnjqH+sjzfJrO6pLkjQkWoOiqt7f/HvJ3JQjSRo2gz5H8Y4kZzff/3qS\n301yUbelSZKGwaC3x34FeDnJFcDfB/4K+HedVSVJGhqDBsVk9cYjvwG4rapuB1Z1V5YkaVgM+hzF\n3iS/Bfw68M4kS4Dl3ZUlSRoWg7YoPkhvePFPVNVz9Oa//medVSVJGhoDzXA3TJzhTpJO3auZ4e5k\nD9z9r6q6NslejvMcRVWNzeZDJUkLx8meo7i2+deOa0k6Qw36HMU1SVb1La9KcnV3ZUmShsWpPEfx\nUt/yvmadJGmRG3g+iurr9a6qwwx+a60kaQEbNCieSPJ3kixvvj4DPNFlYZKk4TBoUNwCvB14GtgJ\nXA3c3FVRkqThMejERbuAGzuuRZI0hAa962lzkj9L8kizfHmSf9xtaZKkYTDopad/BfwW8ApAVT2E\nLQxJOiMMGhRnVdX3Z6ybPN3FSJKGz6BB8UKS19EM45HkA8CznVUlSRoagz4L8UlgG3BZkqeBnwAf\n6awqSdLQOGlQNHNPbK2qX2mmQ11SVXu7L02SNAxOeumpeQr7Hzbf7zMkJOnMMmgfxX9P8g+SXJjk\nvCNfnVYmSRoKg/ZRfJBeR/Zvzlj/2tNbjiRp2AzaotgC3A48CPwQ+BLwyyfbKcl1SR5LsiPJZ1u2\nuzLJZHM3lSRpiAwaFP8WeCPwRXohsaVZd0JJltILl+ub7T+UZMsJtvsC8N8GL1uSNFcGvfT0pqrq\nP8n/zyQ/Osk+VwE7quoJgCR3ATcAM/f7NPD7wJUD1iJJmkODtih+kOSaIwvN7HbbT7LPRuCpvuWd\nzbqjkmwEfo2TTIKU5OYk25Ns75sWQ5I0BwZtUbwNuDfJk83yRcBjSR4Gqqoun+Xn/x5wa1UdTnLC\njapqG70H/hi9YLNJIUlzaNCguG4W7/00cGHf8qZmXb+twF1NSKwB3pdksqr+YBafJ0nqwKDzUfx0\nFu99P3BpkkvoBcSNwIdnvO8lR75P8jXgjwwJSRounc17XVWTST4FfAtYCtxZVY8muaV5/Y6uPluS\ndPpkoXUOj16wufY/8/h8lyFJC0qSB6pq62z2HfSuJ0nSGcqgkCS1MigkSa0MCklSK4NCktTKoJAk\ntTIoJEmtDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1MqgkCS1MigkSa0MCklSK4NCktTKoJAk\ntTIoJEmtDApJUiuDQpLUyqCQJLUyKCRJrQwKSVIrg0KS1MqgkCS1MigkSa0MCklSK4NCktTKoJAk\ntTIoJEmtDApJUiuDQpLUyqCQJLXqNCiSXJfksSQ7knz2OK9/JMlDSR5Ocm+SK7qsR5J06joLiiRL\ngduB64EtwIeSbJmx2U+Av15VbwY+B2zrqh5J0ux02aK4CthRVU9U1UHgLuCG/g2q6t6q+kWzeB+w\nqcN6JEmz0GVQbASe6lve2aw7kU8Af3K8F5LcnGR7ku1VdRpLlCSdzFB0Zid5N72guPV4r1fVtqra\nWlVbk8xtcZJ0hlvW4Xs/DVzYt7ypWTdNksuBrwLXV9XPO6xHkjQLXbYo7gcuTXJJkhXAjcA9/Rsk\nuQi4G/hoVT3eYS2SpFnqrEVRVZNJPgV8C1gK3FlVjya5pXn9DuC3gdcAX24uKU1W1dauapIknbos\ntM7h0Qs21/5nbHxI0qlI8sBs/xDvso9CkjTP9h88xHN7Jl7VexgUkrQAVRV7JiZ5fs8Ez+6e4Lnd\n+3l290Tf8gTP7Zng/738yqv+LINCkobM4cPFiy8f5LndzUl/z1QQPHd0eYKXDx46Zt8156xkw/gI\nm849iysvPo/14yOsHxvhA1+YfT0GhSTNoclDh9m198CMv/7389yeA9NaBa8cmt5/vGxJWDc2wrqx\nlbxx/RjvfsNa1o+NsH58hA3jvX/XrhphxbLTfzOrQSFJp8nEK4emXfqZCoP9R1sCP9t7gMMz7iFa\nuWzJ0ZP9lRefx7qxqZP/+ub715yzkqVL5ueBY4NCkgawd+KVqUtBzUl/ZmvgF8fpD1g1sowN4yOs\nGxvhDetXsX58tBcCfa2B8dHlDPOoEwaFpDNaVfHivoPHdATPbA3sO25/wArWjY2wcfUIb/ul1c3J\nf3Raa+DslQv/NLvwfwJJOoHJQ4f52UsHei2AaSf/ZnnPfp7fc4CDk4en7bd0SVi7aiXrx3utgHdu\nPr85+U+1BtaOrWTlsqXz9JPNLYNC0oI08cohdu050PuLf8/EMZeFnts9wa69E8f0B6w40h8wNsJb\nLzq3d/mnuQx0JAjWzGN/wDAyKCQNnZcOTE6/HXT3BM/umeD5vttFX9x38Jj9Vq1cxrrmuv+la9f0\n+gaO3BU0Nsr68RHOPWu4+wOGkUEhac5UFb94+ZXmr/7pQXCkc/j53RPsPTB5zL7nnb3i6B1Ab7lo\n9dFWwIbxUdaPr2Td2AirRpbPw0+1+BkUkk6LQ4eLF146MHUnUNMKmNk3MLM/YElg7areSf/155/D\nta9fM+PW0FHWjq1kZPmZ0R8wjAwKSSd1YPJIf8D0p4T7O4Z37T3AoRkdAiuWLjl6wr9i02qu++WR\nac8IbBgfZc05K1i2dCjmUNMJGBTSGW7fgcnjPCU8vSXwwkvH9gecvWLp0ZP9O16/ZtpzAUfC4Lyz\nV9gfsAgYFNIiVVXs3v/KtOcCjtca2DtxbH/AuWctZ/34KOvHVnL5ptXHPCW8ftz+gDOJQSEtQIcO\nFz9/qf9S0PFbAwdm9Ack9J4PGBvhkjVn8/bXrZlxKajXGrA/QP0MCmnIHJw8zPN7jh0uunfy33+0\nP2ByRn/A8qU5etJ/86bVvHfLyqPPBRxZf/6qlSy3P0CnyKCQ5tDLByenPSU8sy/g2d0TvPDSgWP2\nG12+lA2re5d+rnnda44+Jdx/Kei8s1awxIfE1AGDQjoNqoo9+yd5tnk24OiDYTNaA3uO0x+w+qzl\nRzuC37RxrO9S0FRrYGxkmZ3CmjcGhXQShw8XL+w7wPO7pw8XMbNVsP+V6YPGJVOTyPzSa87i6tee\nNzV3QPOU8PqxEUZX2B+g4WZQ6Iz2SjOJTNtwEbv2nngSmQ3jI2y5YIz3XLa2GStoqjWw1v4ALRIG\nhRatI5PKP7t7//SO4b7hIl546QA1Y9C4keVLesNCjI1w9SXnHQ2AI08JrxtfyZqzV9ofoDOGQaEF\n59VMKj82sqwZG2iEN64fmzaNZG8U0VHGRu0PkPoZFBoqp3tS+f6ZxNaPj3DWCn/lpVPl/xrNmYU4\nqbwkg0KnyWKdVF6SQaEBnM5J5dfPGC5i2CeVl2RQnNGcVF7SIPyfvEh1Man8kdbAmTSpvCSDYkF6\ntZPKr3NSeUmnwKAYMqdrUvlpTwk7qbykV8GgmCNOKi9poTIoTgMnlZe0mBkUJ/FqJpVfN76SDWOj\nx51Ufv34COefs9JJ5SUNvTM6KE7HpPJvf92aac8FOKm8pMWm06BIch3wL4GlwFer6vMzXk/z+vuA\nl4GPV9UPXu3nOqm8JJ0+nQVFkqXA7cB7gZ3A/Unuqaof9W12PXBp83U18JXm31a79vSf/J1UXpK6\n1GWL4ipgR1U9AZDkLuAGoD8obgC+XlUF3JdkdZINVfXsid70cBVX/dM/m7bOSeUlqTtdBsVG4Km+\n5Z0c21o43jYbgWlBkeRm4OZm8cBPv/D+R2Z+2I5XW+3CtAZ4Yb6LGBIeiykeiykeiylvmO2OC6Iz\nu6q2AdsAkmyvqq3zXNJQ8FhM8VhM8VhM8VhMSbJ9tvt2eS3maeDCvuVNzbpT3UaSNI+6DIr7gUuT\nXJJkBXAjcM+Mbe4BPpaea4Ddbf0TkqS519mlp6qaTPIp4Fv0bo+9s6oeTXJL8/odwB/TuzV2B73b\nY28a4K23dVTyQuSxmOKxmOKxmOKxmDLrY5HeDUeSJB2f94tKkloZFJKkVkMbFEmuS/JYkh1JPnuc\n15Pki83rDyV563zUORcGOBYfaY7Bw0nuTXLFfNQ5F052LPq2uzLJZJIPzGV9c2mQY5HkXUl+mOTR\nJH8+1zXOlQH+j4wn+cMkDzbHYpD+0AUnyZ1JdiU55lmz5vXZnTeraui+6HV+/xXwWmAF8CCwZcY2\n7wP+BAhwDfCX8133PB6LtwPnNt9ffyYfi77t/ge9myU+MN91z+PvxWp6IyFc1Cyvne+65/FY/CPg\nC8335wMvAivmu/YOjsU7gbcCj5zg9VmdN4e1RXF0+I+qOggcGf6j39HhP6rqPmB1kg1zXegcOOmx\nqKp7q+oXzeJ99J5HWYwG+b0A+DTw+8CuuSxujg1yLD4M3F1VTwJU1WI9HoMciwJWNQORnkMvKI4d\nFXSBq6rv0vvZTmRW581hDYoTDe1xqtssBqf6c36C3l8Mi9FJj0WSjcCv0RtgcjEb5PdiM3Buku8k\neSDJx+asurk1yLG4DXgj8AzwMPCZqjrMmWdW580FMYSHBpPk3fSC4tr5rmUe/R5wa1Uddj4QlgFv\nA94DjALfS3JfVT0+v2XNi18Ffgj8DeB1wLeT/EVV7ZnfshaGYQ0Kh/+YMtDPmeRy4KvA9VX18zmq\nba4Nciy2Anc1IbEGeF+Syar6g7kpcc4Mcix2Aj+vqn3AviTfBa4AFltQDHIsbgI+X70L9TuS/AS4\nDPj+3JQ4NGZ13hzWS08O/zHlpMciyUXA3cBHF/lfiyc9FlV1SVVdXFUXA/8F+M1FGBIw2P+RbwLX\nJlmW5Cx6ozf/eI7rnAuDHIsn6bWsSLKO3kiqT8xplcNhVufNoWxRVHfDfyw4Ax6L3wZeA3y5+Ut6\nshbhiJkDHoszwiDHoqp+nORPgYeAw/RmmTzubZML2YC/F58DvpbkYXp3/NxaVYtu+PEk3wDeBaxJ\nshP4HWA5vLrzpkN4SJJaDeulJ0nSkDAoJEmtDApJUiuDQpLUyqCQJLUyKKQ5lOTiIyN7NiO7/tF8\n1ySdjEEhDaB5QMn/Lzoj+YsvnUDz1/9jSb4OPAJ8NMn3kvwgyX9Ock6z3ZXNPCAPJvl+klXNvn/R\nbPuDJG+f359Gmr2hfDJbGiKXAr9B70nWu4Ffqap9SW4F/l6SzwP/CfhgVd2fZAzYT2+I8/dW1USS\nS4Fv0BuHSlpwDAqp3U+r6r4k7we2AP+7GSZlBfA9emMGPVtV9wMcGY00ydnAbUneAhyiN+S3tCAZ\nFFK7fc2/Ab5dVR/qfzHJm0+w398Fnqc3WusSYKKzCqWO2UchDeY+4B1JXg+9FkOSzcBjwIYkVzbr\nVyVZBozTa2kcBj5Kb7A6aUEyKKQBVNXPgI8D30jyEL3LTpc1U29+EPhSkgeBbwMjwJeB32jWXcZU\ny0RacBw9VpLUyhaFJKmVQSFJamVQSJJaGRSSpFYGhSSplUEhSWplUEiSWv1/adVWjZ/lPkAAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x43d69e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.51,.5,.5,.5,.5,.5])\n",
    "pr(y_true, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier C flips the predictions from classifier B\n",
    "Classifier B performed worse than a random classifier by the naive auPRC, whose precision recall curve would be a horizontal line at 1/6, but the same as random when using `average_precision_score`.  We flip the predictions of classifier B to obtain classifer C.  Classifier C attains perfect recall after making 5 predictions (the tied predictions) so the linear interpolation is from 1 to 1/5.\n",
    "\n",
    "Classifier C has naive auPRC comparable to classifier A (slighly better, as expected), but arguably both A and C have inflated auPRC due to the linear interpolation in the naive calculation.  The `average_precision_score` is now 1/5 instead of 1/6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision: [ 0.2  1. ]\n",
      "recall: [ 1.  0.]\n",
      "thresholds: [-0.5]\n",
      "Naive auPRC: 0.6\n",
      "average_precision_score auPRC: 0.2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd0VXXe7/H3N43Qa2gJPQmKikjvHWwzg20UULExCA5F\nxVGfZ8195nnurHvvjAVRUBDRQRwFnbFOUekQ6UE6mpBCrwGkBELa7/6R45BBOBwCp+Tk81orS/c+\nv3P2N78Vzufss/f+bnPOISIicjERwS5ARERCm4JCRES8UlCIiIhXCgoREfFKQSEiIl4pKERExCu/\nBYWZvWNmh8xsy0UeNzN7zcwyzGyTmbX3Vy0iIlJ2/tyjmAXc4uXxW4Ekz88oYJofaxERkTLyW1A4\n55YBR70MGQLMdiVWAbXMrJG/6hERkbKJCuK244HdpZb3eNbtP3+gmY2iZK+DiMo1OlSr14iGNSpR\nPTY6IIWKiJR369aty3HOxZXlucEMCp8552YAMwAS27R1jR9+lV1HT3N9s9pMHJxM91b1glyhiEho\nM7OdZX1uMM962gs0KbWc4FnnVa0qMSyc2If/c+f17D12huFvreb+matYv+uY3woVEanIghkUXwAj\nPGc/dQWOO+d+8rXThURHRnB/l2Ys+U1ffnv7tXy//yR3vrGCke+u5bv9J/xbtYhIBWP+6h5rZnOA\nvkA94CDwOyAawDk33cwMmErJmVGngUecc6mXet2OHTu61NR/H5Z7tpA/Lc/mzWVZnMwr5GdtG/HU\noGRaxVW7qr+TiEh5ZWbrnHMdy/Tc8tZm/EJB8aPjpwuYkZLJn5bvIK+giLvbJzBhYBIJtasEuEoR\nkdCioDhPzqmzvLE4kz+v3olzjmGdmzK2XyL1a8QGqEoRkdCioLiI/cfP8NrCDP6SupuoSOOhbs0Z\n3acVtavG+LlKEZHQoqC4hJ1Hcpm8YDufbdhL1ZgoHuvZgpG9Wug6DBGpMBQUPko/eJJJ89L5ausB\nalWJZnSfVjzUrTmVYyKvcpUiIqFFQXGZNu85zkvz0liafpi46pUY2y+RoZ2bUClKgSEi4UlBUUZr\ndxzlxa/TWJN9lPhalZkwIIm72scTFanu6yISXq4kKCr0O2Kn5nX4cFRXZj/ambrVYnj2400MfmUZ\nX2zcR3Fx+QpQERF/qdBBAWBm9E6O4/Nf9+DNBzsQHRnB+Dnrue21FBZsO0h52+MSEbnaKnxQ/MjM\nuPm6hvxzQi9eHdqOvIIiRs5O5c43VrA8IyfY5YmIBI2C4jyREcaQdvHMf7oPf7jrBg6dyOP+masZ\nNmMV63Z6u72GiEh4qtAHs32RV1DEnDW7eH1xBjmn8unXOo6Jg1tzfXzNgNUgInKldNZTAJzOL2TW\nih28uTSL42cKuO2Ghjw9KJnE+tUDXouIyOVSUATQ8TMFvJ2SxdvfZHOmoIg7b0rgyYFJNKmjxoMi\nEroUFEFw5NRZpi/NZPbKnRQVO+7r1IRx/ZNoWFONB0Uk9CgogujA8TymLt7O3DW7iYwwRnRrxpi+\nidRR40ERCSEKihCw++hpJi/Yzqfr91A5OrKk8WDvltRQ40ERCQEKihCScegkr8zfzj8276dm5Wge\n79OSh7s3p0pMVLBLE5EKTEERgrbsPc6k+eks+v4Q9arF8ETfRIZ3aUpstBoPikjgKShC2LqdR3np\n63RWZh2hcc1Yxg9I4u4OCUSr8aCIBJCaAoawDs3qMGdUV94f2YX6NWJ5/pPNDJq0lM837FXjQREp\nFxQUAdIjsR6fPtGdmSM6EhsdyYS5G7j11RS+3npAjQdFJKQpKALIzBjYpgH/HN+LKcNuoqComMff\nW8cdry9nWfphBYaIhCQFRRBERBg/v7Ex857qzQv3tCXnVD4j3lnDfTNWsXaHGg+KSGjRwewQcLaw\niA/X7mbKogwOnzxLn+Q4nhncmhsS1HhQRK4OnfUUJs7kFzF75Q6mLc3kh9MF3HJdQ54enExyAzUe\nFJEro6AIMyfzCnj7m2xmpmSTm1/IHe3ieXJgEs3qVg12aSJSTikowtSx3HymL8vk3RU7KCxy/LJj\nE8YPSKRRzcrBLk1EyhkFRZg7dCKP1xdn8MGaXZgZD3RpxhP9WlGvWqVglyYi5YSCooLYc+w0ry3c\nzl/X7SE2OpJHejRnVK9W1KyixoMi4p2CooLJOnyKVxZs528b91EjNopRvVvySI8WVK2kxoMicmEK\nigrqu/0neHleOgu+O0jdqjGM6duKB7o2U+NBEfkJBUUFt37XMV6el843GTk0rBHLuAGJ3NuxiRoP\nisi/KCgEgJWZR3hpXhrrdh6jaZ0qPDkwiSHt4omMsGCXJiJBpu6xAkC3VnX56+hu/OnhTlSrFMXT\nH23klsnL+HLzfvWREpEyU1CEGTOj3zX1+fu4nrw+vD3FzjHm/W/5+dRvWJx2SIEhIpfNr0FhZreY\nWZqZZZjZ8xd4vKaZ/c3MNprZVjN7xJ/1VCQREcbtbRsx76k+vPzLGzl+poBH/rSWX05fyaqsI8Eu\nT0TKEb8dozCzSCAdGATsAdYCw5xz20qN+U+gpnPuOTOLA9KAhs65/Iu9ro5RlE1+YTEfpe5myqLt\nHDxxll5J9XhmcGtubFIr2KWJSACE6jGKzkCGcy7L88Y/Fxhy3hgHVDczA6oBR4FCP9ZUYcVERfBA\n12Ys/U0/fnv7tWzdd4Ihry/nV7NT+f7AiWCXJyIhzJ9BEQ/sLrW8x7OutKnAtcA+YDMwwTlXfP4L\nmdkoM0s1s9TDhw/7q94KITY6kpG9WrLs2X48PSiZVZlHuPXVFMbPWU92Tm6wyxOREBTsg9k3AxuA\nxkA7YKqZ1Th/kHNuhnOuo3OuY1xcXKBrDEvVKkUxfkASKc/1Y0yfVszfdpCBk5by3F83sfeHM8Eu\nT0RCiD+DYi/QpNRygmddaY8An7gSGUA2cI0fa5Lz1KoSw7O3XMOyZ/sxolszPl2/l34vLuG/v9jK\noZN5wS5PREKAP4NiLZBkZi3MLAYYCnxx3phdwAAAM2sAtAay/FiTXERc9Ur87ufXseQ3fbm7Qzzv\nrdpJnxeW8Icvv+eH0xc9t0BEKgC/XpltZrcBk4FI4B3n3P8xs9EAzrnpZtYYmAU0Agz4g3Puz95e\nU2c9BUZ2Ti6TF6TzxcZ9VIuJYmSvljzWqwXV1HhQpFxSCw/xm7QDJ5k0P42vtx6kdpVoxvRtxYhu\nzdV4UKScUVCI323c/QMvzUsjZXsO9atXYlz/RO7r1JSYqGCfDyEivlBQSMCszippPLh2xzESaldm\nwoAk7rwpnih1qhUJaaF6wZ2EoS4t6/LR492Y9UgnaleJ4Td/3cTNk5fx9037KC4uXx86RMQ3Cgq5\nbGZG39b1+WJsD6Y/0IHICGPsB+v52ZRvWPjdQTUeFAkzCgopMzPjlusb8uWE3ky+rx2nzhby2Lup\n3D1tBSsycoJdnohcJQoKuWKREcYdN8WzcGIf/u+dN7D/eB7DZ65m+Fur+HbXsWCXJyJXSAez5arL\nKyji/dW7eGNxBkdy8xlwTX0mDm5Nm8Y/6c4iIgGis54kJOWeLWTWih28uTSTE3mF3N62EU8PSqZV\nXLVglyZS4SgoJKQdP1PAW8uyeGd5NnkFRdzVPoEJA5JoUqdKsEsTqTAUFFIu5Jw6y7Qlmby3aifO\nOYZ1bsrYfonUrxEb7NJEwp6CQsqV/cfPMGVRBh+t3U1khPFQ9+aM7tOKOlVjgl2aSNhSUEi5tOvI\naSYvSOfTDXupGhPFoz1bMLJXC2rERge7NJGwo6CQcm37wZNMmp/Ol1sOUKtKNKP7tOKhbs2pHKPG\ngyJXi4JCwsKWvcd5aV4aS9IOE1e9EmP7JTK0cxMqRSkwRK6UgkLCSuqOo7z4dRqrs48SX6sy4wck\ncnf7BDUeFLkCagooYaVj8zrMHdWV9x7rTL1qMTz38WYGvbKMLzaq8aBIMCgoJCSZGb2S4vjs1z2Y\n8WAHKkVFMH7Oem57LYX529R4UCSQFBQS0syMwdc15J/je/Hq0HacLSzmV7NTueONFXyzPUeBIRIA\nCgopFyIijCHt4pn/VG/+ePcNHD6RxwNvr2bojFWk7jga7PJEwpoOZku5dLawiDmrdzF1cSY5p87S\nr3UcEwe35vr4msEuTSQk6awnqbBO5xfy7oqdTF+ayfEzBdx2Q0OeHpRMYv3qwS5NJKQoKKTCO5FX\nwMyUbN5OyeJMQRF33BTPkwOSaVpXjQdFQEEh8i9Hc/OZvjSTd1fsoKjYcV+nJozrn0TDmmo8KBWb\ngkLkPAdP5DF1UQZz1+4iwowHuzZjTN9W1K1WKdiliQSFgkLkInYfPc2rC7fzybd7qBwd6Wk82JKa\nldV4UCoWBYXIJWQcOsUrC9L5x6b91IiN4vE+rXi4e3OqVooKdmkiAaGgEPHR1n3HmTQvnYXfH6Je\ntRie6JvI8C5NiY1W40EJbwoKkcu0bucxXp6XxorMIzSqGcv4AUnc0yGBaDUelDClpoAil6lDs9p8\n8KuufDCyCw1rxvIfn2xm4KSlfLZ+L0VqPCjybxQUUqF1T6zHJ2O68/ZDHakSE8WTH27g1leX8dWW\nA+ojJeKhoJAKz8wYcG0D/jGuJ1OH30RhsWP0n9cx5PXlLE0/rMCQCk9BIeIREWH8rG1j5j3Zmxfv\nacvR3HweemcN9725ijXZajwoFZcOZotcRH5hMR+u3cWURRkcOnmW3slxPDM4mbYJtYJdmshl01lP\nIn50Jr+I91btYNqSTI6dLuDm6xowcXBrkhuo8aCUHyF71pOZ3WJmaWaWYWbPX2RMXzPbYGZbzWyp\nP+sRKYvKMZGM6t2KZc/246mByazIOMLNk5fx5Nz17MjJDXZ5In7ntz0KM4sE0oFBwB5gLTDMObet\n1JhawArgFufcLjOr75w75O11tUchwXYsN583l2Uxa0U2BUWOezsmMK5/Eo1rVQ52aSIXFap7FJ2B\nDOdclnMuH5gLDDlvzHDgE+fcLoBLhYRIKKhdNYbnb72GZc/248Guzfh43V76vriE//nbVg6fPBvs\n8kSuOn8GRTywu9TyHs+60pKB2ma2xMzWmdmIC72QmY0ys1QzSz18+LCfyhW5PPWrx/Lfv7iORc/0\n4c6b4pm9cie9X1jMC199z/HTBcEuT+SqCfbpsVFAB+B24Gbgf5lZ8vmDnHMznHMdnXMd4+LiAl2j\niFcJtavwx3vaMv+p3gxq04BpSzPp+cIipizczqmzhcEuT+SK+TMo9gJNSi0neNaVtgf42jmX65zL\nAZYBN/qxJhG/aRlXjdeG3cSXE3rRtWVdXp6fTu8XFjMzJYu8gqJglydSZj4dzDazSsDdQHNK9gIA\ncM79by/PiaLkYPYASgJiLTDcObe11JhrgamU7E3EAGuAoc65LRd7XR3MlvJiw+4feHleGinbc2hQ\noxLj+idxb8cmxEQFe0deKqJAHMz+nJID0YVAbqmfi3LOFQJjga+B74CPnHNbzWy0mY32jPkO+ArY\nRElIzPQWEiLlSbsmtXjvsS7MHdWVJrWr8NvPtjBg0hI+XrdHjQelXPF1j2KLc+76ANRzSdqjkPLI\nOceS9MO8PC+NLXtPkFi/Gk8PSuaW6xoSEWHBLk8qgEDsUawwsxvKsgERKWk82K91ff42tifT7m8P\nwBPvf8vPp37D4u8PqfGghDRf9yi2AYlANnAWMMA559r6t7yf0h6FhIOiYsfnG/YyecF2dh09Tcdm\ntZk4uDXdWtUNdmkSpvze68nMml1ovXNuZ1k2eiUUFBJOCoqK+Sh1N1MWZnDgRB49E+vxzM2taddE\njQfl6gpIU0AzuxHo5VlMcc5tLMsGr5SCQsJRXkERf161kzeWZHI0N5+B1zZg4uBkrm1UI9ilSZjw\n+zEKM5sAvA/U9/z82czGlWWDIvJTsdGRjOzVkmXP9mPioGRWZx/httdSGDdnPVmHTwW7PKngfP3q\naRPQzTmX61muCqzUMQoR/zh+uoAZKZn8afkOzhYWc3f7eMYPSCKhdpVglyblVCDOejKg9KWlRZ51\nIuIHNatE85ubSxoPPtStOZ9t2Ef/l5byu8+3cOhkXrDLkwrG1z2Kp4GHgE89q+4AZjnnJvuxtgvS\nHoVURPt+OMOURRl8lLqb6Ejjoe7NGd27FbWrxgS7NCknAnUwuz3Q07OY4pxbX5YNXikFhVRkO3Jy\nmbwgnc837qNaTBSP9WrBYz1bUD02OtilSYjzW1CYWQ3n3Akzq3Ohx51zAb/jvIJCBNIPnmTSvHS+\n2nqA2lWiGd2nFSO6NadyTGSwS5MQ5c+g+Ltz7mdmlg2UHvjjBXcty7LRK6GgEDln054feGleOsvS\nD1O/eiXG9k9kaKemajwoPxGQr55ChYJC5KfWZB/lpa/TWLPjKPG1KjNhYBJ33RRPVKQCQ0oE4jqK\nHp5TYjGzB8xskpk1LcsGReTq69yiDh8+3pV3H+1M3WoxPPvXTQyevIy/bdxHsTrVyhXy9ePGNOC0\n5+rsiUAm8J7fqhKRy2Zm9EmO4/Nf9+DNBzsQHRHBuDnruX3KNyzYdlCNB6XMfA2KQlfyVzYEmOqc\nex2o7r+yRKSszIybr2vIPyf04tWh7TidX8jI2ancNW0FKzJygl2elEO+BsVJM/sP4AHgH2YWAeh8\nPJEQFhlhDGkXz4Kn+/D/7rqBA8fzGD5zNcPfWsW6nceCXZ6UI74GxX2UtBd/zDl3gJL7X7/ot6pE\n5KqJjoxgWOemLH6mL//1szakHzzJ3dNW8OistWzddzzY5Uk5oLOeRCqY0/mFzFqxgzeXZnH8TAG3\nt23EUwOTSaxfLdiliR/58zqKb5xzPc3sJBe+jiLgPZAVFCJXx/EzBcxMyeKdb7I5U1DEXe0TmDAg\niSZ11HgwHOk6ChEpsyOnzjJtSSazV+3EOcfQTk0Z2z+RBjVig12aXEWBuMNdV2Crc+6kZ7k60MY5\nt7osG70SCgoR/zhwPI8pi7bz4drdREYYI7o1Y0zfROqo8WBYCERQrAfae06RxXPWU6pzrn1ZNnol\nFBQi/rXryGkmL0zns/V7qRwdyWO9WjKyVwtqqPFguRaQ+1G4UoninCsGosqyQREJbU3rVmHSve2Y\n91Rv+rSO47WF2+n1x8VMW5LJ6fzCYJcnQeBrUGSZ2Xgzi/b8TACy/FmYiARXYv3qvHF/B/4+rift\nm9bij199T+8XljBreTZnC4su/QISNnwNitFAd2AvsAfoAozyV1EiEjquj6/Jnx7pzMdjupFYvyr/\n/bdt9H9pKR+u3UVhUXGwy5MA0FlPIuIz5xzLM47w4rw0Nu7+gRb1qvLkwCR+3rYxERG6O3IoC0T3\n2GQzW2hmWzzLbc3st2XZoIiUX2ZGz6R6fPZEd94a0ZFKURFMmLuB215LYd7WA2o8GKZ8/erpLeA/\ngAIA59wmYKi/ihKR0GZmDGrTgH+O78Vrw24iv7CYUe+t447Xl7Ms/bACI8z4GhRVnHNrzlun0x9E\nKriICOMXNzZm3lO9eeHutuScymfEO2sYOmMVqTsCfqdk8RNfgyLHzFrhaeNhZvcA+/1WlYiUK1GR\nEdzbqQmLnunD//ziOrJycrln+koe/tMatuxV48HyztcL7loCMyg58+kYkA3c75zb6d/yfkoHs0VC\n35n8It5duYPpSzP54XQBt17fkKcHJZPUQLexCRa/XpntuQr7HufcR57boUb82MojGBQUIuXHibwC\n3k7J5u1vssnNL+TOdvE8OTCZpnXVeDDQAtHCI7WsG7jaFBQi5c+x3HymL83k3ZU7KCxy3NupCeP6\nJ9KoZuVgl1ZhBCIo/gDkAB8CuT+ud84F/GiVgkKk/Dp0Io+pizOYs2YXZsaDXZsxpm8r6lWrFOzS\nwl4ggiKbf78fBQDOuZZl2eiVUFCIlH+7j57mtYXb+fjbPcRGR/Jojxb8qndLalZW40F/CURTwDbA\n68BGYAMwBbjOh8JuMbM0M8sws+e9jOtkZoWes6lEJMw1qVOFF395I/Of7kP/a+ozdXEGvf64iNcX\nZ5B7Vmfehxpf9yg+Ak4A73tWDQdqOufu9fKcSCAdGERJf6i1wDDn3LYLjJsP5AHvOOf+6q0W7VGI\nhJ9t+04waX4aC747RL1qMYzpm8j9XZoSGx0Z7NLCxpXsUfjaKvx651ybUsuLzWzbRUeX6AxkOOey\nPEXOBYYA5z9vHPAx0MnHWkQkzLRpXIOZD3Xi213HeHleGr//+zZmpmQxrn8Sv+yYQHSkr19+iD/4\nOvvfeu5yB4CZdQEu9bE+HthdanmPZ92/mFk8cCcwzdsLmdkoM0s1s9TDhw/7WLKIlDftm9bm/ZFd\n+eBXXWhUM5b//HQzAyct5dP1eygqVluQYPE1KDoAK8xsh5ntAFYCncxss5ltuoLtTwae89wI6aKc\nczOccx2dcx3j4uKuYHMiUh50b1WPj8d0552HO1I1JoqnPtzIra8u46st+9VHKgh8/erpljK89l6g\nSanlBM+60joCc80MoB5wm5kVOuc+K8P2RCSMmBn9r2lA3+T6fLnlAJPmpzH6z99yQ3xNJg5Opk9y\nHJ73DvEzv92PwsyiKDmYPYCSgFgLDHfObb3I+FnA33UwW0QupLComM827GPygnT2HDtDp+a1eWZw\na7q0rBvs0sqFQJwee9mcc4XAWOBr4DvgI+fcVjMbbWaj/bVdEQlPUZER3NMhgUUT+/L7O65n55HT\n3DdjFQ++vZqNu38IdnlhTXe4E5FyKa+giPdW7uSNJRkcO13A4DYNmDi4Na0bqvHghfj9yuxQoqAQ\nkdJOnS3knW+yeWtZFqfyC/nFjY15amAyzetVDXZpIUVBISIV3g+n83lzWRazlu8gv6iYX3ZIYPyA\nJBrXUuNBUFCIiPzLoZN5vLE4kw9W7wJgeJem/LpfInHVK3bjQQWFiMh59v5whikLt/OXdXuIiYzg\n4R7Nebx3S2pViQl2aUGhoBARuYjsnFwmL0jni437qBYTxa96t+TRni2oVsnXy8jCg4JCROQSvj9w\ngknz0pm37SB1qsYwpk8rHuzWrMI0HlRQiIj4aOPuH3hpXhop23NoUKMSY/sncV/HJsREhXfjQQWF\niMhlWp11hJfmpbF2xzGa1KnMhAHJ3HlTPJER4dkWJCSvzBYRCWVdWtblo8e7MeuRTtSqHMMzf9nI\n4FeW8o9N+ylWp9p/o6AQkQrLzOjbuj5fjO3B9AfaE2HGrz/4lp9P/YZF3x9Up1oPBYWIVHhmxi3X\nN+KrJ3vzyn03cjKvkEdnpXLP9JWsyMwJdnlBp2MUIiLnKSgq5i+pe3ht4XYOnMijR2Jdnhncmpua\n1g52aWWmg9kiIn6QV1DE+6t38cbiDI7k5jPw2vpMHNyaaxvVCHZpl01BISLiR7lnC5m1YgdvLs3k\nRF4hP2vbiKcGJdMqrlqwS/OZgkJEJACOny7grZQs3lmeTV5BEXe3T2DCwCQSalcJdmmXpKAQEQmg\nnFNnmbYkk/dW7cQ5x7DOTRnbL5H6NWKDXdpFKShERIJg//EzTFmUwUdrdxMVaTzUrTmj+7SidtXQ\nazyooBARCaKdR3J5dcF2Pt2wl6oxUTzWswUje7Wgemx0sEv7FwWFiEgI2H7wJJPmp/PllgPUqhLN\nmD6tGNGtOZVjgt94UEEhIhJCNu85zsvz01iSdpi46pUY1z+R+zo1oVJU8AJDQSEiEoLW7jjKi1+n\nsSb7KPG1KjNhQBJ3tY8nKjLwTTHUFFBEJAR1al6HD0d1ZfajnalXLYZnP97E4FeW8cXGfeWq8aCC\nQkTEj8yM3slxfPbrHsx4sAPRkRGMn7Oe215LYcG28tF4UEEhIhIAZsbg6xry5YRevDq0HXkFRYyc\nncqdb6xgeUZoNx5UUIiIBFBEhDGkXTzzn+7DH+66gUMn8rh/5mqGzVjFup1Hg13eBelgtohIEJ0t\nLOKD1bt4fXEmOafO0v+a+kwcnMx1jWte1e3orCcRkXLudH4h767YyfSlmRw/U8DtN5Q0Hkysf3Ua\nDyooRETCxIm8AmamZPN2ShZnCoq486YEnhyYRJM6V9Z4UEEhIhJmjpw6y/SlmcxeuZOiYsd9nZow\nrn8SDWuWrfGggkJEJEwdPJHHlEXb+XDtbiLMGNGtGWP6JlLnMhsPKihERMLc7qOnmbxgO5+u30Pl\n6MiSxoO9W1LDx8aDCgoRkQoi49BJXpm/nX9s3k/NytE83qclD3dvTpWYKK/PU1CIiFQwW/YeZ9L8\ndBZ9f4h61Srx636tGN6l6UUbDyooREQqqHU7j/HS12mszDpC45qxjB+QxN0dEog+r/FgyDYFNLNb\nzCzNzDLM7PkLPH6/mW0ys81mtsLMbvRnPSIi4aZDs9rMGdWV90d2oX6NWJ7/ZDODJi3l8w17r1rj\nQb8FhZlFAq8DtwJtgGFm1ua8YdlAH+fcDcDvgRn+qkdEJJz1SKzHp090Z+aIjsRGRzJh7gZufTWF\nr7ceuOLGg96PflyZzkCGcy4LwMzmAkOAbT8OcM6tKDV+FZDgx3pERMKamTGwTQP6X1Off2zezyvz\n03n8vXXcmHBl7UD8+dVTPLC71PIez7qLeQz48kIPmNkoM0s1s9TDhw9fxRJFRMJPRITx8xsbM++p\n3rxwT1tyTuVf2etdpbquiJn1oyQonrvQ4865Gc65js65jnFxcYEtTkSknIqKjODejk1Y9EyfK3ud\nq1TPhewFmpRaTvCs+zdm1haYCdzqnDvix3pERCqkK71Xtz/3KNYCSWbWwsxigKHAF6UHmFlT4BPg\nQedcuh9rERGRMvLbHoVzrtDMxgJfA5HAO865rWY22vP4dOC/gLrAG2YGUFjW83xFRMQ/dMGdiEgF\nELIX3ImISPmnoBAREa8UFCIi4pWCQkREvFJQiIiIVwoKERHxSkEhIiJeKShERMQrBYWIiHiloBAR\nEa8UFCIi4pWCQkREvFJQiIiIVwoKERHxSkEhIiJeKShERMQrBYWIiHiloBAREa8UFCIi4pWCQkRE\nvFJQiIiIVwoKERHxSkEhIiJeKShERMQrBYWIiHiloBAREa8UFCIi4pWCQkREvFJQiIiIVwoKERHx\nSkEhIiLhippkAAAFfUlEQVReKShERMQrBYWIiHiloBAREa8UFCIi4pVfg8LMbjGzNDPLMLPnL/C4\nmdlrnsc3mVl7f9YjIiKXz29BYWaRwOvArUAbYJiZtTlv2K1AkudnFDDNX/WIiEjZ+HOPojOQ4ZzL\ncs7lA3OBIeeNGQLMdiVWAbXMrJEfaxIRkcsU5cfXjgd2l1reA3TxYUw8sL/0IDMbRckeB8BZM9ty\ndUstt+oBOcEuIkRoLs7RXJyjuTindVmf6M+guGqcczOAGQBmluqc6xjkkkKC5uIczcU5motzNBfn\nmFlqWZ/rz6+e9gJNSi0neNZd7hgREQkifwbFWiDJzFqYWQwwFPjivDFfACM8Zz91BY475/af/0Ii\nIhI8fvvqyTlXaGZjga+BSOAd59xWMxvteXw68E/gNiADOA084sNLz/BTyeWR5uIczcU5motzNBfn\nlHkuzDl3NQsREZEwoyuzRUTEKwWFiIh4FbJBofYf5/gwF/d75mCzma0wsxuDUWcgXGouSo3rZGaF\nZnZPIOsLJF/mwsz6mtkGM9tqZksDXWOg+PBvpKaZ/c3MNnrmwpfjoeWOmb1jZocudq1Zmd83nXMh\n90PJwe9MoCUQA2wE2pw35jbgS8CArsDqYNcdxLnoDtT2/P+tFXkuSo1bRMnJEvcEu+4g/l3UArYB\nTT3L9YNddxDn4j+BP3r+Pw44CsQEu3Y/zEVvoD2w5SKPl+l9M1T3KNT+45xLzoVzboVz7phncRUl\n16OEI1/+LgDGAR8DhwJZXID5MhfDgU+cc7sAnHPhOh++zIUDqpuZAdUoCYrCwJbpf865ZZT8bhdT\npvfNUA2Ki7X2uNwx4eByf8/HKPnEEI4uORdmFg/cSfg3mPTl7yIZqG1mS8xsnZmNCFh1geXLXEwF\nrgX2AZuBCc654sCUF1LK9L5ZLlp4iG/MrB8lQdEz2LUE0WTgOedcccmHxwotCugADAAqAyvNbJVz\nLj24ZQXFzcAGoD/QCphvZinOuRPBLat8CNWgUPuPc3z6Pc2sLTATuNU5dyRAtQWaL3PREZjrCYl6\nwG1mVuic+ywwJQaML3OxBzjinMsFcs1sGXAjEG5B4ctcPAL8wZV8UZ9hZtnANcCawJQYMsr0vhmq\nXz2p/cc5l5wLM2sKfAI8GOafFi85F865Fs655s655sBfgSfCMCTAt38jnwM9zSzKzKpQ0r35uwDX\nGQi+zMUuSvasMLMGlHRSzQpolaGhTO+bIblH4fzX/qPc8XEu/guoC7zh+SRd6MKwY6aPc1Eh+DIX\nzrnvzOwrYBNQDMx0zoVdi34f/y5+D8wys82UnPHznHMu7NqPm9kcoC9Qz8z2AL8DouHK3jfVwkNE\nRLwK1a+eREQkRCgoRETEKwWFiIh4paAQERGvFBQiIuKVgkIkgMys+Y+dPT2dXf8e7JpELkVBIeID\nzwVK+vciFZL+8EUuwvPpP83MZgNbgAfNbKWZfWtmfzGzap5xnTz3AdloZmvMrLrnuSmesd+aWffg\n/jYiZReSV2aLhJAk4CFKrmT9BBjonMs1s+eAp83sD8CHwH3OubVmVgM4Q0mL80HOuTwzSwLmUNKH\nSqTcUVCIeLfTObfKzH4GtAGWe9qkxAArKekZtN85txbgx26kZlYVmGpm7YAiSlp+i5RLCgoR73I9\n/zVgvnNuWOkHzeyGizzvKeAgJd1aI4A8v1Uo4mc6RiHim1VADzNLhJI9BjNLBtKARmbWybO+uplF\nATUp2dMoBh6kpFmdSLmkoBDxgXPuMPAwMMfMNlHytdM1nltv3gdMMbONwHwgFngDeMiz7hrO7ZmI\nlDvqHisiIl5pj0JERLxSUIiIiFcKChER8UpBISIiXikoRETEKwWFiIh4paAQERGv/j878pSQBjbq\nOAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x43d6b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_true = np.array([0,0,0,0,0,1])\n",
    "y_scores = np.array([.51,.5,.5,.5,.5,.5])\n",
    "# flip the predicted class labels\n",
    "y_scores = -y_scores\n",
    "pr(y_true, y_scores)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}

Compare with version 0.18.1