LogisticRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "There are cases when [linear regression](http://marcodiiga.github.io/linear-regression) simply doesn't cut it. Classification tasks are classical examples for which a linear regression approach wouldn't suffice.\n",
    "\n",
    "Consider the following case as a classification task where we're interested in the probability $h_\\theta(x) = P(y=1|x;\\theta)$. If $h_\\theta(x) \\ge 0.5$ we consider the sample as verifying the hypothesis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEACAYAAABbMHZzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHxJJREFUeJzt3Xt0VeW57/Hvw61EFLYoXsANaEsKpYJiZWO1dqGloEfw\nfgFbWxy2tLXq0LPdIFYJu1pRW6tusA4UrZ6Kl9p6CqdaLwOXbiyUqGi4JARlB0TRAFqhKPfn/PEu\naIxZCcmaWXOtzN9njIyxLm/m+4hZv8y885lzmrsjIiLJ0i7uAkREJP8U/iIiCaTwFxFJIIW/iEgC\nKfxFRBJI4S8ikkCRhL+ZzTKzD8ysopExKTNbbGZLzezFKOYVEZGWsSj6/M3sJOAfwMPuPqiB97sB\nfwW+7e7vmtnB7r4h54lFRKRFItnzd/f5wEeNDBkH/MHd382MV/CLiMQoX2v+pUB3M3vRzMrN7Lt5\nmldERBrQIY/zDAFOAboAC8xsgbu/laf5RUSkjnyF/1pgg7tvBbaa2cvAYOBz4W9mutiQiEgzubs1\nZ3yUyz6W+WrIn4CTzKy9me0H/BtQmW1D7l7QX1OmTIm9BtWpOlWn6tzz1RKR7Pmb2WwgBRxkZmuA\nKUCnkOM+092rzOxZoALYBcx09+VRzC0iIs0XSfi7+7h9GPNL4JdRzCciIrnRGb4tkEql4i5hn6jO\naKnOaKnOeEVykleUzMwLrSYRkUJmZniMB3xFRKRIKPxFRBJI4S8ikkAKfxGRBFL4i4gkkMJfRCSB\nFP4iIgmk8BcRSSCFv4hIAin8RUQSSOEvIpJACn8RkQRS+IuIJJDCX0QkgSIJfzObZWYfmFlFE+OO\nN7MdZnZOFPOKiEjLRLXn/yAwsrEBZtYOmAY8G9GcIiLSQpGEv7vPBz5qYtgVwJNAbRRziohIy+Vl\nzd/MegJnuftvgGbdbUZERKIXyQ3c98GdwMQ6zxv9BVBWVrb3cSqVarP30BQRaYl0Ok06nc5pG5Hd\nw9fM+gBz3X1QA++t2vMQOBjYAvzQ3ec0MFb38BWRgvDee9CzZ9xVNC3ue/gaWfbo3f2ozNeRhHX/\nnzQU/CIihWD1avje9+D442Hz5riraR1RtXrOBv4KlJrZGjMbb2YTzOyHDQzXbr2IFKQNG+Caa2DI\nEOjTByor4YAD4q6qdUS27BMVLfuISL5t2QJ33gm//jVceCHccAMcdljcVe27uJd9RESKyo4dcO+9\n0K8fLFkCCxfCjBnFFfwtla9uHxGRguEOTz4J118PvXvD3Llw3HFxV5VfCn8RSZR582DSJNi5M+zl\njxgRd0XxUPiLSCK88UYI/ZUr4eab4YILoF2CF74T/J8uIkmwahVcfDGcdhqMHh06eC66KNnBDwp/\nEWmjamvhyitDr35pKVRXw+WXQ6dOcVdWGBT+ItKmbN4MU6fCgAHheWUlTJnSdvv1W0rhLyJtwvbt\n4QDunr388nK4+2445JC4KytMOuArIkVt92544onQttmvHzz9NBx7bNxVFT6Fv4gUreefh4kToX17\nuO8+OOWUuCsqHgp/ESk6r70W2jZXrw5tm+edB6Y7hTSL1vxFpGi89VZo0xw9Gs49F5Ytg/PPV/C3\nhMJfRAre+++HNs1hw+Doo8OJWj/6EXTsGHdlxUvhLyIFa9MmuPFGGDgw9OdXVYUDu126xF1Z8VP4\ni0jB2bYttGmWlkJNTVjj//Wv4eCD466s7dABXxEpGLt3w+zZ4Xr6X/kKPPccDPrcjWElCpGEv5nN\nAs4APshyD99x/PMG7puBH7v7kijmFpHi5w7PPhs6eDp3ht/+Fr75zbiratsiuZOXmZ0E/AN4OEv4\nDwMq3f1jMxsFlLn7sCzb0p28RBJk0aLQq79uHdxyC5x1lrp3miu2O3m5+3zgo0beX+juH2eeLgR6\nRTGviBSv6urQpnnOOTBuHCxdCmefreDPlzgO+F4GPBPDvCJSANatC22aX/96uHtWdTX84AfQQUcg\n8yqv/9xmNhwYD5zU2LiysrK9j1OpFKlUqlXrEpHW9/HHcNtt4Z65l14KK1bAQQfFXVVxSqfTpNPp\nnLYRyZo/gJn1AeY2tOafeX8Q8AdglLu/3ch2tOYv0oZs3Qr33APTpsEZZ0BZWbhvrkSnJWv+Ue75\nW+br82+Y9SYE/3cbC34RaTt27YLf/S6cpDV4cLh37le/GndVskdUrZ6zgRRwkJmtAaYAnQB395nA\nDUB34B4zM2CHuw+NYm4RKSzu4bLKkyZB167wyCNwUqMLvRKHyJZ9oqJlH5HitWBBaNvcuDG0bY4e\nre6dfIit1VNEkq2qKrRsXnABfP/78OabMGaMgr+QKfxFpMXefTe0aX7jG3DCCaFt89JL1bZZDBT+\nItJsH30U1vQHDQrtmtXVcO21UFISd2WyrxT+IrLPPv0Ubr89XG1z48awvDNtGhx4YNyVSXPpjzMR\nadLOnfDww6FH/2tfg5dfhgED4q5KcqHwF5Gs3GHOHJg8OSzvPP54WNuX4qfwF5EGzZ8f2jY3bQqX\nZTj9dHXvtCUKfxH5jGXL4Lrrwnr+z38OF18M7dvHXZVETQd8RQSANWtg/HgYPhxSqXDhtUsuUfC3\nVQp/kYT78MPQpnnssdCzJ6xcCddcE+6oJW2Xwl8koT75JLRplpbC5s2wZAncfDN06xZ3ZZIPCn+R\nhNm5E+67L4T+a6/BK6+Ea+z37Bl3ZZJPOuArkhDu8NRToW3z8MPhj3+Eobq2bmIp/EUS4KWXQtvm\n1q1w550wcqTaNpNO4S/ShlVUhLbN5cvhpptg7Fhop8VeQWv+Im1STU1o0xwxIuzlV1WFfn0Fv+wR\nyY+Cmc0ysw/MrKKRMXeb2Uoze8PMjoliXhH5rA0b4Oqr4bjj4MgjQ9vmlVfCF74Qd2VSaKLaD3gQ\nGJntTTM7Dfiiu/cDJgD3RjSvZLF+/XrKy8tZv359QcyRbWw+6kyCLVtCm2b//rB9ezhLd+rUcBtF\nabk2/fPp7pF8AX2Aiizv3QtcWOd5JXBolrEuuZk9+zEvKenu3boN8ZKS7j579mOxzpFtbD7qbOu2\nb3f/zW/cDz/c/cIL3VeujLuitqOYfj4zudm8zG7uN2TdUOPhPxf4ep3nLwBDsoxtpX+eZKitrfWS\nku4Ob3po7nvTS0q6e21tbSxzZBu7fPnyVq+zLdu92/2JJ9z79XP/1rfcX3017oralnx8jqLUkvAv\nyG6fsrKyvY9TqRSpVCq2WopNTU0NnTr15dNPB2VeGUTHjn2oqamhR48eeZ8j29hFixa1ep1t1bx5\n4S5au3bBjBnhoK5EKx+fo1yk02nS6XRuG2nub4tsXzRv2acKLfu0Cu35t12LF7uPHOl+1FHujz7q\nvmtX3BW1XUnY848y/PsCS7K8dzrw58zjYcDCRrbTSv88ybFnrbJr12Nbfc1/X+bINjYfdbYFb7/t\nPm6c+2GHuU+f7r5tW9wVJUMx/Xy2JPwtfF9uzGw2kAIOAj4ApgCdMgXNzIyZDowCtgDj3f31LNvy\nKGpKuvXr11NTU0Pfvn1b7c/U5syRbWw+6ixWtbXhxKxHHoGrrgpX2tx//7irSpZi+fk0M9y9Weds\nRxL+UVL4S9Jt3gx33AF33w3f+Q5cfz0cckjcVUkha0n463w/kQKxfTtMnx6utlldDeXlcNddCn5p\nHQXZ7SOSJLt3wxNPhD38fv3g6afDjVVEWpPCXyRGzz8frrbZvn24xv4pp8RdkSSFwl8kBq+9Fnr1\nV68Ol2U47zxdYlnyS2v+Inn01ltw0UUwenQI/GXL4PzzFfySfwp/kTx4/324/HIYNgyOPjpcbXPC\nBOjYMe7KJKkU/iKtaNMmuPFGGDgwXFa5qioc2O3SJe7KJOkU/iKtYNu20KZZWhrW9V97LfTuH3xw\n3JWJBDrgKxKh3bth9my44Yawt//cczBoUNPfJ5JvCn+RCLjDs8+GDp7OneG3v4VvfjPuqkSyU/iL\n5GjRotCrv24d3HILnHWWunek8GnNX6SFqqtDm+Y554Sboy9dCmefreCX4qDwF2mmdevgRz+CE08M\nN0qvrobLLoMO+jtaiojCX2QfffxxaNP86lfhgANgxYqwxr/ffnFXJtJ8Cn+RJmzdGto0S0vDXv/i\nxXD77dC9e9yVibRcJOFvZqPMrMrMqs1sYgPvdzWzOWb2hpktMbPvRzGvSGvatQseegi+/GV46aVw\n79wHHoDeveOuTCR3Od/MxczaAdXAqcB7QDlwkbtX1RlzHdDV3a8zs4OBFYR7+O5sYHu6mYvEyj1c\nVnnSJOjaFW69FU46Ke6qRLJryc1cojhENRRY6e6rM0U8BpxJuEn7Hg4ckHl8ALCxoeAXiduCBaFt\nc+PG0LY5erS6d6RtimLZpxfwTp3nazOv1TUd+IqZvQe8CVwVwbwikamsDG2aF1wA48dDRQWMGaPg\nl7YrXwd8RwKL3b0ncCwww8x0K2qJ3dq1oU3z5JND62Z1dQj/9u3jrkykdUWx7PMuUPcQ2BGZ1+oa\nD9wC4O5vm9n/AP2BVxvaYFlZ2d7HqVSKVCoVQZki//TRR2Et/7774Ac/CKF/4IFxVyWyb9LpNOl0\nOqdtRHHAtz3hAO6pwDpgETDW3SvrjJkB1Lr7VDM7lBD6g939wwa2pwO+0mo+/TTcJP2228JlGMrK\noFf9RUqRIhPLAV9332VmPwWeIywjzXL3SjObEN72mcBNwG/NrCLzbf/RUPCLtJadO+Hhh2HKFDj+\neHj5ZRgwIO6qROKT855/1LTnL1FyhzlzYPJkOOigsNRzwglxVyUSrbhaPUUK0vz5oW1z8+awzHP6\n6ereEdlD4S9tztKlYU+/ogL+8z/DFTfVvSPyWbq2j7QZa9aENs1TToHhw8P9ci+5RMEv0hCFvxS9\nDz+Ea6+FY48NnTsrV8LVV4c7aolIwxT+UrQ++QSmTQsXXvvHP8Jyz003QbducVcmUvi05i9FZ+dO\nePBBmDo1dO688kq43LKI7DuFvxQNd3jqqXAw9/DD4Y9/hKFD465KpDgp/KUovPRSaNvcuhXuvBNG\njlTbpkguFP5S0Coq4LrrwlU3f/5zGDsW2ulIlUjO9DGSglRTE9o0R4wIe/mVlaFfX8EvEg19lKSg\nbNgQ2jSPOw6OPDK0bV55JXzhC3FXJtK2KPylIGzZEto0+/eHHTtg+fLQzdO1a9yVibRNCn+J1Y4d\ncO+90K8fLFsGCxeGSy4femjclYm0bTrgK7Fwh9//Hn72M+jdG+bODUs9IpIfCn/Ju3nzQtvm7t0w\nY0Y4qCsi+aXwl7xZvBgmTYK334abb4bzz1f3jkhcIvnomdkoM6sys2ozm5hlTMrMFpvZUjN7MYp5\npTisWgXjxoXr6Y8ZEw7mXnihgl8kTjl//MysHTAdGAkMBMaaWf96Y7oBM4Az3P2rwPm5ziuFr7Y2\ntGkef3zo4qmuhssvh06d4q5MRKLY9xoKrHT31e6+A3gMOLPemHHAH9z9XQB33xDBvFKgNm8ObZoD\nBoRLMFRWwo03wgEHxF2ZiOwRRfj3At6p83xt5rW6SoHuZvaimZWb2XcjmFcKzPbtoU2ztDScnFVe\nDnfdBYccEndlIlJfvg74dgCGAKcAXYAFZrbA3d/K0/zSinbvhscfD22bpaXwzDNwzDFxVyUijYki\n/N8Fetd5fkTmtbrWAhvcfSuw1cxeBgYDDYZ/WVnZ3sepVIpUKhVBmdIann8+tG126AD33x9unygi\nrSudTpNOp3Pahrl7bhswaw+sAE4F1gGLgLHuXllnTH/gv4BRwBeAvwEXuvvyBrbnudYkre/VV0Pb\n5po18ItfwLnn6hLLInExM9y9WZ/AnPf83X2Xmf0UeI5wDGGWu1ea2YTwts909yozexaoAHYBMxsK\nfil8b70Vlnf++7/DQdxLL4WOHeOuSkSaK+c9/6hpz78wvf9+uJ7+44/DNdfAVVdBly5xVyUi0LI9\nf51mI43atCns4Q8cCJ07w4oV4TaKCn6R4qbwlwZt2xbaNPv1C+v6r78Ov/oVHHRQ3JWJSBR0bR/5\njN27YfZsuOGGsLf/wgtw9NFxVyUiUVP4CxAusfyXv4T75ZaUwEMPwcknx12ViLQWhb+waFHo1V+3\nDm65Bc46S22bIm2d1vwTbMUKOO88OOeccHP0pUvh7LMV/CJJoPBPoPfegwkT4KSTwhU3q6vhssvC\nWboikgwK/wT5+GO4/vpwALdbt7DnP3Ei7Ldf3JWJSL4p/BNg61a4447QtrluHbzxBtx2G3TvHndl\nIhIX/aHfhu3aBb/7XThJ65hj4MUXQ/umiIjCvw1yhz//ObRtdu0KjzwS1vdFRPZQ+LcxCxaEdfyN\nG8PVNseMUfeOiHye1vzbiMrK0KZ5wQUwfjxUVMCZZyr4RaRhCv8it3ZtaNM8+WQ48cTQtjl+PLRv\nH3dlIlLIFP5F6qOPwvLO4MFw8MEh9P/938OlGUREmqLwLzKffgq33x7ulfvhh2F5Z9o0OPDAuCsT\nkWISSfib2SgzqzKzajOb2Mi4481sh5mdE8W8SbJzJzzwQAj9BQvg5ZfhvvugV6+4KxORYpRzt4+Z\ntQOmE+7h+x5QbmZ/cveqBsZNA57Ndc4kcYc5c0LbZo8e8MQTcMIJcVclIsUuilbPocBKd18NYGaP\nAWcCVfXGXQE8CRwfwZyJMH9+WNffvBl++Us47TR174hINKJY9ukFvFPn+drMa3uZWU/gLHf/DaD4\nasLSpaE//zvfCRdgW7wYTj9dwS8i0cnXSV53AnWPBTQaY2VlZXsfp1IpUqlUqxRVaNasgSlT4Omn\nYdKksMTTuXPcVYlIoUmn06TT6Zy2Ye6e2wbMhgFl7j4q83wS4O5+a50xq/Y8BA4GtgA/dPc5DWzP\nc62p2GzcGG6i8uCD8OMfw7XXhqtuiojsCzPD3Zu1NhDFnn858CUz6wOsAy4CxtYd4O5H1SnyQWBu\nQ8GfNJ98Em6SfscdcO65Ybnn8MPjrkpEkiDn8Hf3XWb2U+A5wjGEWe5eaWYTwts+s/635DpnsdvT\ntjl1ajgr95VXQguniEi+5LzsE7W2vOzjDk89BZMnQ8+ecOut4U5aIiK5iGvZR/bBSy+Fts1t28JS\nz7e/re4dEYmPwr+VVVSEE7QqK+Hmm+HCC6GdLqohIjFTDLWSmhq45JKwhz9qFFRVwdixCn4RKQyK\nooht2ABXXw3HHQdHHQUrV8IVV0CnTnFXJiLyTwr/iGzZEpZ1+veHHTtg+XIoK4MDDoi7MhGRz1P4\n52jHDrj3XujXL/TpL1wI06fDoYfGXZmISHY64NtC7vDkk3D99dCnD8ydG5Z6RESKgcK/BebNC9fe\n2bULZsyAESPirkhEpHkU/s2weHFo23zrrbC+f/756t4RkeKk6NoHq1bBxReHyyqPGRMO5qpfX0SK\nmeKrEbW1cOWVMHQofPnLoW3zJz9R26aIFD+FfwM2bw4XXRswIFyCYflyuPFG2H//uCsTEYmGwr+O\n7dtDm2ZpadjLLy8P1+E55JC4KxMRiZYO+AK7d8Pjj8PPfhaC/5ln4Jhj4q5KRKT1JD78n38+XG2z\nQwe4/34YPjzuikREWl9iw//VV0Ov/po18ItfhDtp6RLLIpIUkaz5m9koM6sys2ozm9jA++PM7M3M\n13wzOzqKeVti5crQpjlmTOjTX7YMzjtPwS8iyZJz+JtZO2A6MBIYCIw1s/71hq0CTnb3wcBNwH25\nzttc778f2jRPOAEGDw6/BCZMgI4d812JiEj8otjzHwqsdPfV7r4DeAw4s+4Ad1/o7h9nni4EekUw\n7z7ZtAluuAEGDoTOncN19SdPhi5d8lWBiEjhiWLNvxfwTp3nawm/ELK5DHgmgnmb9Mkn8JWvwKmn\nwuuvhwuwiYhIng/4mtlwYDxwUmPjysrK9j5OpVKkUqkWzbfffuHA7mGHtejbRUQKUjqdJp1O57QN\nc/fcNmA2DChz91GZ55MAd/db640bBPwBGOXubzeyPc+1JhGRJDEz3L1ZbStRrPmXA18ysz5m1gm4\nCJhTr7DehOD/bmPBLyIi+ZHzso+77zKznwLPEX6ZzHL3SjObEN72mcANQHfgHjMzYIe7N3ZcQERE\nWlHOyz5R07KPiEjzxLXsIyIiRUbhLyKSQAp/EZEEUviLiCSQwl9EJIEU/iIiCaTwFxFJIIW/iEgC\nKfxFRBJI4S8ikkAKfxGRBFL4i4gkkMJfRCSBFP4iIgmk8BcRSSCFv4hIAkUS/mY2ysyqzKzazCZm\nGXO3ma00szfM7Jgo5hURkZbJOfzNrB0wHRgJDATGmln/emNOA77o7v2ACcC9uc67r9avX095eTnr\n16+PfY5s4/JRo4hIXVHs+Q8FVrr7anffATwGnFlvzJnAwwDu/jegm5kdGsHcjXr00cfp06c/I0b8\niD59+vPoo4/HNke2cfmoUUTkc9w9py/gXGBmneffAe6uN2Yu8PU6z18AhmTZnkehtrbWS0q6O7zp\n4A5veklJd6+trY1k+82ZI9u45cuXt3qNItL2ZXKzWdndIYbfN00qKyvb+ziVSpFKpZq9jZqaGjp1\n6sunnw7KvDKIjh37UFNTQ48ePSKpc1/nyDZu0aJFrV6jiLQ96XSadDqd0zaiCP93gd51nh+Rea3+\nmH9tYsxedcO/pfr27cv27TVABTAIqGDHjtX07ds35203d45s44YOHdrqNYpI21N/p3jq1KnN3kYU\na/7lwJfMrI+ZdQIuAubUGzMHuATAzIYBf3f3DyKYO6sePXowa9Y9lJQMp2vXIZSUDGfWrHsi3aPe\n1zmyjRswYECr1ygi0hALy0U5bsRsFHAX4ZfJLHefZmYTCOtQMzNjpgOjgC3AeHd/Pcu2PIqa9li/\nfj01NTX07du31UJ1X+fINi4fNYpI22VmuLs163uiDNooRB3+IiJtXUvCX2f4iogkkMJfRCSBFP4i\nIgmk8BcRSSCFv4hIAin8RUQSSOEvIpJACn8RkQRS+IuIJJDCX0QkgRT+IiIJpPAXEUkghb+ISAIp\n/EVEEkjhLyKSQDmFv5kdaGbPmdkKM3vWzLo1MOYIM5tnZsvMbImZXZnLnCIikrtc9/wnAS+4+5eB\necB1DYzZCVzj7gOBE4DLzax/jvPGKtcbJ+eL6oyW6oyW6oxXruF/JvBQ5vFDwFn1B7j7++7+Rubx\nP4BKoFeO88aqWH4YVGe0VGe0VGe8cg3/Q/bciN3d3wcOaWywmfUFjgH+luO8IiKSgw5NDTCz54FD\n674EOPCzBoZnvfmume0PPAlclfkLQEREYpLTDdzNrBJIufsHZnYY8KK7D2hgXAfg/wHPuPtdTWxT\nd28XEWmm5t7Avck9/ybMAb4P3Ap8D/hTlnEPAMubCn5o/n+AiIg0X657/t2BJ4B/BVYDF7j7383s\ncOA+dz/DzE4EXgaWEJaFHJjs7n/JuXoREWmRnMJfRESKU8Gc4Wtmo8ysysyqzWxi3PU0pNhOWDOz\ndmb2upnNibuWbMysm5n93swqM/+u/xZ3TfWZ2dVmttTMKszsETPrFHdNe5jZLDP7wMwq6rzW5MmX\nBVDjbZn/52+Y2R/MrGucNWZq+lyddd7732a2O7PaEatsdZrZFZl/0yVmNq2p7RRE+JtZO2A6MBIY\nCIwt0BPBiu2EtauA5XEX0YS7gKczjQKDCeeBFAwz6wlcAQxx90GE42QXxVvVZzxI+NzUtS8nX+ZT\nQzU+Bwx092OAlcRfIzRcJ2Z2BDCCsLRdCD5Xp5mlgNHA0e5+NPDLpjZSEOEPDAVWuvtqd98BPEY4\ngaygFNMJa5kf2NOB++OuJZvM3t433P1BAHff6e6bYi6rIe2BLpmutf2A92KuZy93nw98VO/lJk++\nzKeGanT3F9x9d+bpQuCIvBdWT5Z/S4BfA9fmuZysstT5Y2Cau+/MjNnQ1HYKJfx7Ae/Ueb6WAg3V\nPYrghLU9P7CFfFDnSGCDmT2YWZ6aaWYlcRdVl7u/B/wKWAO8C/zd3V+It6omNevkywJwKfBM3EU0\nxMzGAO+4+5K4a2lCKXCymS00sxfN7GtNfUOhhH9RKfQT1szsfwEfZP5KscxXIeoADAFmuPsQ4BPC\nkkXBMLN/IexJ9wF6Avub2bh4q2q2gt0BMLPrgR3uPjvuWurL7IhMBqbUfTmmcprSATjQ3YcB/0Ho\nwmxUoYT/u0DvOs+PyLxWcDJ/+j8J/B93z3ZeQ9xOBMaY2SrgUWC4mT0cc00NWUvYq3o18/xJwi+D\nQvItYJW7f+juu4A/Al+PuaamfGBmhwJkTr6sjbmeBpnZ9wlLk4X6y/SLQF/gTTP7H0IuvWZmhfiX\n1DuEn03cvRzYbWYHNfYNhRL+5cCXzKxPppPiIsIJZIVon09Yi4u7T3b33u5+FOHfcp67XxJ3XfVl\nlibeMbPSzEunUngHqNcAw8yss5kZocaCOijN5/+623PyJTR+8mU+faZGMxtFWJYc4+7bYqvq8/bW\n6e5L3f0wdz/K3Y8k7Kwc6+6F8Mu0/v/z/wucApD5PHV0942NbaAgwj+zR/VTQgfAMuAxdy+0DxiZ\nE9YuBk4xs8WZdepRcddV5K4EHjGzNwjdPr+IuZ7PcPdFhL9IFgNvEj5wM2Mtqg4zmw38FSg1szVm\nNh6YBowwsxWEX1ZNtv3FUON/AfsDz2c+R/fEWSNkrbMupwCWfbLU+QBwlJktAWYDTe7s6SQvEZEE\nKog9fxERyS+Fv4hIAin8RUQSSOEvIpJACn8RkQRS+IuIJJDCX0QkgRT+IiIJ9P8BSqgBkLF8+dIA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x40c3cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def genClassificationData():\n",
    "    x = np.zeros(shape=9)\n",
    "    y = np.zeros(shape=9)\n",
    "\n",
    "    # Empirically adjusted for clarity's sake\n",
    "    x[0] = 1\n",
    "    x[1] = 2.2\n",
    "    x[2] = 2.7\n",
    "    x[3] = 3\n",
    "    y[0] = y[1] = y[2] = y[3] = 0\n",
    "\n",
    "    x[4] = 4\n",
    "    x[5] = 5.2\n",
    "    x[6] = 5.7\n",
    "    x[7] = 6\n",
    "    y[4] = y[5] = y[6] = y[7] = 1\n",
    "\n",
    "    x[8] = 14\n",
    "    y[8] = 1\n",
    "\n",
    "    return x, y\n",
    "\n",
    "def normEquation(x, y):\n",
    "    ll = []\n",
    "    for sample in x:\n",
    "        ll += [[1] + [sample]]\n",
    "    X = np.matrix(ll)\n",
    "    ll = [[sample] for sample in y]\n",
    "    y = np.matrix(ll)\n",
    "    x_trans = X.T\n",
    "    A = (x_trans * X).I\n",
    "    theta = A * x_trans * y\n",
    "    return theta\n",
    "\n",
    "def plotLinearPredictor(x, y, theta):\n",
    "    plt.figure()\n",
    "    plt.scatter(x, y)\n",
    "\n",
    "    y_regression = np.zeros(len(x))\n",
    "    for sample in range(0, len(x)):\n",
    "        y_regression[sample] = theta[0] + theta[1] * x[sample]\n",
    "\n",
    "    plt.plot(x, y_regression)\n",
    "    plt.show()\n",
    "\n",
    "x, y = genClassificationData()\n",
    "theta_coefficients = normEquation(x, y)\n",
    "plotLinearPredictor(x,y, theta_coefficients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establishing decision boundaries\n",
    "\n",
    "Ideally for a well-trained binary classifier we would like to be able to tell if a sample $P$ lies in the region that verifies our hypothesis. For a parametric one-degree polynomial this is trivially $\\theta_0 + \\theta_1 x_1 + \\theta_2 x_2 \\ge 0$.\n",
    "\n",
    "Let us define $h_{\\theta}(x) = g(\\theta^{T}x)$ and consider\n",
    "\n",
    "$$\n",
    "g(x) = \\frac{1}{1+e^{-x}}\n",
    "$$\n",
    "\n",
    "this is known as the **sigmoid** or **logistic** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGJVJREFUeJzt3XuQlPWd7/H3F/CGEVYWRQVEVmMUVoNsQhCCpxUvg66B\n4CVAkl1NckJFOXXWpOpgKnvWWXfLo1aSyjEsZ+XmZStcFBJEAzrx0moWAlhcVAQBFWQGnI0EVBIx\nw8z3/PFrtB1mmO6Znvn18/TnVdXV093PNJ8ahw+P3+d5fm3ujoiIpEu32AFERKT0VO4iIimkchcR\nSSGVu4hICqncRURSSOUuIpJCbZa7mc01s3oze/ko29xnZtvMbIOZDSttRBERKVYhe+4PAFe19qKZ\njQPOdvfPAlOBfy9RNhERaac2y93dfwvsO8om44GHc9uuBnqbWb/SxBMRkfYoxcy9P7Ar73Fd7jkR\nEYlEB1RFRFKoRwneow4YmPd4QO65I5iZFrIREWkHd7diti+03C13a8ky4FZgkZmNBPa7e/1RAhaT\nT46iurqa6urq2DFSQz/Po3v/fXjttXDbvh127IC33gr3+/bBaadBv37hVltbzbhx1fTrB6eeCief\nDL17h1uvXuH+xBPBiqqrymXt+EG1We5mNh/IAH9pZm8DdwDHAu7us9x9uZldbWbbgT8CNxedQkTK\nyrvvwqpVsHIlbNgAmzbB3r1w3nkwdCicey5ccw2cdVa4nX46dMsb8lZXh5vE02a5u/uUAraZVpo4\nIhJDXR089RS88EIo9Pp6GDkSLr4Yvve9UOiDB3+6wKW8lWLmLpFkMpnYEVKlkn6e7rBxIzzyCCxf\nDrt2wRVXwKWXwg9+AEOGQPfu7X//SvpZlivryhm4mblm7iLx7NgBDz0ECxbARx/B174GX/kKjBgB\nPbSrV7bMrNMOqIpIQjU1QU0NzJwZRi5TpoSCHzFCBzTTTOUuklKNjWHscuedcPzxcOutsHAh9OwZ\nO5l0BZW7SMq4h1L/53+Gv/gLuO8+uPxy7aVXGpW7SIq88grccgt8+CH87GfhIKlKvTLpxCaRFPjg\nA7jtNhg7Fr7xDVi9Gq68UsVeybTnLpJwa9aEg6Rf/nK42OiUU2InknKgchdJqMZGuPfeMH6ZOROu\nuy52IiknKneRBNq/H268MZyr/tJLMHBg298jlUUzd5GE2bEDRo8O67w8+6yKXVqmchdJkDVrYNQo\nmDo1nOLYkSUCJN00lhFJiGefDcsFzJsH114bO42UO5W7SAI8/zxMmgRLlsAll8ROI0mgsYxImXvx\nRbjhBli0SMUuhVO5i5Sx1avDKY7z54fleEUKpSV/RcrU22+HD8y4/37N2Ctde5b81Z67SBk6cCCs\ns/6DH6jYpX205y5SZpqawiimTx+YM0frw4g+rEMkFf7pn8KHUS9apGKX9lO5i5SRp5+GBx+E9evh\n2GNjp5Ek08xdpEzs3Qs33QQPPKCVHaXjNHMXKQPuMHEinH02/PjHsdNIudHMXSShZs8OC4ItXBg7\niaSF9txFInvjDfjSl8KVqOefHzuNlKP27Lmr3EUicoerr4ZMBqZPj51GypUuYhJJmCVLYNcu+P73\nYyeRtNGeu0gk778PQ4bAggUwZkzsNFLONJYRSZDbbgsFP3du7CRS7nS2jEhCbNgQVnrctCl2Ekkr\nzdxFuph7mLHfeSf07Rs7jaSVyl2ki9XUQF0dfPvbsZNImqncRbpQUxPcfjvcdRf00FBUOpHKXaQL\nLVoUFgSbODF2Ekk7nS0j0kX+/OdwBercueGiJZFC6SImkTI2axace66KXbpGQeVuZlVmtsXMtprZ\nERdJm1kvM1tmZhvM7BUzu6nkSUUS7ODBMGe/667YSaRStFnuZtYNmAFcBQwFJpvZec02uxXY5O7D\ngEuBn5iZDheJ5Dz4IAwfDhddFDuJVIpCCngEsM3ddwKY2UJgPLAlbxsHTsp9fRKw190PlTKoSFId\nOgT33gv/8R+xk0glKWQs0x/Ylfe4NvdcvhnAEDPbDWwE/mdp4okk3yOPwIABMHp07CRSSUo1OrkK\nWO/ul5nZ2cBvzOxCdz/QfMPq6uqPv85kMmR0dElSzB3uvhvuuSd2EkmSbDZLNpvt0Hu0eSqkmY0E\nqt29Kvf4dsDd/Z68bZ4A/o+7/2fu8TPAdHd/qdl76VRIqShPPAH/+I/hA6+tqBPZRD7RWadCrgXO\nMbNBZnYsMAlY1mybncDluRD9gHOBN4sJIpJGd98drkhVsUtXa3Ms4+6NZjYNqCH8YzDX3Teb2dTw\nss8C/hV40Mxezn3b/3L3P3RaapEEWLUK9uyB66+PnUQqka5QFekkX/86fOELYd12kY7Qh3WIlIl3\n3glLDbz5Jpx8cuw0knRafkCkTMyeDTfeqGKXeLTnLlJiDQ1w1lmwYgVceGHsNJIG2nMXKQNLl8I5\n56jYJS6Vu0iJzZgB06bFTiGVTuUuUkIvvwxvvAETJsROIpVO5S5SQvffD9/9LhxzTOwkUul0QFWk\nRD78MCwQtmEDDBwYO42kiQ6oikS0dGm4aEnFLuVA5S5SIg88ADffHDuFSKCxjEgJ7NwJf/M3UFsL\nxx8fO42kjcYyIpE89BBMmqRil/KhPXeRDmpqgrPPhsWLw967SKlpz10kguefh169wgdgi5QLlbtI\nB82bFw6k6gM5pJxoLCPSAQcOQP/+sH07nHJK7DSSVhrLiHSxZctg9GgVu5QflbtIB8yfD1OmxE4h\nciSNZUTa6d13w1kytbVw0kmx00iaaSwj0oUWL4Zx41TsUp5U7iLttGABTJ4cO4VIyzSWEWmHXbtg\n2DDYvRuOOy52Gkk7jWVEusjChTBxoopdypfKXaQddJaMlDuVu0iRtmyB+nq45JLYSURap3IXKdLi\nxXDDDdC9e+wkIq1TuYsUafFiuP762ClEjk7lLlKEbdvCSGbUqNhJRI5O5S5ShCVLwlkyGslIuVO5\nixRBIxlJCpW7SIHeegvefhvGjImdRKRtKneRAi1ZAl/9KvToETuJSNtU7iIF0khGkkRry4gU4O23\nw2ek7tkDxxwTO41UGq0tI9JJfvlLGD9exS7JUVC5m1mVmW0xs61mNr2VbTJmtt7MXjWz50obUySu\npUthwoTYKUQK1+ZYxsy6AVuBscBuYC0wyd235G3TG1gJXOnudWbW193fbeG9NJaRxNm7FwYPDhcv\nnXBC7DRSiTprLDMC2ObuO929AVgIjG+2zRRgibvXAbRU7CJJtWIFXHaZil2SpZBy7w/syntcm3su\n37lAHzN7zszWmtk3SxVQJLZly+ArX4mdQqQ4pTpjtwcwHLgMOBFYZWar3H17id5fJIqPPoKaGvj5\nz2MnESlOIeVeB5yZ93hA7rl8tcC77n4QOGhmLwCfB44o9+rq6o+/zmQyZDKZ4hKLdKHnn4chQ6Bf\nv9hJpJJks1my2WyH3qOQA6rdgdcJB1T3AGuAye6+OW+b84CfA1XAccBq4Gvu/lqz99IBVUmUadNg\nwAC4/fbYSaSSteeAapt77u7eaGbTgBrCjH6uu282s6nhZZ/l7lvM7CngZaARmNW82EWSxj3M2598\nMnYSkeLpClWRVmzYEJYb2LYNrKh9JpHS0hWqIiV0+CwZFbskkcpdpBU6BVKSTGMZkRbU1sLnPw/v\nvKP1ZCQ+jWVESuSJJ2DcOBW7JJfKXaQFGslI0mksI9LMgQNw+ulhNNO7d+w0IhrLiJRETQ1cfLGK\nXZJN5S7SjEYykgYay4jkaWyE006Dl16CQYNipxEJNJYR6aBVq6B/fxW7JJ/KXSSPRjKSFip3kTwq\nd0kLlbtIzuuvwwcfwPDhsZOIdJzKXSTn8cfh2muhm/5WSAro11gkRyMZSROdCikCvPsunH021NfD\n8cfHTiPyaToVUqSdli+HsWNV7JIeKncRNJKR9NFYRirewYPQrx9s3w6nnBI7jciRNJYRaYdsFi64\nQMUu6aJyl4qnkYykkcYyUtHcYeBAePppOO+82GlEWqaxjEiR1q+Hnj3hc5+LnUSktFTuUtEOj2Ss\nqH0ikfKncpeKpnm7pJVm7lKxdu2Ciy6Cd96BHj1ipxFpnWbuIkV4/HG4+moVu6STyl0qlkYykmYa\ny0hFev/98HF6u3fDSSfFTiNydBrLiBToqadg1CgVu6SXyl0q0mOPwYQJsVOIdB6NZaTiNDSEhcJe\neSWMZkTKncYyIgV48UU45xwVu6Sbyl0qztKlMH587BQinUtn+EpFcQ/z9l//OnYSkc6lPXepKBs3\nhouWhg6NnUSkcxVU7mZWZWZbzGyrmU0/ynZfNLMGM5tYuogipbN0aThLRguFSdq1We5m1g2YAVwF\nDAUmm9kRK1/ntrsbeKrUIUVK5bHHNG+XylDInvsIYJu773T3BmAh0NJfj/8BLAb+q4T5REpm506o\nrQ0XL4mkXSHl3h/Ylfe4Nvfcx8zsDGCCu/8/QP/DK2Vp2TL427/VQmFSGUp1QPVnQP4sXgUvZUen\nQEolKWQfpg44M+/xgNxz+b4ALDQzA/oC48yswd2XNX+z6urqj7/OZDJkMpkiI4sUb98+WLsWrrgi\ndhKRtmWzWbLZbIfeo83lB8ysO/A6MBbYA6wBJrv75la2fwB43N1/2cJrWn5AovjFL+CRR8IBVZGk\nac/yA23uubt7o5lNA2oIY5y57r7ZzKaGl31W828pJoBIV9BIRiqNFg6T1Pvoo7BQ2NatcOqpsdOI\nFE8Lh4m04Jln4K//WsUulUXlLqm3ZAlcf33sFCJdS2MZSbWGBjj9dFi3Ds48s+3tRcqRxjIizWSz\nYe12FbtUGpW7pNrixRrJSGXSWEZS69AhOOMMWL0aBg+OnUak/TSWEcnz4othHKNil0qkcpfU0khG\nKpnGMpJKjY0wYAC88AJ89rOx04h0jMYyIjkrV4arUlXsUqlU7pJKjz4K110XO4VIPBrLSOpoJCNp\no7GMCPDcc6HcVexSyVTukjrz58OUKbFTiMSlsYykysGD4cKlV18N9yJpoLGMVLzly2HYMBW7iMpd\nUkUjGZFAYxlJjffeC8sN7NgBJ58cO41I6WgsIxXtV7+CSy9VsYuAyl1SZMECmDw5dgqR8qCxjKTC\n7t0wdCjU1UHPnrHTiJSWxjJSsR5+OKwAqWIXCXrEDiDSUe4wbx489FDsJCLlQ3vukngrV0L37jBy\nZOwkIuVD5S6JN28efOtbYEVNJEXSTQdUJdEOHICBA2HzZjjttNhpRDqHDqhKxVm8GMaMUbGLNKdy\nl0Q7PJIRkU/TWEYSa/t2GD0aamvhmGNipxHpPBrLSEW5/374u79TsYu0RHvukkh/+hMMGgRr1sDg\nwbHTiHQu7blLxViwAC6+WMUu0hqVuySOO8yYAdOmxU4iUr5U7pI4K1eGsczll8dOIlK+VO6SODNm\nwK23Qjf99oq0SgdUJVH27AlL+771FvTuHTuNSNfotAOqZlZlZlvMbKuZTW/h9SlmtjF3+62ZXVBM\nCJFCzZoFkyap2EXa0uaeu5l1A7YCY4HdwFpgkrtvydtmJLDZ3d8zsyqg2t2PWKNPe+7SEX/6Uzg7\nJpuF88+PnUak63TWnvsIYJu773T3BmAhMD5/A3f/nbu/l3v4O6B/MSFECjFnTrgiVcUu0rZCPqyj\nP7Ar73EtofBb8x1gRUdCiTT35z/Dj38MS5bETiKSDCX9JCYzuxS4Gfhya9tUV1d//HUmkyGTyZQy\ngqTU/Plw7rnwxS/GTiLS+bLZLNlstkPvUcjMfSRhhl6Ve3w74O5+T7PtLgSWAFXu/kYr76WZuxSt\nqSmcITNjBowdGzuNSNfrrJn7WuAcMxtkZscCk4Blzf7gMwnF/s3Wil2kvZYuhc98Bi67LHYSkeRo\ncyzj7o1mNg2oIfxjMNfdN5vZ1PCyzwL+N9AHmGlmBjS4+9Hm8iIFcYe774Yf/lAfoydSDF3EJGVt\n2TL40Y9g40ZdkSqVqz1jmZIeUBUppcbGsMd+zz0qdpFi6a+MlK2HH4Y+feCaa2InEUkejWWkLB08\nGE59XLgQRo2KnUYkLn1Yh6TGv/0bDB+uYhdpL+25S9nZvz/stWezMGRI7DQi8bVnz13lLmXnttvg\nwAGYPTt2EpHyoLNlJPHWrw9LDWzaFDuJSLJp5i5lo6kJvvc9uOsu6Ns3dhqRZFO5S9mYPRt69ICb\nb46dRCT5NHOXslBfDxdcAM88E+5F5BM6oCqJNWUKDBgA994bO4lI+dEBVUmk+fPDgdQ5c2InEUkP\n7blLVDt2hA/gqKmBiy6KnUakPOkKVUmUQ4fgG9+A6dNV7CKlpnKXaO66C044Ab7//dhJRNJHM3eJ\n4umnYeZMWLdOy/mKdAaVu3S511+Hr38dHn0UzjgjdhqRdNI+k3SpP/wBrr02jGQuuSR2GpH00tky\n0mUaGqCqCoYNg5/8JHYakeTQRUxSthob4aabYN8+eOwx6N49diKR5NBFTFKWmprgO9+Bujp44gkV\nu0hXULlLp2pqgqlT4c03Yfly6NkzdiKRyqByl05z6BDccgts3gwrVsCJJ8ZOJFI5VO7SKT74ACZN\nCgdRly+Hk06KnUiksuhUSCm52loYMwb694df/xp69YqdSKTyqNylpF58ES6+OFykdP/9cMwxsROJ\nVCaNZaQkDh2Cf/kXmDUrLN17zTWxE4lUNpW7dNibb8I3vxkOmK5bB6efHjuRiGgsI+128CDceWdY\nj33iRHjySRW7SLnQnrsUzT2cAfMP/xA+73TdOhg0KHYqEcmncpeCuYdPTKquhvfeg/vug3HjYqcS\nkZao3KVNDQ1hPZif/jSsDXPHHXDDDVpGQKScqdylVbW1MG9eOANm8OAwhrnuOpW6SBKo3OVTfv97\nWLIEFiyAV14Je+jLl8OFF8ZOJiLF0JK/Fa6xMRwQXbEi3F57Da6+OiwdUFUFxx0XO6GIdNp67mZW\nBfyMcOrkXHe/p4Vt7gPGAX8EbnL3DS1so3KPbN8++N3vYOVKWLUK1qyBgQNDkY8bF5YNUKGLlJdO\nKXcz6wZsBcYCu4G1wCR335K3zThgmrtfY2ZfAv6vu49s4b1U7iWUzWbJZDItvrZ/P2zfDps2ffq2\nd284L33UqHAbORL69Ona3OXqaD9PKY5+lqXVWR/WMQLY5u47c3/IQmA8sCVvm/HAwwDuvtrMeptZ\nP3evLyaMtK2hIczF6+thzpwsO3dmqK+H3bthx45Pbo2N8Fd/BUOHhtt3vxvuBw/WAdHWqJBKRz/L\n+Aop9/7ArrzHtYTCP9o2dbnnKqbc3cP6Ko2Nn9w3NMBHH4UrOT/8MNzn35o/98c/wvvvh3PI33vv\nk68P3+/fH5bS7dsXTj0VDhwAM+jXL4xWxoyBs84Ktz59wmsiUpm6/GyZqqpQhBDuD9+O9riYbUv5\nvY2Nny7r/PvmzzU1hT3iHj0+ue/RA44//tO3E0448rnDz59wAvTuDWecEe579Qr3+V/36QPdcotG\nVFeHm4hIc4XM3EcC1e5elXt8O+D5B1XN7N+B59x9Ue7xFuC/NR/LmJkG7iIi7dAZM/e1wDlmNgjY\nA0wCJjfbZhlwK7Ao94/B/pbm7cWGExGR9mmz3N290cymATV8cirkZjObGl72We6+3MyuNrPthFMh\nb+7c2CIicjRdehGTiIh0jS5Zz93MrjezV82s0cyGN3vth2a2zcw2m9mVXZEnTczsDjOrNbN1uVtV\n7ExJY2ZVZrbFzLaa2fTYeZLOzHaY2UYzW29ma2LnSRozm2tm9Wb2ct5zJ5tZjZm9bmZPmVnvtt6n\nqz6s4xXgq8Dz+U+a2fnAjcD5hKtbZ5rpBL52+Km7D8/dnowdJklyF+nNAK4ChgKTzey8uKkSrwnI\nuPtF7t78tGlp2wOE38d8twNPu/vngGeBH7b1Jl1S7u7+urtvA5oX93hgobsfcvcdwDaOPIde2qZ/\nENvv44v03L0BOHyRnrSfoU95azd3/y2wr9nT44GHcl8/BExo631i/wdo7eInKc40M9tgZnMK+d81\n+ZSWLtLT72DHOPAbM1trZv89dpiUOPXwGYju/g5walvfULKLmMzsN0C//KcI/5F/5O6Pl+rPqURH\n+9kCM4E73d3N7F+BnwLf7vqUIh8b7e57zOwUQslvzu2NSum0eSZMycrd3a9ox7fVAQPzHg/IPSd5\nivjZzgb0D2lx6oAz8x7rd7CD3H1P7v73ZvYrwuhL5d4x9YfX6zKz04D/ausbYoxl8ufDy4BJZnas\nmQ0GzgF0dL0Iuf/Qh00EXo2VJaE+vkjPzI4lXKS3LHKmxDKznmb2mdzXJwJXot/J9jCO7Mqbcl//\nPfBYW2/QJWvLmNkE4OdAX+AJM9vg7uPc/TUzewR4DWgAbtGawEW718yGEc5Q2AFMjRsnWVq7SC9y\nrCTrB/wqt9RID+AX7l4TOVOimNl8IAP8pZm9DdwB3A08ambfAnYSzjI8+vuoS0VE0if22TIiItIJ\nVO4iIimkchcRSSGVu4hICqncRURSSOUuIpJCKncRkRRSuYuIpND/ByTjFOvgW+4bAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x40b3f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sigmoid = lambda x: 1.0 / (1.0 + np.exp(-x))\n",
    "plt.figure()\n",
    "x = [float(i)/10 for i in range(-100, 100)]\n",
    "y = [sigmoid(i) for i in x]\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now interpret the hypothesis' output as a function of the sigmoid parametrized by $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convexity of the cost function\n",
    "\n",
    "Applying gradient descent or another method that relies on the convexity of the search function will not work with logistic regression if we're to use the same cost function\n",
    "\n",
    "$$\n",
    "J(h_\\theta(x), y)=\\sum^{m}_{i=1}{(h_\\theta(x^{(i)}) - y^{(i)})^2}\n",
    "$$\n",
    "\n",
    "since we're now using a nonlinear function. We need to encode two considerations into a mathematical expression:\n",
    "\n",
    "* if $P(y=1|x;\\theta) = 0$ and $y = 1$, a penalty process has to take place (large cost)\n",
    "* conversely if $P(y=1|x;\\theta) = 1$ and $y = 0$, a similar penalty is needed\n",
    "\n",
    "Therefore\n",
    "\n",
    "$$\n",
    "J(h_\\theta(x), y)=-\\frac{1}{m}\\sum^{m}_{i=1}{y^{(i)}log(h_\\theta(x^{(i)})) + (1 - y^{(i)})log(1 - h_\\theta(x^{(i)}))}\n",
    "$$\n",
    "\n",
    "This formulation has the following additional advantage\n",
    "\n",
    "$$\n",
    "\\frac{\\partial}{\\partial \\theta_j}J(h_\\theta(x), y) = \\frac{1}{m} \\sum^{m}_{i=1}{(h_\\theta(x^{(i)}) - y^{(i)}) x_j^{(i)}}\n",
    "$$\n",
    "\n",
    "this grants us the same formulation as the linear case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiclass classification\n",
    "\n",
    "Classification can occur among many different classes. Two approaches are commonly used:\n",
    "\n",
    "* One vs. all\n",
    "* One vs. many\n",
    "\n",
    "In the first approach the problem is decomposed into a number of binary classification problems that can in turn be solved via logistic regression. Given $k$ classes a problem needs to be solved via binary classification $k$ times. In the latter approach a classifier is constructed for each binary subset (generally slower).\n",
    "\n",
    "Suppose we have the following multiclass training set data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEACAYAAACnJV25AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFEFJREFUeJzt3X+QXXV9xvHn2SRLFmgg6pJUorvSjtFSKTKDpaWduVZS\no7aEsTPW2GnBbjv8UYRqxwHpONlxbEeYaayt8oeyMsGRFaW1kI4/gMGrgx0wFaKJxOiM3UDUXVeF\nVCC4S/LpH+ds2KS7m3vuPXfPPd+8XzN39tyz95zzETfPfvdzvuccR4QAAOnoq7oAAEC5CHYASAzB\nDgCJIdgBIDEEOwAkhmAHgMQUDnbbfbYfsX1P/n6t7Xtt77f9ZdtnlV8mAKBV7YzYr5P02Lz3N0i6\nPyI2SnpA0vvKKAwA0J5CwW57g6Q3S7p13uotknbkyzskXVFOaQCAdhQdsX9Y0nslzb9cdV1ETElS\nRExKOqek2gAAbWg52G2/RdJUROyW5CU+yj0KAKBCKwt89lJJl9t+s6QBSb9i+1OSJm2vi4gp2+sl\n/WShjW0T+ADQhohYajD9/7Q8Yo+IGyPi5RFxnqS3S3ogIv5c0k5JV+Ufu1LS3Uvso7avbdu2VV7D\nqVp/nWun/upfda+/HWXMY/+QpE2290t6Q/4eAFCRIq2YYyLiq5K+mi//XNJlZRYFAGgfV562qNFo\nVF1CR+pcf51rl6i/anWvvx1ut4dT+EB2LNexACAVthXdOnkKAKgHgh0AEkOwA0BiCHYASAzBDgCJ\nIdgBIDEEOwAkhmAHgMQQ7ACQGIIdABJDsANAYgh2AEgMwQ4AiSHYASAxRR5mfZrth20/anuP7W35\n+m22D9p+JH9t7l65AICTKXQ/dtunR8SztldI+rqkayW9SdIvImL7SbblfuwAUFDX78ceEc/mi6cp\ne6zeXFIXOigAoHsKBbvtPtuPSpqUdF9E7Mq/dY3t3bZvtX1W6VUCAFpW6GHWEXFU0mttr5H0edu/\nIekWSR+IiLD9QUnbJY0stP3o6Oix5UajcUo+ixAAltJsNtVsNjvaR9vPPLX9fknPzO+t2x6StDMi\nLljg8/TYAaCgrvbYbb9krs1ie0DSJknftb1+3sfeKmlvkQIAAOUq0or5VUk7bPcp+4VwZ0R8wfbt\nti+UdFTShKSryy8TAJbXzMy0nntuQqtXD6u/f7DqcgppuxVT+EC0YgDUxNTUuPbvH5Hdr4gZbdw4\npnXrtlZSSzutGIIdAOaZmZnWQw8N6ejRw8fW9fUN6JJLDlQycu/6PHYASN1zz03I7j9unb1Kzz03\nUU1BbSDYAWCe1auHFTFz3LqIWa1ePVxNQW0g2AFgnv7+QW3cOKa+vgGtWLFGfX0D2rhxrFYnUOmx\nA8ACemVWDCdPAdRSr4RoL2on2AvdUgAAytZLUwtTwYgdQGFljbB7bWphL2LEDqDryhxhvzC18IVg\nn5taWHawn0rtHoIdQMtmZqa1f/9IPsLOwnj//hGtXXtZW2G5XFMLT7V2D9MdAbSs7It3lmNq4fxf\nRkeOHNLRo4e1f/+IZmamSztGr2HEDqBl3Rhhr1u3VWvXXta1Nslytnt6BSN2AC3r1gi7v39Qa9Zc\n3JWgTeFK0qKYFQOgsLqdiHyhx75KEbO16rFzgRIALKJuv4zmEOwAkJhuPxrvNNsP237U9h7b2/L1\na23fa3u/7S/PPT4PAFCNQiN226dHxLO2V0j6uqRrJf2JpJ9FxM22r5e0NiJuWGBbRuwAUFDXH7QR\nEc/mi6cpmyoZkrZI2pGv3yHpiiL7BACUq1Cw2+6z/aikSUn3RcQuSesiYkqSImJS0jnllwkAaFWh\nC5Qi4qik19peI+nzts9XNmo/7mOLbT86OnpsudFoqNFoFDk8ACSv2Wyq2Wx2tI+2Z8XYfr+kZyX9\nlaRGREzZXi/pKxHx6gU+T48dAArq9qyYl8zNeLE9IGmTpH2S7pF0Vf6xKyXdXaQAAEC5Wh6x236N\nspOjffnrzoj4B9svkvRZSS+TdEDS2yLiqQW2Z8QOAAVxgRIAJKbr0x0BAL2PYAeAxBDsAJAYgh0A\nEkOwA0BiCHYASAzBDgCJIdgBIDEEOwAkhmAHgMQQ7ACQGIIdABJDsANAYgh2AEgMwQ4AiSHYASAx\nRR6Nt8H2A7a/Y3uP7Xfl67fZPmj7kfy1uXvlAgBOpsij8dZLWh8Ru22fKembkrZI+lNJv4iI7SfZ\nnicoAUBB7TxBaWWrH4yISUmT+fLTtvdJOnfu2EUOCgDonrZ67LaHJV0o6eF81TW2d9u+1fZZJdUG\nAGhDyyP2OXkb5i5J1+Uj91skfSAiwvYHJW2XNLLQtqOjo8eWG42GGo1GOzUDQLKazaaazWZH+2i5\nxy5JtldK+k9JX4yIjyzw/SFJOyPiggW+R48dAApqp8detBXzSUmPzQ/1/KTqnLdK2ltwn+jQ9DPT\n2vXDXZp+ZrrqUgD0gCKzYi6V9DVJeyRF/rpR0juU9duPSpqQdHVETC2wPSP2LhjfM66Re0bUv6Jf\nM0dmNLZlTFt/c2vVZQEoSTsj9kKtmE4Q7OWbfmZaQ/88pMPPHz62bmDlgA787QENnjFYYWUAyrIc\nrRj0kImnJtS/ov+4datWrNLEUxPVFASgJxDsNTZ89rBmjswct272yKyGzx6upiAAPYFgL6DXTlIO\nnjGosS1jGlg5oDWnrdHAygGNbRmjDQOc4k7ZHvv0M9OaeGpCw2cPtxSEvXySsuj/FgD1wcnTFhUN\naU5SAqgKJ09bMP3MtEbuGdHh5w/r0C8P6fDzhzVy98iS7RVOUgKok1Mu2NsJaU5SAqiTUy7Y2wlp\nTlICqJNTs8e+d1wjd49o1YpVmj0y2/KJUE5SAlhunDwt4MSQJrQB9CKCvU29PJURwKmNYG8DUxkB\n9DKmO7aBqYwAUnPKBztTGQGk5pQPdqYyAkhNkQdtbJB0u6R1yh6q8YmI+BfbayXdKWlI2YM23hYR\nhxbYvid77HOYFQOgF3X15Gn+CLz1EbE7f6D1NyVtkfROST+LiJttXy9pbUTcsMD2PR3sANCLunry\nNCImI2J3vvy0pH2SNigL9x35x3ZIuqJIAQCAcrXVY7c9rOw5pw9JWjf3jNOImJR0TlnFAQCKKxzs\neRvmLknX5SP3E/sr9FsAoEIri3zY9kplof6piLg7Xz1le11ETOV9+J8stv3o6Oix5UajoUajUbhg\nAEhZs9lUs9nsaB+Frjy1fbukn0bEe+atu0nSzyPiJk6eAkC5uj0r5lJJX5O0R1m7JSTdKOkbkj4r\n6WWSDiib7vjUAtsT7ABQEPeKAYDEcK8YAADBDgCpIdgBIDEEOwAkhmAHgMQQ7ACQGIIdABJDsANA\nYgh2AEgMwQ4AiSHYASAxBDsAJIZgB4DEEOwAkBiCHQASQ7ADQGJaDnbbY7anbH973rpttg/afiR/\nbe5OmQCAVhUZsd8m6Y0LrN8eERflry+VVBcAoE0tB3tEPCjpyQW+VeiRTQCA7iqjx36N7d22b7V9\nVgn7AwB0oNNgv0XSeRFxoaRJSds7LwkA0ImVnWwcEdPz3n5C0s6lPj86OnpsudFoqNFodHJ4AEhO\ns9lUs9nsaB+OiNY/bA9L2hkRr8nfr4+IyXz53ZIujoh3LLJtFDkWAECyrYgodC6z5RG77TskNSS9\n2PbjkrZJer3tCyUdlTQh6eoiBwcAlK/QiL2jAzFiB1BD09PSxIQ0PCwNDi7/8dsZsXPlKQAsYnxc\nGhqSNm3Kvo6PV11RaxixA8ACpqezMD98+IV1AwPSgQPLO3JnxA4AJZmYkPr7j1+3alW2vtcR7ABq\nZ3pa2rUr+9otw8PSzMzx62Zns/W9jmAHUCvt9r2L/jIYHJTGxrL2y5o12dexsWpOoBZFjx1AbbTb\n9x4fl0ZGstbKzEwW0Fu3tn7Mus2KIdgB1MauXdlI/dChF9atWSPdf7908cULb9MrJ0HbxclTAElr\np+9d55Og7SLYAdRGO33vOp8EbRetGAC1U7TvPddjX7UqC/UiPfaq0WMHgEWUfRJ0uU6qEuwAsAw6\nmWVTFMEOAF223LNsmBUDAF1Wh1k2BDsAFFCHWTYEOwAUUIdbDbTcY7c9JumPJE1FxAX5urWS7pQ0\npOwJSm+LiEOLbE+PHUAykpgVY/v3JD0t6fZ5wX6TpJ9FxM22r5e0NiJuWGR7gh0ACurqydOIeFDS\nkyes3iJpR768Q9IVRQ4OAChfpz32cyJiSpIiYlLSOZ2XBADoxMqS97dkr2V0dPTYcqPRUKPRKPnw\nAFBvzWZTzWazo30UukDJ9pCknfN67PskNSJiyvZ6SV+JiFcvsi09dgAoaDkuUHL+mnOPpKvy5Ssl\n3V1wfwCAkhWZFXOHpIakF0uakrRN0n9I+pykl0k6oGy641OLbM+IHQAK4l4xAJAY7hUDACDYASA1\nBDsAJIZgB4DEEOwAkBiCHQASQ7ADQGIIdgBIDMEOAIkh2AEgMQQ7ACSGYAeAxBDsAJAYgh0AEkOw\nA0BiCHYASEwpD7O2PSHpkKSjkmYj4nVl7BcAUFwpwa4s0BsR8WRJ+wMAtKmsVoxL3BcAoANlhXFI\nus/2Ltt/XdI+AQBtKKsVc2lE/Nj2oLKA3xcRD574odHR0WPLjUZDjUajpMMDQBqazaaazWZH+3BE\nlFPN3A7tbZJ+ERHbT1gfZR8LAFJnWxHhItt03IqxfbrtM/PlMyT9oaS9ne4XANCeMlox6yR93nbk\n+/t0RNxbwn4BAG0ovRWz6IFoxQBAYZW0YgAAvYVgB4DEEOx1MD0t7dqVfQWAkyDYe934uDQ0JG3a\nlH0dH6+6IgA9jpOnrZieliYmpOFhaXBweY87NCQdPvzCuoEB6cCB5a0DQGU4edoNVY6YJyak/v7j\n161ala0HgEUwYl9K1SPmqo8PoHKM2MtW9Yh5cFAaG8vCfM2a7OvYGKEOYEnpj9g76Y/3yoi5qh4/\ngMoxYj9Rp/3xXhkxDw5KF19MqANoSboj9jJH24yYAVSknRF7Wfdj7z1z/fH5wT7XHy8azoODBDqA\n2ki3FTM8LM3MHL9udjZbDwAJSzfYe6U/DgDLLN0e+xz64wBqrJ0ee/rBDgA1Vtl0R9ubbX/X9vds\nX1/GPpcdd1AEkIgynnnaJ+mjkt4o6XxJW22/qtP9LivuoAggIR23YmxfImlbRLwpf3+DpIiIm074\nXG+2Ynrl6lIAWEBVrZhzJT0x7/3BfF09VH0/GAAo2bJeoDQ6OnpsudFoqNFoLOfhF8Z8dwA9pNls\nqtlsdrSPsloxoxGxOX9fr1aMlPXUR0aykfrsbDbffevWqqsCgGqmO9peIWm/pDdI+rGkb0jaGhH7\nTvhc7wa7xHx3AD2pknvFRMQR29dIuldZz37sxFCvBe4HAyARXKAEAD2M+7EDAAh2AEgNwQ4AiSHY\nASAxBDsAJIZgB4DEEOwAkBiCHQASQ7ADQGIIdgBIDMEOAIkh2AEgMQQ7ACSGYAeAxBDsAJCYjoLd\n9jbbB20/kr82l1UYAKA9ZYzYt0fERfnrSyXsryd1+nDZqtW5/jrXLlF/1epefzvKCPZCT/aoq7r/\ncNS5/jrXLlF/1epefzvKCPZrbO+2favts0rYHwCgAycNdtv32f72vNee/OsfS7pF0nkRcaGkSUnb\nu10wAGBppT3M2vaQpJ0RccEi3+dJ1gDQhqIPs17ZycFsr4+IyfztWyXtLaswAEB7Ogp2STfbvlDS\nUUkTkq7uuCIAQEdKa8UAAHpD1688tb3Z9ndtf8/29d0+Xplsb7D9gO3v5CeNr626pnbY7ssvILun\n6lqKsn2W7c/Z3pf///DbVddUhO13296bTzj4tO3+qmtaiu0x21O2vz1v3Vrb99reb/vLvTz7bZH6\nb85/fnbb/jfba6qscSkL1T/ve39n+6jtF51sP10Ndtt9kj4q6Y2Szpe01farunnMkj0v6T0Rcb6k\n35H0NzWrf851kh6ruog2fUTSFyLi1ZJ+S9K+iutpme2XSnqXpIvySQUrJb292qpO6jZl/17nu0HS\n/RGxUdIDkt637FW1bqH675V0fj577/uqX/2yvUHSJkkHWtlJt0fsr5P0/Yg4EBGzkj4jaUuXj1ma\niJiMiN358tPKQuXcaqsqJv+BeLOkW6uupah8ZPX7EXGbJEXE8xHxvxWXVdQKSWfYXinpdEk/qrie\nJUXEg5KePGH1Fkk78uUdkq5Y1qIKWKj+iLg/Io7mbx+StGHZC2vRIv/9JenDkt7b6n66HeznSnpi\n3vuDqlkwzrE9LOlCSQ9XW0lhcz8QdTyZ8gpJP7V9W95K+rjtgaqLalVE/EjSP0l6XNIPJT0VEfdX\nW1VbzomIKSkb7Eg6p+J6OvGXkr5YdRFF2L5c0hMRsafVbbi7YwtsnynpLknX5SP3WrD9FklT+V8d\nVv1u/7BS0kWSPhYRF0l6VllboBZsn61stDsk6aWSzrT9jmqrKkUdBwmy/feSZiPijqpraVU+kLlR\n0rb5q0+2XbeD/YeSXj7v/YZ8XW3kf0LfJelTEXF31fUUdKmky23/QNK4pNfbvr3imoo4qGyk8t/5\n+7uUBX1dXCbpBxHx84g4IunfJf1uxTW1Y8r2Oim7dkXSTyqupzDbVylrSdbtF+uvSRqW9C3b/6Ms\nQ79pe8m/mrod7Lsk/brtoXw2wNsl1W1mxiclPRYRH6m6kKIi4saIeHlEnKfsv/0DEfEXVdfVqvzP\n/ydsvzJf9QbV6yTw45Iusb3atpXVX4eTvyf+dXePpKvy5Ssl9foA57j689uJv1fS5RHxy8qqat2x\n+iNib0Ssj4jzIuIVygY7r42IJX+5djXY81HKNcrOSn9H0mciog4/2JIk25dK+jNJf2D7Ue45X4lr\nJX3a9m5ls2L+seJ6WhYR31D2V8ajkr6l7B/rxyst6iRs3yHpvyS90vbjtt8p6UOSNtner+yX04eq\nrHEpi9T/r5LOlHRf/m/4lkqLXMIi9c8XaqEVwwVKAJAYTp4CQGIIdgBIDMEOAIkh2AEgMQQ7ACSG\nYAeAxBDsAJAYgh0AEvN/KXyJ6uQONAkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x80de1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "def generateMulticlassClassificationData():\n",
    "\n",
    "    x = [np.zeros(shape=4), np.zeros(shape=4), np.zeros(shape=4), np.zeros(shape=4)]\n",
    "    y = copy.deepcopy(x)\n",
    "\n",
    "    for i in range(0, len(x)):\n",
    "        x_offset = (i % 2) * 10\n",
    "        y_offset = i * 10\n",
    "        x1 = x[i]\n",
    "        y1 = y[i]\n",
    "        x1[0] = 1 + x_offset + np.random.uniform(0, 2) - 1\n",
    "        y1[0] = 2 + y_offset + np.random.uniform(0, 2) - 1\n",
    "        x1[1] = 2 + x_offset + np.random.uniform(0, 2) - 1\n",
    "        y1[1] = 3 + y_offset + np.random.uniform(0, 2) - 1\n",
    "        x1[2] = 2 + x_offset + np.random.uniform(0, 2) - 1\n",
    "        y1[2] = 1 + y_offset + np.random.uniform(0, 2) - 1\n",
    "        x1[3] = 3 + x_offset + np.random.uniform(0, 2) - 1\n",
    "        y1[3] = 3 + y_offset + np.random.uniform(0, 2) - 1\n",
    "\n",
    "    return x, y\n",
    "\n",
    "def plotScatterData(x, y):\n",
    "    color = {\n",
    "        0: 'r',\n",
    "        1: 'b',\n",
    "        2: 'g',\n",
    "        3: 'y'\n",
    "    }\n",
    "    for i in range(0, len(x)):\n",
    "            plt.scatter(x[i], y[i], color=color[i])\n",
    "    plt.show()\n",
    "\n",
    "x, y = generateMulticlassClassificationData()\n",
    "plotScatterData(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code implements a *one vs. all* logistic regression approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4FcX6wPHvJBAISeiQKCIgghQhhK4RDSIIioCoIBaK\nXGxXrhcVBcsP7GDHgqJXJSiCFBEQRWqkKKIgiFKkJQKSAgFCAunv749NJSch5SR7Ts77eZ48ZPfM\n2X2TE96dndmZMSKCUkqpys/L7gCUUkpVDE34SinlITThK6WUh9CEr5RSHkITvlJKeQhN+Eop5SGc\nkvCNMeOMMX8YY343xsw2xvgYY+oYY1YYY/YYY743xtRyxrmUUkqVTpkTvjHmQmAs0FFE2gNVgGHA\nBGCViFwGrAEmlvVcSimlSs9ZTTregJ8xpgrgCxwBBgLhWa+HA4OcdC6llFKlUOaELyL/AK8Df2Ml\n+lMisgoIFJGYrDLRQMOynksppVTpOaNJpzZWbb4JcCFWTf9O4Nw5G3QOB6WUslEVJxzjOuCAiMQD\nGGMWAVcCMcaYQBGJMcYEAbGO3myM0QuBUkqVgoiYkpR3Rhv+30B3Y0x1Y4wBegE7gSXAyKwyI4DF\nhR1ARNz2a9KkSbbHoPHbH4cnxu/OsVeG+EujzDV8EdlsjFkA/AakZf37IRAAzDPG3ANEAUPKei6l\nlFKl54wmHUTkWeDZc3bHYzX3KKWUcgE60raMwsLC7A6hTDR+e7lz/O4cO7h//KVhStsW5LQAjBG7\nY1BKKXdjjEFs6LRVSinlBjThK6WUh9CEr5RSHkITvlJKeQhN+Eop5SE04SullIfQhK+UUh5CE75S\nSnkITfhKKeUhNOErpSqVmMSYUs8mWdlpwldKVQppGWkEvhZI0OtBxCTF2B2OS9KEr5Rye2sPrsXn\nBR9ik2JZM3wNQf5BdofkkpwyPbJSStlBROjxaQ82HtpIoF8gh8Ydoqp3VbvDclma8JVSbmlHzA7a\nf9AegC8Gf8GwdsNsjsj1acJXSrmdkV+PJHx7OAAJExIIqBZgc0TuQRO+UsptHE44TOM3GwPwynWv\nMD50vM0RuRdN+Eopt/DcD88xKWISAEcfPaods6WgCV8p5dJOJp+kztQ6ADzQ+QGm3zjd5ojclyZ8\npZTL+vS3T7lnyT0A7Pr3LlrVb2VzRO5NE75SyuWkpKdQ75V6JKUl0ad5H5bfuRxjSrR8q3JAE75S\nyqUs37ecfrP7AbBh1AZCLw61OaLKQxO+UsolZEomnT/szG/Rv3FJnUv466G/8PbytjusSkUTvlLK\ndluPbqXTh50AWDhkIYNbD7Y5ospJ59JxAm9vbzp27Ei7du0YOnQoycnJpT7WDz/8wE033QTA0qVL\neeWVVwote+rUKd5///1Sn6soe/bsISQkhE6dOnHw4MFSHWPatGml+l1MmjSJNWvWFFnmfL+b8rJ9\n+3a+++67YpePiorC19eXjh075uxbvnw5rVq1omXLlkydOtXh+5YsWUJwcDAhISF07dqVjRs3ApCc\nnExISAjVq1cnPj6+bD+Mixi6YGhOsk96MkmTfXkSEVu/rBDcW0BAQM73d955p7z55psFymRmZhbr\nWBEREXLTTTcVq+zBgwfl8ssvL16QWdLT04tVbsqUKfLiiy+W6Njn/oxNmzaV48ePOyybkZFRomO7\nipkzZ8pDDz1U7PKRkZHSrl27nO2MjAxp3ry5REZGSmpqqgQHB8uuXbsKvC8pKSnn+99//11atWqV\n7/VmzZoV+rt1FwdPHBQmI0xG3t70tt3huJ2s3FmifKs1fCfr0aMH+/btIyoqilatWjFixAjatWvH\n4cOHWblyJVdeeSWdO3dm6NChnDlzBrBqfK1bt6Zz58589dVXOccKDw9n7NixAMTGxjJ48GA6dOhA\nSEgImzZtYuLEiezfv5+OHTvyxBNPADB+/HjatWtHcHAw8+bNA6y7hquvvpqBAwfStm1bzpw5Q//+\n/QkJCaF9+/bMnz8/38/w3Xff8dZbb/H+++/Tq1cvAN544w3atWtH+/btmTZtGoDDnzHbO++8wz//\n/EPPnj1zjhEQEMBjjz2WE//zzz9P165dad++Pffff3/Oe0eNGpXze2jWrBmTJ0+mU6dOBAcH89df\nfxX43YwaNYqHH36Y0NBQLr300pz3iggPPvggbdq04frrr+fGG2/M9/vN9vbbb9O2bVs6dOjAHXfc\nAcCZM2cYPXo03bt3p1OnTixdupS0tDT+7//+j3nz5tGxY0fmz5/PunXrCAkJoWPHjnTq1ImkpKQi\n/z42b95MixYtaNKkCVWrVuX2229n8eLFAMyYMYMPP/wQgBo1auS8JzExES+v/P9Vxc3ne5+4aiLN\npjUDIPaxWMZ2G2tzRB6ipFcIR19ALWA+sAv4E+gG1AFWAHuA74Fahby3XK+CFcHf319ERNLS0mTg\nwIHywQcfSGRkpHh5ecnmzZtFROTYsWNy9dVXy5kzZ0REZOrUqfL8889LcnKyNG7cWPbv3y8iIkOG\nDMmp4c+cOVPGjh0rIiJDhw6VadOmiYhVk05ISChQe1y4cKH06dNHRERiYmLk4osvlujoaImIiBB/\nf3+JiorKKXfvvffmvC8hIaHAzzR58mR5/fXXRURky5Yt0r59ezl79qwkJiZK27ZtZdu2bRIZGSne\n3t45P+O5mjVrJvHx8TnbxhhZsGBBzvaJEydyvr/77rvlm2++ERGRkSNHysKFC0XEukt47733RERk\n+vTpMmbMmAK/m5EjR8qQIUNERGTnzp1y6aWXiojI/Pnz5cYbbxQRkejoaKlTp07OcfO68MILJTU1\nVURETp06JSIiTz75pMyePVtERE6ePCktW7aUM2fO5DuviMhNN90kP/74o4hYtfJz71zO/YwWLFiQ\n8zOIiHz22Wf5jpfXokWLpFWrVlKvXj3ZtGlTvteKuntyZceSjuXU6h/9/lG7w3Fr2FjDnwZ8KyKt\ngWBgNzABWCUilwFrgIlOOpfLOXv2LB07dqRr1640adKE0aNHA9C0aVO6dOkCwKZNm9i5cyehoaGE\nhIQwa9YsoqKi2L17N5dccgmXXHIJAHfddZfDc6xZs4YHHngAAGMMAQEFJ4vasGEDw4ZZMwY2bNiQ\nsLAwfvnlFwC6du3KxRdfDEC7du1YuXIlEydOZMOGDQ6Pde5xb775ZqpXr46fnx+DBw9m/fr1ADRp\n0iTnZzyX5F7UAahSpQqDB+e2z65evZru3bvTvn171q5dy59//unwODfffDMAnTp1IjIy0mGZQYMG\nAdC6dWtiY2MB2LhxI7fddhsAgYGB9OzZ0+F7g4ODueOOO5g9ezbe3tZTIStWrGDKlCmEhIQQFhZG\namoqf//9d4H3hoaGMm7cON555x1OnDhRoCZeFoMGDWLXrl18/fXXPP300047rl3e/+V96r9aH4B9\nY/fxWp/XbI7I85T5KR1jTE2gh4iMBBCRdOCUMWYgcE1WsXAgAusiUOnUqFGDrVu3Ftjv5+eX872I\n0KdPH2bPnp2vzPbt24t1e16aQSd5j5s3lhYtWrB161a+/fZbnn76aa677rpSJ5S8xz2f6tWr5/wc\nKSkp/Pvf/2br1q1ceOGFPPvss4V28FarVg2wOsfT09OLLAMlb+5YtmwZ69atY8mSJbz44ovs2LED\nEWHhwoW0aNEiX9lNmzbl237iiSfo378/y5YtIzQ0lBUrVtCyZctCz9WoUaN8F47Dhw/TqFGjIuO7\n6qqrOHDgAPHx8dStW7dEP5srOJt2Fr+X/BCEm1vdzFdDCzarqYrhjOpIM+CYMeZTY8xWY8yHxpga\nQKCIxACISDTQ0AnnckmFJZi8+7t3787GjRvZv38/YLUR7927l1atWhEVFZXzJMycOXMcHqtXr15M\nn27NIZKZmUlCQgIBAQGcPn06p0yPHj348ssvyczMJC4ujvXr19O1a9cCxzp69Ci+vr7ccccdjB8/\n3uHFKq8ePXrw9ddfk5ycTFJSEosWLaJHjx5F/uwANWvWJCEhweHvIzk5GWMM9erVIzExkQULFhQZ\nQ0lknyc0NJSFCxciIsTExBAREeGw7N9//80111zDlClTSEhIICkpieuvv5633347p9y2bdsAqx8i\n78904MAB2rZty+OPP06XLl3YvXt3kbF16dIlp48nNTWVuXPnMmDAAADee++9nM84++8EYOvWraSm\nprplsl+yZwk1XqqBIGz+12ZN9jZzxnP4VYCOwL9F5FdjzJtYNflzM4F79zIVobDad9799evXZ+bM\nmQwbNoyUlBSMMbzwwgu0aNGCGTNmcMMNN+Dn50ePHj1ITEwscKy33nqLe++9l48//pgqVarw/vvv\n061bN6688krat29Pv379mDp1Kj/99BPBwcF4eXnx6quv0rBhQ3bt2pXvWDt27GD8+PF4eXnh4+Nz\n3kc7Q0JCGDlyJF26dMEYw7333ktwcDBRUVFF3nmMGTOGvn370qhRI1avXp2vbK1atRgzZgxt27bl\nggsuyHdhyluuOHc255bJ3r7llltYs2YNbdu2pXHjxnTq1IlatWrlK5uRkcFdd91FQkICIsLDDz9M\nzZo1eeaZZ/jvf/9L+/btERGaNWvGkiVL6NmzJ1OmTKFjx45MnDiR9evXs3btWry9vWnbti39+vUr\nMlZvb2/effdd+vTpQ2ZmJqNHj6Z169YA7N69m6uuugqAhQsXMmvWLHx8fPD19c3pgHcXGZkZtJne\nhr+O/8XlDS9n+/3b8TL6jIjdTElvfwscwJhA4CcRuSRr+yqshN8cCBORGGNMELA2q43/3PfLpEmT\ncrbDwsIICwsrU0xKZUtKSsLPz4/4+Hi6devGxo0badiw4m42o6Ki6N+/Pzt27Dhv2QEDBvDVV19R\npcr562HNmjVjy5YtLlnr33R4E1d8fAUA3wz7hhtb3mhzRJVDREREvrvUZ599FhEpUVtvmRM+gDHm\nB2CMiPxljJkEZD9TFi8iU40xTwB1RKRAG74xRpwRg1KO9OzZk5MnT5KWlsYTTzzB3XffXaHnP3z4\nMFdeeSX169c/b9NZcSQnJ3PFFVdw/Phxfv/9d2rXru2EKJ1DRLhpzk0s27uMat7VODnhJNWrVLc7\nrErLGGNbwg8G/gdUBQ4AowBvYB7QGIgChojISQfv1YSvlJvbe3wvLd+1Oqs/7P8hYzqNsTmiys+2\nhF8WmvCVcm8Pf/cwb2+2OrjjH4+njm8dmyPyDKVJ+Dp5mlKqVGKTYgl8LRCAp3s8zfPXPm9zROp8\nNOErpUrszZ/e5JEVjwAQ+XAkTWo3sTkiVRya8JVSxZaUmoT/y/4A3NHuDmYPnn2edyhXoglfKVUs\n8/6cx9AFQwH47b7f6BDUweaIVElpwldKFSk9M51m05pxOOEwXRt1ZdPoTbq+rJvShK+UKtS6qHVc\nM9OaEmvFXSvo3by3zRGpstCEr5QqQES4dta1RERGUKd6HaIfi8bH28fusFQZacJXSuWzM24nbae3\nBWDWoFncHVyxo5NV+dGEr5TKce/Se/lo60cAnJpwiprVatockXImTfhKKf45/Q+N3rDm5X/x2hd5\nsseTNkekyoMmfKU83EvrX+KpNU8BcOSRI1wYcKHNEanyoglfKQ+VkJJArSnW+gD/CvkXHw34yOaI\nVHnThK+UB/ps+2cM/3o4AH8++CdtGrSxOSJVETThK+VBUjNSCXotiBPJJ+jZtCerh6/WQVQeRBO+\nUh5i1YFV9P7MGjj1w8gfuLrJ1TZHpCqaJnylKjkRofvH3dl8ZDONAhoR+d9Iqnjpf31PpJ+6UpXY\n9ujtdJhhTXL25a1fMqTtEJsjUnbShK9UJXXXV3cxe4c1fXHixET8fPxsjkjZTRO+UpXM36f+pslb\n1oIkb/R5g3FXjLM5IuUqNOErVYn839r/4/l11lKDMY/F0NCvoc0RKVeiCV+pSuDE2RPUfaUuAP/p\n+h+m9Ztmc0TKFWnCV8rN/W/r/xizdAwAfz30Fy3qtbA5IuWqNOEr5aaS05OpPaU2KRkp3NjiRpYO\nW6qDqFSRNOEr5YaW/bWM/nP6A/DT6J/oflF3myNS7kATvlJuJFMyCf4gmD9i/6BlvZbsfHAn3l7e\ndoel3IQmfKXcxK///EqXj7oAsPj2xQy4bIDNESl3owlfKTcw+MvBLNq9CC/jReLERHyr+todknJD\nmvCVcmEHThyg+dvNAZh+w3Qe6PKAzREpd6YJXykXNX7FeF776TUAjo0/Rr0a9WyOSLk7pyV8Y4wX\n8CtwWEQGGGPqAF8CTYBIYIiInHLW+ZSqrI6dOUaDVxsAMCF0Ai9f97LNEanKwsuJx3oY2JlnewKw\nSkQuA9YAE514LqUqpXc3v5uT7A/854Ame+VURkTKfhBjLgI+BV4EHsmq4e8GrhGRGGNMEBAhIq0c\nvFecEYNS7uxM2hn8XrJms7ytzW3Mu22ezREpV2eMQURKNNLOWU06bwLjgVp59gWKSAyAiEQbY3QW\nJ6UcWLRrEYPnDQZgy71b6HhBR5sjUpVVmRO+MeZGIEZEthljwoooWmg1fvLkyTnfh4WFERZW1GGU\nqhwyMjNo8U4LDp48SEhQCL/e+ytexpmtrKoyiYiIICIiokzHKHOTjjHmJeAuIB3wBQKARUBnICxP\nk85aEWnt4P1u3aQjIjp/iSqxHw/9SOgnoQB8d+d39L20r80RKXdTmiYdp7Th5wngGuDRrDb8V4Dj\nIjLVGPMEUEdEJjh4j9sm/NOnt7FlS0jOdo0abQkKGkFg4J1Uq3ahjZEpVyUi9Jvdj+/3f4+/jz/H\nxh+jWpVqdoel3JCrJfy6wDygMRCF9VjmSQfvcduED5CcfJjY2NlER4dz5syuAq8b40NQ0HACA0dQ\nq1ao3g14sD3H9tDqPeu5hU8GfMKokFE2R6Tcme0JvzTcPeE7kpmZwvHjy4iOnsXx44sdlqlV6xqC\ngkbQoMGtVKkSUMERqor272X/Zvqv0wE48cQJalevbXNEyt1pwndxiYk7iImZRXR0OGlpcQVe9/G5\nIKtJaAR+fgWeYFVuKDoxmgtevwCAyddMZlLYJJsjUpWFJnw3lJZ2kri4+URHh5OQsNFhmfr1BxMU\nNIK6dfvh5VW1giNUpfXqxld5fNXjABwad4iLal5kc0SqMtGEX0mIZHLy5DpiYsKJjg7H0ROtfn6X\nExg4gsDAu6hWLajig1SFSkxNJOBlq5luePBwwgeF2xyRqow04Vdyycl/ExPzOdHRszh7dk+B142p\nRlDQSIKCRlCzZnftILbBnB1zuOOrOwD4/f7faRfYzuaIVGWlCd8DZWQkc/z4N8TEhHP8+Df5Xqtd\nuyf+/sH4+3fAzy8YP782eHn52BRp5ZaWkUbjNxsTkxRDaONQ1o9ab+sFNzU1juTkSKpXb4qPTwPb\n4lDlRxO+ypGUtJuUlCgSE7eRmLidxMRtJCcfxNe3Jf7+HfD3D8bHJ4iAgE7UqHGZ3eG6tbUH13Lt\nrGsBWD18Ndc2u7bCY8ib4E+cWMWePaMxxgeRVC677GMCA4dVeEwVTQRefhmeeQaOHoWGlXwyF034\nqkgZGWdJSvqTxMRtJCVt58iRdwuUadDgNoKCRlCnzvV4eelyCUUREa6eeTUb/t5AQ7+GHB53mKre\nFd+pHhMzJyfBZ2amAJmIpOa87uXlS/fuUZW2pn/0KPTtC7//bm23awe//QbelXypX034qsREMjh5\nMoLo6FnExMxyWMbPLzjrcdE78PEJrOAIXdMfsX/Q7n2rfX724Nnc0e4OW+JITY1j06YmZGaeLbSM\nt3dNgoNXUbNmlwqMrPwtXAi33pq7/dRT8Nxz4OUh0xFpwldOk5wcRXT0Z8TEhHP27L4Cr3t51cgZ\nQVyzZjeP6iC+Z/E9fLrtUwASJiQQUK18Bs4Vpx0+IeEXtm/vTUZG4WsLVaYa/tmzMHIkzMsze/Sm\nTdCtm20h2UYTvipXGRlnOX58CdHRs4iP/9Zhmdq1e2WNIB6Mt7dfBUdYvg4nHKbxm40BmHrdVB4P\nfbzczpW3maaodnjHNfyqeHlVwZiqiKRVijb8X3+FLnluUG65BcLDwa9y/YmViCZ8VeFEhMTEbURH\nhxMTE056eoHpkqhW7eKsJqHh1KhxqQ1Rlt1zPzzHpAhrlOzRR48S5F9+Yx8cJfGiaum5F4fcBF+n\nznUkJ0fi7e1PRkaiWz6tk5kJzz5rNdNkmzMHbr/dvphciSZ85TLS0o4TG/sl0dHhnD692WGZBg2G\nZnUQ93bZDuJTyaeoPdWa9+aBzg8w/cbp5X5OR80052uHd9T8U9y7BFdz6BD07g17soaatGkDK1ZA\no0b2xuVqNOErlyaSwYkTq4mOnkVs7GyHZfz9OxIUNIKGDYfZXiOduW0moxZbM1ru+vcuWtWvmPmN\nSlrDL69jVLQvvoA778zdfvZZ6xFLD+oeKhFN+MotnT17kJiYz4iODic5+UCB1729/QkMHEFQ0AgC\nAjqXewdxSnoK9V+tT2JqIn2a92H5ncsrvFPaUTNNSWrnpblLsENiItx1FyzOmlTWyws2b4ZOneyN\nyx1owleVRkbGWY4dW0xMTDjx8csdlqlTpw9BQcOpX/9mvL1rOOW83+/7nr6zrdWnNozaQOjFoU45\nbmmUZbSsq9fwf/oJrrwyd/v22+GTT8DX176Y3I0mfFWpWR3EW4mODic6epbDRxGrV29KYOBwgoKG\n4+vbvNjHzpRMunzUha1Ht9KsdjP2jt2Lt5d7j9wp612Cs2VmwpNPwtSpufsWLoTBg20Lya1pwlce\nKTX1GLGxc4mJCef06V8dlmnY8HYCA0dQt25vjMmfyLce3UqnD602hAW3LeCWNreUe8wVxRXm1ImK\ngp494eBBazskBL79FoJ0ktcy0YSvVBarg3gV0dHhxMbOcVjG378T30Vn8tr230hIh6Qnk6hR1TlN\nQwpmzoRReVZxfOklmDBBO2GdRRO+Uudx9ux+oqM/48jRj0lPPVzgdW/vgJxVxwICOnnUCGJnSEiw\n2uO/+87arl7dGgkbHGxvXJWRJnyliuHJ1U/y8oaXAYh9LJa61f04dmwR0dHhnDix0uF76tS5nqCg\nEdSvPwhvb+1ZPNe6dXDNNbnbI0bAjBlQrZp9MVV2mvCVKkL82XjqvVIPgEe6P8Lr179eaFkR4fTp\nX3NGEGdkJBYoU736JTkjiH19m5ZX2C4rIwPGj4c338zdt2QJ3HSTfTF5Ek34ShXig18/4IFlDwCw\nb+w+mtct/hM8eaWmxhIbO4fo6FkkJm51WKZhwzuzRhBfW6CDuDLYv9+qzR85Ym13724l+gb2P+3p\nUTThK3WO5PRk/F/yJ0MyGNRqEIuGLnL6OTIz0zlxYiXR0eHExX3psExAQNesEcS3U7VqXafHUBE+\n/BDuuy93+/XXYdw47YS1iyZ85XLikuKIPBlJ09pNaeBXsVXApXuWMmDuAAA2/2szXRpV7AjTM2f2\nEhMzi+joWaSk/F3g9SpVaueMIPb37+CSHcQnT1pzzq9ebW3Xrg0bN1rz2yh7acJXFaK4SXzOjjmM\nXjIaH28fUjNS+Xjgxwy7vPwH/mRkZnD5+5ez+9huLm94Odvv346XcY1VMdLTE3M6iE+eXO2wTN26\nNxAUNIJ69Qbg7V29giO0rFkDvXrlbt97L7zzDvjoksguQxO+KnfnJvE3r3+Tjhd0LJD845LiaPJW\nE86m5w7t963iS9R/o8q1pv/z4Z/p/nF3AJYOW0r/lv3L7VzOYnUQb84aQRxOZuaZAmV8fS/Nuhu4\nm+rVm5RLHOnp8N//wnvv5e5bvhyuv75cTqfKSBO+KleOkjhAgE8A6Znp+Wrwvxz5hd6f9eZUSu70\nBzWr1WTV3avKpWlFRBgwdwDf/PUNPt4+nJpwiupV7KkdO0tKSnRWB3E4SUnbHZaxppEYQe3aYZhS\n3sXs2QM9ekBcnLV99dWwaBHUdc+uBo+hCV+VK0dJPK+8NfiKrOHvi99Hi3daADCj/wzu7XSvU4/v\nSjIz0zhxYkVWB/F8h2Vq1uxOYOAIGjYcStWqdRyWEbFq8mPH5u577z148MHyiFqVB1sSvjHmImAW\nEAhkAh+JyNvGmDrAl0ATIBIYIiIFMoUmfPdRWA0/27k1+Dl/zGH04tFU8apCakYq0/pO477O9zl8\nb2mNWz6Ot35+C4D4x+Op4+s4wVV2Z87syVmDOCWl4AjiKlXqERQ0Al/fEdx+e3s2bLD2N2xoDZq6\n7LIKDliVmV0JPwgIEpFtxhh/YAswEBgFHBeRV4wxTwB1RGSCg/drwncjeZP46dTT+V5zVIOf8esM\nHl7+MD7ePgWafcoiNimWwNcCAXiqx1O8cO0LZT5mZWN1EH+V1UG8xmGZevX6Exg4gvr1b8LLS4fF\nuhOXaNIxxnwNvJv1dY2IxGRdFCJEpMCSQZrw3U/2Uzpbo7cybvk4qnpXJS0jrUAyL69mnbc2vcW4\n78cBEPlwJE1ql08nprtLTYWHHoKPPsrdt3q10KnTppwRxJmZyQXe5+vbMmsE8V1Ur35xBUasSsL2\nhG+MaQpEAJcDh0SkTp7X4kWkQDeQJnz3VtQjms7uuE1KTcL/ZX8A7mh3B7MHO14m0dP9+ae1uEhC\ngrV93XUwf771DH1hrA7iL7I6iH93WCZ7zEDt2teUuoNYOU9pEr7TVo7Oas5ZADwsIonGmHOzeKFZ\nffLkyTnfh4WFERYW5qywVDlr4Neg0Np609pNSc1IzbcvLSONprWblvg8C3Yu4Lb5twHw232/0SGo\nQ4mPUZmJwBtvwGOP5e778EMYM6Z4769WLYjGjR+hceNHcvZlZqYRH/8d0dGzOHZsITEx1l1BXjVr\nXklQ0AgaNBhC1apFXFFUmUVERBAREVGmYzilhm+MqQJ8A3wnItOy9u0CwvI06awVkdYO3qs1/Eos\nu82/sGaf80nPTOeSaZdwKOEQXRt1ZdPoTS45ItUucXHQv7+1DizARRdBRAQ0L91UQcWSlLQrawRx\nOKmpRwu8XrVq/TwjiNuVXyAezrYmHWPMLOCYiDySZ99UIF5EpmqnrWcr7fQK66PWc/XMqwFYcdcK\nejfvXV4rhxZtAAAXtUlEQVQhup2lS2HAgNztRx6BV14Bb5vmaktPTyAubiHR0eGcOvWDwzL16g0g\nKGg49er11w5iJ7DrKZ1QYB2wA6vZRoAngc3APKAxEIX1WOZJB+/XhK/yERGu++w61hxcQ53qdYh+\nLBofbx3Tn5JiTXEwa1buvh9+sAZKuSIR4dSpjcTEWCOIRdIKlKlRoxWBgdkdxBfZEKX7sr3TtjQ0\n4au8dsXtos10a2au8EHhDA8ebnNE9tu+Hbp1sxI+wA03wJw5ULOmvXGVVkrKEWJirA7iM2f+dFDC\nm6CgEQQFDadWrR7aQVwITfjKrd3/zf3M2DIDgFMTTlGzmptmNCcQgZdfhqeeyt336acwcqRtIZWr\nzMxUjh//lpiYcI4d+9phmVq1rsoaQTyEKlU8928jmyZ85Zb+Of0Pjd5oBMCL177Ikz2etDki+0RH\nQ9++Vq0e4JJLrKmJmza1NSzbJCX9SXS01UGclhZT4PWqVQOz7gZG4OfnWXM2a8JXbufl9S/z5Bor\nwR955AgXBlxoc0T2WLjQmnc+24QJ8OKL4KWtGQWkp58iLm5BVgfx+gKve3vX5IorjlClir8N0VUc\nTfjKbZxOOU3NKdZt+eiQ0fxvwP9sjqjinT0L99wDc+fm7vvxR7jiCvticlcimZw6tSGrX2An7dt/\nX+mbfTThK7fw+e+fc/eiuwH444E/aNuwrc0RVawtW6BLF6udHuDmm+Gzz8DPz964lHuxdaStRzp8\nGDp2tKYa7N3bGsPetStU0V+rI2kZaQS9HkT82XjCmoaxZvgajxlEJQLPPQd5BpXzxRcwrBTzyMXF\nQWSk1a6vC4erktAaflkcO2YNacyetMSRxo2ti0Hv3taacR76P3T1gdVc99l1AESMiOCaptfYHFHF\nOHLE+uh37bK2W7eGFSusEbGlMWcOjB5tLTWYmgoff3z+i4ZeIConbdJxBcePWwuCrlxpfUVGFl2+\nVy/rzqB3bwgJqXS9dCLCFR9fwc9HfqZRQCMi/xtJFa/Kfwc0d27+RDx5MjzzTNk+3rg4aNLEavvP\n5usLUVGFJ/LSXCCUe9CE7+oyMqwG3JUrYdUqa9KTojRvnnt30LMn1HGvxT1+j/md4A+CAZh7y1yG\nXj7U5ojKV1IS3H23tTwggDHWHDedOzvn+L/8Yv0pnMqzjFDNmtafUhcHk4/u2mXVIbIHbMH5LxDK\nfWjCd3dHj1oPXa9aZV0U/vmn8LJVq+ZeDK67Dtq2tTKMi7h70d18/vvnACROTMTPp/L2SG7alP/J\nmqFDrUFSvr7OPU9cnNUUlJpnAlIfH6sr6dwEPmcOjBqVP9lD0RcI5V404Vdmqanw889WA/Dq1fDT\nT0WXDw6GKVOskTtNm1qZoQL8fepvmrxlLUjyWu/XePTKRyvkvBUtMxOeftoaDZttwQK45ZbyO2dc\nHDRqBGl5pqSpWtXqJ8ib8B01/WTTGn7loQnfk0VFWReC7L6D48etmv+BA1YVMCjISv7Nm+f+GxAA\nl14KLVo45e5g0tpJPLfuOQCiH40m0D+wzMd0NVFRVrfL/v3WdnAwfPcdXHBB+Z+7uE06jsoBVKtm\n3XloG37loAlfOZaWBocOWcl///7cfxcudFw+JCS3uSg09LxtEyfOnqDuK9ZiZmO7juXtfm87+yew\n3axZMGJE7vZLL1mjYSuyFa24nbaOylWrBr/9Zj0lpCoHTfiq5ERg377cjuSVKyExsfDy9erlPlXU\nuzf/O7aCMUutZZX2PLSHlvVaVlDg5e/0aas2vGyZtV2tmtWqFhxsX0zZT91UrWpdxwt76qa45ZT7\n0oSvnC8x0Zp0Pfti8Kc1nW2KN9SeAMlV4Ya/4JsvwAB0757bkdy9e4X1HTjThg3Qo0fu9vDhMGMG\nVK9uX0x5Ffe5en3+vnLThK8qxLd7v+XGL24E4Meen3PF1rjcvoO0gotc5Ljggty7g+uuq5iG72LK\nyIDHH7fWhc329dcwcKB9MSlVFE34qlxlSiYdPujAjtgdtKzXkp0P7sTbq4g19U6ehLVrc5uL9u4t\n+gTXXJN7QejcuULW6ztwwDrt4cPWdpcu8M030LBhuZ9aqTLRhK/Kza///EqXj6xHQRYNXcSgVoPK\ndsDMTNi2LffOYPXqoss3bZp7Z9Crl9WXUAYffWQtF5jttdesdWFdaCiDUkXShK/KxS3zbuGrXV9h\nMCQ9mYRvVSePKHIkNjb/FBWHDhVe1pj8g9Dat3c4h8GpU9ac86tWWdsBAdZ0xJdfXk4/g1LlSBO+\ncqoDJw7Q/O3mALx3w3s82OVBmyPKkp5uPWye3VS0vuAiGHmtuWg4vQ6H52z/61/w3ntu2Z+sVA5N\n+MppHl/5OK/++CoAx8Yfo16NsjWhVKgjR0j/fjXjpgbx7l99cnZ/Sz/6sTx/2WrVcu8OeveGVq20\nXUe5BU34qsyOnTlGg1etZ/gev/JxpvaeanNEJfPXX3DVVdYjiWB9//XXeZr8U1KsaSmyHzPdvLno\nA7Zrl3sx6NFDVylRLkMTviqTdze/y9jvxgJw4D8HaFanmc0RFd9778FDD+Vuv/22tV3iyvqBA9bF\nIPuCcPJk4WVr1co3CI1LLilV7EqVhiZ8VSpn0s7g95JVc721za3Mv22+8w4eF2eN6QdrygYnjgCK\nj7eWB1y3ztquX98aNHXZZU47RX5nzlgnyO5I3r696PKdO+deEEJDreYjpZxEE74qsUW7FjF43mAA\nfh3zK50u7OS8g8+ZY01Akz0Yy8cHZs4s8xj/77+Hvn1ztx98EN56y5pGwDYisGdP7p3BypWOp6vM\n1qBB/ieLSrsElvJYmvBVsWVKJld+fCU/H/mZDkEd2HLvFryME1fbiouDiy+G5OT8+0s5P29amtVE\n8+GHuftWrbIeyXcLCQnWgjfZF4Tdu4suHxqae0HQdZKVA5rwVbEcOnWIUYtH8fORn5l36zz6tejn\n/JP88ou1SldSUv79fn7W6NtirsCxcydceWXuVL+9elnzzteu7eR47ZSZCX/8kXtnsGqVNddDYS66\nKP8gtMDKNw21Oj9N+KpIIsLsHbN55PtH+G/3//J46OO568s6e6atMtTwRawmmkceyd33wQdw331l\nD8stxcfnDkJbtcrqWC5Kz565dwchIRUyRYWqeKVJ+IhIuX4BfYHdwF/AEw5eF1X+4pLi5NZ5t0qb\n99rI1n+25n/xiy9EfH1FatWy/v3ii6IPFhsrsnmz9W9RvvhCpGpVESuHi/j4FHnsuDiR7t1zi194\nocjevcX8AT1Verr1Wbz4okjPnrm/vMK+mjcXue8+kQULROLj7Y5elUFW7ixRPi7XGr4xxisr0fcC\n/gF+AW4Xkd15ykh5xqCs2S3HLB3D7W1v58VeL1K9Sp55fou7qka27InWfXysZRfPN9F6MZ7SWbYM\n+vfP3R43Dl59VSumThEdnbsS2qpV1nqIhalSJbepqHdva84JHYTmslyuSccY0x2YJCL9srYnYF2V\npuYpowm/nCSmJvLo94/y/f7vmTloJmFNwwoWKu66eVDyi0MRUlLg/vuth3ay/fADXH11iQ6jyiIt\nzVrRJbsj+ccfiy7funVuU9E111iTESnblCbhO/GxDIcaAXlnvTqctU+Vs2/3fkuHDzqQmpnK9vu3\nO072YLXZp6bm35eWZu0/V2RkwQloqla19hfTjh1Wv2316lay79fPutaIaLKvcFWrWkORJ0+GjRsL\nNgBFRcEnn1h3cPXrw65d1oi2m26yKgXG5H75+cGgQfDuu9bjqVqJc0ku8azX5MmTc74PCwsjLCzM\ntljcXWJqIgEvWzWv+bfN59Y2txb9hgYNrGaZc9fDc1RjL8nFIQ8ReOUVaw3YbJ98AqNGnf/nUTa6\n+GLrQ3L0QSUnWxeJ7LuDLVtg8WLry5EOHXLvDq666rzrJKuCIiIiiIiIKNMxKqJJZ7KI9M3a1iad\ncjT3j7kMW2i1p2+/fzvtA9sX/83FfUqnBIulRkfDDTfkNuE3a2Y9bHKe64NydyKwf791MVixwvr3\n9OnCy9epk38Qmv6BFIsrtuF7A3uwOm2PApuBYSKyK08ZTfhllJaRxsVvXUx0YjShjUNZP2o9pjw7\n285zcVi0CAYPzt1+4gl48UXthFVZEhOtKa2zxx388UfR5bt1y+1IvuIKndc6i8slfABjTF9gGlZ/\nwcciMuWc1zXhl0FEZAQ9w3sCsHr4aq5tdq0tcZw9a1X858zJ3bdxozVoSqliE7H6CvIOQktJKbx8\nUFD+J4tcaJ3k8uaSCf+8AWjCLxURISw8jHVR62jo15DD4w5T1bviJ5PZutV6mCcz09oeOBA+/xz8\n/Ss8FOUJTp60pqjIviAUtk5yTEylX5hYE76H+DP2Ty5/31qX7/ObP+fO9ndW6PlF4PnnYdKk3H2f\nfw53VmwYSuWXmWnNYLpnD9x2W6VvQ9SE7wFGLx7NJ9s+ASBhQgIB1SruWegjR6BPH2t+G7CmIV65\nEho3rrAQlFJZXPE5fOUkRxKOYJ41fLLtE6b0moJMkgpL9l9+aT1qfdFFVrJ/5hlrbq/duzXZK+VO\nXOI5fFW0F9a9wDNrnwHgn0f+4YKA8u+YSkqyprJfuDB33+bNxZ7kUinlgjThu7BTyaeoPdWaB/j+\nTvfzfv/3y/2cP/8M3bvnbg8ZAp9+CjVqlPuplVLlTJt0XFT4tvCcZL/zwZ3lmuwzM+Hpp61mm+xk\nP2+e1Tn75Zea7JWqLLSG72JSM1Jp8GoDElIS6NO8D8vvXF5ug6j+/ttaP2PfPmu7fXv47ju48MJy\nOZ1SymZaw3chK/avoNoL1UhISWD9qPV8f9f35ZLst2+3FhNp0sRK9i+8kPtEmyZ7pSovreG7gEzJ\npOtHXdlydAtNazdl79i9uStROUlKitUBO326NQnifffB0aPWQEWllGfQhG+z347+RscPOwLFnN2y\nhP7+G2bMsOY4u/xyePRRa3ZbXRNbKc+j/+1tNGzhMOb+MReAxImJ+Pn4OeW4GRnWtORr11pzVN19\ntzUavVUrpxxeKeWmNOHbIPJkJM2mNQNgWt9p/Kfbf5xy3IMHrfWro6Ks7Q8+gNmzrbUplFJKE34F\ne2r1U7y04SUAYh+LpYFfyZYGdOSTT6yZKrO9+qrVdKPLkSql8tKEX0Hiz8ZT75V6AIzrPo43rn+j\nTMc7dcoaFLVihbXt7w8//WS10yullCOa8CvAjF9ncP+y+wHYN3Yfzes2L/WxfvgB8q4Aec891pM3\n1aqVMUilVKWnCb8cJacnE/ByAOmZ6Qy8bCBf3/51qY6Tnm410bz9du6+Zcus5QOVUqq4NOGXk6V7\nljJg7gAAfv7Xz3Rt1LXEx9i7F66+2lobFqzVoxYvhvr1nRmpUspTaMJ3skzJpO30tuw+tps2Ddqw\n44EdeJmSDWh+/3148MHc7WnTYOxY7YRVSpWNJnwn2nxkM93+1w2ApcOW0r9l/2K/98QJuPlmq40e\noG5da01YfXZeKeUsmvCdZODcgSzZswQfbx9OTThF9SrVi/W+lSutVaSy3X+/1VZfteKXp1VKVXKa\n8MtoX/w+WrzTAoAPbvyA+zrfd973pKVZTTQzZuTuW7ECevcuryiVUkoTfpnsPrab1u+1BuD448ep\n61u3yPK7dkFoqNV8A9ao2IULoU6d8o5UKaV0euQyCfIPYumwpcgkKTTZi8Bbb1kdrm3aWMl++nRr\n/5o1muyVUhXHiIi9ARgjdsdQHo4dgwEDrNGvABdcYHXItmhhb1xKqcrBGIOIlOjZPa3hO9myZVZt\nvkEDK9k//LDVZv/PP5rslVL20jZ8J0hJgQcesBb7zrZ2bf4pEJRSym6a8MvgzBkIDITERGv7+uut\nRb9r1bI3LqWUckSbdMogKspK9h9/bHXCLl+uyV4p5brK1GlrjHkFuAlIAfYDo0QkIeu1icA9QDrw\nsIisKOQYlbLTVimlypMdnbYrgLYi0gHYC0zMCqQNMARoDfQDphujM8EopZSdypTwRWSViGRmbW4C\nLsr6fgAwV0TSRSQS62JQ8ukilVJKOY0z2/DvAb7N+r4RcCjPa0ey9imllLLJeZ/SMcasBALz7gIE\neEpElmaVeQpIE5E55RKlUkqpMjtvwheRIqf0MsaMBG4Ars2z+wjQOM/2RVn7HJo8eXLO92FhYYTp\nA+xKKZVPREQEERERZTpGWZ/S6Qu8DlwtIsfz7G8DzAa6YTXlrARaOHocR5/SUUqpkivNUzplHXj1\nDuADrMx6CGeTiDwoIjuNMfOAnUAa8KBmdaWUspdOnqaUUm5IJ09TSilVKE34SinlITThK6WUh9CE\nr5RSHkITvlJKeQhN+Eop5SE04SullIfQhK+UUh5CE75SSnkITfhKKeUhNOErpZSH0ISvlFIeQhO+\nUkp5CE34SinlITThK6WUh9CEr5RSHkITvlJKeQhN+Eop5SE04SullIfQhK+UUh5CE75SSnkITfhK\nKeUhNOErpZSH0ISvlFIeQhO+Ukp5CE34SinlITThK6WUh9CEr5RSHsIpCd8Y86gxJtMYUzfPvonG\nmL3GmF3GmD7OOI9SSqnSK3PCN8ZcBPQGovLsaw0MAVoD/YDpxhhT1nO5ooiICLtDKBON317uHL87\nxw7uH39pOKOG/yYw/px9A4G5IpIuIpHAXqCrE87lctz9j0bjt5c7x+/OsYP7x18aZUr4xpgBwCER\n2XHOS42AQ3m2j2TtU0opZZMq5ytgjFkJBObdBQjwNPAkVnOOUkopF2dEpHRvNOZyYBVwBusicBFW\nTb4rcA+AiEzJKrscmCQiPzs4TukCUEopDyciJeobLXXCL3AgYw4CHUXkhDGmDTAb6IbVlLMSaCHO\nOplSSqkSO2+TTgkIVk0fEdlpjJkH7ATSgAc12SullL2cVsNXSinl2mwbaWuMudUY84cxJsMY0/Gc\n19xq0JYxZpIx5rAxZmvWV1+7YzofY0xfY8xuY8xfxpgn7I6npIwxkcaY7caY34wxm+2O53yMMR8b\nY2KMMb/n2VfHGLPCGLPHGPO9MaaWnTEWpZD43ebv3hhzkTFmjTHmT2PMDmPMf7L2u8Vn4CD+sVn7\nS/QZ2FbDN8ZcBmQCM4DHRGRr1v7WwBdAF6yO4FW4ePu/MWYScFpE3rA7luIwxngBfwG9gH+AX4Db\nRWS3rYGVgDHmANBJRE7YHUtxGGOuAhKBWSLSPmvfVOC4iLySddGtIyIT7IyzMIXE7zZ/98aYICBI\nRLYZY/yBLVjjhUbhBp9BEfEPpQSfgW01fBHZIyJ7yWr3z8NdB22500jirsBeEYkSkTRgLtbv3Z0Y\n3GguKBHZAJx7cRoIhGd9Hw4MqtCgSqCQ+MFN/u5FJFpEtmV9nwjswqpQusVnUEj82WObiv0ZuOJ/\nGHcdtPWQMWabMeZ/rnpbmMe5v+PDuMfvOC8BVhpjfjHGjLE7mFJqKCIxYP2HBhraHE9puNPfPQDG\nmKZAB2ATEOhun0Ge+LMfcy/2Z1CuCd8Ys9IY83uerx1Z/95UnuctD+f5WaYDl4hIByAacPlb3Eog\nVEQ6AjcA/85qcnB3LttsWQi3+7vPag5ZADycVVM+93fu0p+Bg/hL9Bk487HMAkSkNKNwjwCN82xn\nD+iyVQl+lo+ApeUZixMcAS7Os+0Sv+OSEJGjWf/GGWMWYTVTbbA3qhKLMcYEikhMVhttrN0BlYSI\nxOXZdPm/e2NMFaxk+ZmILM7a7TafgaP4S/oZuEqTTt42qCXA7cYYH2NMM+BSwKWfwsj6Q8k2GPjD\nrliK6RfgUmNME2OMD3A71u/dLRhjamTVdDDG+AF9cP3fOVh/5+f+rY/M+n4EsPjcN7iYfPG74d/9\nJ8BOEZmWZ587fQYF4i/pZ2DnUzqDgHeA+sBJYJuI9Mt6bSIwGmvQ1sMissKWIIvJGDMLq00tE4gE\n7stuF3RVWY9vTcO66H+cPQ2GO8iqCCzCuv2uAsx29fiNMV8AYUA9IAaYBHwNzMe6o40ChojISbti\nLEoh8ffETf7ujTGhwDpgB9bfjWDNBbYZmIeLfwZFxH8HJfgMdOCVUkp5CFdp0lFKKVXONOErpZSH\n0ISvlFIeQhO+Ukp5CE34SinlITThK6WUh9CEr5RSHkITvlJKeYj/B4NM8nvWNa8CAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x6eed6d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def gradientDescentMC(x, y, setIndex, alphaFactor, numIterations):\n",
    "    theta = np.ones(3)\n",
    "    x_transposed = np.array([item for sublist in x for item in sublist]).transpose()\n",
    "    y_transposed = np.array([item for sublist in y for item in sublist]).transpose()\n",
    "\n",
    "    yl = [int(set == setIndex) for set, sublist in enumerate(y) for _ in sublist]\n",
    "    n_samples = sum([len(set) for set in x])\n",
    "\n",
    "    np.seterr(all='raise')  # Watch out for over/underflows\n",
    "\n",
    "    for i in range(0, numIterations):\n",
    "\n",
    "        sigmoid = lambda x: 1.0 / (1.0 + np.exp(-x))\n",
    "\n",
    "        hypothesis = np.zeros(n_samples)\n",
    "        offset = 0\n",
    "        for set in range(0, len(x)): # We're keeping this verbose for readability's sake\n",
    "            for sample in range(0, len(x[set])):\n",
    "                hypothesis[offset + sample] = sigmoid(theta[0] + theta[1] * x[set][sample] + theta[2] * y[set][sample])\n",
    "            offset += len(x[set])\n",
    "\n",
    "        error_vector = hypothesis - yl\n",
    "\n",
    "        gradient = np.zeros(3)\n",
    "        gradient[0] = np.sum(error_vector) / n_samples\n",
    "        gradient[1] = np.dot(x_transposed, error_vector) / n_samples\n",
    "        gradient[2] = np.dot(y_transposed, error_vector) / n_samples\n",
    "\n",
    "        theta = theta - alphaFactor * gradient\n",
    "\n",
    "    return theta\n",
    "\n",
    "'''\n",
    "    Boilerplate plotting helpers\n",
    "'''\n",
    "def drawMCPredictors(theta_vector):\n",
    "    color = {\n",
    "        0: 'r',\n",
    "        1: 'b',\n",
    "        2: 'g',\n",
    "        3: 'y'\n",
    "    }\n",
    "    plt.text(-5, 60, \"Predictors for training sets [0;\" + str(len(x) - 1) + \"]\")\n",
    "    for i in range(0, len(x)):  # Graph all predictors\n",
    "        y_regression = np.zeros(25)\n",
    "        x_regression = np.zeros(25)\n",
    "        for sample in range(-5, 20):\n",
    "            x_regression[sample] = sample\n",
    "            y_regression[sample] = -(theta_vector[i][0] + theta_vector[i][1] * sample) / theta_vector[i][2]\n",
    "        plt.plot(x_regression, y_regression, color=color[i])\n",
    "\n",
    "def getMCThetaVector(x, y):\n",
    "    theta_vector = []\n",
    "    for i in range(0, len(x)):\n",
    "        theta_vector.append(gradientDescentMC(x, y, i, 0.01, 30000))\n",
    "    return theta_vector\n",
    "\n",
    "def plotScatterData(x, y):\n",
    "    color = {\n",
    "        0: 'r',\n",
    "        1: 'b',\n",
    "        2: 'g',\n",
    "        3: 'y'\n",
    "    }\n",
    "    for i in range(0, len(x)):\n",
    "        plt.scatter(x[i], y[i], color=color[i])\n",
    "    plt.show()\n",
    "\n",
    "x, y = generateMulticlassClassificationData()\n",
    "theta_vector = getMCThetaVector(x, y)\n",
    "drawMCPredictors(theta_vector)\n",
    "plotScatterData(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "New samples will be classified according to\n",
    "\n",
    "$$\n",
    "\\max_{i} h_\\theta^{(i)}(x)\n",
    "$$\n",
    "\n",
    "where $i$ is the class that maximizes $h_\\theta(x)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Considerations\n",
    "\n",
    "Other advanced algorithms are available for logistic regression tasks, for instance *conjugate gradient* or *BFGS* to solve unconstrained nonlinear optimization problems. Upside is speed and no need to choose an $\\alpha$ parameter, downside is usually the added complexity to implement them."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}