XGBoost + Dask example

xgboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dask and XGBoost\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dask and XGBoost can work together to train gradient boosted trees in parallel. This notebook shows how, and is easily runnable.\n",
    "\n",
    "XGBoost provides a powerful prediction framework, and it works well in practice though it's not well understood. It wins Kaggle contests and is popular in industry, because it has good performance (i.e., high accuracy models) and can be easily interpreted (i.e., it's easy to find the important features from a XGBoost model).\n",
    "\n",
    "The goals of this notebook are to show Dask and XGBoost working together, and explain a little bit of what they do together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"http://dask.readthedocs.io/en/latest/_images/dask_horizontal.svg\" width=\"30%\" alt=\"Dask logo\"> <img src=\"https://raw.githubusercontent.com/dmlc/dmlc.github.io/master/img/logo-m/xgboost.png\" width=\"25%\" alt=\"Dask logo\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3>Client</h3>\n",
       "<ul>\n",
       "  <li><b>Scheduler: </b>tcp://127.0.0.1:60822\n",
       "  <li><b>Dashboard: </b><a href='http://127.0.0.1:8787/status' target='_blank'>http://127.0.0.1:8787/status</a>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3>Cluster</h3>\n",
       "<ul>\n",
       "  <li><b>Workers: </b>8</li>\n",
       "  <li><b>Cores: </b>8</li>\n",
       "  <li><b>Memory: </b>17.18 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: scheduler='tcp://127.0.0.1:60822' processes=8 cores=8>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dask import compute, persist\n",
    "from dask.distributed import Client, progress\n",
    "\n",
    "client = Client()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create a bunch of synthetic data, with realistic sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.array as da\n",
    "from dask_ml.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "num_samples, num_features = int(100e3), 20\n",
    "da.random.seed(42)  # so results are deterministic; not important\n",
    "\n",
    "X = da.random.normal(size=(num_samples, num_features),\n",
    "                     chunks=num_samples // 100)\n",
    "w_star = da.random.uniform(size=num_features,\n",
    "                           chunks=num_features)**3\n",
    "y = da.sign(X @ w_star)\n",
    "\n",
    "# for binary:logistic objective, only [0, 1] labels allowed\n",
    "y = (y + 1) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's separate into a training set and testing set, which will allow for good evaluation after training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try to do something with this data using [dask-xgboost][dxgb].\n",
    "\n",
    "[dxgb]:https://github.com/dask/dask-xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask_xgboost\n",
    "import xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dask-xgboost is a small wrapper around xgboost, and will behave the same as xgboost.\n",
    "\n",
    "During training Dask will take care of loading, cleaning, and pre-processing the data. XGBoost will leverage their own distributed training system using all the workers that Dask has available. Dask sets XGBoost up, gives XGBoost data and lets XGBoost do it's training in the background using all the workers Dask has available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do some training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.04 s, sys: 58.6 ms, total: 1.09 s\n",
      "Wall time: 2.47 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "params = {'objective': 'binary:logistic', 'nround': 1000, \n",
    "          'max_depth': 5, 'eta': 0.01, 'subsample': 0.5, \n",
    "          'min_child_weight': 0.5}\n",
    "\n",
    "bst = dask_xgboost.train(client, params, X_train, y_train)\n",
    "bst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `bst` object is a regular `xgboost.Booster` object. This means all the methods mentioned in the [XGBoost documentation][2] are available. Here's one example with plotting:\n",
    "\n",
    "[2]:https://xgboost.readthedocs.io/en/latest/python/python_intro.html#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAAHwCAYAAACG+PhNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XmYXGWZ9/HvnQ4QlrAIgQECdFCEkBBCCIm8CiZKlE1RQMcQ3xFRMwiIOOiQcWFTXnFERQaUQcdBGQy7yAAiIgQQwRAg7EQQIgkwEDKEJQTMcr9/1Elomu6kk9NVp6v7+7muulJnqXPuU5V66tdPPadOZCaSJEmS1ky/qguQJEmSmpmBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNR9UETsFRGzqq6jIxExLiLmrmT5uyPi0Yh4JSI+0sjayurJz7uk3ikiZkfEPg3e58kR8V+rsb7tupqegbqJFA3joqLRWX47uwuPy4h4x/LpzLw1M3esU43nR8S36rHtwqnA2Zm5QWZeWWZDjf6gqefzvrpW9YeLpK6JiE9ExJ8iYmFEPFfcPyoiouraVqU72utuakts17uB7Xq1DNTN50NFo7P8dkzVBTXYdsCDVRcBEBH9q65hTTRr3VJPExHHAz8Evgv8HbAFcCTwbmDtTh7T0rACS2pgW2G7XlKz1t2rZKa3JrkBs4F9Oln2DuBm4EXgeeDiYv4tQAILgVeAvwfGAXPbbfcrwH3Fev9B7YPhN8DLwA3AJm3WvxT4n2JftwDDivmTgcXA34p9/XcxfyvgcmAe8ARwbJttrQucD7wAPFTUMbeTY/wLsAxYVGx/HWCjot5ngKeAbwEtxfpvB24E5hfPyYXAxsWyC9pt65/bPy/tn3PgZOAy4L+Al4DPUvujdEpR23zgEuBtndS/xs870Fq8jpOBp4vjPb7NttYBziyWPV3cX6ftfoETitft0uK4lxXH/krxGo0BbgcWFNs/G1i7zT6SWlh4tHi9zgGizfLPAQ8XtT8EjFrV6+/NW7PeirZnIXDIKtY7H/gxcG2x/j7FY39RvCf+Cnwd6FesfzLwX20ev/y937+YngZ8E7iteK9dD2zWZv3/W2xzPvA1OvncoPP2enbRVtwHvA70L/b/jnbH9C1g/U7akpOptYW/KGp8EBjdyfNju2673itulRfgbTVerJUH6qnUGs9+wADgPW2WtW8MO2oA7ije9FsDzwF3A7sVb+gbgZParH8EMLDNm31mm2XnA99qM90PuAs4kVqPzfbA48AHi+WnA7cCbwO2AR6gk0Dd0XMAXAn8O7WGfXNgOvCPxbJ3ABOKOgdRC/9nrmRb49rvm7c2vIuBjxTHtS5wXPHcDS728+/A1E5qX+PnnTca3qnFse5SNGTLazu12NbmxbH+Efhmm/0uAb5TbHfdTo51d+Bd1D5AW6k1ose1+390NbAxsG2x/32LZR+j9sG3BxDFc7/dql5/b96a9QbsW7yv+q9ivfOpdT68mzfa518Av6bWjrYCfwY+U6x/MqsO1H8B3lm8l6cBpxfLdqYWpPYu3uvfL2rs7HPjfNq018W82cBMau3xusW8DgN1cb+jtuRk4DVgf6AF+DZwx0qeo9nYrtuuN/nNIR/N58qIWNDm9rli/mJq/9G3yszXMvMPq7ndf8vMZzPzKWoB90+ZeU9mvg78ilpjAEBm/iwzXy6WnQzsGhEbdbLdPYBBmXlqZv4tMx8HfgJ8olj+ceC0zPzfzJwDnNXVgiNiC2A/ao3Dwsx8DvjB8m1n5mOZ+bvMfD0z51H7cHlvV7ffidsz88rMXJaZi4B/BL6WmXPbPB+HrsbXb11+3gunFMd6P/CfwMRi/iTg1Mx8rjjWU6j1VC23jFoj/npR91tk5l2ZeUdmLsnM2dQ+RNo/X6dn5oLMfBK4CRhZzP8s8K+ZeWfWPJaZf2XVr7/UrDYDns/MJctnRMQfi3Z5UUTs3WbdX2fmbZm5jFpb/ffAvxTt6Gzge7z5/boq/5mZfy7ey5fwxvvwUODqzLylaEO+Qe29v7rOysw5nbUVXfSHzLw2M5dS6znetSsPsl23XW9WjrlpPh/JzBs6mP/P1L4GnB4RLwDfy8yfrcZ2n21zf1EH0xvAivF/p1H7y3UQbzTWm1HrhWlvO2CriFjQZl4LtUYGal8bzWmz7K+rUfN2wFrAM23O/+m3fHsRsTm1gL4XtZ6gftS+0ipjTrvp7YBfRUTbD62l1HonnurC9rr0vHey/79S69GA2vP413bLtmozPS8zX1tZIRHxTmofTqOB9ai1D3e1W+1/2tx/tU1921DrNWtvVa+/1KzmA5tFRP/loToz/w9AcWJY2w6rtu/bzaj16rV/v269Gvvu7H34pvY0MxdGxPzV2O5y7du5NdG+xgFtn6uVsF23XW9K9lD3Epn5P5n5uczcitpf1z9q+8se3egw4CDeGAfYWsxf3vJlu/XnAE9k5sZtbgMzc/9i+TPU3rTLbbsatcyhNsZvszbb3jAzhxXLv13UMyIzNwQ+2abOjmpdSK3BqR1Q7Y+HQe3W6ej49mt3fAOKnol6aP9cPV3cf5paI9fRMnhr3e2noTbO8xFgh+L5+ipvfr5WZg61sY0dzV/Z6y81q9uptT8HdWHdtu+353njG8XltuWNoPamdojayY5d9ab2NCLWAzbtYl0rm//qSmrqbBtrynbddr0pGah7iYj4WEQMLiZfoPbGWlpMP0ttjFN3GEitsZtPrZH6f+2Wt9/XdOCliDghItaNiJaIGB4RexTLLwH+JSI2Ker/QlcLycxnqJ2Q872I2DAi+kXE2yNi+ddZA6mNJ1wQEVtTO1FkZbX+mVovygERsRa1E4XWWUUZ5wKnRcR2ABExKCK68gG7pr4REetFxDDg08DFxfypwNeL/W9GbWzbyn4H9llg03ZDdQZSOynnlYjYCfj8atT1U+DLEbF71LyjeE5W9fpLTSkzF1D7Cv5HEXFoRGxQtEEjqY2H7exxS6m1e6dFxMDiffJPvPF+nQnsHRHbFu/Pf1mNsi4DDoyI90TE2tTG4K7sc76rnw0zgcOK9+++vHnIQEdtyRqzXbddb1YG6ubz3/Hm36H+VTF/D+BPEfEKcBXwxcx8olh2MvDzYmzfx0vu/xfUvnZ6itoZv3e0W/4fwM7Fvq4sPjw+RG1M1hPUemd+Sq13G2ofSH8tll1Pbazd6vgHal+fPkTtD4nLgC3bbHsUtaEo1wBXtHvst6k1Vgsi4suZ+SJwVFHfU9R6Nlb1m54/pPZ8Xx8RL1N7Psau5jGsjpuBx4DfA2dk5vXF/G8BM6idWX4/tZNgOv192cx8hFpj/Xhx/FsBX6b2DcTL1MbDXdzZ4zvY3qXUhgL9snj8ldTOil/V6y81rcz8V2ph+J+pnXz2LLUxqidQO4GsM1+g1r48DvyB2vvmZ8U2f0ftvXcfta/mr16Neh4Eji629wy1NnFlbdib2uuVrPdFau/jBdTG9a5Yt5O2pCzb9Rrb9SYSmd39bY2k7hYRrdQarrW6MAZRktTD2a73LvZQS5IkSSUYqCVJkqQSHPIhSZIklWAPtSRJklSCgVqSJEkqoSmvlLjxxhvnO95Rj2uW9DwLFy5k/fU7/UnTXqcvHW9fOlboW8e7smO96667ns/M9heW6NU222yzbG1trboMSVptXW2zmzJQb7HFFsyYMaPqMhpi2rRpjBs3ruoyGqYvHW9fOlboW8e7smONiL92uKAXa21t7TNttqTepatttkM+JEmSpBIM1JIkSVIJBmpJkiSphKYcQy1Jam6LFy9m7ty5vPbaa1WXoh5iwIABDB48mLXWWqvqUqTVZqCWJDXc3LlzGThwIK2trURE1eWoYpnJ/PnzmTt3LkOGDKm6HGm1OeRDktRwr732GptuuqlhWgBEBJtuuqnfWKhpGaglSZUwTKst/z+omRmoJUmSpBIcQy1JqlzrlGu6dXuzTz9gleucddZZ/PjHP2bUqFFceOGFXd/27Nn88Y9/5LDDDitT4iqdeOKJ7L333uyzzz513U9bZ555JpMnT2a99dZr2D6l3sAeaklSn/SjH/2Ia6+9drXCNNQC9S9/+cvV3t/SpUtXa/1TTz21oWF66dKlnHnmmbz66qsN26fUWxioJUl9zpFHHsnjjz/Ohz/8YU477TSOOOII9thjD3bbbTd+/etfA7XgvNdeezFq1ChGjRrFH//4RwCmTJnCrbfeysiRI/nBD37A+eefzzHHHLNi2wceeCDTpk0DYIMNNuDEE09k7Nix3H777dx11128973vZffdd+eDH/wgzzzzTKc1Hn744Vx22WVA7fLtX/3qV9lzzz0ZPXo0d999Nx/84Ad5+9vfzrnnngvULnm/995789GPfpSdd96ZI488kmXLlgEwdepUdtllF4YPH84JJ5ywYh9t6zvttNN4+umnGT9+POPHjwfg85//PKNHj2bYsGGcdNJJKx7X2trKSSedxKhRo9hll1145JFHAHjllVf49Kc/zS677MKIESO4/PLLAbj++uvZc889GTVqFB/72Md45ZVX1vzFk3ogA7Ukqc8599xz2WqrrbjppptYuHAh73vf+7jzzju56aab+MpXvsLChQvZfPPN+d3vfsfdd9/NxRdfzLHHHgvA6aefzl577cXMmTP50pe+tNL9LFy4kOHDh/OnP/2JsWPH8oUvfIHLLruMu+66iyOOOIKvfe1rXa55m2224fbbb2evvfZaEbbvuOMOTjzxxBXrTJ8+ne9973vcf//9/OUvf+GKK67g6aef5oQTTuDGG29k5syZ3HnnnVx55ZVvqe/EE09c8ZzcdNNNAJx22mnMmDGD++67j5tvvpn77rtvxb4222wz7r77bj7/+c9zxhlnAPDNb36TjTbaiPvvv5/77ruP973vfTz//PN861vf4oYbbuDuu+9m9OjRfP/73+/ycUvNwDHUkqQ+7frrr+eqq65aEQpfe+01nnzySbbaaiuOOeYYZs6cSUtLC3/+859Xe9stLS0ccsghAMyaNYsHHniACRMmALUhFltuuWWXt/XhD38YgF122YVXXnmFgQMHMnDgQAYMGMCCBQsAGDNmDNtvvz0AEydO5A9/+ANrrbUW48aNY9CgQQBMmjSJW265hY985CNvqq8jl1xyCeeddx5LlizhmWee4aGHHmLEiBEAHHzwwQDsvvvuXHHFFQDccMMNXHTRRSsev8kmm3D11Vfz0EMP8e53vxuAv/3tb+y5555dPm6pGRioJUl9WmZy+eWXs+OOO75p/sknn8wWW2zBvffey7JlyxgwYECHj+/fv/+KoRXAm35LecCAAbS0tKzYz7Bhw7j99tvXqM511lkHgH79+q24v3x6yZIlwFt/ei4iyMxOt9m2vvaeeOIJzjjjDO6880422WQTDj/88Dcd2/IaWlpaVuw/M99SQ2YyYcIEpk6d2tVDlZqOQz4kSX3aBz/4Qf7t3/5tRfC85557AHjxxRfZcsst6devHxdccMGKkwoHDhzIyy+/vOLxra2tzJw5k2XLljFnzhymT5/e4X523HFH5s2btyJQL168mAcffLBbj2X69Ok88cQTLFu2jIsvvpj3vOc9jB07lptvvpnnn3+epUuXMnXqVN773vd2+Pi2x/bSSy+x/vrrs9FGG/Hss8/ym9/8ZpX7/8AHPsDZZ5+9YvqFF17gXe96F7fddhuPPfYYAK+++uoa9fZLPZk91JKkynXlZ+7q5Rvf+AbHHXccI0aMIDNpbW3l6quv5qijjuKQQw7h0ksvZfz48ay//voAjBgxgv79+7Prrrty+OGHc9xxxzFkyJAVJ/2NGjWqw/2svfbaXHbZZRx77LG8+OKLLFmyhOOOO45hw4Z127HsueeeTJkyhfvvv3/FCYr9+vXj29/+NuPHjycz2X///TnooIM6fPzkyZPZb7/92HLLLbnpppvYbbfdGDZsGNtvv/2KIRsr8/Wvf52jjz6a4cOH09LSwkknncTBBx/M+eefz8SJE3n99dcB+Na3vsU73/nObjtuqWqxsq+Ceqodd9wxZ82aVXUZDTFt2jTGjRtXdRkN05eOty8dK/St413ZsUbEXZk5urEVVWv06NE5Y8aMN817+OGHGTp0aEUV9U7Tpk3jjDPO4Oqrr666lDXm/wv1NF1tsx3yIUmSJJXgkA9Jkip09NFHc9ttt71p3he/+EU+/elPr9Z2xo0b12e+BZJ6GgO1JEkVOuecc6ouQeqxWqdc0+3brMc5Gw75kCRVohnP4VH9+P9BzcxALUlquAEDBjB//nxDlIBamJ4/f36nv/Ut9XQO+ZAkNdzgwYOZO3cu8+bNq7oU9RADBgxg8ODBVZchrREDtSSp4dZaay2GDBlSdRmS1C0c8iFJkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEkl9K+6gDWxaPFSWqdcU3UZDXH8Lks4vI8cK/St4+1Lxwo993hnn35A1SVIkpqcPdSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJK6F91AZIkSWqM1inX1GW7s08/oC7bbRb2UEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UktfPDH/6Q4cOHM2zYMM4880wAvvGNbzBixAhGjhzJBz7wAZ5++umKq5Qk9RR1DdQRcWxEPBwRl0fE7RHxekR8uc3yARExPSLujYgHI+KUetYjSavywAMP8JOf/ITp06dz7733cvXVV/Poo4/yla98hfvuu4+ZM2dy4IEHcuqpp1Zdal1ExL4RMSsiHouIKR0s3zYiboqIeyLivojYv4o6JaknqXcP9VHA/sDngWOBM9otfx14X2buCowE9o2Id9W5Jknq1MMPP8y73vUu1ltvPfr378973/tefvWrX7HhhhuuWGfhwoVERIVV1kdEtADnAPsBOwMTI2Lndqt9HbgkM3cDPgH8qLFVSlLPU7dAHRHnAtsDVwGTMvNOYHHbdbLmlWJyreKW9apJklZl+PDh3HLLLcyfP59XX32Va6+9ljlz5gDwta99jW222YYLL7ywt/ZQjwEey8zHM/NvwEXAQe3WSWD5XxcbAY59kdTn1S1QZ+aR1Bra8Zn5g87Wi4iWiJgJPAf8LjP/VK+aJGlVhg4dygknnMCECRPYd9992XXXXenfv3YNrNNOO405c+YwadIkzj777IorrYutgTltpucW89o6GfhkRMwFrgW+0JjSJKnnqvykxMxcmpkjgcHAmIgY3tF6ETE5ImZExIxXXnqpsUVK6lM+85nPcPfdd3PLLbfwtre9jR122OFNyw877DAuv/zyiqqrq47GsbT/1nAicH5mDqY2pO+CiHjLZ0nbNnvevHl1KFWSeo7KA/VymbkAmAbs28ny8zJzdGaO3qDNWEZJ6m7PPfccAE8++SRXXHEFEydO5NFHH12x/KqrrmKnnXaqqrx6mgts02Z6MG8d0vEZ4BKAzLwdGABs1n5DbdvsQYMG1alcSeoZ+le584gYBCzOzAURsS6wD/CdKmuSpEMOOYT58+ez1lprcc4557DJJpvw2c9+llmzZtGvXz+22247zj333KrLrIc7gR0iYgjwFLWTDg9rt86TwPuB8yNiKLVAbRe0pD6tIYE6Iv4OmEHtRJZlEXEctTPItwR+XpxZ3o/ameNXN6ImSerMrbfe+pZ5vXSIx5tk5pKIOAb4LdAC/CwzH4yIU4EZmXkVcDzwk4j4ErXhIIdnpieTS+rT6hqoM7O1zeTgDla5D9itnjVIkrouM6+ldrJh23kntrn/EPDuRtclST1ZjxlDLUmSJDUjA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIkldC/6gLWxLprtTDr9AOqLqMhpk2bxuxJ46ouo2H60vH2pWOFvne8kqS+wx5qSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKqF/1QWsiUWLl9I65Zqqy2iI43dZwuF95Fihbx1vXzpWaNzxzj79gLrvQ5KktuyhliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSer0f/OAHDBs2jOHDhzNx4kRee+01brzxRkaNGsXw4cP51Kc+xZIlS6ouU5LUpCoJ1BFxbEQ8HBEXFtN7RMTSiDi0inok9V5PPfUUZ511FjNmzOCBBx5g6dKl/PKXv+RTn/oUF110EQ888ADbbbcdP//5z6suVZLUpKrqoT4K2D8zJ0VEC/Ad4LcV1SKpl1uyZAmLFi1iyZIlvPrqq6y//vqss846vPOd7wRgwoQJXH755RVXKUlqVg0P1BFxLrA9cFVEfAn4AnA58Fyja5HU+2299dZ8+ctfZtttt2XLLbdko4024uMf/ziLFy9mxowZAFx22WXMmTOn4kolSc2q4YE6M48EngbGA5cAHwXObXQdkvqGF154gV//+tc88cQTPP300yxcuJALL7yQiy66iC996UuMGTOGgQMH0r9//6pLlSQ1qao/Qc4ETsjMpRGx0hUjYjIwGWCTTQexYQOKk9T8brjhBoYMGcKgQYMAOPjgg/njH//IJz/5SW699VYArr/+ev785z9XWaYkqYlV/Ssfo4GLImI2cCjwo4j4SEcrZuZ5mTk6M0dvsKFxWlLXbLvtttxxxx28+uqrZCa///3vGTp0KM89Vxtl9vrrr/Od73yHI488suJKJUnNqtJAnZlDMrM1M1uBy4CjMvPKKmuS1LuMHTuWQw89lFGjRrHLLruwbNkyJk+ezHe/+12GDh3KiBEj+NCHPsT73ve+qkuVJDWpqod8SFLdnXLKKZxyyilvmvfd736X7373uxVVJEnqTSoJ1EWPdPt5hze+EkmSJKmcqsdQS5IkSU3NQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJfSvuoA1se5aLcw6/YCqy2iIadOmMXvSuKrLaJi+dLx96Vih7x2vJKnvsIdakiRJKsFALUlaISL2jYhZEfFYREzpZJ2PR8RDEfFgRPyy0TVKUk/TlEM+JEndLyJagHOACcBc4M6IuCozH2qzzg7AvwDvzswXImLzaqqVpJ7DQC1JWm4M8FhmPg4QERcBBwEPtVnnc8A5mfkCQGY+1/AqpR6mdco13b7N2X3kXLHewiEfkqTltgbmtJmeW8xr653AOyPitoi4IyL2bVh1ktRD2UMtSVouOpiX7ab7AzsA44DBwK0RMTwzF7xpQxGTgckA2267bfdXKkk9iD3UkqTl5gLbtJkeDDzdwTq/zszFmfkEMItawH6TzDwvM0dn5uhBgwbVrWBJ6gkM1JKk5e4EdoiIIRGxNvAJ4Kp261wJjAeIiM2oDQF5vKFVSlIPY6CWJAGQmUuAY4DfAg8Dl2TmgxFxakR8uFjtt8D8iHgIuAn4SmbOr6ZiSeoZHEMtSVohM68Frm0378Q29xP4p+ImScIeakmSJKmUpuyhXrR4aV1+87EnOn6XJRzeR44Ves/x+vuhkiT1HfZQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIkldCUv0MtSZJ6r3pda8JrBKhe7KGWJEmSSjBQS5IkSSU45EOSpD6gHsMoHEIh1dhDLUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSV4JUSpQZpbW1l4MCBtLS00L9/f8444wz+/u//nlmzZgGwYMECNt54Y2bOnFlxpZIkaXXULVBHxLHA54GHgK2AUcDXMvOMYvmOwMVtHrI9cGJmnlmvmqSq3XTTTWy22WYATJs2jYsvfuMtcPzxx7PRRhtVVZokSVpD9eyhPgrYD1gIbAd8pO3CzJwFjASIiBbgKeBXdaxH6rEyk0suuYQbb7yx6lIkSdJqqssY6og4l1qP81XApMy8E1i8koe8H/hLZv61HvVIPUFE8IEPfIDdd9+d8847703Lbr31VrbYYgt22GGHiqqTJElrqi491Jl5ZETsC4zPzOe78JBPAFNXtkJETAYmA2yy6SA2LF+m1FC33XYbW221Fc899xwTJkzgiCOOYNy4cQBMnTqViRMnVlugJElaI5WflBgRawMfBv5lZetl5nnAeQDbbv+ObEBpUrfaaqutANh888356Ec/yiOPPALAkiVLuOKKK7jrrruqLE+SJK2hnvCzefsBd2fms1UXItXLwoULefnll1fcv/766xkyZAgAN9xwAzvttBODBw+uskRJkrSGKu+hBiayiuEeUrN79tln+ehHPwrUeqQPO+wwxowZA8BFF13kcA9JkppY3QN1RPwdMAPYEFgWEccBO2fmSxGxHjAB+Md61yFVafvtt+fee+9907xp06YBcP755ze+IEmS1G3qFqgzs7XNZIffZWfmq8Cm9apBkiRJqreeMIZakiRJaloGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKWO1AHRGbRMSIehQjSZJTZywUAAAbgklEQVQkNZsuBeqImBYRG0bE24B7gf+MiO/XtzRJkiSp5+tqD/VGmfkScDDwn5m5O7BP/cqSJEmSmkNXA3X/iNgS+DhwdR3rkSRJkppKVwP1qcBvgb9k5p0RsT3waP3KkiRJkppD/66slJmXApe2mX4cOKReRUmSJEnNoqsnJb4zIn4fEQ8U0yMi4uv1LU2SJEnq+bo65OMnwL8AiwEy8z7gE/UqSpIkSWoWXQ3U62Xm9HbzlnR3MZIkSVKz6Wqgfj4i3g4kQEQcCjxTt6okSZKkJtGlkxKBo4HzgJ0i4ingCWBS3aqSJEmSmsQqA3VE9ANGZ+Y+EbE+0C8zX65/aZIkSVLPt8ohH5m5DDimuL/QMC1JkiS9oatDPn4XEV8GLgYWLp+Zmf9bl6pWYd21Wph1+gFV7Lrhpk2bxuxJ46ouo2H62vFKkqTm19VAfUTx79Ft5iWwffeWI0mSJDWXrl4pcUi9C5EkSZKaUZcCdUT8Q0fzM/MX3VuOJEmS1Fy6OuRjjzb3BwDvB+4GDNSSJEnq07o65OMLbacjYiPggrpUJEmSJDWRrl4psb1XgR26sxBJkiSpGXV1DPV/U1x2nFoI3xm4tF5FSZIkSc2iq2Ooz2hzfwnw18ycW4d6JEmSpKbS1SEf+2fmzcXttsycGxHfqWtlkiRJUhPoaqCe0MG8/bqzEEmSJKkZrXTIR0R8HjgK2D4i7muzaCBwWz0LkyRJkprBqsZQ/xL4DfBtYEqb+S9n5v/WrSpJkiSpSaw0UGfmi8CLwESAiNic2oVdNoiIDTLzyfqX+FaLFi+ldco1Vey64Y7fZQmH95JjnX36AVWXIEmS1O26NIY6Ij4UEY8CTwA3A7Op9VxLkiRJfVpXT0r8FvAu4M+ZOYTapccdQy1JkqQ+r6uBenFmzgf6RUS/zLwJGFnHuiRJkqSm0NULuyyIiA2AW4ELI+I5ahd4kSRJkvq0rvZQHwS8ChwHXAf8BfhQvYqSJEmSmkWXeqgzc2FEbAfskJk/j4j1gJb6liZJkiT1fF39lY/PAZcB/17M2hq4sl5FSZIkSc2iq0M+jgbeDbwEkJmPApvXqyhJkiSpWXQ1UL+emX9bPhER/YGsT0mSJElS8+hqoL45Ir4KrBsRE4BLgf+uX1mSJElSc+hqoJ4CzAPuB/4RuBb4er2KkiRJkprFSn/lIyK2zcwnM3MZ8JPiJkmSJKmwqh7qFb/kERGX17kWSZIkqemsKlBHm/vb17MQSZIkqRmtKlBnJ/clSZIkseorJe4aES9R66let7hPMZ2ZuWFdq5MkSZJ6uJUG6sz08uKSJEnSSnT1Z/MkSZIkdcBALUmSJJWwqjHUUl289tpr7L333rz++ussWbKEQw89lFNOOYVjjz2Wfv1qf+c999xzjBkzhiuvvHIVW5MkSapOJT3UEXFsRDwcEb+KiP+OiHsj4sGI+HQV9ajx1llnHW688UbuvfdeZs6cyXXXXccdd9zBWWedxcyZM5k5cyZ77rknBx98cNWlSn1KROwbEbMi4rGImLKS9Q6NiIyI0Y2sT5J6oqqGfBwF7A/cCTyUmbsC44DvRcTaFdWkBooINthgAwAWL17M4sWLiXjjZ89ffvllbrzxRj7ykY9UVaLU50REC3AOsB+wMzAxInbuYL2BwLHAnxpboST1TA0f8hER51K7SMxVwC+BgVFLUhsA/wssaXRNqsbSpUvZfffdeeyxxzj66KMZO3Ys06ZNA+BXv/oV73//+9lwQ3+ZUWqgMcBjmfk4QERcBBwEPNRuvW8C/wp8ubHl9SytU66py3Znn35AXbYrqX4a3kOdmUcCTwPjgbOBocX0/cAXM3NZR4+LiMkRMSMiZrzy0ksdraIm09LSwsyZM5k7dy7Tp0/ngQceWLFs6tSpTJw4scLqpD5pa2BOm+m5xbwVImI3YJvMvHplG2rbZs+bN6/7K5WkHqTqX/n4IDAT2AoYCZwdER12SWbmeZk5OjNHb2CvZa+y8cYbM27cOK677joA5s+fz/Tp0zngAHtppAaLDuatuEpuRPQDfgAcv6oNtW2zBw0a1I0lSlLPU3Wg/jRwRdY8BjwB7FRxTWqAefPmsWDBAgAWLVrEDTfcwE471V76Sy+9lAMPPJABAwZUWaLUF80FtmkzPZjaN4jLDQSGA9MiYjbwLuAqT0yU1NdVHaifBN4PEBFbADsCj1dakRrimWeeYfz48YwYMYI99tiDCRMmcOCBBwJw0UUXOdxDqsadwA4RMaQ4QfwT1M53ASAzX8zMzTKzNTNbgTuAD2fmjGrKlaSeoerfof4mcH5E3E/tq8YTMvP5imtSA4wYMYJ77rmnw2XLT0yU1FiZuSQijgF+C7QAP8vMByPiVGBGZl618i1IUt9USaAuejaW+0AVNUiS3iozrwWubTfvxE7WHdeImiSpp6t6yIckSZLU1AzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKqHqn82TJGmF1inXdPs2Z5/uVVcl1Zc91JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIkldC/6gLWxLprtTDr9AOqLqMhpk2bxuxJ46ouQ5IkSZ2wh1qSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJfSvuoA1sWjxUlqnXFN1GQ1x/C5LOLwHHevs0w+ougRJkqQexR5qSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQK019tprrzFmzBh23XVXhg0bxkknnQTApEmT2HHHHRk+fDhHHHEEixcvrrhSSZKk+qkkUEfEsRHxcERcExG/ioj7ImJ6RAyvoh6tmXXWWYcbb7yRe++9l5kzZ3Lddddxxx13MGnSJB555BHuv/9+Fi1axE9/+tOqS5UkSaqbqnqojwL2Bx4CZmbmCOAfgB9WVI/WQESwwQYbALB48WIWL15MRLD//vsTEUQEY8aMYe7cuRVXKkmSVD8ND9QRcS6wPXAVtWD9e4DMfARojYgtGl2T1tzSpUsZOXIkm2++ORMmTGDs2LErli1evJgLLriAfffdt8IKJUmS6qvhgTozjwSeBsZT65E+GCAixgDbAYMbXZPWXEtLCzNnzmTu3LlMnz6dBx54YMWyo446ir333pu99tqrwgolSZLqq+qTEk8HNomImcAXgHuAJR2tGBGTI2JGRMx45aWXGlmjumDjjTdm3LhxXHfddQCccsopzJs3j+9///sVVyZJklRflQbqzHwpMz+dmSOpjaEeBDzRybrnZebozBy9wYYbNrROdWzevHksWLAAgEWLFnHDDTew00478dOf/pTf/va3TJ06lX79qv6bTZIkqb76V7nziNgYeDUz/wZ8FrglM+1+bhLPPPMMn/rUp1i6dCnLli3j4x//OAceeCD9+/dnu+22Y8899wTg4IMP5sQTT6y4WkmSpPqoNFADQ4FfRMRSar/48ZmK69FqGDFiBPfcc89b5i9Z0uGoHUmSpF6pkkCdma3F3eeBHaqoQZIkSeoODnCVJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBL6V13Amlh3rRZmnX5A1WU0xLRp05g9aVzVZUiSJKkT9lBLkiRJJRioJUmSpBIM1JKkFSJi34iYFRGPRcSUDpb/U0Q8FBH3RcTvI2K7KuqUpJ7EQC1JAiAiWoBzgP2AnYGJEbFzu9XuAUZn5gjgMuBfG1ulJPU8BmpJ0nJjgMcy8/HM/BtwEXBQ2xUy86bMfLWYvAMY3OAaJanHMVBLkpbbGpjTZnpuMa8znwF+U9eKJKkJNOXP5kmS6iI6mJcdrhjxSWA08N5Olk8GJgNsu+223VWfJPVI9lBLkpabC2zTZnow8HT7lSJiH+BrwIcz8/WONpSZ52Xm6MwcPWjQoLoUK0k9hYFakrTcncAOETEkItYGPgFc1XaFiNgN+HdqYfq5CmqUpB7HQC1JAiAzlwDHAL8FHgYuycwHI+LUiPhwsdp3gQ2ASyNiZkRc1cnmJKnPcAy1JGmFzLwWuLbdvBPb3N+n4UVJUg9nD7UkSZJUQlP2UC9avJTWKde8Zf7s0w+ooBpJkiT1ZfZQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJvTZQz5kzh/HjxzN06FCGDRvGD3/4w6pLkiRJUi9USaCOiGMj4uGIyIi4r7j9MSJ27a599O/fn+9973s8/PDD3HHHHZxzzjk89NBD3bV5SZIkCYD+Fe33KGA/YEvg4cx8ISL2A84DxnbHDrbccku23HJLAAYOHMjQoUN56qmn2Hnnnbtj85IkSRJQQQ91RJwLbA9cBYzNzBeKRXcAg+uxz9mzZ3PPPfcwdmy3ZHVJkiRphYb3UGfmkRGxLzA+M59vs+gzwG+6e3+vvPIKhxxyCGeeeSYbbrhhd29ekiRJfVxVQz7eJCLGUwvU71nJOpOByQCbbDqIrkTjxYsXc8ghhzBp0iQOPvjg7ilWkiRJaqPyX/mIiBHAT4GDMnN+Z+tl5nmZOTozR2/QhZ7mzOQzn/kMQ4cO5Z/+6Z+6sWJJkiTpDZUG6ojYFrgC+L+Z+efu3PZtt93GBRdcwI033sjIkSMZOXIk1157bXfuQpIkSap8yMeJwKbAjyICYElmju6ODb/nPe8hM7tjU5IkSVKnKgnUmdla3P1scZMkSZKaUuVjqCVJkqRmZqCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBL6V13Amlh3rRZmnX5A1WVIkiRJ9lBLkiRJZRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5IkSSUYqCVJkqQSDNSSJElSCQZqSZIkqQQDtSRJklSCgVqSJEkqwUAtSZIklWCgliRJkkowUEuSJEklGKglSZKkEgzUkiRJUgkGakmSJKkEA7UkSZJUgoFakiRJKsFALUmSJJVgoJYkSZJKMFBLkiRJJRioJUmSpBIM1JIkSVIJBmpJkiSpBAO1JEmSVIKBWpIkSSrBQC1JkiSVYKCWJEmSSjBQS5JWiIh9I2JWRDwWEVM6WL5ORFxcLP9TRLQ2vkpJ6lkM1JIkACKiBTgH2A/YGZgYETu3W+0zwAuZ+Q7gB8B3GlulJPU8BmpJ0nJjgMcy8/HM/BtwEXBQu3UOAn5e3L8MeH9ERANrlKQex0AtSVpua2BOm+m5xbwO18nMJcCLwKYNqU6SeqjIzKprWG0R8TIwq+o6GmQz4Pmqi2igvnS8felYoW8d78qOdbvMHNTIYroqIj4GfDAzP1tM/19gTGZ+oc06DxbrzC2m/1KsM7/dtiYDk4vJHalPm91M/6eaqVZornqttX6aqd561dqlNrt/HXbcCLMyc3TVRTRCRMzoK8cKfet4+9KxQt863iY+1rnANm2mBwNPd7LO3IjoD2wE/G/7DWXmecB5daoTaK7nuZlqheaq11rrp5nqrbpWh3xIkpa7E9ghIoZExNrAJ4Cr2q1zFfCp4v6hwI3ZjF91SlI3atYeaklSN8vMJRFxDPBboAX4WWY+GBGnAjMy8yrgP4ALIuIxaj3Tn6iuYknqGZo1UNf1a8Qepi8dK/St4+1Lxwp963ib9lgz81rg2nbzTmxz/zXgY42uqxPN9Dw3U63QXPVaa/00U72V1tqUJyVKkiRJPYVjqCVJkqQSmipQr+qSuM0uIraJiJsi4uGIeDAivljMf1tE/C4iHi3+3aTqWrtLRLRExD0RcXUxPaS4nPGjxeWN1666xu4SERtHxGUR8UjxGu/ZW1/biPhS8X/4gYiYGhEDetNrGxE/i4jnIuKBNvM6fC2j5qyi3bovIkZVV7kkqR6aJlB38ZK4zW4JcHxmDgXeBRxdHOMU4PeZuQPw+2K6t/gi8HCb6e8APyiO9QVqlznuLX4IXJeZOwG7UjvuXvfaRsTWwLHA6MwcTu3ktk/Qu17b84F9283r7LXcD9ihuE0GftygGlWhiNgpIt4fERu0m9/+/02PExG/qLoGNV5ErB0R/xAR+xTTh0XE2RFxdESsVXV97UXE2yPiyxHxw4j4XkQcGREbVVVP0wRqunZJ3KaWmc9k5t3F/ZepBa6tefOlfn8OfKSaCrtXRAwGDgB+WkwH8D5qlzOG3nWsGwJ7U/uFBDLzb5m5gF762lI74Xnd4neK1wOeoRe9tpl5C2/97eXOXsuDgF9kzR3AxhGxZWMqVRUi4ljg18AXgAciou1n1f+rpqqORcRV7W7/DRy8fLrq+tRQ/0ntM/mLEXEBtZOP/wTsQfE53VMU77FzgQHU6luX2u/j3x4R46qoqZl+5aOjS+KOraiWuouIVmA3av+Zt8jMZ6AWuiNi8wpL605nAv8MDCymNwUWFJczho4ve9ystgfmAf8ZEbsCd1Hrne91r21mPhURZwBPAouA66kdb299bZfr7LXs7HLezzS4PjXO54DdM/OVoi2/LCJaM/OHQFRa2VsNBh6iFpiSWn2jge9VWZQqsUtmjig6Qp4CtsrMpRHxX8C9FdfW3ueAkUV93weuzcxxEfHv1P6Y3a3RBTVTD3VHjVCv/ImS4ivCy4HjMvOlquuph4g4EHguM+9qO7uDVXvLa9wfGAX8ODN3AxbSC4Z3dKQYO3wQMATYClif2rCH9nrLa7sqvfn/dcNExEYRcXpxDsL84vZwMW/jqutrpyUzXwHIzNnAOGC/4oO/pwXq0dT+4P0a8GJmTgMWZebNmXlzpZV1ICL+LiJ+HBHnRMSmEXFyRNwfEZf0xG9+IuLuiPh6RLy96lq6oF9xbstAat8sLh8+sQ7Q44Z88Ean8DoUHXOZ+SQV1dpMgborl8RtesU4pcuBCzPzimL2s8sbiuLf56qqrxu9G/hwRMymNnznfdR6rDcu/jqG3vUazwXmZuafiunLqAXs3vja7gM8kZnzMnMxcAXwf+i9r+1ynb2WfaLtaoBLqI29H5eZm2bmpsD4Yt6llVb2Vv8TESOXTxTh+kBgM2CXyqrqQGYuy8wfAJ8GvhYRZ9Ozv70+n1qP+hzgJmrfgh0A3EptCEBPswmwMXBTREwvTtjequqiOvEfwCPATGp/YF0aET+hdgXVi6osrAM/Be6MiPOA24GzASJiEG8djtcQTfM71MUH8Z+B91P7KuJO4LDMfLDSwrpRMYb458D/ZuZxbeZ/F5ifmadH7ddN3paZ/1xVnd2tGO/05cw8MCIuBS7PzIsi4lzgvsz8UbUVdo+IuBX4bGbOioiTqfXcQi97bSNiLPAzauPaFlH7AJxBbQx5r3lti6/yry5OvOz0fRoRBwDHAPtTG6Z2VmaOqajsphURszJzx9VdVoXi/JAlmfk/HSx7d2beVkFZXVL8f313Zn616lo6EhH3FN/yERFPZua2bZbNzMyRnT+68SLi7swcVdzfC5gIHEztHKmpmdmjLpyyPOxn5tPFNz/7AE9m5vRqK3uriBgGDAUeyMxHKq+nWQI1QETsT60Xc/klcU+ruKRuFRHvofZX9v3AsmL2V6mNo74E2JbauNSPZWYlf4HVQ7tAvT21v4Tf9v/bu5vQuMooDuPPv/UDtVCoSnClgrqsorgrtYhRwU1EoaKiBkRx4VJwIdiKgtitK1st1oWuJBQ3LSoS8AMqTVpdm3VawQ9qg6gcF3MHhpBmkUnmzs08v02G973cOZf7Lk7unPseYAF4tqr+bjO+zdI8sToGXAP8Qu+J0A624b1Nchg4SG/nmgXgRXp1w9vi3ib5lN7P+DcBy8CbwBxr3MvmH+X36e0KchmYraof24i7y5KcBr4EPq6q5WZsCngBmK6qh1oMTyOS5FxV3d18fruq3hiY+6mqxuoXgMGEemBsJzANHKyq2XYi02brVEItSZpMTW3+6/Tq8/svfC4DJ4F3q+q3tmLT6CR5C3ivX6M+MH4HvXXwZDuRrS3JZ1X1VNtxaOuZUEuSOi3JbFUdbzsOtatr66Br8Wp9JtSSpE5bXUurydS1ddC1eLW+cX6TV5IkAJKcv9IUMDXKWNSerq2DrsWrjTOhliR1wRTwCL1t8gYF+G704aglXVsHXYtXG2RCLUnqgi+AXVW1uHoiyTejD0ct6do66Fq82iBrqLWtJPmP3raDfTNNpzJJkqQtYUKtbSXJparaNcLvu6qq/h3V90mSpPHTpdbj0tCS3JJkPslikp+bzlUkeTTJ2STnknzVjO1JMpfkfJIfkuxtxg8l+aBpNHEiyc4kR5KcaY59ucVLlCRJI2YNtbab65L0a9WWqurxVfNPA6eq6p2mW9X1SW4GjgL7q2opyZ7m2MPAQlXNJHkQOAH029reB+yrqpUkLwF/VNX9Sa4Fvk1yuqqWtvJCJUnSeDCh1nazUlX3rDN/BvgoydXAXFUtNq3P5/sJ8EDr733AE83Y10luTLK7mTtZVSvN54eBvUn6Hbp2A3cCJtSSJE0AE2pNlKqaT7IfeAz4JMkR4HdgrZcJstYpmr9/rTru1ao6tanBSpKkTrCGWhMlya3Ahao6CnwI3At8DzyQ5PbmmH7JxzzwTDN2APi1qv5c47SngFeap94kuSvJDVt6IZIkaWz4hFqT5gDwWpJ/gEvAc1V1samD/jzJDuACMA0cAo43na4uA89f4ZzHgNuAs0kCXARmtvIiJEnS+HDbPEmSJGkIlnxIkiRJQzChliRJkoZgQi1JkiQNwYRakiRJGoIJtSRJkjQEE2pJkiRpCCbUkiRJ0hBMqCVJkqQh/A+goHluxFh3XAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import dask\n",
    "\n",
    "w_star = dask.compute(w_star)[0]\n",
    "idx_top = np.argsort(w_star)[-7:]\n",
    "top = w_star[idx_top]\n",
    "\n",
    "fig, axs = plt.subplots(figsize=(12, 8), ncols=2)\n",
    "\n",
    "axs[0] = xgboost.plot_importance(bst, ax=axs[0], height=0.8, max_num_features=9)\n",
    "\n",
    "axs[0].grid(False, axis=\"y\")\n",
    "axs[0].set_title('Estimated feature importance')\n",
    "\n",
    "df = pd.DataFrame({'feature_importance': top}, index=idx_top)\n",
    "df.plot.bar(ax=axs[1])\n",
    "axs[1].set_title('Ground truth feature importance')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from this example that XGBoost does a good job at feature *support recovery*, or figuring out which features are important. Notice that in both plots, feature `9` shows up in both plots (in the left plot as `f9`). It has them misordered a bit, but it still does a pretty decent job: the top 5 estimated most important features: the top 5 *estimated* most important features are *actually* the top 5 most important features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, we can use a fancier metric to determine how well our classifier is doing by plotting the Receiver Operating Characteristic (ROC) curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.array<_predict_part, shape=(15000,), dtype=float32, chunksize=(150,)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = dask_xgboost.predict(client, bst, X_test).persist()\n",
    "y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAFNCAYAAABSVeehAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VFX6wPHvm17pRSD0buhSRIHQCSIKIkWUZgErqy6rrihr96colhUVFIFFXVRWUJRqQkQFpCgg0psQaighvZ/fHxOGBEIYkszcmeT9PE8e7rlz5s57k/Dm3HPPPUeMMSillCo6L6sDUEopT6eJVCmlikkTqVJKFZMmUqWUKiZNpEopVUyaSJVSqpg0kSqlVDFpIlVOJSIHRSRVRJJE5LiIzBGRkIvq3CAi0SKSKCLnRGSxiFx7UZ1yIvK2iBzKPdbe3HKVy3yuiMhEEdkmIskiEisiX4lIS2eeryqbNJEqVxhojAkB2gBtgX+ef0FEOgMrgG+AmkB9YAvwi4g0yK3jB0QB4UAkUA64ATgNdLzMZ74D/A2YCFQCmgCLgAFXG7yI+Fzte1QZY4zRL/1y2hdwEOidp/w68H2e8k/A+wW8bynwn9zte4ETQIiDn9kYyAY6FlInBrg3T3ks8HOesgEeAvYAB4APgTcuOsY3wOO52zWB/wFxufUnWv291y/XfWmLVLmMiIQB/YG9ueUgbC3Lrwqo/iXQJ3e7N7DMGJPk4Ef1AmKNMeuLFzGDgE7AtcDnwHAREQARqQj0BeaLiBewGFtLulbu5z8qIv2K+fnKQ2giVa6wSEQSgcPASeBfufsrYfsdPFbAe44B5/s/K1+mzuVcbf3LedUYc8YYk4qt5WyArrmv3Q6sNcYcBToAVY0xLxhjMowx+4GPgBElEIPyAJpIlSsMMsaEAt2BZlxIkGeBHKBGAe+pAZzK3T59mTqXc7X1L+fw+Q1jjAHmA3fk7hoJfJa7XReoKSLx57+Ap4HqJRCD8gCaSJXLGGN+BOYAb+SWk4G1wNACqg/DdoMJ4Aegn4gEO/hRUUCYiLQvpE4yEJSnfE1BIV9U/i9wu4jUxXbJ/7/c/YeBA8aYCnm+Qo0xNzkYr/JwmkiVq70N9BGRNrnlp4AxuUOVQkWkooi8BHQGns+tMw9bsvqfiDQTES8RqSwiT4vIJcnKGLMHeB/4r4h0FxE/EQkQkREi8lRutc3AbSISJCKNgHuuFLgx5ndsN5M+BpYbY+JzX1oPJIjIkyISKCLeItJCRDoU5RukPI8mUuVSxpg44D/As7nln4F+wG3Y+jX/wjZEqktuQsQYk47thtNOYCWQgC15VQF+vcxHTQTeA6YD8cA+YDC2m0IAbwEZ2EYDzOXCZfqV/Dc3ls/znFM2MBDb8K4D2LokPgbKO3hM5eHE1vWjlFKqqLRFqpRSxeS0RCoin4jISRHZdpnXRUTezX3Ub6uItHNWLEop5UzObJHOwfY43+X0x/YESmNgPPCBE2NRSimncVoiNcasBs4UUuVWbI8AGmPMOqCCiJTE2D+llHIpK/tIa5FnwDMQm7tPKaU8ipWz2kgB+wocQiAi47Fd/hMcHHxds2bNnBmXUqWKMZBtDNk5tq+cHEO2sf2bA6RnZWNMbr0cQ05u3azz9UvxyB6Tk03WmaO27az0U8aYqkU5jpWJNBaonaccBhwtqKIxZiYwE6B9+/Zm48aNzo9OKTdzNjmD4wlpJKRmkpCWxbnUTBJSMzmbkkFcYjqJaVkkpGXa/01Itf2bkZVT6HH9XBR/QbwEfLy98PUSvL2EYH8fgvy87f/6enuRO08M3gJeInh5CV75tvOU82575d8nYvuM868nnzvDrKfv5vjJdKrUrMupo3/9VdTzsDKRfgs8LCLzsT1ud84YUxITTSjl1owxxKdkciQ+laPxqRyJT+Wv0ykcPJ3MmeQM0jNzSM/KJj0rh4ysHNKzbOXMbGtahl4C5QJ9CQ3wwVuEg6dT6NakKn7eQsUgP6qG+lM11J9gfx8CfL0J8PHCP/ffAF9v/H29CPDxtm37eOHv64WPlxc+XrZEaIVjx47Rq9cQjh/cTbNmzYiKiqJWraL3LDotkYrIf7FNUlFFRGKxzfjjC2CM+RBYAtyEbUq1FGCcs2JRypWycwxH41OJPZtqT5bnE6ZtO43UzGyXxePrLZQLsCXCcoG+9m1b4vPC38cbgCA/b5pdU45ygT6UD/SlfKAvlUP8CfbztrcKS4MjR47Qs2dPdu/eTXh4OFFRUVSvXrz5ZZyWSI0xd1zh9fMT5yrlsc4mZ7D/VDJbDsfz64HT7D2ZxKEzKU5pPfr5eFGnUhAVAn0pl5voygf6Ui7Ah6rlAuzboQG+lA/0yU2YvgT4epWqRFgcp06dolu3buzfv5/WrVuzcuVKqlYtUrdoPrqEglJXEJ+Swb64JPbHJbN6zylC/H04eCqZHccTiE/JLNIxg/y8qVUhkJq5X7UrBdKgSjDVywXkuQS2/evn42X711sTYnFVqlSJnj17UqlSJZYvX06lSpVK5LiaSJXKIy0zm+V/HmftvtPsi0tiX5yt3/JqVQ31p06loNxEGWBLmuVtSbNWhUDKBfpoUrSAl5cXM2bMIDk5mdDQ0BI7riZSVabl5BhmrznIn0fOkZCWxS97T11V/2Wgrzf1qgTToEow19WtSNs6FWhcPZQQf/2v5S527NjBpEmTmDdvHpUqVcLLy6tEkyhoIlVl0ImENLbGnmNfXBIfxOzjXGrhl+cBvl40qBJCg6rB1KoYSEJqFl0bV6FVWHlqlg+07M6zurI//viDXr16ERcXx/PPP88777zjlM/RRKpKLWMMf51OYcexBHYcT2TnsQS2H0sg9mxqoe+rHOzHoLa16NakKg2rBmuy9FC///47ffr04fTp0/Tt25dXX33VaZ+liVSVKsYYVu06ScyuOOavP0xGduGD0QFC/H1oFVaeIe3CqFs5iJZh5e1DgpRn2rBhA3379iU+Pp4BAwawYMECAgICnPZ5mkhVqbD3ZBKf/foXs385eMW6Ab5e1KsczOnkDCb2asygNjUJDfB1fpDKJdasWUP//v1JSEhg8ODBzJ8/Hz8/5z6/pYlUeZy9J5PYePAMB04ls+dkEtE7TxZav3o5f25pXZNm15SjWY1QGlUL0RZnKfa///2PhIQEhg0bxqeffoqvr/P/SGoiVR5jx7EExs/byOEzhfdxhvj7UCnYj4gmVXm4ZyOql3PeJZ1yP1OnTiU8PJzRo0fj4+OaFKeJVLm9vScTeWP5bpb9ebzQetfWKMdT/ZtxY6MqeOvNoTIlJiaGFi1aUKVKFby8vLj77rtd+vmaSJXbMcawLy6Jr387wvsx+wqsE+TnzbD2tcnIzqFh1RBuaV2TqqH+Lo5UuYPFixdz++23c+211/LTTz8REhLi8hg0kSq3YIxh7b7TzN9wmC2x8fx1OqXAenUqBfH0Tc3pF15dnwxS/O9//2PEiBFkZWXRrVs3goODLYlDE6myTGZ2Dgs2xbLkj2P8tOfUFev3bl6dj8e0d0FkyhP897//ZdSoUWRnZ/OPf/yD1157zbI/rppIlcvtPZnEJ78c4PNfDxVar0n1EPx8vBjTuR63tQvTfk9l95///Idx48aRk5PDM888wwsvvGDpFYomUuV0OTmGPSeTWH/gNIu3HGP9wcLWRIRR19dlQkQDwioGuShC5UnWrl3L2LFjMcbwwgsv8Oyzz1odkiZS5TwbDp7hro9/Jf0KS10MaRdGn2ur07VxFYJ1sg91Bddffz0TJkygXr16PPnkk1aHA2giVSXMGMP6A2eYtGBLoeM9ezevRr3KwTwR2Qw/HysXs1WeIi0tjYCAAESE999/361uNmoiVSUiMS2T77Ye45lF28jOKXh2+FB/H1rVLs/4bg2JaFL8WclV2fH666/z2WefER0dTeXKld0qiYImUlUMSelZLPz9CKt3x/HznsvP4zn37o50a1zF7X75lWd48cUXmTJlCiLCqlWruP32260O6RKaSNVV23DwDHfP3kBietZl61QM8mXa8Db0aFrNhZGp0sQYw5QpU3jppZcQEWbPnu2WSRQ0kSoHpWRksXL7CVbtPMmizUcLrNO0eihD24cxqG0tqoToU0aq6IwxPPnkk0ydOhVvb2/+85//MHLkSKvDuixNpKpQCWmZfLflGE8v/OOydYZeF8awDrVpX7eiXr6rYjPG8Nhjj/HOO+/g4+PDf//7X7dtiZ6niVQVKDM7h9veX8O2o+cwBdw7alenAlMGhtOmdgXXB6dKvezsbHx9ffnqq6+49dZbrQ7nisQU9L/EjbVv395s3LjR6jBKtS2H47njo3WkZFx686h17Qo8GdmUzg3c786pKj2MMfzxxx+0atXKZZ8pIpuMMUV6BllbpCqfcbPXs2pX3CX77+1Sn5Gd6tCgqutn1lGlX3Z2Ns8//zyPPPIIVatWRURcmkSLSxOpAuDwmRRu/vfPl6yoWatCICse66ZPHCmnyczMZPTo0cyfP5+oqCh+/vlnj7va0f8dZdzafae546N1Bb72xfjr6dSgsosjUmVJRkYGd9xxB19//TWhoaG8/vrrHpdEQRNpmZWakc2Ij9ax5XB8ga9vfKa3DmFSTpWens7QoUNZvHgx5cuXZ8WKFXTs2NHqsIpEE2kZs/SPY3y58XCB/aAA/+jXlDs71aFCkHNXXVRlW2pqKoMHD2b58uVUqlSJlStX0q5dO6vDKjJNpGVEVnYOzy3+k0/XFTwH6Op/9KBOZZ22TrnGvHnzWL58OVWrVuWHH37wqBtLBdFEWgacTEyj48tRl+yvVSGQYe1r80D3hjoDk3Kp++67j9jYWEaMGMG1115rdTjFpom0lFvx53HGz9uUb5+fjxevDWnJoDa1PLJjX3mmc+fOkZaWRvXqtvW2XnjhBatDKjGaSEupbUfOMWHeJo7E558TtNk1oXzz8I34+3hbFJkqi86ePUu/fv1ISUlh1apVVK1auqZR1ERayhhj+Hz9ISYv3HbJa6H+Pix7tJsFUamy7NSpU/Tp04fNmzdTv359UlIKXiHWk2kiLUVW7TzJv6P38NuhS4c0TYhowD/7N7cgKlWWnTx5kl69erFt2zYaN25MdHQ0YWFhVodV4jSRlhI/7Ylj3JwN+fbVrxLM8A61mdCtgfaFKpc7duwYvXr1YseOHTRr1ozo6Ghq1KhhdVhOoYm0FPh03V+8sHh7vn39wqszbVgbfbRTWSI+Pp6IiAj27NlDixYt+OGHH6hevbrVYTmN/i/zYMnpWYybs4H1By4sb+zn7cWkfk0Y362hhZGpsq58+fIMGDCAmJgYVq5cSZUqVawOyak0kXqofXFJ9Hrzx0v2L320Kw11hiZlMRFh2rRpJCcnExJS+n8fdRS2hzHG8Pmvhy5JovUqB/HdI100iSrL7Nq1iz59+nDixAnAlkzLQhIFbZF6lFNJ6bR/6YcCX4v+e3e8vPSGkrLG9u3b6dmzJydOnGDKlCnMmDHD6pBcSlukHmJfXFKBSbRzg8oc/L8BmkSVZbZu3Ur37t05ceIEvXr1Ytq0aVaH5HLaInVzmdk5fBizjzdX7s63v9k1ocwYdR11KwdbFJlS8Ntvv9GnTx/OnDlDZGQkX3/9NYGBgVaH5XKaSN3cGyt2MePH/fn23XV9HV4a1NKiiJSyWb9+Pf369SM+Pp5bbrmFL7/8En//sjmHrSZSNzZ//aF8SdTP24vnbglnZKc6FkallM33339PfHw8Q4YM4fPPP8fPr+zOYauJ1E19uu4vnll04Xn5WhUCiZ4UoZONKLfx3HPP0bhxY0aMGIGPT9lOJU692SQikSKyS0T2ishTBbxeR0RWicjvIrJVRG5yZjyeICs755IkGl6zHAse6KxJVFkuJiaGY8eOAbbhTXfddVeZT6LgxBapiHgD04E+QCywQUS+NcbkfZbxGeBLY8wHInItsASo56yY3N3Xv8Xy4nfbOZtyYSXPKiH+zLunE5WCy+5lk3IPS5Ys4bbbbqN+/fqsWbOGihUrWh2S23Bmi7QjsNcYs98YkwHMB269qI4ByuVulweOOjEet7Zy+wke/3JLviQK8MbQVppEleW++eYbBg0aRHp6Oj169KB8+fJWh+RWnNkmrwUczlOOBTpdVOc5YIWIPAIEA70LOpCIjAfGA9SpU7putETvPMHdczZesr9N7Qr835CWNLumXAHvUsp1vvrqK0aOHElWVhZ/+9vfeOutt3Q2sYs4M5EW9J02F5XvAOYYY94Ukc7APBFpYYzJyfcmY2YCMwHat29/8TE81qyfD/Did/lnbapTKYh593TU8aHKLXz++eeMGjWKnJwcnnjiCf7v//5Pk2gBnHlpHwvUzlMO49JL93uALwGMMWuBAKB0TxOTa8exhEuSaMta5VnwQGdNosot/Pbbb9x1113k5OTw7LPPahIthDNbpBuAxiJSHzgCjABGXlTnENALmCMizbEl0oIXXC9FsnMM/d/5Kd++uXd3JKJJ6VrHRnm2tm3bMmnSJEJDQ3n22WetDsetOS2RGmOyRORhYDngDXxijPlTRF4ANhpjvgX+DnwkIo9hu+wfa4wpNZfuBUnPyqbpM8vy7Xv1tpaaRJXbSE1NJTAwEBHhtdde01aoA5w6AMwYswTbkKa8+6bk2d4O3OjMGNxJcnoW4f9afsn+OzqWrhtoynO99dZbfPjhh8TExFCjRg1Nog7S2Z9cJC0zm/s/3XTJ/t0v9bcgGqUu9dprr/H444+ze/duVq1aZXU4HkUTqQvEJaYzfOY6ftpzyr4voklV9r9yE34++iNQ1nvxxRd56qmnEBE+/vhjRo68+HaGKow+2+VkP+6OY8wn6/Pte7xPEyb2amxRREpdYIzh2Wef5eWXX8bLy4s5c+YwatQoq8PyOJpInWjDwTNMmJd/sH3lYD9NosotGGN48sknmTp1Kt7e3nz22WcMHz7c6rA8kiZSJ5m+ai9Tl++yl4P8vLmzUx2evqm5hVEplZ+vry++vr7Mnz+f2267zepwPJZ42mij9u3bm40bL32k0p1sPhzPoOm/2MtVQvz46v4bqF9FB9or92KMYfv27YSHh1sdiuVEZJMxpn1R3qt3OkrYX6eT8yXRBlWC+e9912sSVW4hOzubyZMnc+TIEcA2FZ4m0eLTRFrCIqbG5Cu/Maw1jauHWhOMUnlkZWUxZswYXnnlFQYOHEhOTs6V36Qcon2kJej3Q2fzlf/Rrynt6uicjcp6mZmZ3HXXXXz55ZeEhITw9ttv4+Wl7aiSoom0hKRmZDP4/TX59j0Q0dCiaJS6ICMjgxEjRrBw4ULKlSvH0qVLueGGG6wOq1TRRFoCjDE89sXmfPu+nNBZ15pXlktLS+P222/n+++/p0KFCqxYsYIOHTpYHVapo4m0BNzx0TrW7T9jL1cJ8adj/UoWRqSUzYIFC/j++++pXLkyK1eupG3btlaHVCppIi2mnByTL4mO7FSHlwe1sDAipS648847OXToEAMHDqRly5ZWh1NqaSItptlrDuYrT7n5Wp0xR1kqMTGRc+fOERYWhojw9NNPWx1SqaeJtBj+8dUWvtoUay+P7FSHAF9dMllZJz4+nv79+xMXF8ePP/5IrVq1rA6pTNDxD0X04Y/78iXRlrXKM6lvUwsjUmXdmTNn6N27N+vWrSMzM5P09HSrQyoztEVaBL/sPcX/Ld1pLze7JpQvJ3Qm0E9bo8oacXFx9OnThy1bttCgQQOio6OpW7eu1WGVGZpIi+DVpTvylf99R1tNosoyJ06coFevXvz55580adKEqKgowsLCrA6rTNFEepXOJGew7UiCvTxNHwFVFkpKSqJ79+7s3LmT5s2bExUVRY0aNawOq8zRPtKr9PVvF/pFa1UI5LZ2+pdfWSckJIRhw4bRsmVL+zpLyvU0kV6FlIwsPv7pgL08vENtC6NRyua5555j7dq1VKtWzepQyixNpA7KyTF0fDmK4wlpAFQI8uWeLvUtjkqVRXv37iUiIoJDhw4BtqnwgoN1mkYraSJ1QHpWNu1eWklSepZ930PdGxHsr13MyrV27txJREQEq1evZvLkyVaHo3JpJrgCYwzXTllOdk7+lQTu7aqtUeVa27Zto3fv3pw4cYKIiAg++OADq0NSubRFegWPf7klXxLt2awaO1+M1MdAlUtt2bKFHj16cOLECXr37s2SJUsICQmxOiyVSxNpIeIS01n4+5F8+6aPbKePgSqX2rRpEz169ODUqVP079+fxYsXExQUZHVYKg9NpIWY8s22fOUtU/rqwHvlcqtWreLs2bPccsstLFy4kICAAKtDUhfRPtLLMMawdNtxe3lir8aUD/K1MCJVVk2aNInatWszePBg/Pz8rA5HFUBbpJcx6aut+cr9W1xjUSSqLFq9ejUHDx60l4cPH65J1I1pIr2MQ2eS85Wb1yhnUSSqrFmxYgX9+vWjZ8+exMXFWR2OcoAm0gKcSEhjw8ELK4J+P7GLhdGosmTJkiXccsstpKWl0atXLypXrmx1SMoBDiVSEfETkUbODsZddHolyr4dXrMc4TXLWxiNKisWLVrEoEGDSE9P58EHH2TGjBm6ZLKHuOJPSUQGAH8AK3PLbURkobMDs0pKRla+8shOdSyKRJUlX331FUOHDiUzM5PHHnuM9957T5OoB3HkJ/UC0AmIBzDGbAZKbev0w5h9+crD2+vEJMq5duzYwYgRI8jKyuKpp57izTff1Ac+PIwjw58yjTHxF/1gzeUqe7KcHMO70Xvt5a6Nq+Djra0C5VzNmzfnX//6F9nZ2Tz33HOaRD2QI4l0h4gMA7xEpD7wN2Cdc8OyxpyLVgR9ZbAuX6ucJzk52T5r05QpUyyORhWHI82th4HrgBzgayANWzItVZLSs/jklwtzjYb6+1C7kj6Gp5zjvffeo0WLFvz1119Wh6JKgCOJtJ8x5kljTNvcr6eA/s4OzNW+/i2W2LOpAAT5ebPqH92tDUiVWtOmTeORRx7h4MGDrFq1yupwVAlwJJE+U8C+UjcR4pRv/rRv33V9XaqE+FsYjSqtXn31Vf7+978DMGPGDMaOHWttQKpEXLaPVET6AZFALRGZluelctgu80uNZduO5Sv3C9fHQVXJMsbwwgsv2G8mzZo1i3Hjxlkdliohhd1sOglsw9Yn+mee/YnAU84MytXyrlEf4u9DuzoVLIxGlUbPPPMMr7zyCl5eXsydO5e77rrL6pBUCbpsIjXG/A78LiKfGWPSXBiTSyWnZ3EyMd1eHnNDXR1+okpcaGgoPj4+fPbZZwwbNszqcFQJc2T4Uy0ReRm4FrBPhGiMaeK0qFzon1//QUpGtr08vmtDC6NRpdVTTz3FoEGDaNasmdWhKCdw5GbTHGA2INju1n8JzHdiTC5z/Fwa3245ai+/OKiFzjmqSkROTg5PP/00Bw5cGFKnSbT0ciSRBhljlgMYY/YZY54Bejg3LNf4Ze8p+3aLWuUYdX1dC6NRpUV2djZ33303r776KjfddBNZWVlXfpPyaI4k0nSxdRruE5H7RWQgUM2Rg4tIpIjsEpG9IlLgDSoRGSYi20XkTxH5/CpiL7YNB8/Yt7s3ceiUlCpUVlYWo0ePZu7cuQQFBTF9+nR8fHQhitLOkZ/wY0AIMBF4GSgP3H2lN4mINzAd6APEAhtE5FtjzPY8dRoD/wRuNMacFRGXZbOcHMPK7Sfs5e5Nq7rqo1UplZmZyciRI1mwYAEhISEsWbKErl27Wh2WcoErJlJjzK+5m4nAKAARCXPg2B2BvcaY/bnvmQ/cCmzPU+c+YLox5mzuZ510PPTiiUtK53RyBgBeAm3rVHTVR6tSKD09neHDh/PNN99Qrlw5li1bRufOna0OS7lIoZf2ItJBRAaJSJXccriI/AfHJi2pBRzOU47N3ZdXE6CJiPwiIutEJPIqYi+WV5fssG/XqxKMt5cOeVJF9/333/PNN99QoUIFfvjhB02iZcxlE6mIvAp8BtwJLBORycAqYAu2BHglBWWmi6ff8wEaA92BO4CPReSS0fAiMl5ENorIxpJawyY7TySh/tqHpYrntttu46233iI6OpoOHTpYHY5yscIyyK1Aa2NMqohUAo7mlnc5eOxYIO+syGG5x7i4zjpjTCZwQER2YUusG/JWMsbMBGYCtG/fvthzoWZl57B694WE/Nwt4cU9pCqDkpOTOXnyJPXr1wfg0UcftTgiZZXCLu3TjDGpAMaYM8DOq0iiYEuGjUWkvoj4ASOAby+qs4jcoVS53QdNgP1X8RlF8sOOE5xLzQTgmnIBtKmtj4Sqq5OQkEBkZCTdunXLN1ZUlU2FtUgbiMjXudsC1MtTxhhzW2EHNsZkicjDwHLAG/jEGPOniLwAbDTGfJv7Wl8R2Q5kA/8wxpwuxvk45JlF2+zb1zeopI+EqqsSHx9PZGQkv/76K2FhYTpOVBWaSIdcVH7vag9ujFkCLLlo35Q82wZ4PPfLJdIyszmVlGEvtwrT1qhy3JkzZ+jbty+bNm2iXr16REdH2y/tVdlV2KQlUZd7zZN9s/lIvrKuEqocFRcXR58+fdiyZQsNGzYkOjqaOnX090c5NiC/1DDG8G7UhcXtejStSoCvt4URKU+RlpZGz5492bZtG02bNiUqKopatS4ezafKqjK1ROa0lbs5Ep9qLz8RqZNIKMcEBAQwduxYwsPDiYmJ0SSq8nE4kYqIx6+98e88Sy23qV2B5jXKWRiN8gS2bnybv//972zYsIFrrtEVFFR+V0ykItJRRP4A9uSWW4vIv50eWQnbdTwxX/mNoa0tikR5igMHDtC1a1f27r3wBzgwMNDCiJS7cqRF+i5wM3AawBizBQ+cRu/nPFPmBft506haiIXRKHe3Z88eunXrxi+//MI///lPq8NRbs6RROpljLl48e3sAmu6sZhdF+ZDmTzgWgsjUe5ux44dREREEBsbS5cuXZg1a5bVISk350giPSwiHQEjIt4i8iiw28lxlais7Bw2/XXWXu7WpIqF0Sh3tm3bNrp3786xY8fo0aMHS5cupVw57UtXhXMkkT6AbcB8HeAEcH3uPo+x83iifV2mGuUDCKsYZHFEyh1t3ryZ7t27c/KPMw1mAAAgAElEQVTkSfr06cN3331HSIh2Aakrc2QcaZYxZoTTI3GiT3658Cx0O513VF3G2rVrOX36NAMGDGDBggUEBARc+U1K4Vgi3ZA7K9MXwNfGmMQrvcGdGGPYePDslSuqMu+BBx7gmmuuYcCAAfj5+VkdjvIgV7y0N8Y0BF4CrgP+EJFFIuIxLdS1+09z6EyKvXxvV30uWl3w888/s3v3hS7/wYMHaxJVV82hAfnGmDXGmIlAOyAB24TPHmHkR7/atwe2rqlLiii76Oho+vXrR8+ePTl69OKpcpVynCMD8kNE5E4RWQysB+KAG5wemRPc2LCy1SEoN7F8+XIGDBhASkoKffr0oXr16laHpDyYI32k24DFwOvGmJ+cHE+Jmv1L/gl3+7eoYVEkyp189913DBkyhIyMDMaPH88HH3yAl1eZmnZClTBHEmkDY0yO0yMpYcYYnl98YcHSaqH+lA/ytTAi5Q4WLlzI8OHDyczM5OGHH+bdd9/Vib1VsV02kYrIm8aYvwP/E5FL1km60gz5Vjt6Li1f+ZOxuiBZWbd//36GDRtGVlYWjz/+OG+88YYmUVUiCmuRfpH771XPjO8OfsvzJFPzGuVoUau8hdEod9CgQQOmTp3KiRMneOWVVzSJqhJT2Az563M3mxtj8iXT3LWY3HoG/eidF56t73ut3kgoyxITEwkNDQV0pU/lHI70sN9dwL57SjqQkrZ234U19G5spM/Wl1UffPABTZs2zTdWVKmSVlgf6XBsSyjXz7t6KBAKxDs7sOJIzcjmeMKFPtKm1UMtjEZZ5Z133rG3QFetWkWTJk0sjkiVVoX1ka7HNgdpGDA9z/5E4HdnBlVcP+2Jy1fWu/Vlz9SpU3niiScA+Pe//82ECRMsjkiVZoX1kR4ADgA/uC6ckvHn0QT79h0ddZXHsubll1/mmWeeAWDGjBmMHz/e4ohUaVfYpf2PxpgIETkL5B3+JNiWpK/k9OiKaEvshZ6HhlWDLYxEudpzzz3H888/j4gwa9Ysxo0bZ3VIqgwo7NL+/HIiHnenZl9ckn27diWde7QsqVq1Kt7e3sydO5c777zT6nBUGVHYpf35p5lqA0eNMRki0gVoBXyKbfISt3P4TAqHz1xYcjmiSVULo1Gu9tBDD9GvXz8aNWpkdSiqDHFk+NMibMuMNAT+AzQHPndqVMWwZt+FRe66NalKgK+3hdEoZ8vJyWHy5Mns2LHDvk+TqHI1RxJpjjEmE7gNeNsY8whQy7lhFd0/v/7Dvn19A7ftxlUlICcnhwkTJvDKK68wYMAAMjIyrA5JlVEOLTUiIkOBUcCg3H1uOZ7IGENOnttiXRvpZX1plZ2dzT333MPcuXMJCAjgww8/1AmZlWUcfbKpB7Zp9PaLSH3gv84Nq2j2nkzKV25RS1d/LI2ysrIYNWoUc+fOJSgoiCVLltC3b1+rw1Jl2BVbpMaYbSIyEWgkIs2AvcaYl50f2tXLm0jb1amgk1KUQpmZmYwcOZIFCxYQEhLC0qVL6dKli9VhqTLuiolURLoC84Aj2MaQXiMio4wxvzg7uKu14WD+GZ9U6RMVFcWCBQsoX748y5Yt4/rrr7c6JKUc6iN9C7jJGLMdQESaY0us7Z0ZWFEcPJ1s326mibRUioyMZMaMGVx33XVcd911VoejFOBYIvU7n0QBjDE7RMTtevUzs3P47dCFFmn7urrIXWmRkpLCkSNHaNy4MYA+8qncjiM3m34TkRki0iX36wPccNKS2LOpxKdkAhDq76MzPpUSSUlJDBgwgC5durBz506rw1GqQI4k0vuBfcATwJPAfsDtptJJz8q2b19TPgAvL73R5OkSEhKIjIwkJiYGHx8fvXmo3Fahl/Yi0hJoCCw0xrzumpCK5nxrFCA0wJEeC+XOzp49S2RkJOvXr6d27dpER0frE0vKbV22RSoiT2N7PPROYKWIFDRTvts4kWci52vKB1gYiSqu06dP06tXL9avX0/9+vVZvXq1JlHl1gprut0JtDLGJItIVWAJ8Ilrwrp6eRNptVBNpJ4qIyODXr16sWXLFho3bkxUVBS1a9e2OiylClVYH2m6MSYZwBgTd4W6lkvJuNBHWk4v7T2Wn58fDzzwAM2bNycmJkaTqPIIhWWcBnnWahKgYd61m9xtXfvM7Bz7tq+3W+d8VQBjjP1m0oQJExgzZgwBAXploTxDYYl0yEVlt17fPiv7wmwlvj6aSD3JX3/9xfDhw5k1axbh4eEAmkSVRylsYme3Xrf+Ynn7SIP9dA5ST7F//3569OjBoUOHePLJJ/nuu++sDkmpq1Zqmm6bD19Yp6lFrfIWRqIctXv3brp168ahQ4e4/vrr+eyzz6wOSakiKRWJ9HRSOgdPp9jLOmGJ+9u+fTsREREcOXKELl26sGLFCsqX1z+AyjM5nEhFxP9qDy4ikSKyS0T2ishThdS7XUSMiBRpIpQNB8/YtxtUCdblRdzc1q1b6d69O8ePH6dHjx4sW7aM0FB9pFd5rismUhHpKCJ/AHtyy61F5N8OvM8bmA70B64F7hCRawuoFwpMBH69ytjt8g59SkjLKuphlIts3ryZuLg4+vbty3fffUdwsC6ZrTybIwMu3wVuxvaUE8aYLSLSo/C3ANAR2yTQ+wFEZD5wK7D9onovAq8DkxwN+mJ5Vw0dcp3bLielco0ePZrKlSvTq1cvvTuvSgVHLu29jDF/XbQvu8Ca+dUCDucpx3LRonki0haobYwp1q3aQ2cu9I/Wrqjr2LujtWvXsmXLFnt5wIABmkRVqeFIi/SwiHTEtiSzN/AIsNuB9xU0VY99sKeIeGGbNHrsFQ8kMh4YD1CnTp1LXt97MtG+XbuSJlJ3s3r1am666SYCAwPZsGED9erVszokpUqUIy3SB4DHgTrACeD63H1XEgvkfb4vDDiapxwKtABiRORg7nG/LeiGkzFmpjGmvTGmfdWql64MmveOfaNqIQ6EplwlKiqKyMhIkpOT6d+/P2FhYVaHpFSJc2Txu5PAiCIcewPQOHfV0SO5xxiZ57jngCrnyyISA0wyxmy8mg9JTMvkXKptCj1fb6FGOb1cdBfLly9n0KBBpKWlcffddzNz5ky8vXVEhSp9HFn87iPyXJKfZ4wpdL0HY0yWiDwMLAe8gU+MMX+KyAvARmPMt0WMOZ8j8RduNNWuFKQTOruJxYsXc/vtt5ORkcH999/P9OnT8fIqFcOWlbqEI32kP+TZDgAGk/8m0mUZY5Zgm34v774pl6nb3ZFjXuxQnsv6muUDi3IIVcJiY2MZOnQoGRkZTJw4kbfffltnt1elmiOX9l/kLYvIPGCl0yK6SnFJ6fbtWhU0kbqDsLAwpk+fzq5du3jttdc0iapSrygTd9YH6pZ0IEWVnnlh+rxAnazEUufOnbM/5nnPPfdYHI1SruPIk01nReRM7lc8ttbo084PzTEpGReeZNJEap1Zs2bRqFEjtm7danUoSrnclRa/E6A1trvuADnGmEtuPFnpdHKGfbtSkJ+FkZRd77//Pg899BAAMTExtGrVyuKIlHKtQlukuUlzoTEmO/fLrZIowJm8iTRYE6mrvf322/YkOm3aNCZOnGhxREq5niPjUdaLSDunR1JEmkit8/rrr/PYY48BMH36dPu2UmXNZS/tRcTHGJMFdAHuE5F9QDK2Rz+NMcYtkqsmUmu8/PLLPPPMM4gIM2fO5N5777U6JKUsU1gf6XqgHTDIRbEUyVlNpJYICwvDx8eHjz/+mDFjxlgdjlKWKiyRCoAxZp+LYrlqxpj8N5s0kbrMmDFj6Nq1Kw0aNLA6FKUsV1girSoij1/uRWPMNCfEc1VSMrJJz7KNI/X38SJIhz85jTGGZ599liFDhtC2bVsATaJK5SrsZpM3EIJtlqaCviyXt3+0crCfPkHjJDk5OTz00EO8/PLL3HzzzaSmpl75TUqVIYW1SI8ZY15wWSRFcOzchSWYK4XoZb0zZGdnM2HCBGbNmoW/vz8ff/wxgYH6KK5SeV2xj9SdbY29sARz0+q6cmhJy8rK4u6772bevHkEBgby7bff0rt3b6vDUsrtFJZIe7ksiiLaGnvOvt26ti7lW5IyMzMZPXo08+fPJzg4mO+//56IiAirw1LKLV02kRpjzlzuNXfx59ELibRlLU2kJWnNmjV88cUXhIaGsnTpUm688UarQ1LKbRVl9ie3YIzhaPyFPtIGVXSJkZIUERHB3Llzadq0KR07drQ6HKXcmscm0vSsHFIzbYuZ+noL5QI99lTcRmpqKvv37yc8PByAUaNGWRyRUp7BY9d+SEq/MH1eiL+PDn0qpuTkZG6++Wa6dOnC77//bnU4SnkUz02kaXkSaYC2RosjMTGR/v37Ex0djb+/P/7+/laHpJRH8dgMlL9F6mthJJ7t3Llz9O/fn7Vr11KzZk2io6Np2rSp1WEp5VE8NpEm5mmRhvp77GlY6syZM/Tr14+NGzdSp04doqOjadiwodVhKeVxPDYD5W2Rhuql/VXLzs62J9H69euzatUq6tZ1m6W4lPIonttHmp5p39Y+0qvn7e3No48+StOmTVm9erUmUaWKwWMT6emkCxOWlA/UPlJH5V0t5s4772TLli2EhYVZGJFSns9jE+nhMyn27bCKOomGI2JjY+nUqRObNm2y79M79EoVn+cm0rMXpnKrXTHIwkg8w8GDB+nWrRsbNmzgySeftDocpUoVz02keVqktStpIi3Mvn37iIiI4MCBA3To0IGvvvrK6pCUKlU8MpEaYzh8VhOpI3bt2kVERASHDh2ic+fOrFy5kooVK1odllKlikcm0rikdNIybUuMlAvw0ZtNl7F9+3YiIiI4cuQI3bp1Y/ny5ZQvr7NkKVXSPDKRHj5zoX+0TmVtjV7Ojh07iIuLo2fPnixZsoTQULdYIUapUscjB2DG5rmsD6ugifRyhgwZwrJly+jSpYsuD6KUE3lkIj2eZ62mmhU0QeT166+/AtCpUycA+vTpY2U4SpUJHplI8y56V6N8gIWRuJeff/6Zm266CS8vL3799VedfEQpF/HIPtK8LdLqmkgBiImJITIy0j4lnk4+opTreGQiPZty4fHQKroMMz/88AM33XQTycnJjB49mk8//RQfH4+82FDKI3lkIj2XemHCknIBZXvo05IlS7j55ptJTU3l3nvvZfbs2Xh7e1sdllJlikcm0oQ8ibQsjyE9efIkQ4cOJT09nQceeIAZM2bg5eWRP1KlPJpHXv8l5JnUuSy3SKtVq8asWbPYsGEDb7zxhq5bpZRFPDKR5ltmpAzORXr27Fn7Y54jRoxgxIgRFkekVNnmcdeB2TkX5tMM8ffB26tstcLmzp1LgwYNWL9+vdWhKKVyeV4izTMxcVlbYuSjjz5i3LhxxMfH8+OPP1odjlIql8cl0pyLWqRlxfTp0xk/fjzGGF577TX+8Y9/WB2SUiqX5yXSPC3SstI/+tZbb/Hwww/bt5944gmLI1JK5eVxmSg758J2WWiRTp061Z44p0+fzoMPPmhxREqpi3lcJsrXIi0DibRBgwb4+vry/vvvc++991odjlKqAB6XicpaH+mQIUPYs2ePLpeslBtzah+piESKyC4R2SsiTxXw+uMisl1EtopIlIhcMVvkvWsfXAoTqTGGKVOmsGbNGvs+TaJKuTenJVIR8QamA/2Ba4E7ROTai6r9DrQ3xrQCFgCvX+m4OaW4j9QYw2OPPcaLL77ILbfcQmJiotUhKaUc4MwWaUdgrzFmvzEmA5gP3Jq3gjFmlTHm/HT364CwKx00bx9poF/pmZwjJyeHBx98kHfeeQc/Pz9mz56tS4Mo5SGcmUhrAYfzlGNz913OPcDSgl4QkfEislFENianXFivqbQMyM/Ozua+++7jww8/JCAggG+++YaBAwdaHZZSykHOzEQFPbtpCtiHiNwFtAciCnrdGDMTmAlQtX5z+zGqhfoXP0qLZWVlMW7cOD799FMCAwNZvHgxvXr1sjospdRVcGYijQVq5ymHAUcvriQivYHJQIQxJv1KB83MvpCLq4Z6/uz4mzZtYv78+QQHB7NkyRK6detmdUhKqavkzES6AWgsIvWBI8AIYGTeCiLSFpgBRBpjTjpy0LyTllQM8vwp9Dp16sT8+fOpUaMGN9xwg9XhKKWKwGmJ1BiTJSIPA8sBb+ATY8yfIvICsNEY8y0wFQgBvsqdS/OQMeaWwo6b92ZTkJ9n9pGmpaWxc+dO2rRpA9jGiiqlPJdTM5ExZgmw5KJ9U/Js977aY+ZpkBLk73l37VNTUxk0aBBr1qxhxYoVdO7c2eqQlFLF5NGTlgT5elYiTU5OZsCAAaxYsYLAwEBCQkKsDkkpVQI889oY8PPxwsfbc/4OJCYmMmDAAH766SeuueYaoqKiuPbai59PUEp5Io9NpEEeNBg/Pj6e/v37s27dOmrVqkV0dDRNmjSxOiylVAnxnCbdRYI95EZTTk4OAwYMYN26ddStW5fVq1drElWqlPHYROopj4d6eXkxadIkmjZtyo8//kiDBg2sDkkpVcI8NpG6+6W9yXNTbPDgwfzxxx86i5NSpZQmUic4evQoHTp04KeffrLv8/X1/IcHlFIF8+BE6p59pIcPHyYiIoJNmzbx9NNP52uZKqVKJw9OpO7XIj148CARERHs3buXdu3asWjRInKf2FJKlWIem0jd7a793r176datGwcOHKBTp05ERUVRuXJlq8NSSrmAxybSCsHu0+e4c+dOIiIiOHz4MDfeeCMrVqygQoUKVoellHIRz02kgX5Wh2B34MAB4uLi6N69O8uWLaNcuXJWh6SUciH3uj6+CsFuNGFJ//79WblyJR06dCAoKMjqcJRSLuaxLVJfi5+z37hxIzExMfZyRESEJlGlyiiPbZFamUjXrVtHv379yM7OZs2aNbRq1cqyWJRS1vPgFqk1w4p++ukn+vTpQ0JCApGRkTRr1sySOJRS7sNjE6mfBS3SVatWERkZSVJSEnfccQfz58/Hz899bnoppazhsYnU1Zf2K1as4KabbiIlJYXRo0czb948fHw8tmdEKVWCPDeR+rgu9LNnzzJ06FDS0tK49957mT17Nt7e7jNqQCllLY9tUrmyj7RixYp8+umnREVFMW3aNLy8PPbvj1LKCTw2kbqij/T06dP2xzwHDhzIwIEDnf6ZSinP47FNK2f3kX7++efUr1+fH3/80amfo5TyfJpICzBnzhzuuusuEhMT+fnnn532OUqp0sFjE6mfj3P6SGfOnMm4ceMwxvDSSy8xefJkp3yOUqr08Ng+Ume0SN977z0eeeQRAKZOncqkSZNK/DOUUqWPxybSkl7T/p133uHRRx+1b0+cOLFEj6+UKr08NpGW9PCnpk2bEhAQwNtvv82ECRNK9NjqUpmZmcTGxpKWlmZ1KKqMCQgIICwsrETXUfPYRFrSw58iIyPZu3cvtWrVKtHjqoLFxsYSGhpKvXr1dDkW5TLGGE6fPk1sbCz169cvseN67M2m4vaRGmP417/+RVRUlH2fJlHXSUtLo3LlyppElUuJCJUrVy7xKyGPbZEWJ5EaY5g0aRLTpk2jXLlyHDhwgEqVKpVgdMoRmkSVFZzxe+fBibRo34ycnBz+9re/8d577+Hr68ucOXM0iSqlisUjL+19vaVIf1VycnK4//77ee+99/Dz8+Prr79m8ODBTohQeQJvb2/atGlDixYtGDhwIPHx8fbX/vzzT3r27EmTJk1o3LgxL774IsYY++tLly6lffv2NG/enGbNml12qJyj9ZzFGEPPnj1JSEhw6edejU2bNtGyZUsaNWrExIkT832fzzt37hwDBw6kdevWhIeHM3v2bPtrTz75JC1atKBFixZ88cUX9v0jRoxgz549LjkHjDEe9eV3TSPT/Nml5mplZWWZMWPGGMAEBASY5cuXX/UxVMnZvn271SGY4OBg+/bo0aPNSy+9ZIwxJiUlxTRo0MD+O5KcnGwiIyPNe++9Z4wx5o8//jANGjQwO3bsMMYYk5mZaaZPn37J8R2tdzlZWVlFO7E8vvvuO/Poo49e1XtK4nOvRocOHcyaNWtMTk6OiYyMNEuWLLmkzssvv2yeeOIJY4wxJ0+eNBUrVjTp6enmu+++M7179zaZmZkmKSnJXHfddebcuXPGGGNiYmLMvffeW+BnFvT7B2w0RcxLHnlpX5T+0W3btjF//nyCgoJYvHgxPXv2dEJkqijqPfW904598P8GOFSvc+fObN26FbDNs3DjjTfSt29fAIKCgnjvvffo3r07Dz30EK+//jqTJ0+2r47g4+PDgw8+eMkxC6s3duxYbr75Zm6//XYAQkJCSEpKIiYmhueff54aNWqwefNmBg4cSN26de3ve+655wgNDeXvf/87U6dO5csvvyQ9PZ3Bgwfz/PPPXxLDZ599xvjx4+3lQYMGcfjwYdLS0vjb3/5mfy0kJITHH3+c5cuX8+abbxIYGMjjjz9OUlISVapUYc6cOdSoUYOPPvqImTNnkpGRQaNGjZg3b16x1io7duwYCQkJdO7cGYDRo0ezaNEi+vfvn6+eiJCYmIgxhqSkJCpVqoSPjw/bt28nIiICHx8ffHx8aN26NcuWLWPYsGF07dqVsWPHkpWV5fS5gz300v7qw27dujULFy5k2bJlmkRVPtnZ2URFRXHLLbcAtsv66667Ll+dhg0bkpSUREJCAtu2bbvk9YI4Wu9i69ev5+WXX2b79u2MGDEi3+Xql19+ydChQ1mxYgV79uxh/fr1bN68mU2bNrF69epLjvXLL7/ki+GTTz5h06ZNbNy4kXfffZfTp08DkJycTIsWLfj111/p1KkTjzzyCAsWLGDTpk3cfffd9kelb7vtNjZs2MCWLVto3rw5s2bNuuQzV61aRZs2bS75uuGGGy6pe+TIEcLCwuzlsLAwjhw5ckm9hx9+mB07dlCzZk1atmzJO++8g5eXF61bt2bp0qWkpKRw6tQpVq1axeHDhwHw8vKiUaNGbNmyxdFvfZF5ZIvUz8EbTenp6WzdupUOHToAXPJXTpVtqamptGnThoMHD3LdddfRp08fwNbddbk+eFeMNOjYsaN9jGPbtm05efIkR48eJS4ujooVK1KnTh3effddVqxYQdu2bQFISkpiz549dOvWLd+xzpw5Q2hoqL387rvvsnDhQgAOHz7Mnj17qFy5Mt7e3gwZMgSAXbt2sW3bNvv3Izs7mxo1agC2Pw7PPPMM8fHxJCUl0a9fv0vi79GjB5s3b3boXE0B/aEFfY+XL19OmzZtiI6OZt++ffTp04euXbvSt29fNmzYwA033EDVqlXp3LlzvtZntWrVOHr0aJH+oF0Nj0ykjsyOn5aWxpAhQ4iOjub777/XVqgbc/Tyu6QFBgayefNmzp07x80338z06dOZOHEi4eHhl7Tu9u/fT0hICKGhoYSHh7Np0yZat25d6PELq+fj40NOTg5gSyYZGRn214KDg/PVvf3221mwYAHHjx9nxIgR9vf885//vOJTeOc/x8vLi5iYGH744QfWrl1LUFAQ3bt3t4+nDAgIsK/6YIwhPDyctWvXXnK8sWPHsmjRIlq3bs2cOXPyLUl+3qpVq3jssccu2R8UFMSaNWvy7QsLCyM2NtZejo2NpWbNmpe8d/bs2Tz11FOICI0aNaJ+/frs3LmTjh07MnnyZHuLeeTIkTRu3Nj+vrS0NAIDAwv9HpWIonauWvXld00j0+vNmAI7kM9LTk42ffr0MYCpXLmy+f333wutr1zP3W42/fbbb6Z27domIyPDpKSkmPr165uVK1caY2w3nwYMGGDeffddY4wxW7ZsMQ0bNjS7du0yxhiTnZ1t3nzzzUuOX1i9F1980X7zZOHChcb2X9GYVatWmQEDBuQ7zrZt20znzp1N48aNzdGjR40xxixfvtx07NjRJCYmGmOMiY2NNSdOnLgkhk6dOpk9e/YYY4xZtGiRufnmm40xxuzYscP4+/ubVatWXfK9SE9PNw0bNjRr1qwxxhiTkZFhtm3bZowxpnLlyubEiRMmIyPD9O7d24wZM+YK3+Ura9++vVm7dq39ZtP3339/SZ3777/f/Otf/zLGGHP8+HFTs2ZNExcXZ7KyssypU6eMMbbvd3h4uMnMzLS/r0WLFvbvWV56s4nC+0iTkpIYOHAgMTExVKtWjaioKFq0aOHC6JQnatu2La1bt2b+/PmMGjWKb775hkceeYSHHnqI7OxsRo0axcMPPwxAq1atePvtt7njjjtISUlBRBgw4NJWdWH17rvvPm699VY6duxIr169LmmF5hUeHk5iYiK1atWyX2L37duXHTt22G/ShISE8Omnn1KtWrV87x0wYAAxMTE0atSIyMhIPvzwQ1q1akXTpk25/vrrC/w8Pz8/FixYwMSJEzl37hxZWVk8+uijhIeH8+KLL9KpUyfq1q1Ly5YtSUxMvPpv9kU++OADxo4dS2pqKv3797d3wX344YcA3H///Tz77LOMHTuWli1bYozhtddeo0qVKqSlpdG1a1cAypUrx6effmq/tD9x4gSBgYH275lTFTUDW/Xld00jc8u/f7rkr4kxxpw7d87ceOONBjA1atSwDztR7scdWqRlwdGjR03v3r2tDsMS06ZNMx9//HGBr5V0i7TU3LU3xjBo0CB++eUXwsLC+PHHH+3DTpQqq2rUqMF9993n1gPynaVChQqMGTPGJZ9Vai7tRYSnnnqKo0ePsnTp0hKd2UUpTzZs2DCrQ7DEuHHjXPZZnplI89y1P39HEmz9Rtu2bXP64FtVMkwhw4yUchZTwJCr4vLIS/vz40iPHz9Ohw4dWLFihf01TaKeISAggNOnTzvll1qpyzHGNh9pQEBAiR7XI7OOr7cXR44coWfPnuzevZvJkyfTu3dve8tUub/z4wfj4uKsDkWVMednyC9JTk2kIhIJvAN4Ax8bY/7vojcTpOkAAAjISURBVNf9gf8A1wGngeHGmINXOm7a2RNERAxn3759tGrViiVLlmgS9TC+vr7aj61KDXHWpZWIeAO7gT5ALLABuMMYsz1PnQeBVsaY+0VkBDDYGDO8sOP6Va1nAiWThLijtGvXjhUrVlC5cmWnnINSquwQkU3GmPZFea8zm3Edgb3GmP3GmAxgPnDrRXVuBebmbi8AeskV7j5knjlCQtxROnXqRFRUlCZRpZTlnJlIawGH85Rjc/cVWMcYkwWcAwrPjDnZhDVry4oVK6hQoULJRauUUkXkzD7SglqWF/cjOFIHERkPnJ9UMT125+/bypcvX8zw3FYV4JTVQThRaT6/0nxuUPrPr2lR3+jMRBoL1M5TDgOOXqZOrIj4AOWBMxcfyBgzE5gJICIbi9qP4Qn0/DxXaT43KBvnV9T3OvPSfgPQWETqi4gfMAL49qI63wLnn+G6HYg2OrBQKeVhnNYiNcZkicjDwHJsw58+Mcb8KSIvYJsc4FtgFjBPRPZia4mOcFY8SinlLE4dR2qMWQIsuWjflDzbacDQqzzszBIIzZ3p+Xmu0nxuoOd3WU4bR6qUUmWFPg6klFLF5LaJVEQiRWSXiOwVkacKeN1fRL7Iff1XEann+iiLzoHze1xEtovIVhGJEpG6VsRZFFc6tzz1bhcRIyIedSfYkfMTkWG5P78/ReRzV8dYHA78btYRkVUi8nvu7+dNVsRZFCLyiYicFJFtl3ldROTd3HPfKiLtHDpwUWeEduYXtptT+4AGgB+wBbj2ojoPAh/mbo8AvrA67hI+vx5AUO72A55yfo6cW269UGA1sA5ob3XcJfyzawz8DlTMLVezOu4SPr+ZwAO529cCB62O+yrOrxvQDth2mddvApZiG+N+PfCrI8d11xapUx4vdSNXPD9jzCpjTEpucR22cbiewJGfHcCLwOtAmiuDKwGOnN99wHRjzFkAY8xJF8dYHI6cnwHK5W6X59Lx4W7LGLOaAsaq53Er8B9jsw6oICJXXPTJXROpcx4vdR+OnF9e92D7K+kJrnhuItIWqG2M+c6VgZUQR352TYAmIvKLiKzLnQXNUzhyfs8Bd4lILLZROY+4JjSXuNr/m4D7zkdaYo+XuimHYxeRu4D2QIRTIyo5hZ6biHgBbwFjXRVQCXPkZ+eD7fK+O7YriZ9EpIUxJt7JsZUER87vDmCOMeZNEemMbSx4C2NMjvPDc7oi5RV3bZFezeOlFPZ4qZty5PwQkd7AZOAWY0y6i2IrriudWyjQAogRkYPY+qG+9aAbTo7+bn5jjMk0xhwAdmFLrJ7AkfO7B/gSwBizFgjA9hx+aeDQ/82LuWsiLe2Pl17x/HIvf2dgS6Ke1MdW6LkZY84ZY6oYY+oZY+ph6/+9xRhT5OecXcyR381F2G4WIiJVsF3q73dplEXnyPkdAnoBiEhzbIm0tCx18C0wOvfu/fXw/+3db2iVZRjH8e+P6I8WCb4okqAlRqGko39IvgizpD8EJeKMZS2SmBRh4ZuwFxW9kP5BZrZCQgOTMVGQIspiWshsjpiuhmSZL4JRvhAJWSDr6sV9j57W0XPOntHO4veBAzv3eZ5zX88528V9P8/u6+F0RAxV3Wuyr6Kd5+rafaTC0D8B63Pby6Q/OkhfXhfwI9ALzJ7smCf4+L4AfgX682PPZMc8Ucc2Ztt9TKGr9jV+dwLeBAaBAWDlZMc8wcc3FzhAuqLfDyyd7JjrOLYdwBBwljT6fAJoB9oL3907+dgHav3d9MomM7OSGnVqb2Y2ZTiRmpmV5ERqZlaSE6mZWUlOpGZmJTmRWlWSRiT1Fx5N59m26VyVdersc1+uQHQ4L7Ws+8ZkktolPZp/bpM0q/DaFklzJzjOQ5Kaa9hnraTpZfu2xuFEarUYjojmwuPEf9Rva0QsIBWnea3enSOiIyI+zE/bgFmF11ZHxOCERPl3nJupLc61gBPp/4gTqY1LHnl+Lenb/Li9wjbzJPXmUewRSdfl9kcK7e9JuqBKd18Bc/K+S3IdzIFcW/Li3L6hUL/19dz2oqR1kpaT6hVsz31OyyPJWyStkfRqIeY2SW+PM84eCgUuJL0rqU+pJulLue0ZUkLvltSd25ZK6smfY5eky6r0Yw3GidRqMa0wrd+d234D7o6Im4AWYGOF/dqBtyKimZTIfslLCluARbl9BGit0v8DwICkS4CtQEtE3EgqDrJG0kzgIWBeRMwHXinuHBE7gT7SyLE5IoYLL+8ElhWetwCd44zzHtLy0FHrI92+eD5wh6T5EbGRtHZ7cUQszktIXwDuyp9lH/BclX6swTRq9SdrLMM5mRRdCGzK5wRHSOvJx+oB1ku6GtgVEcckLQFuBg7l8rHTSEm5ku2ShoETpFJt1wM/R8QP+fVtwFPAJlJd0y2SPgFqLs8XESclHc/rqo/lPg7k960nzktJRZGLFdVXSHqS9Hd2FWlp5ZEx+y7M7QdyPxeRPjebQpxIbbyeJdUCWECa2fyrQHNEfCTpG+B+4DNJq0lrmbdFxPM19NEahWImkirWm4106+/bSIU0VgJPA3fWcSydwArgKLA7IkIpq9UcJ2nd+QbSOu1lkq4F1gG3RsQpSVtJ9SHGErA3Ih6uI15rMJ7a23jNAIYi1aBcRRqN/YOk2cDxPJ3dQ5rifgksl3RF3mamar8f1VGgSdKc/HwVsD+fU5wR6fbfa4FKV85/J5Xwq2QX8CCpzmZnbqsrzog4S5qiL8ynBS4HzgCnJV0J3HuOWA4Ci0aPSdJ0SZVG99bAnEhtvDYDj0k6SJrWn6mwTQvwnaR+4AbSLRwGSQnnc0lHgL2kaW9VEfEH8DjQJWkA+BPoICWlj/P77SeNlsfaCnSMXmwa876nSJWaromI3txWd5z53OsbwLqIOEy6b9P3wAek0wWj3gc+ldQdESdJ/1GwI/dzkPRZ2RTi6k9mZiV5RGpmVpITqZlZSU6kZmYlOZGamZXkRGpmVpITqZlZSU6kZmYlOZGamZX0Fy+dPJ3kpR7LAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "fpr, tpr, _ = roc_curve(y_test, y_hat)\n",
    "ax.plot(fpr, tpr, lw=3,\n",
    "        label='ROC Curve (area = {:.2f})'.format(auc(fpr, tpr)))\n",
    "ax.plot([0, 1], [0, 1], 'k--', lw=2)\n",
    "\n",
    "ax.set(\n",
    "    xlim=(0, 1),\n",
    "    ylim=(0, 1),\n",
    "    title=\"ROC Curve\",\n",
    "    xlabel=\"False Positive Rate\",\n",
    "    ylabel=\"True Positive Rate\",\n",
    ")\n",
    "ax.legend();\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Receiver Operating Characteristic (ROC) curve tells how well our classifier is doing. We can tell it's doing well by how far it bends the upper-left. A perfect classifier would be in the upper-left corner, and a random classifier would follow the horizontal line.\n",
    "\n",
    "The area under this curve is `area = 0.89`. This tells us the probability that our classifier will predict correctly for a randomly chosen instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "* XGBoost documentation: https://xgboost.readthedocs.io/en/latest/python/python_intro.html#\n",
    "* A blogpost on dask-xgboost http://matthewrocklin.com/blog/work/2017/03/28/dask-xgboost\n",
    "* Similar example with real world dataset: https://dask-ml.readthedocs.io/en/latest/examples/xgboost.html\n",
    "* Recorded screencast of a core Dask developer stepping through the example above at https://www.youtube.com/watch?v=Cc4E-PdDSro \n",
    "* ROC curve: https://en.wikipedia.org/wiki/Receiver_operating_characteristic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}