Skip to content

Instantly share code, notes, and snippets.

@pmbaumgartner

pmbaumgartner/Calculating ROC before and after transformations.ipynb

Created May 21, 2017 17:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save pmbaumgartner/5f65c1cbded30eb20b86d4a1b41bc3d8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save pmbaumgartner/5f65c1cbded30eb20b86d4a1b41bc3d8 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Calculating ROC before and after transformations.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"title: The Impact of Model Output Transformations on ROC\n",
"date: 05/21/17\n",
"category: Blog"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Risk Assessment tools are currently used to assist in decision-making at several points in the criminal justice system. These tools take in some data about an individual and to provide a 'risk score' for an individual that's reflective of their likelihood of committing a specific behavior in the future. A standard outcome of interest is recidivism, or `a person's relapse into criminal behavior, often after the person receives sanctions or undergoes intervention for a previous crime` ([NIJ](https://www.nij.gov/topics/corrections/recidivism/Pages/welcome.aspx)). One thing the field seems to have settled on is using area under the receiver operating characteristic (AUC ROC) scores in evaluating the accuracy of these tools. AUC is a sensible evaluation measure since it's indifferent to base rates and has a straightforward interpretation. \n",
"\n",
"The hypothetical example below explains what AUC ROC (Ranking Accuracy) measures.\n",
"\n",
"<img src=\"https://cdn.pbrd.co/images/90KfTPdS0.png\" width=\"600\">\n",
"\n",
"## COMPAS & Risk Assessment Scores\n",
"In the world of risk assessment, the most well known risk assessment tool is probably COMPAS by Noirthpointe. COMPAS is a black box algorithm that is used to predict recidivism, and was the subject of an [investigation by ProPublica](https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm) as well as a [lawsuit in Wisconsin](https://www.nytimes.com/2016/06/23/us/backlash-in-wisconsin-against-using-data-to-foretell-defendants-futures.html). \n",
"\n",
"Northpointe [claims](http://www.northpointeinc.com/files/downloads/FAQ_Document.pdf) that the with the COMPAS tool \"AUCs ranged from .68 to .71 in the full sample\" for general recidivism, with an \"AUC 0.67 for violent recidivism\". In reading these claims, it is important to consider there is [some evidence of authorship bias](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0072484) in model evaluation, to the extent that \"[...] studies authored by tool designers reported predictive validity findings around two times higher those of investigations reported by independent authors\".\n",
"\n",
"Scores from the COMPAS tool go through several transformations before they end up in a decision maker's hands. The first output is a `raw score` from the model itself, which appears to be a normally distributed score. These raw scores are converted to a `decile score`, which I assume is based on the full range of scores. Finally, Northpointe also privdes a mapping of decile scores to qualitative risk levels following this mapping:\n",
"\n",
"- **1-4**: Low risk\n",
"- **5-7**: Medium risk\n",
"- **8+**: High risk\n",
"\n",
"In evaluating model accuracy, it's relevant to look at the model accuracy of the information presented to the decision maker. That is, if a judicial official views receives a report with the qualitative risk level (High, Medium, Low) then we should be concerned with the accuracy of those model scores. The the accuracy of the `raw score` or the `decile score` doesn't really matter -- the user of this tool never sees this underlying information, though the reported accuracy of the tool may depend on these pre-transformed measures.\n",
"\n",
"We'll evaluate model accuracy at each one of these transformations. Using the [data](https://github.com/propublica/compas-analysis/) published by ProPublica that contains COMPAS scores and outcomes for individuals in Broward County, Florida.\n",
"\n",
"## Outline\n",
"This analysis is comprised of 3 parts:\n",
"\n",
"1. Processing & Exploring the COMPAS dataset from ProPublica\n",
"2. Analyzing AUC ROC scores by raw/decile/qualitative transformation\n",
"3. Analyzing the general impact of binning on model accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Data Processing & Exploring\n",
"The final dataset ProPublica uses doesn't have the raw score, but they do provide the raw COMPAS data, which we will merge into the clean dataset. I couldn't find a clear ID or case number to merge on at a glance, so we're merging on `first name`, `last name`, and `compas screening date`. This merging scheme may lead to some discrepancies between the dataset ProPublica used for analysis, but should match most cases in general. In general, I follow [their process](https://github.com/propublica/compas-analysis/blob/master/Compas%20Analysis.ipynb) of filtering observations based on data quality assumptions."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import seaborn as sns; sns.set_context('notebook', font_scale=1.5);\n",
"import matplotlib.pyplot as plt; plt.style.use('ggplot')\n",
"import numpy as np\n",
"import scipy\n",
"\n",
"from sklearn.metrics import roc_auc_score\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def read_pp_compas_data(csv_url):\n",
" d = (pd.read_csv(csv_url, parse_dates=['compas_screening_date'])\n",
" .query('days_b_screening_arrest <= 30')\n",
" .query('days_b_screening_arrest >= -30')\n",
" .query('is_recid != -1')\n",
" .query('c_charge_degree != \"O\"')\n",
" .query('score_text != \"N/A\"')\n",
" .assign(score_text=lambda x: x['score_text'].map({'Low' : 1, 'Medium' : 2, 'High' : 3}),\n",
" v_score_text=lambda x: x['v_score_text'].map({'Low' : 1, 'Medium' : 2, 'High' : 3}))\n",
" )\n",
" return d"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"compas_recid = read_pp_compas_data('https://raw.githubusercontent.com/propublica/compas-analysis/master/compas-scores-two-years.csv')\n",
"compas_recid_v = read_pp_compas_data('https://raw.githubusercontent.com/propublica/compas-analysis/master/compas-scores-two-years-violent.csv')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"compas_recid_simple = compas_recid[['first', 'last', 'compas_screening_date', 'decile_score', 'score_text', 'is_recid']]\n",
"compas_recid_v_simple = compas_recid_v[['first', 'last', 'compas_screening_date', 'v_decile_score', 'v_score_text', 'is_violent_recid']]\n",
"\n",
"compas_scores = pd.merge(compas_recid_simple, compas_recid_v_simple, on=['first', 'last', 'compas_screening_date'], how='outer')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>first</th>\n",
" <th>last</th>\n",
" <th>compas_screening_date</th>\n",
" <th>decile_score</th>\n",
" <th>score_text</th>\n",
" <th>is_recid</th>\n",
" <th>v_decile_score</th>\n",
" <th>v_score_text</th>\n",
" <th>is_violent_recid</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>miguel</td>\n",
" <td>hernandez</td>\n",
" <td>2013-08-14</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>kevon</td>\n",
" <td>dixon</td>\n",
" <td>2013-01-27</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>ed</td>\n",
" <td>philo</td>\n",
" <td>2013-04-14</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>marsha</td>\n",
" <td>miles</td>\n",
" <td>2013-11-30</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>edward</td>\n",
" <td>riddle</td>\n",
" <td>2014-02-19</td>\n",
" <td>6.0</td>\n",
" <td>2.0</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" first last compas_screening_date decile_score score_text \\\n",
"0 miguel hernandez 2013-08-14 1.0 1.0 \n",
"1 kevon dixon 2013-01-27 3.0 1.0 \n",
"2 ed philo 2013-04-14 4.0 1.0 \n",
"3 marsha miles 2013-11-30 1.0 1.0 \n",
"4 edward riddle 2014-02-19 6.0 2.0 \n",
"\n",
" is_recid v_decile_score v_score_text is_violent_recid \n",
"0 0.0 1.0 1.0 0.0 \n",
"1 1.0 1.0 1.0 1.0 \n",
"2 1.0 NaN NaN NaN \n",
"3 0.0 1.0 1.0 0.0 \n",
"4 1.0 NaN NaN NaN "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"compas_scores.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Read Raw Score Data"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def read_compas_raw(csv_url):\n",
" columns = ['LastName', 'FirstName','Screening_Date','RawScore','DisplayText']\n",
" colnames = {'FirstName' : 'first',\n",
" 'LastName' : 'last',\n",
" 'Screening_Date' : 'compas_screening_date'}\n",
" \n",
" d = (pd.read_csv(csv_url, parse_dates=['Screening_Date'], usecols=columns)\n",
" .rename(columns=colnames)\n",
" .assign(last=lambda x: x['last'].str.lower(),\n",
" first=lambda x: x['first'].str.lower())\n",
" )\n",
" d_recid = d.loc[lambda x: x['DisplayText'] == 'Risk of Recidivism']\n",
" d_viol = d.loc[lambda x: x['DisplayText'] == 'Risk of Violence']\n",
" \n",
" d_recid = d_recid.rename(columns={'RawScore' : 'r_raw_score'})\n",
" d_viol = d_viol.rename(columns={'RawScore' : 'v_raw_score'})\n",
" \n",
" del d_recid['DisplayText']\n",
" del d_viol['DisplayText']\n",
" \n",
" return d_recid, d_viol"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"recid_raw, viol_raw = read_compas_raw('https://raw.githubusercontent.com/propublica/compas-analysis/master/compas-scores-raw.csv')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"complete_scores = pd.merge(compas_scores, recid_raw, on=['first', 'last', 'compas_screening_date'], how='left')\n",
"complete_scores = pd.merge(complete_scores, viol_raw, on=['first', 'last', 'compas_screening_date'], how='left')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>first</th>\n",
" <th>last</th>\n",
" <th>compas_screening_date</th>\n",
" <th>decile_score</th>\n",
" <th>score_text</th>\n",
" <th>is_recid</th>\n",
" <th>v_decile_score</th>\n",
" <th>v_score_text</th>\n",
" <th>is_violent_recid</th>\n",
" <th>r_raw_score</th>\n",
" <th>v_raw_score</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>miguel</td>\n",
" <td>hernandez</td>\n",
" <td>2013-08-14</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>-2.78</td>\n",
" <td>-4.31</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>kevon</td>\n",
" <td>dixon</td>\n",
" <td>2013-01-27</td>\n",
" <td>3.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>-0.76</td>\n",
" <td>-3.07</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>ed</td>\n",
" <td>philo</td>\n",
" <td>2013-04-14</td>\n",
" <td>4.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>-0.66</td>\n",
" <td>-2.26</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>marsha</td>\n",
" <td>miles</td>\n",
" <td>2013-11-30</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>-1.93</td>\n",
" <td>-3.30</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>edward</td>\n",
" <td>riddle</td>\n",
" <td>2014-02-19</td>\n",
" <td>6.0</td>\n",
" <td>2.0</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>-0.16</td>\n",
" <td>-2.89</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" first last compas_screening_date decile_score score_text \\\n",
"0 miguel hernandez 2013-08-14 1.0 1.0 \n",
"1 kevon dixon 2013-01-27 3.0 1.0 \n",
"2 ed philo 2013-04-14 4.0 1.0 \n",
"3 marsha miles 2013-11-30 1.0 1.0 \n",
"4 edward riddle 2014-02-19 6.0 2.0 \n",
"\n",
" is_recid v_decile_score v_score_text is_violent_recid r_raw_score \\\n",
"0 0.0 1.0 1.0 0.0 -2.78 \n",
"1 1.0 1.0 1.0 1.0 -0.76 \n",
"2 1.0 NaN NaN NaN -0.66 \n",
"3 0.0 1.0 1.0 0.0 -1.93 \n",
"4 1.0 NaN NaN NaN -0.16 \n",
"\n",
" v_raw_score \n",
"0 -4.31 \n",
"1 -3.07 \n",
"2 -2.26 \n",
"3 -3.30 \n",
"4 -2.89 "
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"complete_scores.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Some final checks"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Has Recid Outcome:\t 6209\n",
"Has Violent Recid:\t 3998\n"
]
}
],
"source": [
"complete_scores = complete_scores.dropna(subset=['r_raw_score', 'v_raw_score'])\n",
"\n",
"print(\"Has Recid Outcome:\\t\", complete_scores['is_recid'].notnull().sum())\n",
"print(\"Has Violent Recid:\\t\", complete_scores['is_violent_recid'].notnull().sum())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore Distributions of Scores\n",
"Below we'll take a look at the distributions of each score to get a better understanding of what the data looks like that's coming out of the model. We examine the general recidivism cases below, the violent recidivism scores are not pictured.\n",
"\n",
"#### Raw Score Distributions"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEbCAYAAAA/P/RsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHlRJREFUeJzt3XucHFWZ//FPAuYyiozxmiZowOgBQUVFWIPcUVFcdUUf\ncAkEbyBGwOWqGDRKvABZV1w3oKIiBpVHQZYfN0UhiYgxsgiyGo+ghGWZGC55DQIZMkDy++PUbJrO\n9Ez1TE/3zJnv+/WaV89UnVP1dBGeOv1U9akJmzZtQkRE8jSx3QGIiMjIUZIXEcmYkryISMaU5EVE\nMqYkLyKSMSV5EZGMKcmLNEEIYWYIYVMIYdcmbW9pCGFRM7bVTCGEBSGEWwZYvyiEsLSFIckgtm53\nACLSr3cDT7Q7iH4sAv693UFIeUryIqNQjHFdu2PoT4zxUeDRdsch5SnJj2MhhJnA3cCZwEnAihjj\n2wbpczRwAvBH4B3A2cXPZ4EjgO2AdcAPgJOBtwNLgOfEGJ8MITwfWAt8IsZ4TrHN84HJMcYPlIj5\nTOAY4PnAKuCMGOO1xboXA18FDgB6gO8Dpxb73QZYCBwKTAOWAyfGGGPRd1Ox/hjgb8BrgZcW29sH\neAC4FDgzxrhhgBDfFkK4EngR8DPguBjjmhDCYuCVMca9q97L+4HPADvEGJ/21fOi5HFLjPGUEEIF\n+HoRxybgemBejPH+EsdrAbAnqTT7D8AJMcbvhhBOAY4HngvcBpwSY1xR9JkIfAr4cHGsVhb7W1Vs\n7+0xxt2LtgcD5wKzirjWDhaTtJZq8gLwNlIiOLVk+9cA3aRE+D3gNOBIYC7wsmI7HyOdBH4BTAL2\nKPruV7zuU7W9twDXDLbTEMK7im2/H9gJuBr4UQjh2SGEyaQkM7XY9qHFz6eL7j8mJf/3Fe/1ceBn\nIYSOql28D9i/eB/PAH4K3Fm83yOBg4HzBgnzeOBYYC9Sov9usXwJsFcIYUbN/r5fm+D7sbiIZ8/i\nvc0E/nWQPtUOBpaSkvw1IYRjizg/Qnpv1wA3FCd9SCeeE4CPF+vXAFeHELaq3mgI4eXAlaRj+2rg\nV8AHG4hLWkAjeQE4L8Z4Z4N9zooxrgUIIfwRODrGuKxYtzqEcCqwa4zxJyGEm0gJ9mZSEr2GlPAm\nkkbLM0ij3sHsQKpT3xNjXB1C+BxpRP4EcBAp+b0xxvhAEdexwEuKi6FvBl4fY7ylWHcEcA/p08c3\ni+1/M8b4x2L9+4HeGOPxxboYQvgI8MsQwmkxxr/XifG0GOP1xTY+DPw+hDArxnhzCOFuwIAvhxBe\nWByTj5d83xFYHWN8PITwPmCbEv36rAe+1HcyCSGcAXyy7xMQ8IUQwn7AvBDCacBxpP++lxft55E+\n7T2nZrsfBG6PMX62+PvsEML+wJQGYpMRppG8APy1wfaP9SV4gBjjFcBTIYSzQwhXhBD+AuwK9I38\nriUlNEhJ/jxgMvAq0ij+VwMkzWqXkMomfw4h3ArMB/4cY+wBXkFKgg9UxXVtjPGCYl0v8F9V6x4D\nfgfsUrX96uOwC/DSEMKjfT+kTwoTSZ9W6llR9ft/AxtIxwLSaP6w4ncD7ug7qQzi86RPRQ8VpaB9\ngD+U6NdndVWCfxbwYuDCmve2P+nT0fNIpbDf9nWOMXbHGE+OMT5Ys91dgVtrlq1sIC5pASV5gVS/\nbsTj1X+EED4DXEH6ZHg5cAipztvnWuANIYQdSKPtX5KS4d6kEfbVZXZa1KB3Ad5EKj8cAdwWQngV\nKYlPKBNvlYlsPhHB04/D1sCvgd2qfl5NSvADJeanqn6fUOyjt/h7CfD6oixyePH3oGKMTvq087Ei\nxq+SjmlZte8LUkmq+r3tTCrf9MVa71hW29RPu97+Gkr7KMlLM5wCnFyM9i4GVgMvoUgAMcY/kEbg\npwMrY4yPA8tIJZYDKFGPBwghvA04Psb48xjjSUAAHgbeCvyZVJp5blX7uSGE35Au0E4Cdq9a90xS\n0l5VZ3erSAn9f2OMd8UY7yKNcs8ttlXPq6t+fx2plr6qOA53kkbIHwBeT7o4XeZ9nwXsGGP8Tozx\nMOCfgAOKkk9DYozdpAvL2/W9r+K9fRx4S4zxYeB+0vWWvv1PDSH8LYSwR83m7mDztZY+r0VGFdXk\npRnuAw4JISwHng0sINVvJ1e1uZZ0wfTs4u+lpAt895QsWUA6aXwphLCWVN/fk3Rx87ekk8ZdwHeK\nmvO0Io7vxhjvDCFcDnw7hPBR0kXjBcCTwA/r7GsJ6aLtd4sk+2zgW8Bfi0RYz3khhEdIdfBvApfG\nGO+uWv890olieYyxq+T73gl4S1Xs/0w6kT4wUKcBnAN8OoSwBrgFmEO6WLxvsf7fgPnFNYQ7SfX4\nv5M+nVXfffV14IQQwjnAhaRPZW8n/beRUUIjeWmGo0kXB+8glW1WkxLi66raXEsaAfddnP0N6aN9\nqVE8QIzxatKnhrNII/cvkEb2N8QYnyLVrbcm1YUvJSXws4ruHyiWX0kqw0wB9ql3P3pRs38z6WS1\nEvjPot8Rg4R5NimR30A6HsfWrP8h6ThcUupNJx8hncCuA35Pqqm/Pca4sYFtVDuP9KWmc0i1/XcD\nh8YY+5LzItJ/vwtJNffnA4fEGJ9WiokxriZdUzmwiOtQ4PwhxiQjZIKeDCXSOiGEXUifPF4YY3yk\n3fFI/lSuEWmBEMILSHfFHE+6N14JXlpCSV7+TwjhJOBzAzRZG2N86Qjt+1A2f3GonucO8m3T0exZ\nwLeBP5G+TTpsIYSvAB8aoMnKGOMBA6yXcUBJXqp9m1SzrufJEdz3T0m38g1kzN6eF2P8K+nibTN9\nHvjaAOvr3Toq44hq8iIiGRttI3mdcUREhqbfL7CNtiRPV1fZW4efrlKpDLlvbnQsNtOx2EzHYrPc\njkWlUqm7TvfJi4hkTEleRCRjSvIiIhlTkhcRyZiSvIhIxpTkRUQypiQvIpIxJXkRkYwpyYuIZKzU\nN17NbCtgIenhENuQHl4wz93X1mnvwHtrFv/C3Q8aeqgi49OSJR1D7jtnzvomRiJjUdlpDRaQHvx7\nFPAQsBi4DHhjnfavBD7B06eOHatTxIqMWdUniM5O6O4uf8LQCSIPgyZ5M5sEnAic4O7XF8sOB+42\ns9nufnNN+8nALGClu/9tBGIWEZGSytTkdyOVaJb2LXD31aTneO7dT/udSCePVcOOTkREhqVMuWZG\n8XpfzfIuYPt+2u9KerjDZ83srUAP8CNgobvrIQYiIi1UJsl3ABvd/Yma5RtIT7yvtQtpXuM/kZ5a\n80rgy6QTwtzBdjbQlJkj2Tc3OhabjfVj0dnZzG2V31il0sQdj0Jj/d9FWWWSfA8w0cy2dvfqx79N\nBh7rp/18YJG7ryv+vsPMngJ+aGYnuftDA+1M88kPn47FZjkci0Yulg6ks7OT7u7u0u27uvK98JrD\nv4tqA52wyiT5e4vX6VW/A1TYsoSDu28E1tUsvqN43Z50d46IiLRAmQuvtwOPAPv2LTCzmcBMYHlt\nYzNzM/tJzeLdSeWdu4YaqIiING7Qkby7bzCzxcAiM3sQuJ90n/wyd19R3GI5DVjn7r3AjylKM8B/\nAq8BFpFKOI+O1BsREZEtlf0y1HzgGcCS4vU6YF6xbjZwI7A/sNTd3cymAKcCnyedFM4DvtjEuEVa\najjfOgV9sUjap1SSLy64nlz81K5bSs1Twt39YuDiJsQnIiLDoAnKREQypiQvIpKxsjV5kTGvY9VN\nw+j95qbFIdJKGsmLiGRMSV5EJGMq14i0wHBvwRQZKo3kRUQypiQvIpIxJXkRkYwpyYuIZEwXXkVK\nmLRmeBOo9k6f1aRIRBqjkbyISMaU5EVEMqYkLyKSMSV5EZGMKcmLiGRMSV5EJGNK8iIiGdN98tJy\nw5nXff3Ob2xiJCL500heRCRjSvIiIhlTkhcRyZiSvIhIxnThVUT6NdynWc2Zs75JkchwaCQvIpIx\nJXkRkYwpyYuIZEw1eRlTGv4iVVcnHQ93j0wwImOARvIiIhkrNZI3s62AhcDRwDbAdcA8d19bou9V\nwLPcfb+hhykiIkNRtlyzAJgLHAU8BCwGLgMGnEjEzI4FDgGWDT1EGY2GM/+MiLTOoOUaM5sEnAic\n4e7Xu/utwOHAXmY2e4B+s4AvAL9uVrAiItKYMjX53UglmqV9C9x9NbAa2Lu/DkV552LgbOCPw4xR\nRESGqEy5Zkbxel/N8i5g+zp9PglsAhYB32gkoEql0kjzpvXNzYgfi67Okd1+E3VuO/xYO6ZOHV7/\nztFxvDpbGEelMjrecz3jJV+USfIdwEZ3f6Jm+QZgSm1jM3sdcDLwenffaGYNBdTV1dVQ+z6VSmXI\nfXPTimMxVm5L7Ny2k+4mxLq+Z9qw+vd2t/94dXZ20t3COLq6Ru+0Brnli4FOWGXKNT3ARDOrPSFM\nBh6rXmBmU4DvAfPd/a4G4xQRkSYrM5K/t3idXvU7QIUtSzh7AjsDZ5vZ2cWyyaSTxKPAK9z9f4YR\nr4iINKDMSP524BFg374FZjYTmAksr2m7EngZ6WJt389PgFuK3/P5fCQiMgYMOpJ39w1mthhYZGYP\nAveT7pNf5u4rilsspwHr3L0HeFqZxsz+DvSofCMi0nplpzWYD1wCLAFuBO4B3lOsmw2sKV5FRGQU\nKfWNV3d/knTHzMn9rFsKTBig74eGGpxILiatGfoH2d7ps5oYiYw3mqBMRCRjSvIiIhlTkhcRyZiS\nvIhIxpTkRUQypiQvIpIxPeNVxo2Lrtqx3SGItJxG8iIiGVOSFxHJmJK8iEjGlORFRDKmJC8ikjEl\neRGRjCnJi4hkTEleRCRjSvIiIhlTkhcRyZiSvIhIxpTkRUQypiQvIpIxJXkRkYwpyYuIZExJXkQk\nY3poiIwpjT74o2PqVNb3TBuhaERGP43kRUQypiQvIpIxlWtEZEQsWdIx5L5z5qxvYiTjm0byIiIZ\nKzWSN7OtgIXA0cA2wHXAPHdfW6f9B4BTgR2AvwLnuvt3mhGwiIiUV3YkvwCYCxwF7APMAC7rr6GZ\nHQqcD5wN7Ax8Gfimmb1juMGKiEhjBk3yZjYJOBE4w92vd/dbgcOBvcxsdj9dngd8xt0vcve73f1C\n4A7gwGYGLiIigytTrtmNVKJZ2rfA3Veb2Wpgb+Dm6sbu/vW+381sa+CfSCP6M4cfrjRLx6qb2h2C\niLRAmSQ/o3i9r2Z5F7B9vU5mtjuwAtgK+BZw9VACFBGRoSuT5DuAje7+RM3yDcCUAfrdDewOvAY4\nD1gLfGqwnVUqlRIhNb9vbgY9Fl2drQmkyTqmTm1Jn9Gko7N5/606m7itkVSpjHyc4yVflEnyPcBE\nM9va3Z+sWj4ZeKxeJ3d/CHgIuM3MXgB8xsw+7e5PDbSzrq6uEiFtqVKpDLlvbsoci46Hu1sUTXM1\nOkVBmtagZ4SiaY3e7ub8t+rs7KS7SdsaaV1dI3uffG75YqATVpm7a+4tXqfXbpctSziY2b5mtlvN\n4juAqYAmERERaaEySf524BFg374FZjYTmAks76f96aR76qvtAdwPPDiUIEVEZGgGLde4+wYzWwws\nMrMHScl6MbDM3VcUt1hOA9a5ey/wFeA6MzsFuIJ0cjgNOMndN43UGxERkS2V/TLUfOASYAlwI3AP\n8J5i3WxgTfGKu/+sWHckqUxzOnC8u1/QvLBFRKSMUtMaFBdcTy5+atctBSbULLscuLwJ8YmIyDBo\ngjIRkYwpyYuIZEzzyYuMcpPW3DXkvr3TZzUxEhmLNJIXEcmYkryISMZUrhnD6s4k2dU5ZqctEJHm\n0kheRCRjSvIiIhlTkhcRyZiSvIhIxpTkRUQypiQvIpIxJXkRkYwpyYuIZExJXkQkY0ryIiIZU5IX\nEcmYkryISMaU5EVEMqYkLyKSMSV5EZGMKcmLiGRMSV5EJGNK8iIiGdPj/6TlLrpqx3aHIDJuaCQv\nIpIxJXkRkYwpyYuIZKxUTd7MtgIWAkcD2wDXAfPcfW2d9ocBnwReBqwBLgTOdfenmhCziIiUVHYk\nvwCYCxwF7APMAC7rr6GZvRW4hJTYXwV8AjgdOGOYsYqISIMGHcmb2STgROAEd7++WHY4cLeZzXb3\nm2u6fAS4zN2/Vvz9FzPbGXg/cFbzQheRwUxac9fmP7qnMqmnp3Tf3umzRiAiabUyI/ndSCWapX0L\n3H01sBrYu5/2C4HP1izbCDxnKAGKiMjQlanJzyhe76tZ3gVsX9vY3X9b/beZPRs4jlTHFxGRFiqT\n5DuAje7+RM3yDcCUgTqaWQdwBTCVVJsfVKVSKdOs6X3HpK7Ouqs6t62/rt06pk7Nen+jWSPHoqOz\nff+GKpWR3/d4yRdlknwPMNHMtnb3J6uWTwYeq9fJzJ4HXAm8AniTu99TJqCurq4yzbZQqVSG3Hes\n6ni4u9/lndt20l1n3Wiwvmday/bVMXUq6xuoQ+es0WPR292+f0NdXetHdPu55YuBTlhlavL3Fq/T\na7fLliUcAMxsJnAzsAOwT20JR0REWqNMkr8deATYt29BkcRnAstrG5vZC4Abi23PdvffNyNQERFp\n3KDlGnffYGaLgUVm9iBwP7AYWObuK4pbLKcB69y9F/gP4HnAAUCPmb2o2NSmel+eEhGRkVF2Fsr5\nwDOAJcXrdcC8Yt1s0sh9fzP7DfBu0ih+Zc02nmpgfyIi0gSlkm5xwfXk4qd23VJgQtWirZoSmYiI\nDJtG1m3WseqmdocgIhnTLJQiIhlTkhcRyZiSvIhIxpTkRUQypiQvIpIx3V3TBLpDRkRGKyV5GZKL\nrtqx3SFIxpYs6Rhy3zlzRnZys7FG5RoRkYwpyYuIZExJXkQkY0ryIiIZU5IXEcmYkryISMaU5EVE\nMqYkLyKSMSV5EZGMKcmLiGRMSV5EJGNK8iIiGVOSFxHJmJK8iEjGlORFRDKmJC8ikjEleRGRjCnJ\ni4hkTEleRCRjesbrOKVntEquyjwftrMTuru3bJfj82E1khcRyVipkbyZbQUsBI4GtgGuA+a5+9pB\n+r0UuB3Yyd3/d3ihishYMmnNXUPu2zt9VhMjGd/KlmsWAHOBo4CHgMXAZcAb63Uws5cD1wLPHF6I\nItIOw0nSMnoMWq4xs0nAicAZ7n69u98KHA7sZWaz6/Q5EbgF6G5msCIi0pgyNfndSCWapX0L3H01\nsBrYu06fdwLHACcPKzoRERmWMuWaGcXrfTXLu4Dt++vg7gcAmNl+jQZUqVQa7dKUvsPS1dme/Q6g\nc9uBY+qYOrVFkbTfeHqvgxkrx6Kjc+T/n+rsZx+Vyuj7f3m4yiT5DmCjuz9Rs3wDMKXZAXV1dQ2p\nX6VSGXJfgI5VNw2572jTuW0n3Q8PXClb3zOtRdG0V8fUqazv6Wl3GKPCWDoWvd0jW+nt7Oyku599\ndHWNzVsoBxrglinX9AATzaz2hDAZeGwYcYmIyAgrk+TvLV6n1yyvsGUJR0RERpEySf524BFg374F\nZjYTmAksH5GoRESkKQatybv7BjNbDCwysweB+0n3yS9z9xXFLZbTgHXu3juy4YqISCPKTmswH7gE\nWALcCNwDvKdYNxtYU7yKiMgoUuobr+7+JOme9y3ue3f3pcCEOv3qrhMRkZGnCcpERDKmqYbHsHrT\nBaf7ocfHffAiMjCN5EVEMqYkLyKSMSV5EZGMKcmLiGRMSV5EJGNK8iIiGVOSFxHJmJK8iEjGlORF\nRDKmJC8ikjEleRGRjCnJi4hkTEleRCRjmoWyzerNJCki0gwayYuIZExJXkQkY0ryIiIZy6cmf8vP\n6Hi4u91RiIiMKhrJi4hkLJ+RvIhkY9Kau4bct3f6rCZGMvYpyYuIFJYs6Rhy3zlz1jcxkuZRuUZE\nJGMayTeBvtAkIqOVRvIiIhnTSF5EslLqom33VCb19GyxOMeLthrJi4hkrNRI3sy2AhYCRwPbANcB\n89x9bZ32uwPnAa8B7gPOcveLmxHwSFFdXURyVLZcswCYCxwFPAQsBi4D3ljb0MyeD/wU+D7wQeBN\nwLfM7G/u/rMmxCwiMuoM5/ZLGLlbMAct15jZJOBE4Ax3v97dbwUOB/Yys9n9dPkQ8DBworv/yd3/\nHVgCnNLEuEVEpIQyI/ndSCWapX0L3H21ma0G9gZurmm/N7Dc3TdWLVsKLDazCe6+aTgB1/ONy7dj\nfc+0kdi0iIwTOX7TtkySn1G83lezvAvYvk773/XTtgN4LvBgIwGKiIwFwzlBJJWmxFGrTJLvADa6\n+xM1yzcAU+q0f7yfttRp/zSVytDe6DFfGJkDJCIylpVJ8j3ARDPb2t2frFo+GXisTvvJNcv6/u6v\nfbUJJeIREZGSytwnf2/xOr1meYUtSzh97ftr+yjpgqyIiLRImSR/O/AIsG/fAjObCcwElvfT/iZg\nHzOrHpXvD/yq5mKsiIiMsAmbNg1+s4uZfYn0RaijgftJ98k/7u77FbdYTgPWuXuvmb0QiMClwFeA\ng4B/BQ529xtG4k2IiEj/yk5rMB+4hHS/+43APcB7inWzgTXFK8W3YA8mfdv1d8DHgKOU4EVEWq/U\nSF5ERMYmTVAmIpKx7KYaNrO9gLNJ5aJuUonpTHfvbWtgbWBmrwXOAXYH1gPXAKe5+7q2BtYmZjYZ\nWAmc6+5L2h1PKzQ6ueB4YWYXAFu7+4faHctIy2okb2YvIf0jXgm8ijSp2pHAl9oZVzuYWQX4OXA3\n8AbgvcAegLczrnYxs22An5D+XYwnC9g8ueA+pG+kX9bOgNrJzCaY2eeAY9sdS6tkleRJt3Ve7u4n\nuftf3P3npLt8DmxvWG1xGOmbxx9x91Xu/itgHnCgmb24vaG1lpkdBNwGvLDdsbTSECYXzJqZ7Qjc\nABwH/E+bw2mZrJK8uy9z97l9fxflincB43GK4yuBw9z9qaplfd9TeE4b4mmnfwQuprgDbBzpd3JB\nYDVpIsHxZjbpy5qvJH3CHReyq8n3MbNuYFvSbZwL2xxOy7n7X4C/1Cw+nfQt5f9ufUTt4+4n9v1u\nZu0MpdUanVwwa8V1mCUwvv4djKkkX3zTtt4ZeIO7TynaTSQ9rGQa6QlVV5vZ3iM1zXE7lD0WVe2/\nBLwdeFfN6H5Ma/Q4jDONTi4oGRpTSZ40Itm5zrr/mzKhmD7htwBmNhdYQbr4WDv3/VhW6lgUd1d8\njXSh6Th3v7IFsbVSqeMwTjU6uaBkaEwl+WJE8qd6683sFcB27n591eI7itftRjK2VhvsWACY2RTS\n3TQHA3Pc/futiK2VyhyHcax6csF7q5bXm1xQMpTVhVdSOeIHRXLrs0fx+sc2xNM2RcnqR6Q7i/4x\nxwQvg2p0ckHJ0JgayZdwMXAa8O3iXtjtgQuAS939D22NrPWOI530PgTcbmYvqlr3UD91WsmMu28w\ns8XAIjN7kM2TCy5z9xXtjU5aJauRvLv/DTgAeAGpJv890hdg5g7UL1NHFK8XkiaQq/7Zs11BScsN\nNLmgjAOaoExEJGNZjeRFROTplORFRDKmJC8ikjEleRGRjCnJi4hkTEleRCRjSvIiIhlTkhcRyZiS\nvIhIxnKbu0YyZWb7kZ4N8BjwTGAPd9/QT7uLSM8ReClwFfAt4D+AZ5FmX7yN9GjELwKPufv8Yl6f\nLuAgd7/BzI4A3unudZ8sYWbvJk0ZsBF4CjjV3ZcX27oA2KlYd4G7f9XMZgDnkyYHmwB8193PLSYM\n+yWwqli3L7AD6WH0zyy2scDdr2r8qIloJC9jy67A+9z91f0l+Cod7r6Lu58OfJiUUN8AzCIl0ENI\ncxodXLQ/GFgLHFT8/U7gx4PEci7wUXffHTgT2K9Yvhj4s7vvRHqGwTFmNos0f8yN7v5KYC9gjpkd\nXvSZAZzl7i8nPZf3O8CR7v5a4B3A+ePtubzSPEryMpbc6+73lGh3U9XvpwMPmNlppJF0hTSqvwmY\nYWYvICX5hcCbiodf7wtcM8g+fgj8xMwuJD0z95xi+UHANwDc/WF335U0KdxepE8UuPvDwEXAW4s+\nTwK/Ln5/A2n+9yvM7LYijk3Aq0q8b5EtKMnLWPLoENr9ADiGNPvivwG3AhOKp4f9P9Ko/h+Ab5KS\n63uBX7v7gPty90+REvctwNHAr4s5/J8kJWUAzGxHUll0Qs0mJgLPKH7fUPXkpq2AVe6+W99PEd9P\nS753kadRkpfcvQX4nLtfSkq+e5ISKaSSzWnAHe7eC9xAqtUPWKoxs63NbDXwTHe/APgo6RGEzwB+\nDry/aLct8AtSmWgFMK9q+VHA9VtsPLV7mZntU7TdDbiT9AlEpGFK8pK7M0hllVtIF0SXkZIupAS8\nHZuT7U+BF5JG+HUVo+6PA983s1tJT+D6QHGd4GPAzmb2e+BXwBfd/b9I8/sfaGZ3ACuBy0glm9pt\nPwAcCpxrZreTnolwZMkylcgWNJ+8iEjGdAuljClmFoBL66yO7n5Yk/azP6mG358b3f1fmrEfkZGm\nkbyISMZUkxcRyZiSvIhIxpTkRUQypiQvIpKx/w8RvXaZ4+Se2AAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x116bc9080>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def bin_plot(df, values_column, binary_by_column, n_bins=25, bins=None):\n",
" d_true = df.loc[lambda x: x[binary_by_column] == True][values_column]\n",
" d_false = df.loc[lambda x: x[binary_by_column] == False][values_column]\n",
" \n",
" d_min, d_max = df[values_column].min(), df[values_column].max()\n",
" \n",
" if not bins:\n",
" bins = np.arange(d_min, d_max, (d_max + 1 - d_min) / n_bins)\n",
" \n",
" fig, ax = plt.subplots()\n",
" sns.distplot(d_false, ax=ax, bins=bins, kde=False, norm_hist=True, color='coral')\n",
" sns.distplot(d_true, ax=ax, bins=bins, kde=False, norm_hist=True, color='blue')\n",
" sns.plt.title(\"%s by %s\" % (values_column, binary_by_column));\n",
" \n",
"bin_plot(complete_scores, 'r_raw_score', 'is_recid')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Raw scores look to be normally distributed and also indicate the model works -- there seems to be a significant shift in raw scores in individual who actually recidivated.\n",
"\n",
"#### Decile Score Distributions"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEbCAYAAAA8pDgBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucVVX9//EXgiBj1oRKMuJX1HRbYqL5rSTBS+bX/JmZ\n2UczBTQr80Zl2kUrC7S8pmV8MzUrsV9+yi5+U1FKQdHQSiNT3KUyhIxCiBgCcp3vH2sdO9/DOTP7\nnJnDOcx+Px+PeZyZtdda57P3zOzP2WvtS7/Ozk5ERCSftmh0ACIi0jhKAiIiOaYkICKSY0oCIiI5\npiQgIpJjSgIiIjmmJCCZJUlyVJIkvXJOcZIk7UmSnBW/n5EkyRW90W8zS5LkoiRJ/thLfY1IkqQz\nSZKRvdFfbyr+3VZYviRJkgmbMCTpwoBGByC59Z/AikYHsRlbAAwDljQ6kDL0u92MKAlIQ6Rp+s9G\nx7A5S9N0PfBCo+MoR7/bzYuSgFSUJMmbge8Bo4GngZ+ULN8B+DbwPuAV4A7g3DRNX47L/yMuPxRY\nFdufl6bpuiRJ2oEr0jS9tsz7jgMuBIYDTwFfTtP0jowxtwHXAWOBTmA6cGaapovj8uOALwN7AM8A\nX0zT9H/isv2Ay/n3J9kfAxemabo2Dl+cAzwJHA1cmqbpxTXE2j9JkmuAU+M2+06appckSTKIsFP/\nXJqmNxatz0xgZpqmXylZzxHAPGDvNE3/miTJMcBk4M3A88CUNE0vz7jN2gEHTohF+wCD6IXfbZIk\n/YFL4vpuAXw9S0yy6WhOQMpKkmRL4E7CDmB/4CLg/JJqv4ivBwDvB3YDfhrbDyLsgAcTdsgfil9f\noQtJkvwXcA1hR703YYf+8yRJDsgY+hRgS+Cd8X1HAFfGvg8BbiXs3PcGfhD73jNJkt2B+4G5wDuA\njwMnE3ZgBfsCy4D9gJtrjHUUYRjnncBngQuSJDkxTdPVwM+A44u2xXDgQGBqVyucJMlQwk78OiAB\nzgMuTpLkPV21K3EacCxwbJqmL9F7v9svA+MJ2/IQ4APAtlXEJXWmIwGp5DDgP4B3pWm6FHgySZI9\ngG/AazvUtwEHp2m6JpZ9FFiYJMlehJ3vCODAwvBAkiSfBHbu5n2/BFyepumt8ednkiR5O2GH+eEM\nce8CpEB7mqavJknyEWCbuOwM4Ndpml4Zf74qSZKtgdcRPgXPA85O07QTeCpJks8QdvYXFfU/KU3T\nRXF9bq4h1peAU9I0XUHYpvvHuH5C2NnfmyTJ9nGbnQD8KU3Tv3WzzjsSEl9HmqbzgflJkiwCumtX\n7NY0Tf8Y16tXfrdJkvQDTidss2mxbBzQXkVcUmdKAlLJSGB+TAAFjxR9vxfQAixNkqS07Z7AroQd\n8Wvjw2ma3pXhffcC3pkkyZeKyrYk+w7tYsIn/ReTJPkd8Cv+/Un6rZR8qk7TdBJAkiRfB2bHBFAw\nK7737vHnFYUE0INYH48JoOBPwMfi9w8AzwHHAf8NfCSuS3f+TPjk/vMkSeYThm6mlsTanWeLvu+t\n3+12wJuAR4vqPZckSUcVcUmdaThIKukE+pWUrSn6fgAwnzC8Ufy1O3BPrFvaPosBwBdL+twLOCpL\n4zRNnTA+fxZhrPrbQGEH1VVMr5YpK/x/9K9Qp5ZY15d5jzUx9k7gFsDi8NQ+xCGYrqRp2pmm6Yfi\n+19PGK6aVeVpmKuKvu+t320hoXb1dyQNpiQglTwOjEiS5E1FZfsVfT8XaAOWp2n6dJqmTwNrgW8B\nQwmfhndOkuS18d8kScYnSfJwN+87F9i50Gfs9yPxq1tJkkwCdk3T9KY0TY8HPggcGtfjbyXrQJIk\ndydJMjG+77viEEbB6LhOz/RirHslSVJ8BH4AYbK5YCowhjCO/tssn+bjnMY1aZrOSdP04jRNDyAk\nj+O7a1tBb/1uXyRMUr+zqN52wE41xiV1oOEgqeR3hJ3Bj5Mk+RzhH/eLRcunA08AP43L1wPfJYy/\nt8evp4Gb4nDJEMLk8o+6ed/LgP+fJMlT8T3eC3wVODFj3HsC/5UkyRmESdwTYyz/BK4GHogXMt1F\nmPAcC3waeJlw9s+3kyT5LmHI40rgR2maLiszLFJrrNsBP0iS5FLCzvFjwDGFhWmazk2S5M/AuYTJ\n6SxeAj6WJMlKwpHADoTkcnPG9qV65XebpmlnkiTfIkx+PxvbXIr2O01FRwJSVpqm64AjCZ8AZxM+\nBV5ZtHwD4VTJpcB9wAzCKY5Hpmm6Pp7HfjThH/4Rwlk5PwUmdfO+vwTOJkyuPgl8BvhEHObJ4nTC\nzmYa8BfC5PZRaZpuSNP098A4wlDRE4RP28ekaTo3TdMO4Ajg7cAcws70R8CZvRzrdGA1YZt8DTin\nMGla5CfABsJ8Rrfi0cLRhMn8vwK/BG4nzI9UrZd/t1cQku91wEPAY8Dfa4lL6qOfniwm0lySJLkc\nGJam6UmNjkX6Ph2WiTSJJEneQbje4JOEozCRulMSkM1GkiS/JIy7V/LzNE0nbKJw6uFQwsVV30nT\ndFZvdJgkyRJgqy6qTCy+QlnyR0lANidnAFt3sXz5pgqkHtI0/SbwzV7u9h10Pfe3uJffTzYzmhMQ\nEcmxze1IQBlLRKQ2ZS/w29ySAB0dfe+K87a2tj65XvWi7VUdba/q9MXt1dbWVnGZrhMQEckxJQER\nkRxTEhARyTElARGRHFMSEBHJMSUBEZEcUxIQEckxJQERkRxTEhARybHN7orhnmiZ2ys3Zux9bdbo\nCEQkp3QkICKSY0oCIiI5lmk4yMz6A5OBCYSHTU8DznT3RRXqnwqcB+wCPAtc7u43FS0fClwLHA6s\nAW4CLnD3dTWviYiIVC3rkcBFhIdyjwPGAsOB28pVNLMPAf8NXAq8BbgKuN7Mji6qdhuwA3AQIbGc\nQnjotoiIbELdJgEzGwhMBL7k7tPd/VHgBODdZja6TJPtgK+6+w/dfZ673wA8Drwn9ncAcCAw3t3n\nuPudhKOGs81sUO+sloiIZJFlOGgUYQhoRqHA3dvNrB0YAzxUXNndryt8b2YDgA8Sjgi+HIvHAPPd\nfV5RsxnxPUYBD1e5DiIiUqMsSWB4fF1YUt4B7FSpkZntD8wG+gM3AncU9VeuL2J/XSaBrh6O0K2O\n1trb1lmP1iuHtL2qo+1VnTxtryxJoAXY4O5rS8pXA1t10W4esD+wL3ANsAi4IPb3anFFd19rZp3d\n9Af07MliLS8vq7ltPbXSN5+YVi998clP9aTtVZ2+uL26SmpZksAqYAszG1By9s4gYEWlRu7+IvAi\n8Od4NtBXzewrsb//M/ZvZlsSnn9ZsT8REel9Wc4OWhBfh5WUt7HxsA5mdpCZjSopfhwYDAyJ/ZXr\ni3L9iYhI/WRJAnOA5YTTOQEwsxHACOD+MvU/T7imoNg7gMXAEmAWsKuZFc8nHBLf488Z4xYRkV7Q\n7XCQu682synAFWa2hLAznwLMdPfZ8RTSIcBSd18DXA1MM7PPAb8iJI/zgc+6e6eZ/Z4wYXyrmZ0F\nvAm4DLgqthcRkU0k68ViFwK3AFOB+4D5wHFx2Wjg+fiKu98Tl51MGAb6PHC2u38vLu8knDa6CHiA\ncLXwDcDXe746IiJSjX6dnZ2NjqEanT06O6hJ7yLa+h7rc2cj1FNfPHujnrS9qtMXt1c8O6hfuWW6\ngZyISI4pCYiI5JiSgIhIjikJiIjkmJKAiEiOKQmIiOSYkoCISI4pCYiI5JiSgIhIjikJiIjkmJKA\niEiOKQmIiOSYkoCISI4pCYiI5JiSgIhIjikJiIjkmJKAiEiOKQmIiOSYkoCISI4pCYiI5JiSgIhI\njg3IUsnM+gOTgQnANsA04Ex3X1Sh/vHAF4HdgeeBG4DL3X19XH4G8N2SZuvdPVM8IiLSO7IeCVwE\njAfGAWOB4cBt5Sqa2fuAWwg7/rcBXwA+D3ypqNrewO3AsKKvHauOXkREeqTbT95mNhCYCJzj7tNj\n2QnAPDMb7e4PlTQ5HbjN3a+NPz9jZm8BTgEmxbKRwL3u/kJvrISIiNQmy5HAKMIQ0IxCgbu3A+3A\nmDL1JwNfKynbALyx6Oe9gLnZwxQRkXrIMgY/PL4uLCnvAHYqrezufyj+2cxeD3yKMI+Ame1ISAjv\nM7OLgK2BmcD57t5RTfAiItIzWZJAC7DB3deWlK8GtuqqoZm1AL8CBhPmBiAcBQCsBU4AtgMuAX5n\nZvu5+6qu+mxra8sQcgUdrbW3rbMerVcOaXtVR9urOnnaXlmSwCpgCzMb4O7risoHASsqNTKz7QiT\nv28F3uvu8wHc/R4z297dlxTVfYJwpHEkFSacCzo6aj9YaHl5Wc1t66mVnq1X3rS1tWl7VUHbqzp9\ncXt1ldSyzAksiK/DSvtl4yEiAMxsBPAQsAswtnSIqDgBxJ+fB5ZQZnhJRETqJ8uRwBxgOXAQMBVe\n28mPAO4vrWxmQ4H7gPXAaHefV7L8HMLQ0M6FISYz2xnYHniixvUQEZEadJsE3H21mU0BrjCzJcBi\nYAow091nx1NIhwBL3X0N4SKw7YBDgVVmtkPsqjNeXHYHcDFwo5ldAmwLXAPMKpyCKiIim0bWi8Uu\nJFwANpXwKX8+cFxcNppwVfBoMxsMHAu8Dngklhe+FgK4+zPAewlDP48Q5g3+Ahzd89UREZFq9Ovs\n7Gx0DNXo7NHE8NxZvRhK72l9j/W5iah66osTd/Wk7VWdvri94sRwv3LLdAM5EZEc0w3bRESKfP/7\nsGxZS6PD2MhJJ62sS786EhARyTElARGRHFMSEBHJMSUBEZEcUxIQEckxJQERkRxTEhARyTElARGR\nHFMSEBHJMSUBEZEcUxIQEckxJQERkRxTEhARyTElARGRHFMSEBHJMSUBEZEcUxIQEckxJQERkRxT\nEhARybFMzxg2s/7AZGACsA0wDTjT3RdVqH888EVgd+B54AbgcndfH5cPBa4FDgfWADcBF7j7up6s\njIiIVCfrg+YvAsYD44AXgSnAbcCBpRXN7H3ALcCngbuAfYHrgS2BSbHabUAncBCwI/BDYB1wQU1r\nISIiNel2OMjMBgITgS+5+3R3fxQ4AXi3mY0u0+R04DZ3v9bdn3H3nwNXAafE/g4gJI/x7j7H3e8E\nzgPONrNBvbNaIiKSRZY5gVGEIaAZhQJ3bwfagTFl6k8GvlZStgF4Y/x+DDDf3ecVLZ8R32NUhnhE\nRKSXZBkOGh5fF5aUdwA7lVZ29z8U/2xmrwc+RZhHKPRXri9ifw9niElERHpBliTQAmxw97Ul5auB\nrbpqaGYtwK+AwcAXivp7tbieu681s87u+gNoa2vLEHIFHa21t62zHq1XDml7VUfbqzqtrc23r2hr\nq09MWZLAKmALMxtQcvbOIGBFpUZmth1wO/BW4L3uPr+ov0EldbcE+nXVX0FHR0d3VSpqeXlZzW3r\nqZWerVfetLW1aXtVQdurWm0sW9Z8+4qOjpU1t+3qQ0CWOYEF8XVYab9sPKwDgJmNAB4CdgHGlgwR\nLajQF5X6ExGR+siSBOYAywmncwKv7eRHAPeXVo7XANwX+x7t7n8pqTIL2NXMiucTDonv8ecqYhcR\nkR7qdjjI3Veb2RTgCjNbAiwmXCcw091nx1NIhwBL3X0N8F1gO+BQYJWZ7RC76owXl/0emA3camZn\nAW8CLgOuiu1FRGQTyXrbiAsJF4BNJXzKnw8cF5eNJlwVPNrMBgPHAq8DHonlha+FAO7eCXwQWAQ8\nQLha+Abg6z1fHRERqUamK4bjhPC58at02QzCpG5B/wz9vUBIBCIi0kC6gZyISI4pCYiI5JiSgIhI\njikJiIjkmJKAiEiOKQmIiOSYkoCISI4pCYiI5JiSgIhIjikJiIjkmJKAiEiOZbp3kNTZH+9pygfe\nrHzLgY0OQUTqTEcCIiI5piQgIpJjSgIiIjmmJCAikmNKAiIiOaYkICKSY0oCIiI5piQgIpJjSgIi\nIjmW6YphM+sPTAYmANsA04Az3X1RN+12A+YAe7r7c0XlRwJ3lGmyU3E9ERGpr6xHAhcB44FxwFhg\nOHBbVw3MbA/gHmDrMov3Bh4DhpV8dWSMR0REekG3RwJmNhCYCJzj7tNj2QnAPDMb7e4PlWkzEZgE\n/L1CtyOBx939hZojFxGRHstyJDCKMAQ0o1Dg7u1AOzCmQpsPAJ8Azq2wfCQwN2OMIiJSJ1nmBIbH\n14Ul5R3ATuUauPuhAGZ2cOmyOL+wJ/B2M5sDbA/8ATjf3dNsYYuISG/IkgRagA3uvrakfDWwVQ3v\nuVtsNwj4ODAQuBB4wMxGuvvirhq3tbXV8JZRR2vtbeus9Q3NF1trT7Z1nfXo7yCHtL2q09rafP+P\nbW31iSlLElgFbGFmA9x9XVH5IGBFtW/o7n8zs22BZe6+AcDMjgX+AZwMXNlV+46O2ueOm/Ge/RAS\nwLImjG1lD7Z1PbW1tfXo7yBvtL2q1cayZc33/9jRsbLmtl19CMgyJ7Agvg4r7ZeNh4gycfelhQQQ\nf14JPEuF4SUREamPLElgDrAcOKhQYGYjgBHA/dW+oZkdY2bLzWz7orJtgD2AJ6rtT0REatftcJC7\nrzazKcAVZrYEWAxMAWa6++x4CukQYKm7r8nwnjOBfwE3m9n5MYZLgCXAzTWuh4iI1CDrxWIXArcA\nU4H7gPnAcXHZaOD5+Notd38JOAxYSzjtdAZhbuFQd381YzwiItIL+nV2djY6hmp09mhieO6sXgyl\n9zTtxHCTPmheE53V0faqzm9+05wTwyed1OOJ4X7llukGciIiOaYkICKSY0oCIiI5piQgIpJjSgIi\nIjmmJCAikmNKAiIiOaYkICKSY0oCIiI5piQgIpJjSgIiIjmmJCAikmNKAiIiOZbl8ZKSU81611Xa\nrNERiPQZOhIQEckxHQmI9HHf/z4sW9bS6DA20pP740vvURIQkYaYOrX5EhNAa2ujI9i0lARks6NP\ntiK9R0lANj/tTzBw1apGR1FGW6MDEKmaJoZFRHJMSUBEJMc0HCTSSzTRKZujTEnAzPoDk4EJwDbA\nNOBMd1/UTbvdgDnAnu7+XFF5C3A1cGyM4WfAZ9z9lRrWIbMf/mbXenZfs09/dGmjQxCRnMo6HHQR\nMB4YB4wFhgO3ddXAzPYA7gG2LrP4OuBA4Cjg/cDBsUxERDahbo8EzGwgMBE4x92nx7ITgHlmNtrd\nHyrTZiIwCfh7mWXDgROB97j77Fh2GnCfmZ3v7gt7skKbo+//YkdWrhrS6DA2MuGoZxsdgojUWZYj\ngVGEIaAZhQJ3bwfagTEV2nwA+ARwbpllo4ENwINFZQ8C6wlHByIisolkmRMYHl9LP6F3ADuVa+Du\nhwKY2cEV+lvs7muL6q8zs8WV+ivW1lb7udgtg1+quW29tQwe3OgQNtL6huadUWzG7dXSxDOwrU0c\nWzNqxu3V1lafmLIkgRZgQ/FOO1oNbFXDe7YAr5Ypz9RfR0dHDW8ZrGzKC4zCDq0ZY1v28rJGh1DB\njk25vdYsa87t1drayrImja0ZNev26uio/Yr0rj48Z0kCq4AtzGyAu68rKh8ErKghnlWxbala+5M6\nadazqVqa7yBAZLOVZU5gQXwdVlLexsZDRFksAIbG004BMLMBwNAa+xMRkRplSQJzgOXAQYUCMxsB\njADur+E9HyQcgRxQVHZgjOXBsi1ERKQuuh0OcvfVZjYFuMLMlgCLgSnATHefHU8hHQIsdfc1Gfpb\naGYO3GhmpwL9gOuBm/N4eqiISCNlvVjsQuAWYCpwHzAfOC4uGw08H1+zOg14CLgT+DVwL/CpKtqL\niEgv6NfZ2dnoGKrR2ZOzg/zy2tvWU7OeHdSstL2q0/KW/ZvybJdm1axnB/XkeRXx7KB+5ZbpLqIi\nIjmmJCAikmNKAiIiOaYkICKSY0oCIiI5piQgIpJjSgIiIjmmJCAikmNKAiIiOaYkICKSY0oCIiI5\npiQgIpJjWZ4sJiKSH+1PMLApb1BY+/PVu6IjARGRHFMSEBHJMQ0HifR1TTq8sWbYmxsdgqAjARGR\nXFMSEBHJMSUBEZEcUxIQEckxTQyLSEMMfP7pRodQ3uDBjY5gk8qUBMysPzAZmABsA0wDznT3RRXq\n7w9cA+wLLAQmufuPi5YfCdxRpulO7v5cNSsgIiK1yzocdBEwHhgHjAWGA7eVq2hm2wN3A48C+wHf\nBm40s8OLqu0NPAYMK/nqqHoNRESkZt0eCZjZQGAicI67T49lJwDzzGy0uz9U0uQ04GVgortvAJ4y\ns/2AzwH3xDojgcfd/YVeWg8REalBliOBUYQhoBmFAndvB9qBMWXqjwHujwmgYAbwbjPrF38eCcyt\nOloREelVWeYEhsfXhSXlHcBOFeo/VqZuC7Ctmb0E7Am83czmANsDfwDOd/c0a+AiItJzWZJAC7DB\n3deWlK8GtqpQ/9UydYn1d4uvg4CPAwOBC4EHzGykuy/uKpi2ttrvpNcy+KWa29ZbS87OSOgpba/q\naHtVpxm3V0/2fV3JkgRWAVuY2QB3X1dUPghYUaH+oJKyws8r3P05M9sWWFYYMjKzY4F/ACcDV3YV\nTEdH7XPHK5vw/ikQ/uCaNbZmpO1VHW2v6jTr9urJvq+rBJJlTmBBfB1W2i8bDxEV6per+wphwhh3\nX1o8Z+DuK4FnKT+8JCIidZIlCcwBlgMHFQrMbAQwAri/TP1ZwNiiSWCAQ4AH3X2DmR1jZsvjqaSF\n/rYB9gCeqHoNRESkZt0OB7n7ajObAlxhZkuAxcAUYKa7z46nkA4Blrr7GuBG4Hzge2Z2NXAYcCJw\nROxyJvAv4GYzOz/GcAmwBLi5V9dORES6lPVisQuBW4CpwH3AfOC4uGw08Hx8JV5FfAThauHHgLOA\nce5+b1z+EiExrCWcOjqDMLdwqLuXTiiLiEgd9evs7Gx0DNXo7MnkiF/enBckN+tEVLPS9qqOtld1\nmnV72Xm1nx0UJ4b7lVumu4iKiOSYkoCISI4pCYiI5JiSgIhIjikJiIjkmJKAiEiOKQmIiOSYkoCI\nSI4pCYiI5JiSgIhIjikJiIjkmJKAiEiOKQmIiOSYkoCISI4pCYiI5JiSgIhIjikJiIjkmJKAiEiO\nKQmIiOSYkoCISI4pCYiI5NiALJXMrD8wGZgAbANMA85090UV6u8PXAPsCywEJrn7j4uWtwBXA8fG\nGH4GfMbdX6l5TUREpGpZjwQuAsYD44CxwHDgtnIVzWx74G7gUWA/4NvAjWZ2eFG164ADgaOA9wMH\nxzIREdmEuk0CZjYQmAh8yd2nu/ujwAnAu81sdJkmpwEvAxPd/Sl3/w4wFfhc7G84cCJwhrvPdvcH\nYpuPmNmOvbJWIiKSSZYjgVGEIaAZhQJ3bwfagTFl6o8B7nf3DUVlMwhJox8wGtgAPFi0/EFgPeHo\nQERENpEsSWB4fF1YUt4B7FShfrm6LcC2cflid19bWOju64DFFfoTEZE6yTIx3AJsKN5pR6uBrSrU\nf7VMXWL9csu76u//aGtr665KRZ/+Vu1tRUT6oixJYBWwhZkNiJ/YCwYBKyrUH1RSVvh5RYXlXfVX\nrF/34YqISFZZhoMWxNdhJeVtbDzsU6hfru4rhAnjBcDQeNopAGY2ABhaoT8REamTLElgDrAcOKhQ\nYGYjgBHA/WXqzwLGxknggkOAB+Nk8YOEI5ADipYfGGMpniwWEZE669fZ2dltJTP7JuFCsQmECdwp\nwKvufnA8hXQIsNTd15jZm4AUuJVwQdhhwJXAEe5+b+zvp4QLyU4lDPHcREgSE3pz5UREpGtZLxa7\nELiFcL7/fcB84Li4bDTwfHwlXkV8BGEn/xhwFjCukACi04CHgDuBXwP3Ap/qyYqIiEj1Mh0JiIhI\n36QbyImI5FimG8hJ74tzJ5cBhwODgYeBc939rw0NbDNgZu8inIBwmLvPaHA4TcvMTgPOJ1yE+SRw\nXsmwrERmtjXwTeBDhGuZfk/4f3yyoYFtAjoSaAAz2wL4JbAH8AHCfMrLwO/MbNtGxtbs4j/rzUD/\n7urmmZmNB75L2LHtDcwEbo9n9snGriGcxPJhwpmLrwLTzKzbC1g3d0oCjbEP4Q/tVHd/JH7aOBl4\nHfD/GhpZ87sKeK7RQTSzeHr214BL3f0H7v404QaOTxNP4JCNHANMcfcH3X0ucAHhCOqtjQ2r/jQc\n1Bj/INxGOy0qK9xw742bPpzNg5kdSUiS7wP+0uBwmlkC7Ew4TRuAeI3OqIZF1Pz+CRxvZrcCy4CP\nAS8BzzY0qk1ASaAB3P1F4I6S4nMIcwP3bPqImp+ZbQfcCJxC+OeUyvaIr61mdi8wEngK+IK7P9S4\nsJraJwinwC8i3NF4JXC4uy9raFSbgIaDmoCZHQ18A7gqHorKxq4Dbnf3aY0OZDPw+vj6I+AGwnU7\nfwXuNbO3NCyq5vZm4AXCkea7CQ/G+nl8/kmfpiTQYGY2gfCUtlsJZ3JIiTjJuS9wbqNj2UwU7vh7\nsbv/JD4I6kzg7+iizI2Y2S7A9YQHYd3p7g8THnz1KvCZhga3CSgJNJCZXUC4Zcb3CFdVb+imSV5N\nIDyH4gUze4V/z6XcZWbfa1hUzatwI8bHCwXu3gnMBXZpSETNbX/C2WZ/LBTEW+c/RjhC6NM0J9Ag\nZnY+MBn4irtPanQ8Te4kwnxJwQ5A4bGk0xsSUXN7lHBb9v8k7tjiGUNvBX7bwLiaVeFss7cRtl3x\n9rqrUUFtKrptRAOYWeGP7UeEU9GKLXf37p6rkGtxnHYBcIguFivPzCYRhoBOIxwRnAGcDoxy97Sr\ntnkTb2s/C9iasJ2WAJ8GPgqMdPf5DQyv7jQc1BgnEA4/TyXcfK/4q8+PQcom8RXgcsKdfB8nXJdy\nuBLAxtx9PfB+wlX7PwVmE4aBxvT1BAA6EhARyTUdCYiI5JiSgIhIjikJiIjkmJKAiEiOKQmIiOSY\nkoCISI4pCYiI5JhuGyF9kpkdB5zl7gfX0PZOwkNYhgLXuvvIXg5PpGkoCYiUcPcjAcxsaKNjEak3\nJQHpM8zs64T7vbxIuG0yZjYQuBQ4iHCrjseAc9z9X2a2B+E5BUMJT3ab7O63mlk7cFxJ3xX76SKe\n1xHuErvrgsK5AAACPklEQVR77P9PwCfdfYOZnUq4NfZ6wr1qxrv7AjP7BOEBQ+sJDzg5y93/ZmY/\nBIYAuwG/Ab5cbTwi5WhOQPoEM/sA8CHCIxRHA2+Ii74ArAPe7u77AB2Eh69DuE/Mz9x9L+BI4BIz\nez3lddVPJR8EtnH3UYQ7egLsamb7EHbgR7j724DbgQvM7FDCMyUOie/xE+BX8Y6WAC3uvpe7f77G\neEQ2oiMB6SsOA37h7ssBzOwHhE/URwGtwHvNDGAgsNjMhgD7EJ68hbsvIHzKJtYrVbafbmKaRUgs\nMwi3vL7a3Z82s88Cd8f3xN2vju97GXCru/8zlv/QzK4BRhT115N4RDaiJCB9RSfQr+jndfG1P+GJ\nUXfBa0M0WxUtf+0OimaWAP+o0H+lfipy93lm9mbgYOBQ4LdmdnZ87+L3HUx4MHy5I/N+wJbx+1d6\nEo9IORoOkr5iGvBhM2s1sy2Ak2P53cBZZjYwll8PfCOOnf8JGA9gZjsBD/LvYaRSZfvpKiAz+xRh\nTuCeOIRzN+Gh7/cBh5nZsFj1k8BlcfnxZrZ9bH8KYX7j6d6IR6QcJQHpE9z9TuAHhCdpPQy8HBdN\nAtoJE6dPEj5ZF55VfCJgZjYH+B/gNHd/ocJbdNVPJT8mfGJ/0sz+SHgA/DXu/jhwHjAtvvcRwOnu\nPh34FuGB8E8QEtRRFR47Wks8IhvR8wRERHJMcwIiPWBmtwJJhcXH60le0ux0JCAikmOaExARyTEl\nARGRHFMSEBHJMSUBEZEc+1/HiadWFmAsKgAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x116bc6080>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"bin_plot(complete_scores, 'decile_score', 'is_recid', n_bins=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Decile scores are uniform among individuals who did recidivate and right skew among those that did not.\n",
"\n",
"#### Qualitative Scores"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEbCAYAAAA/P/RsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHj5JREFUeJzt3XuYHFWd//H3JDEhWTEjigtj0KwLHiHhKooGuUQREK/L\nul9YRQg/+YEuSB5BA0JQEEREVK4RBRTZoPBVvKJcskJAxXARRRA4LkpiZBDEMAhkTCCZ/ePUaNPp\nnqnu6emZOfN5PU+enqk+VXX6TOXTp09Vn+ro6+tDRETyNGGkKyAiIsNHIS8ikjGFvIhIxhTyIiIZ\nU8iLiGRMIS8ikjGFvIx5IYTtQghzW7StTUMI7xng+aUhhLNatK95IYTHWrGtVgohzAwh9IUQZtd5\nfnbx/Mw2V02aoJCXHHwPmNWibZ0J7N+ibQ3mSmCbNu2rESuBzYH7R7oiMnSTRroCIi3QMUq3NaAY\nYy/Q2679lRVjXAf8aaTrIa2hkJeaQgiHAwuALYAHgdNjjJcVz20CfB54R1H8B8BRMcYnQwiTgY8B\n80i9wTuAY2OMtxbrLgccOLBYd3tgCnAu8BbgKeCHxTpPlKjnUuDlwHkhhHfHGPcMIWxWb3shhIOA\nrwGvjTH+IoTQCdwDXFGUPaTYbl+MsV7gbxpCuAaYW7TNghjjD0IIrwVuBbaMMf6u2M4/AY8Cb48x\n3lBV93nAWTHGFxe/nwQcDmwK3AecEGO8pkQbzCzqcRJwDLAsxrhfCGEX0t/p1aTe+UXF/tYX6+0J\nfJr0N3gY+FSM8SsV29s2xnhPCGFT4EvA3sAjxTZljNBwjWwghLAj8EXgeOCVwDnApSGErYoi3wa2\nBd4KvIkUEhcUz50HHAYcCewI/AZYEkLYvGIXh5GGRPaPMT5ebA/g9cDbgX8lhW4Z+wN/BE7gH8Ms\ndbcXY1wMXAN8MYQwgRRYPcCJwFmkN6CrSW9Q9RwM/KR43d8CrgohvDzGeBvwW+CAirLvBFYBSwd6\nESGEdwEfBQ4FXkV6Y/pmCOEFA77659oP2AX4aAjhJcB1xXZmA0eT/iYLiv2F4vmfADsAHwcuDCG8\nqcZ2vwlsBrwBOKJ/GzI2qCcvtcwE+oCVMcYVpEB8APhzCGEWsAewfYzx1/D3Xv9eRa/4/cCBMcYf\nFc99kBQOR5GCFODKGOMdxfNzge2APWOMa4tl7wUeCiHMijH+ZqCKxhhXhRDWAU8WP5fZ3gdIbz5f\nIwXy62KMa4A1IYReYGKMcaDhiutjjKcXP58UQtgP+P/AQuBy0qeU/uffA3yjv/c8gH8BngFWxBiX\nhxA+CdxcLCvrnBjj/xav+RTgtop6PhBC+BjpDfsM0hvtb2KM/YH92xDCC6nq+IUQtib9vWf3/y1C\nCMcB32igXjKCFPJSy7WkYYdlIYT7Sb3BS2OMPSGENwNrgLv7Cxc92NuK4YqJwM8rnlsfQriF554Y\n/X3Fz7OAacCq1Ll8jleRwrgRg24vxvjHIvAuAM6IMd7Z4D6WVf3+C1JvGWAxcEoRjo+QhjhOZHCX\nk3rJvw0h/JL0aeIrxbh9WdXtOjeE8FTFsgnA1BDCi0gnfG+vXDnGeD78ffin32xgTdWb7W0N1ElG\nmIZrZANFsLwB2JU09LE3cEcR8GsHWPVvdZZPIIV/v8rgmgSsIA0ZVP7bCri+ieqX3d4OwDpgz2LY\nphHrqn6fQNEuMcbfA7cABvx7WhTvGmyDMcZHScH8ZtLQznuBX4UQtmugXtXtehXPbYPtSO3wRFHf\nMieZ+4COEEJl2YGOARllFPKygRDC64GTY4y3xBhPjDFuR+qt7k8ac55CRc88hDA3hLACWE4aXphT\n8VwHaWz8vjq7uw/oIg23PBBjfKDYxheAl5SscuV82YNuL4TwRtLY91tJ5xzm19lWPdv3/1C8vtcB\n91Y8v5g0Fv92Ug99UMWQz4dijP8TYzwGCKQwfkuZ9Wu4Dwj9bVC0wyzgZGA96e+4U1UdLgohfK5q\nO3cDk0lvEv12QsYMDddILU8DJ4QQ/gJ8nxSE2wAXxxjvCyFcB1wcQjiKFIqfA26MMf41hHAe8Pli\nmOBB0gm/maQrO2pZQhqSuSKE8BFSL/kCYGPSm0YZTwFbFycbB9xecbXLxcCiGON1IYTjgbNDCN8r\neuFPAbNDCDNjjPX2/84QwrGkYazDgZcBiyqevxI4mzQ8dFTJ19ABnBFCeIT0SWAX0snO2wdcq74L\ngKNDCOcWP7+cdIXMVcUQ2heB+SGEU4HLgNcC7yOdvP27GGMMIfwIuCSEcATpDf7MJuskI0A9edlA\ncUL1vaQAux+4BPhcjPGrRZH3kQL4BtL4/e38I8w+Rgq5S4E7KcaG+08I1tjXetKlmKuAG0lDFX8C\n9iuu1y7jHOAg4LoS2/sUKagWFuteDPyK9KbVUdR7M+De4lLMWs4F3gbcRbqMcr9iuKX/Na0iXbly\ne4zxD2VeQIzxh8BHgFNJvezTST37GwZcsf72/gjsA+xc1PNS0ieMDxfPLyd90ngrqbf+ceD9dfb3\nn6RPKjeQTrie20ydZGR06M5QIq0XQrgd+FKM8eKRrouMbxquEWmhEMK+pBPWW5I+0YiMKIW8jFoh\nhNeQhlwGslOM8bftqE9JR5PGtw+LMT451I2N0TaQUUQhL6PZr3nuVR21rGhHRcqKMe43eKmGjLk2\nkNFFY/IiIhkbbT15veOIiDSn5pfbRlvI093d3dR6XV1dTa87Hqm9GqP2apzarDFDaa+urq66z+k6\neRGRjCnkRUQyppAXEcmYQl5EJGMKeRGRjCnkRUQyppAXEcmYQl5EJGMKeRGRjI26b7w27Y7rmfZE\nz0jXYuzo7mxLe63e+g3Dvg8RqU89eRGRjCnkRUQyppAXEcmYQl5EJGMKeRGRjDV8dY2ZXQhMcvfD\nBiizM3AOsCPwEHCqu1/WdC1FRKQppXvyZtZhZp8Ejhik3KbAdcCdwE7AucAlZrb3UCoqIiKNK9WT\nN7NXAJcAs4E/DFL8MOAJYL67rwfuN7OdgI8A1w+hriIi0qCyPfk5wEpgW+DBQcruBtxcBHy/pcCu\nZlbzHoQiIjI8SvXk3X0xsBjAzAYrPgP4ZdWybmAa8CLgscaqKCIizRqOaQ2mAX+rWrameNxosJUH\nuiHtgLrvoXN6Z3PrjlPtaK/OZv+eo1DTx+Y4pjZrzHC013CEfC8wpWpZ/+9PD7Zy03crB3o0d01p\nndM729Jeq5v8e442XV1dTR+b45XarDFDaa+B3hyG4zr5lcDm1XUAniKdkBURkTYZjpD/KbB71UnW\nucDPqk7GiojIMBvycI2ZTQY2AVa5+1rSpZYLgAvN7GxgL+A9wL5D3ZeIiDSmFT35OcDDxSPu/ggp\n0HckXWVzFHCwu9/Qgn2JiEgDOvr6+ka6DpX6mj7x0H2PTrw2oG0nXjO5aYhOIjZObdaYFpx4rfk9\nJE1QJiKSMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIi\nIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8\niEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhmbVKaQmU0ETgPmARsD1wJHuvsjdcq/ETgDmAX8\nCfgS8Fl372tBnUVEpKSyPfmTgUOAg4HdgRnAVbUKmtmWwNXFv22B44BPAP81xLqKiEiDBg15M5sM\nzAdOcPcl7n4ncCCwq5nNqbHKvkCvu3/S3X/v7t8Cfgjs08qKi4jI4MoM1+xAGqJZ2r/A3Zeb2XJg\nN+CWqvJ/BjYxs/8ErgS2IfX+F7WgviIi0oAywzUziseHqpZ3A1vUKH8VcAlwObAWuBu4iTSmLyIi\nbVSmJz8NWO/uz1QtXwNsVKN8JzATOJPUk98WOJs0Lv+JwXbW1dVVoko1dN9D5/TO5tYdp9rRXp3N\n/j1HoaaPzXFMbdaY4WivMiHfC0wws0nu/mzF8inA0zXKfwZ41t2PL37/pZlNAi40s3Pd/S8D7ay7\nu7tMvTfQBfQ80dPUuuNR5/TOtrTX6ib/nqNNV1dX08fmeKU2a8xQ2mugN4cywzUri8fNq7fLhkM4\nAK8D7qhadivwPOBlJfYnIiItUibk7wKeBPboX2BmM0lDMjfXKP9HYLuqZbOB9cDvmqmkiIg0Z9Dh\nGndfY2aLgLPM7DHgUdKVMje5+7LiEstNgFXuvhY4B7jazBYCXyddXfN5YJG7/3W4XoiIiGyo7Jeh\nFpKullkM3AisAN5dPDcHeLh4xN1/BOwPvAv4Nemk65eAY1pWaxERKaWjr29UzTTQ1/SJh+57dOK1\nAW078br1G4Z9H+2gk4iNU5s1pgUnXjtqPacJykREMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJ\nmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVE\nMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkR\nkYxNKlPIzCYCpwHzgI2Ba4Ej3f2ROuVnAGcD+wC9wLeAj7j76hbUWURESirbkz8ZOAQ4GNgdmAFc\nVaugmU0BlgCbALsCBwBvA84cYl1FRKRBg4a8mU0G5gMnuPsSd78TOBDY1czm1FjlPcDmwL+7+6/d\n/UbgE8BrW1hvEREpoUxPfgfSEM3S/gXuvhxYDuxWo/w+wBJ3f7yi/FfdXSEvItJmZcbkZxSPD1Ut\n7wa2qFH+lcANZnYqcBDQB3wbWOjufxtsZ11dXSWqVEP3PXRO72xu3XGqHe3V2ezfcxRq+tgcx9Rm\njRmO9ioT8tOA9e7+TNXyNcBGNcq/AHg/cA3wH8BLgfOBl5DG9AfU3d1dokob6gJ6nuhpat3xqHN6\nZ1vaa3WTf8/Rpqurq+ljc7xSmzVmKO010JtDmeGaXmCCmVW/IUwBnq5R/hlgFfA+d7/D3b8HfBh4\nn5m9qFyVRUSkFcqE/MricfOq5V1sOIRDsew+d19Xseze4nFmQ7UTEZEhKRPydwFPAnv0LzCzmaTA\nvrlG+Z8AO5jZ8yqWzQbWkU7WiohImww6Ju/ua8xsEXCWmT0GPAosAm5y92XFJZabAKvcfS1wIfAh\n4DIzO4V04vazwGXu/pfheiFf/vZLWd27yXBtPjvTpk5tS3vZ1sO+CxEZQNkvQy0ELgcWAzcCK4B3\nF8/NAR4uHim+Bbs7KfjvBL5O+uLUB1tWaxERKaWjr69vpOtQqa/Zs8tXn/84q3t7W1ydfKWe/PC3\nl300j0vodKVI49RmjWnB1TUdtZ7TBGUiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuIZEwhLyKS\nMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGFPIiIhlTyIuI\nZEwhLyKSsUkjXQHJ2+LF00a6Ci3R2Qk9PXm8loMOWj3SVZA2Uk9eRCRj6smLjDPt+nSV06efdliw\nYHi2q568iEjGFPIiIhlTyIuIZEwhLyKSMYW8iEjGSl1dY2YTgdOAecDGwLXAke7+SIl1rwae7+57\nNl9NERFpRtme/MnAIcDBwO7ADOCqwVYysyOAtzZbORERGZpBQ97MJgPzgRPcfYm73wkcCOxqZnMG\nWG9L4HTg562qrIiINKZMT34H0hDN0v4F7r4cWA7sVmuFYnjnMuAzwL1DrKOIiDSpzJj8jOLxoarl\n3cAWddb5GNAHnAV8uZEKdXV1NVK8wuNMmzq1yXXHp3a017TOzmHfR7t0ZvRa2kVt1pjm86++MiE/\nDVjv7s9ULV8DbFRd2MxeDRwLvMbd15tZQxXq7u5uqHyl1b29Ta873kybOrUt7bW2p2fY99EOnZ2d\n9GTyWtpFbdaozqbzb6A3hzLDNb3ABDOrfkOYAjxducDMNgL+G1jo7g80WE8REWmxMj35lcXj5hU/\nA3Sx4RDOLsDWwGfM7DPFsimkN4mngG3c/Q9DqK+IiDSgTE/+LuBJYI/+BWY2E5gJ3FxV9jZgK9LJ\n2v5/3wHuKH5ufixGREQaNmhP3t3XmNki4Cwzewx4FFgE3OTuy4pLLDcBVrl7L/CcYRoz+yvQq+Eb\nEZH2K/tlqIXA5cBi4EZgBfDu4rk5wMPFo4iIjCIdfX19I12HSn3Nnl2++vzHdXVNA9p2dc3mWw77\nPtpBV4o0Tm3WmAULhnx1TUet5zRBmYhIxhTyIiIZU8iLiGRMIS8ikjGFvIhIxhTyIiIZU8iLiGRM\nIS8ikjGFvIhIxkrdyFukWZMfzmTKop6pTM7kG9W5fAtZylFPXkQkYwp5EZGMKeRFRDKmkBcRyZhC\nXkQkYwp5EZGMKeRFRDKmkBcRyZhCXkQkYwp5EZGMKeRFRDKmkBcRyZhCXkQkYwp5EZGMKeRFRDKm\nkBcRyZhCXkQkY6XuDGVmE4HTgHnAxsC1wJHu/kid8gcAHwO2Ah4GLgY+6+7rWlBnEREpqWxP/mTg\nEOBgYHdgBnBVrYJm9hbgclKwbwccDxwHnDDEuoqISIMG7cmb2WRgPnC0uy8plh0IPGhmc9z9lqpV\nPgBc5e7nF7//zsy2Bg4FTm1d1UVEZDBlevI7kIZolvYvcPflwHJgtxrlTwNOqVq2HnhhMxUUEZHm\nlRmTn1E8PlS1vBvYorqwu99e+buZvQD4IGkcX0RE2qhMyE8D1rv7M1XL1wAbDbSimU0DvgtMJY3N\nD6qrq6tMsRoeZ9rUqU2uOz6pvRqTS3tN6+xs274627ivHDSff/WVCfleYIKZTXL3ZyuWTwGerreS\nmb0Y+D6wDfBmd19RpkLd3d1litW0ure36XXHm2lTp6q9GpBTe63t6WnLfjo7O+lp077y0Nl0/g30\n5lBmTH5l8bh59XbZcAgHADObCdwC/Auwe/UQjoiItEeZkL8LeBLYo39BEeIzgZurC5vZS4Abi23P\ncfdft6KiIiLSuEGHa9x9jZktAs4ys8eAR4FFwE3uvqy4xHITYJW7rwUuAF4MvBHoNbPNik311fvy\nlIiIDI9S33gFFgLPAxYXj9cCRxbPzSH13Oea2a3A/qRe/G1V21jXwP5ERKQFSoVuccL12OJf9XNL\ngY6KRRNbUjMRERkyTVAmIpIxhbyISMYU8iIiGVPIi4hkTCEvIpIxhbyISMYU8iIiGdOXk0TGmckP\nP9CeHfVMZXImk7q1x87DslX15EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkR\nkYwp5EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJeRCRjCnkRkYwp5EVEMqaQFxHJmEJe\nRCRjpe7xamYTgdOAecDGwLXAke7+SJ3yOwPnADsCDwGnuvtlraiwiIiUV7YnfzJwCHAwsDswA7iq\nVkEz2xS4DrgT2Ak4F7jEzPYeamVFRKQxg4a8mU0G5gMnuPsSd78TOBDY1czm1FjlMOAJYL673+/u\n5wGLgY+0sN4iIlJCmZ78DqQhmqX9C9x9ObAc2K1G+d2Am919fcWypaQ3hY4m6ykiIk0oE/IziseH\nqpZ3A1vUKV+r7DTgRQ3VTkREhqTMiddpwHp3f6Zq+Rpgozrl/1ajLHXKP0dXV1eJKm3o8NObW09E\nZLRoNv8GUibke4EJZjbJ3Z+tWD4FeLpO+SlVy/p/r1W+koZzRERaqMxwzcricfOq5V1sOCzTX75W\n2adIJ2RFRKRNyoT8XcCTwB79C8xsJjATuLlG+Z8Cu1edZJ0L/KzqZKyIiAyzjr6+vkELmdkZpC9C\nzQMeBRYBf3P3PYtLLDcBVrn7WjP7ZyACVwJnA3sBnwP2dfcbhuNFiIhIbWW/DLUQuJx0vfuNwArg\n3cVzc4CHi0eKb8HuS/q26y+Bo4CDFfAiIu1XqicvIiJjkyYoExHJWKkJykYTM7sQmOTuhw1QRhOk\nVSjZZg78R9XiH7v7XsNauVGgOI90JrA3MBW4FTjW3e+pU37vonwA/hc4zt2vaVN1R4Um2uw24DVV\niy8Z6JjMiZnNAL4AvInUub4WOMbdu+uUb9kxNmZ68mbWYWafBI4YpJwmSCuUbbPCtsDxpMtf+/9V\nh352zGwC8B3glcA7SeeWngB+bGYbfEPbzLYBvg98k9SJ+B7wXTOb1bZKj7Am2qwDmAW8l+ceX8e0\nq84jqXj9PwReSLrScA/S6/9BnfItPcbGRE/ezF4BXALMBv4wSPHKCdLWA/eb2U6kCdKuH9aKjiKN\ntJmZTQG2BG5z9z+1oXqjyfbA64Ft3P0+ADN7H7AKeCtQ/QlwPrDM3T9V/H6Smb2hWH54e6o84hpt\ns1eQvgn/83F4fAH8M3AfcHwx7xdm9nlScL/Q3R+vKt/SY2ys9OTnkL5ktS3w4CBlNUFa0kibvYr0\nhn/fcFdqFPoD8DbSZb/9+o+dF9YovxsVk/UVllJ7sr5cNdpms0nfhF8xzPUaldz9T+5+YEXAzyB9\nur69RsBDi4+xMdGTd/fFpMs3MbPBis8gXbpZqXKCtMdaXb/RqME2mw2sBU4xs7eQ/kN+EzjN3avn\nIcqKu/+F9FG60tGkceZan/zqTcBXa7K+LDXRZrOBHuByM9sD+AvwVeDs8fYFSTP7LmmI63HS0E0t\nLT3GxkpPvhFDmiBtnJpFmjfoftLH7VNIw15fGslKjQQzewfwaeDz/UMRVeodX+P22CrRZrOA55PO\nle0DXEA6xj7RtkqOHicBu5BmBlhiZi+tUaalx9iY6Mk3aCgTpI1XC4Gz3H1V8fvdZrYOuMLMjil6\nbtkzs3nARcAVwII6xeodX+Py2CrZZgcDz3f3nuL3u81sOnCimZ3s7uPmyzrufjeAmR1IGk49BDi9\nqlhLj7EcQ14TpDWo+Mi8qmrx3cXjFqSP11kzsxNJ9zE+Hzh6gOCpd3zVmqwva2XbrJi9tqdq8d2k\nmxFNr/FcVorLTee6+xX9y9x9tZn9DqjVk2/pMZbjcI0mSGuQmbmZfadq8c6kj4gPjECV2srMFpDC\n6uPu/qFBepY/pWKyvsJcak/Wl61G2szMlpnZOVWLdwa6K3r3OXs58I3i+zsAFJ9kAnBvjfItPcbG\nfE++eoI00mWDC4ALzax/grT3kObTEWq22bcohmZI1+TuCJxFGsJ5auRqOvzMbDvSx+WvABeZ2WYV\nTz9JumpkOvBnd18HnAf8wsxOAb5BOrZ2AT7Y1oqPoCba7NvAJ83sF8DPgD2B40iXBI4HdwA/AS42\ns8OBZ4AzgD8DXzOzqQzjMZZDT14TpDWuus2cNMPoocA9pFlDzwE+PkL1a6cDgYnA/yO1SeW/DwMH\nFD9vAX8fU/030gR9vwLeAby9zgnHXDXUZsBngRNI535+Qwr4D7v7xe2t9sgoRhD2Jx0vVwM3AX8F\n9ig6UcN6jGmCMhGRjOXQkxcRkToU8iIiGVPIi4hkTCEvIpIxhbyISMYU8iIiGVPIi4hkTCEv0iJm\ndr2ZvXgI619kZq9uZZ1EFPIirfPmFqw/nm5sI22gb7zKmGVmzyfdfGIr0nwpvyDdcWcecCywjnST\nmEPcfWUxb8jRxfJHgKPc/bdmdilpLp9/JX3t/CTgM6RJoiaSpsc42t3/OkBdvlrs9x5gv6I+5wMv\nA54HXOHup5vZXNINWbYnfZX9x8CNpKlkP0q6i9fB7n7rkBtIBPXkZWz7N2Bjd98BeE2xbFtSQO/r\n7tuRboh8opm9kTRx3Vx33x74Oukem/0952nuPsvdjyPd0PxZ4NVF2W7ShFJ1ufuhxY9z3X0l8N/A\nV9z91cBrgb3MzNz9RtLNWC4izeWyhnQHrhOL/bxXAS+tpJCXseynwCwzW0oK5rOBNwHXFUGLu5/t\n7h8gTVp3pbv/uVh+KWku75kV2+r3NtIt2n5pZr8C3gVsU7ZSZvZPpE8BpxbrLyP16HcoinyCdCvK\n/wIO0hTYMpzG/FTDMn65+4NmtiVp6to3Av8DfBH4+xhkMY3ry6ndoekgDaVAuqlMv4nAfHe/ptjG\n82ns1msTi23PcffVxTZezD9u6TYd2Iw0pLMV4+S+wzIy1JOXMcvMPkgak7++GGa5jjTWvZeZ9d9Z\n5wjgzOK5A8xs02LdQ0l3vKp1U5TrgKPMbLKZTSANrXy6RJXWAc8rxu6XAccU++okzaP+zqLcV0jD\nOYeSbm49vVj+LP940xFpCYW8jGWXkXrN95rZHcALSDcg/yhwrZndRRqm+YC7LwG+ANxgZr8h3Vvz\nbXWGSk4FlpNOuN5L6pUfW6I+3wZ+amazSTd6eJ2Z3Q3cCnzD3S83syNJ84af4u7Xkd5Qvlys/13g\nSjPbu8F2EKlLV9eIiGRMY/IiJZnZlaT7ctZygLvHdtZHpAz15EVEMqYxeRGRjCnkRUQyppAXEcmY\nQl5EJGP/B7i6RAqH1FfzAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x116ae8668>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"bin_plot(complete_scores, 'score_text', 'is_recid', bins=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"Qualitative scores a mapped onto a 1, 2, 3 (low, medium, high) scale. We see a similar pattern to the decile scores since they're just a collapsed version.\n",
"\n",
"## 2. Calculate AUC ROC Data for each Outcome and Transformation\n",
"Now that the data is merged, we'll calculate the AUC ROC value for each transformation. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def calculate_roc(outcome, score, transformation_name, outcome_name):\n",
" results = complete_scores[complete_scores[outcome].notnull()]\n",
" auc = roc_auc_score(results[outcome], results[score])\n",
" results = {'transformation' : transformation_name,\n",
" 'outcome' : outcome_name,\n",
" 'auc' : auc}\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"roc_data = [\n",
" calculate_roc('is_recid', 'r_raw_score', 'Raw', 'General Recidivism'),\n",
" calculate_roc('is_violent_recid', 'v_raw_score', 'Raw', 'Violent Recidivism'),\n",
" calculate_roc('is_recid', 'decile_score', 'Decile', 'General Recidivism'),\n",
" calculate_roc('is_violent_recid', 'v_decile_score', 'Decile', 'Violent Recidivism'),\n",
" calculate_roc('is_recid', 'score_text', 'Qualitative', 'General Recidivism'),\n",
" calculate_roc('is_violent_recid', 'v_score_text', 'Qualitative', 'Violent Recidivism'),\n",
" ]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>outcome</th>\n",
" <th>transformation</th>\n",
" <th>auc</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>General Recidivism</td>\n",
" <td>Raw</td>\n",
" <td>0.725</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Violent Recidivism</td>\n",
" <td>Raw</td>\n",
" <td>0.724</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>General Recidivism</td>\n",
" <td>Decile</td>\n",
" <td>0.714</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Violent Recidivism</td>\n",
" <td>Decile</td>\n",
" <td>0.716</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>General Recidivism</td>\n",
" <td>Qualitative</td>\n",
" <td>0.678</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Violent Recidivism</td>\n",
" <td>Qualitative</td>\n",
" <td>0.665</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" outcome transformation auc\n",
"0 General Recidivism Raw 0.725\n",
"1 Violent Recidivism Raw 0.724\n",
"2 General Recidivism Decile 0.714\n",
"3 Violent Recidivism Decile 0.716\n",
"4 General Recidivism Qualitative 0.678\n",
"5 Violent Recidivism Qualitative 0.665"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"roc_df = pd.DataFrame(roc_data)\n",
"roc_df[['outcome', 'transformation', 'auc']].round(3)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAF9CAYAAADvF6QBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucznX+//HHjMOgop1IZlR0YkVhy0ZJOqD9tbItL1To\n4Fs6UbbT6qS2bUsHHSTZnIrUS4ftrFWora10olYUhYRRUkIO0/D74/2Z6XJ9LsPgMmY877fb3Gbm\nfb2vz+d1XfOZ6/V5v9+fz/udsXHjRkRERBJllnYAIiKy61FyEBGRGCUHERGJUXIQEZEYJQcREYlR\nchARkZiKO2tHZjYMqOjuvYupcxRwH9AMWAT8zd0f3UkhiohIJO0tBzPLMLNbgAu3UK8W8CrwEdAc\nuB8YYWbt0h2jiIhsKq0tBzM7CBgBNAa+3kL13sAKoJ+7bwBmm1lz4Erg3+mMU0RENpXulkMrYCHQ\nBJi3hbqtgTejxFBoKnCsmWWkJzwREUklrS0Hdx8LjAUwsy1Vrwt8nFS2GKgG7AMs29HxiYhIarvS\n1UrVgLVJZeui71V2ciwiIru1nXa10lZYA2QllRX+vnoLz9XsgSIi2yZlt/2ulBwWAnWSynKAVYSB\n6mItXrw4HTGJiJRbOTk5m31sV+pWegs4PmnwuS3wdtIgtYiIpFmptRzMrDKQDSx39/WES16vBoaZ\n2b3AycCZQIfSilFEZHdVmi2HVsCS6DvuvpSQCJoRrlq6FOjp7pNLLUIRkd1URjlZCW6jxhxEREom\nGnNIOSC9K405iIjILkLJQUREYpQcREQkRslBRERilBxERCRGyUFERGKUHEREJEbJQUR2qKuuuooV\nK7Y4Hdp2+fbbbzn33HM5//zzmTlzZlr3lWzMmDG89dZbAIwcOZJXX311p+5/Z9mVJt4TkXLggw8+\nSPs+Pv74Y7Kzs7n77rvTvq9U+65Xrx4A55133k7f/86i5CAiO8wdd9wBwBVXXMGCBQs4/vjj+eqr\nr+jduzcVK1Zk3Lhx5Ofn8+OPP9K+fXvOO+88pk+fziOPPEJOTg7z5s0jPz+ffv360axZMz799FOG\nDh1KQUEBGRkZnHnmmVSvXp2RI0eyevVqrrjiCgYPHswLL7zAM888Q2ZmJtnZ2fTt25f999+f22+/\nnZUrV7Jo0SJatmzJDz/8QFZWFrNnz2b58uWccMIJ7L333rzzzjssX76cK6+8kubNm7Nw4ULuu+8+\n1qxZw/fff8/BBx/MTTfdxEsvvcTnn3/OsGHDyMzM5O2336Z+/fp07dqVTz75hGHDhrFu3ToqVqzI\n+eefT4sWLZg4cSL/+c9/yMzM5JtvvqFSpUr89a9/pX79+qX81yqeupVEZIe55pprABg8eDC1atWi\nfv36jBkzhuOOOw5359prr+Xhhx/mwQcfZNy4cUXdT7NmzcLM+Oc//8kf/vAHRo8eDcCoUaPo0qUL\nw4cP5+qrr+bjjz+mWbNmnHvuuTRp0oTBgwfz0Ucf8cQTTzB48GBGjBjBSSedxA033EDh1EBr165l\n9OjRXHjhhQDMmTOHBx98kIcffpinnnqKqlWrMmTIEM444wzGjx8PwEsvvUT79u158MEHGTt2LHl5\nebzzzjv86U9/okGDBvTp04fWrVsXve4VK1YwcOBALr30UkaMGMG1117L3//+d5YsWQLAjBkz6Nu3\nL6NGjaJx48Y88cQTO+XvsT2UHEQkbY444ggAMjIyuO222/jiiy8YM2YMQ4cOBWDNmjUA7Lfffhxy\nyCEAHHrooaxcuRKAE044gfvuu49bb72VL774gt69e8f2MW3aNNq2bcvee+8NQIcOHVi2bBl5eXkA\nNGnSZJP6rVq1omLFimRnZ1OlShVatGgBQG5uLj/99BMAF1xwATVq1GD8+PEMHjyYZcuWFcWayqxZ\ns8jJyaFRo0YA1K9fn8aNGzN9+nQADjvsMGrVqhV7fbsyJQcRSZuqVasCIQn83//9H1988QWHHnoo\nffr0oUKFCkX1KleuXPRzRkZG0Vl/x44dGTFiBEcddRTvv/8+vXv3ZtWqVZvsI9XkoRs3buSXX37Z\nJIZClSpV2uT3ihXjvet/+9vfePHFF6lduzadO3fmsMMOK/Z1bimGrKxfF7lMfH27MiUHEdmhMjMz\niz4UCy1atIiff/6Z888/n1atWjFjxgzy8/MpKCgodluXXnopc+bMoUOHDvzlL39h1apVseRw9NFH\nM2XKFH788UcAXnnlFapXr05ubu42v4b333+fnj17cuKJJ5KRkcGsWbOKYq1QoULs9TVq1IiFCxcy\na9YsAObNm8cnn3xC06ZNtzmG0qYBaRHZoVq3bk3fvn036YY56KCDOOaYY+jVqxd77rknOTk5HHjg\ngSxatGiTVkOyCy+8kAceeICRI0eSkZFBz5492W+//Tapc9RRR9G5c2f69+/Phg0b2HvvvbntttvI\nzNz2c9/evXtz4403stdee1GlShWOPPLIoqWIW7ZsybBhw8jPzy+qX6NGDQYOHMj999/PunXryMjI\n4Oqrr2b//fff6Zfa7ihaz0FEZDel9RxERKRElBxERCRGyUFERGKUHEREJEbJQUREYpQcREQkRslB\nRERilBxERCRGyUFERGI0fYaIsOSq+GynO0OdOx8p8XO6devG0qVLNynLysqidu3anHbaaXTp0mVH\nhbdbU3IQkTKne/fudO7cuej3FStW8MILLzB06FD22WcfTjzxxFKMrnxQt5KIlDlVq1YlOzu76Kt+\n/fr07duXnJwcpk6dWtrhlQtKDiJSblSqVKloNtbp06fTr18/Tj31VNq1a0fv3r2ZNm0aADfeeCMD\nBw4set5HH31E27ZtmTRpUlHZvffey3XXXbdT49+VKDmISJm3du1annjiCRYsWMApp5zCt99+yzXX\nXMMRRxzBiBEjeOihh6hduzb/+Mc/yM/P55hjjuHjjz9mw4YNQEgOGRkZzJgxo2ib06ZNo1WrVqX1\nkkqdxhxEpMx57LHHePzxx4t+X79+PQcddBA33HADxx57LIsWLeK8887DzMjICDNSF6758MMPP3DM\nMcdw1113MWfOHBo0aMCHH35YtAgRwNdff83SpUtp2bJlqby+XYGSg4iUOaeffjqdOnWioKCAt99+\nm8cee4wOHToUDUTn5ubSrl07nnrqKb766isWLVrEnDlzANiwYQPZ2dk0aNCAjz76iNzcXObMmcPl\nl19Onz59WLZsGe+99x4NGzYkOzu7NF9mqVJyEJEyJ3EZ0AMOOIDMzEyGDBnC3nvvzUknncS8efPo\n27cvjRo1onnz5rRt25aCggIGDBhQtI1jjjmGDz74gP33358DDzyQBg0aUKdOHaZPn77bdymBxhxE\npBzo0qULTZo04d5772X58uW88MIL7Lvvvtxxxx107dqVFi1a8N133wFQuPply5Yt+fTTT3nvvfdo\n3rw5AM2bN+edd95hxowZSg6lHYCIyPbKzMzkyiuvZN26ddx///3UqlWLpUuX8sEHH5CXl8ekSZMY\nMWIEQNHaz4cddhg1atTg1VdfpWnTpkBIDlOnTqVWrVrUr1+/1F7PrkDJQUTKhQMOOICzzjqLN954\ngwMPPJDjjjuOW265hd69e/Pss8/Sv39/qlSpwuzZs4ue8/vf/56CggKOPPJIAJo1a8bGjRt3+1YD\nQEZhE6uM27h48eLSjkFEpEzJyckByEj1mFoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIi\nMUoOIiISo+QgIiIxSg4iIhKjWVlFpEzp1q0bS5cuLfq9UqVK1KxZk9atW9OrVy+qVau23fuYOHEi\nd955J6+//joAbdu2ZcCAAZxyyinbve2yQslBRDhnzDulst/RvbZtMZ3u3bvTuXNnANasWcPnn3/O\nQw89xMyZMxk8eDCVKlXarrjatm1LixYttmsbZZ2Sg4iUOVWrVt1kIZ7c3Fzq1q1Lnz59eOWVV+jY\nseN2bT8rK4usrKztDbNM05iDiJQLhx12GE2aNGHy5MkAzJs3j6uuuooOHTrQpUsX7r77blatWlVU\n/5dffmHEiBGYGaeeeiqXXnopn332GRC6lU466aTN7uvFF1+kZ8+etG/fnvPOO4+JEyem98WVAiUH\nESk36tevz7x58/juu++4/PLLOfjggxk+fDg333wzCxYs4MYbbyyq+8ADD/Dyyy/Tt29fHnnkEQ45\n5BCuvvpqfvzxx2L38dxzzzFixAjOP/98Ro4cSffu3RkyZEi5SxDqVhKRcmOvvfZi9erVPP/889Sp\nU4c+ffoUPXbDDTdgZsycOZP69evzyiuv0L9/f4477jgALrvsMrKysvjpp5+K3cfYsWPp2bMnbdq0\nAUKX1tKlS3n88cfp0KFD+l7cTqbkICLlxurVq9lzzz2ZM2cOc+bM4dRTT43VWbBgARUrViQ/P5+G\nDRsWlVeoUIGLLroIoKh7KdmPP/7IsmXLGDZsGMOHDy8qLygooKCggPz8/O0eDN9VpDU5mFkF4Fbg\nHGAvYCJwibsv3Uz9DlH9hsA84DZ3H5/OGEWk/JgzZw6HHnoolSpV4uijj+ayyy6L1alRowZ5eXnb\ntP2KFcNHZt++fYuWFk1UoUKFbdrurijdYw4DgV5AT+B4oC7wdKqKZnYs8DLwFnAU8A/gYTPrkeYY\nRaQcmDt3LjNnzuSkk06iXr16LFiwgNq1a5Obm0tubi6ZmZkMGTKE7777jtzcXCpUqMDnn39e9PwN\nGzbQo0ePogHtVPbcc09q1qxJXl5e0XZzc3P58MMPefLJJ8nMLD/DuGlrOZhZZaAf0NfdJ0Vl3YB5\nZtbK3f+b9JSrgP+6++XR77PNrD5wC/BYuuIUkbJnzZo1LF++HIC1a9cye/Zshg8fzhFHHMEpp5zC\nihUrePbZZ7n99ts588wzyc/P595772X16tXUrVuXSpUq0alTJ0aMGEGNGjWoW7cuEyZMYOXKlTRt\n2pRp06Ztdt89evRg6NCh7LvvvjRv3pxZs2YxdOhQunXrtrNe/k6Rzm6lpoSupKmFBe4+38zmA62B\n5ORwKPB8UtnHQD0zO8Ddv05fqCJSlowfP57x40OPc7Vq1ahduzannXYanTt3pkKFCmRnZ3P33Xfz\n8MMPc/HFF5OVlUWzZs246KKLisYELrjgAjIzMxk0aBBr1qyhQYMGDBo0aJP7J1Lp2LEj69ev58kn\nn+SBBx6gZs2a9OjRgzPPPDPtr3tnSmdyqBt9X5RUvhjYP0X9VOX1ou/7AkoOImmyrXcql4Ynnnhi\nq+o1aNCAe+65Z7OPV65cmYsvvpiLL7449liHDh02ufJoypQpmzzeuXPnoju0y6t0JodqwAZ3z08q\nXwdUSVH/MWCEmT0PPAM0Aa6MHquctihFRCQmnclhDZBpZhXd/ZeE8ixgdXJld3/UzA4ERgPjCS2F\nu4AhwIot7SwnJ2dHxCwiIqQ3OSyMvtdJ+Bkgh3hXEwDu/jcz+wehG2kJ8EegAFiwpZ0tXrx4u4IV\nEdndFHdSnc7kMANYCbQBxgKYWT3COMKbyZXN7FLgkOhqpcVRWSfCFUyrkuuLiEj6pC05uPs6MxsK\n3GVmy4BvgaHAG+7+bnSpazaw3N3XA7OBwWb2AeFeh+7A2cDuM4G6iMguIt13bFwPjCO0HKYQuocK\nh/hbEbqOWgG4+2vARYQb5z4D/gT80d3fSHOMIiKSJGPjxo2lHcOOsFFjDiIiJRONOWSkeqz83Ost\nIiI7jJKDiIjEKDmIiEiM1nMQEV59fkmp7Ld9xzrb9Ly1a9cyYcIEJk+eTF5eHlWrVqVx48Z07dqV\nww8/fIfFd/nll5Obm8tVV13FxIkTufPOO3n99dcBmD9/PkuWLKFly62bemTt2rVMnDiRTp06ATB6\n9GgmTZrEuHHjdli8O5JaDiJSpqxatYpLLrmESZMm0atXL0aNGsVtt91G9erV6devH6+88kpa9tu2\nbVsmTJhQ9Pv111+/yZTfW/LUU09tMi9U165dGTp06A6NcUdSy0FEypQHH3yQ1atXM3z4cKpXrw7A\nfvvtR8OGDcnOzubee++lSZMm1K1bdwtbKpmsrCyysrKKfi/plZ7J9atWrUrVqlV3SGzpoJaDiJQZ\nq1at4vXXX6dLly5FiSHR2WefTaVKlXjxxReZOHEiJ5100iaPJ5fNnTuXa6+9ltNOO41TTjmFnj17\n8uqrr6bcd+JzL7/8chYvXsyYMWOK1nHIy8tj4MCBnH766Zx88sl069atqKUwceJERo4cydKlS2nb\nti3Tp09n9OjRnHXWWUXbu/322zfZ39SpU+nQoQOrV4ep6F588UV69uxJ+/btOe+885g4ceK2vIVb\nTclBRMqM2bNnk5+fT+PGjVM+XrlyZRo1asTMmTO3uK01a9Zw9dVXs88++zB06FBGjBjBEUccwd13\n3120kNDm3HLLLey3336YGcOGDQNgwIABrF+/nsGDBzNmzBjatWvHww8/zNy5c2nbti3du3enVq1a\nPP3007Fxkfbt2/PWW2+xfv36orLXXnuNY489lj322IPnnnuOESNGcP755zNy5Ei6d+/OkCFD0pog\nlBxEpMxYsSJM0Lznnntutk716tWL6hVn7dq1dOnShcsuu4wDDjiAAw44gLPOOov8/Hy++eabYp9b\nvXp1MjMzqVq1KnvvvTfr1q2jQ4cOXHHFFRx00EHk5ubSq1cvMjMz+eqrr8jKyqJq1apkZmaSnZ1d\ntOBQoTZt2lBQUMC7774LwMqVK5k2bRrt27cHYOzYsfTs2ZM2bdqQm5vLKaecQrdu3Xj88ce3+Dq3\nlcYcRKTMqFGjBhA+PDdn1apV7LHHHlvc1m9+8xs6duzIv//9b+bMmcOiRYuYO3cuENaTLomsrCw6\nderElClTmD17Nt988w1ffvklGzZs2KptVatWjdatWzN58mSOP/54pk6dSvXq1fnd737Hjz/+yLJl\nyxg2bBjDhw8vek5BQQEFBQXk5+fHks2OoOQgImVGw4YNqVSpEp9++ikNGzaMPb5+/XpmzZoVG2so\nVFBQUPTzsmXLuPTSS6lZsyYtW7akZcuW7LPPPvTp06fEca1Zs4a+fftSUFBAmzZtaNq0KY0aNSrR\nutLt27dnwIAB/Pzzz7z++uucfPLJVKhQgYoVw8d03759adq0aex5FSpUKHG8W0PJQUTKjD333JN2\n7doxYcIE2rVrR40aNVi1ahUXXHABZ5xxBj///DOrVq3i9NNPZ86cOWzYsIG1a9dSpUpYfDKxu2jy\n5Mn8/PPP3HfffUUfsNOmTQO27kqkjIxfpySaNm0ac+fO5bnnnisaKP/666/ZsGFD0bYS66fSrFkz\natSowcsvv8ynn35Kv379il5zzZo1ycvLIzc3t6j+888/z9y5c+nfv/8WY90WGnMQkTLloosuIjs7\nm8suu4ypU6eyatUqOnXqxLBhwxg1ahTdunXjwAMPpFGjRmRkZDB69Gjy8vKYPHnyJlci1apVi9Wr\nV/Pmm2+Sl5fH22+/XbTmdH5+8urGcdWqVWPhwoUsW7aMfffdFwiDyHl5eXz44YfcfPPNm2yratWq\nrFq1iq+//nqTgedCmZmZnHLKKYwaNYqDDz6Y+vXrFz3Wo0cPJkyYwAsvvMCiRYt47bXXGDp0KNnZ\n2dv+Rm6BWg4iUqbsscce3H///UyYMIExY8awZMkSqlSpwtFHH02dOnV45plnWLduHRdffDFXXHEF\n48aN45lnnqFJkyZceOGFDBo0CIATTjiB2bNn88ADD7BmzRpycnLo2bMnjz/+OLNnz6ZFixbFxtGl\nSxfuv/9+PvjgA5599ln69OnD+PHjGT58OLVr1+bUU09l2rRpzJ49m44dO3L88cfz8ssv07t3b667\n7rqU22zfvj2PP/447dq126S8Y8eOrF+/nieffJIHHniAmjVr0qNHD84888wd86amoCm7RaRcWbhw\nIdOmTePPf/5zaYeyyytuym4lBxGR3ZTWcxARkRJRchARkRglBxERiVFyEBGRGCUHERGJUXIQEZEY\nJQcREYlRchARkRglBxERiVFyEBGRGCUHERGJUXIQEZEYJQcREYlRchARkRglBxERiVFyEBGRGCUH\nERGJUXIQEZEYJQcREYlRchARkRglBxERiVFyEBGRGCUHERGJUXIQEZEYJQcREYlRchARkRglBxER\niVFyEBGRGCUHERGJUXIQEZEYJQcREYlRchARkRglBxERiVFyEBGRGCUHERGJUXIQEZEYJQcREYlR\nchARkZiK6dy4mVUAbgXOAfYCJgKXuPvSzdQ/EbgdOBzIAx4G7nT3jemMU0RENpXulsNAoBfQEzge\nqAs8naqimR0CvBh9NQGuAW4CLk5zjCIikiRtycHMKgP9gAHuPsndPwK6AceaWasUT+kArHH3W9z9\nK3d/CngJaJ+uGEVEJLV0dis1JXQlTS0scPf5ZjYfaA38N6n+d0C2mXUHngQaEVobQ9MYo4iIpJDO\nbqW60fdFSeWLgf1T1H8aGAGMA9YDnwJvEMYsRERkJ0pny6EasMHd85PK1wFVUtTfG6gHDCK0HJoA\n9xLGHW7a0s5ycnK2J1YREUmQzuSwBsg0s4ru/ktCeRawOkX9O4Bf3P3a6PePzawiMMzM7nf374vb\n2eLFi3dI0CIiu4viTqrT2a20MPpeJ6k8h3hXE8AxwAdJZe8BlYADdmxoIiJSnHQmhxnASqBNYYGZ\n1SN0Hb2Zov43wBFJZY2BDcCXaYlQRERSyti4MX33l5nZ7YQb4M4BviVcebTW3U+ILnXNBpa7+3oz\n+wPhHocbgccJVys9DDzj7pdtYVcb1a0kIlIyUbdSRqrH0n0T3PWEq4/GAlOABUDn6LFWwJLoO+7+\nMnAG0An4hDAY/TDQP80xiohIkrS2HHYitRxEREpom1sOZnazmZ2U8PujZrbFy0pFRKRs22xyMLOr\ngXbA/ITiB4AOZqauHhGRcqy4lsNZwB/cvehKIXd/nzAmcE6a4xIRkVJUXHIocPcfkguj6bYL0heS\niIiUti2NOVROUZZFuDFNRETKqeKSw4vAIDMrGsmOfr4deD3dgYmISOkpbm6lvwP/Ar4ys/cIieRo\nYDbw550Qm4iIlJIt3udgZm2A3xGmsXjX3d/dGYGVkO5zEBEpoeLuc9iaWVlnE6bT3siml7WKiEg5\ntaUB6duArwjTYPwdmBPNlyQiIuVYcTfBnQu0BOq7+9Hu3gRoCLQ0s947K0AREdn5ims5XAic6e7f\nFha4+yLCzXEXpjswEREpPcUlhyx3X5Jc6O7fkHqZTxERKSeKSw57FvOYboITESnHiksOH5jZ2cmF\nZtYDmJa+kEREpLQVdynrAOANMzucsKxnJaAt0IUwUC0iIuXUZlsO7j4POB74DXAHcCthrOEYd1+4\nc8ITEZHSsE0rwZlZLXf/Lg3xbCvdIS0iUkLbdIe0mf0GuBL4HrjP3QvMLBO4DLgJyN7xoYqIyK6g\nuDGH0YT5lGoBFczseeAJoA6gleBERMqx4pLD4cChhHmVXgMuB14ArnX3H3dCbCIiUkqKSw4r3X0j\n8IOZ1QWucffROycsEREpTcVOvJfgWyUGEZHdR3HJIfEypvx0ByIiIruOzV7KamY/A3OjXw9J+BkA\ndz8ivaGViC5lFREpoW1d7OfUtEQjIiK7vG26CW4XpJaDiEgJFddy2NoBaRER2Y0oOYiISIySg4iI\nxBSbHMzsXDNrkfD7IDPrlf6wRESkNG02OZjZeYQ1HdYnFL8FXB8t+CMiIuVUcS2HS4CT3H16YYG7\nPw+0J8yzJCIi5VRxySHD3b9OLnT3r4AK6QtJRERKW3HJoUK0fsMmorJK6QtJRERKW3HJYSpwRYry\n/sD7aYlGRER2CcVNn3Ej8IaZdQLeJiSSlkB14OSdEJuIiJSSzbYc3H0F8HvgEaAaUBl4EDhqF1s/\nWkREdjDNrSQispvapllZzWwKm67pUAB8D7zs7o/uyABFRGTXUtyYw5Ck3zOB2sBlZlbT3e9JX1gi\nIlKaStytZGa/Aaa6+5HpCWmbqFtJRKSEtnWxn5Tc/QczKxcDFYmWXNW7tEMokTp3PlLaIYhIOVbi\n5GBmGegmOJGdpqyduIBOXsqD4gaks1MU7wP0Bd5JW0QiIlLqims5LCNcrVTYH7UxKnsZ6JfmuERE\npBRtNjm4e6p5lSoCBkwi3CAnIiLl0FaNOURXKF1ImMZ7T+CBdAYlIiKlq9jkYGYNCJPvnQ3MB6oC\n9aKpNUREUjpnTNkalhzdq2Vph7DLKW5A+mXgd8CTwAnu/oGZzVNi2DWUtX8+0D+gSFlS3JTdTYGP\ngP8Bc6Kycnd/g4iIxBWXHA4ARgLdgSVmNoHQrSQiIuVccVcr/QJMACaYWSOgD1DVzOYAd7v7sJ0U\no4iI7GRbdbWSu38G9DWzawmD032ALSYHM6sA3AqcA+wFTAQucfelKepOBdpsZlNt3P3NrYlVRES2\nX4mmz3D3n4Hh0dfWGAj0AnoSpvseCjwNHJei7hmEBYUKZQIvAj8B/y1JnCIisn1KPLfS1jKzyoQ7\nqfu6+6SorBswz8xaufsmH/juvjzp+dcABwENoy4uERHZSYobkN5eTQldSVMLC9x9PuF+idbFPdHM\n9gOuBwa4e17aIhQRkZTSmRzqRt8XJZUvBvbfwnOvAb4FHt7RQYmIyJalrVsJqAZscPf8pPJ1QJXN\nPcnM9gLOA65294Kt3Vm0aMU2W7Jdz5atsb1/o92Vjs3007EZl87ksAbINLOKSWMGWcDqYp53ehTX\n2JLsTCvB7fr0N5Jd1e56bBaXFNPZrbQw+l4nqTyHeFdTotOBF929uAQiIiJplM7kMANYScK9C2ZW\nD6gHFHfPQmtgchrjEhGRLUhbt5K7rzOzocBdZraMMMA8FHjD3d+NLnXNBpa7+3oAM6sD1AY+TVdc\nIiKyZelsOUC4HHUcYfxgCrAA6Bw91oow1tYqoX5hF9Qm9zyIiMjOlc4B6cL5mf4SfSU/NpVflyAt\nLPsouUxERHa+dLccRESkDFJyEBGRGCUHERGJUXIQEZEYJQcREYlRchARkRglBxERiVFyEBGRmLTe\nBCeS6NWw5+dyAAAT80lEQVTny97k0+07Js8bKbJ7UMtBRERilBxERCRGyUFERGKUHEREJEbJQURE\nYpQcREQkRslBRERidJ+DiOz2dA9OnFoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoO\nIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIi\nEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj\n5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMRXTuXEzqwDc\nCpwD7AVMBC5x96WbqV8XuBdoD6wBngKudPef0xmniIhsKt0th4FAL6AncDxQF3g6VUUzywImAdnA\nsUBX4DRgUJpjFBGRJGlLDmZWGegHDHD3Se7+EdANONbMWqV4yplAHeDP7v6Ju08BbgJapCtGERFJ\nLZ0th6aErqSphQXuPh+YD7ROUb89MMndf0ioP8rdlRxERHaydI451I2+L0oqXwzsn6L+YcBkM/sb\ncDawEXgGuN7d16YtShERiUlncqgGbHD3/KTydUCVFPWrA+cDrwBdgFxgCLAvYcxCRER2knQmhzVA\npplVdPdfEsqzgNUp6ucDy4Ee7l4AfGBmlYAJZnaFu39f3M5ycnK2K9gl2/VsKa+297jaEXRsSirp\nPjbTmRwWRt/rJPwMkEO8q4mobG2UGAp9Fn2vBxSbHBYvXrxtUYoUQ8eV7Kp2xLFZXIJJ54D0DGAl\n0KawwMzqET7o30xR/z9A06i1UKgxUEAYxBYRkZ0kbS0Hd19nZkOBu8xsGfAtMBR4w93fjS51zQaW\nu/t6YBhwGfComd1MGNC+E3h0S11KIiKyY6X7JrjrgXHAWGAKsADoHD3WitCd2gogumv6eELC+Ah4\nnHDD3EVpjlFERJKkdfqMaCD6L9FX8mNTgYykss8I9zuIiEgp0sR7IiISo+QgIiIxSg4iIhKj5CAi\nIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIx\nSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoO\nIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIi\nEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj\n5CAiIjFKDiIiEqPkICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiElMxnRs3swrArcA5wF7A\nROASd1+6mfoOdEkqft3dT05nnCIisql0txwGAr2AnsDxQF3g6WLqNwGuBeokfCUnCxERSbO0tRzM\nrDLQD+jr7pOism7APDNr5e7/TaqfBRwCTHP3vHTFJSIiW5bOlkNTQlfS1MICd58PzAdap6jfkJCs\nZqUxJhER2QrpHHOoG31flFS+GNg/Rf3GwHrgZjM7FVgDTABudfe1aYtSRERi0pkcqgEb3D0/qXwd\nUCVF/cOBDGA2MIQw/nAPIZH0SmOcIiKSJJ3JYQ2QaWYV3f2XhPIsYHWK+tcDd7n78uj3T82sAHjC\nzPq7+/fF7SwnJ2e7gs0Z9/J2PX9n+3dpByA7TVk7NkHHZ3mQzuSwMPpeJ+FngBziXU24+wZgeVLx\np9H3/YHikkPGNsYoIiIppHNAegawEmhTWGBm9YB6wJvJlc3MzezZpOKjCN1Qc9MWpYiIxGRs3Lgx\nbRs3s9sJN8CdA3wLDAXWuvsJ0aWu2cByd19vZgY8AVwJPAc0i+oPd/fr0xakiIjEpPsmuOuBccBY\nYAqwAOgcPdYKWBJ9x92dkETOBf4H3A3cB9yY5hhFRCRJWlsOIiJSNmniPRERiUnrxHtlmZllAhcQ\n7rFoSLg3Yy4wHrinLNyYZ2YbgR7uPjbFY+cAo5KKNwA/AG8D/d39yx0Ux1Rgrrv33szj1wO93b3e\nluJOel49YB7Q2t3f2hGxliVmNgWo6u7HbObxycBqd//j1r6n0fPOAR5x9x32+WBm/w+Y5+6fbebx\n+cCBScVrCF3Rw9198A6K4wRCF/f+7v7NZur8QjgeR5vZQOBsdz9kK7Y9GqhbXiYKVcshBTOrCLwM\n3EIYM2kJ/Bb4B9AbeNHMysPlswVsOslhPcK4T1Pg+R34Gs8A+pegfh3gqa2otzCq+962BFUOjAJ+\nb2YHJT9gZvsDJwAjo6KtfU93ODPLBV4E9t1C1TvY9Hg8GngNuMfMuu6gcP4bbXvxVta/C0iZfFPo\nRzmaKFQth9SuBNoCv3P3/yWUzzez9wh3cf8BeKk0gtuRUkxyuNDMfgM8SrhL/ZMdsI/k+1dKGtPm\n6hUAu/MkjU8RZhPoBtyW9NhZwHeED+Wtfk/TZGtPMlYlxZkHXBZNp9MVeHJ7A3H39ZTgmHH3VcCq\nray7Ylvj2hUpOSSJzpYvAR5NSgwAuPuXZvZb4MuE57QmnPU0I1yB9SRwc2HXU9SkP59wVt6CcFnv\nre4+PGEb/0dISgcQuq/ucvcx0WMnENbCuJVwBj7D3dua2Z8JU5w3BjYCHwOXu/v72/k2rIu+F93Z\nXlx80eOHEqY7aRM9/7kollXJ3UrRZcs3E1oqU4HPE3de2AUCfEPoAjjI3eclPD4TeBZ4hIRuJTM7\nhnCVW1NgLaH118/dlye8h2cRWoD7E7rPziG8hz2i5wx299u36V3bydz952gNlO7Ek0MPYGzh9DXJ\n3Upmdh7hWDqYcFPqve4+JNV+opOFu4HTCR/07wJXuPvn0eOjCa3Q1YT3Nwt4HrjQ3Vfy602wU8xs\njLufU8KXup5Nj8X9gcFAO0LX0xRCN+ji6PEM4HLgYiAX+AIY4O4vJ3crmVk2IcH+P+BnwrGQ+NoH\nEnUrRcfx/MT4zawLMAbYD7ifqFspWsvmDsLfpibhGP+bu0+Injc1eh8PiN7XFcBN/Dp90KHAR0Cv\nHdW9W1LqVoqrT5g0cPLmKrj7XHffCGBmTYFXgWcIZ9q9gT8CDyU97Q7CH71RVPchMzsw2sZFwN+B\n6wgf9HcA95lZ4pxSWYTWTAugr5kdDTgwmtDl1Ybwj/vPbXzdRLEcDtwAfEg4ULcYn5ntDbxBSFCt\ngdMIlyg/nGL7xxPuZxkDHEmYaeHSzYTzBqHPuVvC85sS3sNHk7ZbgfCB9Dphnq4/ELol7kqoVpnw\nz38mcCLQnNAyWhXVfRj4h5k1Ku492sWMAhqbWePCAjNrTniPRqZ6gpn1JxyL9wJHAHcCd5rZX1LU\nzSQk2RygPXAc4W/ylpntk1D1bKACoQvWgI6EbhYI7zPAnxPKtsjMqpnZVYTjuzCp7UE4oVhDOMba\nE/6uk6N7pwCuJpx8/I3wPzkBeDY6tpNNiOp0iGK+NHodqYwB/mRmiXPDnQX8y91/Sqp7MfAnwmtu\nEO1nvJnVT6jTn/B/1oRwMvVg9NWXsP5NLvGkv9Oo5RBXO/q+LLHQzGYQzrIKjXX3PoSz6ZfcvfBD\naK6ZXUj45xng7kui8pHRvRyY2U2Ef5IWhH+06wgtjcI+4S+jxDGAcEAWGuTuc6NtNAUudvfCD+D5\nZvZPwtn01qpgZolN5sqED8oXgKuiKU3Yivi6AnsSzrB+iuLrDaQamLsEmJJwdv6FmbWM3otNuPtG\nM3uUcPb1j6j4LOBdd/8iGpAuVINwhpYHLHD3+Wb2p+g1FcognEF+EMX4OuEu/L9G+/oHITEeDqQc\nON3VuPvbZjaH8B5dFxX3IKyLMjO5fnRWfTWhpVB4rMyJxi2uNrN7kp5yIiFxZid8AF5kZicRLtgo\n/Lt8T1i7pYDwN51ESBQQurcg3PBaXNfLDWZWeOaeQbgI5BOgu7s/H5V3B/YAzon2hZl1J/y//tnM\nniD8b93j7oUnEH83s0qEYzTxvfht9PqOd/d3orJeQOx9ixR24/0BeCZqUZ1KSCrJDiG0ROa7e56Z\n3QpMY9Mpgt5397uj/Q4B+hBarm9EZU440SoVSg5xhXM4ZSeV/5FfP2jG8OvMss2AQ5M+ZAv7WH9L\n6GaC0LQFQt9k6FmhspnVIpwh3GVmdyRsoyJQMeFsCOCrhG1MN7MfzeyvhLPEQwndKSVpDRZEz4Fw\nZnhn9PMAd/8WYCvjawLMTjx7ihZz2mRBp0hj4mM175IiOUQeJXxoHE5Y66M7oRWziajr6G7CmdfN\n0YfTC8QHYROnYllNuIJmY7SNNdHfJWszseyqRhFarNdFLajuhFUYU6lFOAFK/tu8SUgayYPGzQhn\n0ouj96ZQFcLxXejLwg/ryArCcVMSDxJmRahA6Gq5ARjl7k8kxVMLWJEUT7Uonn0IA87TEh9094FQ\n1EVbqLC19WFCvc/MbGWq4Nx9pZk9Q3h/nyEMPn9PGDRPNpRwIcYiM/uA0KU5Lik5Jh+LkNBdTWgd\nldqxqG6luC+BpSQtSOTuX0fdSXMJf7RC6wnJomnC15GED+vEq2jWEZcRPR/gsqRtNCZcQps4o23R\nfs3sREK3T1PgfeCvlKDJnvC65kZfbxLOguoAL1lYma/w9W0pvuRp2YuzkfgA5fpUFQvjI4wNdCc0\ntWsSuqVS1b2K0C04kHB2OYrQ5ZcoOdYNlH2PAvXM7PeEfvjqhEuuU1mzmfLCrpTk92c94Wy3adJX\nQ0IyKbS547sklkfH4ufuPojQNXRf1DJIjGdmingOI/T5l/RYTBXnZo9Hwv/6aWa2F6F7cmxSUgQg\nGo85iDCWUXj8zoxaXIVSxbrLHI9KDkmiP/QQ4Nyo2bmJ6Ey5VkLRTOC3CR+yc6PH7yKshLel/a0g\nDAjWS9rGycCVCV07ya4A/u3uXd39fnefQhjgLew6KLHoqqLehOQ2sATxzQIamFlRs93M2pvZ12ZW\nLWk304mmTElw1BZCG0PouzXgRXf/IbmCmR1sZg8Bee7+oLufTli7vK2ZbekSyjLN3RcRxm66EMZn\nntlc9000QPwNcGzSQ8cRuuSS39uZRK3ohL/9PMLFEcdvZYjbOg3DPcBbwFAz2y8hnvrA9wnxfBvV\nbRK97iUkHVNmNiUav0g0I/reKqFePULrY3MmE7qwziOcQD6aqpKZXQz82d0nunt/QjL9nHAclwnq\nVkrtdkI/69tRX+GrhDOuYwgDmg2BB6K6dwAfRX21wwlN9keARSW4fPBWwrXcXxMGVH9PONgHFfOc\nhYQzmGMILZ3TCFdoQGiKbtNNeu7+76if/0ozG+/un2xFfOMIc2AV3jS0F+FqkqnRFTWJu7gXeMfM\n/k740G9L+EAr7r1ywlnh2YQxh1SWEcY+ssxsEOFssCuhJbhsM88pT0YR+v9/w6/zl23OrcBgM/uS\nMLjbltAyvDEae0ms+zqh28/NrB/hWLuW0M9+y1bGVthNc4SZfepbWJulkLtvsHCV3HTC398Ix9p1\nUTx/JRzntxO6JQvHCgYBA83sC0Krujvhf/dSEk7s3H2OmT1HSD69CV1h91PM2XsU02OEwe7pnuKK\nxkhNQvfmKsLSA80JSe3OzdTf5ajlkIKHxYk6EbppTiNcNfMZ4Wz6DeDwwsE8d/+U0HQ8lnAQe1Tn\nTyXY3zBCt9BV0X5uJRx8NxfztBsJl7q9SugzPYNfV8w7emv3vRn9CWeQj5hZhS3F5+6rCVeNVCf0\n9f6LcLngRckbjgaD/0h4zz4hDJ7eXVww0dngvwgfBK8UU+dUwkUD70VxZAF/KKb1VZ48B+xN+ICb\nUlzF6CKGAYS/6UzC37u/u8c+uKLxmE5RvecIl0sfBrT3zdztnGIbPxFOpu6gZBdM4O6zCVfsdDGz\nP7r7GuAUwmDvZEKXTUXgxMJxMsIH/KDo63+E8Ys/phqgJ5xsTCEcX68RxqmWpKiXaAzhBChlqyFy\nGzCCMPbwBSEp3OQJl3/v6jTxnoiIxKjlICIiMUoOIiISo+QgIiIxSg4iIhKj5CAiIjFKDiIiEqPk\nILKNzKx3dCesSLmj5CCy7Y4jTPgmUu7oJjiRiJldQJhLv4AwTcSlhDuJ/1c4JbuFhW3+R5iWYwRh\nWpXbCGtBDCLcUf8LYdbTiwnzCt0DnBRt9z3CQjkrLayb/DjhbvF9CIu9HAv8jjApW0d3X2xhmc0h\nhIVhKgFPuHupzfMvuwe1HEQomuX2aqCtux9J+ND+F5uZWdTdnyUsLjTY3R8kJILfESYtbEyYXqEr\ncD1hOvQjo69MNp1fp0q0v78Q5ua6L/p9IWGVOoDHCOuB/I4wh9DJljQBksiOpon3RIIOwJPu/h2A\nu482s/sIE7dtjZOBx6K5fyAkBsxsGnCd/7pc5wOEpFPo6ej7l4QZZWck/J4drXzWJvr5b9FjexKm\nqfYSvkaRrabkIBKkakVnELpyEmferJyiHoSupKI+WjOrHW0zebuZhK6hQonrIKSa379CFEcrd/85\n2nZNtnHWXZGtpW4lkeBVoGu08h1mdi5hla/3idYGiD6UExeB+oVfP+hfA840sywL6y4/RJgq+lWg\nj5lVisovASZtbVDRjKbvEmZOLVyv+23CTKMiaaOWgwjg7pPMbDBhofpMwrrHpwE/AePM7HNgPmH9\ng0KvAEOi7v9BhMWWPiSc6U8lTB1dibDw03TC/9s0wtoJJXFmtJ9PCS2X8e4+rqSvUaQkdLWSiIjE\nqFtJRERilBxERCRGyUFERGKUHEREJEbJQUREYpQcREQkRslBRERilBxERCTm/wOe8UXFcWxWQQAA\nAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x116b2aba8>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sns.plt.figure(figsize=(6,6));\n",
"sns.barplot(y='auc', x='outcome', data=roc_df, hue='transformation')\n",
"sns.plt.ylim((0.5, 1))\n",
"sns.plt.ylabel('AUC ROC');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Accuracy of COMPAS scores by transformation\n",
"For the COMPAS tool, the accuracy of the model decreases as the number of bins gets smaller. The raw scores have the highest ROC values, but not much higher than the decile scores. In Northpointe's rebuttal, they achieved AUC scores on the ProPublica sample of 0.69 for general recidivism and 0.68 for violent in evaluating the decile scores. ([source, p. 3](http://go.volarisgroup.com/rs/430-MBX-989/images/ProPublica_Commentary_Final_070616.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. What's the general case?\n",
"What happens to ROC scores as deciles are composed out of model scores? To find out, we can generate some simulated data. This simulated data is not from a particular model, but is meant to represent a hypothetical model output of scores for positive and negative cases. For simplicity, we'll choose a beta distribution for the model output scores and assume our model has some predictive power, where the scores from the model predicting positive classes is generally higher than those where the model predicts a negative outcome.\n",
"\n",
"In addition, any cuts of data will be determined from the total distribution of scores. Comparitively, we don't know the original distribution of COMPAS scores (from when the assessment was developed) to map to the original decile scores -- we only use those available in the data.\n",
"\n",
"We use a `Beta(2,2)` distribution to represent the scores of negative cases, and a `Beta(3.5, 2)` distribution for positive cases. There's no rationale for the parameters in these distributions except for applying them gives us hypothetical AUC scores in the same range as the COMPAS model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What does this hypothetical model output look like?\n",
"Below is a single simulated example from a hypothetical model -- the blue indicating the distribution of model scores for positvie cases and coral for negative cases. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAENCAYAAAABh67pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGbFJREFUeJzt3X24ZEV94PHv8DbMDep1SHRpIcsStBQGBRYlOwqCYRE3\nJGuysTTIyxCJqKD4QgARFYQY1BFFCSHoJIizu1gIygY3iAFGxJe4KCoaOCuBUeTiG5OZADMwApM/\n6lxyabpvv9x+Pf39PM99erpunT5V5/b8uvp36tRZtGXLFiRJ1bXVsBsgSeovA70kVZyBXpIqzkAv\nSRVnoJekijPQS1LFbTPsBmg8hRC2AU4GjgV2BTYA1wHvLorijrLOmcDhRVHs14f9rwBWFkXx623W\nfymwoSiK73S5vxOBk4ui2LXJ7+ebp3xsURSXdLPffun0+Gm8OaJXt94PvA54KxCAw4GnADeGEKbL\nOiuBlw+neU+yBti5z/s4Ctipwc9n+rxfaV6O6NWt1wEnFUXx9+XztSGECPwceCVwSVEUDwAPDKuB\nQ7C+KIqfDrsRUj0Dvbr1GPA7IYTLiqJ4BKAoio0hhL2BX8ATUzchhIOAy8jfAFYC08AlwEeBVcCL\ngFuBo4qiKBqlFkIIlwA7FEXxR/WNCSG8HDgLeAGwBfgG8IaiKP5/CGFtWe3vQgifKopiRQjhOcDH\ngAPL9n6GnHZ6uHy9FwIfL1/vW8BNCzlYIYQdgX8CLiuK4qSy7C+BVwAvKIri/hDC24A3klNhDwBX\nA28siuLB8ni8gXwM30X+v/sh4FrgYuC5wNeBI4qi+FlZ/61lv/4MeJR8nN9VFMWjDdo37/HQeDN1\no259GFgB/CSEcEkI4ZgQwjOLorijKIoNTbbZETiOnM55PXAC8A/AR4D9gcXAX3TakBDCrsD/AS4H\n9gBeBiwlB0KAF5aPRwEnhRC2B74I/BDYpyw/DDi/fL2lwDXAd8rf/y05aHatKIr7gBOBE0IIe5fn\nDI4HVpRB/gjgTODtwLPJx/aV5OM0ax/gt4Hl5NTZOeQPy5OBg8kptFPn1H8eOaX2MvK5lOOAd9e3\nrdXx0PhzRK+uFEVxbgjhh+QR6BHAMcCjIYS/At7aaNRIfr+dUhTFD4AfhBA+Any+KIrPA4QQVpOD\nUae2JZ8o/Xj5/K4QwqeBN5Vt/UUIAXJqZUMI4Vhgc1EUb57tTgjhDcBXQginAK8GHgbeXBTFr4Db\nyxH+YS3acXkI4Un9Lopih/Lx8hDCa4ALgWcA5xdFcWNZ7V5y0L+6fP6jEMKXgWVzXmo74ISiKO4L\nIVwAfBD4ZFEU1wOEEK4C9pxTfxvgtUVRrAW+E0I4BzgthHBWXRP/eL7jURTFv7bot0acgV5dK4ri\nCuCKEMIOwEHA0eRR6z3AuU02u3POvzc1eL64i3b8MITwYBmkl5FHtnuTg2cjewK/FUKYe/5gEfkb\n7rPL17i1DPKzvknrQP8W4IYWdU4ACmAdOQUz24cbQgj7hhDOJqdh9iwfL52z7YbymwFFUWwqP7zm\nO353l0F+bh+eCdTPtGl1PL7Vok8acQZ6dSyE8HzgT2dHgOVJ16uBq0MIl5EDYrNA/0jd88ea1Gs0\nXbHh+zWEsBfwVXIa6MvkXPT+lCP6Jq/zdXI6o9495b4X1ZVvbvJac907O7V0HrsDU8AS8gfSd+Hx\n6Y5/RU4TXQP8OTmVM1f9sYPmx69R/a3Lx/pvHa2Oh8acOXp1Y2vgxDLPXO9fKU/GLtBm4NdCCHMD\n7m5N6r4e+E5RFH9YFMX5RVF8mXxCsz5Yz7qNPFL9SXlO4Q7yKPdD5PTIrcALQghzR8f7dt+VrMyF\nrwIuIJ9U/ZvyegSAdwAfKoriTUVRrCJ/ADx7nj604zdDCHNH7/uTR/nr6uq1Oh4ac47o1bGiKG4J\nIVxJzkmfTk5XPAU4BHht+bhQNwPbA6eGED4DRPKJwpkGde8BXh1COKD8938H/hS4b06dB4BlIYSv\nAauB9wCfKlMlTyUH4DvLHP5lwHuBT4QQ3k8O8seR0y3zmQ4h/IcG5ZvKE9RnAb9W7nsJcDv5ROq5\nZbsPDiHsQQ7ubyOfWL61xT7nsy1wSQjhVHIgP538TaHevMdjAfvXiHBEr24dQZ6O9zZyMPoKOWVz\nWFEUX1/oixdF8cPytd9Svv4e5KmYjXyMnLb5O3I++XDyVMRnhBBmL5JaSQ5mq4qieBA4FHg6OW99\nFTl18dpy3xvIH1b/EbiFHIw/0kazP00+L1D/c34IYT/yqP2koijuL4ri58A7gfeGnGw/iZwyurns\ny+wMpIV8k7iv7N/XgL8kn7w9r75Sq+Oh8bfIO0xJ1eMSB5rLEb0kVZyBXpIqztSNJFWcI3pJqrhR\nm17p1wtJ6k7Tay5GLdAzM9NomnRrtVqt623H2aT2Gya37/Z7srTT71qtNu/vTd1IUsUZ6CWp4gz0\nklRxBnpJqri2TsbGGJ9JXifjUPJiTP8IvCOl9P0m9Q8t6wfyXWtOTSn9faO6kqT+ajmijzFuBXwO\neA55VcDlwAbguhjjjg3q78G/39ZtH/ICSZ+PMe5ZX1eS1H/tjOhfAPwXYI+U0m0AMcajyEu2/i5P\nvAMO5FX4vpFSml0O9d0xxpeU5a9HkjRQ7eTof0xe9rWYUzZ7V5unN6h/ALCmrmxNWS5JGrCWI/qU\n0n3AF+qK30LO1V/bYJOdefLtx2aAXdppUKuJ//3adpxNar9hcvtuvyfLQvvd8ZWxMcbfJ98Q4bzZ\nVE6dKeChurKHyXcLaskrYzszqf2Gye37KPZ79eqpjuofeeTGjvcxiv0ehF5cGdtRoI8xrgA+Qb7f\n5SlNqtXfiZ7y+YOd7EtSfzULzt0EYY22tufRxxjfRb5D/UXA0SmlZnefvxvYqa6shneTl6ShaHce\n/SnAOcB7Ukpnt6h+E/BSYG69g4Ebu2qhJGlBWgb6GOPzgfcDfwN8IsY49y7395Nn4DwN+EVK6VHg\n48C3YoxnAf+bfBPp/YE39rjtkgas01y8RkM7I/rXAFsDf1L+zPVu4CfklM5/AtamlG6NMf4B+crY\nU4Hbgd9rcuJW0ogxmFdPO9MrTwdOb1HtkrptvsCTp2RKkobARc0kqeIM9JJUcQZ6Saq4kbtnrKTJ\n1uxk8CnNLtFUSwZ6SWPh4oth/fonfwh4JW9rpm4kqeIM9JJUcQZ6Sao4c/RSxXmlqxzRS1LFGegl\nqeIM9JJUcQZ6Sao4A70kVZyBXpIqzumVUkU4jVLNGOilAWgWhLtZp6XZmi9SM6ZuJKniHNFLGmu9\n/LZUVY7oJaniHNFLQ+RoVINgoJc0cM4QGixTN5JUcQZ6Sao4A70kVZyBXpIqzkAvSRVnoJekijPQ\nS1LFGeglqeIM9JJUcQZ6Sao4l0DQxJu67aaG5Ruf95IBt0TqD0f0klRxBnpJqjhTN9IAbHfvHQ3L\nN++0e8Py+VZ3nJ7uSZM0QQz0kirJtf7/nakbSao4A70kVZyBXpIqruMcfYzxImCblNJx89RJwKvq\niq9LKR3S6f4kSQvTdqCPMS4CzgKOB1a1qL4XcBrwqTllD3fcOqkLXgAlPVFbgT7GuBs5uC8Dftyi\n7mJgd+CbKaWfLriFUjM3X8vUhvXDboU08trN0S8H7iaP1O9qUfe55A+Q2xbQLklSj7Q1ok8prQZW\nA8QYW1VfBmwGzooxvgLYBFwOnJNSeqjVxrVarZ0m9XzbcTap/Wbm+0w/rf2rh6abHaeZxq/RtP48\nLr64cfnUkn9pXN7l1U/TE3rVVC/6XauN37Fb6P/xflwwtSewCLgduID8LeA8YBfgmFYbz8zMdLXT\nWq3W9bbjrCr97iavXgPWd5C62djkODVL/zSrP5/16xtfpLPdpk0Nyzev7zz1ND09zfoutht3ver3\nzMx4XTDVzv/xVh8E/Qj0ZwArU0rryue3xhgfBS6LMb49pXRfH/YpSWqi54E+pfQYsK6u+NbycRfA\nQK+2NRvpA9BB2kaaZD0P9OUc+m1TSn8wp3g/8vTKxis7SWOm+QfQoQNth9SOBQf6GON2wFJgXUpp\nM/BZyjQNcBWwD7CSnM55YKH7kyR1phdLICwH7i0fSSklYAVwLPB94MPA+cB7erAvSVKHOh7Rp5QO\nqnu+hjzLZm7ZpcClC2mYJKk3XI9eE2PeE7tShRnopSaG+cHQ7I5UAKxf0nBefrO7VUkGeg2Uo2pp\n8Az00hDNO3If0r79ZlA93nhEkirOEb3UQ8McoUvNGOilLlxy9W7DboLUNgO9pImyenXjFUYBjjxy\nvFa2bJeBXpJKzT4Exv0DwJOxklRxjujVtfnmxHsjbml0GOjVF14YNb66mTnk3PvRZqDX47q5nZ9G\nhxdAqRlz9JJUcY7oNTGazX1fcfidA26JNFiO6CWp4gz0klRxpm5UOS5PID2RgV5qwg8MVYWBXqo4\nV9SUOXpJqjgDvSRVnIFekirOQC9JFWegl6SKM9BLUsU5vVItueSwNN4c0UtSxTmil7RgnV6U5Rr5\ng2Wgn0CmYqTJYupGkirOQC9JFWegl6SKM9BLUsV5MlYTz3XnVXWO6CWp4gz0klRxpm4kDdx8F1h5\nMVXvGeglqYXVq6calh955MYBt6Q7HQf6GONFwDYppePmqbMfcD6wD3APcHZK6dKuW6mueAWsJOgg\nRx9jXBRjfB9wfIt6vwF8Efg2sC/wMWBVjPHQhTRUktSdtkb0McbdgFXAMuDHLaofB2wATkopPQbc\nHmPcFzgZuHYBbZWe4OIrn8XGTUuH3Qxp5LU7ol8O3A3sBdzVou4BwI1lkJ+1BnhxjHFRxy2UJC1I\nWyP6lNJqYDVAjLFV9Z2BW+rKZoApYEfgl501UZK0EP2YdTMFPFRX9nD5uH2rjWu1Wtc7Xsi246xp\nv2emB9uQIZhasmTYTRiKKvd7arr5+3Z6nt8NQ602mPYsNLb1I9BvAhbXlc0+f7DVxjMzM13ttFar\ndb3tOJuv31Mb1g+4NYP2LDZu2jTsRgzc1JIlle735vWN37fT09Osb/K7YZmZ6f/0ynZiW6sPgn5c\nGXs3sFN9O4AHyCdpJUkD1I9AfxNwYN2J14OBr9adoJUkDcCCUzcxxu2ApcC6lNJm8jTMU4CLYowf\nBQ4BjgAOW+i+JE2wtT9guwYpK5dMaK0XI/rlwL3lIymln5GD+j7k2TcnAkenlK7vwb4kSR3qeESf\nUjqo7vkaYFFd2TeAFy2kYZKk3nCZYkmqOFevHBNNFyirtbyATdKEc0QvSRVnoJekijPQS1LFmaMf\ndzdfOwFLHUida3a7wkmcd2+glzRSmt5PtsILufWbgV6SujQu95I1Ry9JFeeIXiPhkqt3a1i+4vA7\nB9wSqXoc0UtSxTmiHzFNr4CdUM1G+gBTnpuT2uKIXpIqzkAvSRVnoJekijPQS1LFGeglqeIM9JJU\ncQZ6Sao459H3WbN58Ruf95IBt0TSpDLQD4kXRkm90XS1Sz3OQC9JPTZqq1qao5ekijPQS1LFmbrR\nQM23SJmk/nBEL0kVZ6CXpIozdaO+MEWjUTXfdMzNO+0+wJYMjiN6Sao4A70kVZyBXpIqzkAvSRXn\nyVh1zROu0nhwRC9JFeeIXpJKzaZejvu0S0f0klRxjuh7wLXlJY0yR/SSVHEGekmqOAO9JFVcWzn6\nGOPWwDnACuApwDXACSmlnzWpn4BX1RVfl1I6pPumStJwjPtsnHZH9GcCxwBHAwcCOwNXzFN/L+A0\nYKc5P/WBX5I0AC1H9DHG7YCTgLeklL5Ulr0GuCvGuDyl9LW6+ouB3YFvppR+2oc2S5I60M6Ifm9y\numbNbEFKaS2wFjigQf3nkj9Abltw6yRJC9ZOjn7n8vGeuvIZYJcG9ZcBm4GzYoyvADYBlwPnpJQe\n6rahkqTutBPop4DHUkq/qit/GNi+Qf09gUXA7cAF5Hz9eeQPhWNa7axWq7XRpN5vuyAz08PZb2n6\nacPZ/9SSJUPZ76i1YRjs92iYmu7s/16t1t3/1YXGtnYC/SZgqxjjNimlR+aULwYebFD/DGBlSmld\n+fzWGOOjwGUxxrenlO6bb2czMzPttPtJarVa19su1NSG9UPZL+Qgv35I+9+4aelQ9jtraskSNm7a\nNNQ2DIP9Hh2b13f2f29mZmPH+2gntrX6IGgn0N9dPu40598ANZ6cziGl9Biwrq741vJxF2DeQD/K\nXOpA0jhq52Tsd4H7gZfOFsQYdwV2BW6srxxjTDHGz9UV70dO9TS/K68kqS9ajuhTSg/HGC8EVsYY\nfwn8HLgQ+HJK6Rvl9MulwLqU0mbgs5RpGuAqYB9gJTmd80C/OiJJaqzd1SvPALYFVpeP1wAnlL9b\nDtwAHAysSSmlGOP2wJ8Bf07+YDgf+IsetlsD5J2kpPHWVqAvT8K+o/yp/90a8iybuWWXApf2oH2S\npAVyPXo9zpG71F+rV081/d2RR3Y+I6ddrl4pSRXniF6SujQuq1o6opekijPQS1LFGeglqeIM9JJU\ncZ6MbcA1bSRViSN6Sao4A70kVZypmwnkFbDSZHFEL0kVZ6CXpIoz0EtSxRnoJaniPBkrST02aoud\nOaKXpIpzRD/mLr7yWWzctHTYzZA0whzRS1LFGeglqeIM9JJUcROdo3eVSkmTwBG9JFWcgV6SKs5A\nL0kVN9E5+nHSbGnhqSUDboiksWOgl6QBabY0Qlbr235N3UhSxRnoJaniTN2MGG/zJ6nXJiLQe2GU\npEk2EYF+mByhSxo2c/SSVHEGekmqOAO9JFWcgV6SKq46J2NvvpapDeuH3QpJGjmO6CWp4gz0klRx\n1UndDIBz4iWNo7YCfYxxa+AcYAXwFOAa4ISU0s+a1N8POB/YB7gHODuldGkvGixJ6ky7qZszgWOA\no4EDgZ2BKxpVjDH+BvBF4NvAvsDHgFUxxkMX2lhJUudaBvoY43bAScDpKaUvpZS+DbwGeHGMcXmD\nTY4DNgAnpZRuTyl9HFgNnNzDdkuS2tRO6mZvcrpmzWxBSmltjHEtcADwtbr6BwA3ppQem1O2Brgw\nxrgopbRlIQ0eBHPxkqqkndTNzuXjPXXlM8AuTeo3qjsF7NhR6yRJC9bOiH4KeCyl9Ku68oeB7ZvU\nf6hBXZrUf4JarcvbadVqTHe35ZO89Xd69EKS1ANdx8VSO4F+E7BVjHGblNIjc8oXAw82qb+4rmz2\neaP6cy1qoz2SpA60k7q5u3zcqa68xpNTNLP1G9V9gHySVpI0QO0E+u8C9wMvnS2IMe4K7Arc2KD+\nTcCBMca5o/ODga/WnaCVJA3Aoi1bWk+CiTGeS75YagXwc+BC4KGU0kHl9MulwLqU0uYY4zOBAvgM\n8FHgEODDwGEppev70QlJUnPtXjB1BvA/yfPhbwB+BPxR+bvlwL3lI+XVsoeRr4q9BTgRONogL0nD\n0daIXpI0vly9UpIqbixWr5zkRdW66PurgXcCzyan1D4JfCil9OhAGtwjnfa7bturgR1SSgf1s439\n0MXfe2fyubCXk6c2fxY4OaW0cSAN7pEu+v0y4FxgT+CnwF+T3+djm6KIMV4EbJNSOm6eOl3FtnEZ\n0Z/J5C6qdibt9/0V5HMpnwSeD5wGnAqcPoiG9tiZtNnvuWKMxwO/29eW9deZtP/3Xgx8iTwZ4sXA\nq4HDgQ8OoqE9dibt93t34OryZy/ye/y9wJsG0dBeizEuijG+Dzi+Rb2uY9vIB/pJXlSti76/Abgi\npXRBSumfU0qfBc4Djh1cqxeui37Pbrc78H7g64NpaW910e8jyNes/I+U0vdSSjeQA96LBtboHuii\n34cBm1JK70sp3Vm+z79A/lYzVmKMuwHXA28EftyietexbeQDPU0WVQPWkhdQq9dsUbUX183tHwed\n9v0c4Ky6sseAp/eldf3Tab9nv/pfCnwA+Kd+N7BPOu33y4EvpZT+ZU79v00pjVWgp/N+/wJYGmP8\n4xjjVjHGZeRvATf3vaW9t5x8kelewF0t6nYd28YhR9/Nomq3NKg7u6jaL3vauv7qqO8ppf8393mM\n8ankkcI1fWld/3T6N4d8XmILsBK4uE/t6rdO+/0c4PoY49nAkeT+XwmckVKqX29qlHXa7yuAVeQ0\n5aeBrYFEHuiMlZTSavKonBhjq+pdx7ZxGNEPdFG1EdNp3x8XY5wCPg8sIefqx0lH/Y4x/mfgHcAx\nY371dad/76cCrwN+C3gV8DZynn7cPug67fc0+cr8DwIvJOf2/ys5bVVlXce2cQj0jy+qVlfej0XV\nRk2nfQcgxvjrwD+QT9gcllL6Uf+a2Bdt9zvGuD15VHdGSumOAbWvXzr9e/8KWAcclVK6OaV0FTnY\nHxVjHKclwTvt9weAR1JKp6WUbilnnZwMvHPM+t2prmPbOKRu5i6qdvec8klYVK3Tvs+uQ3QtOed5\nYErpe/1sYJ900u/9gecBH4gxfqAsW0wOHA8Ae6SUWp3kGhWd/r3vIS9FMnfq7Oz5iV2B+3rdwD7p\ntN+/DXyuruwfgW2B32R8+t2prmPbOIzoJ3lRtY76HmN8BnmJiq2A5WMa5KGzfn+TfM3A3nN+Pkc+\nMbc3OYc5Ljp9r38F2DvGuO2csmXAo+QTmeOi037/hDx9eK5l5IkH/9yXFo6GrmPbWCyBMMmLqnXY\n98vJU89exhNHRlvaudBolHTS7wbbfhLYfUwvmOr0vf4D8lz6s8gn61YB16WU/mQIze9ah/3+b+Q5\n9O8B/hewB/mCqStTSm8eQvN7Isa4Brhj9oKpXsa2cRjRw2QvqtZW32OMS4A/BHYgj3LvnfPTMM0z\n4tr+m1dMp+/1A8nB4NvkoHcFeabVuOmk3/+X/F5/JfA9ctD7a+Dtg21y3/Usto3FiF6S1L1xGdFL\nkrpkoJekijPQS1LFGeglqeIM9JJUcQZ6Sao4A70kVZyBXpIq7t8AF21amu7GM4MAAAAASUVORK5C\nYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x116a865c0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"SAMPLE_SIZE = 10000\n",
"\n",
"neg = scipy.random.beta(2, 2, size=SAMPLE_SIZE)\n",
"pos = scipy.random.beta(3.5, 2, size=SAMPLE_SIZE)\n",
"fig, ax = plt.subplots()\n",
"bins = np.arange(0, 1, 0.02)\n",
"sns.distplot(neg, ax=ax, bins=bins, kde=False, norm_hist=True, color='coral')\n",
"sns.distplot(pos, ax=ax, bins=bins, kde=False, norm_hist=True, color='blue')\n",
"sns.plt.title(\"Simulated Example\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run simulations\n",
"The simulation below runs 500 simulations with 10000 samples for each simulation, resulting in 10000 simulated 'positive' cases and the same for 'negative' cases. We will apply quantile cuts ranging from 2 to 15 quantiles on the combined positive and negative predictions, as well as the case where no cuts are applied. From each simulated dataset the ROC is calculated."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def run_simulations(alpha_pos, alpha_neg, beta, qcuts, sample_size, n_sims):\n",
" sim_data = []\n",
" cuts = list(qcuts) + ['None']\n",
" for qcut in cuts:\n",
" for sim in range(0, n_sims):\n",
" pos = scipy.random.beta(alpha_pos, beta, size=sample_size)\n",
" neg = scipy.random.beta(alpha_neg, beta, size=sample_size)\n",
" roc_scores = np.hstack((neg, pos))\n",
" if qcut != 'None':\n",
" roc_scores = pd.qcut(roc_scores, q=qcut, labels=False)\n",
" roc_true = np.array([0] * len(neg) + [1] * len(pos))\n",
" auc = roc_auc_score(roc_true, roc_scores)\n",
" data = {'sim' : sim, 'AUC' : auc, 'qcut' : qcut}\n",
" sim_data.append(data)\n",
" return pd.DataFrame(sim_data)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"sim_df = run_simulations(alpha_pos=3.5, alpha_neg=2, beta=2, qcuts=range(2, 16), sample_size=10000, n_sims=500)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot Data"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.text.Text at 0x116a28b00>"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAt4AAAH1CAYAAADBF/k3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8W9d14PHffQD3VSQlihQlUevTLse75X2JndRO6jQJ\n0uxpM+2nnTZd0jTtdDppOplmMlMnnTSZJNPWiRMvseElXmVt1mJt1r5YlPS0UaIkSiLFVdxBAvPH\nfY8CKYAEQKzW+X4++ogEHoALECTOPe/cc1UgEEAIIYQQQgiRWEaqByCEEEIIIcT1QAJvIYQQQggh\nkkACbyGEEEIIIZJAAm8hhBBCCCGSQAJvIYQQQgghkkACbyGEEEIIIZLAneoBCCHSg2mabuAbwO8B\ntUAH8A7w3yzLOmEf823gMcuybk7A438FeMKyrIoIj78X6LAsa3+Mj/enwDcsy6od57gvAb8E/otl\nWd+LdMymaT4FFFqW9amgyyqB/wp8HKgEGoBf2ffRP8YYfgf4JrAMGAR2A/9kWdY74z7RFDNN8z5g\nA1BkWVZXHO5vIzAPWGhZVueo6y6jf6ZPTfRxQjzut0nQez+KMRQALwAPAbssy7p71PXfBv4h6KJB\n4DTwY8uyfhh0XAD4mGVZbyZ6zEKIkSTjLYRwfBf4KvAXgAk8BhQB75qmWWof8wTwSGqGd42NQE0S\nHucLwHH0hCRmpmnWAnuBOcCXgEXA3wF/BDw/xu0eBZ4G/g0deN8O7ARWmaZ550TGlMGqgf+Z6kGk\nwOPooPtOwBPmmANAlf1vHvDfgO+ZpvnJoGOqgLUJHKcQIgzJeAshHF8F/tyyrLft70+bpukBmtAf\n+E/ZGcsJZy0zhWmaVcAD6OD716Zp3mVZ1pYY7+6ngAV83LKsIfuyetM0zwPbTdN8xLKs1SFu9/vA\nry3L+nnQZX9rmuZtwB8AW2McTyY7DfyRaZpPW5b1XqoHk0SlwCXLsvaMccygZVkXg74/bZrmY8Dn\ngZcBRl0vhEgiCbyFEA4/8KBpms9bljUIYFlWj2maNwDNMPJ0u11C8Dw6Q/4EOih4Cvg/wJPArcD7\nwBcty7JClWWEKscIuu4R4B+B5UAAeA/4I8uyjpmmedo+7A3TNH9pWdZXTNOcD/wrcI893hfQZTL9\n9v3dAvzIvr89QCQB9GeBK8CLwLfQQXDUgbdpmtPQZwoeDQq6AbAs6z3TNO9Hl4+E4gduNU1zkmVZ\nbUGXfwEYvi/75/E/0c/vAroU5ef2dXPRP6N77ft7Bfi6ZVlX7Nu9BPwC+EN0kP9Hpml+1L4/EziF\n/tn9wr6/IuBnwEeAXGAz8DXLso6P8TL8nmma/wV9FuUV4E/tx18JtFiW9cWg5/KPwP2WZd0T5r5e\nBm4A/s00zRud92uwMKU+p+3n8WP7vbwQqAf+GOgF/ov92v0QnVVfg37/9th34TJN84fo90EX8CPL\nsr4bdP9jvWbfBm5Dn2m+Hfgzy7J+OWrMCvgT4GvATOAo8HeWZa0MLiOxS0V+L4qSmm4gP+hxhktN\n7NKdTejX82HgMvCPlmX9h33sXcAPgKVAG/AMuuxqxPtYCBEZKTURQji+D3wFOGea5lOmaX7ZNM1K\ny7JOWJbVEeY25cB/QgeVf4gOGtYB/4IOMnKIoSTALst4HR3wLkJnncuAf7YPucX+/4vAn5ummQus\nRpeEfMi+/CPoAArTNMuAVcB++/pfoCcM4/kC8IYdZLwCfNo0zcJonw+6REShS0SuYVnWxjHqn38E\nLED/XF4zTfPPTdNcYFnWeSdzaZqmiX7+m9EB1LeAn5mm+aBpmpPQkwUfcDfwO8BdQHAGvRyYC9wI\nfN80zcXo4PYnwBLgv9uX/659/HfQQesD6NdzaNT9hfKH9mN/FFiBfr+BDuR+2/4ZOj5rXz6WP0KX\nUnxjnOPG8jj6PXojehL5E/Rz+6I91ofR72/HDegyjduArwP/1TTNzwFE8JqBfk9uRAfeK0OM52/t\nx/8H9HvmVeB10zSXoydOfwecs8fwQiRP0DTNm+zn8vQYh/0N+vdjMfAb4CemaVaapukCXgPWo3/e\nX0SfZflKJI8thLiWZLyFEABYlvU90zSPo7N/nwO+DAyZpvlT4C/CZLjcwDcty6oD6kzT/BfgVcuy\nXgUwTfMZRgYukcpCL5L7kf19vWmaTwP/2R5rs441abcsq8M0zd8DBizL+przdEzT/CNgs2ma3wQ+\nA/Sjs7I+4KidAf9IuAGYprkQHVT+d/uil9ALIz+NDtyjMcn+P9wEJizLst41TfN24K/RQevH7fGt\nA75gWdYl9GtcZ1nWN+2bHbMDbgNdYuACvmRZVq99268A75mmOS/oob5nWdZJ+/pfAs9YlvVv9nUn\nTdOcA/wVOkCdhc6inrZf/z9AL8gdyx86ZSGmaX4deMk0zb9AB5f/D/gt4BXTNG9GZ3tfHOd1OWGa\n5neAb5mm6bUs69Q4jx9KD/p9NmSa5v8D/hx9pmCnPc530cGoow2dae4GDttj/c/Ac+jFr2O9Zs7j\nfc+yrMDogdjZ7q/bj+8c/227pOiblmV93jTNK8DQOKUiHzJN05nEZQHZ6MWtq8a4zQbLsn5qj+Pv\n0Rn35eizMJOAS8AZy7JOm6b5MLr8TAgRAwm8hRDDLMt6GXjZzureh14E+KfAeeB7YW4WHPD0hvg+\nJ4ZxHDdNs9sOmpegT93fgC4DCGUxMCco4ACdYTbQWdElwPt20O3YyRiBNzq7143OJGNZ1n7TNE+g\nywycwNtH+DOHhn096NP3oIOY5jEeMyTLsvYCnzV155lbgU+iA75f2s9hEbBr1G1+DGCa5ieA/U7Q\nbdsFDKBft3b7suCf22JgqZPNtbmDns/30Wckmuzg9HV0mVE4Q6PGtwcdEM63X9dX0JOjV9DZ7pWj\nymrC+Wf0JPGnxLbo90zQhNJ5fcZ6/75vB92OPei1ETD+awZ6onJN0G2bDFQA20ddvgU92YvUYeAT\nQY9fi144/Trw4TC3GS4Rsiyr057UZlmW1WqX1vwA+BvTNN8Gnrcsa1eY+xFCjENKTYQQmKa5zDRN\nJ7uMZVldlmW9aVmWB31Ke6wAdXR9rT/McaECjpCTf9M0l6LrW29HBzffRHdnCMeNDlhuCPq3HB10\nH7YfW426zUC4O7Ozj58DCoArpmkOmqY5iO5IcldQprgNKDRNM9Tf0klcDWr3YNdqh3m8Z03T/HyI\nywtN0/yRnTnFsqxBy7K2WZb1V8BfAg/Z5QADIZ6foy/M5QqdCXcEB+ZudIlL8Ou5BH0GAMuy3gWm\noydm59G1+O+ZppkX5rEgqB6dq589zs/gGeAxU7fL+wzjl5lgj8OHLmH5cIjXL5L32zW14YR//8LI\n5wD6eTjPYczXzNZLeOF+TgYjf07j6bfLw05YlnXUsqxV6BKwh+xymFBC/S4oAMuy/hKYD/xvYAaw\n0jTNb0UxHiFEEAm8hRCgP9j/1NS9sUfrJIYsbQgDQIEd1Dpmhzn2D9FZ2t+xLOuHlmVtQmfuwgWX\nR9BB9jkn6EBnD/8ZnVl9H1hummZw9vLGMcZ6N7rc4Q8ZGUg9iA7Mft8+bj/6dP4twTc2TdO5bB+A\nZVkt6Jreb4wO0k3dEvBz6CB+tF50qcgXQlzXCbTZGdtjo5+PaZr/bprm99GvzQ2jguJb7HEfCfP8\njwBzgwK4E/Zz/1P7vv8CuNuyrBcsy/oKuuZ5EbouORQXOgh13IEONJ3s8jvoMpy/BgqBiPtLW5a1\nDV2q8i/ohZ6OAfRCTuwxFwJTIr3fMBbbZx0cd6AndjDOazYeS/ckb0TXvwdbQfifU6Sc35toAnjs\nOu//C1y0LOsHlmU9iD7z9dkJjkeI65aUmgghsCxrn326/0XTNP8Oe8MTdM/gz9v/T9RudGD0N6Zp\nvoDuQ/whdLAx2nngM6Zp3m1//dvoRV0tQcd0AUtM09yGzpB+C/ilXfdbjO6scsquQX4evWDt303T\n/C46SP1PQGuYsX4BvYjt56Nr2+0uHF8yTfPvLctqNE3TCzxtmuafoQOkaejFagPo2l/H19Gt/143\nTfN76LKZu9CTgxeBtxnFrj3+B+AHdsDnRdeq32zf7n/Zh/4Uvcj0O+gNeW5Fl8r8FrobzLeAX5m6\nW0iZffw6y7IOm6YZKhh9Athhvxe86J/T99GZbezn+DW7tOcsusd5J7pdYigB4CnTNP8YHVh/H/ih\nZVl99vP0m6b5nP26PWuNsZlQGH+Lfo8UBF22C/iiaZq/BZy0xz7RThwVwM9N0/xf6MnGV9ELNGH8\n1ywS3wP+u2maDeie759FL/C8L4r7cJumOTXo+xn2OOrQE9BotKKfX75pmv8T/fo+TJhFwkKI8UnG\nWwjh+By6Hd9foj+gN6NLTD5iWdboutOo2a3m/hL4M/v+F6FbD4byr+juKG+gyzQeQ3exmGKaprNp\nzhPogPJJu+72YXR5x050J4bt6EkDdleWh9BZ7H3oThj/EuqB7az4p4D/CLOg9EfoVnNO+c2X0LXJ\nP0Fnnn+DnhTcFdypxH7+K9CTh+eBQ+iA8fvolnUha3/tBaZfQncQ2QIcRGeGv2VZ1hP2MaeBjwGP\nol/bbwFftSxrvd0K7xGgBB2Mvgy8i+50EZKl+0R/Cl32UYd+rf/J/h902c8qdBnSEXRm91HLstqv\nvTdALyr8OfCW/Vq9ao8x2HPoidmz4cY1xng70Asjgz1t3+fz6Pfyfibe83wteuKzEx1Q/5ldyhHJ\naxaJH6NLOv4Z/XP8GLp95+Yo7sNpJ3kBPWl9Cz2JfGyM+vKQ7FKeR9GLafegu5scRv8OCyFioAKB\nqH4PhRBCiLgz9Q6dPwVmRhsgCiFEppBSEyGEECljmuYsdM35t4B/k6BbCPFBJqUmQgghUmk6ugzl\nHLptnRBCfGBJqYkQQgghhBBJIBlvIYQQQgghkkACbyGEEEIIIZLgellcKfU0QgghhBAiWUJu+Ha9\nBN40Nobao0MIIYQQQoj4qa6uDnudlJoIIYQQQgiRBBJ4CyGEEEIIkQQSeAshhBBCCJEEEngLIYQQ\nQgiRBBJ4CyGEEEIIkQQSeAshhBBCCJEEEngLIYQQQgiRBBJ4CyGEEEIIkQQSeAshhBBCCJEEEngL\nIYQQQgiRBBJ4CyGEEEIIkQQSeAshhBBCCJEEEngLIYQQQgiRBBJ4CyGEEEIIkQQSeAshhBBCCJEE\nEngLIYQQQgiRBBJ4CyGEEEIIkQQSeAshhBBCiOvK0aNHeeGFFwgEAkl9XHdSH00IIYQQQogU+/73\nv8/g4CB33HEHM2bMSNrjSsZbCCGEEEJcVwYHB0f8nywSeAshhBBCCJEEEngLIYQQQgiRBBJ4CyGE\nEEIIkQQJXVzp8XhcwP8AvgIUAauAP/F6vZfCHF8D/B/gEaAXeAn4htfr7bGvn2pf/yDgB7zA33q9\n3u5EPg8hhBBCCCEmKtEZ728DXwa+BNwD1AAvhzrQ4/HkAGuBMuBO4DPAY8D/tq/Psq9fCDwOfBS4\nEXgtkU9ACCGEEEKIeEhY4O3xeLKBPwf+zuv1rvV6vXuB3wXu9Hg8K0Lc5HNAFfBJr9d70Ov1bgD+\nAbjVvv5RYAnwKa/Xu9W+v88AD3g8nnsT9TyEEEIIIYSIh0RmvG9Al5dsdC7wer2ngdPA3SGOfwRY\n6/V624KO/4XX63UC73nARa/Xezzo+nPAZUACbyGEEEIIkdYSWeNdY/9/ftTljcD0EMfPB9Z7PJ7v\nAF8AAsArwN97vd4++3ZlHo+nwKnp9ng8RejSlCkJGL8QQgghhPgA+yDtXJkP+L1er2/U5f1Abojj\ni4GvAm8DnwamAT9GB9Vfsi/vBP7N4/H8CTow/4n9f/Z4g6muro7tWQghRBwEAgH8fj8ulyvVQxFC\nCGErLy9PaoyYyMC7FzA8Ho/b6/UGbwuUA4TqQuIDWoEver3eIWC3vaDyRY/H85der7fF4/F8HPil\nfVwv8CNgP9Ax3mAaGxsn9myEEGICnnzySU6ePMl3v/vdVA/lA6uxsZF169bx6U9/mry8vFQPRwiR\nAZqbm+MeI44VyCcy8D5r/18V9DVANdeWn2Bf1mcH3Y7D9v+1QIvX690OzPd4PFOAK0Afusb7yTiO\nWwgh4m7Lli2AznwrpVI8mg+mp556iuPHj1NZWckjjzyS6uEIIcQ1Ehl4H0AHx/cCzwB4PJ5adBD9\nbojjNwN/4PF4soLKU5YAQ8Bpj8czD/gF8HGv19tk3989QCmwLnFPQwgh4kfKTRKnvr4egM7OzhSP\nRAghQktY4O31evs9Hs9PgCc8Hs9loAldk73J6/W+Z7cbLANavV7vAPAz4GvArzwezz+iF2f+M/Ar\nu8ykE133/SOPx/MP6AWaTwNPer3eE4l6HkIIEU/JXsgjxPVuaGiItrY2KioqUj0UIRK+gc7fA8+i\nM94bgDPAp+zrVgAX7P+xd7O8Bx2M7wWeQ2+288f29T70hjpV6LruXwFPAf85wc9BCCHixu/3p3oI\nQlxXvF4vf/3Xf83586GqXMX17oPU1QR7UeVf2f9GX7cRUKMuO4zu5x3u/uqAB+I7SiGESJ6hoaHx\nDxJCxM2aNWsAOH36NNOmTUvxaMT1LtEZbyGEEEEk8E4cKeMRY5H3hwgl2e8LCbyFECKJpNQkcaRb\njBiLvD9EKMn+myyBtxBCJJFkvIVIDcl4i1AGBwfHPyiOJPAWQogkkox34khgJcYiGW8RSrKTIRJ4\nCyFEEknGW4jUkImZCEUy3kII8QEmgXfiSEZTCBEtn883/kFxJIG3EEIkkWTdhEgNmZiJUCTwFkKI\nDzCp8U48mdwIISIlpSZCCPEBJkFh4snkRoSSSb97J06c4OzZs6kexgdW8Hsh2RnvhO5cKYQQQojw\nAoEAgUAAw5A8WKJlUqnJP/3TP5GTk8PPfvazVA/lAyl4rY2UmgghxAdYJmXdROL94Ac/4Lvf/W6q\nh3FdyLTfvf7+/lQPIWItLS1873vf4/jx46keSkSCA28pNRFCiA+wTMq6ZapMeo0PHTrEyZMnUz2M\niAUCAerq6ujp6Un1UEQaWb9+PZZl8fTTT6d6KBEJDrylj7cQQggxAVK2kTiWZfHEE0/wzDPPpHoo\nIo042fnW1tYUjyQywetAZMt4IYQQKTcwMMB3vvMd3n333VQP5bqQKWUQjY2NAGzfvj3FI4leppwJ\nyZT3QiaTwFsIIa4TmfLhf/LkSU6dOsUvfvGLVA8lYpkcsCS7zvR6lCnvj0zeZCtTXuPgcUrgLYQQ\nH2CZEniLxAsOsDJpIZ1IrGR32YgHJ5DNlL9vwb97EngLIcQHWKZ8MGVK5ipYpry2jr6+vpBfi8TI\nlLMKmRh4OzLl70bwe0EWVwohxAdYpgWHmSRTPvQdvb29w19Ll5DEy5SzCsHjzJTJQqYJntxIH28h\nRNL19fXJbn9JkimBd6aMM5RMGXtXV9fw193d3SkcyfUhUwLvTDwTkinjdKTyNZbAW4jrnM/n4+tf\n/zpPPvlkqocSsStXrrBy5coRGUMRX7LAK/GCA+/gr0ViZMrkJhPPhDivbab87gW/F5L9uyeBtxDX\nuZ6eHnp7e9m2bVuqhxKxV155hRdffJFVq1aleihRy5Rs7MDAQKqHELVMeW0dV65cGf66s7MzhSP5\n4AouIwh+vdNZcFCYKZMFJ3jNlDOn7e3tIb9OBgm8hRAZx9npr6GhIcUjiV6mBIeZduoYMifb5ggO\ntjMlKMw0wa9xR0dHCkcSueD3QqacCem2x9nf358Ri0NbWlqGv25taU3q3w4JvIUQQlwjU+phgzmT\nmkz44IeRQWGmZLwzJaPpCN5JsaUlM3ZVDJ4gZMpkoasrdaUbsbh06RIAJdkFDPgGaGtrS9pjS+At\nxHUu07KEwTIlexwsU8acifXzzns5U7L1wZnNTMl4Z1oJUnNz8/DXLS2XM2LisGvXruGvM2ELdr/f\nT3dPZgXe586dA2DAPzji+2SQwFuI61wmB96ZOPZMDLwzZaGlE1RJ4J04mXYmxMls5uYWMTg4mBGB\n7MWLF4e/vnz5cgpHEpmurq4Rf4vT/b3c39/PhQsXcCsXbsMFJLdsUQJvIa5zmRJUfVBkymQhOPDO\nlOy389pmWicIyIwsIWRe4H3hwgUA8vNKR3yfzpy/yQpFU1NTikczvtGBdroH3g0NDfj9ftzG1cC7\nvr4+aY8vgbcQ17lMOPX6QSKBd2IETyAzK/BWoIyM6V6RCe+FYI2NjQD09LaP+D5d+Xw+/H4/LsPN\npIIKLjSm/0TB6QriUq4R36erEydOAJDlcuFSikm5RZw4cSJpf5sl8BbiOpfJgXemlG0Ey5QzDMHB\nayYEspmYPe7q6gKlQKmMCbwtyxr+Ot13VRwcHOTixYsYhhvDcAPJreWNhZORdxluphRPo/NKZ9ov\nvHXKd7Ls1zjdy3mOHz8OgNtwA4p5k2ro7OxM2tkFCbyFuM5lSgb2gyLdgxVHpm3iMaJeOgMC70Ag\noINtO/D2+XwZsXAxuA1buk9wLly4wNDQEIbhsv+5OX/+fKqHNSan1thluKgqnQHA2bNnUzmkcTkB\na5aRNeL7dBQIBDh+/DjlecW4lA6BzbIaYOSkMpEk8BZCZKxMmTQEjzNTWt1l2nbmwYF3X29v2r/O\nvb29+uyHHXhDZrQUDD5Dlu4lBU522+XSmdjS0mrOnz+f1mf5Tp8+Dehs7LTSWiC59cexcMp3sgw3\nhVkFaV3O09jYSFdXF2bZ9OHLzDI9wXEy4YkmgbcQQiRYcCYzUzpuBLdhS/fMJlwbBKZ7/+Ph8Spj\nOPBO9zH7/f4RQWu6lxQ4gbdTZlJWNp2BgYER7+10c+rUKUCXmkwvmwOkf+B95swZFAqlDGYUTaOp\nqSltz5I5We0FdrANUFM0mYKsXI4ePZqUMUjgLYTIWIaRGX/CMq1eOhAIjJgspHuXAuDqBhh2EJvM\nDTFiMVyyYSiw38fBZRzpaPTkJp1LCoDhshLD7lxRPklnOdO1zntgYICGhgZcdu1xSX4ZxXmTkrrw\nL1odHR20tLTgNtwoYFZxLZC+kwUnuF5YfjXwNpTCLJvO5cuXk9K+MTM+tYQQIoR0/TAaLdP6NQ9n\nuO0gNt1LCiAo++rSQVa69z8ebmtnuMDuBpHure6c8Sl3NpD+HULOnz9PXl4Jyq7lLStL78D79OnT\nDA0N2Yv+tJnl85K68C9aTgbZWVg5v1Rn6ZOVPY6G3+/nyJEjTMotYkr+pBHXLSqfCcCRI0cSPg4J\nvIW4zmViZxBHpgTewdnXzApi9UdEumePISj76tYBQDqXE0DQhh2GMfw6J3MTj1g44zPc2SjDndaL\n/np6erh8+fJwsA3pH3gfO3YMALcra/iy2or5QPLqj6N1+PBh4OrCSnPSHAxlJCWAjVZDQwNdXV0s\nqai95nNvccUsAA4dOpTwcUjgLUScdXV1pfUH0miZUq6RyYJLCNI9EwtBQathQJY77YNYsLOx6mrZ\nRrpnj48fP47KzkEZLpQyUAVFHD+eviUFcLX+WLnc5JTXcPbcubRdxOoE1+VBtbwFBWXk5BSk7QRn\nOPA2ggNvc8R16SQQCHC4ro48d95wlj7Pncfs4pnU19enXVndwYMHAVg2ec4111UXllOWW8yhQ4cS\n3vJVPnGFiLOf/vSnfOtb30q7PzrhZHLgnc5BSjBn22pI/7pYCBqvYUBxIZcuXUrrThB9fX16QmMY\n+p/bnbZZTdATm6amJoypQXWmVbVcudKZtpP2QCCggz+lQLnImzqXocFBTp48meqhheR0B6moqB2+\nTClFRXltWi7+Gxoa4vjx40wuqhoujQGoKp1BblZ+0lrdRaOpqYnmy5dZXGYSnD9eUr4Qv98/nA1P\nF3v27MGlDJZMnnXNdUopPlQ5l56enoS/1pn7iStEmnL+2GRCJwjIzFITJyORbh+e4QxnX/OyaGxs\nTPsJw3DQahioScX4fL60ztSfOXNGv6b2JFKVl9PY2Ji2fbH37t0LgDFj3vBlrhlzR1yXbs6dO0dn\nZyfKLoPIn7YAgLq6ulQOKywnOz9lVHZz8uTZQPot/jtz5gx9fX3MmrxgxOWGMqitmE9TU1PalXw5\nP/sl5SPHvLR84Yjr08HFixdpaGhgcUUtBVm5IY+5Zap+Hjt37kzoWCTwFiJBMiWgzZRxBnMC7kzo\nLw1w9tw5KMiGqcX09PSk3QfoaMNZV0OhykqA9K4/Hq5/tRdWqslT8Pv9w8FXutmxYwcohbvWHL7M\nNWMeuNy89957aTkxO3DgAHB1YWX+tAUow83+/ftTOayQnOx8bm4RxcWVI66rrNSTnXQr3XAWI86e\nvPCa65zL0q1u2qmHXlI+csyzimeS585Lq8B769atANxRvTjsMQvKpzMpt4idO3bS39+fsLG4xz8k\ndh6PxwX8D+ArQBGwCvgTr9d7KczxNcD/AR4BeoGXgG94vd4e+/rJwA+AjwAKWA983ev1pu85RSHS\nXKYF3oFAYLjfcSZ0COns7KS9rQ2KcqBX18M2NDRQVlaW4pGF1tfXp7tVOCVIFXr1f319PTfffHMK\nRxbecAcFO/A2qqrwH3qfI0eOsGDBgjFumXyNjY3U19djTJ+Dyi8cvlxl5+CqNbl0so5Tp04xZ861\ndaiptHv3bpThwrAz3kZWLvk1CznX8D6XLl2isrJynHtInkuXLtHW1sasWbdc8/etauoCQHHkyBE+\n8YlPpGaAIThnSkMF3nOmLAJ04L1ixYqkjiucoaEhjh45SmX+ZCbnVYy4zmW4WFQ2nz1NB2hqamLK\nlCkpGqU2ODjI5s2byXPncHOVGfY4QxncNW0pb5zcxq5du7jrrrsSMp5EZ7y/DXwZ+BJwD1ADvBzq\nQI/HkwOsBcqAO4HPAI8B/zvosF8Ds4CHgYeAauA3iRm6EBOTjlmrUDJlnI6enp7hMXd1dSV8IcxE\nDWeK+weho3fkZWmovr5ev752pw01RU8Q0rWWt7+/H8uyUGVlw+0PVXU1KMX777+f4tFda9OmTQC4\nzRuuuc5tLh9xTLq4cOECZ86cIb9mkd7wx1Y0R0/EduzYkaqhheQsoptes+ya63JyCpgyZQ4nTpxI\nm3LAgYENpWJUAAAgAElEQVQBjh07RmVxDcV5pddcX1U6g4KcIg4dOpQ2f69PnTpFb18viyaFDmQX\nl6VPKdLu3bvp6OjgnunLyAnqGBPK/TNvQCnF2rVrE/ZaJyzw9ng82cCfA3/n9XrXer3evcDvAnd6\nPJ5QU7bPAVXAJ71e70Gv17sB+AfgVvv+ioAHgP/l9Xr3eb3e/cB3gZs9Hk96po7EdS1TMsnp8oc8\nUsHt+AKBQNpvsz2ibZyh3xNnzpxJ4YjGNnwK3inbyM6C8hJOnTqVlh0sDh06xODgIGrG1YWKKjsH\nNbWK+vr6tCrrGRgYYOvWrai8Alwzrw1YjGmzUUWl7NixI63WL2zbtg2A4nm3j7i8cNaNKHc2W7du\nTau/I3v27AEUM2Z8KOT1tTNvJBAIpE2ZzJEjR/D5fCyoWh7yekMZzK9cRnt7e9pM2ofLTCquzdDD\n1brvZLTnG0sgEGDlypUopXho5k3jHl+RV8ItU00aGhoStjg0kRnvG9DlJRudC7xe72ngNHB3iOMf\nAdZ6vd62oON/4fV6b7W/7QO6gC97PJ5ij8dTiM6knwDSvzGuuO6k0wfRWNI9Yzza6G2q033b6uF6\naZfSGdnc9O64cbVs4+rHg6qajM/nS8ua6d27dwNg1M4ecblRqzsX6CAsPezevZvu7m5c5nKUPbEJ\nppTCvfBGBgYGhoPdVBscHOTdzZtx5eRTOGtkIOvKzqNo9k00NTWlzYYpLS0tHD9+nKlT51MwapMU\nx+xZtwHpk6l33sMLq0NPFIKvS5f38/79+3EpF4vLQme8K/OnUJk/hbq6upRO2A8cOMDZs2e5deoC\nKgtCvx9Ge3SOnmC+8frrCfkcT2TgXWP/f37U5Y3AdK41Hzjj8Xi+4/F46j0ezymPx/OEx+PJBfB6\nvT50rfh96EC7A12+8lGv15u+fa7EdSud268FS8cs5liGu2sYo75PU+fOnQO3MVwGQVkBzc3NCV28\nE6u+vj6OnzgBFaVXxwuoabp+N9XZq9H6+vrYu3cvqrgYVTGyztSYPQuU4r333kvR6K61ceNGANwL\nbgx7jHv+cjAMNm3alBaT93379tHZ0UHRvNsx7IWVwUoW3gPAhg0bkj20kLZv304gEGD+vPD1uSUl\nU5kyZS51dXUpPyPi8/nYu3cvxXmTmFE+L+xxC6qWk+XKZseOHSl/XzQ3N9PQ0MCisvnkufPCHnfj\n5KX09/en7O+G3+/n1VdfRQG/Pe/OiG83q6SK5ZPnYB07lpAFrYlcXJkP+O2AOVg/EKqXSzHwVeBt\n4NPANODHwBR0ZhtgAfA+unZ8CL1w8zcej2eF1+sdc5VVdXV1bM9CiBhVVFRkxPsueIFiJoy3t1fX\nSSs3BAZ08JWu4/b7/VxqaoLSPOgbBECV5hNo7MDv96fduLdt28bQ4CBqehWB41fLYVT1FDAMDh8+\nzNe+9rUUjnCk1atXMzAwgLFk6TWlXSq/ADVtGidPnmRwcJAZQaUoqdDQ0MDx48cxps3GKA6feVP5\nhbhmzudc/VGuXLmS8sWh7777LgCli+8PeX3e1LnklE9nz549uN3ulC6kCwQCbN++HZcrazirHY45\n726amk5w8OBBPvvZzyZphNfasGEDPT093Gs+gKHC50Kz3bksmXYL+xq20tLSwrJl19avJ4szgby1\ncuzSjdsqb+LtM++wf/9+Hn300SSMbKRNmzZx5swZbqtaSE3R5Khu+zvm3RxoPslrr73Ggw8+GNfS\n0UQG3r2A4fF43F6vdzDo8hwgVA8wH9AKfNHr9Q4Buz0eTxbwosfj+UtgEfAdYLrX620E8Hg8jwNn\n0JnwH401mMbGxgk+HSGic+nSJdzuhDYOiovgmsFM+D1xWscpFwTQNcnpOu7W1lZ8AwOokmICffYE\np0RniOrq6sjLC58tSoW1a9cCYMysYig48M5yo6ZN4dSpUxw4cIDJk6P7EEuU119/HQDX/NCnu435\nCxg6d46XXnqJ3/3d303m0K7xm9/oPgDuBdcuqhzNbX6Iofqj/OY3v+HLX/5yoocW1unTpzl06BD5\n0xeTM6kq5DFKKSYtfZCLG5/iueeew+PxJHmUVx09epTGxkbmzb2LnJyCMY+dM3cF23c8y1tvvcXd\nd9+dso3EXnzxRQBumXXvuMfeMvs+9jVs5YUXXqBi1BmeZPH7/ax86y2yXdncUjn2e7m2eAbVBVPZ\ntm0bR48epbi4OEmj1GcS/v3f/x2XMviUeU/Ut59VUsWtVQvZeewIv/nNb7j99tvHv1GQsZIqiXyn\nOdtvjf5treba8hPsy47YQbfDqWyvBW4HLjhBN4DX620HjgFz4zFgIeIp1acDIxXcCzsTymOczWiU\nSzdYSNegG4LKYIqunuRTxTkAabcNu8/nY9++fVCQB5OvXa+uZk0DrtajptrZs2c5ceIEavp0VFFR\nyGOM2lpUbi5btmxJaUmV3+9n2/btumVgiEWVoxk1s1H5hezYuTOl4161ahUAZcs+POZxRXNvw5VX\nxMaNG4fPSKWCk4lduPCBcY/Nyc5nzuw7aG5uTll/7GPHjnHy5EkWVN1ARdHUcY+fVWFSXTqTPXv2\ncPHixSSM8FoHDx6k+fJl7ph685hlJqAnZffX3KXXCdhnTpJlzZo1NDc38+DMG6ksiK3/hmfBvbgN\nF16vN66lgYkMvA8AV4DhaZzH46lFB9GhfgKbgRvsLLdjCbqk5DRwDqj0eDzD57E8Hk8+MBs4Ht+h\nCzFxmRJ4B3cFSfcNafr7+/WW6/a6NNckOH/+fNpOGIbrRwtzrl5YkDPyujTx/vvv09vbi5ozPeRp\nVVU7DQzF9jSpmX7nnXcAcC1cFPYY5XKhzAV0d3endCHd0aNHaW9rw5i9CBXBWTBlGLjmLqG3p2d4\n45pka25uZufOneSU15BfE37TEQDDncWkJQ/S29ubslaIXV1d7Nmzh9KSKqZWzo/oNgsW3AfA5s2b\nEziy8JwzNvcv+FhExyuluG/BxwgEArz11luJHFpYq95+G4APT78vouPvrr6dXHcu69atS9ok8vLl\ny7z++usUZefzifmx9+Kekj+Jj866jba2Nl577bW4jS9hgbfX6+0HfgI84fF4PuLxeG4Engc2eb3e\n9zweT7bH45lqtx0E+Bm69vtXHo9ngcfjeQj4Z+BXXq+3BXgDnUV/wePx3OTxeJYCz6JLWn6VqOch\nxAddcACYbsHgaOfOnSMQCAy3Es4q15naS5dC7smVcs7rqQqCFqXZgXdwW8R0sH37dgCMuaFroVVu\nDqpmKmcbGlLelaWrq4vt27ejiopQ08eu3XYtWgQJ7ss7HmfXPPe8pRHfxj1P1/A6P5dkW7VqFYFA\ngLIbPhpRfWvpkvsxsnJYvXp1SrL0O3fuZHBwENO8L+J63Mop8ygtrWbPnj1JTzocPXqUuro65k5Z\nzMyKyCYKAEtqbqGyuIatW7cOn/1LluPHj2MdO8ay8kVML5oW0W3y3Hk8UHM3HR0dbNmyJcEj1Amv\np59+moGBAT638AEKsiZWzvfxeSuoyCth9erVcWvlmOiipr9HB8fPABvQ9difsq9bAVyw/8fezfIe\n9AY6e4Hn0Jvt/LF9fRdwP9CMXoC5Hr175d1erze9G/kKkcZaWlpCfp2O6uvr9Rd2xjvLLnM8ffp0\nSsYznuGJTEFQxjsvC5RKqzaIXV1d7Nu/HyYVQ/m1G3g41PxagJS3utu0aZNeVLloCWqc2lxVWISq\nraWhoSEl24T39vaye88eVFEpxtTIF3ga5ZWo8koOHDiQ9F71HR0dbN68mayiiuFNcsbjyimgZNG9\ntLe3p+T9sWXLVpRSzIuie4VSCnPePQwODrJz584Ejm6kQCCA16trux9Z+umobmsog4eXfIpAIMBL\nL72UiOGF5WR9Pzbrkahu98iM+8kysnjrzbcYHBwc/wYTsG3bNg4ePMjiilpWTFsy4fvLcWXxlSWP\n4Pf7efLJJ+My/oSu/LIXVf6V/W/0dRvRgXPwZYfR/bzD3d8ZIHUrN4T4AAquNW5qakrhSMbnBN5O\nxjvbLjw7deoUd9xxR4pGFd5wcB1UaqIMBQXZaRV479y5k6HBQYz5tWNmC9XMKsjJZuu2bXzyk5/E\nFaIXdaINDg6ybt06yMrCMMevlwZwLVnKYH09a9aswYzwNvGyc+dOBvr7yVp6e9SdEdzzl+Pbvoat\nW7fy0Y9+NEEjvNaaNWvw+XxMuf0RlBH5z7hs2cO0v7+elStXJnXB4sWLF6mvP8X0mmVhe3eHM3fu\nCnbseoFt27Zz//2hO7fE265du6ivP8XSmluZXjYn6tsvqr6RmeXz2bt3L8eOHWP+/Mgz5rE6fvw4\ndXV1LCqbz/xJ0S2rK80p4f6au1jTsIHNmzcn7HVubW3lueeeI8eVxe8vjexMTSSWTZnD3TXL2Nxw\nkDfffJPHH398QveXmmW8Qoi0EAgERpw+S9eSDcfJkydR2VcD76wKwEjf7cybmpp0D++8UdsUF+XS\n3t7OwMBAagY2yrZt20ApVJgyE4dyuVCza+js6EjYrm7j2blzJ+3t7RimicrJGf8GgKqcipo8hX37\n9iV1UVogENCTBGXgCrFF/Hjc85aBy8369euTto6hp6eH9es34M4rpsSMPHsM4C4opdhcQVNTU1IX\n4ToZ9nlj9O4Op7CwnOrqRZw4cTwpiQefz8dLL72Ey3DxyNLY8ohKKR5drlsgPv/8CwkvoQoEArzy\nyisAfGLOYzHdx6O1HybblcUbr7+RkFKkQCDAz3/+c3p6evjcogeZnB/+zF0sPr/oQcpyi3njjTeu\nnnmNkQTeQlzHOjs7GRwcxElqpXOHkM7OTi5dukR25dXLlFuRVaG3YE+3DWn8fr8O8kryru0xPSmP\nQCCQss4EwZqbm/WEpnoyqmD8ekhj3kwgNbv+BQIB3WlDKVxLIu9jrJTCtWwZgUCANWvWJHCEIx0+\nfJhz587hmrUAoyD6VmoqNw/XvKVcvnw5aTsWbtiwgb6+XkqXfTjkhjnjKbvhEVCKt956Kyk19X6/\nn23btpGVlUttbWRlMaM5m+0ko0Rmw4YNNDc3c/uch6gorBz/BmHMKJ/LsprbqK8/xa5du+I4wmsd\nPnyYo0ePsqx8EfNLo8/Qg856PzT9Ptra24YXRsfTxo0bqaurY9nk2dw3PfJJrj/C92h+Vi5/sPxR\n/H4///Ef/zGhyYME3kIkSCZ0NTl/Xnf2NAzIK0zvwNvp3509qutWdpX+8E237cwbGxsZGBhAVRRe\ne6V92UQzJ/HgBHRqTqgNhUOoLIeCPPbt25fwes3R6urqOHv2LMasWWFbCIajavVtNm/ZkpSa6UAg\nMNy1Imv5ipjvJ2vZ7YDijTfeSHjW2+fzsWbNGozsPEoXjd9XOpTskkqKZt9EQ0MDdXV1cR7htY4c\nOUJLSwuzZ91GljuyMyCjzZ51K253Dps3b0noa9zT08Prr79OblY+Dyz87Qnf3yNLP43LcPHSSy8l\n7HfR7/cP9xr/1LyPT+i+Hq39MPnuPN588016enriMTxAr03yvvAC+Vm5/P6y34qoxOTclWba+q7Q\n2tfJNzf+P85dGb+96+KKWh6aeSONjY3Dv9uxkMBbiARJ1xZ3wZzuFMqA4lK9i2VHR0eKRxXa0aNH\nAR1oB8upGnl9uhgeT+W1AaKqLB55TArt378fADUzsl00lVKomdX09PQMT4aS5W27lZmxLPqyDWUY\nGEuXMejz6fKPBDt06BDHjh3DmDEPY3LozWciYZRW4Jq7mLNnzyY8673FnpSULroXV05+zPdTdoOu\nR1+5cmW8hhaWs1X9ggWx1w1nZeUyd84dtLa28P7778draNd4++236e7u5r4Fj1GQE93EMZTywkpu\nnf0Azc3NCeuTvWPHDs6cOcMdU29mZlGEk/MwCrMKeKz2Ybq7u3nzzTfjNEJ45pln6Ovv53MLH6Qs\nN7LX9V/3vMJQQH9GX+xu5Ud7Xonodp4F91ORV8LbK1fG3N1JAm8hEmRoaGj8g1Ls7Fm9z5VhQJG9\nx0C8WibF2+Ejh1FuRpSagB2IK1K2CUY4+/btA0DNCLF5Q1k+FORw8ODBpGeNg/X29urgefIkVF7u\n+DewqRk6kDx06FCihnaN06dPc/jwYVT1NIwYd840TBOVm8s777xDX19fnEd4ld/v58WXXgIU2bfc\nN+H7y7rpXjCMhGc23161CmW4mbT0oQndV+7kmeRPW8iRI0cSeibq8uXL7N27l/LymVROmdg+eosX\n6U2CnN1b462zs5O1a9dSlFvCirkPx+1+H1j422S5snnjjTfivmZkYGCAl196Gbfh5pNzJ5btdnx4\nxn2U5U5i7dq1VzcYm4BDhw6xf/9+FpbN4O6ayNp1tvd1cbF75OL2C92ttPd1jXvbXHc2X1r8MEN+\nP7/+9a9jOrMtgbcQCZLKHeci5ZQ6GAaU2PFhOrbma2tro/F8I9lVoFwjTyMaOYqsyXqBZSp3zQt2\n+fJlPRGoLEIVXnv6WymFmlNBT0/PcICeCidOnMDv96OmRVdrqqomg6GwLCtBI7uWk+12LV8e830o\ndxbGosX09PQkdCe97du3c7ahAdfcJRjl4+9IOB6jpAz3ghtpamoazvDG265du2huaqLYXIG7YOIL\n08o+9FtAYrPea9asIRAIsCwOHSwqKmqprlpIXV1dQpIPq1evpr+/n/sWfIzsGEtiQnEC+fb29ri/\np9esWUNLawsPz7ifyXnlcbnPbFc2n5r7cQYHByfcDjEQCPDiiy+igM8teiji94DPH3ryGu7y0W6o\nnMvSybM4fPhwTIvMJfAWIkHSbbHfaD09PTQ2NuJ0/Cq1k4gnTpxI3aDCOHjwIAA5NaGvz5muM3bJ\nzMCOxdmsRS0MX2KgFuqAbPXq1SlbD+CUiqipFVHdTmW5oXwS9fX1SenM0tzczK5du1Dl5ahpYd4E\nETIWLwa3m9WrVyfkrFR/fz8vvfwyuNxk3Rq/tmlZN92Dys7htddfj2t9LOgA5s033wSl9OLIOMif\ntoDcybXs2bNneC1JPHV2drJx4yYKC8uZM3v8VqKBwPilf8uX6x0k41kGAbpP/jvvrKcot5RbZ0f2\nnohkvI67zY+S5cpm5cqVcTsj0tnZyVtvvUVRViEfq43sPeGPcMx3TL2ZWcUz2LFjx4TOiBw+fJiG\nhgZuqVrIzJLYF6rG4pPz9RqIWCaWEngLEUfBAVS6ZF/DOXbsGIFAYLijSW4e5Bfpy9OtTMbJCufW\nhr7euXzv3r1JGc9Y2tradFayMAc1L3xJhJqUD7XlnDx5MmUTBmeSpSqjz2apynL8fn9SzpA4mU1j\n2fIJZzZVbh7GfJPW1taEdINYu3Yt7W1tuJfeilFYEtFtIpl4qbwC3DfcSXdXV9y3C9+3bx/nzp2j\naM6tZMcpgFFKUX6Tbj33xhtvxOU+g61atQqfb4Dlyx7D5Qq/JUlr61m6u9vo7m7lee83aG09G/bY\n6TXLqKiYxe7du+M6WdiwYQP9/X3cPf8jZLnG7hRzseMsnb1tdPS28sTbf83FjvDjdRTmFHPb7Ado\na2uL206nb731Fn19ffz27I+SP87uj+e6Gmnrb6e1v52/2fqPnOsae5G+oQw+M+8TABPKem/atAmA\nj86+Neb7iNXs0ioWlM3g8OHDUZfMSOAtRBwFZ/+SvQVxtJxgL3h/jIoq6OvrS6sOIT09PRyqO4S7\nHNwloYOurApwFcH+A/tTXuLz0ksv4fP5UDfNQLnG/hNr3Kz7Zv/6179Oeq33cCeYScWonOjbxjnB\neqJ7qPf09PDu5s2oggKM2bPjcp+upboW1Ano46Wrq4u33lqJys0j64bxe2D7W5sIdF+B7k56X/i/\n+FvH7iPtXnIrqqCINWvXXt0VdYICgYC9I6Gi/KZH43KfjoKZy8kpn87OnTvj2jGps7OT9evXk58/\niQXmfWMeu2bdDwkEdCKho+MCa9f9MOyxSilu+tDjBAKBuE0WnA2fcrLyuHX2A+Me/8y2f8Vvj/dy\n10We3f6jiB7nrvkfwVCuuLyn29raWL9+PRV55dxfM35v9B8d+PerCxV7mvjxgf8Y9zYLy+aztFyv\nA4hlkXl/fz/79++nqqCc2SWxL16eiDtr9M6Y0e56KoG3EHF05cqV4a+Tvc1zNPx+P/v27SUrG4I3\nH5xin8VPVs/gSOzatYuhwSHyxlg7pZQibw709fYNd+lIhSNHjuhewBUFKHP8zKGqKEQtnMqFCxdY\nvXp1EkZ41dmzZ+nv70dNia120wm8E93ZZMuWLQz092MsWhzVLopjUcUlqJkzqa+vj2tLx9WrV9PX\n14v7hjtR2eMvVu1f+yLYAUugo5X+tWNn/5Q7i6yb7mXQ54tbOcTevXtpaGigaO6t5EyKrLNNpGUQ\nSinKb/54UHAfH0699A3LH8M9Rq/xnp52OjoujLisveMCPT3tYW8zc+ZNlJfPZOfOnVy4cCHscZHa\nvXs3nZ2d3DLrPnLHyRxf6WvnctfI3v7NVy5wpS/8eB2l+eUsqbmZc+fOTXjtxerVqxkcHORjtY/g\nNsbe4Ly9v4OLPSMnjBd6LtHeP353rE/M0RO9WN7L9fX1+Hw+lk+ZE7cdKqO1fLLuaX7s2LGobieB\ntxBxFNyKL13b8oEOllpb26gc1R2qogrc2bBz5460aYe4efNmAPLnjX1c3vyRxydbf38/Tz31FCgw\n7pmnt4aPgLqtFvKyefW11+LyQR8pJ8ukqqKr73aownwozMeyrIS9VwKBABs3bgTDwDAXRHqjiA5z\nLVwMELfFit3d3axdt06XhCwafyOXQE8XgY6RnRUCHS0EesburOCavwxVPIlN77474ay33+/n1Vdf\nBaWouPlj4x7f33qewe42hrrbOPXr/0p/6/jlGIW1N5BTMYNdu3bF3H4tWHd3t53tLmHhgrEzyEND\noc9+hbscnKz3J67WvU+QUw5xewTZbl+YcYW7fLTb5zw04jFj0d/fz7vvvktJTjF3Vo9fwjGRhYpz\nSmaxYNI86urqov7b5yyAnV2ammw3QGluIWW5xVEvxpXAW4g4am29+kHa3j5+liJVnNXv00aduXe5\noGomtLW1p8VCxYaGBk6ePEnOdHAVjh3IZpUpsip1CU0ytn4e7eWXX6apqQm1bBpqSuQ9elVuFsZd\nsxn0+XjyySeTNuFxNjdR1VNivg81rZKenp6E1XmfOnWKCxcuYNTOQuWNnS30t7ZCdzd0dzPgfV5/\nPwZVU4MqKmLXrl1xWQi9ceNG+vv6cC+7A+XOGvf4wFDowCTc5Q5luMhavoIhu4RhIvbv38+5c+co\nnnsb2aXjd185v/onwxl6X8clGtf8dNzbKKWosLPe8Qhk9c6afSxd8ltjZrsnorb2JiaVTuO9996j\npaUl5vu5fPkyR48eZfbkhVQUTby7zXhmVZhUFE5lz549Ma8xcm57b/UKsozx38cT9UDN3UD0CRMn\nsVWWG/2OsPFUlltEZ0dHVH+3JfAWIo6CA+949ChNhM7OTnbu3ElBMYTqdDbTzhyvX78+uQMLwemp\nW7AksuMLFussaSK2JB7LsWPHWLtuHZTmoW6ZGfXt1ZzJqDkVnDx5MmF9hIP19vbqdodlJTpzHSM1\nQ7+BEtUS0dmW3pg/f9xjB9etuZrt7uhgcN3Yr6NSCjV33nCt6EQMDQ2x7p13UFnZuBd+aEL3FQnX\nvGWo3Hw2btw4oa4yTkcGp/XfWAZ7OvB1XBpx2UD7RQZ7xj+zF1zrPZG/i069dHZ2HosWjp9BjpVS\nBsvt7cEn8vvoLN69YUbsO5dGQynFDTNX4PP5Yv6ddMa8oio5CxY/NHkZue5cdu7cGVVtutMAwG3E\nHsZmZ2dTUVFBdnbsEziXYTAUZbJEAm8h4qi5+eq2s5cvX07LbeM3bdrE4OAgM00IVRpXUq5bCx48\neJCLFy9ee0CSdHR0sP297bhLIGdGZLfJmwOufHj33U1xb7kWjs/n4+c//zkEAhj3zUe5Y6tDVnfN\ngdys4cx5IjnbvRuzpk3oflTNVHC72BHlh2YkAoEAu/fsgewc1LSxxxno6YHRpV0d7fryMTiLNSe6\npuHgwYO0t7XpgDiC2u6JUm43rgUfoqenh927d8d0H6dPn+bkyZMUzFhGTtn4td2BwdDlDuEuD6aU\nYtKyDxMIBCY0od+1axcdHR0sMO8jOzv2CWMk5s65k7y8YjZv3hzzGZG9e/ehlGLxtJviPLrwlk67\nBYhtMuzz+Th8+DDVBVOpKkhOe75sVxbLyhfR0tISVblJnn0GrGsgtsx+dnY2jz/+OE8++SSPP/54\nzMF3j6+P3JwcjCgmABJ4CxFHTsDkMnStXPBiy3Tg8/l45511uLOgZk7442Yt1IFPMrKv4axZs4ah\nwSEKlhHx4hnlUuQvhb6+/oRtNDLaqlWruHTpEmpJNWpq7Kc9VV426s7Z+Hw+nnvuuTiO8FrOaV01\nd5wZzTjBtMpyo2qn0dzUFPdFlo2NjbS1tmJMnz7+ospw7S/HaYupJpVBYSF1dXUTKvFxWri5zei3\nso+V29QbCW3bti2m22/duhWA0sX3xm1MYymaewuunHy2b98e82vt1C4vsneZTCSXy80C8356enpi\nmpj19vZy6tRJppfNicv28JGaUjyN0vwKjhw5EvXrfPr0aQYGBlhcZiZodKEtLtPrN6JZFFpdrSeL\nZ680j3NkaMXFxXz+858nNzeXz3/+8xQXR/+3e9A/RGNXC9XjJAZGk8BbiDhyAm9n8nvp0qUxjk4+\nnTHqZPo8yBpjgj91BuQV6I4SqWiL2NPTw/r16zHyID/Kz4CCRaCyYM3aNQlvLdjR0aHrVvOyYyox\nGU3NnQzVJRw4cGC4Bjvezp8/z9GjR1HVk1EloQOCQGsHdPdCdy+DL7ytvw/DWKCzxvEu73E+hFX1\nxLLyY1FKYUyroaenJ+bdCn0+HwcPHkQVl6EqEl/H6zBKylEVUzl69GjUZ3cCgQD79u3DyMmnoGZx\ngsA06cwAACAASURBVEY4kuHKonDWjbS3t8e0JuDy5ctYlkV11UJKipOTjXVaFW7ZsiXq2546dQq/\n38/syREuCo4TpRSzJy+gu7s76gWLThvZuaXxadsZqXn240XTmnS2fbaq7vLpmB6zs7OTZ599lr6+\nPp599tmYupAdbW1gKOAfHkukJPAWIk4GBwdpbm7GMK4G3qks1RgtEAiwZs0alILacT4LDANmmrov\neSK31g7HWUBVsAyUO7pWUUaOomAxdHZ0xvSBGY033niDgYEB1C0zUDljt92KhFIKY4X+I/7yyy8n\npFRp1apV+rGWhG8TM7R2W1C9dBdDa8fYlKOqAspL2L1794hSq4lyPoSNysQGWcq+/1h719fX19Pf\n349revLbmrlmzGNoaCjqsw2tra20tLSQX22ixth8Jt7yaxYB0WU2HU7t8dy54/dHj5fi4ilUTpnH\n0aNHow7Mzp7VG99Ul9YmYGRjq55UO2IMkTpz5gwAtUUR1vbFSVVBJdmubBrsx49EeXk5tTNncqTl\nDJ390SeHBgYGePXVV/nqV7/Kq6++GtNaiR2NRwC48cYbo7qdBN5CxElzczN+v39E4J3M9nDjqa+v\n58yZM0ypgfzC8Y+fMU9vrrNx44akthYcGBhg9erVGNk6ex2LgmWgXHrxWKJ24ezq6tKTkqLciHp2\nR0pVFMLscurr6yfcj3e05uZmXZpQWoSaGbquN9DTBx2jWtp1XNGXhxqvUhjLF+D3++O61fbZs2fB\n7YbS0rjdZyiqQrdTjLXVnROwG1Onj3Nk/LnsfqDRThqcXRlzKpIbYOXajxfLZjoHDhwAFLUzk1cv\nDTBr1i0EAgHef//9qG7nLCItL0zuVuYAFfZjRtuR5dy5c2S7spmSH1uL0VgZymBaQRWNFy5EtZHY\nXXffzVDAz4aG2BZHDwwMcPny5ZiC7q6BXrY31jG5ogLTjO60rATeIq11dnby1FNPxTWTlijOh5mh\nrgbe8dytbaKc+siZ4zeIACArB6proampOaadxWK1ZcsWrly5Qv5inb2OhStfkWfqD79EbAsOuk7a\n5/OhllaPu0PlsAgz2MYyvZPRRNvFjfbaa6/pyeGNi8JnZ2Ool1azp0NpEVu2bInLWR6/38/FixdR\nJaUJzyKrEh3Yx/q76pSTGZOSG6wAKPsxoy1pc1qdZhWUxX1MY3EX6seLtv94f38/x4+fYHLFLPLy\nkts+bnrNMgAOHz4c1e26uvTktTAn+vFOtNtGfnbhiDFEwvmdq8qvxFDJDw2nFUxlaGgoqq43K1as\nIC83j3Vn9jAQYa/zeHnnzF4GhgZ54MEHo1pYCRJ4izT39ttvs2nTJp5//vlUD2VcTsbMMEABebmx\nZ9Hirb+/nx07dpBXABWRbU4HwHS7GiFZ5SZ+v59Vq1ahXFCwdGL3VXgDoGDl2ysT0nFjy9YtYCjU\n/PH7YAdau6F7ALoHGPr1bv39WCqLoCyf/fv3R/XhOZazZ8/qbHdZiQ6U40gZCuPmxfj9fl555ZUJ\n319HR4ee1MSw4Clayu2GggKaY2xz5yygVnkRnEYKYSJBlvOY0S7idjJ8RlZO1I85EcqVBaioM4wN\nDQ34/UNMnRph1iCOJk2aRnZ2flT1x8Dw3xwVZRAbj24bTuAczZnK1tZWfD4fU/Nj7+s/EVPtLirR\nTNzz8vJ44MEH6OjvZtPZA4ka2jX6BwdYfXoXBfn53Htv9IuTJfAWac3JjDjZ5HTmLM5ykp/lpfpU\nXyoWJ462e/du+vv7qZkTuoVgOJMmQ0Ex7N27Jynt+fbs2UNzczN583XWeiLcxYrc2XC24WzcFype\nvHiRxvONMKMMlTv+JhP+1UeCaqZ78a85MubxSinUvCkMDQ1NuMe0w+v1EggEMG5dGvGumtFQs2pg\nchm7du2KOkgZbbgffmFswWy0VEEh7W1tMZVUDZcyuaJvIznhIMvOtEVbTuWyxxrwJ6YMK6yAHwgM\nP36knL//5eWxlcZMaHKjDMrLZtDU1BTVYm2n3V2fL7q/m/HottHn6x0xhkg4Z5Ur8ydH/Xgw8Sz9\nlDx99ibaVqoPP/ww2VnZvHVyB4NJej9vaNhP10AvD334w1G9xg4JvEVau5o1SO6ipWgFAgHq6+vJ\nz7sa2FbYZ3Hr6+tTNzCbU2YyVgvBUJSC6XPB5xscbpmWKIFAgLfffhuAwuXxuc9Cu7ubs6AwXpxg\nWM0qH/fYQM8AdIzqNdveqy8fg3Pf8Qi8Dx06xKFDh1DTpqCmJ6bzhlIK1+36tPzzzz8/obMMTimE\nyi+Iy9jGVVCA3++Pqf2nE2gEfNHXiU44yLJ7aEcb7BTaE5qhvuS2Ox3q6xrx+JFygsLi4ujfu/HI\nIBcXVxIIBKIqgyi11ya090RXZx2PbhvOY5ZGsT7CqQcvz42+/Cger3FFnn7c1nF2nB2tuLiYe+69\nh9a+TnZeSHxJ5JDfz6r6XeTk5PDQQw/FdB8SeAsRBy0tLbS3tzMlKA5zvj5x4kRqBmU7e/Ysx48f\np6IK8mNoJ1szB5Shd7JM5IZAx48fp76+ntxacJeOP9GKZCjZkxXZ1Xp79FjbxYUyvN369EnjHzwY\nJosa7nKbKs2HolwOx9CPN5jf7+fFF18EwLhteUInsapqMqq2mhMnTkxoN8vhYCM/+mxSLJzt6DtG\nb8ITASe4CXRHH8RONMgKdOvjS0pKorpdWZkOcnxXot8OfSKZTd+VyyMeP1LO65KfF93zhPhkkPPz\n9c84monZNLu388WO6DqLxKPbhvOYTq/rSDjPrSSGmvR4vMYl2fo2sUw0HnroIZRSrDs9sY2wIrG/\n6QStfZ3ceeedUU8gHRJ4i7TmBAnpuANkMGfx4dSgs3SV9lqreHemiJaT7a1dGNvtc/L0IsvGxkYO\nHjwYv4GN4mS7C8bJdvtaAwz9f/bePDiO88zTfLJO3IX7JkDwAC+Q4AmekkhRFEWJkinRKh2U1Fbb\nbrt7trvd27M7u7PTM7MxHR27PR0b45iN3p7d8HjDY63bZdmiLbdtWRJ1keIBkuABHjgIgLjv+yjU\nkd/+kZWFAgigKutClbqeCAaARFbmx0JW5vu93+/9vVMgT0HfTwTO4eWvDTV7/sEHH4RjmLhcLhob\nGyErBSkl+FbDgSCVWJiZntZsC+bL9evXaW9vR1pXhpQbWYcQAN2erSBJvPfee0FPGLy66aToBt7B\nZLwLPHaEYlS7RjzUIEv2nLOwUFsmWN3fMaLNdSnUzKZjpHfe+QPFbrd7zq/9eghHBtloVLqRaulg\nWVFRAcDDIe2NpUJx2wBoH2pGr9NTXh54b4GZGWVVLsWwMu9xsue86t9aCwUFBWzevJnm0S76p7UV\n7mrly656gKC03SqJwDtBXBDrUhO14r3YxzkqySyRm6VkcoO5mYSDnp4eLl68SFom5IfQh2SNp8fG\ne++9F5FJUG9vLzdv3sSYDyY/z+ThDwDPENxjMPL75fc3l4EhEy5fvqzZTWEx2tvblcK/oii4KxQp\nGb5gu0IKIbwWf7qdQXozakTKykBau4rOzs6gJ2reADhJW/Ff0NnYJCWwCqaQVQ1u5P7gXFFCCbLU\nc2oJsABSU1PJyspidkjbhC7UzKZ6vlWrtBX3qhM4SdKuow9HBlnn6ZyqRUufk5NDdnY2rQP3kEX0\n7Fjtzmk6R1ooX12O2Rz452fuPdb+rA3LexxEQagve/fuBaC2J3KJLofbyY3+BxQXFWm+hn1JBN4J\nYppYz3SDcqO4ffsWKcmQvWAltLRIuVlrtaIKF2pB3Ybt2ooqF5KRBUWrlQYLly5dCtv4VD788EOE\nEKT5aQ/vnha4F6gBXKPK9qWQJInUbcrf4dy5cyGP1dt1ryAKjhv5ijYo2DqB5uZmHj58iFRRgpQZ\nvbbVuu1Kh6Zg7RDVAFjyBMSBEFI21hOgBBN4l5WVYTSacPcE3vwjXMg9D9HpdJo754ESrLumx3BN\nBy6vCTWzaR9sR5IkzUGLWowpB1k8F2oGWZZd88YRCJIksXXrVqYdU3QMRU9u2NRXjyxkqqqqNL1O\n/bw43MG9R6G+x7NuZTXBaPRfrL4Y27Yp9SX1g5GrqWoY7sApu9hWHZpkLxF4J0gQIo2NjUxOTlFW\n/GjQWO7JMl+/fj3q47px4wY3btwguwAKwuAet3GH0lDnpz/9aVgdTqampjh//jz6NEjyEz+IJXor\nLLVdJaUSdElKR0wty8WLoXZ3k3Kj4LiRmQxGvfecWlE7d0qbNVbVhoiUbYGCHO7cuaO5WAqYcwIy\nBx54h5KNlTznCcaByGg0snHjBsTIAPKkdo14sAj7DHJ/NxUVFaSkpGh+vZolnx0MPOsdSmZTCJnZ\nwXYKCws1ZWIBkjwTMIcj8s5Ki+FwzMwbR6Bs365Ud9/pjrz2WOVOl3KuHTt2aHqdWicwMjsa9jEF\nwuis8tnRUhDqi8VioaSkhKaRroi5mzQMK5+VTZuC1G16SATeCeKCWM58qw1aKhYJbvOyITUF6uqu\na7KiCpWZmRl+9KMfodNBVU1o2W6VlHRYt1UpQLPZbKEf0MPnn3+Ow+EgtYqI2NyB0nY+ZbMSWIWa\nse/q6lK6JGVGXn8sSRJkpdCjsaMbKCsxdXV1kJyEVBR9b17dOsX6LRhXlqmpKcUqzxB4O/OQsrGe\nQDBY6081wHK3Ra+ew93eCEL2nlsrZWXK38euUW4SbGbTNTGM7LR7z6sFdRI1MxO9iY0v057zpqdr\nWzXavHkzZnMS9Z1Xo/IMc7md3OupIyc7R7P8qKioCIDOyZXptqyeVx1HMKxfvx6H20n7uDZLwkBp\nGu5EkiTWrVsX0nESgXeCBCHgcrmorb1CkhmKF4ltJElizSqYnp7R3HI4FP7xH/+RkZER1m6F9ACM\nNwJl7RbleJ999llYvLFlWebcuXNIBkgJLYngl9TNgAQff/xx0A9BIQTdPT2QkRR4t8oQkTKTkWVZ\nk5UZKPr+iYkJpNKCoCY0ofryqraFwXQ9nZ6eBpNJ03JuKNlYyaz8H4MNvHfu3IkkSbgfhNcvfjlc\nnnPt2bMnqNerjhdaCyyDZXa0Z955tZCTo1hEjU9o69AZLibG+5EkyTuOQDGZTGzfXs3wVD+dI5G3\nlW3ovcWsc4Y9NXs0SyHKysrQ6/Q0jqyMC1eD57yrV68O+hjr1ysd3xpHlp9MGnWLT+iX2g7gkt08\nGOumpKQkqBUmXxKBd4KYJhQbtWhQX1/PxMQka8tAt0Rws86TePjyyy+jNqbPP/+cjCxYF6DML9A4\nVKeH6gOKveB//a8/8FbCB8vdu3cZHBwkeV3w7eEDRZ8mkbRasVdsaWkJ6hhTU1PMTE+DJTpuG4D3\nXFrbgnslMfkr48tLeiokmec08RqYmZmBIM4ZtM7UqJwr2CLozMxMNm/ejNzXiTyq3aJPK/LUBHJn\nCxUVFV5XFa2oQaRrUrsUKBhcHuvCvDztDVpKS0sBGBoKnyVooAghMzTcTkFBQVD643379gFQ9/BC\nuIf2CDfaL8w7pxaSkpJYt34drePtjNijKzeRhczNoXrSUtNCCrwrK5XOpvcGl5fmZSalUZg6/75Y\nlJpNZtLS8sEHo9043C7vOUIhEXgniGmi0S0xFFQN7fqKpffJyYIsi7LkHozNkhbsdjs//OEPkXRQ\nfVAJlJdjYgTs08q/T88qP/vDkgNrq2B4eIR33303pPFeuKA8KFI2hnSYgFGz6sFOgtSss5SuTesZ\nEhlJ884dKN5APYiiynD48kqSBJY0BgcHNctkZh0OpZV7tPAEVKHo/w8dOgSA637w/uWB4m64AULw\n2GOPBX0Mo9FIamoqrijJN1wzyr0vmGuprKwMg8FATxQapCxkaLgdh2OatWuDq5OoqqoiPT2dmx0X\nccnaPgdamHZMcre7juLi4qDkPAA1NTUIBF/2XAnz6Jbn9tA9xmbH2b1nNzpd8GFpbm4uhYWF3Bl6\niMO9vLTzz3a9hN7jpFKUms2f7npp2f1v9CsZ+a1btwY9PpVE4J0gplFdBuz20DKrkWBsbIy6ujqy\nMyF3GTmHJElsWKO4akQ66/3rX/+a4eFh1myBjAASndc+m8t2T40rPwfCuq2QZlGKFYMt/HM6ndTV\n1aHPAGNwSTvNmEtBl6zo8oNZTVG7u5GurTgsFKQ05VxaixTVZjBSEE1owuHLCyClJiOE0OwW4na5\n/M8aw4nnYa91guDLrl27SE1Lw914AxHCcfwhZBnX/TpMZnNQmU1fkpOTkR3RsTqVPQWKwSzTm81m\nKisrGRxqY2JiINxDW5Y2T1OWLVu2BPV6g8HA/v37mZqd4F535Irsrz+8gFt2cejQoaAdN/bt24fJ\nZOLjzs+j1n4d4MP2T4DQvLFVdu7cicPt5Hrf8haspel5ZCWlk52Uwf9++DuUpi+9EiMLwaXuuySZ\nzSEXVkIi8E4Q40xNKQ/sycmpmCuw/OKLL5BlmY1r/Xufrl8Neh18+uknEZPPjIyM8MEHvyMpFdYH\nMCm3zyjBti9T48p2f+j1sGWPonkOttDy/v37zM7OklQRPZ92SSeRVK74RAdj0ef1AU+NXuCNJ/D2\nBv0B4pUBmbQvj4fDl9f33FolSdH37VfuLaFk24xGI088/jjCPoO7JXJab3d7I2JyjIMHDpCcHJrk\nyWg0IvxkBsOFcCuTEUOQKxmqT3ND4+dhG5M/ZFmmsekLTCaTZpcQX9SA8vKD0O1MF0MIwZUH59Dr\n9Rw8eDDo46SkpPD4448zZB/hQs/lgF4TjF7alwdjbdweuseGDRtCkpmoHDhwAIBP228GtL8ugHvN\nncFWhmbG2b1nj2ZHnkXPGfIREiSIEEIIJsaVRhputztkPXE4cblcnDv3MQaDElT7I8kssaYM+vr6\nqa+vj8iYPvjgA1wuN+u3gT6Ae95SCY1AEx25xZBbpOi0g9FM37t3D1Cy0NFEPZ96fi2oWWc1C62F\noIsVPd0xtTb/8U7wggwmQ/Xl9T23lsYj4PHyjWDW+BFc7rnzhsCRI0eQJAnXndpwjGpRXHeuAnD0\n6NGQj6XX6yFKzV2E5zzBBt579uwhJSWFO3c/xOkKzRI0UFpbrzAxMcD+/fs1Wwn6UlxczMaNG2nu\nv0P/eNeS+xn1i19/S21XeTBwl/6Jbnbv3h2UlMeXZ599FqPRyNmWfwrI0zvTbKEwZb6zQFFKAZlm\nyxKvmEMIwc+afgnAqVOnghvwAkpKSti4cSN3h9roCJO7yQetyuc5HJ85SATeCWIYu92Ow8eCL9L6\naC3U1tYyMjLKhgowGQPLzlVtUL6Gq3W5L4qM5QKmJCjV3ksjaNZ4miGqWnctNDc3g+S/U2W4MRX7\nnF8jXrlHqrbgOZRiRUmvgxST5sDb2+xjJQuU5eCCrbS0NEQ05WWeSX1aWmje7Lm5uVRXVyMP9OAO\nspPlcsijg8hdrWzYsIGSkhBa0XowGAzeTHSkUTPrWprQ+JKcnMzRo0ex2ye4fes3Ab1Gv0TAutR2\nX9xuF7VXf4ZOp+OZZ57RNNbFUIO2L5s/XHKf9KRMctPm3xDz0otIT1re2/rLJqV971NPPRXiKCEr\nK4tjx44xbB/lN22BNcD60+pvz+mlUwr476q/FdDrrg/c4t5II9u2bWPjxvAV+qh/r/cfhC7tbB3r\n4dZACxsqK8OSkYdE4J0ghlE1qkv9vFIIIfjNb36DJM0F04GQmyVRnK9kiIPtRLgULS0tTExMUlgW\nXWlsThGYkuDGDW0FZbIs8/DhQwxZoAtw4hIu9CkS+jSCctvwBr8p2gLvkIsVU00Mj4xoklt5ZQiO\n6PnHP8Ksc/5YAiQvLw/sdkSIzY4CRYwr95bc3NyQj/Xkk08C4Lq/vJ5XWmJZaqntyjEVT/QjR44E\nObr5JCUlIbscUQm+VS15KJnjEydOkJ6ezo2b7zMx6b/YOCUlE4tlvi90pqWIlBT/TVrq73zA2Hgv\nhw8fprAw9OzAjh07yMnO4VrbF0w7lratfOPAn6GTlJt4XnoRZ/b/6bLHHZrs5153HRUVFUEXgC7k\n5MmTWDIs/Lrt9/RP+3+fS9OKyTJnkm3O5H87+G8pTfNvGTnrnuWdhnfR6/W8+uqr4Ri2l23btlFW\nVsbl7nt0TWgrSl/I2SbFAOD5F14Ix9CAROCdIIYZHR1d9ueVoq6ujs7OTtaUQUaatqBxuydD/Ktf\n/SqsY1ID+ewoFSmq6HSQlQsjI6OaJkaqhMGozRY3bBiylYncxMSEpteNjo5Cikmzh3fIxYqpZtwu\nl6YiRbUTnZhaOWcgMTWNTqfTnElW7eOERieXYFHPo543FLZs2UJWdjbygzsI19KTHiklDckyvwJa\nsuQgpSz+XglZxt10i9TUVHbu3BnyOGHuGtHSNj5YXFMj6PX6kFYVkpOTsVqtOF2znD//w4Amok8/\n9edInkA201LEsaf+3O9rxsb7uHrtXdLS0njxxReDHq8ver2eo08dxel2cLX10yX3K7SsIiM5C0ty\nNn/5zN9SaFm+7fDF5g8RCI4dOxa22ojk5GRefe1VnLKT/3b/pwFP+HVS4PfFsy2/Zcg+zPHjx0Nq\nmrMYkiRx6tQpBPBe0xdBH+fBaDd1fU2sX7+ezZs3h218EfVrslqteuCvgW8A6cDvgH9hs9kWNaS1\nWq2lwH8CjgMzwLvAv7TZbNNWq/UbwA+XONUPbTbbH4Z39AlWGjW7aNCBS9bu6hAJZFnm7NmzAOwI\n4nNYXAAFuYq1YGtrKxUVy/gQakCdlCzxzI4oyR63upGREe+D3B89PUozDUNw3YExmUxkZGQwPj4e\nlAbZkAWz7dDb26upG93Y2BikBV+seO7cOe+YtYTuUooRgRLABzpeb5ZuZBxKo6znwdNtdnSCvLw8\nzfKCDRs28Lvf/Q65uxNdGCQV/pC7FN1tWDx6dTr279vHb37zG9ztTRjWLH2jMB97GfvP/x8QMpIl\nB/Oxry89xu42xMwUNUeOhKxFV1HlKvbBdozpy8+CJcPi51xquy9CduMY7qK4qCikAlaAgwcPcvHi\nRe7evUFD42ds3HB42f2zs1eRmpqFEDKvWP/O7/FlWebTz/4LLpeD11//RsjyI18ef/xxzr53lksP\nPuZQ5YllA1UpgCB21mXnatvnWCyWoBspLcXevXv54osvuHX3Llf6rrO3cFfYjt0+0cXvHn5Mbm4u\nL4Qxk+zL9u3bqaio4ErrfTrG+1mVob1773uNStD+4osvhrXgO9IZ738P/AHwFvA4UAr8fLEdrVar\nGfgQyAYOAq8AJ4G/9ezyU6Bowb//BZhGCdYTfMVQXRwMnmd2LATetbW1dHR0sG41ZFm0fxAlSWKX\nx3HkF7/4RdjGpbajD1I+GRLqOZ3OwCUNqse0IbA4fR7haO6inre/P/DiG5fLpfg8JwWXrwipWNGs\nBDdaOiuuWaOI/UVvdLLGjzAyDg5nUMvfmzZtwmQyIVpaIu5mJKanED3drF27NuTCNBXVgcPdurzv\ntC47Hyk1HVIzSH7lT9BlLx0cuFuVYuCampqwjBHmOv1Nd/h3YTGkWDBa5i+pmTILMaT4/xDP9DQh\nuxze84WCJEn84R/+IUlJyXx58b8xHmABXSCBLMDt+t/S29vA7t27Q7ZrXEhqair7D+xneGqAxt5b\nIR/vZvtF7M5pDh8+HHTR6lJIksRbb72F0WjkncZ3mXaGp+ZCFjL/772fIAuZN998MywuIYuhZr0B\nzjZpr0F6MNLFrYEWNm7cGBYLQV8iFnhbrVYT8OfAv7bZbB/abLbrwKvAQavVemCRl7yOEkyfttls\nt2w22yfAvwNqAGw224zNZutV/wEpKIH3X9psttCv4AQxx8CA4tdq0M3/eaVwu928994v0EmwKzhL\nVwBKChStd319PQ0NDWEZm3rzWmZlO2Ko59Si3VQnVXrtvV3C0txFPa8Wiz7vxMKwArMbz4dAy+Qm\nLy+PvLw8REcfwhU9T14V0aZkkYPxPzabzezatQsxPo7oXtoFwstSM84AZqKyp6W9akMWDlatWkVO\nTg5yxwNEAMWt/rJpQgjc7U2kpqaGJSuvsn79etLT05louYrs8j8hLDn+J0rbWpSgu/jpPw7oPGON\nSpHbrl3hyZrm5OTwxhtncDrtfPLZP3gdU0JlaLidK7U2MjIyeOuttyJia3n48GEALrd8EvKxrrR8\ngiRJPP744yEfazEKCgo4efIkY7PjvNfyT2E55hfdl3gw1sqePXvYtm1bWI65FFu3bqWiooKrvQ2a\ntd6/enARgK997WthH1ckM97bUeQln6obbDZbG9AGLNZu6zjwoc1mG/HZ/4c2m22p6f3fAreB/zs8\nw00Qa6hZUb0Okk2KLGAluXDhAn19/WxYCxnpod2Q91QrX3/+85+HJaOXna1oRWcCT4iGjRmP7Dgr\na5kuQgtQ9eD6VO3nC0dzF33K/HEEQiz4yGsZgyRJSubV5UI0R7fVtpBl5PutGI1Gtm/fHtQxVIcG\n903/frxSSgoslDlZMpXty43T6US+e4fk5GT2798f1DgXHY8ksW3bNoTDjtwfwMTBD2JkADE1wdat\nW0OWavii1+t5/PHHcdsnGWvw39LcnF2CITULfWoWFa/+NeZs/zIg19QoE01XKCgoCGvm8MCBA+za\ntYve3gZu14fuFOWWXXz66X9Bll28/fbbmiRoWli9ejVlZWU09Nxgwh68tr53rIPOkVa2bdvmvf9H\nghMnTlBQUMBHHZ/RORmaU8+0c4afNf+SJLOZ1157LUwjXBpJknjuuecQzFkCBkLP5BB1fU2sXbuW\nDRs0OCgESCQDb7VKZeFdpxtYrFqgEnhotVr/g9VqbbVarS1Wq/XvrFbrI2k0q9VaDZwG/mebzbaC\nXlkJIoUQgu7ubnQSIEFuusTAwICmjF84cblcvP/+r9DrgtN2LyQ/R6KsGJqamrh7927Ix1P1vBMr\nUH86OabYsGnRQtrtisOBpF0lEpbmLjrP6ub0dOCFh15JizP62WNc8vwxBMiRI0fQ6XTINxsC5PZL\nogAAIABJREFUyrwCIWWPVURzO0xOc+jQoaA6FYIildm0aROiqxO5x/8D3/DU06BmKC2ZGJ465vc1\n8p07iJkZnnrqqZCb0SxEzfS7O7V73C9EPUaw3ROX49ixY5hMJoav/RrZGZiLTKCyDYDBq79CyC6e\nffbZsE4aVClEWloatVd/FnJHy9u3fsvgUBuHDh0KerIYKI899hiykLnZfjHoY1x/qMgnDh06FK5h\nLYrRaOTVV19FFjL/2PheSMd6v+0DJhyTPHfypKZETSjs2LGDvLw8LnTVM+EI7H7/YZvilf/MM89E\nZNUjkoF3CiDbbLaFkdIssNiadAbwTWAt8DLwFyg678Uy2t8DLnnkKAm+gqiOE+p9uiBDQpZlurpC\nzx4Fw5UrVxgcHGLDWkhNCc8HcWeV8vXXv/51yMcqKysDYDwIGXzQjV0ApwOmJ6G8vFz7iQGCfCvD\n0txFIwaDQZlcTEXvnF6mlIAoM1NbNWp2drbSNW9sAnH3QUCvkVKSwLJgEmVJV7YHgHC6kGvrMRgM\nPPvss5rGu5DTp08D4L58yW+2X5edDampkJqKyfqK8vNy45yZQb5ZR0pKSlh8mheyceNGJElC7m4L\n+VhuzzHC6aygYrFYeOaZZ3BNjzF0PTxyAhX7QBtj976gqKgopI6KS5GRkcFrr72GyzXLhS9/FPRx\nJiYGuFb3C9LTM8JubbcYNTU16HQ66tqD85mWhcyN9oukpKRQXV0d5tE9SnV1NZs2beL20F3ujyzf\nin0pRuyjfNj+KdnZ2Tz99NNhHuHS6HQ6nnzySZyyi4td/msZZt1OLnTdISsrK6RupcsRSVeTGUBn\ntVoNNpvN1yTUDCy2IO4EhoE3bTabG7hqtVqNwM+sVutf2Gy2IQBPBvzrwJ9pGUxxsX9fyQSxQ3u7\nsjSuurYVZUnQqrh3RPtvKYTgk08+QZJgW/g8/snLligpEN7W6aE6nBQWFjI81IsQc4k/f6iFimfO\nnOGdd97xOLYEHliOeWRzVVVVmv4uarZDzALhMWjQhOz5L+bk5Gga96pVq7jXcB/hciNFUestRqYx\nGAxs2bJFs6PFd7/7Xa7U1jJdW49YXYKU5j8DrT92APfPPwQhwJKO/ljgMgy59jZMzWB9/fWQNZzF\nxcUcPnyYTz/9FLmpEX1lAMu+AV787qu1CIeDb3zrW6xbty6kcS7FunXraHrQgnA6kIxBLO+gOIKI\nnoeUlJRQVVUV5hEqfPOb3+TixUsM3vyAjPV7A5KQ+EPIMn2f/xgQfO9732PVquVt8YLl9OnTXL58\nmVu3rtPZeZvS0q2aj3G59qe4XA6+850/C0sBqD+Ki4vZtWsXtbW1DE70kpuuzXWobbCB8ZkRnnnm\nmeCTHhr5zne+w/e+9z1+2fJbNu7S/h795uFHOGUnb775Ztga0QTKSy+9xM/f/Tmfd97m6Yrl3V+u\n9TYw45rl1DMvReyajWTg3eH5WuTzPUAxj8pP8Gyze4JuFXUNfjWgVkEdBUyApjWP7u7wdxFLEDlq\naxU9llpYWZKlPEyvXbsW8WXAhbS1tdHc3Ex5CaSnBvZQD1SKu3k9dPXBz372M954440QRqkEhb29\nvcxMBW4ruLBQ8dy5c0DgRSijnk9lXl6eps+YKj9wjYF+BSwQXR5pZVJSkqZxl5SUKK3m+yehOAhL\nliAQTjcMTVG2uiLoAuNXX3mFH/zgB8ifXEH33BNIuuWvYynbAqnJIASGVwLPBsudfYj6ZgoLCzl8\n+HBY7rsnT57kyy+/xHnlMrry1UhhcEGQBwaQ79+juLiYHTt2ROz5sH79epqampB7HqIvCy6gk/u7\nEE4HGzZsiOhz7PXXX+P73/8+vZ/9iLJT/0qTnGQxRm5/hH2gjf3795Ofnx/RsZ8+fZpbt25xufan\nlJRUaZIHDA618eDBRVavXs2mTZuiFitUV1dTW1vLzY5LHN2srV36zfZLgFI8GK3xWiwWtmzZwp07\nd2gZe8gaS+AB/6Rzik+7LpCVlUVVVdWKxGNbqrZw8+ZNuieHKE5b2jrzYrcSdob63i6X0Imk1OQm\nMAE8oW6wWq2rUYLozxfZ/wtguyfLrVIFuFEKMlUeA67bbLbY6KaSICI0NjYiMZfxzrcojmpNTcEt\nc4XCpUvKTW5DAK3Yh0cFUzMwNQO2fxIMjy4fgZcVQ5IZrly5jNsdmnZYzSKMBW7UEXKh4tjw/HMH\nirq/Y1FH/8jj9JxX6yqDmnEUDzW8yaHSPgyyCCnbefDgQXbu3InoGUDcXN7ibh4aAhgxY0d8WotO\np+OP/uiPwuY1nZOTw/PPP4+YmcF9/WrIxxNC4P5S0ce+8cYbYbdh82XrViX76u4ITOazGOprI6Hv\n9mX79u3s2bMHe98DRu98FtKxnOODDNWeJTUtLSrSjbKyMmpqahgcbOVhu7YuuteuKzm806dPh1WD\n7o+dO3diMBi42XFJ0+vcsov6rloyMjLC2mY9EE6cOAHA79vPaXrdZ50XcLgdHD9+PKKft+VQbTiv\n9i7tJDbjmuXOYBurSkvD3tTHl4hdZTabbRb4e+DvrFbrM1ardSfwj8BnNpvtktVqNVmt1kKP7SDA\nP6Bov39ktVo3Wq3Wp4D/CPxIlZl42IHiZpLgK4rD4aCl5QGFmXPPfZ0kUZYj0dfXN9e2O0rU1dVh\nNATWg+SjC3PZ7rEJ5efl0OkkVpfCxMQkLS2hFWGpS45adN6hFiqODyvetFqr6jdt2oROp8Pepull\nYUEIgb1NKRrSKjHYvHkzycnJiKYBhDvAYkXDErfZpbYvQG5QZgmhWLFJksTbb79NVlYW8tU7iJ7w\nWnMKIZA/uYKYnuH06dNhawylcvz4cfLz85Hv3EEO0c9fbmxA9PdTU1MTdn/ehaxbt46k5GTcDxuD\ndsVxP2xEbzBERN+9kDNnzpCcksLglV+E1M2y78JPkF0OXn/ttbB5o/vj+eefB+DmzfcDfs3IaDdt\nbdeoqFgT8YnNQpKTk6murqZ/vIvesQ7/L/DwoP8uU7MTXp14NNm8eTPFRUXU9t9g3BFY119ZyJzr\nOo/JZOKxxxYztIsO1dXV6HU6rvc1LrnP7YFWXLKbHWHqDLsUkf6r/RvgHeDHwCfAQxR9NsABoMfz\nFU83y8dRGuhcB/4/lGY7C41Ci1C04Am+ojQ1NeFyuVmdN//yrMhTovBwuIAEyuDgIP39/RQXgF6/\nfPZvekYwtuBeNDahbF+OVZ6JdX19fShD9ba71upsEmyhotsF0xPKebVWfqelpbF582ac/eAcjq5N\nn6MXXKNKhk9r8waj0ag8PKYdiAeBSXKkFBNYFjhmZCYr2/0gRqehfYS1a9d6C2iDJS0tje9+97vo\nJAn53GXETGAOFoEgbtxHdPaxdevWiBQqGo1GxX5MCNyXgneCEE4n8tVajEYjVqs1jCNcHIPBQPW2\nbYjJMcSgdjtUeWwIMdxP1ZYtmnzyg8VisfDy17+O7Jhh4NK7QR1j8uEtph7eZOPGjWG1aPRHaWkp\n27Zto7evkf6BwFYY6ut/BwhOnIiMe4U/1CzsrY7LAb/mpmffcDZSChRJkjh85Agu2cX57sDGfGe4\ngcGZIfbu3Ru0w1E4SE1NZd369bSO9izpbnJ7QEl+RbpgNaI5f09R5V96/i383acs8DSw2Wx3Ufy8\nlztmZB3XE6w4agC6Nl/irk81wNoCHdyWqa+vj0iF/GI0Niqz46IAus0upRTxpyBRjx2qjCYjI4PU\n1FQmR6Nj5j3pSYiVBNnS+8iRI9TX1zN1CzIPh29c/pjytNt68skng3r9sWPH+Pjjj3FffYhYm4uk\n95+/0B3fhPyzOmU5JDMZ3dOBZVrFVaXIOFzBbGVlJS+99BLvvvsu8qdX0D1zKOSAQ/QOIl+9Q2ZW\nJt/+9rcjloWrrq72akzlzk50paX+X7QA+dZNxPQ0z37ta+TkLN8iPVzU1NRw+fJlXM23MeVpW752\nNdV7jxEtnnjiCT799FPaGy+SVXWUpPzVAb9WyG4GLtqQJIk33ngj6sHssWPHuHXrFnfufEj+4eU7\nps46pmlsOk92dg47I5zhXIrq6mpMJhO3Oi5xbMtpv++XS3Zxt+sqWVlZQXWEDQf79+/HZrPxRfdF\nTpQf9TvmL7qUifITTzyx7H7RoKqqioaGBu4OPmRv8fx7sBCC+sFWUlNTI178Gd11igQJAuD27VsY\n9FCeN/8DnZ8BaUlKYC4H6kkcIqr8Iz+Cz2izSSIzA1pbW0P6f0mSRHFxMVOT/oP9cKAG3sFq4bZv\n305hYSEzjeAaj07W2zkssLcq2u5gGyPk5uYqD5FxO+JOT0CvkbJTIdUEqSb0r+5WfvaD6B1HNA9Q\nXl4eto5/oOg0t2zZgujoRdxpDulYwuHEfe4yEvDH3/3jiDUdAeX6fvnllwFw117WLN0Q9hnct2+R\nnp4ekaz8Umzbto20tDTcTbcRGj6YQgjcjTcxmc0RszVbDJ1OxyuvvALAQK023+bxxos4Rnt54okn\ngp6Qh8LmzZspKCjgQcsl7PblpRBNTedxuWY5cuQweg0e9eHEbDazbds2Bif76Bvv9Lv/g/67zDin\n2b17d9RlJippaWlKQfJULy3jD5fdd8o5zbWBmxQVFbFmTQBFUhFGlZbdHWp75Hf90yMMzYx7ZZCR\nJBF4J4gpBgcH6erqpiJPwrhA2iFJEusLJSYnJ2ltbY3KeB4+fIgkQY42+2TN5GYpTWX6+/tDOk5J\nSQkImApenhkwqqSlNIjMIygP+Oeffx4hw8S1wF4jLbFGt9T2hUx4mpe98MILIWXjTp06RUpKCqL2\nIWJSg2QjwHMKt4z8hRIUnzlzJqyZQ51Ox7e//W3S0tKQL99GjGjv/KkiX6iDyWmef/75sLYyX4ry\n8nL27NmDGBxEtC//0F+I+/YtcDo5efJkVGQbKgaDgYMHDyLs07hb7wX8OrmjGTE5xt6amrA39/HH\n5s2b2bRpE9Mdd5jpC6z2RMhuhq7/EwaDgRdeeCHCI1wcnU7H4cOHcbudNDUvXWAjhOD+/U/Q6fQr\nqjuGudqNO13+b4J3PfuEcyIeDOqK8wU/cpMrfddwyS4OHjy4IlKehaxevZoks5n7Q4928r3n2Rbp\nug9IBN4JYowbN24AUFm0+Ie0slC5ZOvqtFWuB4Msy7S3t5OVAQZDZG8auZ7axIcPtQUTC1F9R8ej\nUH+qFnEGG3gD7Nu3j+KSYmYawTniP4OpT5HQL3DxM2Qq2/3h6FOy3WvXrg1Zw5eenq5kX51u5PPN\nYW8nL252wtAUjz32WER8hS0WC2+//Ta43bg/q0XI2scvP+xGND1k9erV3sK2aKAGde6bNwJ+jXA4\nkO/eJT09fUWWvI8cOYIkSbjqrwT8Gqdn36NHj0ZqWMty8uRJAIZvBtaOfaLlOs7xAQ4dOhS1roSL\nceDAAXQ6PQ2Ni5mnKQwNPWRouJ3t26uxWKJjC7oU27ZtQ6/Tc7f7+rL7CSG411NHWlpaxHznA6Wq\nqgqLxcKlvqs45aW7SZ/vvowkSVHV+i+HXq9nfWUlPVPDjNon5/3u/rASeEeiRfxCEoF3gpji2jVl\nRr+haPFLc22BhFGv7BfuYGchvb29OBwOcqLwDMn1nCPUwFt1NhkN3Io7KIRQrARzcnI0tYpfiE6n\n4/RLp0HMZaP9kX0cb3WIIROyAmyCNu6Jeb7+9a+HJfvy+OOPK3ZebcOI5vC5hIjhKcS1djIsGd4l\n/0iwc+dO9u3bB/3DmiUnwulCnL+OXq/nW9/6VlQtwkpLS6murkb09SH3BeZHKTfcB4eDY8eOaS6o\nDQcFBQVUV1cj93fh7vXvYCEP9yF3tlBZWRm1BikL2bRpE2VlZUy21uGc9O9nMHpHsZg7fnzZMq2I\nk5GRwbZtW73B9WI0Nil2ktGqFVqOlJQU1leup2uklQn70pXxPWPtjM+MsHXr1hWTxqjo9Xr27dvH\nlHOaGwOLmwL0TvXRPNbK5s2bNbteRRI1sG4cmf85bBzuJC0tLaI2giqJwDtBzDA2NkZDQwOrciQy\nkhcPjEwGiXWFEr29vXR2+tfEhUJbWxswl42OJKqUJdTAu6ysDINBz0h43eIeYXoCHHbCknnZsWMH\nFRUV2FvAOeh/MmXMltCngi4V8l+VMGb7D6JnuwWOLsULOVzetzqdjrfffhuT2YQ4/wARhlbyQhbI\nnzSCW/D2N94mNdW/FjwUXnvtNVJSUpCv1iOm7QG/Tr52FzE1w3PPPbciWl615bR8138LaCEE8t07\nGAwGDh8+HOGRLY2qK3fd9N8m3HlTKUhTfZNXAkmSlAJkITN2//yy+86OdDPT08SWLVsoLNTWhTES\n7Nu3D4CWlkelEELItLReIiUlJeTOquFCHUdT39LOVo29iouy6g2/0hw6dAiACz2Ly03O9yiZjliY\n3PiiriA2Ds/FD8Mz4wzOjLFu3bqoaOcTgXeCmOHyZaVgqqp0+UCqqlS5bC9eDN5WLBDUIDg3Chlv\nk0kiI00J9kPJ5BuNRioq1jAxAs7Q48AlGfIkGsMhg5AkiRdffBEIXOutvC7wfSc8fVfU84SL/Px8\nXrG+ArMu5C+aQl6FETc6YWCSAwcORKVDa0ZGBqdPnwanC/lqYHaWYnwSUd9ETk4Ozz33XIRHuDib\nNm2isLAQubUFMbu8xl70dCPGx6mpqYlo8ac/KisrWbt2Le6HjcjLzIzlyTHcD+5QXFy84oHh3r17\nMZvNjDdcQIilC7/H7it66scffzxaQ1uW7du3YzQaaWt7tOHSwGArU1Mj7NixY8WauSxE9RBv6l36\nM9jUd3vevitNaWkp5eXl3By884intyxkLvRcJjkpecX16AupqKjAYDDQODIXeDeNKPZpkZD1LUYi\n8E4QEwgh+OKLL9BJsKV0+cuyskgiyQhffnkh5G6Py9HerixTZke4sFIlJwump6cZDrFByKZNmxAC\nhrTbBgfMkMfMI1zZ46qqKtasWYO9Nfy+3o4+gaNbeWBFwoLr8OHDXskJLcFrfMTojFdi8tprr4Vx\nhMvzxBNPUFxcjGhoQ4z6b4ohX70DsszLL7+MyeTfizwSSJKkZNzcbuQHy3s2yx5L0JUOCiVJ4tln\nnwXAeXvpojTX7Ssgy5w4cWLFnCtUkpKS2L17N86JIWZ6FpcjCVlmovkyySkpUXVfWQ6z2cymTZsY\nHulkYnL+Z7K9XakNiLRXsxZKS0vJyMigqb9+0cm7wzVL22AjZWVlUWtIFAgHDx5EFjIXe+brBO+P\nNDFsH2FPzZ4Vu0cshdFopLy8nI7xfu973TyqBN7R0s4nAu8EMcGDBw/o7OxkQ7FEWtLyqUyjXmJb\nmY6xsfGIFll2dnaSkQYmY3SqsdUAP1QJjZoRGQzM6U4zQijHzsrKpLi4OCzHlCTJmz2dvBmWQ3qZ\n9FwiarFYuNHpdPzBH/wBBoMB+UILwuHSfAwhBPL5ZnDLvHHmjZB081rR6/W89NJLIATy9eWbU4nR\nCURzO6vKytizZ0+URrg4Bw4cQJIk5AdL+98Llwu5rZWcnJyouK74Y/v27eTn5+NuuoWYedRvXzhm\ncTfUYbFYvHKJlebAgQMAjDcvPlmY7mnANTVKzZ49GI3GaA5tWdT7YHf3/Gu6u+cekiRFpRNooOh0\nOrZs2cKkfWxRW8G2wQbcsitmst0qe/fuRafTcbF3fuCtBuLqtRNrVFRU4BYyLqEk7lpGu9HpdFGr\np0gE3gligo8++giAPWsCuyR3e/ZTXxdupqammJycJDOKyYUsz7l6e0NLVa9Zs4akpCQGusMwqEUY\nGwLHLGzZUhVWi6jt27dTWFSIvQnck+HJertGlfbwa9asiWi1emFhoTJxmHYg6oKYOD0chs5Rqqqq\n2L17d/gH6IedO3dSUlKCeNCBmFi6AZN8qwGAF55/fsWzsVlZWVRWViJ6exHTi3eiE12d4HRSU1MT\nE3ZmOp1OcSlxu3E1POrK4mq+jXDM8uSTT8aMDGLjxo1kWCxMtlxDyI+uME40K0FWrEwUVNSJVl/f\n3MTMLbvo72+mpKQ04vUTWqmqqgLmtNy+qNtiLfDOyMhgy5YttI634/ZIkVyyi6v9N8jKyoqadEMr\nFRUVALhkNwLBw/E+SoqLo1Z4nQi8E6w4Q0ND1NbWkp8hedvC+yM/Q2JNvkRDQ4O3CDKcDA4qy5Pp\nQdybTSYTubm5mpfY0tPmnztYDAYDW7ZsYXoCJpexaNYtURi/1HaVAU830XDrT3U6HSeeOYGQYfLR\nZ09QTHpimxMnTkQ88Dpx4gRZWVmIW12avL2FLJAvtaLT6XjttddWJECUJEkp/hMC+d7ivs1i1oFo\nbic3N3fFOv0tRB2HvISnt+y5N8SSzvTQoUMYjUZcDTcekRW4799Ap9OtuCzGF51Ox+5du3DbJ5nu\nuj/vd0J2M9l6nYyMjJhYUfCltLQUo9HIwMDc9Tw60oXb7WTt2pVv5rIQNahu6H10ya+h9yYmkynm\n3mPAmyiYdStFRfdGmph2zaxokx9/lJWVAUrgLcsyDreLsii6B8Xmu5LgnxUffPABsiyzf71OU9Bx\noFK5fH/729+GfUzj40rEmqyxz4bJZOLUqVP84Ac/4NSpU5qCb/Vc6rlDQQ2K+5dJviYlQ+qCjH5q\nhrJ9Ofq75pZGw83+/fvJzMpk+i64Z0LLersmBDONipVbNAJFs9nMqVOnwC0j6vxbxqmIpn4YneGx\nxx4Lm3QnGGpqakhOSUE0ti3q6y0edIDLzeHDh2PmgarqdOWOR23jhBCIzg7S09O9Ga5YICUlhV27\ndiHGhpF9lqXk0UHkwR6qqqrIzIxSYUmAqMHVZOt8ad9MbzNu+yS7du2KmWtCxWAwUFJSwrBPEd3g\nkDJBUwOvWMJisVBevpq2wYZ5E7KhyX4GJnrYvHlzTEl5VNTPoFNWAu+bHnvBaBSHB0thYSF6vR63\nLOPyZOpD6Uehldj6pCT4Z8fExASfffYZ6cmwtUxbpm9tvkShBWpra+kL0M83UGY9Tgla73MZGRmc\nOXOGpKQkzpw5o6kQxmiYf+5QUG+GfX7iv11PzLmDpGYoPy+HfUbxCK+srCQlJSXkcS7EaDQqWW/n\nXLY6WCavg5Dh+SjKIg4ePEhefh7ifh9i2r+tjBACUdfh7eK5kphMJmr27IFpO6LnUdcNubkdSZJi\nSrdZUFBAbm4uorsHIS9w3RgdRUxPs3nz5pgLCmtqagBwt81lkN1tioxn7969KzKm5VA+76lMPpyf\njZ1sUz6ksVJUuZDS0lJk2YXskciMeILwaAZZWqiu3oZbduOS5+4d93uU93ilHW6WwmKxUFZWhlN2\nIRDcG2nEZDLFrMwElElZQUEBbuHG7bk2opn0iK27UYJ/dnz00Uc4HA4Ortdh0GkLvCVJ4tBGPUKI\nsGe91YyD1kX/8fFx3nnnHex2O++8846m7LUaAMsLA4ggsFgsrFmzhpF+mF3Gnjk9C5JSlH+HTyk/\nL0efJ7EYyQftkSNHyMrKYrpeyVoHg3NEMH0fCosKo6o91ev1nHjmhJL1vh2AyP7hMIzOsH//fnJy\nciI/QD+oBZOirWvedjEzC72DrFu3bkW7Ei7Gpk2bwDGLGJnvBiT3dM/9PsZQs5fu9jmnEHd7E5Ik\nxWSApdfr2bZtK66pkXk676mOekwmU9jcjcKNGkzJslLwPDLaPW97rOHNHrvmAu+GGA+8Ya4pjdPt\nonOymzVr1sRkdt6XgoICBOD0XM8FBQVRO3ci8E6wYtjtdj766CNSTBI7K4K7FDeXSGSnSVy4cJ6R\nkfD1SVdvGi6NboUOh4OzZ8/yzW9+k7Nnz+JwBG6m7fKYYYTLfmnPnj0IAb0B9OQJVOHT3aZ8jaRm\n1mg08tJLLyHcMLG069qSCCEY/xIQ8PLXX456l7eDBw+SmpqKuNeLcC0/iZLrlUBgpbv9qWzYsAGz\n2YzonF/gK7qUFaVYsmBTUS3AxIJVL/XnWMy8mc1m1q9fjxjuV2yCEMj9XZSXl0fV0UYLqrRMeLS8\nzslhHCM9bNy4MWaDLLULoZrxHhvtIS0tLWbf49WrV5OWloZTfY/dDloG71NaWhoTE/OlUKVcdreS\n5VmzJvY09AvJz88HwCm7kCQpqu9vIvBOsGJ8/vnnTE9PU7NOwmQIrqBMJ0kcrNThcrn58MMPwzY2\ntdGGPQjVh8PhYHBwUFPQ7XuucD0UVCeHzuVtjgNmehKG+5TgLNI3qQMHDlBRUcFMM8x2act629tg\ntkPJdK7EErjJZFKK4+xOROvShbJibAY6R6msrGTVqlVRHOHSGAwGJXs5NukJCBVEdz9ATFmwqaje\n7GKgf952eWCA5OTkmOikuBjqhEC4XQi3G2Q5JovnVNSVA+FyAjDTrfijx2q2G/D+7ZXAWzA+0R+z\n2W6Yq50RCGTZRdtgIy63M+bcTBaidq91yM55P8cyvs+wzMzMqLoIJQLvBCuC2+3m9x98gFEPNQFa\nCC7FtjKJtCT49JNPmJmZCcv41OKmycVdyiKCeq5wLeVnZ2dTVVXF6CCMh9aTB4B25TnrbRUcSXQ6\nHW+99RaSJDH2OQhXYMG37BCMn1eWxt98880Vs5BTXSnE/aWtIcV9JSO7km3MF8ObIXbPZetF3xAm\nszkmi9KKioowmc0IHzcg4XDA2Cjl5eUxp+9WWb16tfKN7Fb++W6LQXJycsjOzka4laW5mX7FLSSW\nJwt5eXno9Xpkj3uFEHLMTsRU1AmOU3bSOtAwb1usomaPl/o5FvF9zkZbPhebd6QEX3muX7/O0PAw\n28t1pJhDC46MeomatTpm7Ha++OKLsIwvKysLg8HAuP9GfmFDPVc4b1pPPvkkAK33QjuO2wUdTZCa\nmuotDIs0q1ev5ujRo7jGYCLAPkkTteCegueee867zLwSFBYWKgFs1xhi4lGRvRAC0dRPUlJSTFnd\nwdyyMZ6usMLlgtEJKlavjrpsJxB0Oh2rSksRo6PebareOxYnCireAj9ZVv4R+5lCRUL6tuDtAAAg\nAElEQVQgQMjYB9rQ6fQx/R7r9Xry8ws8gbcyYVjJ+0IgqNIpt9tFx7BSAxCJjrvhJCkpaV6SI9bq\nQBbDYrEs+n00SATeCVYEtfHN3nXhuQR3Vegw6ODcuXOLttzVik6no6ioiNFxkBexVosEw2PK13A+\nfLdt20ZBQQFdrbBIo7yAaW9SmuYcPXo0qi2AX3rpJTIzLUzVKc1wlsM5KJiqh/yC/Ih1qdSC6v4h\nHiwiN+kZh8lZ9uyJvZbK3kBKve6Hx0GImJHDLEZpaem8AFYMD89tj1Gys7OV5W2fcUezwCsY1GtA\nuF04hjopLi6KWX23SlFRISBwuxUZRKxnvNWJgVt20TPWQW5ubsxq0n3xDbxjqa39Uvi+p9F+fxOB\nd4Ko09PTQ2NjIxV5Ernp4ZECpJolNpdK9PX1cf/+ff8vCIBVq1bhcsP4ZFgO55fh0bmAP1zodDqe\nffZZhAwP6oM7htsNLXcU7fLRo0fDNrZASE5O5vXXzyBkGPty6f2EEIydBwS89eZbMREMqA0kxINH\nrfnUbbFoHZeWlqbUOKhB7JiyFBPLmUKvblcd8+jI/O0xiE6nIzs7W/G8lGXS0zOi1jkvWNSkgOx2\nIrscMZ+hh7lAW/KEO7EeeOt0OsVjWriZtI/F9OfOFzV41ev0MXH/9Ydv59JodzFNBN4Joo4qB9kV\noLY70AT2Lo8zyvnz54Ma10LKPZ2sBsOgj/aHLAuGR5UHW7hvWgcOHCA3N4eO5uCy3h1NYJ9WbP5W\nIpOxe/duNm3axGw7zHYufjHYW8HRq3QyjJVCpLS0NKXwbGByXidLIQSibYiUlJSItrEPhfz8/LkP\nnmfmGcu6TW9w4g28R+dvj1GysrKU91nIZGXFVtOcxVCvAeGxu4vla0JFHaPDOY0kSeTm5q7wiPzj\nK+mKh/HC3GqN0RT7QTcoSZ3Fvo8GicA7QVSRZZlLly6SZIQNRctnu/vGBBMzMD4D//kDJ31jy0fg\nZTkSWalw7drVsDShUbWu/VEIvEfGFOvCSHTYMxgMPP/8C8huaNbYit3tVjLlZrOJZ599NuxjCwRJ\nkrBarQCMX+ERKZEQgolaJVP08ssvr8QQl0R1VREPh+Y2Dk3BlIPq6uqoVtJrwVvxLwRicmb+thjE\nm8X0ZunHyMjIiEiTp3DiO5GNh+V5bxDo6fYXD0Gh7xjV2p1Yx1dyFGtdTJdCnSzodbFXB7IYvgmu\npCSNLapDJBF4J4gqzc3NjIyMsqlEwqhfPvC2XXJ5ZaZDk8rPyyFJElWrdMzOOrh161bIY1UdEQaG\n/O8bKn2ec0TK//TAgQPk5+fRqTHr3d6oZLuPHn1qRQOD1atXs3PnTpz94FjQl8beBq4R5f8Ya8vI\natML0THnMS/aR+b9LhbxFkfJAqaVwDuWA4CcnBzlwa82n5qcjHm9NMzZlkL0dabBkJycHHdFdNnZ\n2d7v42G8MP/eEG0ZRLDEqntQIES7ziZ+36kEccm1a9cA2Fyy/KU3YRcMLdBWD00q25dDPa56nlAw\nmUyUl5czOAKuAO3sgqXPU3+nVrSHG2/WW1b02oEge7TdZrOJZ555JiLj0oI6hqkF45+qn//7WCI/\nP5+8/HzoHvNuE12KDCIWPbFVvAGhEIgZO0ajMerLsVrQ6XRKRl4IZbIgBHl5eSs9LL/4BlXxEHjD\n/CK6WJ6Mqayke0Ww+GZgY133r+K9LlbGwTUkor0Kkgi8E0QNIQR1dXWYDVCRt/ync6mOkf46SRZa\nIDMFbt28icu1fIY8ENavX48sQ38Es95CCHoHlIdwJDWp+/btIycnhw6PQ4k/utuUbPfhw0fmZeZW\ninXr1lFSUoK9DdRpkHtS4Oia+10ssmnjRnC457KxfeOUlpbGtLTAN/DG7iA9hseqkpub6wm840cG\n4Rt4x7osRsX3uo2HyYJvEBsv2WNfGUQ8FCqCT8Y7OiZgYSXa73Ei8E4QNbq7uxkYGGBtgYTBj8wk\nWCRJYkOR4und0NAQ8vHUxgVdfX52DIGJSZicUjrARXK5zmAwcOzYMdxupWDSH633lPfz2LFjERuT\nFiRJYt++fSCD8MypZlqVr/v371+5gflhriGNJyh0yTHZxtwXb0AlBMw6SIuDgMUrKchQJg3xICvw\nXUWIl8Dbt8lPPATevvfUWF618cW3uDIWvfMXw/s+x2HGO9oymUTgnSBq3LhxA4ANRZG97Co9RZs3\nb94M+VgbNmzAYNDT0eN/36Xuj/7um+2eY0fDjePQoUMYjQY6mpd3ixkbVrpdbt++PaaK6qqrq5Vv\nPIH3bMeC7THIXEMaWQm+ie0OheCTGRQCnK64yBSqsgfJnDTv51jGN9iOl6DQd5zxko1ViTXP/KXw\nDQTjRTsdLxOExYj22OPjL5ogbPzkJz/h448/XpFz19XVIUmwrjCyU+LyPAmzEerqrofcTCc5OZmN\nGzcxNALjk8sfKyVZwrJAkWFJV7YvR5sneNy+fXsoQw2I1NRUduzYydQ4jC0jn+lWukFHpT28FkpL\nS0lPT0d4JEeOXsUBIJYmBwspLCxUAhRZeJvSxHK3P1gQeBMfS/SqBEJtnhPLUh4V38A7XjLe0XaA\nCCfx4GgC83X0icA78iQy3gkihizL/P73v+fHP/5x1M89MjJCS0sLZTkSqSG2iPeHQSexvkBicHCI\n9vb2kI+ntkhvbvO/71MHQb1nWtKVn5djckrQMwCVlZVRWxrfs2cPAL0di/9eCOV3ZrOZrVu3RmVM\ngSJJktI+WYBwg3DEfjtlvV6vOGzIc/rjWPeX9gbanvHGQ+DtlT1MKVXZ8RB4x6PUJF6yxosRj4G3\n7/exTDxfF4nAO0HEcLv9VCZGkGvXriGEYHNJdG4imzzuJlevXg35WLt378ZsNtHQ4r99fHamRGoy\npCaD9TmJ7Mzl/7/3HyhfDx70E6GHka1bt6LX6xnsXvz30xPKv6qqqphcSva2rXbN/zmW8TYacQss\nFkvMOxXMabyVL3EVeHuIhzH7Bt7xkkmOxXtCoMRL9jhegm1f4jnwTkhNEkQMp9O5Yue+fPkyEnMB\ncaRZXyhhMsCVK5fDIjc5cOAgk9PQ1hXYawK5b7pcgnsPlExXNFuHm81m1q1bx9gQOB2P/n6oV/ka\nq3Z3arZYDbxjzbt7MXy9hGNZFqNiNBoxmc1eqUk8FNH5ZowlSYqLQHYlu+cFS0JSEHniMeOtjjPU\n5+1KEO33OD6uwgRhYaUC76GhIZqbmynPk8jwo3cOFyaDRGWRRH//AK2trSEf7+mnn0aSJG7cDd+N\n5X4L2GfhySefjHoGVHXVGBl49HfqtsrKyiiOKHC8/syeP0M8tK2Otw6FAOk+wXYs2En6Y2GhYjwE\nLL6f+3iYKED8yDUWI14mDfFYXBkPn7elSEhNEkSMcLRRD4YrV64AUFUa3cttq+d8ly9fDvlYhYWF\n1NTUMDQCDwPMei+HyyW4eU9pTvP000+HfkCNqLroxQosRweVgKC4uDjKowoM3+wxxIdtnG/GOB4k\nEDB/zPGW8Y5HvXSsy49U4iUQXIxE4B154jEAT0hNEkQMu93u/V5Wm3lEgdraWnQSUdN3q6wtkEgy\nKucPR5b6hRdeQJIkrt72r/X2x50mpRP3U08dW5FsompxtzDwdjlhckyxu4vVG75vxthkNsXFEr1v\nNjMexgvzJwjxMFmIx/fY9zMWLxrZWL0vBEIi8I488Sg1SWS8E0SM6elp7/e+QXgkGRkZobW1lfJc\niZQg3ExMJhO5ublBPZQMekVuMjIywsOHDzW/fiHFxcUcPHiQkbHAHE6WYtYhuHEPUlKSOXHiRMjj\nCgaLxYLFksH48PztE0o385i2uzMYDN5AMCM9PmQbvtnMeMlsxlsG2WQyeR+g8RJ4+xKPQWG8ES/v\ncTw20FGJx4x3QuOdIGJMTU15v5+cnIzKOe/cuQPMNbXRgslk4tSpU/zgBz/g1KlTQQXflYXKJV5f\nX6/5tYtx6tQpjEYDV+sVuUgw3LgLDgecPPn8imYSy8rKmZma30hHDcRjOfCGuQxsPGiPYX42M15c\nIeKt8E+SJO8DNB7Gu5B4CVjiOfCOF+I58E7gn8Qn6J8RvsH2xMREVM7Z2NgIQEW+9kstIyODM2fO\nkJSUxJkzZ4IqSludJ80bR6jk5OTw1FPHmJpW5CJamZwS3GmE7OwsnnrqqbCMKVhUGz5f1VG8BN5q\n1jheAqx4DLx9pRvxUvinEi/XRTwSLxOExYgXGUQi8P5qkwi8/xlx/vx57/djY2NROWd7ezsGPeQF\nkZgcHx/nnXfewW6388477zA+Pq75GGlJEpYUwtJIR+XkyZOkpKRw454iG9HCtXqlc/iLL7604gGY\nGlwL38B7RLnRx2ph5ULiRRfr6wQRL2OOR3mMSjwF3kVFRXHR3j5B9PC9X8SLi4w6IYu3SfpKEB9/\n0QRhoaNjrlXh6OhoVM45ODhIZgroddqzJA6Hg7Nnz3Lu3DnGx8dxOBwEc8lmp0q0DozhdDrDEuym\npKRw8uRJbDYbN+9BTXVgrxsZEzS1QUlJMQcOHAh5HKGiBt5qxluWlcC7tKQk7m72sY7vdbfSE65A\n8Q224y3rFk8P/7/6q7+KarF7gtjH9/MWL/fiw4cPc+XKFV5//fWVHkrME9G/qNVq1QN/DXwDSAd+\nB/wLm83Wt8T+pcB/Ao4DM8C7wL+02WzTnt9LwP8EfBfIBa4Bf2az2W5E8v/xVcH35j40tIiPXASY\nnp4mOzv44MjhcDA4OBjSGJI9CcaZmZmwBT1PPvkkH3zwAXeaxti6QZCc5P//eL1e0VO/9NLpmNBJ\nFhQUYDabPBMamBoH2R37MpN4xDfLHS8Z73iZICxGvEzIIL6y8xBf7+1C4mXs8ZjxLikp4fvf//5K\nDyMuiPTT/98DfwC8BTwOlAI/X2xHq9VqBj4EsoGDwCvASeBvfXb7t8C/Av4c2Al0Ab+xWq3xUWG1\nwvi2jB8YWKRzSpgRQiBJEistq4vE+c1mMydPnsTlglv3/e8/PCZo6YDy8nJ27NgR/gEFgU6no7R0\nFUIovWjiRd/tS7xoNuMx8I6XcSZIECjxcr+Ix8A7QeBELPC2Wq0mlAD5X9tstg9tNtt14FXgoNVq\nXWyd/XWgCDhts9lu2Wy2T4B/B9R4jpcG/I/Af2+z2c7abLYG4DvALEoQnmAZ3G43sixjkCRMeh19\nfYsuOoQVSZJIT09j0r6yN7vJWWUs4XYQeeKJJ8jMtHC32b/W++Zd5evXvva1mMq6+Oq8J0aUbWrR\nZYLw4SvbiJeANp4z3gkS+JKbmwvEjwQpHqVp8YjVaqWgoICSkpKonjeSU6ntKPKST9UNNputzWq1\ntgGPAV8u2P848KHNZhvx2f+HwA89Px4CklDkJ+rvx4GKCIz9K0d/fz8Aeh3kp5ro7elBluWISx5K\nSkq5e/cuMw5Bsin6AacsBP1jgqKi4rDrVI1GI08/fRybzcbdJtixZfH9JqcED9qVpbjq6gAF4VGi\ntLQU8Oi7R+dvSxA+4rFQMR4f+LE0qU0QO/zJn/wJX375Jdu3b1/poQSEb5Y73uor4okTJ06sSC+N\nSAbe6tN7YYPtbmCxlFolcM5qtf4H4A2U1e9fAP/GZrPZPb8fAPZ69qkA6lAy4HcjMP6vFF1dyp9B\nr5MoSTfTMT5Gf38/hYWFET1vZWUld+/epblPsHVV9B+K7YOCWRds2LAhIsc/fPgwv/zlL7nXPEv1\nJoFukSLSu82K3OX48eMxoe32RXUvEbLSsTIjIyMu2oOrxEug5fvwTATeCRJEl4qKCm+33njAN/BO\nfA6/ekQy8E4BZJvN5lywfRYlc72QDOCbwG+Bl4ES4P8E8lE04hkoGfT/DPwPQC9KoeXnVqt1k81m\nW1a0HC/2aJFieFgR8BokiTJLEpe6xpiYmGDnzsiqdJ599lnOnj3L9VaZrauiH3Reb1MKSo8fPx6x\na+Dpp5/m/fffp6MHyhesWMmyoLEVMtLTefHFF2NOZqAGgbIMM5Owbmt5XHxW1IeR2WyOi/H6Ulpa\nGhdjVifrED/3z8zMTMVJKTMzbsYcb1gsFu/3ifc4MvgW3CZWIL96RDLwngF0VqvVYLPZXD7bzcDU\nIvs7gWHgTZvN5gauWq1WI/Azq9X6F57fpwB/7NF/Y7VazwAdwJvA/7HcYLq7u0P9/8Q1t2/fBsCg\nkyi3KPOeuro61q9fH9HzGo1GNm3axL179+gYklmVE73ge2hSUN8hKCoqIj8/P2LXwO7du3n//fdp\nan008O7ogRk7HDy0N2R3lkgghECn0yE8jjcWiyUuPitOpzKft9vtcTFeX0ZHR+Ni+djX6z9e3uM3\n3niDn/70p2zfvj1uxhxvxON1EW/4dplOvMfxyXKT0khGQappdNGC7cU8Kj/Bs+2eJ+hWUSUkq31e\nc1v9pUeC0kpC570ssizT2tKCTlKW5isyk5GAlpaWqJz/1KlTAHxwS0aOUlW5EILf33IjC3jxxRcj\nKkkoKyujqKiIjh5wLmgj3+r5FOzbty9i5w8FSZIoKCjwOr/k5OSs7IA0Ei9SE18SS8eRo7q6mr/5\nm78hLy9vpYeSIEHQJJxMvtpEMvC+CUwAT6gbrFbrapQg+vNF9v8C2O7JcqtUAW6gDVDbLu7xOV4y\nsBZ4EL5hf/Xo6+tjanoag0d/nGzUU5xuprW1FZfL5efVoVNZWUlNTQ2dw4IrDwJrFGFYIiG41PaF\n3OkSNPQINmzYwO7duwMcafDs2LEDlxt6++e2CSHo7FV007GsL8zKylr0+wSRIV4eqvE4qUmQ4KtA\nPKyIJQieiD0BbDbbrNVq/Xvg76xW6yDQD/w98JnNZrvksRvMBoZtNpsD+AfgT4EfWa3W/xWlOPM/\nAj+y2WxDwJDVav0x8H9ZrdZvAZ0odoNu4P9n787DpKru/I+/q/eN7mZrdmhAkKURRCEKsogIrtEE\ncmLELZpkxpjEmDGTmYyTmEkmmaiT5DeTOMlMEidGTHJQx7jgggIiIgiCgiwiIPsOTTe9L1W/P+4t\nLJrq7uql6lZ1fV7Pw1Pdt27d+nYD3Z869T3nPBGtr6Mr2L59OwDpIRP7RvbI4cCeUvbu3cuwYcOi\nXsPNN9/Mls2bee2DSoYVpVCU3/Iv9W5ZPnrmwYmKT471zHOOt6asKsCLG/xkpKdzxx13xCRAjBkz\nhsWLF3MwJHifKnfaTMZPGBN3kypD5efnn/k4tH9TokO/VCWRBX9fzJ071+NKui79jOjaop0GHgAW\n4gTjZcAeYL573xTgkHuLu5vldJwwvh54EmeznbtDrvclnOUEn3DPKQIut9bGX/NsHNm6dSsA6amf\nBNDze+WedV+0FRQU8MU776ShERatbqSuofWWE3NJGsFFQnrmOZ+3ptEf4Om1jVTXBbjpC1+I+qot\nQcOGDcPn83Hs5CfHgh8PHz48JjW0V+gqJp291rl84qabbmLq1KlxN8FWpC2Ki4v5t3/7N+bPn9/6\nydIuerepa4vqe57upMq/c/80vW854GtybAvOet7NXa8WuN/9IxHw+/1s3bKFwqw0UkP+M4/q6QSs\nLVu2cO2118aklokTJzJ79mxee+01XtzQyI0Xp7b4A6ZPgY9u2c5SfF+fG1lf7LItfvYeDzBp0iRm\nzpzZSZW3Ljs7m169enGq/Bip7svZUncOUrzvBBkathMteCfKTnSQeCOEwZYY9aRLU3369PG6BJGE\nFb/vf0un2LNnD+WnT1PS++y1mQuy0hhckMWHH35IdXV1zOoxxjB06FDe3xtgw+7IQlOkL/63H/Kz\n8kM/Rb17x6zFJFTfvn2prnEWoAcod9tk4v2XVOjSVaEfS3IbPnw4l19+Od/85je9LkVEpMtQ8O7i\n1q9fD8D4Pt3OuW9Cn240NjaeWWowFtLT0/nqV79KTk42L73fyJGyzhmxLKsK8H/r/KSlpXHP175G\nTk5Op1y3LYLbEgfc+aMVlc6oYWgPdTwK3UY5UbZUDtJbstGTkpLCbbfdxpgxY7wuRUSky1Dw7sIC\ngQDvvPMOGakpjCs6dzfCi/o5gXDt2rUxratXr17cddeXqG+Ep99ppL6xY+HbHwjwjNvXffPNN3vW\n2lFYWAhwZmm+6hrnWLyHw9CwnSi7Kl511VUATJ8+3eNKREREIpcY61pJu+zatYujR4/yqQEFZKad\n+xprUH4m/btl8t57G6isrIxpf+/EiRO5/PLLWbZsGUs3+5l7Qftnca/+yM+e4wEmTpwY077uprp1\nc95VOBO8a6Fv//hfJSQ0bMfz6iuhpkyZQklJSdy/myAi0h5f+cpXPHnnVqIvMX7LSru88cYbAFw2\nqDDs/T6fj8sGFdLQ0Mjbb78dy9IA+PznP09RURGrd/jZfzKy9b2bOlERYOkWP9265XH77bd7Oroc\nDIF+nD5vv/+TMB7PEnWVDYVuEemqLr30UsaPH+91GRIFCt5dVEVFBWtWr6Z3Tjpjejc/kj11UCFp\nKT6WLl2K39++8NtemZmZ3HnnnQQC8MKGtu9qGQgEWLyhkYZGWLDgFs+DWOiId/BLSYTgrVUrRERE\nYkPBu4t64403qKuvZ1ZxD1JaGAXOz0xjcv98Dh06xAcffBDDCh3nn38+U6dO5fCpAOs/blvw3n4o\nwM6jAcaOHcvkyZOjVGHkgsE/0YJ3oo54i4iIJBoF7y6ovr6eJUteJSsthelDWt8CfO7wngC89NJL\n0S4trPnz55ORkcHyrf6INtYBZ0Llax/48fl8fOELX4iLCYzhgnci7ASpEW8REZHYUPDugt566y3K\nysqZOaQ7OemtT1ocXJDN2N65bNu2jZ07d8agwrMVFhYyZ84cKmoCrNsVWbvLB/sCHDsd4LLLLmPA\ngAFRrjAy2dnZpKenJ1zwDm6UIiIiItGl4N3FNDY2snjxYtJSfGdGsiNx3YjeADz//PPRKq1Fc+fO\nJTMzk7c/CtDgb3nUOxAIsPJDPykpKVx//fUxqrB1Pp+P7t27E/BD8Evo3r31dxy8lpra/hVlRERE\nJHIK3l3M6tWrOXbsGNMGF1KYFXkLwfk9czivezbvv/8+e/fujWKF4eXl5TFz5kxO1wTYvL/l4P3x\nsQBHy51t4Xv37h2jCiPTvXt3AnyyiU5wbe94puAtIiISGwreXYjf7+eF558n1efjmvN6temxPp+P\nT5/v7aj3rFmzAHj345bbTYLtKLNnz456TW0VDNqN7peQCCPe8dAfLyIikgwUvLuQtWvXcvjIEaYM\nKqBXTttXqijpncfQwmzWrVvHgQMHolBhy4qKihg9ejR7jwcorQw/6l1TH2D7oQD9+vVj+PDhMa6w\ndcGgHQg4o/iJMHExJyeH7OxspkyZ4nUpIiIiXZqCdxfh9/t54YUX8Pk+6dduK5/Px/UjnZHyF154\noTPLi9gll1wCwJYD4Ue9tx8K0OB3zovHkdrQEe5EGO0GZ1WTn//859x1111elyIiItKlKXh3ERs2\nbGD//v1cMqCAotz2r8s8oU83BuVnsmbNGg4fPtyJFUb4/BMm4PP5+Ohw+BHv7YecQH7hhRfGsqyI\nhfZ0J0J/d1BmZmbCbBcvIiKSqPSbtgsIBAI899xz+Gj/aHeQM+rdm0AgwIsvvtg5BbZBfn4+gwYN\nYt+JAPWNZ4fvQCDAx8cCFBQUMHDgwJjXFonQ5QMTYSlBERERiR0F7y7gvffeY+/evUwekE//bpkd\nvt5F/ZzrrFq1iiNHjnRChW0zcuRIGv1w+NTZwftUFVTWwogRI+KyzQQ4a9t6r7ewFxERkfii4J3g\n/H4/zz77LD7g+pGds7Reis/HDSN74/f7ee655zrlmm0xZMgQ4NzgHfy8uLg41iVFLHSL+ETYLl5E\nRERiR8E7wb3zzjvs3buXSwYWMKBbVqdd9+L++QzKz+Ttt99m3759nXbdSATbSI6dPvv4sdNO8I6X\nnSrDycnJOfNxXl6eh5WIiIhIvFHwTmD19fU88/TTpKb4+Mz5RRE9xh9oeXOaoBSfj/mj+xAIBFi0\naFFHymyzoiLnazlZcXatwc/79OkT03raInSCYmgIFxEREWkxeBtjvmiMmRzy+UPGmNujX5ZE4tVX\nX+XY8ePMLu5B71ZWMtlfXkNpTT2lNQ384+sfsb+8ptXrjyvKY0yvXDZt2sTGjRs7q+xW5eTkkJOT\nw6mqc3u8AXr1atvmQF7Jzs72ugQRERGJI80Gb2PMncB3gbqQwyuBB4wxt0a7MGlZaWkpzz//HHkZ\naRH1dv9q7T78bo49XFnHr9a13j7i8/m4qaQvPh88+eST1NfXd7TsiPXq1YuyqrOPlVUFyM/PT4hN\nacBZok9EREQkqKUR73uAK6y17wUPWGufA+YC34x2YdKyP/3pT9TW1jF/dBG5GaktnltWU8/hyrqz\njh2uqKOspvUgPSg/iyuKe3DkyBFeeeWVDtXcFj169KC+0dkBEpylBE9XO8cTRUZG+9dTFxERka6n\npeDts9bubXrQWrsLaDnpSVRt2bKFtWvXMqx7NtMGt75JS70/fF93c8eb+syoIvIz03j+uec4fvx4\nm2ptr2DADpZYWQsN/sQK3okyMi8iIiKx0VLwTjXGnHO/e0yJwiMNDQ0sfOIJfMCt4/qREoP1rHPS\nUzFj+lBXX89f/vKXqD8fQM+ePYFPgnew3ztR+ruBuF1rXERERLzRUvBeDtwX5vi3gLVRqUZatWLF\nCg4eOsSMId0pLozd5L0pAws4r3s269atY/v27VF/vt69nb51v7ND/JmJlcFAHs9GjBgBaFUTERER\nOVtaC/d9D3jDGHMj8BZOSL8UyAdmx6A2aaKuro6//vWvZKWlcGOEywd2Fp/Px01j+/KjlR/zzNNP\n8w//+I9Rfb7gyHZwxLu00vkguNRgPPva177GyZMntY63iIiInKXZEW9rbRnwKeC3QA6QAfwKuNha\neyw25UmoVatWUV5ezqziHhRktfSaKTqG98hhXFEeH27fzs6dO6P6XMG1uoMj3mRfDCwAACAASURB\nVImwhndQfn5+XO+uKSIiIt5oMb1Za2uBP7h/xGNvvfUWPh/MHubdBMM5w3qy6WgFb731FsOHD4/a\n8+Tk5JCXl0dVZQUAJyqcUfdE6vEWERERCdVs8DbGLANCl71oBE4Ai621j0e7MDlbZWUlO3bs4Pye\nOXTP8m5u6+heueRlpLLx/fej/lx9+vRh504neJ+sCNCzZy+tFCIiIiIJq6XJlb/EaS0J/vlvnA10\nvm6M+VYMapMQ+/Y5G94M6+7tboipKT6GFmZz4uRJKisro/pcwbaS4l4+KmoSo79bREREpDnNjnhb\na58Od9wYsxBnxZOfRakmCaOqylnWoyCz/b3dGRkZ5OfnU15eTl1dXesPaEY3d8Oe6upqcnNz232d\n1gRXNulT4AMCCt4iIiKS0Foa8Q7LWlvK2S0oEgOpqU7YrW9s37c+IyODG2+8kd/97nfceOONHdpV\nscFdaiRYU7QEg/fuY4m3hreIiIhIU20O3sYYH9pAJ+aCo72HKmrb9fj8/HwWLFhAVlYWCxYsID8/\nv921HKqoIy0tjYKCgnZfIxLBNbv3nnCCdyLtWikiIiLSVEuTK8OlnJ7AN4C3o1aRhNWnTx9yc3L4\n8EQVgUCgzbsilpeXs3DhQhYsWMDChQspLy8Hure5joq6BvaX1zDy/PNJSWnz67Y2KSwsBKCm3vlc\nwVtEREQSWUsNw8dxWkqCCS/gHlsM3BvluqSJlJQUSsaNY82aNewuq2FoG3etrKur49lnn2Xp0qUd\n6vFef+g0AWDcuHHtenxbBIN3ULRH2EVERESiqaXJlecMZxpj0gADLMHZXEdi6JJLLmHNmjWs2neq\nzcEbnPB9/PjxDtWwav8pAD71qej/9WdmZpKenk59vTPk3ZH2GBERERGvRbREhjGmO/A3wD1AHvCf\n0SxKwispKSE/P5+395fxuTF9yEiNbqtHU4cravnwRBVjxoyJyURHn89HXl4epaWlpKSkkJ3t7VKK\nIiIiIh3RYvA2xpwP3AfcAuwGsoFidzv5VhljUoEfAXcA3YCXgXustUeaOX8g8AtgLlANPAXcb62t\ncu+/BngxzEMHWWv3R1JTIktLS+Oyyy5j8eLFrDtYzpRBha0/qBOt2FMKwPTp02P2nLm5uZSWlpKT\nk9PmvnYRERGReNLskKkxZjGwAqgDZlprS4DTkYZu14PA7cBtwHRgINDc+uCZOC0sPYCpwOeB64CH\nQk4bB2wA+jX5c7ANNSW0GTNmAPDm3tKYPm+DP8Bb+8vIy81l4sSJMXve4Ci3RrtFREQk0bU04j0B\nWA98AHzkHot4EWljTAbOJMxvWGuXuMduAj42xkyx1q5q8pCbcUL0FHetcIwx3wfuDjmnBNhkrT0c\naR1dTVFREaNGjWLbtm0crayjKLf963G3xcYjpymvbeDK6ZfHdNv2nJwcgKivoCIiIiISbS2lmcHA\n74EvAIeMMYtwWk0iNQGnvWR58IC1djdOy8q0MOfPBZYEQ7d7/mPW2skh55QAW9tQQ5c0depUAN52\nJzrGQnBS5WWXXRaz5wQ6tNGPiIiISDxpaVWTBmARsMgYMwb4WyDbGPMR8O/W2l+3cu2B7u2BJscP\nAoPCnD8SWGqM+SFOT3kAeAZ4wFpb4/aLjwIuMsa8D/QG1gJ/b639sJVaupSLLrqIPz7+OG/vL+PT\nI3tHvfe5sq6R949UMGBAfwYNCvdXFz3q6xYREZGuIqJVTay1W4BvGGP+AScU/y3QWvDOAfzW2vom\nx2uBrDDn5wN3AS8BnwMGAL8EinB6xIe7j8sEvgxkAA8AbxpjSqy1R1sqpn///q2Um1imTJ3K8uXL\n2XWqmuHdc6L6XGsPltHgDzB37lUMGDAgqs/VVLC3Oy0trcv9HYqIiEhyiSh4B7mri/y3+6c11UCK\nMSbNHT0PygQqw5xfD5wEbrXWNgLrjDHpOCPu91lrtxtjegKnrLV+AGPMZ4G9wK3Av7dUzMGDXWv+\n5cSJE1m+fDkr956KevBeue8UPp+PsWPHxvz7WF1dDUBDQ0OX+zsUERGRrqelgcJozljb5972a3K8\nP+e2n+Ae2+qG7qAt7m0xgLX2ZDB0u59XAbsI37rSpY0dO5YePXqw+kAZ1fWNrT+gnfaW1bCztJqS\nkhJt2S4iIiLSAdEM3u8Dp4EZwQPGmGKcEL0izPlvAhPcUe6gEqAR2G2MudEYc9oY0zvket1wesM3\nd3r1cS4lJYUZM2ZQ0+Dnzb0tT7JMTwnfJ93c8VBLdp0AYNasWW0vshOox1tERES6iqgFb2ttLfAo\n8Igx5ipjzETgz8Ab1trVxpgMY0xfd9lBcHrGs4DHjTGjjDGzgYeBx621J4A3gHLgj8aYC9zrLQKO\nA3+M1tcRzy6//HIy0tN5ZdcJGvz+Zs8ryEqnb5NlB/vmZVCQ1fKygCeq63n7QBl9+/blggsu6JSa\nRURERJJVtBdHfgBYCDwBLAP2APPd+6YAh9xb3N0sp+NsoLMeeBJns5273ftLgdk4veDL3T+VwCxr\nbU2Uv4641K1bN2Zefjknq+tbHfW+Z9IgggPcffMyuOfi1rtzXvzoGI3+ANdee61n62gHAhEvHS8i\nIiIS13xJEmwCXXViXllZGX//7W+Tmwo/mXUemWnNB+T7l2zHHwjwsznnt3rdo5V1fHfpDnr27s2P\nf/xjUlNTO7PsiP3Xf/0X77zzDn379uUnP/mJJzWIiIiIRMqdXBm2V1bbASa4goICrpwzh9Kael77\n+ESr56dE2DP9zLajNAYCfPazn/UsdMMnPd5J8gJRREREujAF7y7gmmuuITc3lxd3nKCirqH1B7Ri\n96lq1hwoo3jIECZNmtQJFbbfwIHOPkyjR4/2tA4RERGRjlLw7gJycnK47rrrqK5vZPFHxzt8vae3\nOnsRzf/c5zzr7Q668sorWbBgAfPmzfO0DhEREZGOUvDuImbNmkX37t15fXcpZTXtH/XefqKSD45V\nMHr0aMaOHduJFbZPZmYms2fPJi8vz+tSRERERDpEwbuLyMjI4LrrrqOu0c/LO9s/6v3c9mMAfOYz\nn+ms0kREREQEBe8uZdq0aRQWFLB8TylV7djNck9ZNZuPVTJq1ChGjBgRhQpFREREkpeCdxeSnp7O\n7CuvpKbBz4o9pW1+/Ks7nVVRrrnmms4uTURERCTpKXh3MTNmzCAjPZ3Xd5/E34Yl+MprG1hzsJx+\n/fpRUlISxQpFREREkpOCdxeTl5fHpy65hONV9Ww6WhHx41bsLaXRH2DWrFln1s4WERERkc6j4N0F\nzZo1C4DlEbab+AMBVuwpJSMjgylTpkSzNBEREZGkpeDdBRUXF1M8ZAjvHzlNaXV9q+dvPVbJsap6\nJk+eTE5OTgwqFBEREUk+Ct5d1PQZMwgEYOW+U62e+8ZeZ2R8xowZ0S5LREREJGkpeHdRl1xyCRkZ\nGby5t7TFSZanaxtYf/g0Awb0Z/jw4TGsUERERCS5KHh3UdnZ2UyaNIljVfVsP1HV7HlvHyij0R9g\n2rTpmlQpIiIiEkUK3l3YtGnTAHhzb/OTLFfuLSU1NZVLL700VmWJiIiIJCUF7y5s5MiR9O7Vi3WH\nTlPTcO5OlnvLathXXsv48ePJz8/3oEIRERGR5KHg3YX5fD4unTKFukY/Gw6fPuf+1QeciZca7RYR\nERGJPgXvLu6SSy4B4J2D5WcdDwQCrD1YTnZWFuPHj/eiNBEREZGkouDdxfXr148BAwaw+WgFoWub\n7C2v4XhVPeMnTCA9Pd2z+kRERESShYJ3EpgwYQL1/gD1jf4zxzYeqThzn4iIiIhEn4J3Ehg3bhwA\n9f5Pxry3HKvA5/MxduxYr8oSERERSSoK3klg2LBhpKWlnRnxbvD72VlazcCBA8nLy/O4OhEREZHk\noOCdBNLT0xkyZAiN7oD3gdO11PsD2qlSREREJIYUvJPE4MGDAWj0B9hXVnvWMRERERGJPgXvJNG3\nb18AGgMBjlTWnnVMRERERKJPwTtJ9OrVC3CC9/GqegB69+7tZUkiIiIiSUXBO0kUFhYC4A9AeW0D\nAAUFBV6WJCIiIpJUFLyTRHD1kkAgwOm6RrIyM7VxjoiIiEgMKXgnidzcXMAZ8a6qbyQ3L9fjikRE\nRESSi4J3ksjOzgYgAFQ1+MnJUfAWERERiSUF7ySRkpKCD6fVpKa+8UwQFxEREZHYUPBOIplZWYAz\n6p2Tk+NtMSIiIiJJRsE7iRQVFZ3ZvVJbxYuIiIjEloJ3EunWrduZjxW8RURERGJLwTuJhAbv0I9F\nREREJPoUvJNIfn5+2I9FREREJPoUvJNIaNjWrpUiIiIisZUWzYsbY1KBHwF3AN2Al4F7rLVHmjl/\nIPALYC5QDTwF3G+trQpz7nxgETDUWrs7GvV3NWo1EREREfFOtEe8HwRuB24DpgMDgafDnWiMyQSW\nAD2AqcDngeuAh8Kc2w/4TVQq7sI0uVJERETEO1EL3saYDOBe4LvW2iXW2vXATcBUY8yUMA+5GegH\nzLPWbrTWLgO+D0wOc+7vgY1RKr3LCm4b3/RjEREREYm+aI54T8BpL1kePOC2hOwGpoU5fy6wxFpb\nGnL+Y9bas4K3MearOAH9h51ecRcXultllruZjoiIiIjERjR7vAe6tweaHD8IDApz/khgqTHmh8At\nOBssPgM8YK2tATDGjAT+FZgBaFmONgoN2z6fz8NKRERERJJPNIN3DuC31tY3OV4LhBtuzQfuAl4C\nPgcMAH4JFAG3GWPSgD8CD1lrNxpjLmtLMf37929j+V1P6Dbx+n6IiIiIxFY0g3c1kGKMSbPWNoQc\nzwQqw5xfD5wEbrXWNgLrjDHpwCJjzH3A1wA/YSZbRuLgwYPteViXUlX1yeIw+n6IiIiIdL6WBjej\n2eO9z73t1+R4f85tP8E9ttUN3UFb3NtinCUJJwJlxpgK4FX3vs3GmO92RsFdXVpaVFePFBEREZEW\nRDN4vw+cxunHBsAYU4wToleEOf9NYII7yh1UAjTiTMicCYzFmbQ5Afiie841wK87s/CuKiVF+yWJ\niIiIeCVqQ6DW2lpjzKPAI8aY48BR4FHgDWvtane5wR7ASWttHU54/jrwuDHmBziTMx8GHrfWngBO\nhF7fGNPX/XCPtfZktL6OrkQTKkVERES8E+0h0AeAhcATwDJgDzDfvW8KcMi9xd3NcjpOGF8PPImz\n2c7dUa4xaSh4i4iIiHjHFwgEvK4hFgKaTAiNjY186UtfAuCxxx7zuBoRERGRrsedXBl2tFNNv0kk\nSV5kiYiIiMQlBe8kouAtIiIi4h0F7yTi9/u9LkFEREQkaSl4JxEFbxERERHvKHgnEQVvEREREe8o\neCcR9XiLiIiIeEfBO4loxFtERETEOwreHXDo0CGWLl2qkWQRERERaVXUtoxPBo8++ij79+9n8ODB\nnHfeeV6XIyIiIiJxTCPeHbB//34AKisrPa4kMikp+usWERER8YqSWBLx+cLuXioiIiIiMaDgnUQ0\n4i0iIiLiHSWxTtDY2Oh1CRFR8BYRERHxjpJYJ6itrfW6hIgoeIuIiIh4R0msE1RXV3tdQkQUvEVE\nRES8oyTWCaqqqrwuISIK3iIiIiLeURJrp9C+7oqKCg8riZxWNRERERHxjoJ3O4Wu3Z0o63iLiIiI\niHe0c2U7hYbtRBnxBpg1axZ9+vTxugwRERGRpKPg3U6hYTuRRrxvvfVWr0sQERERSUpqNWmnRB3x\nFhERERFvKHi3U3l5+ZmPT58+7WElIiIiIpIIFLzbqays7MzHFRUVNDQ0eFiNiIiIiMQ7Be92OnHi\nBACpPudbePLkSS/LEREREZE4p+DdTkePHgUgIzX1rM9FRERERMJR8G6HQCDAgf37SfH5SHN3g9y3\nb5/HVYmIiIhIPFPwbodjx45Rfvo0aSkpZ4L3rl27PK5KREREROKZgnc7bN68GYD0lFRSfSn0zM5l\n65Yt+P1+jysTERERkXil4N0OGzZsAD7p757QZwCVVVVs377dy7JEREREJI4peLdRWVkZmzdvZmhh\njzMrmkweMBiAVatWeVmaiIiIiMQxBe82WrlyJX6/n2mDhp85NrpXX3pm57L2nXeorq72sDoRERER\niVcK3m3g9/tZtmwZGalpTB007MzxFJ+Py4vPo6a2lrfeesvDCkVEREQkXil4t8GGDRs4ceIElw0a\nSm5Gxln3XV48grSUFF5/7TVNshQRERGRcyh4t8GSV18FYO6wUefcV5CZzaUDh3L4yBE2bdoU69JE\nREREJM4peEdo7969fLh9O+OK+jEgvzDsOVe5gfy1116LZWkiIiIikgAUvCO0dOlSAOaEGe0OGlLY\ng/N7FvHBBx9w5MiRWJUmIiIiIglAwTsCNTU1rFm9mp7ZuYzv07/Fc68YOhKAFStWxKI0EREREUkQ\nCt4RWL9+PTW1tUwfPJwUX8vfsov7DSYnPYNVq1ZpkqWIiIiInJEWzYsbY1KBHwF3AN2Al4F7rLVh\n+zCMMQOBXwBzgWrgKeB+a22Ve/8Y4GfAFKAWeBr4jrW2LJpfx5o1awCYOmhoq+dmpKYyuf9glu/Z\nwfbt2xk1qvnWFBERERFJHtEe8X4QuB24DZgODMQJy+cwxmQCS4AewFTg88B1wEPu/XnAa8BJYDLw\naWAa8Fg0v4Cqqio2b97MkILu9M3Lj+gxnxowBIB33303mqWJiIiISAKJWvA2xmQA9wLftdYusdau\nB24CphpjpoR5yM1AP2CetXajtXYZ8H2ckA0wBFgJfNlau81a+zbw38AV0foaADZt2kRjYyMX9RsU\n8WNG9+pLTnoGGzZsIBAIRLE6EREREUkU0Ww1mYDTXrI8eMBau9sYsxtnpHpVk/PnAkustaUh5z+G\nO6Jtrd0MmOB9xpiRwK3Aq9Ep37Fx40YAJvaNPHinpaRwQVF/Vh/Yzf79+xk0KPLHioiIiEjXFM3g\nPdC9PdDk+EEgXBIdCSw1xvwQuAUIAM8AD1hra0JPNMa8B4wH9gCf6cyiQ/n9fjZu3Ej3rGyGFHRv\n02Mn9B3A6gO72bhxo4K3iIiIiEQ1eOcAfmttfZPjtUBWmPPzgbuAl4DPAQOAXwJFOD3ioe4EcoGf\n4oT18cEJmM3p37/lZQDD2bx5MxUVFcwcch4+n69Njx3fZwA+n48tW7bw5S9/uc3PLSIiIiJdSzSD\ndzWQYoxJs9Y2hBzPBCrDnF+PM3HyVmttI7DOGJMOLDLG3GetPRE80e0XxxgzD9gP3Ag82VIxBw8e\nbPMX8MorrwC0qb87qFtGJiN79Gbr1q1s3bqVgoKCNl9DRERERBJLS4O90VzVZJ9726/J8f6c236C\ne2yrG7qDtri3xcaYYmPMDaEPsNYeAk7gjI53Kr/fz7q1a8lKS2ds76ZfQmQm9R9MIBBg7dq1nVyd\niIiIiCSaaAbv94HTwIzgAWNMMVAMhNvW8U1ggjvKHVQCNAK7cVY3edoY0yfkekOB3nwS0DvNtm3b\nOHHyJJP7DyYjNbVd17hkQDEpPh8rV67s5OpEREREJNFErdXEWltrjHkUeMQYcxw4CjwKvGGtXe0u\nN9gDOGmtrQN+DXwdeNwY8wOcyZkPA49ba08YY14AdgELjTH34ayY8p/A2zh94Z1q6dKlAFxePKLd\n1yjMyubCPgN4d88edu7cyfDhwzurPBERERFJMNHeQOcBYCHwBLAMZxWS+e59U4BD7i3ubpbTccL4\nepye7aeBu937q4A5OKPoK4AXgPeAq621nbo3++HDh1m/fj1DC3twXvdeHbrWlcOdnStfeqnTXxuI\niIiISALxJckGL4G2TK78zW9+w+rVq/nGpOlMdnehDOebrzwDwC/mfrb5Jw4EeHDFS+wsPcGDDz7I\nkCHNX09EREREEps7uTLscnjRHvFOOLt27WL16tUUF/Tg4v6DO3w9n8/H/NETAPjLX/6inSxFRERE\nkpSCdwi/38/CJ54AYMG4i0mJYO1ufwRBelxRf8b3GcDWrVtZt25dh+sUERERkcSj4B1i+fLl7Pr4\nYy4dWMzoXn1aPHdfeSmlNVWcrKni/tf+yr7y0hbPv3XcxaSnpPLkk09SVdXiXj8iIiIi0gUpeLtO\nnjzJokWLyEnP4OaSi1o9//+9s4JGd7T7cEU5//FOuBUSP9E3L59Pjyzh1KlTLFq0qFNqFhEREZHE\noeCNMwHyD3/4AzU1Ndw8diLds3JaPP9UTTWHK8rPOnaoopxTNdUtPu76kWMZ0K2A5cuXs23btg7X\nLSIiIiKJQ8EbWL16NRs3bmRs777MGHJeq+fXNza26XhQWkoqX5k4BZ/Px2OPPUZtbW276hURERGR\nxJP0wfv06dM8+eSTZKSmcteES/BFMKGyI4Z378VVw0Zx9OhRnnvuuag+l4iIiIjEj6QP3k899RQV\nFRXMGzWeotxuMXnOeaPH0ysnl1defpkDBw7E5DlFRERExFtJHbx3797Nm2++ycD8Qq4aPjpmz5uV\nls5t4ybR6Pfzpz/9KWbPKyIiIiLeSerg/dRTTxEIBLhl3MWkpsT2W3Fh34GMK+rH5s2b2bx5c0yf\nW0RERERiL2mD965du9i8eTMlvftS0rtfzJ/f5/NhxlwIwAsvvBDz5xcRERGR2Era4P36668DcN3I\nEs9qGFrYk3FF/di2bRv79+/3rA4RERERib6kDN719fW8++67FOXmMbZXX09ruXzICADWrFnjaR0i\nIiIiEl1JGbw//vhjamtrmdBnYIeWD8zIyKBXr15kZGS0+xrjivqT4vOxdevWdl9DREREROJfUgbv\nQ4cOAVBc0KPd18jIyODGG2/kd7/7HTfeeGO7w3d2ejp98/I57NYkIiIiIl1TUgbvqqoqALplZrb7\nGvn5+SxYsICsrCwWLFhAfn5+u6/VLSOTSrcmEREREemakjJ4p6enA1DTUN/ua5SXl7Nw4UJqampY\nuHAh5eXl7b5WTUMDGW5NIiIiItI1pXldgBf69OkDwIHTZe2+Rl1dHc8++yxLly6lvLycurq6dl2n\nwe/ncEU5vfv2aXctIiIiIhL/knLEe/jw4fh8Pj442rG+6rq6Oo4fP97u0A2w4+QxahsbGDlyZIdq\nEREREZH4lpTBOycnh1GjRrGj9DhHKk97Wsuq/R8DcOGFF3pah4iIiIhEV1IGb4Bp06YBsGTXh57V\nUFlXy8p9H9OzRw/Gjh3rWR0iIiIiEn1JG7wnTZpEYWEhy/d8xOm6Wk9qWPLxh9Q1NnDF7NmkpCTt\nX4WIiIhIUkjatJeWlsbVV19NTUMDL+/YEvPnr6qv4+Wd28jNzWXmzJkxf34RERERia2kDd4AM2bM\nID8/n1d2beN0bU1Mn/vVXduoqKvlqquuIjs7O6bPLSIiIiKxl9TBOzMzk2uvvZaahgYWx3DUu6q+\njsU7tpKbm8sVV1wRs+cVEREREe8kdfAGmDlzJgUFBSz5+MOYjXq/umsbVfV1Gu0WERERSSJJH7wz\nMjK45pprnF7vnVuj/nzV9fVners12i0iIiKSPJI+eIPb692tG6/u+pDKCDbDSU9NbdPxUK/v3k5F\nXS1z5szRaLeIiIhIElHwxun1nnvVVVQ31PPqrm2tnl+YlU3fvPyzjvXLy6cwq+UgXev2kmdnZzN7\n9uwO1SwiIiIiiUXB2zVr1ixyc3N5Zdc2quvrWz3/3snTSfX5ACd0f2Py9FYfs3T3dspra5g9ezY5\nOTkdrllEREREEoeCtysrK4u5c+dSUVcb0aj3oPzudM/KoUdWDg/PvoFB+d1bPL+2oYEXdmwhKzOT\nOXPmdFbZIiIiIpIgFLxDzJ49m9ycHBbv2EJVfeu93gAp7qh3a17fvZ2ymmqumD2bvLy8jpQpIiIi\nIglIwTtEdnY2V119NZX1dby0o/NWOKmur+f57R+QnZXFVVdd1WnXFREREZHEoeDdxOzZs8nv1o2X\ndm6hrLa6U6750s4tnK6r5aqrr9Zot4iIiEiSUvBuIisri0/fcAM1DQ08u21Th69XVlvN4h1byO/W\nTb3dIiIiIklMwTuMGTNmUFRUxNLd2zlcUd6haz27bRM1DQ18+oYbyMrK6qQKRURERCTRKHiHkZaW\nxrx582gMBHhq63vtvs7hinKW7t5OUVERM2bM6MQKRURERCTRKHg3Y9KkSQwdOpTVB/bw8akT7brG\nU1vfozEQYN68eaSlpXVyhSIiIiKSSKKaBo0xqcCPgDuAbsDLwD3W2iPNnD8Q+AUwF6gGngLut9ZW\nufefBzwCXAYEgOXA31lr93Z27T6fj3nz5vHII4/w1Nb3+PalV7Tp8XvKTrL6wB6GDBnCxRdf3Nnl\niYiIiEiCifaI94PA7cBtwHRgIPB0uBONMZnAEqAHMBX4PHAd8JB7fy7wCpAKzMIJ572Al9zHdrqx\nY8cyatQo3j9ykB0nj7Xpsf+3bSMA8+bNIyVFbyyIiIiIJLuoJUJjTAZwL/Bda+0Sa+164CZgqjFm\nSpiH3Az0A+ZZazdaa5cB3wcmu/fPAQYDC9z71+ME+jHAp6L1ddxwww0A/PXDyFc42VdeyrpD+xg2\nbBglJSXRKk1EREREEkg0h2In4LSXLA8esNbuBnYD08KcPxdYYq0tDTn/MWttMHi/A1xjrQ1dZsTv\n3ra8X3sHjBo1ihHnnceGIwfYV17a+gOAFz/aAsD111+PL8KdLUVERESka4tmj/dA9/ZAk+MHgUFh\nzh8JLDXG/BC4BaeH+xngAWttjbX2QJhr/QNQCbzZaVWHcfU11/DRf/wHL+/cxpcvvLTFc0trqnj7\nwG769+vHBRdcEM2yRERERCSBRDN45wB+a219k+O1QLgFrfOBu4CXgM8BA4BfAkU4LSVnMcbcDXwN\n+Lq19mRrxfTv379NxYfq27cvTz31FKv2f8xNYyfSLaP5lvKlH39Eo9/PvPnzGThwYLPniYiIiEhy\niWbwrgZSjDFp1tqGkOOZOKPUTdUDJ4FbrbWNwDpjTDqwyBhzn7X2zJp+1SXf+QAAFmhJREFUxph/\nwlkt5SfW2l9GUszBgwfb+3UAMH36dP785z/z5p6dXDNiTNhzGv1+lu/5iOysLMaOHdvh5xQRERGR\nxNLSYG80e7z3ubf9mhzvz7ktI7jHtrqhO2iLe1sMYIxJMcb8Gid0f8da+93OK7dlU6dOJS0tjeV7\nPiIQCIQ9570jByitqebSKVPIzIzKQisiIiIikqCiGbzfB04DZ7ZsNMYU44ToFWHOfxOY4I5yB5UA\njTgTMsFpPfkS8EVr7UOdXnEL8vLyuOiiizhYUc6O0uNhz1mxdyeAdqkUERERkXNErdXEWltrjHkU\neMQYcxw4CjwKvGGtXe0uN9gDOGmtrQN+DXwdeNwY8wOcyZkPA49ba08YY64F7gZ+ALxsjOkb8nSn\nrLU10fpagqZOncqaNWtYuW8XI3r0Puu+03W1vHd4P4MGDmTw4MHRLkVEREREEky0d3Z5AFgIPAEs\nA/YA8937pgCH3Fvc3Syn44Tx9cCTOJvt3O2ev8C9/b77uNA/wWtG1ZgxY8jPz2fNgT00+v1n3bf2\n4B4aAwEunRJuiXIRERERSXa+5vqVu5hAZ010fOKJJ3j99df5zpQr+N2G1QD8Yu5n+be3lvDBscM8\n/PDD9OrVq1OeS0REREQSizu5MuxGLtrLvI0uuugiAN49tO/Mscq6WrYcP0JxcbFCt4iIiIiEpeDd\nRiNHjiQ7O5v3jnwygr7p2CH8gQATJkzwsDIRERERiWcK3m2UmprK2LFjOV5VQWPA6fP+4OghAMaN\nG+dlaSIiIiISxxS822H06NEA1Dc6S45vO36E7KxsiouLPaxKREREROKZgnc7jBgxAoB6vx8/AQ5X\nnmb4ecNJSdG3U0RERETCU1Jsh/79+5ORnk6D30+Du6zgsGHDPK5KREREROKZgnc7pKam0n/AABoD\nnwTvgQMHelyViIiIiMQzBe92ctdopL6x4azPRURERETCUfBup6KiIsDp8wbo3bt3S6eLiIiISJJT\n8G6nHj16nPk4Ly+PjIwMD6sRERERkXin4N1OhYWFYT8WEREREQlHwbud8vLyznzcrVs3DysRERER\nkUSg4N1Oubm5Zz7OycnxsBIRERERSQQK3u2UnZ0d9mMRERERkXAUvNspKyvrzMeZmZkeViIiIiIi\niUDBu53S0tLOfKzgLSIiIiKtUfBuJ5/Pd+ZjLSUoIiIiIq1R8O4ECt4iIiIi0hoF704Q2nYiIiIi\nIhKOgncnUPAWERERkdYoeHeC9PR0r0sQERERkTin4N0JFLxFREREpDUK3p1AwVtEREREWqPg3QlS\nU1O9LkFERERE4pyCt4iIiIhIDCh4i4iIiIjEgIK3iIiIiEgMKHh3gtDt40VEREREwlHw7gSBQMDr\nEkREREQkzil4i4iIiIjEgIJ3B5SUlADQo0cPjysRERERkXjnS5I2icDBgwc7/aJlZWXs3buXcePG\ndfq1RURERCTx9O/fHyDsBEAFbxERERGRTtJS8FariYiIiIhIDCh4i4iIiIjEgIK3iIiIiEgMKHiL\niIiIiMRAWjQvboxJBX4E3AF0A14G7rHWHmnm/IHAL4C5QDXwFHC/tbaqyXk+YDHwlrX2R1H7AkRE\nREREOkm0R7wfBG4HbgOmAwOBp8OdaIzJBJYAPYCpwOeB64CHmpyXAfwWuCpaRYuIiIiIdLaoBW83\nIN8LfNdau8Raux64CZhqjJkS5iE3A/2AedbajdbaZcD3gckh15wIrAEuB05Fq3YRERERkc4WzRHv\nCTjtJcuDB6y1u4HdwLQw588FllhrS0POf8xaOznknCuBFe61yzq9YhERERGRKIlmj/dA9/ZAk+MH\ngUFhzh8JLDXG/BC4BQgAzwAPWGtrAKy1Pw2ebIzp9IJFRERERKIlmsE7B/Bba+ubHK8FssKcnw/c\nBbwEfA4YAPwSKMLpEe8QdxchERERERFPRDN4VwMpxpg0a21DyPFMoDLM+fXASeBWa20jsM4Ykw4s\nMsbcZ6090ZFitGW8iIiIiERbS4O90ezx3ufe9mtyvD/ntp/gHtvqhu6gLe5tceeWJiIiIiISW9EM\n3u8Dp4EZwQPGmGKcEL0izPlvAhPcUe6gEqARZ0KmiIiIiEjCilqribW21hjzKPCIMeY4cBR4FHjD\nWrvaXW6wB3DSWlsH/Br4OvC4MeYHOJMzHwYe72ibiYiIiIiI16K9gc4DwELgCWAZsAeY7943BTjk\n3uLuZjkdJ4yvB57E2Wzn7ijXKCIiIiISdb5AIOB1DbEQ0ORKEREREYk2d3KlL9x90R7xFhERERER\nkmjE2+sCRERERCRphB3xjuY63vEk7BcvIiIiIhIrajUREREREYkBBW8RERERkRhQ8BYRERERiQEF\nbxERERGRGFDwFhERERGJgWRZ1aRTGWP6AA8Bc4BsYA3wd9baDzwtrAXGmIHAz4ErcF5wvQx8y1ob\n9zsLGWMuAVYCs621yz0up1nGmDHA5jB3TbPWrox1PZEyxnwJ+HtgELAF+La1dqm3VYVnjJmJswtu\nOMustbNiWE5EjDG5wL8B84Ac4G2cnxdbPC2sBcaYfJyfcZ8GsoDFOD8vjnpaWBjGmF8DadbaL4Uc\nm4NT//nAR8B3rLUveVTiOcLVHHLfdOBla21O7CsLr5nv8deAr+H83NgD/Mxa+1uPSjxH05qNMT7g\nO8DfAH1xftZ9z1r7ondVnq2VfxfpwDvA+9baO2JdWzjN/Lt4B5jU5NTfhfuaOuH5dwONwDhrbVWT\n+5YDO6LxvB2lEe82MsakAP8HjARuwNnyvgx43RjT08vamuP+wHkR6A5cDswA+gHPe1lXJNzQ8kcg\n1etaIjAOOI7zvQ39s8bLolpijLkd+BVOMBwHvAE8Z4wp9rKuFqzi3O/vbYAf+KmHdbXk/wGzgc8B\nlwI1wMvGmCxPq2rZIuBq4IvANCAPWGaMyfS0qhDGGJ8x5l9wglTo8THAczhfw4XAX4FnjTFjY1/l\n2ZqrOeT+KTi/X+Lid3ML3+O7cX5m/Ai4APgZ8Kgx5tbYV3m2Fr7H3wL+AfgmMBZ4GvirMWZijEs8\nR2v/Llz/AkyIUUktauHfhQ/ne7uAs39GfyuK5QwDfhzF63c6jXi33XicX55jrLVbAdwfNieBa4HH\nPaytOX2ArcA/WGt3Axhjfobzy6i7tbbUy+Ja8TNgP3Ce14VEoATYYq097HUhkXB/SP4A+Km19vfu\nsfuBWTgvKHd7V1141to64Mz31xhTgDOy+bC19hXPCmvZjcAPrLVvARhj/gnnnZExwHovCwvHGDMB\n5928K621r7nHbgH2ATcBf/CwPACMMcOA3+H8n9vb5O57gdXW2n91P/9nY8xl7vGvxK7Ks7VUs/t/\n8ac4NQb/bXiqle/x3wK/stY+4X6+0xhzKc4LtT/GrsqztVJzDs47TX91P/+xMebbOANRnv0/bKXm\n4DlTgTuBTTEsLaxW6h2G+65eDH8P7gK+boyx1tpVMXrODlHwbru9wHXAhyHH/O5t99iX0zr3P8BN\nwc/dtpO/AdbGc+g2xlyD82LmamCjx+VEogTnBU6iOB8YAvwleMBa6ydORlUi9M9ALc5oULw6Bnze\nGPMX4BRwF1CK8wsjHo1wb8+0R1lrK4wxH+GEFM+DN84Lw33AF4A/N7lvGmCbHFtOyM9Aj7RUcyrO\ni51rgKHAL2NbWlgt1fsNzg1dfrz/HdhszdbaHwY/dt9tuhMnJC6PYX3htPR9xhiThzOg93U8fOEY\noqV6S4BqnNajWPlfnP87vzPGXGitrWl6gjFmMM4AzRU4rXOv47TO7XLv3w38JzDTPacc+C9r7Q9C\nrnEjzkDV+TiDUr/Faa8K5r+IKXi3kbX2BE7bRqhv4PR6vxr7itrGGPMsTotMKU7bSVwyxvTCeVX9\nRZxaE0EJkGWMWQ0UAx8A37XWvuNpVc0b6d4WGmOW4tS/DeedkbgfOTDGFOH0mN7dtL8vznwFeAI4\ngtOPWAXMsdae8rSq5gXnfQwEdgAYY1Ldz+Oix9sdaX0CwBjT9O6BwIEmxw7i9CJ7pqWarbUNuC94\n3TkXnmul3jdCP3eDzRdwwotnWvl3gXv8s8BTODtaf89auyFmBYYRQc2/wBkks8YYz4N3K/WW4Awu\nLDTGzABOAI8Bv2hPQI1QAGcw433gQZx2ojPc+Spv4QyKzcX5e38EeMMYU2KtLXNP/SFOG9K9wHzg\np8aY1621K91BwIU4L37ewGmn+RWQixPG2yQu+sgSmTHm08BPcF75JMJo5z8Dn8IZzVpijBngcT3N\n+Q3wnLX2Za8LiYQxJhvnbbYC4Ns4k9IO4vznHu1lbS3Id2//gPPq/SqcFwtL47jmUHfjBMEnWjvR\nY+fhtMdcC0wFXgGect95ikdrcV6A/doY08/9t/0ToDeQ4WllkcnB6aMPVYsz0iWdzBjTG2cw6jBO\n33e8W4PT+/8t4HvxEGab4+aLa4Cvel1LhMbizAd5BSfk/gonmH4/mk9qrd0OfA+43xhzUZO7b8F5\nJ+Yma+16a+27OPNterj3BT1vrf1va+0ua+1DOC8gLnXv+y7wqLX299bandba53AC/nfceX9touDd\nAcaYO3AmaPwFZ1WIuGet3eSOwN6E8/bm7R6XdA53wt+FwN95XUukrLXVuJNXrbVvut/jO3DaCeL1\nh2a9e/uv1tonrbXrgXtwVoG427uyInYL8Ji1tr7VMz1ijBkK/A9wr7V2sbV2DXAzTjC8z9PimuH2\n0X8GKMR58XgK6IWzsklZCw+NF9VA00mgmUClB7V0aW6/70qcfytzQkYP45a19oC19n1r7c9xBhzu\n97qmcNwXNP8DfNlae9LreiJ0GzDYWvu/btb4Nc4E3PvceQzR9DPgXeAxdwWYoBJgW+j30Fp7HGdV\nm5KQ87Y3uV4Znww0XAh8wxhTEfyD83eTjfPudpsoeLeTO0HqMeDXwG1RfBulw4wxfYwxZ/U3um/N\n7wTiccT7Dpy3iw+7/8CD/fQvucsXxSVrbbm1tjbkcz/ORClP3+JuQfDt+DMTdqy1AZy35IZ6UlGE\n3BUqziNMT2ScuRjnBe664AH3hcIG4njCsLV2m7X2YpzA3ctaeyfOv+Od3lYWkX04KymE6s+57SfS\nAe5qIG/j9HZPCfbLxitjzLXGmKb/5zYRn78DwRnpLgL+EhL2ZgIL3I/jjrW2IUwL3SagG867wdF8\n7kacvv3zgX8Kuau6mYek8sngEzjvijUVfLFQh7NyyoSQPxfgzIfZ39Za1ePdDsaYv8d5Ffe90Akb\ncWwI8CdjzA5r7To4sxrE+cTHRKmmbsF5JRnUF3gT+BKwxJOKWuG+vbUMZ8T7XfdYKs5/0EVe1taC\n9TijgJNwg6E7KjEGeM3DuiIxDTiUAO1dwR/KF+CunBDyPY6bdaVDuT2RzwP3BPcmcJeXHE+cjg42\nsRJnEmjoz+bLgRXelNP1GGNG4fws3gFc4859incPA0tx5oUETcYZ+YxHz+D0Jof6A05Lz3diX07r\n3PlNa6y194Ycvhg4GIs5LdbazcaYH+G01B7B+fe5BfgbY0yP4Ki3O4fsfOC/I7z0ZmCEtXZH8IAx\n5jM4nQO3tbVOBe82MsZcgPPK5/fA/xhj+obcfdpaG49vZ67DCa6/dfvZ6nF68Y4Rh8HbWnvWyJQx\nJtiveSAeN/BwvY8z0/k3xph7gAqcH469cNZxjjvW2ipjzM+BfzXGHMEZmfgqMBxns5d4diFOP3q8\newdYDfyvMearOOu8fxMYjMcT0ZpjrS13XzT+whjzDZyezd8Dr8frxkpN/CfwrjHmB8CfcFp7PkVi\ntE8lisdx2qVuBdJDfg82uG/jx6PgWuPrcF6cfQZnvekbPK2qGdba08Dp0GPGmGqcnLEj/KM89wzw\nL8aYd3FeNMzE+T14b0sP6mTBzcrGu58vxBkB/7MxJjjx8iGcRRsifcf0R8CLxpgPcNqLR+LMQ1sc\n+i53pNRq0nbB3ug7gUNN/sRrz6Yf+CzwHvACzqzccmCGtTYu37JKNO6qBFfjtMU8jxO4+gLT4/jF\nAjgTUh7GmTm/CWcyyRxr7YctPsp7/XDWzo9r7tuf1+NM6PozTgg/D2c301guudVWN+G8eHwbZzOa\nN3B+hsQ9a+0mnFA1H+dn3qeB6xPg3ZGEYIwZifMuWX+cn3ehvwNXe1hai6yzq+Y3gH/EedF+CzDP\nWrvY08K6lodxJiI+gDNK/B3gPhvDHU3dVr4vAg3u5zU4Ez1rcd71WorTvz0t0lF4d5GHW3FexH+A\nE7ofp+UNj5rlCwQC7XmciIiIiIi0gUa8RURERERiQMFbRERERCQGFLxFRERERGJAwVtEREREJAYU\nvEVEREREYkDBW0REREQkBrSBjoiIB9zdID8Gvhy6zq0x5n6gxFp7Ryc9z25gfnDX2mhyd718CSjE\n2dn36ZD7/he4EmfjLh+QibP74X3W2gZjzG+BP1tr433XVBGRdlPwFhHxjh94xBizwlq73etiOsEE\noI+19rxm7v+5tfYRAGNMFrAK+Dyw0Fr7pRjVKCLiGQVvERHvVAP/DvzJGHOptbYu9E53lPiDkLB6\n5nN3JPtJ4FqgJ/B9YCpwEVAPfNpae9C91D3GmPE4o8z/bq39vXu963F2mcsAqoD7rbVvG2MexNnF\ntB+w0Vp7S5O6bnSfLxVnF9xv4ewG9//bu7/Qu+c4juPPEf1q5WLtZmnUtFoJ20hc2MXExUrKxdsF\nF1OSlEXiRrlZtNzIn4RQLmhe6ieR2sVvTbkwaTYboqZQLHKhzGL4ufh8jg7OzpzfxTm/6fmoU+d7\nzuff91yc8zqf3ud8XwTOr6qDwNVJTow595V9Pcf6mPuAp4APgAXgbdql3lcBDyZ5tao2AC8Ac7Rd\n8+eTPD1mDklaVqzxlqTZehg4DjyyhL5zSS4D7gOeAx7vx18D24fanUiymVbqsauqLq6q9X3ObUk2\nAXcA81W1sve5ENg8InRvAJ6hXW77UuAh4A3aJcNvB44m2XiK0H1vVR2sqsN9jd8C745otw7Yk+RK\n2mWnH+2P3w+8meRyYBuwpar8HJN0xvANS5JmKMkfwK3AbVV13YTdBzXUR4FjSQ4NHa8aavdsn+sb\nYA9wLS2ErwEW+g71y7TSl0GZyHtJfhsx51ZgIckXfcy9wHe0nfbTeayH8kuA1cAPwBMj2p2k7XgD\nHBg6l9eBB6pqHrgJ2NFfP0k6Ixi8JWnGknwF3Am8RAukA4u0koqBc//R9Zeh+yfHTPH70P0Vve3Z\ntAC9cXADrgKO9HY/nWKsUZ8bZwHnjJn/X5L8TDvfLSOe/nUoUP/1GiR5C1gPBNgEHK6qiyaZV5Jm\nyeAtSctAktdo/whyz9DD3wNXAFTVauCaJQ6/vY9xAW2newHYC1zfS0eoqm3AR7T66XEG/db1fluB\ntcD+SRbUS0RuBN6foM8rwM1JdgN30erL104yryTNksFbkpaPHcCXQ8dPAmuq6jNaKci+JY47V1UH\naOUbdyf5PMnHtLru3VV1CNhJ+0Hm8XEDJfmEFnrnq+oIsAu4IcmP/2EdgxrvD4FPgfP4+xeN09kJ\n3NLXu59WevLOBP0laaZWLC4uznoNkiRJ0v+eO96SJEnSFBi8JUmSpCkweEuSJElTYPCWJEmSpsDg\nLUmSJE2BwVuSJEmaAoO3JEmSNAUGb0mSJGkK/gRML3CJXUqzngAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x1169dd320>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"d = pd.DataFrame(sim_df)\n",
"sns.plt.figure(figsize=(12,8))\n",
"sns.violinplot(data=d, x='qcut', y='AUC');\n",
"sns.plt.xlabel('Number of Bins')\n",
"sns.plt.title('Simulated AUC Scores by Number of Bins')"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>qcut</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>10</th>\n",
" <th>11</th>\n",
" <th>12</th>\n",
" <th>13</th>\n",
" <th>14</th>\n",
" <th>15</th>\n",
" <th>None</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>median</th>\n",
" <td>0.626</td>\n",
" <td>0.652</td>\n",
" <td>0.662</td>\n",
" <td>0.666</td>\n",
" <td>0.670</td>\n",
" <td>0.671</td>\n",
" <td>0.672</td>\n",
" <td>0.673</td>\n",
" <td>0.673</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.675</td>\n",
" <td>0.674</td>\n",
" <td>0.676</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>0.626</td>\n",
" <td>0.652</td>\n",
" <td>0.662</td>\n",
" <td>0.666</td>\n",
" <td>0.670</td>\n",
" <td>0.671</td>\n",
" <td>0.672</td>\n",
" <td>0.673</td>\n",
" <td>0.673</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.674</td>\n",
" <td>0.676</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.003</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" <td>0.004</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"qcut 2 3 4 5 6 7 8 9 10 11 \\\n",
"median 0.626 0.652 0.662 0.666 0.670 0.671 0.672 0.673 0.673 0.674 \n",
"mean 0.626 0.652 0.662 0.666 0.670 0.671 0.672 0.673 0.673 0.674 \n",
"std 0.003 0.004 0.004 0.004 0.004 0.004 0.004 0.004 0.004 0.004 \n",
"\n",
"qcut 12 13 14 15 None \n",
"median 0.674 0.674 0.675 0.674 0.676 \n",
"mean 0.674 0.674 0.674 0.674 0.676 \n",
"std 0.004 0.004 0.004 0.004 0.004 "
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame(d.groupby('qcut')['AUC'].agg(['median', 'mean', 'std']).round(3)).T"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### How the binning scheme impacts ROC scores\n",
"Binning schemes can generate ROC scores lower than the same model with no binning. In the hypothetical example above, the model results with no binning scheme achieved a median AUC of ~0.676, while the models with binary or ternary schemes achieved median AUC values of ~0.626 and ~0.652 respectively. This hypothetical example shows that binning schemes can have a negative impact on AUC values in comparison to the untransformed data. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"The findings above show that when reporting model accuracy with AUC, a reader should be wary of any transformation scheme applied to the model output since that scheme can change the AUC value. \n",
"\n",
"We explored the impact of transformation schemes through the COMPAS dataset provided by ProPublica, demonstrating that when the final output of the model is qualitative scores, incomparison to decile scores or raw scores, the model has lower AUC. We then analyzed a hypothetical example to determine what the general case is when differing binning schemes are applied, inferring that binning schemes with a small number of bins in our hypothetical example can result in lower AUC scores.\n",
"\n",
"In cases where the output of a model presented to a decision maker is transformed model results, there may be a discrepancy in the accuracy of those presented outputs in comparison to reported accuracy from evaluations. It is important to analyze risk assessments and other models in practice in context of what information is presented to decision makers."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Helpful Resources\n",
"- Tom Augspurger's [post on method chaining](https://tomaugspurger.github.io/method-chaining.html) was instrumental in refactoring the data reading steps"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python [Root]",
"language": "python",
"name": "Python [Root]"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment