Skip to content

Instantly share code, notes, and snippets.

@pb111

pb111/Data Preprocessing Project-Imbalanced Classes Problem.ipynb

Created April 15, 2019 06:50
Show Gist options
  • Save pb111/3f654b410b0a0e7256d42362566651f2 to your computer and use it in GitHub Desktop.
Save pb111/3f654b410b0a0e7256d42362566651f2 to your computer and use it in GitHub Desktop.
Download ZIP
Data Preprocessing Project-Imbalanced Classes Problem
Display the source blob
Display the rendered blob
Raw
Data Preprocessing Project-Imbalanced Classes Problem.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Preprocessing Project - Imbalanced Classes Problem\n",
"\n",
"\n",
"Imbalanced classes is one of the major problems in machine learning. In this data preprocessing project, I discuss the imbalanced classes problem. I present Python implementation to deal with this problem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Contents\n",
"\n",
"\n",
"I have divided this project into various sections which are listed below:-\n",
"\n",
"\n",
"\n",
"\n",
"1.\tIntroduction to imbalanced classes problem\n",
"\n",
"2.\tProblems with imbalanced learning\n",
"\n",
"3.\tExample of imbalanced classes\n",
"\n",
"4.\tApproaches to handle imbalanced classes\n",
"\n",
"5.\tPython implementation to illustrate class imbalance problem\n",
"\n",
"6.\tPrecision - Recall Curve\n",
"\n",
"7. Random over-sampling the minority class\n",
"\n",
"8.\tRandom under-sampling the majority class\n",
"\n",
"9.\tApply tree-based algorithms\n",
"\n",
"10.\tRandom under-sampling and over-sampling with imbalanced-learn\n",
"\n",
"11.\tUnder-sampling : Tomek links\n",
"\n",
"12.\tUnder-sampling : Cluster Centroids\n",
"\n",
"13.\tOver-sampling : SMOTE\n",
"\n",
"14.\tConclusion \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Introduction to imbalanced classes problem\n",
"\n",
"\n",
"Any real world dataset may come along with several problems. The problem of **imbalanced class** is one of them. The problem of imbalanced classes arises when one set of classes dominate over another set of classes. The former is called majority class while the latter is called minority class. It causes the machine learning model to be more biased towards majority class. It causes poor classification of minority classes. Hence, this problem throw the question of “accuracy” out of question. This is a very common problem in machine learning where we have datasets with a disproportionate ratio of observations in each class.\n",
"\n",
"\n",
"**Imbalanced classes problem** is one of the major problems in the field of data science and machine learning. It is very important that we should properly deal with this problem and develop our machine learning model accordingly. If this not done, then we may end up with higher accuracy. But this higher accuracy is meaningless because it comes from a meaningless metric which is not suitable for the dataset in question. Hence, this higher accuracy no longer reliably measures model performance. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Problems with imbalanced learning\n",
"\n",
"\n",
"The problem of imbalanced classes is very common and it is bound to happen. For example, in the above example the number of patients who do not have the rare disease is much larger than the number of patients who have the rare disease. So, the model does not correctly classify the patients who have the rare disease. This is where the problem arises.\n",
"\n",
"\n",
"The problem of learning from imbalanced data have new and modern approaches. This learning from imbalanced data is referred to as **imbalanced learning**. \n",
"\n",
"\n",
"Significant problems may arise with imbalanced learning. These are as follows:-\n",
"\n",
"\n",
"1.\tThe class distribution is skewed when the dataset has underrepresented data.\n",
"\n",
"2.\tThe high level of accuracy is simply misleading. In the previous example, it is high because most patients do not \n",
" have the disease not because of the good model. \n",
" \n",
"3.\tThere may be inherent complex characteristics in the dataset. Imbalanced learning from such dataset requires new \n",
" approaches, principles, tools and techniques. But, it cannot guarantee an efficient solution to the business problem.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Example of imbalanced classes\n",
"\n",
"\n",
"The problem of imbalanced classes may appear in many areas including the following:-\n",
"\n",
"\n",
"1.\tDisease detection\n",
"\n",
"2.\tFraud detection\n",
"\n",
"3.\tSpam filtering\n",
"\n",
"4.\tEarthquake prediction\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Approaches to handle imbalanced classes\n",
"\n",
"\n",
"In this section, I will list various approaches to deal with the imbalanced class problem. These approaches may fall under two categories – dataset level approach and algorithmic ensemble techniques approach. The various methods to deal with imbalanced class problem are listed below. I will describe these techniques in more detail in the following sections.\n",
"\n",
"\n",
"1.\tRandom Undersampling methods\n",
"\n",
"2.\tRandom Oversampling methods\n",
"\n",
"3. Tree-based algorithms\n",
"\n",
"4. Resampling with imbalanced-learn\n",
"\n",
"5. Under-sampling : Tomek links\n",
"\n",
"6. Under-sampling : Cluster Centroids\n",
"\n",
"7. Over-sampling : SMOTE\n",
"\n",
"\n",
"I have discussed these methods in detail in the readme document."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Python implementation to illustrate class imbalance problem\n",
"\n",
"\n",
"\n",
"Now, I will perform Python implementation to illustrate class imbalance problem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import Python libraries\n",
"\n",
"I will start off by importing the required Python libraries."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# import Python libraries\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import warnings\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import dataset\n",
"\n",
"\n",
"Now, I will import the dataset with the usual Python `read_csv()` function."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"data = 'C:/datasets/creditcard.csv'\n",
"\n",
"df = pd.read_csv(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset description\n",
"\n",
"\n",
"I have used the **Credit Card Fraud Detecttion** dataset for this project. I have downloaded this project from the Kaggle website. This dataset can be found at the following url-\n",
"\n",
"\n",
"https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
"\n",
"\n",
"This dataset contains transactions made by european credit card holders in September 2013. It represents transactions that occurred in two days. We have 492 fraudulent transactions out of total 284,807 transactions. This dataset is highly unbalanced, the positive class (frauds) account for only 0.172% of all transactions.\n",
"\n",
"\n",
"Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount, this feature can be used for example-dependant cost-senstive learning. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise. So, our target variable is `Class` variable.\n",
"\n",
"\n",
"\n",
"Given the class imbalance ratio, it is recommended to measure the accuracy using the `Area Under the Precision-Recall Curve (AUPRC)`. Confusion matrix accuracy is not meaningful for unbalanced classification.\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exploratory data analysis\n",
"\n",
"\n",
"Now, I will conduct exploratory data analysis to gain an insight into the dataset."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(284807, 31)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check shape of dataset\n",
"\n",
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that there are 284,807 instances and 31 columns in the dataset."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Time</th>\n",
" <th>V1</th>\n",
" <th>V2</th>\n",
" <th>V3</th>\n",
" <th>V4</th>\n",
" <th>V5</th>\n",
" <th>V6</th>\n",
" <th>V7</th>\n",
" <th>V8</th>\n",
" <th>V9</th>\n",
" <th>...</th>\n",
" <th>V21</th>\n",
" <th>V22</th>\n",
" <th>V23</th>\n",
" <th>V24</th>\n",
" <th>V25</th>\n",
" <th>V26</th>\n",
" <th>V27</th>\n",
" <th>V28</th>\n",
" <th>Amount</th>\n",
" <th>Class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.0</td>\n",
" <td>-1.359807</td>\n",
" <td>-0.072781</td>\n",
" <td>2.536347</td>\n",
" <td>1.378155</td>\n",
" <td>-0.338321</td>\n",
" <td>0.462388</td>\n",
" <td>0.239599</td>\n",
" <td>0.098698</td>\n",
" <td>0.363787</td>\n",
" <td>...</td>\n",
" <td>-0.018307</td>\n",
" <td>0.277838</td>\n",
" <td>-0.110474</td>\n",
" <td>0.066928</td>\n",
" <td>0.128539</td>\n",
" <td>-0.189115</td>\n",
" <td>0.133558</td>\n",
" <td>-0.021053</td>\n",
" <td>149.62</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.0</td>\n",
" <td>1.191857</td>\n",
" <td>0.266151</td>\n",
" <td>0.166480</td>\n",
" <td>0.448154</td>\n",
" <td>0.060018</td>\n",
" <td>-0.082361</td>\n",
" <td>-0.078803</td>\n",
" <td>0.085102</td>\n",
" <td>-0.255425</td>\n",
" <td>...</td>\n",
" <td>-0.225775</td>\n",
" <td>-0.638672</td>\n",
" <td>0.101288</td>\n",
" <td>-0.339846</td>\n",
" <td>0.167170</td>\n",
" <td>0.125895</td>\n",
" <td>-0.008983</td>\n",
" <td>0.014724</td>\n",
" <td>2.69</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.0</td>\n",
" <td>-1.358354</td>\n",
" <td>-1.340163</td>\n",
" <td>1.773209</td>\n",
" <td>0.379780</td>\n",
" <td>-0.503198</td>\n",
" <td>1.800499</td>\n",
" <td>0.791461</td>\n",
" <td>0.247676</td>\n",
" <td>-1.514654</td>\n",
" <td>...</td>\n",
" <td>0.247998</td>\n",
" <td>0.771679</td>\n",
" <td>0.909412</td>\n",
" <td>-0.689281</td>\n",
" <td>-0.327642</td>\n",
" <td>-0.139097</td>\n",
" <td>-0.055353</td>\n",
" <td>-0.059752</td>\n",
" <td>378.66</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>-0.966272</td>\n",
" <td>-0.185226</td>\n",
" <td>1.792993</td>\n",
" <td>-0.863291</td>\n",
" <td>-0.010309</td>\n",
" <td>1.247203</td>\n",
" <td>0.237609</td>\n",
" <td>0.377436</td>\n",
" <td>-1.387024</td>\n",
" <td>...</td>\n",
" <td>-0.108300</td>\n",
" <td>0.005274</td>\n",
" <td>-0.190321</td>\n",
" <td>-1.175575</td>\n",
" <td>0.647376</td>\n",
" <td>-0.221929</td>\n",
" <td>0.062723</td>\n",
" <td>0.061458</td>\n",
" <td>123.50</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>-1.158233</td>\n",
" <td>0.877737</td>\n",
" <td>1.548718</td>\n",
" <td>0.403034</td>\n",
" <td>-0.407193</td>\n",
" <td>0.095921</td>\n",
" <td>0.592941</td>\n",
" <td>-0.270533</td>\n",
" <td>0.817739</td>\n",
" <td>...</td>\n",
" <td>-0.009431</td>\n",
" <td>0.798278</td>\n",
" <td>-0.137458</td>\n",
" <td>0.141267</td>\n",
" <td>-0.206010</td>\n",
" <td>0.502292</td>\n",
" <td>0.219422</td>\n",
" <td>0.215153</td>\n",
" <td>69.99</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
" Time V1 V2 V3 V4 V5 V6 V7 \\\n",
"0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 \n",
"1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 \n",
"2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461 \n",
"3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609 \n",
"4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941 \n",
"\n",
" V8 V9 ... V21 V22 V23 V24 \\\n",
"0 0.098698 0.363787 ... -0.018307 0.277838 -0.110474 0.066928 \n",
"1 0.085102 -0.255425 ... -0.225775 -0.638672 0.101288 -0.339846 \n",
"2 0.247676 -1.514654 ... 0.247998 0.771679 0.909412 -0.689281 \n",
"3 0.377436 -1.387024 ... -0.108300 0.005274 -0.190321 -1.175575 \n",
"4 -0.270533 0.817739 ... -0.009431 0.798278 -0.137458 0.141267 \n",
"\n",
" V25 V26 V27 V28 Amount Class \n",
"0 0.128539 -0.189115 0.133558 -0.021053 149.62 0 \n",
"1 0.167170 0.125895 -0.008983 0.014724 2.69 0 \n",
"2 -0.327642 -0.139097 -0.055353 -0.059752 378.66 0 \n",
"3 0.647376 -0.221929 0.062723 0.061458 123.50 0 \n",
"4 -0.206010 0.502292 0.219422 0.215153 69.99 0 \n",
"\n",
"[5 rows x 31 columns]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# preview of the dataset\n",
"\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `df.head()` function gives the preview of the dataset. We can see that there is a `Class` column in the dataset which is our target variable.\n",
"\n",
"\n",
"I will check the distribution of the `Class` column with the `value_counts()` method as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 284315\n",
"1 492\n",
"Name: Class, dtype: int64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check the distribution of Class column\n",
"\n",
"df['Class'].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So, we have 492 fraudulent transactions out of total 284,807 transactions in the dataset. The `Class` column takes value `1 for \n",
"fraudulent transactions` and `0 for non-fraudulent transactions`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I will find the percentage of labels 0 and 1 within the `Class` column."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 0.998273\n",
"1 0.001727\n",
"Name: Class, dtype: float64"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# percentage of labels within the Class column\n",
"\n",
"df['Class'].value_counts()/np.float(len(df))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the `Class` column is highly imbalanced. It contains 99.82% labels as `0` and 0.17% labels as `1`. \n",
"\n",
"Now, I will plot the bar plot to confirm this."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0x6b5b677400>"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAC15JREFUeJzt3V+Infldx/H3ZxOjF60tmFFq/jSBpmgUYWWIhV640orJCslNkQSKWpbmxijSIkaUVeONthcFIf4JWqsFN8Ze6FAjuahbBDU1s7QuJiE6xGqGiDttlwUpmsZ+vZixHk5Ocp5JTnKSb94vGDjP8/xyzpcwefPMc84zSVUhSerlmXkPIEmaPeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJamhrfN64e3bt9eePXvm9fKS9ER65ZVXvlRVC9PWzS3ue/bsYXl5eV4vL0lPpCT/OmSdl2UkqSHjLkkNGXdJasi4S1JDxl2SGpoa9yQfT/Jakn+8y/Ek+c0kK0leTfL9sx9TkrQZQ87cPwEcvMfxQ8C+ja/jwG8/+FiSpAcxNe5V9dfAV+6x5AjwR7XuIvDWJG+b1YCSpM2bxU1MO4AbI9urG/v+fXxhkuOsn92ze/fuGbz0w7fn5F/Me4RWvvjrPzrvEaSnwizeUM2EfRP/1+2qOlNVi1W1uLAw9e5ZSdJ9mkXcV4FdI9s7gZszeF5J0n2aRdyXgB/f+NTMu4A3quqOSzKSpEdn6jX3JC8BzwHbk6wCvwx8E0BV/Q5wHngeWAG+CnzgYQ0rSRpmatyr6tiU4wX81MwmkiQ9MO9QlaSGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLU0KC4JzmY5FqSlSQnJxzfneTlJJ9P8mqS52c/qiRpqKlxT7IFOA0cAvYDx5LsH1v2S8C5qnoWOAr81qwHlSQNN+TM/QCwUlXXq+oWcBY4MramgG/dePwW4ObsRpQkbdaQuO8Aboxsr27sG/UrwPuTrALngZ+e9ERJjidZTrK8trZ2H+NKkoYYEvdM2Fdj28eAT1TVTuB54JNJ7njuqjpTVYtVtbiwsLD5aSVJgwyJ+yqwa2R7J3dednkBOAdQVX8HfAuwfRYDSpI2b0jcLwH7kuxNso31N0yXxtb8G/AegCTfzXrcve4iSXMyNe5VdRs4AVwArrL+qZjLSU4lObyx7MPAB5P8A/AS8JNVNX7pRpL0iGwdsqiqzrP+RunovhdHHl8B3j3b0SRJ98s7VCWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJamhQXFPcjDJtSQrSU7eZc2PJbmS5HKSP57tmJKkzdg6bUGSLcBp4IeBVeBSkqWqujKyZh/wC8C7q+r1JN/+sAaWJE035Mz9ALBSVder6hZwFjgytuaDwOmqeh2gql6b7ZiSpM0YEvcdwI2R7dWNfaPeCbwzyd8kuZjk4KQnSnI8yXKS5bW1tfubWJI01ZC4Z8K+GtveCuwDngOOAb+X5K13/KGqM1W1WFWLCwsLm51VkjTQkLivArtGtncCNyes+fOq+lpV/QtwjfXYS5LmYEjcLwH7kuxNsg04CiyNrfkz4IcAkmxn/TLN9VkOKkkabmrcq+o2cAK4AFwFzlXV5SSnkhzeWHYB+HKSK8DLwM9V1Zcf1tCSpHub+lFIgKo6D5wf2/fiyOMCPrTxJUmaM+9QlaSGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLU0KC4JzmY5FqSlSQn77HufUkqyeLsRpQkbdbUuCfZApwGDgH7gWNJ9k9Y92bgZ4DPzXpISdLmDDlzPwCsVNX1qroFnAWOTFj3a8BHgP+a4XySpPswJO47gBsj26sb+74hybPArqr69L2eKMnxJMtJltfW1jY9rCRpmCFxz4R99Y2DyTPAx4APT3uiqjpTVYtVtbiwsDB8SknSpgyJ+yqwa2R7J3BzZPvNwPcCn03yReBdwJJvqkrS/AyJ+yVgX5K9SbYBR4Gl/ztYVW9U1faq2lNVe4CLwOGqWn4oE0uSppoa96q6DZwALgBXgXNVdTnJqSSHH/aAkqTN2zpkUVWdB86P7XvxLmufe/CxJEkPwjtUJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqaFBcU9yMMm1JCtJTk44/qEkV5K8muQzSd4++1ElSUNNjXuSLcBp4BCwHziWZP/Yss8Di1X1fcCngI/MelBJ0nBDztwPACtVdb2qbgFngSOjC6rq5ar66sbmRWDnbMeUJG3GkLjvAG6MbK9u7LubF4C/nHQgyfEky0mW19bWhk8pSdqUIXHPhH01cWHyfmAR+Oik41V1pqoWq2pxYWFh+JSSpE3ZOmDNKrBrZHsncHN8UZL3Ar8I/GBV/fdsxpMk3Y8hZ+6XgH1J9ibZBhwFlkYXJHkW+F3gcFW9NvsxJUmbMTXuVXUbOAFcAK4C56rqcpJTSQ5vLPso8CbgT5N8IcnSXZ5OkvQIDLksQ1WdB86P7Xtx5PF7ZzyXJOkBeIeqJDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGBsU9ycEk15KsJDk54fg3J/mTjeOfS7Jn1oNKkoabGvckW4DTwCFgP3Asyf6xZS8Ar1fVO4CPAb8x60ElScMNOXM/AKxU1fWqugWcBY6MrTkC/OHG408B70mS2Y0pSdqMrQPW7ABujGyvAj9wtzVVdTvJG8C3AV8aXZTkOHB8Y/M/k1y7n6E10XbG/r4fR/FnuqfRE/G9+QR5+5BFQ+I+6Qy87mMNVXUGODPgNbVJSZaranHec0jj/N6cjyGXZVaBXSPbO4Gbd1uTZCvwFuArsxhQkrR5Q+J+CdiXZG+SbcBRYGlszRLwExuP3wf8VVXdceYuSXo0pl6W2biGfgK4AGwBPl5Vl5OcAparagn4feCTSVZYP2M/+jCH1kRe7tLjyu/NOYgn2JLUj3eoSlJDxl2SGjLuktTQkM+56zGT5LtYvyt4B+v3E9wElqrq6lwHk/TY8Mz9CZPk51n/FRAB/p71j6oGeGnSL3WT9HTy0zJPmCT/BHxPVX1tbP824HJV7ZvPZNK9JflAVf3BvOd4Wnjm/uT5OvCdE/a/beOY9Lj61XkP8DTxmvuT52eBzyT5Z/7/F7rtBt4BnJjbVBKQ5NW7HQK+41HO8rTzsswTKMkzrP8q5h2s/6NZBS5V1f/MdTA99ZL8B/AjwOvjh4C/rapJP3XqIfDM/QlUVV8HLs57DmmCTwNvqqovjB9I8tlHP87TyzN3SWrIN1QlqSHjLkkNGXdJasi4S1JD/wvQ9V7if0uClQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# view the distribution of percentages within the Class column\n",
"\n",
"\n",
"(df['Class'].value_counts()/np.float(len(df))).plot.bar()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above bar plot confirms our finding that the `Class` variable is highly imbalanced. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Misleading accuracy for imbalanced classes\n",
"\n",
"\n",
"Now, I will demonstrate that accuracy is misleading for imbalanced classes. Most of the machine learning algorithms are designed to maximize the overall accuracy by default. But this maximum accuracy is misleading. We can confirm this with the following analysis.\n",
"\n",
"\n",
"I will fit a very simple Logistic Regression model using the default settings. I will train the classifier on the imbalanced dataset."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# declare feature vector and target variable\n",
"\n",
"X = df.drop(['Class'], axis=1)\n",
"y = df['Class']"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# import Logistic Regression classifier\n",
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"\n",
"# instantiate the Logistic Regression classifier\n",
"logreg = LogisticRegression()\n",
"\n",
"\n",
"# fit the classifier to the imbalanced data\n",
"clf = logreg.fit(X, y)\n",
"\n",
"\n",
"# predict on the training data\n",
"y_pred = clf.predict(X)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I have trained the model. I will check its accuracy."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy : 99.90%\n"
]
}
],
"source": [
"# import the accuracy metric\n",
"from sklearn.metrics import accuracy_score\n",
"\n",
"\n",
"# print the accuracy\n",
"accuracy = accuracy_score(y_pred, y)\n",
"\n",
"print(\"Accuracy : %.2f%%\" % (accuracy * 100.0))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Accuracy paradox\n",
"\n",
"\n",
"Thus, our Logistic Regression model for credit card fraud detection has an accuracy of 99.90%. It means that for each 100 transactions it classified, 99.90% were classified as genuine.\n",
"\n",
"\n",
"It does not mean that our model performance is excellent. I have previously shown that our dataset have 99.90% genuine transactions and 0.1% fraudulent transactions. Our Logistic Regression classifier predicted all transactions as genuine. \n",
"Then we have a accuracy of 99.90% because it correctly classified 99.90% transactions as genuine.\n",
"\n",
"\n",
"Thus, this algorithm is 99.90% accurate. But it was horrible at classifying fraudulent transactions. So, we should have other ways to measure the model performance. One such measure is confusion matrix described below."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Confusion matrix\n",
"\n",
"\n",
"A confusion matrix is a tool for summarizing the performance of a classification algorithm. A confusion matrix will give us a clear picture of classification model performance and the types of errors produced by the model. It gives us a summary of correct and incorrect predictions broken down by each category. The summary is represented in a tabular form.\n",
"\n",
"\n",
"Four types of outcomes are possible while evaluating a classification model performance. These four outcomes are described below:-\n",
"\n",
"\n",
"**True Positives (TP)** – True Positives occur when we predict an observation belongs to a certain class and the observation actually belongs to that class.\n",
"\n",
"\n",
"**True Negatives (TN)** – True Negatives occur when we predict an observation does not belong to a certain class and the observation actually does not belong to that class.\n",
"\n",
"\n",
"**False Positives (FP)** – False Positives occur when we predict an observation belongs to a certain class but the observation actually does not belong to that class. This type of error is called **Type I error.**\n",
"\n",
"\n",
"\n",
"**False Negatives (FN)** – False Negatives occur when we predict an observation does not belong to a certain class but the observation actually belongs to that class. This is a very serious error and it is called **Type II error.**\n",
"\n",
"\n",
"\n",
"These four outcomes are summarized in a confusion matrix given below.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Confusion matrix:\n",
" [[284240 75]\n",
" [ 203 289]]\n"
]
}
],
"source": [
"# import the metric\n",
"from sklearn.metrics import confusion_matrix\n",
"\n",
"\n",
"# print the confusion matrix\n",
"cnf_matrix = confusion_matrix(y, y_pred)\n",
"\n",
"\n",
"print('Confusion matrix:\\n', cnf_matrix)\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Interpretation of confusion matrix\n",
"\n",
"\n",
"Now, I will interpret the confusion matrix.\n",
"\n",
"\n",
"- Out of the total 284315 transactions which were predicted genuine, the classifier predicted correctly 284240 of them. It means that the classifer predicted 284240 transactions as genuine and they were actually genuine. Also, it predicted 75 transactions as genuine but it were fraudulent. So, we have `284240 True Positives(TP)` and `75 False Positives(FP)`.\n",
"\n",
"\n",
"- Out of the total 492 transactions which were not predicted as genuine, the classifier predicted correctly 289 of them. It means that the classifer did not predict 289 transactions as genuine and they were actually not genuine. SO, they were fraudulent. Also, it did not predict 203 transactions as genuine but they were genuine. So, we have `289 True Negatives(TN)` and `203 False Negatives(FN)`.\n",
"\n",
"\n",
"\n",
"- So, out of all the 284807 transactions, the classifier correctly predicted 284529 of them. Thus, we will get the accuracy of\n",
"`(284240+289)/(284240+289+75+203) = 99.90%.`\n",
"\n",
"\n",
"\n",
"- But this is not the true picture. The confusion matrix allows us to obtain a true picture of the performance of the algorithm. The algorithm tries to predict the fraudulent transactions out of the total transactions. It correctly predicted 289 transactions as fraudulent out of all the 284807 transactions. In this case the accuracy becomes `(289/284807)=0.10%.`\n",
"\n",
"\n",
"\n",
"- Moreover, we have `203+289=492` transactions as fraudulent. The algorithm is correctly classifying 289 of them as fraudulent while it fails to predict 203 transactions which were fraudulent. In this case the accuracy becomes `(289/492)=58.74%.`\n",
"\n",
"\n",
"So, we can conclude that the accuracy of 99.90% is misleading because we have imbalanced classes. We need more subtle way to evaluate the performance of the model.\n",
"\n",
"\n",
"There is another metric called `Classification Report` which helps to evaluate model performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Classification report\n",
"\n",
"\n",
"\n",
"**Classification report** is another way to evaluate the classification model performance. It displays the **precision**, **recall**, **f1** and **support** scores for the model. I have described these terms in later sections.\n",
"\n",
"\n",
"\n",
"We can plot a classification report as follows:-"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Classification Report:\n",
"\n",
" precision recall f1-score support\n",
"\n",
" 0 1.00 1.00 1.00 284315\n",
" 1 0.79 0.59 0.68 492\n",
"\n",
" micro avg 1.00 1.00 1.00 284807\n",
" macro avg 0.90 0.79 0.84 284807\n",
"weighted avg 1.00 1.00 1.00 284807\n",
"\n"
]
}
],
"source": [
"# import the metric\n",
"from sklearn.metrics import classification_report\n",
"\n",
"\n",
"# print classification report\n",
"print(\"Classification Report:\\n\\n\", classification_report(y, y_pred))"
]
},
{
"attachments": {
"Precision.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAAA4CAYAAAAGnO/aAAAFBElEQVR42uydj5G7KBTH2ZttgCvBLcEtwZTglcCVkJRgSnBLyJZgStASYgmxhNw487h5ww8QMUaSfD8zzmSUh2D4Cjz585cAACTJJx4BSJBCCNFE2p6FEDv6PcaR0THSCSEGI3xG50e7Hzx6AKbFeRVCKCGEZOfH3zc6pEPQF0t82sbFnu53ssQLAGAoOsRMoUkS2VxxaoGOYSo8fgDcjALJI8Q5UkeKk9fKWQoPAQ4hkCpdpN0QKa7B6IdCnABY6BeKWm54fwDelpAmaoxNSWFOeMQApCPOnLy8TUre2g/81+AJxTm37Gqbo0OYQ4rfOTEIAbwbh2dJKBxCAECcAACIEwCIEwAAcQIAcQKQLNLxGwCwEQ0dVzag4MrOVx67C7O5TYRPCv4hV88E8E1OTXli6pi2lsZFfkfG0VI83xhfCVJtOviGOxXsrZVS06KIHNrF0fYFigFIlalCrhKdmLqnAcyxlBQHAE8rzuQmpgLwasR6a5ObmAoAxPkncJwAsAKfC/pmI79MnIrO59Ts/WD9t4qm6/AZAQWFH1gN7PIC5xTXwML3LGxO99CeZnM6UWVplneUhn/oXOOx1+nbG60GSfnqjbRWdC1nyzTq55JRHAfLMo0ALOpzSvrk4PLWnsi2ZNfN+BR5fHNLvKaQtPMpt5yvHPc2hVk40tkE2As2GTcPPM/jUpbrrWMhKgCCxVkZR02FTnlsKyZOXtOWrEDfHHGUhpMp83iFW8syiJVFXK6XTGYRp80+o/tUnvxeLX1vHdfeYzMH82P63AO+gRcSZ+xnkspR02lqT2GRRoGuPd8dWzqmxHWleGy1/D7Avp4o3PoFUjvikp5nBMDd+5whdBP9VV/tK42wZ0uY0FFABxKOong61l89BtgXE46v3ghnMjzpyxlsy8cWy5RIJhqXmMywS/ghAZUkoIK9HHYB4gltDr5SsxFrSyXAFrNShpXC+kQz1pj/CiG+hBB/0+8s0CkzPDCtAGwqzt+JZiCvhabCysA+sCmiH/qEUs5Ir6sPnRvh1gIOIYhzdY4kkNLTxytYk1R4wqqA+5UOEZ8Da7vjRBpKI9xafFFzM/bAYJEXEKe8U39PehwoByrUylJjFkyUHTVBbbtO5TMKnK35ug+c9tZTGkrHd86SrvcLancAvB1/23xO7SXdBcSlmNNFF2otxM7RdN0b/bXBUQMVTJw9hetZU9K8t55zeqDPLQdjNJJ5r5zlX7K07yzpDR0hZMZ1YGI2n9EORREAEEq1oH9bsReVuRpBy1YwaNj1OrF5tDH5b4zKak7e4RMAswpna+mzu3aXlhS2jdgjUzIxpDZHOPfspm3m++q4Hpp3hWIHQvvprprsNnNYpAhcpeJkGfqZAqGb714X2Kead5AgpwWFNXZ3aT22+nKH9F8eLE7BZkYtyfv/Q1KxNCZwscWnFz59cClb9OG6BfcdWDMa4gSribN/whdDDMoiznfJO0iUtXaX9k2zi0nj2vmtVsg7nEIgOXEWd/bWri1OOaNfG5V3bJ4LHo1NfDkbjHFOOO2NkWZ5h7zrZWtSzzt4g5qzeGAa18xvMWM1i6jFzuEQAs9KPTFaRwSOYoolxS1JAEDNSag1a04AIM71xYVmLQCvBMQJAMQJXoR32116aX6xGzdYlTxgd2nlsbtYbFTifU7bfMzLjJ2xffYYBQTemhuatQCkCQaQAwBQcwLwVvwXAAD//wtFB2TtUkP9AAAAAElFTkSuQmCC"
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Precision\n",
"\n",
"\n",
"Precision can be defined as the percentage of correctly predicted positive outcomes out of all the predicted positive outcomes.\n",
"It can be given as the ratio of true positives (TP) to the sum of true and false positives (TP + FP). \n",
"\n",
"\n",
"Mathematically, precision is defined as\n",
"\n",
"\n",
"![Precision.png](attachment:Precision.png)\n",
"\n",
"\n",
"\n",
"Precision is more concerned with the positive class than the negative class.\n"
]
},
{
"attachments": {
"Recall.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAMoAAAA4CAYAAAC8P2e6AAAEsElEQVR42uydi23jOBCGZw/XAFvQlcArQVuCWnBKWJdAl2CX4JQQlyCVYJUQl5BDgBEwmONDIuWYsv8PEHZtkZKp8Ndwhq+/CACQ5G88ApDgg4jazLy/iehCRDsi6oio4eObgYhuKn3D6Q9ENOLRg60JpeeKLjFE9MWHUd9/C+OT/w3l8fF93vH5HR492BK9EoIkVul3XOmX5Jk4c5qulocAHwWkuHmaSHN4L7jnlPcPhAK2wlAgMFOQ9xsLoYCtMD5AZGvcG4BqmONv5OSBQw8glESels+7mgqKfhTwKHxCaNg/mfpfAIBF2RJw5gGAUACAUACAUACAUACAUMCLYAL/XzsPAJtjx8PuP3gY/RTq/RTf20i+q8hz5e820fv+S312XNDYBBvDnUGnysfiWC7PVJZf6g/XBc4BsMishjqGtja55hzp4DpvsfML1EWqB/VY2+SaAC5SDgehgHs78xdR2QCAUAJMfkuDxwgglDCNsiwAPC0lw+x3bFX2kTQt+zDS+kwRM43lOdJyjvboSduoIILlyNwef05QkzPfcfz7mOg0mpaskXF1w6t6OE/aL08MXq/kMUXc9H3PfF048+BhQnHquAYqq7YOofBxx+caYSFCM9p6FpsWlPNYmVAEbk2hNOK55BxXVKvXsSjyDR4Sy1GJgZRV+BJL0Uxp24BQetWU+wosYxMSGywKeJiPsheW5S1gNSjRIWlUWl9Q4F/1+SJ60Q1bLvvE44Yg4jo45FoUEmN8cvLmpp2aP0cxTsjCooB7Uzp6OPQmX7Ky4JK0lptiAy9AcFph7SgA7i4UCvgW75Fz0jLMSWuU7xMKL5Ny+u8BnHkIZTGDqvCysh/YUnQRcU1pT8pXiVV6GxixbAMiXJuRfaTc4x9UuecSyhzn+OSppJ2oUHv+vPNU4lbkHzggsPOk1cK4cF49Aajj65iMZiIASebMRxn5+O3J33Go9iKaUYMSxR/li9wCUYRWCGXkdKNaFd3w9ay6z4Hv5dT1nYiKjULAln97O6OMAIANUOITyRegnmHYi1mJ07lzhXOMcsot/V09I7OP3EvP3OxrWlEfpCvK2RPsCO14NVlxX4dvaserVkzrramp2gTKqpvpx0iHtOOKn1rX2BRuxwceVEH6GW9bClSMdmEeigwTqsWypHCR/jQrrmMT11gt6gV+Rii50xhy+5fkblel0cNHhMMPEaszCP/1uEbUC9RDboUfMyv6TQm1VOg/TWorvT0/U7vEakIodXMrWOlmjRELW9nxSgcgUlb4TVhNOOsv4uivve1CJwIIa/y+e5fVzLQMzvM5FAWDjwKiGNEv9rZRazKXRU0w7Lj1ujiPSAyPmDhV/Ls/lA/UpIbBJ5pgPb8c3jHAFk0vncf94O+7Z1kNR69yml6xJhiaXmBVmhkjA+aOHsgNdpQuKjKrCQahgBJSo6lpxohqWkEspdG5ZBQMQgHPQKlPld0RCeCjbMFHKQle+OhDU9xhUQD4fxPMQCjPQ8mOV+ZJyr0kj50xIngoCDWDinCBHa+u4vtYPjkfJZa+pqaX/t2p3b18+ZfOMelR1cBP+1BPAZpe4J6MeAQAvBCwKADM4L8AAAD//08Owr0ITzbtAAAAAElFTkSuQmCC"
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Recall\n",
"\n",
"\n",
"Recall can be defined as the percentage of correctly predicted positive outcomes out of all the actual positive outcomes.\n",
"It can be given as the ratio of true positives (TP) to the sum of true positives and false negatives (TP + FN). **Recall** is also called **Sensitivity**.\n",
"\n",
"\n",
"Mathematically, recall can be given as\n",
"\n",
"\n",
"\n",
"![Recall.png](attachment:Recall.png)\n",
"\n"
]
},
{
"attachments": {
"f1-score.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAA8CAYAAADmDDgpAAAJmElEQVR42uydjbGbOhOGlUwa4CuBlEBKwCVwSsAl4BI4JeASTAlQgl2CKcGUwDe5s5q7o6tfjAzY7zPD5AQECAm93tXP8lMAAAAIBuIJQBi1EGISQhQzzy/o/BpF+dI66wzl3lmOQTwBWJBM+ffV54NwTrTpOFiOAQAWFs+/Fkoy8/yEzod4vh6bdRlsef5CeQIQxI22uYywdN4DuO0AAADxBACA1wC3HaxFSSPPGbnBR9onqF8wI/e2V87r2PED+zuntNwlzuiaI9uXUJpRue7ftBXtl9sghGhZmprd61vjfteKe/6XlK7Rs2c2nS/TV5o8f1N+1LzI8vvSlN9Zyf+SrFkP/B1K2fVkOd3QvMAncBFC3IUQjbI/p078UnNOxo5JwZpo4w3roRmYqWh/oqTV3SvR7MsMgws15Vn3fLnH+fLYXZNn2/6JhEydOiWnRKUR627Neqg05Z05ppEtOmAEwNrIeZO60euGjuWGl/2qiEXBrDdbY+BiLdM2mnRXauA+DW0y3CvV5F93fkr3qi3l9NCI4UTPkxjyWUWuv7Xq4a4RX/nOXF8hnujzBFth1OxrWYPUMShpW2aVCHJbheG6hZK29biHK/+NpjEPnteo6FxTns90vDI8z2g4LwnsSulmTKNaox4GwzOPr5oGhj5PsGV6Jp7HAGErmBiY3M1ESdtr0v0JyOuJxLOka90of2dP8cwdzzQo6WLQMCv3MFM8X1UPB+U6KesDFhBPAMKtJ57+FOHawmIZDmxAKGfCcbBYhtxt9yFmH+aZ8tsuXG+x6qFkP1Y9DRZVkX9gIJ5gV6I5zugCSCKkdYlazyynhIS0Jovua6F8jBHL+2iw8J/piolVDw0b6R/XeDnR5wm2jLQgQi2h1sPFTT3TJp7WXq0RgzOJZhGQ58zi4oqIU49iEKseCrI4jx7CWUI8wSe65jlzJ0P4pkZVOCwXmVZY0lae4lcYnqH3tIxc+SiUdHsgVj3IH5KbR7dGCvEE706tsRikdRE66Xkgd64wWB4164cb2AR93ZzOLEC8G0OjP3vm+cgWDqj5kINmw47qNFY99AarsmTlk9IWrbx+oM2CDYhmRSOqBbNC5bQd3QqjlFkUcmTbtFqn1DQi3Qh4yqbLjBrrSb33yO57o7mFJ3qGUXHfv9mz5kwce3JXz5p8+KwwUq8l86Mro3aGBW9jzXqQXknJViFJS7Rlq69kftYuq/+88LVjUi/YFhU18I4mGF+eCOi79Ls0oXrA3vj5hHt1UvqlwHZp6Jf1D7lRv+mX92JwNQEAC5NYlsuBbVKSSJpEdYo5IgnLE4B/GyJe9H1xdUx/mSxrgSGeACz4oncohl0xGQJK8OPTSj/EHbv/Hf3n4J3p8ILvjjuJU7Yx8QRg1/gsz0xZn1hOUwvUQaMlLVuVkQVOVV3OUgk622umGPhO/eDBeROaxlWxfKnBa3NlWorp/msjA9Xq5krKvuseTQGAeOSRB4tswWR1E6h1AVYbJW1ocFl5v0lZMeIT4DWhvsM9WebNk98gBwB4EHuwyBZMtlb+bwpc+mBBU+cGl601glIoS8NMI9Qh0bvvTJTnbM8uO7MFnwUALGwZ3iNe/2EIJisUi9QWXfzKRo4bh8iYxKN29BHarpu8KHr3ElxnjLLPDZYLwEfTOUbaMzqeP2nZTmxgqjSI7BRg2bms3btBPG0iP7FVVuq2h2+hyE8VJDO8gylwxsWEDdsbbd1cy7C2uNWFxSKsLZlRLcxGcWnVRu47Ojx5iucUKJ57H51uLBa+bx9pCVsCAD8Sz4EFk3imLLq2uvE06j3lwMxlhuX58BTPx0zLc49Uhj7OFK84AHHIHf2ALvH04eK4N083OYKmcivJtbqmCRTPxuM5tzhgVFoGhzB3F4BI1AGu8lzxnCyu5EMjpo1FbIXHZ09rgwi5njVlg1um+2/NrS0do+oYcQcgEo1nR+mz4nkxuJq6eZ66vrdUEcPSMc+ztIhq4hCjh+H+W7PiMsprZ9iusDwBCMc3GPKVVqGcPATwMHPFik8wWZOFN7BPvd40ghaywoh/AnZgwW6F4bqCXXvc4GcSHh6DQ6edfd4BgN3guwoF4erAu1A/ufqq2Mm0tU+v485QT53l2D/4BEPG+mfwiWTKv68+H8TnZPGmDy5P2xQY5EKi+T/24aURZQ0+rGEVT3RnyPNaFOWmuc08ZnW/a2a+ph7mb80GfWpMpAZgl9w/8Jlt3SvBXS85m1CNiDsAfJaQQDw9jpnc9h59nAAAYOYXigBEhgeZvgkhjqxLJ6H9J8P32eXxA/s710ybUwNjCxZAW+2rz9jUNbkNSt9kze71rRk44JYID4bdUt74tLdvy7fMfb/PnrHy+9KU33lDfatr1ht/51J2PVmuNzRHsEcu1J+mrmbKLcFGeOzUmrlRrsDUghqaOsfVtLgi0ezLDC6bLWh37nG+mBGkO2MRfdRutGKh+K5Luu1r1lulqZ/MMe1s0T5PAJbGtnLLFi9A/bonD0ztWoLLxdoW/PmqCRBjaji2oN25x/lzg3TL8ImJIZ/VhsRzzXq7GxaGNJb4tbPE8yfaNHgxuilvLWtgOgYlbcusDGH5blTLrlkp97Ldw5V/XUi/wfMaFZ1ryvOZjleG5zFNGUw2WNdr1NtgKKNx6Tm36PMEW6Bn4nkMELaCuXUm9zFR0uoGQv8E5PVE4lnSteSy4LOneOaOZxqUdEtz93DxbdZnyPLrNertoFwnZX3GAuIJ3pVkZvpThGsLi2U4sAGhnAnBwWMxiW/fZKwYq789hPPHi+o5Vr2V7Metp8GiaukfJIgn2JJojjO6AJIIaV2ixqfyJSSkNVmkXwvlY3zzrptY9dawkf6oZYg+T7AFpEUQOt2m9XBxU8+0iae1V2sa95lEswjIsy1It3jzZZ2x6q0gi/PoIZwlxBO8g2ueM5c4hG9qJIXDEhFsrbkpbeUpfoXhGXpPS8eVj0JJ947Eqjf5w3Pz6AZJIZ5gb+gCW0trIXQS80DuWWEJbH1iaY/sfqq1lwWId2NoxGfPPB/ZwgE1H3LQbHjjdyBWvfUGq7Jk5SkDpj9dvj/QlsELRbOiEdKCWaFy2o5uhRH/MoAc2Tat1ik1jUI3Aq4LYs2tIfXeI7vvzTNot5xIn7G8t4q4hqwwUq8l86Mro3aGBc95dsBozXoTLEj6yPbfqFxqtlLrtELZAjBbPCcUw+ZBHcFtBwDMdKkBxBMAEMhvFAHEEwAAANgtJfuQlgxugSg1AAAAANx2AAAAEE8AAIjF/wMAAP//asJejAc3Jj0AAAAASUVORK5CYII="
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"### f1-score\n",
"\n",
"\n",
"**f1-score** is the weighted harmonic mean of precision and recall. The best possible **f1-score** would be 1.0 and the worst \n",
"would be 0.0. **f1-score** is the harmonic mean of precision and recall. So, **f1-score** is always lower than accuracy measures as they embed precision and recall into their computation. The weighted average of `f1-score` should be used to \n",
"compare classifier models, not global accuracy.\n",
"\n",
"\n",
"Mathematically, f1-score can be given as\n",
"\n",
"\n",
"![f1-score.png](attachment:f1-score.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Support\n",
"\n",
"\n",
"**Support** is the actual number of occurrences of the class in our dataset. It classifies `284315 transactions as genuine` and `492 transactions as fraudulent`."
]
},
{
"attachments": {
"fpr-formula.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAANkAAABPCAIAAADp4/d/AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAPJSURBVHhe7d0xUuMwGIZhs2dJKBhOEE6Q0GxFSxfKpNmOkm4bKElHS7XNwgnwCRgKkrtkjaPIkoMde21Fv5T3aeIxngQ8H5L1W4pP1ut1AgjwQ70CvpFFSEEWIQVZhBRkEVKQRUhBFiEFWYQUZBFSkEVIQRYhBVn0YfVwcVLl4mG1Oej1Ru0p0QfEhixKk34s89fV53v+uiOdD+MMJFmUZnQ2VFt1skDevKrtWJBFz6Yva9vbbKB+pBXHLO9Hal+SLP5EFkayGJTB7KlI4/tnXP00WQzM4PRcbUWHLAamckgTPrLo2WKiSjVbtUOS15vhPFXbo6vLnSvLoJHFABR5nSzUrmxAc7s7yAkbWRSmWUlndL98HKvtaJBFz8o1nW9KOrYshg2OChFZDICV1yhjmCOLkIIsQgqyCCnIIqQgi5CC73aCFLSLkIIsQgqyCCn6yGLPK4mqjtwzhQWhc90utl9JVDNBL5+vQiBj5TqL/a8kWkxIY5x6zmK/K4nMd3uZqp3fH4rweR27tFlJNH4s0uh00dHm0hQmdWoc85rFViuJjOvI89OqaVN1w6iN+g4+O0BtwXCY0+I5i41XEq0erouFHo2uQRGanrPY70oi492MA++fop1NeuTU0KALcwhSth1+1B3z5fsBTVk+v94x9VEwqFPjmOM+OsCVROrEwKBOjWM9Z7Fc0+m2kki9W9FSpvPrWL/wDT7GLlZem6wkGszedD0nnf+uuwLtOo6GR57H0U2Nf+ri4uKOpjFOgWQxGf8qOurnv4QxRqFkMRlcXhVXjZX9dNadq86/SnxftxCNYLJohpE70lEKJ4tZm3dbXDQSxgiprgtVGtXejVlEFkfF+aqPK9lULOw/oFx1y+h3O8SthBoBtYsCBfHMgeXH9vbpF8HTP8liF0E+c0Ds9Q1ZbGGnf+s0U3iHLtTvbUjHj+ozckaPbf+Gds1gNJ1ufiGpFVqy6Iy0Zw6kyZmqROy5eeULWXRI2jMHTnUlQmTTSBYdajxT+GD07SuJTSNZbCGCZw4URVp5TSNZ7FmrZw7Y04p0dPNKkNbzGFxu00gWOzjMTOG+hz3FmEpY00gWWyjXdLrNFG6qetXj/9IdtaymkSz2zMrrvhja04p0QdK+F+diZpHuqCUVvsnicdJNY9ZPJ2fqf8Azsnistk1jOr97znd4RxaPVnHVmJqTJ/whi0fMWLghAVmUQ49kWo68h/p6r+23u1i3zL0X5HmOAaSgXYQUZBFSkEVIQRYhBVmEFGQRUpBFSEEWIQVZhBRkEVKQRciQJP8A/URylqZJTvYAAAAASUVORK5CYII="
},
"tpr-formula.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAPQAAABSCAIAAADsCG5aAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAANCSURBVHhe7drtleIwDEBR6qIg6qEampliGD5MLDuy4xhDFPHuvw0WYXMeWWdmD1fAKeKGW8QNt4gbbhE33CJuuEXccIu44RZxwy3ihlvEDbeIG24RN9wi7i2dDi1Ol/vaS2Hx8fz3eC/MEPeWVsT9dz6GP2ooXEPcWxoW983z/g6BuM0Q+w4lVBF3fDUpnrpzxG1GR9zJYbYmOeI2oytuMUXcOeI2gzv3aMRtRkfcYoS254jbjOa4VcrIzyNuM96Im7u2irjN6IubrsuI24yOPTeqiNsM4h6NuM0g7tGI2wziHo24zSDu0YjbjJiv9hMQ4l6PuOEWccMt4oZb/XHHx5+a5/5QPCsl5pvL0kp2mljtK3GLhyGFLLy+ksKxhoG4b6ZmF1eSN5oN2nN3/Iw26fh1WF2ZblaoG422izs5/NqaFFbKE/Df4NBoy7iVYksri9FLYlFBeTYXBlAVLpZVu7hzy2orN+5hcYfVaBAumUnbxS1GYrFLfVb3JMS9gXDJTPp23Ko4UltZDXuocEI0CJfMpO3jTpq1EfdNOCeqwsWyatO457mKleF95OxX88bufTtu5VVJXVl/86j0HYoWzg5f9hC3fPfazZu4kdhF3PJwpW7iRmIfcSfdUija7CRu6sZ6g+KO7WnbhgFxL31/gJlBcQP2EDfcIm64Rdyt4pa/5vk4IJ4PEp/8Fat85M5N500XKc8u0yf/5Ef9FuJutSLuWmcfy6Z60lfG+bduljdx/6Rhcd8ot8y3Nd25Z/+k5J+EuCEiUUIVncVXk/ha6p4G2jpTT5p7fe7j6aS/OXGjI+7kcEs6n4z7cDpPy5PVxI2uuMXU9nFf9I6JG11xi8MW4hYTcYC40RG3GGkrpz/umelTJHFrKRM3muNWKSMPS3MPPcOluMVQOEDceCPuSjVNcZfmK8NxJI9bjD0XETf64l4Kpilu5XwPYri0RIs7O0bcaI673Nmi7J66pDvuOHk7SNxwFXc8fDxf1p3UNuLu4ixuca4jcf86b3Enf6MH4v5Z/uKW83fE/bNiCVoEY+Je6d2407w9tE3c8Iu44RZxwy3ihlvEDbeIG24RN9wibrhF3HCLuOEWccMt4oZT1+s//0PZRPTTOpwAAAAASUVORK5CYII="
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"### ROC Curve\n",
"\n",
"\n",
"Another tool to measure the classification model performance visually is **ROC Curve**. ROC Curve stands for **Receiver Operating Characteristic Curve**. \n",
"\n",
"\n",
"The **ROC Curve** is plotting the **True Positive Rate (TPR)** against the **False Positive Rate (FPR)** at various threshold levels.\n",
"\n",
"\n",
"**True Positive Rate (TPR)** or **Recall** is defined as\n",
"\n",
"\n",
"![tpr-formula.png](attachment:tpr-formula.png)\n",
"\n",
"\n",
"It is also called **Recall**.\n",
"\n",
"\n",
"\n",
"\n",
"**False Positive Rate (FPR)** is defined as\n",
"\n",
"\n",
"![fpr-formula.png](attachment:fpr-formula.png)\n",
"\n",
"\n",
"\n",
"The **Receiver Operating Characteristic Area Under Curve (ROC AUC)** is the area under the ROC curve. The higher it is, the better the model is. \n",
"\n",
"\n",
"In the ROC Curve, we will focus on the TPR (True Positive Rate) and FPR (False Positive Rate) of a single point. This will give us the general performance of the ROC curve which consists of the TPR and FPR at various probability thresholds."
]
},
{
"attachments": {
"precision-formula.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAABPCAIAAABJfJ+RAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABcWSURBVHhe7Z15XE/Z/8ct45t1ZIvIvs8YE5qyTMi+y9ZgpBhkmBikQcgydhMPI5LsFaMsRRqaJJqxE8a+JstYBiMzzNrv2ef9cecjalrwK5/z+uM+zj3n3HPPvff1fp3X+35u5EhUUDACKKIrGAUU0RWMAoroCkYBRXQFo4AiuoJRQBFdwSigiK7wDP7++2+2f/31V3h4uIWFRVRUFLtPnjzRNWZjKKIrPIM///yT7fXr10eNGmVubq6IrvAGAjn/9ddff//99927d9esWbNWrVoxMTGouxCdVspEggSD4a6sA1kZiugKL8Dhw4dLlSo1cuTIX375RV/1LLI+s5NBEV3hXzx+/Pj27dsPHjwIDQ0tVKiQj49PfHy8aDytKPdvv/0G9dlK50ePHtH54cOHf/zxh26ArAtFdIWk1JMtZA0ODh48ePAHH3xQpkyZ3Llzt27d2tbWduHChRcvXqTPli1bFi1a1Lt37zFjxtDfz89v7ty5/fv3d3V19ff3l0FkmwWhiK6Q+M8//7DFjVy5cuXQoUNeXl4dOnQwMzMLDw+PjIw8duwYok6HgQMH4t09PT1xNc7OzidPnoyNjd2/fz/s//DDDwkGGUpGy2pQRFdIjoiIiB49etjZ2en3dTp9+fLllStXUj5y5EilSpXs7e0xOdJKJBQsWDAsLEziIWvad0V0hX8hb1cwJHXq1Bk7diz+G3cOy/HicXFxV69epXXevHn4mX379umOSAKiXrJkyU8++YRgYBcrL/VZCoroCv9CHPbUqVOtra137Ngh2mxouxMSEvr162dpaWlYuXPnThMTExcXF4kERXSFrAvNWGNIunfvDtFRcakxBPlo165d27Rpo9/XISAgIFeuXL6+vvIu0jAGsg4U0RWSoBE9JCQEKrds2VIqRdSllTJ+plmzZoMHD07qqhPvU6dOubu7m5ubnzt3jm5ZU85B+oguV4Jv0/DHs6Dm9ecinJHzMjHtaaUOusls09jfGKDdismTJ8Pyvn37yq6/v/+uXbukDHr27IneBwYGcrcBNXRo0KCBBAbgQUghq0EpukISNKL36dOnffv2ixYtonzr1i14D5WlCUFprUNkZKTU/PTTT8OHD69bty6WBoozSNb0LSDdiv7w4UNs3M9Pcffu3Xv37hmWX/OlMiVu8f379/GUaVxMkCKu4sGDB6JJCkC7dU5OTgMGDIDc3J9Vq1adOXPm+vXr0oRLwbtXrFhx8eLF8oPo3LlzHR0d3dzcpIMWLVkQ6SM6fCKCCxYsWLJkyWLFihUvXrx+/fqkJs2bN7ezs7OxsbGysiIH1/d+xZCIgqzbtm2rU6fOkCFDtEfywjuuVR4/frxXr162traHDh2Sytdvt7IatDsQERGBEX///fdHjBixYcMGNFtu0ZMnT7y8vMqVK4fer169ul+/fl988cX06dM3btwor8+zONJHdLTzyJEjc+bM6dSpE4k2VxsUFBQdHc3dgd9sHRwcaBo9erT2Yeeri3IhOk+IKVlaWrq6usqPcyB1oqNMLi4uxCQFqcyyC+7rB4/4/PnzUVFR3NWbN29qdwYJ79KlC1q2YMEC6vfu3RsbG3vp0iV5OZP1lSIjHp00DuuWP39+Iltf9RSnT58m4k1NTadNmyY1dJbCqwP3fenSpSEhITwMfVWqSEhIwFP6+vpeu3ZNal5dNL4BkE+4wLvvvtu1a9fw8HDZ1ZAtHGD6iE7gclXEsbu7O4ru7e2NNYfK3AuUQILbz88vX758pUqVYtVjl3qOApAJSBnoxtND6g076BsMoPVJqUMyaP0NC/q2lKEb+8Vn0SqBbjA9ZFffKfuDa+ER89TYIufapeHIS5QoMWXKFPGHjx8/pgPg8qVDFke6ic726tWr+LPcuXOTmxv+RsCWDkuWLClQoAA3JS4uLumYVw8eBsHGTU874ZiqRKB+XyEFCO8Rsl27dr311ltff/01pjRdtzqL4CUTHUB0CwsLFP3u3btJx6SKlO4X42hN/0lHIioyMpLMUktGU8edO3eOHTu2Y8eOc+fOyeCcTppeHThFkgCmGVmBSXJzyN0nTZrEM82RI4e5uTnL+JkzZ6RDNqJ7Bok+ZswYiL5w4cJ79+7xVFjI0FTJvlesWFG4cOFq1apRJq3BrFtbW9vb2x8+fJgsftSoUaSqs2fP1pwfB+7fv3/kyJH4fm4oIbRs2TJpYmQpwBK0xNnZmT6cmq1YfzqQFYwbN65hw4aMEBoaKv3JMjdu3Ojh4TFr1ixamQNT5ew0XblyhbKdnR15M4GqJc2AoGVwwLNkGhMmTNA36FZq8rM2bdo0adKESX777bfLly8nKWdkZuXj46Pv92ZBeEzmA9dRdNSEmwDLRd2yFzJC9Pj4eDw6RA8ICJB6DTCgfv360I7skN0bN25AuFatWuHa4Rz827x5c9++fWvUqHH79m3u4/379yMiIuAK1F+5cuWqVasYuWPHjkFBQSwIcjp02s3Nbc2aNRDL39+fFWP48OFbt249cuQIAcBkiIF69erZ2trKfAghasaPH88cAgMDCZsvv/wSTo8dO5ZWModNmzaRVJUuXXrixIl0lsfJaLCWB8kpGJxwZaryhoFW4urEiRNQ38TExNLSkhMRnOt06NChg6OjY3BwsMRessVBdokToouQGzFiBAHJNhXQgWC7cOECB2qhngzUY71eNbgitvpTPgvDDq8a+lNmDhlUdMQSx4Y8wwykOiYmZs+ePWwh8Xvvvefl5SX9BWfPnoXodevWhejM+6uvvurZsydLAU34B5jdunVr6QngxODBg6lhQHY5l6+v7//+979t27ZJB8CaALOhqX7/6U/TW7ZsoYzGDxw4cO3atdIErl27Bs/gkH4/MZEx4SuRI6sQTobAqFSpkqHAX758+e233169erVh5UcffcR6xQTIwqUmLCysRYsWXB1By67QXYPskqYPGTKEgG/QoAEqwDYl0AqaN29O4GmHK2QeGSE6nhjzkCdPnurVq3/88cdDhw51cnLq0qULMolwYlekM4C1HMKSB9F5eCdPntQ3PB2Kcd555x20HN8MUQALJZw2NTUlBuhAmbyWs4jCcRTrZp8+fZo1a+bn50cNekxl+/btmUBISIgc0rJlS0KRNQE6CpW/+eYbeeNJf4DG16lTZ+bMmVCQXbSW+GTyoqDy+ogyawsmB5fCLmehplOnTixHBKfUINjR0dG9evVifZDQfVkKlDo4F0uTkUB/zZlDBhUdZuTMmZPtpUuXYB58gqPcfZgtPQWihah+oUKFXF1d5b21DAIwNi4uLsQAaocnBiwR6DQOAW2TqMDYcCJPT08igV2IBS/lB3zORVk3UiL+AQrCZsqclKAqWLBgzZo1iUCiSPIBjmIrVJ4+fTprAm4KyWT+MBX6kpvSBERHGYd4qFWr1vz586UeCPVZEPT7iYmYV2SeLO01EF27XrIFhIa7yvaNh1xyJpFBj47fhX8Ya8NlXYNGZXnqEJ3lHkGV5Z5K6YA5QXpJW5F80h38A+s1BYADhp0oK6fIlSsXxkMSoGQOWDsRRBdnL7sHDhyAnVWqVClXrlyRIkUwAzRJZ+EKRMdsMDgDHjp0iMDAyZCn6o7WBwOXhssiYAyz0nbt2jFncgn9vu7qyGvLlCnzOhWd6Fq/fj2JAds3HvprzhwypegYD1YWaAEnUEEAbzTyAQkDqIAVwScI0UUvwcGDB2Fn7dq1Dd2OISA6lJLXt4ZE10YwJHrnzp3JMuGx8BW9R8unTp2Kn+YsiO7ixYulMxCio+iM8OOPP3IsRCeApVWITqRNmTKFmXOZUg/wSLgmSbUFougpEV0mzGRebjKqkF6km+gwCY/Ok4Do0AhHQU0yodUgRN+5c6d8FCCd5eFRwH4MGDCgWLFiy5Ytoyf1+GlpPXr0qETFd999lzt3bjc3N3n7odHo9u3bCQkJQnSGguhkCKGhoRyOBnAiOVxAkkom0LRpU8p0ZstkxLpAQQhNDo38s6TouifxUgrDhg0jQxVacyCn40TIP+mBdmrCGKLj0UkJ2E1GdInJl56MUs8dMxLorzlzSB/RBag4qzlEx8LK5yXy1J+H1MfExEB0fIK4ZKkUQnt5eVWtWhWiSMoolYg3tIBAlNlWqFDBysoK6rOrRVRAQAAxIGWAQdc+w1iyZAkTkzcwGlBKuK7fSUxkMqLoYt9ZbVB0kgShqdALhbaxsWnRokVgYGDSMTpwIqyL/D28AAOGxbewsJAUNqWYV/j/RUaIDl9nzJhBloBTF+6mDsQJo7xgwQJDEkiZXBYKmpiYnDhxQuoBq/acOXMOHz5MGdJjr3Ev69atk1bAsWvXrpWXIYIePXpgPzZv3kx5zZo1jRo14nTSJKA/YqzfSUykFaLjSUTgz549i2EoUaKE4a+55MFMTPsRSsCJsOmG7y7JB0h5K1asqN9XSAGyJPLsBHLnXxvSR3T0m/ysVq1aZJA5cuSoXLly48aNoZf2T3wItGvAfEPHGjVqkFByFJ3JO7VFnwLby5cvYzZw0gMHDmRwNB5ffu7cOc3GUEC/Ed1u3brNmzePASEfOnr9+nXCbPfu3Y6OjoxPLFWvXh3u+vj4YHVQWZYdAoYxkWoS3DNnznB/ST09PT2LFi2aP39+bIm7uzsTYBo4dV9fXzQeu4KDHzRoEIdzFlk6mQPloUOHEt6cyNra2sPDg9FWr17dpk0bMzMzJmBvb09syOuw558iF5su6A9LA+jMEsRk9PsKL0L6iM7z3rBhA+I36Slgw969eyVT1KA9Zoz19u3bMfTSmWwsNjZWmoD2OHlI8+fPh8QMTkIZFham1UsBBAcHkxLQgQFDQkLkjBgPlgLYzMiMD4P9/f2h/r59+wg/AobVgAMZGc2mP2ekgOQzJRJNJr9ixYpbt27pzpAUxt7e3iS+CDZbwxUDuhMJJJQcgi8iEjiQ0bBPRCaOf+LEiVSydsnEnie6kYNHSTSiMsgQdjQ6Olpy93SFdGaQEevyGsD1a7cgJa16iRqWrXlJ6ox8yLumV3EhjJkZOmrHiuiwumIynZ2d5Z9ASpa7vzqkm+jMG0dhCGpSur9yj/T9dHhhTyphrYD+bPUNTyEdOJxWgTaO1iSQEQAFw3Iq/ZM1GUJrAskOZNjnK9kF0v9VQ5sAAol9YkFjlznoGrMQ5J4wsebNm2Mjcbnv67B//35asy7RFbII5NUQZtLV1RXeYKjYzWpEl/nAZqwgiY38dH3p0qU7d+6ILuh6vQ4oomdLIOfkJxCI3NfS0rJhw4ZkFFBHBDJpFdNlqNouyw67INkylTo4kO3SpUtHjx4tNemlJrkNs/r5558dHR0bNWqU7HtmGY2z0Efmxq6skOyylQm8FCiiZ3uULVt27NixWkqdDDBGX0o/oBrbTz/9tHbt2lKTYebZ2Ng4OTn98MMP+v0UYub5yvSGVkpQRM+WwOkeO3YsMjIyNDS0SJEiZHibN28+cuSI0B0/c+PGjYMHD8qvbAjqhQsXDh06tHfvXjqIcKaFQNJz/Pjxbdu2lZq00056xsXFHThwICQkpGjRor179/b19WWSN2/e1MbhFEzvxIkTTE9eWF28ePH48eOEBHNOSEiQbpmHInp2gsaP3bt3T5s2zcrKqmrVqjlz5qRgZ2c3bNiwmJgYWmF/QEBA3759cTUwOywszM/P77PPPnNwcJg1a5b8La+QOHWIoo8ZM4Y8UmrSrugyfnh4+KBBg5gek6xZs2aDBg2Y5J49e7Szx8fHBwYGenp6tmzZcu7cuZB+1apVEyZM6NWrF+d9WV90AUX0bAk0+9GjRwi2s7Nz6dKlYTYUgdNiVEaMGEENMmlmZlahQoUdO3Zg5dHLc+fOWVhYbN26VQb5T3nODNEBnfHo9+7dCwoKMjU13bJlC3MgDZXMQQbft28f1ujBgweY+IIFCw4cODA2Nlb+MmHcuHHt2rXTvrTLJBTRszFOnz6NkDdt2lR+DhNQhivyDVyJEiUwDNrHoTCsTJkyLAXyaeTzok4HADsFxBJbNzc3ziI1ZMBSAPRMnfcyPrG3cOHC4sWLyzcdGggzYoCZrFixgl2Cs0mTJgsWLJAAALNnz2ZFkj+PzDwU0bMl5Fs0CA2BIOLVq1ehnbypuHbtmvAvIiKicOHCBIPuCD3KlSvXvXt3+R4uLe4FzJgxo3379vqd9ID5MJNTp0716NGjcuXKUVFR1BA82kry8OFDpn337l3qGzdu7OHhIdclE5s/f36NGjVwPrq+mYUievYDRIEZFDAhBQoUwPI+b0JgDEqZN29ekXbpgLcpVqzYgAED5McajegSGIyJxd+2bRu5o2Djxo1YoG7dutWuXVtqNm3aJAWA9Zdv4FLSdUjMlmTU2tqaZUc+PJbKZEDscfBELGXtWjDuZcuWZf2R3UxCET37QaiAr50zZ06+fPm0P4wSIOp0+P777/v3749+a04AwO88efLgfeXljEZQCRv0FU5DOEw/DsdcBzw91tnExIQa6qmRVlCtWjV5XfhC7gLsDVuCh0FcXFxknobLiMyNmJw0aVK9evUM/6aeqxs6dGjFihWTfW6dYSiiZz8IVxDyfv36QQWRVZhh+Mn0smXLEFGSOflKHk7HxcUNHz6cwNi+fTs1z7OT8MA0kywSBrdv32Z7/fp1CsOGDbO1taUAWB/YSgctrUwJQnQUHQfl7e0t0zMkugQYM2/VqlWzZs20P1BktuvWraPGwcHBMFAzA0X07Afhir+/f4sWLWCzvH5eu3atoSJCrAYNGpAFQnTh05kzZ6pUqYJvkX9zOHWOGmLy5Mltnv1Pi9IIWXkCAwPz588vXy8ngzaH8uXLjxo1ipRDpgqcnZ07duzIVchu5qGInv0gRJ89e3bDhg3l71mRXtyz4d+aYHDNzMw06t+4cQOfg+UlQ4X6yOQLjTX1kE/Dr7r/HH306NGIq9Q8fvxYCgLNTz8PGZ8g9PPzw/zIsmMo50D6kEmbmpo6OjpqeTO5spWV1eDBg2lSim68kGe/fPly5Bnvu3fv3qCgIARbU03csKurK8Z67NixOByyQB8fnxEjRozR/dfmQBPO1CEnyswPRmSZWG1SBakkNqQAJEg4BZkDKXXnzp1Zoyjj+93d3XFZ8oeRyWIjw1BEz34QtsXHxy9dutTGxgYtXL9+/alTp6QVYHbxLTQFBwfDs88//xw5h0Y0cWx6yWpI9FQk/IUgAu3t7bHgGHF2DeVZysQkyo2iT5kyZdGiRU5OTuPHj58+fbr8e7FpDMi0QBE9uwLOke2RPt6/f5+CpnzUI4eYDXQRomAe6PCb7lNH6ZB2yJjjxo1r+fQ/nUsX0ekMZe3s7PBUcnZDeRYSk6oSk40bN6aALyLHTUhIkLfpLxeK6G8O0EiRSQcHB9JHw3/HRpB2LRfIaCwI9evXl5rUR5AwIOru3LmDvaYcGhqKJ9E8lWGckACwPXnyJJkDi1Kyl6QgXUH1n1BEz66ABwgkuggddX7kXwra2tpi34Ve9KED29Q5+kLIIVFRUdo/+JE6+USkY2NjWVK8vb3JgAm2rl27SivHaodrhejoaAy69vmNLE1A6/CyoIj+JkBHoSQnc/78eWxx+fLlUXQIh7K+dMakAvEnBEbx4sXbtm07a9asmTNnam/HDWdCmTg8evQovihXrlxLliy5desW85cm6fNyoYj+JgAJZHvz5s158+aZm5vnyZMnb968zs7OcE7sRwbkXAPHyiD/Cel2+vTpkSNHenh4dOvWTf6RM5meQONxSEgIjqVQoUIQ3cLCYtOmTYZfK7x0KKIrGAUU0RWMAoroCkYBRXQFo4AiuoJRQBFdwSigiK5gFFBEVzAKKKIrGAUU0RWMAoroCkYBRXQFo4AiuoJRQBFdwSigiK5gFFBEVzAKKKIrGAESE/8Pray+mg/mOAsAAAAASUVORK5CYII="
},
"recall-formula.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAMkAAABBCAIAAAD0RxfnAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABOWSURBVHhe7Z15VFVlF8bJoVLCEbFScSoxzVJxmRJiDohprnQZqKhpjsUgTmlACFLOaJBDKSrOpoRDDmiKmrNlqKCS4hSUgWKKgppZfD/Z13fdT7yXywx2nj/e9c7nPWc/59l733tRswwNGgoGGrc0FBQ0bmkoKGjc0lBQ0LiloaCgcau4459//qG8e/fumjVrKlasmJiYSPOvv/7KHCzW0LhV3PH3339TxsbGenh4WFpaatzSkA/4999/79+/n56eDpPCwsKaNGnSvHnzhIQElOzevXtMoMIE6qJt1KUJHaWnaKFxq2Rg6dKljRs3njBhgq5dEqBxq/gC3bp9+3ZycnJqampQUFDDhg2XLFmSlJR0584dlIlRJOrWrVuMit9E4WjeuHEjLS0NAZNNihAat4ojhBmQBrlydna2tbWtXLlyhQoVunbt2rZt2/Dw8GvXrkGjDRs2TJo06b333lu2bBnzZ8+eHRAQ0LdvXz8/v23btmXupEsFigQat4oj0CRKZOnixYsHDhzw9fWFUsRb1KOiouLj4xGq8+fPjxo1Kjo6unfv3tbW1p988snZs2ePHTt26NAhHx8fFxeXlJQU2Up2K3xo3CoBCA0N7dGjxwcffKBrZ2TgK+Pi4nCU1KdPn04o5u7ujiuUUVwnRIyMjBT9Kyrp0rhVfIHeCDkGDBjQoUMHXB4JI4Arf/75Jx4ToWLU1dV1+PDhKBx1mT9//vxGjRqhavhN1Vn40LhVAtCrV68+ffqcOHFCmvo+7sKFC02bNh00aBB11R8cHFyjRo0xY8aIkkmkX/jQuFVMoRzZmTNncHkokzQFMkr59ddfv/HGGwRb0g9QqfHjx1erVo0AH5GjR/OJGv4P4sgohT0jR46kqRRIJCo9PZ2ssE2bNrNmzZJ+Juzdu5fMkSXkkkwrKmIBg9ziWBxUHLyAtEWBJqNFdW7OxkOXU6lggsPI2YrwYPkIoRG3g79zcHDw9/eX/lWrVp0+fVrqoHnz5kOGDNm9e7e6a29vb4Iz4SJQz6fwkVfdegKsWDwh3OLlsbe3R4e2bNlCMyEhgQh9//79mVMybty4UatWrcGDByu2EX5BLGdn58OHD8sORWggY7p18+bNq1evIq2ClJQU0hOpX7lyBUHWTS108Lzu3LnDYXi4VGhy2rt373IwOsnP5bGWaKBYlNxUu3btEKFdu3YlJydv3LiRiP769esMccvHjh2rWrUqPnHdunU8Cozi5eXl4uKCG2VCkb/2BrnFDXBQCwsLKysrbqBmzZp2dnadOnVydHRs27ZtgwYNiB/1xblwwAOl/P3335cvX06a3axZMzLzS5cuofzbt2/nFbexsZkwYYJ6s0suxJdR4gTxeq1btw4MDKSuXmleexwlBvL09Jw3b97AgQOxCHOOHDmC2hUHf2KQW5wvOjp6xowZ3bp1K1269Mcff7xp06Y9e/bs2LEjKioKiXZycqpbt+7MmTN1CwoRaNWvv/66du3aChUq8ALExsbCOQjHy/3yyy9z4M2bN8tM4WKJRlpaWkxMDE/+1KlTSUlJut6MjLi4uLfeesvW1ha78Hbt27ePafhEueVizS0ByhwQEGBubh4REaHreoht27ZBrxYtWiDUcieFcz+KLpcvX65YsSKhLtySHtCqVSvEVX2b9kRCpIvYq3bt2oRWcE76FYowfteHQW5BFKIW3gMUq1SpUrgeQhmohmYgaZTMmTNnTrly5Vq2bCmf/6ooB/MDdlCl9CtkjuuGpKIbeAjpF2TdgYiKksvhrEeMGHHy5EnpB7zKpOWRkZHSfGRhSQS3z2PnmXO/kEbd0fHjx5FtvGFqaipNLMIEUHxu2Ri3KHk5xo0bV6ZMGcLDrF8gIGkMEeLI7XFj0l/Q4EFT4hYrVark5uamuMVjJS4hmH2SuPUIuCNuH+lasWKFmZkZwQkW0edc8UH23Bo/fjzx1ldffUUmQic3JkPcD0PcnqurqxIS7tDQTcoqkO1TUDMfgVpI9kSZlVsgW26xeebrbSoMHaYIQVg5evRo4gEePgnNN998Q6ypGytOyIZbiYmJ+ES4RYYi/Qo4oyZNmnTv3v3cuXMPCPXwi1VAbjxlyhRvb29KpuFPRfMkrxYQbkNKcjofHx8SHPmqVeHMmTOEqGQSQUFBZEArV678448/dGN6ulW5cmV3d/dHuNWrV68nWLcAEf3Ro0d/+OEHkip4RiAv73ZxQzbcwn5+fn5ly5bt169fWFgYYfvSpUvpAXDi888///nnn9V8gKKQXQ4fPhzShIaGwonJkyd37dqVOqGbkO/mzZu+vr4LFiyAecuWLVuyZAn8Cw8P3759u2wFWZFJCEcmyEsJwxAn/K+il0R7ueMW9uBIo0aNGjlyJKURjBkzhjNwMN1KA+AScL0wobvw/4N+CcsKB7qrGkX23PL394dbjo6O0CUkJOTTTz9t3779U089hbHxkswRNZIyPj4euWrTpo36LREYO3Zsz549Fy1aRB0Bg6NwIjAwUEYBTOrcuTMSKM3FixfDzqFDh0oTELHa29svXLhQTiWJUu64tXz5cpymnZ0dGSWTjcDBwYE7hdm6lRpyiOx9IrE8PhFCSI8AyXn66ac9PT3VJy6iJYSWJC/Tpk1j4fWH2LdvH7mkGAlCVK9eHart3buX5oMo9P59JBBu4XwfbJSRgVz16NEDrWKI5BRMnToVS/fp00cYLB42d9zKdxCT8RD+a9DdvFGYyi1cmDh1sRa+r0OHDvXq1UOTHszOBCafM2fOM888g3VRLxYyilv57LPP3nnnHZwpcwgOUMFhw4ahcDTlKogcEqg+cZYvbXCdBw4c+PDDD9kB5j3//PPdunUrVtwS13Dq1CnuCJQvX14q/wXIEzCObLilnydibEwlORrAZ1laWhKHSROcPXsWJXvuuefWrFnDEz9+/DhB/YkTJ6j/+OOPkJ3lq1evLlWqFPHW5cuXWaKvhfog9urSpQt+E3qxA+Fds2bN8FDq6oCzFS235PCpqakILSAyk8p/AfIEjCN73YJbsEG4RY8oB97Kw8PD3NwcBkg/88+fP09MRidOkJ6s4EUnu0HYUDVJm8X2qFTmuA4RERE4RBsbm/Xr10sPEf3bb7+NW+S6bCJ/ZUBqmQtu5Xssr8EQjHELw+B35LPT+fPni24JtwDcgnNOTk6nT5/G5LKEaYgc+ePVq1fpwdmJ44iNjUVmqPz2229wq3///vI9N7sJiQmqZAno1KlTx44dhViyfO7cuRCLTg5AoHbw4EEqeNVKlSp99NFHMTExwiG2Em4R9sm2WZHvsTyXRk3/a9DdvFFko1uk/QTa+vGW4hYOi6i8UaNG69atU/07d+4kMMIk8mcCIkg8fS8vL3agfu3atfr169va2n777bc0FVi4YsUKqWPUvn37HjlyRJosJ3Vo2LAh0kUTzZOUE95XqVJlxIgRv/zyS+bEB3jzzTcJ+b///ntpCuc0FAmy4RZyQiSO0oSEhEgmqPwXwtO7d28CWAxPc/PmzcgJoRXzn3322e+++06mAZzXypUrhUzo0KZNm6pWraqyQgFulKBK6u3atYMiRGbSRM9mzZr10ksv4X9pEqsJt/CMVlZW7APJMic+AGsHDBggSSjQuPVY8FgA3kZgSOPzCIPc4tp4FjxOnTp1zMzMMC2RjTgj6MWBqKxatcrZ2blp06bEZDADNqBtBOkoEDENrmf69On4FPoJ6qGp8JISqSOAw3lNmjQJokBEIqRz587JpUkFcLgkoUQ806ZNQ4TwesyvXbs2CeaGDRv2798fFhbm6OiIU65VqxZEnDdv3o4dOwYOHIj7hnBt2rRBVjkku0mpwLF5lKaD+bqVpoEluIycrnoiYZBbgOQuNDQU80+cOBE1oi5BkmL6lStXCI0hEPxAt/R9E9E3C0kokCuoI2GTvpkPHz7MtrBz27ZtW7duJQ/QDWRO27NnD+TAjbIDIoSp6ORCyCcqRWoG4Yjq/P39AwICKLl6dHT0zJkzoTjXhdkopRxSSg0KPEzAq75kyRIyFQxx9OhR3Vi+whi3jIPz6Wp6oFM5zUegbGzI2IYWlixcv3599+7d+p8F5iPy/p7IQybqXbZsGYmXq6sr4engwYNlNH/fQ2Pc4kqoBacRKLlS4NnJBEr9Uf1+OkHWp0ynoQmGtpUeRgGdNBVkB0Z17UL//YJc7tatWxEREeXKlZNPhk3MpwoT4kCIH8zNzZOTk7/44gtra2v1x488Q6nkC3KvWxr0AZsp8S8oQeXKlSV2LG7c4t2jJN8aOXJktWrVqKOvcXFx8gllvkPjVl4hepmeng6TcDSNGjUit7hw4QJKpqJMcO/ePUr9JnSUnmwhn+/Exsb2799ffgwiVM4pyPTZauPGjfjBxo0bZ92EM8vZ5OTSZBqgJ6eqpnErP4HZbGxsgoODde0syEomU3y3WHrXrl1k6+JtpSd3ID0ijyYH17UNMPWxnTmCxq3cQ95jbHDx4kWSWXLbcePG1apViyCGNJZ8Fo/DnKtXr5Jfk4uRVjMfd4lXOnToEM5IfqQEjEuCMOngwYOvvPLKpUuXqIuSmQjZHFpzhgMHDri7u9etW9fNzY0MPSEhQb74l2kkjxyPCx07dowe8hKaMTEx5OAszCmhNW7lHiJC2IZkHi/z6quvVqlSpXz58k5OTm3btl21ahWsgj04ysmTJ3fp0sXb2zs1NXXhwoWBgYH9+vXz8vIKDQ0Vw5vCrf3799evX19+oJsjM8s5cdlQv1evXkRaZBs4bgcHh/DwcPnRgGDlypVTp07t2bPna6+9xpsQGRk5d+5cPz8/T0/Phg0b/vTTT7p5pkHjVl6BU0Of4BAM69atW8uWLXndSfIpGcWFDRo0CDvhgypUqODi4nLy5ElGmT9z5kx8k3w5JvtIJSvyyC0AdwExO+eB0+3ateMAHCMtLY3rKmZDehSXObwh9vb2GzZsQHFZlZiYaGdnN3DgQPnDz6ye/bHQuJVvCAoKQpyGDRuma2f+epugfs2aNdRDQkJatWqF8RQtli5d2qBBg0WLFsmHYfoGk9gZpRGwD+Xu3bvr1at35swZ1SNgZraxkSLu8ePHhwwZ0rVrV2kqSIy/Y8cO2AaJzczMkCuuJaPsz2vg6OiIsNHUuFVIwGzyrEeMGNGhQwe4gp0wBp3YCUhmhzPCqAQ0mYseAM6hQ0OHDpXObEMoQh8ShWxp9FiIMnGkL7/8Eoq8++67MJIrAqEd2xICJiUlUVm3bp25uTkBlr6Ucun3338fflPX7zcCjVt5gvImPG7Y07t3b/0vrxTwgwQ3rq6u1NWSefPmEfogDxJN63OLYB8JQUgE8r3ZlClTXnjhhQULFtATEREhQ/Rv2bJFvoszArko5ahRo9q3bz969Gjpz6pAaBVx4YsvvihN3g1KAkdCSQ6ckpLCJhq3CgPKZiRWLVq0gFvSLxAbUM6aNQtujR8/XvpBeno6rLKwsFi/fr1Ikb6Zif1bt25do0YNbAygVM2aNS0tLcuWLWtlZUU/PTJkbW2NY502bRqr5DCPhWKDs7Nz586dySekqb+EY9DkPHhM7kX9nA6QOZYuXVr/ty2mQONWniCEwCQSmJPeSyeRsoqrsBkWxV2qz72wNGKDCSGQyNUjSoBaoBDJyckkAYBQmnLTpk116tSRzzKkhwkAEqjPEQxB7Q/7kU/cnzT1ryvcmjFjhq2trYeHhygWIGQcMGBA9erVkVLpMREat/IEJTbkUCRfEx7+nyj+/v7qN2SgWbNmbm5uWFTF3WPHju3YsSPSJROy+qasiI2NRaJgra5tMpQ4sRZBUg4RPMItSnwuEqv/w10Cr7p16/J6UI+KigoLC5P+bKFxK09QnECBCHWFT+RihMxKG9AVbNO9e3f1UxYSMYiFhJw9e1YpnwwJ6ETPIKIAWaLcs2cPeSJLVI+CcWrK5pRcF27Jn2Y9ckVAD9v6+PgQkMlfJAjbuKny5cvLr4spvby8Hsw2ARq38gR5+lilT58+Y8aMmTt3LgTauXNnQkKC/EkcE6Kjo3EoqBop5JEjR2hiXZLKR37VbQSwhzKPn51C1pCQEN4B+Yd39emoeIbXI5CX344DuQU4jV6SPdy8eTMyMnLr1q1ZeflYaNzKE5Tj2LdvHzGKvb09bnH16tXqby2JigjC5DfcixcvRtuI6Am95aMjEymSlVv6SaWJuH37No6bQJ4XgKa+N1Rcge6+vr5Il4zezfwdB0nrsEygWMR89BiXSQWNW/kDjIFDuZb5XziJSUTS4uPjW7ZsiWghZnfu3IFqvP3yPWPmOpMg3CLuQT9yza3U1NTGjRsTbIlr1ueWAifnrVAvhgAmEdezHORILzVuFRTEQgRblpaWffv2jYmJkX4F0+klFsU3NW3aVD4/y9bGanMOQF2+gHr99dfVn47miNz6MH2hxq38ATKAUCEnvOVAGSAuLq5MmTLh4eHKyzCNCY+VDUNgPmViYuJXmf8KmuoxArV/QEBAUFDQ2rVrUVA7OzsISqeh5XIXQNfOBPcinSBHx9a4VSDAHqRaRCqEVmZmZoTPSUlJOXWFeYG6kIuLi42NDTEfZwgODpaf6MASGS1QaNzKZyijRkVFEd1bWFjALSsrqxUrVohdc00vFprOCSUws2fPnjhxIifp2LGj/IVpLmK13EHjloaCgsYtDQUFjVsaCgoatzQUFDRuaSgYZGT8D836qDXIJPz1AAAAAElFTkSuQmCC"
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Precision - Recall Curve\n",
"\n",
"\n",
"\n",
"Another tool to measure the classification model performance is **Precision-Recall Curve**. It is a useful metric which is used to evaluate a classifier model performance when classes are very imbalanced such as in this case. This **Precision-Recall Curve** shows the trade off between precision and recall.\n",
"\n",
"\n",
"\n",
"In a **Precision-Recall Curve**, we plot **Precision** against **Recall**.\n",
"\n",
"\n",
"Precision is defined as :\n",
"\n",
"\n",
"\n",
"![precision-formula.png](attachment:precision-formula.png)\n",
"\n",
"\n",
"\n",
"Recall is defined as :\n",
"\n",
"\n",
"\n",
"![recall-formula.png](attachment:recall-formula.png)\n",
"\n",
"\n",
"\n",
"The **Precision Recall Area Under Curve (PR AUC)** is the area under the PR curve. The higher it is, the better the model is."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Difference between ROC AUC and PR AUC\n",
"\n",
"\n",
"- Precision-Recall does not account for True Negatives (TN) unlike ROC AUC (TN is not a component of either Precision or Recall). \n",
"\n",
"\n",
"- In the cases of class imbalance problem, we have many more negatives than positives. The Precision-Recall curve much better illustrates the difference between algorithms in the class imbalance problem cases where there are lot more negative examples than the positive examples. In these cases of class imbalances, we should use Precision-Recall Curve (PR AUC), otherwise we should use ROC AUC.\n",
"\n",
"\n",
"So, we can conclude that we should use PR AUC for cases where the class imbalance problem occurs. Otherwise, we should use ROC AUC.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Precision - Recall Curve \n",
"\n",
"\n",
"In the previous section, we conclude that we should use `Precision-Recall Area Under Curve` for cases where the class imbalance problem exists. Otherwise, we should use `ROC-AUC (Receiver Operating Characteristic Area Under Curve)`.\n",
"\n",
"\n",
"Now, I will compute the `average precision score`. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average precision-recall score : 0.47\n"
]
}
],
"source": [
"# compute and print average precision score\n",
"\n",
"from sklearn.metrics import average_precision_score\n",
"\n",
"average_precision = average_precision_score(y_pred, y)\n",
"\n",
"print('Average precision-recall score : {0:0.2f}'.format(average_precision))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Precision-Recall Curve` gives us the correct accuracy in this imbalanced dataset case. We can see that we have a very poor accuracy for the model.\n",
"\n",
"\n",
"Now, I will plot the `precision-recall curve`."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.legend.Legend at 0x6b5ed6a940>"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VHW+//HXJ40ESCWAtASkSguQqHgtiNjYZbFcFLFd77qLDXRX11293p9tdYuirgVF1oIiKuhdFdvqWlZ3UZSE3psEAigtBEIv398f52QMIZABMpnMzPv5eOTBzJzvnPM5k5BPvt/POd+vOecQEREBiAt3ACIiUn8oKYiISICSgoiIBCgpiIhIgJKCiIgEKCmIiEiAkoJEJDObZ2Zn1tAmx8zKzSy+jsKqdWbW1sycmSX4z/9pZr8Id1wSvRLCHYBEFzNbATQH9gHbgA+Akc658to8jnOuWxBtVgKNa/O4ItFOPQUJhZ855xoDfYATgf+t2sA8UfPzV/GXfDSIpnORIxc1/yml/nHOrQY+BLpDYOjjQTObAmwHjjezdDN73szWmtlqM3ug8nCPmf3SzBaY2VYzm29mffzXV5jZ2f7jk8ys0My2mNkPZvao/3rVoZeWZjbZzDaZ2VIz+2Wl49xrZpPM7GX/WPPMrOBQ5+a3f9PMXjGzLcA1ZhZnZneY2TIz2+jvL6vSe04zs6/MbLOZrTKza/zXf2pmM/z4V5nZvUfzeZtZvJn9j3/8rWZWZGZtqn4Olb4Xv/AfX2NmU8zsMTPbBPzej7F7pfZNzWyHmTXznw8ys5l+u6/MrOfRxCz1j5KChIyZtQF+Asyo9PJVwHAgFSgGXgL2Ah2A3sC5QMUvq0uAe4GrgTRgMLCxmkM9DjzunEsD2gOTDhHSa0AJ0BIYAvzBzAZU2j4YeB3IACYDT9VwihcAb/rtJwA3AxcC/fxjlAKj/XPJwUuQTwJNgV7ATH8/2/xzzAB+CtxgZhfWcOzq3AoMw/vM04Cf4yXfYJwMLAeaAfcDf/P3VeFS4Avn3Do/Mb8AXAc0AZ4FJptZg6OIWeob55y+9FVrX8AKoBzYjPdL/2kgxd/2T+D+Sm2bA7sqtvuvDQM+9x9/BNxymOOc7T/+ErgPyK7Spi3g8GpnbfDqHKmVtv8RGOc/vhf4pNK2rsCOw5znvcCXVV5bAAyo9LwFsMc//p3AW0F+hn8BHqt6DpU+w18c4n2LgAuqef2AfVTdD3ANsLLKe84Glld6PgW42n/8DPD7ao7dL9w/f/o69i/1FCQULnTOZTjncp1zNzrndlTatqrS41wgEVjrD0Nsxvurs5m/vQ2wLIjjXQt0Ahaa2TQzG1RNm5bAJufc1kqvFQOtKj3/vtLj7UCymSWY2RX+VUzlZvbhIc6l4nzeqnQuC/ASUfPDnYuZnWxmn5vZejMrA64Hsms864MF+3lVp+q5fAak+LHl4vVs3vK35QK3VZynf65t8D5jiXAqKEldqzwt7yq8nkK2c25vNW1X4Q0HHX6Hzi0BhvmF64uBN82sSZVma4AsM0utlBhygNVB7H8C3vDQQZuqiffnzrkpVRua2SrgpEMc4lW8oaqBzrmdZvYXji4pVHxec6u8vs3/tyGwxX98XJU2B5yLc26/mU3C67n9ALxX6XNbBTzonHvwKGKUek49BQkb59xa4GPgETNL8wu17c2sn9/kOeA3ZpbvX63Uwf+r9QBmdqWZNXXO7ccbtgLvL/TKx1oFfAX80cyS/cLotVT/y/5ojQEerIjRL85e4G+bAJxtZpf6vY8mZtbL35aK14vZaWYnAZcf5fGfwysSd/Q/r55m1sQ5tx4v+V3pF6N/ThDJFi9ZDQWu8B9X+Ctwvd+LMDNr5BfLU48ybqlHlBQk3K4GkoD5eIXZN/HG4nHOvQE8iPcLaSvwNpBVzT7OB+aZWTle0fky59zOatoNwxtfX4M3FHKPc+4ftXguj+MVqD82s63AVLwCLs67Z+InwG3AJrwic57/vhuB+/333M2hC+U1edR/78d4PYLngRR/2y+B2/EK9d3wEuRhOee+wetltMQrkle8Xujv7ym879lSvLqERAFzTovsiIiIRz0FEREJUFIQEZEAJQUREQlQUhARkYCIu08hOzvbtW3bNtxhiIhElKKiog3OuaY1tYu4pNC2bVsKCwvDHYaISEQxs+Jg2mn4SEREApQUREQkQElBREQClBRERCRASUFERAJClhTM7AUzW2dmVafxrdhuZvaEecsizvZXcxIRkTAKZU9hHN7slYcyEOjofw3HW81JRETCKGRJwTn3Jd4UwYdyAfCy80wFMsysRajiKSou5clPl/DVsg1s313dei4iIhLOm9daceASgCX+a2urNjSz4Xi9CXJyco7qYIUrNvHIPxYDEB9ndG2RRn5uJn1yMynIzaRlRkoNexARiX7hTApWzWvVLu7gnBsLjAUoKCg4qgUgruvXnstOzGH6ylKKir2vidNWMe6rFQC0SE8OJIj83ExOaJFGYrzq8CISW8KZFErwFvuu0BpvRayQSW+YSP8uzejfxVsXfs++/Sxcu5XC4k0UFZcyvbiU92d7HZWUxHjy2qSTn5tJQW4WvXMyyGiYFMrwRETCLpxJYTIwwsxex1uysMxfs7fOJMbH0aN1Oj1ap/Pfp7YDYM3mHYGeRFFxKWO+WM6+/csA6NCsMQX+kFN+bibHZzfCrLoOj4hIZApZUjCz14AzgWwzKwHuARIBnHNjgA/w1qxdCmwH/jtUsRyJlhkptMxI4Wd5LQHYvnsvM1dtZrqfJD6Ys5bXp3mlkMyGieTnZpKfm0V+biY9W6eTnBgfzvBFRI5JxK3RXFBQ4MI5S+r+/Y5l68spKi6l0B9yWr5hGwCJ8Ua3lhVDTl5vollacthiFRGpYGZFzrmCGtspKRy7jeW7mL5ysz/ktIlZJWXs3rsfgNaZKYEE0Sc3ky7HpREfpyEnEalbSgphtHvvfuatKQvUJQqLS1m/dRcAjZLi6Z2T6Q87ZdIrJ4O05MQwRywi0U5JoR5xzlFSusNPEJsoKt7Mou+3sN+BGXRunhpIEgW5WbTJSlEBW0RqlZJCPbd15x5mrtoc6E3MWLmZ8l3endbZjRscMOTUvVUaDRJUwBaRoxdsUoi45TijRWpyIqd3bMrpHb0lU/ftdyz+YWugeF1UXMrf530PQFJCHD1bpQd6E31yM8lu3CCc4YtIlFJPoR5bt2Vn4A7swuJS5q4uY88+7/vVLrsRffzaREHbTDo0bUycCtgicggaPopCO/fsY+7qMgr9nsT04lI2btsNQGpyAn1yfrwUNq9NBo0aqCMoIh4NH0Wh5MR4CtpmUdA2C/AK2Cs2bg9cCltUXMoj/1gPeJP+ndAilfycTPLbejfXtdKkfyJSA/UUokzZ9j1MX+X1IgpXlDJz1WZ27NkH/DjpX36ON+SkSf9EYod6CjEqvWEi/Ts3o39nb9K/vfv2s/D7rRSu2ETRys0HTPqXnBhHXusMCtr6BeycTE36JxLj1FOIQWvLDpz0b96aLezb7/0cdGjW2B9y0qR/ItFEhWYJ2vbde5m1quyAtSbKduwBfpz0r2LYKa9Nhib9E4lAGj6SoDVMSuCU9k04pX0TwJv0b/mGcgpX+EliZSmfLFgHQEKc0a1VeuAqp/zcTJpr0j+RqKGeggRl07bd3k11K0spWlHKrJLN7Ko06V/FzLCa9E+kflJPQWpVVqMkzu7anLO7Nge8Sf/mr91C4YpNTF9ZytfLNvLOTG/hvIpJ/yoWI+qtSf9EIoaSghyVpIQ4erXJoFebDODHSf+mrywNDDs99dmSAyb9q7wGdk5WQxWwReohDR9JyJTv2stMf52JwuJNzFy5ma2VJv3Lz80IrFynSf9EQkvDRxJ2jRskcFrHbE7rmA14k/4tWbeVwhWlgfrER/N+AA6c9K9i2EmT/onUPfUUJKzWbd3J9OLN/rDTJuau3sLufV4Bu22ThoH1r/NzM+nYTJP+iRwt3acgEali0r/KN9dVnfSv4konTfonEjwNH0lEqm7Sv+KN2w+YGfaxTxbjHMQZnNAiLXApbEHbLFqmJ6uALXIM1FOQiFO2Yw8zVvqT/hV7k/5t3+1N+ndcWnJguCk/N5OuLTXpnwiopyBRLD0lkTM7N+PMKpP+VR5yen/OgZP+VSxGpEn/RA5PPQWJSmvLdjC9eDOFxZuY7k/6t9ef9K9900YUVBSw22rSP4kNKjSLVLJj9z5mlWw+oDdRedK/PhUzw+Zk0rN1BilJumdCoouGj0QqSUmKp+/xTeh7fOVJ/7YFVqwrKi7l04UHTvqXX2kNbE36J7FCPQUR36Ztu5nhTx9eWFzKrFU/TvrXKiPlgMWIuhyXSoIK2BJB1FMQOUJZjZIYcEJzBpzw46R/C9ZuodC/FHbq8gMn/euVkxG4uU6T/km0UE9BJEjOOVZvPnDVugVrtxw06V/FGtia9E/qExWaRepA+a69zFq1OTDkNKO49BCT/mXSvVW6Jv2TsNHwkUgdaNwggVM7ZHNqB2/Sv/37HUvWlVPoF7CnF1ea9C8+jh6t0wNJok9OJk1TNemf1C8h7SmY2fnA40A88Jxz7k9VtucALwEZfps7nHMfHG6f6ilIpFm/ddcB61/PKSkLTPqX26RhIEkU5GZp0j8JmbAPH5lZPLAYOAcoAaYBw5xz8yu1GQvMcM49Y2ZdgQ+cc20Pt18lBYl0u/YePOnfhvIfJ/3rnfPjYkS9NOmf1JL6MHx0ErDUObfcD+h14AJgfqU2DkjzH6cDa0IYj0i90CAh3r9q6cdJ/1Zu2u6tWLey+kn/Ks/n1CojRQVsCZlQ9hSGAOc7537hP78KONk5N6JSmxbAx0Am0Ag42zlXVM2+hgPDAXJycvKLi4tDErNIfVG2Yw8zV1Xcgb2JGSt/nPSveVoDbujXnmtObRfmKCWS1IeeQnV/ylTNQMOAcc65R8zsFGC8mXV3zu0/4E3OjQXGgjd8FJJoReqR9JRE+nVqSr9OTQFv0r9FP3iT/r0/ey33vjufVpkNOadr8zBHKtEmlLdklgBtKj1vzcHDQ9cCkwCcc18DyUB2CGMSiUgJ8XF0a5nO1ae05aWfn0TP1un8euJMlq4rD3doEmVCmRSmAR3NrJ2ZJQGXAZOrtFkJDAAwsxPwksL6EMYkEvGSE+MZc2U+yYlxDB9fyJade8IdkkSRkCUF59xeYATwEbAAmOScm2dm95vZYL/ZbcAvzWwW8BpwjYu0u+lEwqBlRgqjL+/Dyo3b+fXrM9m/X/9tpHbojmaRCPbSVyu4Z/I8bh7QkVvP6RTucKQeC7bQrGkeRSLY1afkMiS/NU98uoSP5n0f7nAkCigpiEQwM+OBC7uT1zqdWyfOZOm6reEOSSKckoJIhEtOjGfMVfmkJMUz/OUiFZ7lmCgpiESBFukpPH1FPis3qfAsx0ZJQSRKnNQui7t/1pVPF67jL58sDnc4EqGUFESiyFV9c7kkvzVPfLaUv89V4VmOnJKCSBQxM35/YXfy2mRw26SZLPlBhWc5MkoKIlEmOTGeZ6/MJyUpgeHjiyjbocKzBE9JQSQKHZeezDNX9mHVpu386vUZ7FPhWYKkpCASpU5sm8U9g7vx+aL1PPYPFZ4lOEoKIlHsypNzGFrQhqc+X8rf564NdzgSAZQURKKYmXH/hd3o1SaDWyfNYrEKz1IDJQWRKNcgwZtqu1GDBIa/XEjZdhWe5dCUFERiwHHpyTxzRR9Wb97BLRNVeJZDU1IQiREFbbO452fd+Oei9Tz6j0XhDkfqqVCu0Swi9cwVJ+cwb00Zoz9fRreW6fykR4twhyT1jHoKIjHEzLh3cDf65GTwmzdmseh7FZ7lQEoKIjGmQUI8z1QUnser8CwHUlIQiUHN05IZc2Uf1mzewUjd8SyVKCmIxKj83CzuG9ydLxevZ9THKjyLR4VmkRh2+ck5zFldxjP/XEb3lun8tKcKz7FOPQWRGHfv4K6BwvPC77eEOxwJMyUFkRhXccdzanICw18uYvP23eEOScJISUFEaJaWzDNX5rO2bAcjX1PhOZYpKYgIAPm5mdx/QXf+tWQDD3+kwnOsUqFZRAKGnZTD3NVljPliGd1bpTGoZ8twhyR1TD0FETnAPT/rRkFuJre/MZv5a1R4jjVKCiJygKSEOJ6+sg9pKQlc90ohpdtUeI4lSgoicpBmqcmMuTKfH8p2cfPrM9i7b3+4Q5I6oqQgItXqnZPJ7y/spsJzjAlpUjCz881skZktNbM7DtHmUjObb2bzzOzVUMYjIkdm6Ik5XNk3h2e/XM7kWWvCHY7UgZBdfWRm8cBo4BygBJhmZpOdc/MrtekI3Amc6pwrNbNmoYpHRI7O3YO6sej7rfz2zVl0aNqYri3Twh2ShFAoewonAUudc8udc7uB14ELqrT5JTDaOVcK4JxbF8J4ROQoJCXEMfqKPmSkJDF8vArP0S6USaEVsKrS8xL/tco6AZ3MbIqZTTWz86vbkZkNN7NCMytcv359iMIVkUNplprMM1f2Yd2WXYx8TYXnaBbKpGDVvFb13vkEoCNwJjAMeM7MMg56k3NjnXMFzrmCpk2b1nqgIlKz3jmZPHBRd/69dAMPqfActUKZFEqANpWetwaqVqpKgHecc3ucc98Bi/CShIjUQ5cWtOHqU3IZ++Vy3pm5OtzhSAiEMilMAzqaWTszSwIuAyZXafM20B/AzLLxhpOWhzAmETlG/29QV05qm8Xv/m8289aUhTscqWUhSwrOub3ACOAjYAEwyTk3z8zuN7PBfrOPgI1mNh/4HLjdObcxVDGJyLFLjPcKz5kNkxj+chGbVHiOKuZcZE2RW1BQ4AoLC8MdhkjMm7VqM5c8+zUFuZm8/POTSIjXvbD1mZkVOecKamqn76KIHJW8Nhk8eGF3vlq2kT99uDDc4Ugt0dTZInLULilow9zVZTz37+/o0TqdC3pVvepcIs1hk4KZ3Xq47c65R2s3HBGJNP87qCsLvt/Kb9+cTfumjeneKj3cIckxqGn4KLWGLxGJcYnxcTx9RR+yGiVx3XgVniOdCs0iUitml2xmyJivyc/JZPy1KjzXN8EWmmsaPnricNudczcfaWAiEp16ts7gjxf14LY3ZvGHDxZy98+6hjskOQo1FZqL6iQKEYkK/5nfmjmry3hhynd0b5XGxX1ahzskOUKHTQrOuZfqKhARiQ53/fQEFqzdwp1/m0On5qkqPEeYoAb9zKypmY0ysw/M7LOKr1AHJyKRp+KO5yZ+4Xlj+a5whyRHINhK0AS8qSraAfcBK/DmNhIROUh24wY8e1UBG8p3cdOr09mjqbYjRrBJoYlz7nlgj3PuC+fcz4G+IYxLRCJcj9bp/PHiHkxdvok/fLAg3OFIkIK9o3mP/+9aM/sp3hTYqiCJyGFd3McrPL84ZQU9WqWr8BwBgk0KD5hZOnAb8CSQBvw6ZFGJSNT4n5/8WHju2CyVHq1VeK7Pgho+cs6955wrc87Ndc71d87lO+eqro0gInKQxPg4Rl/eh+zGDbhufCEbVHiu14K9+uilystkmlmmmb0QurBEJJo0adyAZ6/KZ+O23dw0QYXn+izYQnNP59zmiifOuVKgd2hCEpFo1L1VOn/+z558890mHnxfhef6KtikEGdmmRVPzCwLTbstIkfowt6tuPa0doz7agVvFK4KdzhSjWB/sT8CfGVmbwIOuBR4MGRRiUjUunNgFxas3cJdb8+lU/NU8tpk1PwmqTPBFppfBv4T+AFYD1zsnBsfysBEJDolxMfx1OV9aNq4Ade/UsT6rSo81ydHMrdtFrDNOfcksN7M2oUoJhGJclmNknj2qnxKt+/WHc/1TLBXH90D/A64038pEXglVEGJSPSrKDx/+90mHnhvfrjDEV+wPYWLgMHANgDn3Bq08pqIHKMLerXiF6e146Wvi1V4rieCTQq7nbdEmwMws0ahC0lEYskdA7twaocm3PX2XGat2lzzGySkgk0Kk8zsWSDDzH4JfAI8F7qwRCRWJMTH8eQwr/B83XgVnsMt2KuPRgFvAv8HdAbuds4ddqlOEZFgZTVKYuzV+WzesZsbJxSxe68Kz+ES9NVHzrl/OOdud879BvjMzK4IYVwiEmO6tUznoSF5TFtRyu9VeA6bwyYFM0szszvN7CkzO9c8I4DleDewiYjUmsF5LRl+xvGMn1rMpGkqPIdDTXc0jwdKga+BXwC3A0nABc65mSGOTURi0G/P68z8NVv437fn0rF5Y3rnZNb8Jqk1NQ0fHe+cu8Y59ywwDCgABikhiEioeIXn3jRP9+54Xrd1Z7hDiik1JYWKFddwzu0DvnPObQ1tSCIS6zIbJfHslQWU7djDja9MV+G5DtWUFPLMbIv/tRXoWfHYzLbURYAiEpu6tkzjoSF5FBaXcv9788IdTsw4bFJwzsU759L8r1TnXEKlx2k17dzMzjezRWa21MzuOEy7IWbmzKzgaE5CRKLT4LyWXHfG8bwydSUTp60Mdzgx4UgmxDsiZhYPjAYGAl2BYWbWtZp2qcDNwDehikVEItdvz+/C6R2z+X9vz2P6ytJwhxP1QpYUgJOApc655c653cDrwAXVtPs98BCgapKIHCQ+znhyWG+OS0/m+vFFrNuiXxWhFMqk0AqofKFxif9agJn1Bto459473I7MbLiZFZpZ4fr162s/UhGp1zIaenc8b925lxsmqPAcSqFMClbNay6w0SwOeAy4raYdOefGOucKnHMFTZs2rcUQRSRSdDkujYcv6UlRcSn3vqvCc6iEMimUAG0qPW8NrKn0PBXoDvzTzFYAfYHJKjaLyKEM6tmS6/u159VvVvLatyo8h0Iok8I0oKOZtTOzJOAyYHLFRudcmXMu2znX1jnXFpgKDHbOFYYwJhGJcLef15kzOjXl7nfmUlSswnNtC1lScM7tBUYAHwELgEnOuXlmdr+ZDQ7VcUUkusXHGU9c1osW6Snc8EoRP6jwXKvMWzsnchQUFLjCQnUmRGLdwu+3cPHTX9HluFReG96XBgnx4Q6pXjOzIudcjcPzoRw+EhEJmS7HpfHwkDymr9zMvZM11XZtUVIQkYj1054tuOHM9rz27Upe/UaF59qgpCAiEe0353amX6em3DN5LkXFm8IdTsRTUhCRiOYVnnvTMiOF61+ZrsLzMVJSEJGIl94wkbFXFbBt116uf6WIXXv3hTukiKWkICJRofNxqTxySR4zVm7mnnfmEWlXVtYXSgoiEjUG9mjBTf3b8/q0VUxQ4fmoKCmISFS59ZzOnNm5Kfe9O4/CFSo8HyklBRGJKvFxxuOX9aaVX3j+vkyF5yOhpCAiUSc9JZGxVxewY7cKz0dKSUFEolKn5qk8cmkeM1dt5u63VXgOlpKCiESt87u3YET/DkwsXMUrKjwHRUlBRKLar8/pRP/OTblv8jy+/U6F55ooKYhIVIuPM/5yWW/aZDXkxglFrC3bEe6Q6jUlBRGJeukpiYy9Kp8du/dx/fgidu5R4flQlBREJCZ0bJ7Ko0N7MaukjP/39lwVng9BSUFEYsZ53Y7j5rM68EZRCeOnFoc7nHpJSUFEYsqvzu7EgC7NuP/d+XyzfGO4w6l3lBREJKbExRmPXdaLnKyG3PTqdNZsVuG5MiUFEYk5acmJjL06n5179nPDKyo8V6akICIxqUOzVB69NI9ZJWXc9ZYKzxWUFEQkZp3b7ThuHtCR/5tewktfrQh3OPWCkoKIxLRfDejI2Sc04/fvL2CqCs9KCiIS2+LijMeG9iK3SUNumjCd1TFeeFZSEJGYl5rsrfG8a+/+mL/jWUlBRATo0Kwxjw3txZzVZfzPW3NitvCspCAi4juna3N+dXZH/jZ9NeNitPCspCAiUsnNZ3XknK7NeeD9BXy9LPYKz0oKIiKVxMUZj16aR9sm3h3PsVZ4VlIQEakiNdlb43nP3v1cN74wpgrPIU0KZna+mS0ys6Vmdkc12281s/lmNtvMPjWz3FDGIyISrPZNvcLz3NVbuPNvsVN4DllSMLN4YDQwEOgKDDOzrlWazQAKnHM9gTeBh0IVj4jIkTq7a3N+fXYn3pqxmhemrAh3OHUilD2Fk4ClzrnlzrndwOvABZUbOOc+d85t959OBVqHMB4RkSM28qwOnNu1OX/4YAFfLdsQ7nBCLpRJoRWwqtLzEv+1Q7kW+LC6DWY23MwKzaxw/fr1tRiiiMjhxcUZjw7tRbvsRox4dQYlpdtrflMEC2VSsGpeq3ZQzsyuBAqAh6vb7pwb65wrcM4VNG3atBZDFBGpWeMGCYy9Kp89+/Zz3fgiduyO3sJzKJNCCdCm0vPWwJqqjczsbOAuYLBzblcI4xEROWrHN23M45f1Yv7aLdz5t9lRW3gOZVKYBnQ0s3ZmlgRcBkyu3MDMegPP4iWEdSGMRUTkmJ3VpTm3nt2Jt2eu4fl/fxfucEIiZEnBObcXGAF8BCwAJjnn5pnZ/WY22G/2MNAYeMPMZprZ5EPsTkSkXripfwfO69acP364kK+WRl/h2SKtC1RQUOAKCwvDHYaIxLDyXXu5aPQUNpTvYvKI02iT1TDcIdXIzIqccwU1tdMdzSIiR6hxgwTGXl3A3v0u6grPSgoiIkehXXYjnrisNwu+38Lv/i96Cs9KCiIiR6l/l2b85tzOTJ61huf+FR2FZyUFEZFjcOOZ7RnY/Tj++OEC/r0k8gvPSgoiIsfAzBh1SR4dmjVmxGvTWbUpsu94VlIQETlGjRokMPaqAvbvdwyP8MKzkoKISC1om92IJ4b1ZuH3W/htBBeelRRERGrJmZ29wvO7s9bw138tD3c4R0VJQUSkFt14Znt+0uM4/vThQv61JPJmdVZSEBGpRWbGw0Py6NgslZGvzWDlxsgqPCspiIjUskYNEhh7db5feC5k++694Q4paEoKIiIhkNvEKzwv+mErt78ZOYVnJQURkRA5s3MzfnteF96fvZZnv4yMwrOSgohICF3f73h+2rMFD/19IV8urv+FZyUFEZEQ8grPPenU3Cs8F2/cFu6QDktJQUQkxBomeXc8A1w3vqheF56VFERE6kBOk4Y8Oaw3i+t54VlJQUSkjpzRqSm/Pd8efO1hAAAP0klEQVQrPI/5on4WnpUURETq0HVnHM+gni146KOFfFEPC89KCiIidcjMeGhITzo3T2Xkq9NZsaF+FZ6VFERE6lhF4dnMGD6+kG276k/hOSHcAdSGPXv2UFJSws6dO8MdikSY5ORkWrduTWJiYrhDkRiT06QhT13em/964Vtuf3MWoy/vg5mFO6zoSAolJSWkpqbStm3bevGhSmRwzrFx40ZKSkpo165duMORGHR6x6bcMbALf/hgIU//cxk39e8Q7pCiY/ho586dNGnSRAlBjoiZ0aRJE/UwJax+efrxDM5ryaiPF/H5onXhDic6kgKghCBHRT83Em5mxp//syddjkvjltdmhL3wHDVJQUQkUqUkxTP2qnzi4rzCc3kYC89KCrUkPj6eXr160b17dy655BK2bz/2hTUKCwu5+eabD7l9zZo1DBky5JiPE0rjxo1jxIgRANx7772MGjUqzBGJ1E9tshry1LA+LF1Xzu1vzArbHc9KCrUkJSWFmTNnMnfuXJKSkhgzZswB251z7N+//4j2WVBQwBNPPHHI7S1btuTNN988qngPZ+/e+nN5XE0iKVaRmpzWMZs7B57Ah3O/5+l/LgtLDFFx9VFl9707j/lrttTqPru2TOOen3ULuv3pp5/O7NmzWbFiBQMHDqR///58/fXXvP322yxatIh77rmHXbt20b59e1588UUaN27MtGnTuOWWW9i2bRsNGjTg008/paioiFGjRvHee+/xxRdfcMsttwDeGOSXX37Jxo0bGTRoEHPnzmXnzp3ccMMNFBYWkpCQwKOPPkr//v0ZN24ckydPZvv27SxbtoyLLrqIhx566KCYx40bx/vvv8/OnTvZtm0bn332GQ8//DCTJk1i165dXHTRRdx3330AvPzyy4waNQozo2fPnowfP553332XBx54gN27d9OkSRMmTJhA8+bNg/q8fvjhB66//nqWL/du+3/mmWdo2bJl4NwARo0aRXl5Offeey9nnnkm//Ef/8GUKVM466yzePHFF1m+fDlxcXFs376dzp07s3z5clauXMlNN93E+vXradiwIX/961/p0qVL0N9HkXD4xentmLumjFEfL6JrizT6d2lWp8ePuqQQbnv37uXDDz/k/PPPB2DRokW8+OKLPP3002zYsIEHHniATz75hEaNGvHnP/+ZRx99lDvuuIOhQ4cyceJETjzxRLZs2UJKSsoB+x01ahSjR4/m1FNPpby8nOTk5AO2jx49GoA5c+awcOFCzj33XBYvXgzAzJkzmTFjBg0aNKBz586MHDmSNm3aHBT7119/zezZs8nKyuLjjz9myZIlfPvttzjnGDx4MF9++SVNmjThwQcfZMqUKWRnZ7Np0yYATjvtNKZOnYqZ8dxzz/HQQw/xyCOPBPWZ3XzzzfTr14+33nqLffv2UV5eTmlp6WHfs3nzZr744gsApk+fzhdffEH//v159913Oe+880hMTGT48OGMGTOGjh078s0333DjjTfy2WefBRWTSLiYGX+6uCdLfijn5tdnMHnEabTLblRnx4+6pHAkf9HXph07dtCrVy/A6ylce+21rFmzhtzcXPr27QvA1KlTmT9/PqeeeioAu3fv5pRTTmHRokW0aNGCE088EYC0tLSD9n/qqady6623csUVV3DxxRfTunXrA7b/+9//ZuTIkQB06dKF3NzcQFIYMGAA6enpAHTt2pXi4uJqk8I555xDVlYWAB9//DEff/wxvXv3BqC8vJwlS5Ywa9YshgwZQnZ2NkCgfUlJCUOHDmXt2rXs3r37iK77/+yzz3j55ZcBrzaTnp5eY1IYOnToAY8nTpxI//79ef3117nxxhspLy/nq6++4pJLLgm027VrV9AxiYRTSlI8z16Vz+Cn/s3wlwt566ZTadygbn5dh7SmYGbnm9kiM1tqZndUs72BmU30t39jZm1DGU8oVdQUZs6cyZNPPklSUhIAjRr9mOGdc5xzzjmBdvPnz+f555/HOVfjpZF33HEHzz33HDt27KBv374sXLjwgO2HK0o1aNAg8Dg+Pp69e/fy1ltv0atXL3r16kVhYWG1sd55552BWJcuXcq11157yFhHjhzJiBEjmDNnDs8+++wxX/ufkJBwQA2m6v4qxzp48GA+/PBDNm3aRFFREWeddRb79+8nIyMjEP/MmTNZsGDBMcUkUpfaZDXkqcv7sHzDNm6bNJP9++um8ByypGBm8cBoYCDQFRhmZl2rNLsWKHXOdQAeA/4cqnjqg759+zJlyhSWLl0KwPbt21m8eDFdunRhzZo1TJs2DYCtW7ceVEBdtmwZPXr04He/+x0FBQUHJYUzzjiDCRMmALB48WJWrlxJ586dDxnLRRddFPhlWVBQcND28847jxdeeIHy8nIAVq9ezbp16xgwYACTJk1i48aNAIHho7KyMlq1agXASy+9dESfy4ABA3jmmWcA2LdvH1u2bKF58+asW7eOjRs3smvXLt57771Dvr9x48acdNJJ3HLLLQwaNIj4+HjS0tJo164db7zxBuAluVmzZh1RXCLhdmqHbO4c2IWP5v3A6M+X1skxQ9lTOAlY6pxb7pzbDbwOXFClzQVAxW+QN4EBFsV3EzVt2pRx48YxbNgwevbsGfiLPykpiYkTJzJy5Ejy8vI455xzDvrL+C9/+Qvdu3cnLy+PlJQUBg4ceMD2G2+8kX379tGjRw+GDh3KuHHjDughHKlzzz2Xyy+/nFNOOYUePXowZMgQtm7dSrdu3bjrrrvo168feXl53HrrrYB3uekll1zC6aefHhhaCtbjjz/O559/To8ePcjPz2fevHkkJiZy9913c/LJJzNo0KAaC8RDhw7llVdeOWBYacKECTz//PPk5eXRrVs33nnnnSP/IETC7NrT2nFhr5Y8+sliPlv4Q8iPZ6G6FtbMhgDnO+d+4T+/CjjZOTeiUpu5fpsS//kyv82GKvsaDgwHyMnJyS8uLj7gWAsWLOCEE04IyXlI9NPPj9R3O3bv46ZXp3NT//bk52Yd1T7MrMg5d/CwQBWhrFxU9xd/1QwUTBucc2OBsQAFBQX1cw07EZEQSUmK54VrTqyTY4Vy+KgEqHyJS2tgzaHamFkCkA5sCmFMIiJyGKFMCtOAjmbWzsySgMuAyVXaTAb+y388BPjMHeV4Vn1dBFvqN/3ciBwoZEnBObcXGAF8BCwAJjnn5pnZ/WY22G/2PNDEzJYCtwIHXbYajOTkZDZu3Kj/4HJEKtZTqHojoEgsC1mhOVQKCgpcxXX1FbTymhwtrbwmsaI+FJrrTGJiolbOEhGpBZolVUREApQUREQkQElBREQCIq7QbGbrgeIaG1YvG9hQY6voonOODTrn2HAs55zrnGtaU6OISwrHwswKg6m+RxOdc2zQOceGujhnDR+JiEiAkoKIiATEWlIYG+4AwkDnHBt0zrEh5OccUzUFERE5vFjrKYiIyGEoKYiISEBUJgUzO9/MFpnZUjM7aOZVM2tgZhP97d+YWdu6j7J2BXHOt5rZfDObbWafmlluOOKsTTWdc6V2Q8zMmVlEX74YzPma2aX+93memb1a1zHWtiB+rnPM7HMzm+H/bP8kHHHWJjN7wczW+StTVrfdzOwJ/zOZbWZ9ajUA51xUfQHxwDLgeCAJmAV0rdLmRmCM//gyYGK4466Dc+4PNPQf3xAL5+y3SwW+BKYCBeGOO8Tf447ADCDTf94s3HHXwTmPBW7wH3cFVoQ77lo47zOAPsDcQ2z/CfAh3sqVfYFvavP40dhTOAlY6pxb7pzbDbwOXFClzQXAS/7jN4EBZlbd0qCRosZzds597pzb7j+dircSXiQL5vsM8HvgISDS51UP5nx/CYx2zpUCOOfW1XGMtS2Yc3ZAmv84nYNXd4w4zrkvOfwKlBcALzvPVCDDzFrU1vGjMSm0AlZVel7iv1ZtG+ctBlQGNKmT6EIjmHOu7Fq8vzQiWY3nbGa9gTbOuffqMrAQCeZ73AnoZGZTzGyqmZ1fZ9GFRjDnfC9wpZmVAB8AI+smtLA60v/vRyQq1lOoorq/+KtedxtMm0gS9PmY2ZVAAdAvpBGF3mHP2czigMeAa+oqoBAL5nucgDeEdCZeT/BfZtbdObc5xLGFSjDnPAwY55x7xMxOAcb757w/9OGFTUh/f0VjT6EEaFPpeWsO7lIG2phZAl6383DdtfoumHPGzM4G7gIGO+d21VFsoVLTOacC3YF/mtkKvLHXyRFcbA725/od59we59x3wCK8JBGpgjnna4FJAM65r4FkvEnjollQ/9+PVjQmhWlARzNrZ2ZJeIXkyVXaTAb+y388BPjM+RWcCFXjOftDKc/iJYRIH2uGGs7ZOVfmnMt2zrV1zrXFq6MMds4VVr+7ei+Yn+u38S4owMyy8YaTltdplLUrmHNeCQwAMLMT8JLC+jqNsu5NBq72r0LqC5Q559bW1s6jbvjIObfXzEYAH+FdvfCCc26emd0PFDrnJgPP43Uzl+L1EC4LX8THLshzfhhoDLzh19RXOucGhy3oYxTkOUeNIM/3I+BcM5sP7ANud85tDF/UxybIc74N+KuZ/RpvCOWaCP8DDzN7DW8IMNuvldwDJAI458bg1U5+AiwFtgP/XavHj/DPT0REalE0Dh+JiMhRUlIQEZEAJQUREQlQUhARkQAlBRERCVBSkJhkZvvMbKaZzTWzN8ysYS3ss8DMnjjM9pZm9uaxHkcklHRJqsQkMyt3zjX2H08Aipxzj1babnj/P6J5ugSRg6inIAL/AjqYWVszW2BmTwPTgTZmdq6ZfW1m0/0eRUUiOdHMvjKzWWb2rZmlmtmZZvaev72f3xOZ6c/1n+rvf66/PdnMXjSzOf72ijuRrzGzv5nZ381siZk9FKbPRGKUkoLENH/uq4HAHP+lznjTEvcGtgH/C5ztnOsDFAK3+lMuTARucc7lAWcDO6rs+jfATc65XsDp1Wy/CcA51wNvUreXzCzZ39YLGAr0AIaaWRtE6oiSgsSqFDObifeLfiXe1CcAxf4c9eBNotcVmOK3/S8gFy9xrHXOTQNwzm3xp2CvbArwqJndDGRUs/00YLz//oVAMd5cRQCf+nM37QTm+8cUqRNRN/eRSJB2+H/FB/hzQm2r/BLwD+fcsCrtelLDVMXOuT+Z2ft4c9RM9WeorbzQz+EWdao8g+0+9P9U6pB6CiKHNhU41cw6AJhZQzPrBCwEWprZif7rqf4wVICZtXfOzXHO/RmvN9Klyr6/BK7w23YCcvCmuhYJKyUFkUNwzq3HW6TnNTObjZckuvhLQw4FnjSzWcA/8KZsruxX/uWus/DqCVVXunsaiDezOXj1iWuiYI0LiQK6JFVERALUUxARkQAlBRERCVBSEBGRACUFEREJUFIQEZEAJQUREQlQUhARkYD/D6iCVFDf1/t9AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"from sklearn.metrics import precision_recall_curve \n",
"\n",
"precision, recall, thresholds = precision_recall_curve(y_pred, y)\n",
"\n",
"# create plot\n",
"plt.plot(precision, recall, label='Precision-recall curve')\n",
"plt.xlabel('Precision')\n",
"plt.ylabel('Recall')\n",
"plt.title('Precision-recall curve')\n",
"plt.legend(loc=\"lower left\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Random over-sampling the minority class\n",
"\n",
"\n",
"\n",
"**Over-sampling** is the process of randomly duplicating observations from the minority class in order to achieve a balanced dataset. So, it replicates the observations from minority class to balance the data. It is also known as **upsampling**. It may result in overfitting due to duplication of data points. \n",
"\n",
"\n",
"The most common way of over-sampling is to resample with replacement. I will proceed as follows:-\n",
"\n",
"\n",
"First, I will import the resampling module from Scikit-Learn."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"# import resample module \n",
"\n",
"from sklearn.utils import resample"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I will create a new dataframe with an oversampled minority class as follows:-\n",
"\n",
"\n",
"1. At first, I will separate observations from Class variable into different DataFrames.\n",
"\n",
"\n",
"2. Now, I will resample the minority class with replacement. I will set the number of samples of minority class to match \n",
" that of the majority class.\n",
"\n",
"\n",
"3. Finally, I will combine the oversampled minority class DataFrame with the original majority class DataFrame."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"# separate the minority and majority classes\n",
"df_majority = df[df['Class']==0]\n",
"df_minority = df[df['Class']==1]"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# oversample minority class\n",
"\n",
"df_minority_oversampled = resample(df_minority, replace=True, n_samples=284315, random_state=0) "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"# combine majority class with oversampled minority class\n",
"\n",
"df_oversampled = pd.concat([df_majority, df_minority_oversampled])"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1 284315\n",
"0 284315\n",
"Name: Class, dtype: int64"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# display new class value counts\n",
"\n",
"df_oversampled['Class'].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can see that we have a balanced dataset. The ratio of the two class labels is now 1:1.\n",
"\n",
"Now, I will plot the bar plot of the above two classes."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0x6b5f3a2a90>"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAC1BJREFUeJzt3VGInflZx/HvbxPihS29MIPUJNMJNiBRi8Ux61UVXTGhkAiukIDQlcpQaKhaL5qihDbe6Ar2KheNuFKEmq69GutIwGovRLZmVpdKNqQdwmqGgKZ2WRGxadzHi0zbw+lJ5j0zZzKbJ98PBM7/ff+c82wYvrx555yzqSokSb08tdsDSJJmz7hLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWpo72698P79+2thYWG3Xl6SHksvv/zy16tqbrN9uxb3hYUFVldXd+vlJemxlORfh+zztowkNWTcJakh4y5JDRl3SWrIuEtSQ4PinuR4khtJ1pKcm3D+uSR3kryy8efXZz+qJGmoTd8KmWQPcBH4BWAduJpkuapeHdv6uao6uwMzSpKmNOTK/RiwVlU3q+oucBk4tbNjSZK2Y8iHmA4At0bW68DTE/b9cpL3AV8Ffquqbo1vSLIELAHMz89PP+0uWDj3V7s9Qiuv/f77d3uEPj7xjt2eoJdPvLHbE8zUkCv3TDg2/n/V/ktgoareA/wN8JlJT1RVl6pqsaoW5+Y2/fSsJGmLhsR9HTg0sj4I3B7dUFX/WVXf3Fj+MfCTsxlPkrQVQ+J+FTiS5HCSfcBpYHl0Q5J3jixPAtdnN6IkaVqb3nOvqntJzgJXgD3AC1V1LckFYLWqloGPJDkJ3AO+ATy3gzNLkjYx6Fshq2oFWBk7dn7k8ceBj892NEnSVvkJVUlqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDQ2Ke5LjSW4kWUty7iH7nk1SSRZnN6IkaVqbxj3JHuAicAI4CpxJcnTCvrcDHwG+POshJUnTGXLlfgxYq6qbVXUXuAycmrDv94Dngf+d4XySpC0YEvcDwK2R9frGse9I8l7gUFV9YYazSZK2aEjcM+FYfedk8hTwKeC3N32iZCnJapLVO3fuDJ9SkjSVIXFfBw6NrA8Ct0fWbwd+DPhSkteAnwaWJ/1StaouVdViVS3Ozc1tfWpJ0kMNiftV4EiSw0n2AaeB5W+frKo3qmp/VS1U1QLwEnCyqlZ3ZGJJ0qY2jXtV3QPOAleA68CLVXUtyYUkJ3d6QEnS9PYO2VRVK8DK2LHzD9j7s9sfS5K0HX5CVZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8ZdkhoaFPckx5PcSLKW5NyE8x9K8i9JXkny90mOzn5USdJQm8Y9yR7gInACOAqcmRDvz1bVj1fVTwDPA38080klSYMNuXI/BqxV1c2qugtcBk6Nbqiq/xpZfj9QsxtRkjStvQP2HABujazXgafHNyX5MPBRYB/wc5OeKMkSsAQwPz8/7aySpIGGXLlnwrHvuTKvqotV9cPAx4DfnfREVXWpqharanFubm66SSVJgw2J+zpwaGR9ELj9kP2XgV/azlCSpO0ZEverwJEkh5PsA04Dy6MbkhwZWb4f+NrsRpQkTWvTe+5VdS/JWeAKsAd4oaquJbkArFbVMnA2yTPAt4DXgQ/s5NCSpIcb8gtVqmoFWBk7dn7k8W/MeC5J0jb4CVVJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0NinuS40luJFlLcm7C+Y8meTXJV5J8Mcm7Zj+qJGmoTeOeZA9wETgBHAXOJDk6tu2fgcWqeg/weeD5WQ8qSRpuyJX7MWCtqm5W1V3gMnBqdENV/V1V/c/G8iXg4GzHlCRNY0jcDwC3RtbrG8ce5IPAX29nKEnS9uwdsCcTjtXEjcmvAovAzzzg/BKwBDA/Pz9wREnStIZcua8Dh0bWB4Hb45uSPAP8DnCyqr456Ymq6lJVLVbV4tzc3FbmlSQNMCTuV4EjSQ4n2QecBpZHNyR5L/Bp7of9P2Y/piRpGpvGvaruAWeBK8B14MWqupbkQpKTG9v+EHgb8BdJXkmy/ICnkyQ9AkPuuVNVK8DK2LHzI4+fmfFckqRt8BOqktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1NCguCc5nuRGkrUk5yacf1+Sf0pyL8mzsx9TkjSNTeOeZA9wETgBHAXOJDk6tu3fgOeAz856QEnS9PYO2HMMWKuqmwBJLgOngFe/vaGqXts49+YOzChJmtKQ2zIHgFsj6/WNY1NLspRkNcnqnTt3tvIUkqQBhsQ9E47VVl6sqi5V1WJVLc7NzW3lKSRJAwyJ+zpwaGR9ELi9M+NIkmZhSNyvAkeSHE6yDzgNLO/sWJKk7dg07lV1DzgLXAGuAy9W1bUkF5KcBEjyU0nWgV8BPp3k2k4OLUl6uCHvlqGqVoCVsWPnRx5f5f7tGknSW4CfUJWkhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1NCguCc5nuRGkrUk5yac/74kn9s4/+UkC7MeVJI03KZxT7IHuAicAI4CZ5IcHdv2QeD1qno38CngD2Y9qCRpuCFX7seAtaq6WVV3gcvAqbE9p4DPbDz+PPDzSTK7MSVJ09g7YM8B4NbIeh14+kF7qupekjeAHwC+PropyRKwtLH87yQ3tjK0JtrP2N/3W1H8N92T6LH42eSTj8316LuGbBoS90n/xbWFPVTVJeDSgNfUlJKsVtXibs8hjfNnc3cMuS2zDhwaWR8Ebj9oT5K9wDuAb8xiQEnS9IbE/SpwJMnhJPuA08Dy2J5l4AMbj58F/raqvufKXZL0aGx6W2bjHvpZ4AqwB3ihqq4luQCsVtUy8CfAnyVZ4/4V++mdHFoTebtLb1X+bO6CeIEtSf34CVVJasi4S1JDxl2SGhryPndJGizJj3D/U+sHuP95l9vAclVd39XBnjBeuUuamSQf4/5XlAT4R+6/lTrAn0/60kHtHN8t00ySX6uqP93tOfRkSvJV4Eer6ltjx/cB16rqyO5M9uTxyr2fT+72AHqivQn80ITj79w4p0fEe+6PoSRfedAp4Acf5SzSmN8Evpjka3z3CwfngXcDZ3dtqieQt2UeQ0n+HfhF4PXxU8A/VNWkKyfpkUjyFPe/KvwA938m14GrVfV/uzrYE8Yr98fTF4C3VdUr4yeSfOnRjyN9V1W9Cby023M86bxyl6SG/IWqJDVk3CWpIeMuSQ0Zd0lq6P8BshdYHBwkBd8AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# view the distribution of percentages within the Class column\n",
"\n",
"\n",
"(df_oversampled['Class'].value_counts()/np.float(len(df_oversampled))).plot.bar()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above bar plot shows that we have a balanced dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I will train another model using Logistic Regression and check its accuracy, but this time on the balanced dataset."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy : 93.76%\n"
]
}
],
"source": [
"# declare feature vector and target variable\n",
"X1 = df_oversampled.drop(['Class'], axis=1)\n",
"y1 = df_oversampled['Class']\n",
"\n",
"\n",
"# instantiate the Logistic Regression classifier\n",
"logreg1 = LogisticRegression()\n",
"\n",
"\n",
"# fit the classifier to the imbalanced data\n",
"clf1 = logreg1.fit(X1, y1)\n",
"\n",
"\n",
"# predict on the training data\n",
"y1_pred = clf1.predict(X1)\n",
"\n",
"\n",
"# print the accuracy\n",
"accuracy1 = accuracy_score(y1_pred, y1)\n",
"\n",
"print(\"Accuracy : %.2f%%\" % (accuracy1 * 100.0))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now have a balanced dataset. Although the accuracy is slightly decreased, but it is still quite high and acceptable. \n",
"This accuracy is more meaningful as a performance metric."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Random under-sampling the majority class\n",
"\n",
"\n",
"The **under-sampling** methods work with the majority class. In these methods, we randomly eliminate instances of the majority class. It reduces the number of observations from majority class to make the dataset balanced. This method is applicable when the dataset is huge and reducing the number of training samples make the dataset balanced.\n",
"\n",
"\n",
"The most common technique for under-sampling is resampling without replacement.\n",
"\n",
"\n",
"I will proceed exactly as in the case of random over-sampling."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"# separate the minority and majority classes\n",
"df_majority = df[df['Class']==0]\n",
"df_minority = df[df['Class']==1]"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"# undersample majority class\n",
"\n",
"df_majority_undersampled = resample(df_majority, replace=True, n_samples=492, random_state=0) "
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"# combine majority class with oversampled minority class\n",
"\n",
"df_undersampled = pd.concat([df_minority, df_majority_undersampled])"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1 492\n",
"0 492\n",
"Name: Class, dtype: int64"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# display new class value counts\n",
"\n",
"df_undersampled['Class'].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can see that the new dataframe `df_undersampled` has fewer observations than the original one `df` and the ratio of the two classes is now 1:1.\n",
"\n",
"Again, I will train a model using Logistic Regression classifier."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy : 93.90%\n"
]
}
],
"source": [
"# declare feature vector and target variable\n",
"X2 = df_undersampled.drop(['Class'], axis=1)\n",
"y2 = df_undersampled['Class']\n",
"\n",
"\n",
"# instantiate the Logistic Regression classifier\n",
"logreg2 = LogisticRegression()\n",
"\n",
"\n",
"# fit the classifier to the imbalanced data\n",
"clf2 = logreg2.fit(X2, y2)\n",
"\n",
"\n",
"# predict on the training data\n",
"y2_pred = clf2.predict(X2)\n",
"\n",
"\n",
"# print the accuracy\n",
"accuracy2 = accuracy_score(y2_pred, y2)\n",
"\n",
"print(\"Accuracy : %.2f%%\" % (accuracy2 * 100.0))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, we can see that we have a slightly decreased accuracy but it is more meaningful now."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Apply Tree-Based Algorithms"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"# declare input features (X) and target variable (y)\n",
"X4 = df.drop('Class', axis=1)\n",
"y4 = df['Class']\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"# import Random Forest classifier\n",
"from sklearn.ensemble import RandomForestClassifier\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"# instantiate the classifier \n",
"clf4 = RandomForestClassifier()\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
" max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
" min_impurity_decrease=0.0, min_impurity_split=None,\n",
" min_samples_leaf=1, min_samples_split=2,\n",
" min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=None,\n",
" oob_score=False, random_state=None, verbose=0,\n",
" warm_start=False)"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# fit the classifier to the training data\n",
"clf4.fit(X4, y4)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"# predict on training set\n",
"y4_pred = clf4.predict(X4)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy : 99.99%\n"
]
}
],
"source": [
"# compute and print accuracy\n",
"accuracy4 = accuracy_score(y4_pred, y4)\n",
"print(\"Accuracy : %.2f%%\" % (accuracy4 * 100.0))"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROC-AUC : 0.9999990706517691\n"
]
}
],
"source": [
"# compute and print ROC-AUC\n",
"\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"y4_prob = clf4.predict_proba(X4)\n",
"y4_prob = [p[1] for p in y4_prob]\n",
"print(\"ROC-AUC : \" , roc_auc_score(y4, y4_prob))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10.\tRandom under-sampling and over-sampling with imbalanced-learn\n",
"\n",
"\n",
"\n",
"There is a Python library which enable us to handle the imbalanced datasets. It is called **Imbalanced-Learn**. It is a Python library which contains various algorithms to handle the imbalanced datasets. It can be easily installed with the `pip` command. This library contains a `make_imbalance` method to exasperate the level of class imbalance within a given dataset.\n",
"\n",
"\n",
"Now, I will demonstrate the technique of random undersampling and oversampling with imbalanced learn. \n",
"\n",
"\n",
"First of all, I will import the `imbalanced learn` library.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"# import imbalanced learn library\n",
"\n",
"import imblearn"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then, I will import the `RandomUnderSampler` class. It is a quick and easy way to balance the data by randomly selecting a subset of data for the targeted classes. "
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"# import RandomUnderSampler\n",
"from imblearn.under_sampling import RandomUnderSampler\n",
"\n",
"# instantiate the RandomUnderSampler\n",
"rus = RandomUnderSampler(return_indices=True)\n",
"\n",
"\n",
"# fit the RandomUnderSampler to the dataset\n",
"X_rus, y_rus, id_rus = rus.fit_sample(X, y)\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removed indices: [ 5836 202223 199184 217162 156765 119722 9747 96096 46243 260470\n",
" 137149 9617 186704 172923 53824 264531 186954 148131 274313 50064\n",
" 87518 235078 91080 29077 141762 270962 270486 116652 263295 153042\n",
" 197270 105049 204953 53555 163533 115226 19172 128673 208375 107624\n",
" 20913 36306 90589 148390 18284 100420 211724 144616 279522 197900\n",
" 96253 13239 141860 114292 256405 119907 29937 24333 256844 226620\n",
" 41229 235689 75305 23532 13676 26830 132834 238893 131701 125698\n",
" 159945 196198 276122 10986 58949 7352 256147 168740 130487 98300\n",
" 161448 257972 117239 25459 123127 231101 77134 281171 50599 2473\n",
" 153956 31603 31999 283559 153970 251528 99568 129949 101736 190313\n",
" 118997 41125 31133 71329 160298 249008 126580 146635 230144 104093\n",
" 273563 267528 125426 39093 75909 117904 241892 56816 225206 33650\n",
" 5263 105116 211599 265515 21761 250814 29125 275722 44086 257401\n",
" 71780 132076 102377 214299 278445 268138 236507 95319 106287 273761\n",
" 142529 146819 135435 103856 141023 74960 181619 8311 168238 54159\n",
" 106336 229618 147865 200897 107247 255643 113321 66650 32975 20393\n",
" 180125 11994 256435 127328 41987 57728 90030 243368 41344 29823\n",
" 143290 38265 236063 217520 283773 150366 218249 122243 220355 189274\n",
" 27530 87580 221250 41056 107837 84258 50854 9063 189150 26381\n",
" 244868 55951 62812 207094 230748 192377 242018 178620 116709 188443\n",
" 208235 188563 44668 137090 197103 136151 175938 243614 267850 123543\n",
" 63129 164108 284346 218596 126940 259856 243974 207932 43625 257522\n",
" 196663 251069 173453 41363 257675 59752 212712 222609 52795 207421\n",
" 67880 47771 69700 115211 282664 246172 264648 241070 257206 75210\n",
" 238341 43663 16425 150562 63409 71794 213739 74121 128287 226199\n",
" 62370 8528 205549 112187 89638 71309 114325 84435 216889 228287\n",
" 130425 117058 280060 153016 205429 230355 20712 23043 122137 132379\n",
" 11661 176513 177958 94888 130708 36296 93731 14570 261935 251092\n",
" 252165 263671 162757 65270 277561 25114 100992 197227 44143 20917\n",
" 80714 278852 96638 90850 84163 6425 129699 136579 90116 106625\n",
" 188972 211079 101738 120161 40106 206659 258639 280200 126335 247343\n",
" 226440 130027 194130 278418 186462 82078 271763 35570 183145 271267\n",
" 279317 271526 252925 124545 190201 56605 18503 89200 166734 167539\n",
" 60858 226381 260764 82075 107619 10853 273426 191654 169236 178160\n",
" 95083 210584 63167 70200 226357 107241 160240 249633 253367 198873\n",
" 279417 37280 10344 209171 181792 135543 151665 133518 135882 212357\n",
" 280926 21195 218363 178371 215814 8014 32246 204065 22996 54619\n",
" 18406 151371 278843 251113 167432 175717 143750 253487 33354 17997\n",
" 167254 106361 20423 6917 18525 252277 96308 272383 281469 212870\n",
" 12459 18541 61863 257403 206573 167346 194616 165149 165896 59201\n",
" 10103 146561 3827 4515 140176 24129 89804 251580 111608 46719\n",
" 158238 7169 234934 121245 270536 135760 206508 26492 98444 248432\n",
" 81023 50882 277044 221110 177975 154267 257395 174818 149312 279392\n",
" 33443 205591 166024 106117 91340 95683 78512 184548 215361 129105\n",
" 74797 228909 182643 87382 1180 25699 247498 283012 192458 267131\n",
" 159556 65971 132476 133966 78890 67625 107435 48701 191604 158442\n",
" 117564 84450 129364 194944 100406 228483 76078 193637 284602 36073\n",
" 22438 57904 15759 83488 170064 191446 241572 65327 73440 96669\n",
" 240512 171764 206452 270464 251364 9144 182378 147063 128548 43986\n",
" 121838 210254 541 623 4920 6108 6329 6331 6334 6336\n",
" 6338 6427 6446 6472 6529 6609 6641 6717 6719 6734\n",
" 6774 6820 6870 6882 6899 6903 6971 8296 8312 8335\n",
" 8615 8617 8842 8845 8972 9035 9179 9252 9487 9509\n",
" 10204 10484 10497 10498 10568 10630 10690 10801 10891 10897\n",
" 11343 11710 11841 11880 12070 12108 12261 12369 14104 14170\n",
" 14197 14211 14338 15166 15204 15225 15451 15476 15506 15539\n",
" 15566 15736 15751 15781 15810 16415 16780 16863 17317 17366\n",
" 17407 17453 17480 18466 18472 18773 18809 20198 23308 23422\n",
" 26802 27362 27627 27738 27749 29687 30100 30314 30384 30398\n",
" 30442 30473 30496 31002 33276 39183 40085 40525 41395 41569\n",
" 41943 42007 42009 42473 42528 42549 42590 42609 42635 42674\n",
" 42696 42700 42741 42756 42769 42784 42856 42887 42936 42945\n",
" 42958 43061 43160 43204 43428 43624 43681 43773 44001 44091\n",
" 44223 44270 44556 45203 45732 46909 46918 46998 47802 48094\n",
" 50211 50537 52466 52521 52584 53591 53794 55401 56703 57248\n",
" 57470 57615 58422 58761 59539 61787 63421 63634 64329 64411\n",
" 64460 68067 68320 68522 68633 69498 69980 70141 70589 72757\n",
" 73784 73857 74496 74507 74794 75511 76555 76609 76929 77099\n",
" 77348 77387 77682 79525 79536 79835 79874 79883 80760 81186\n",
" 81609 82400 83053 83297 83417 84543 86155 87354 88258 88307\n",
" 88876 88897 89190 91671 92777 93424 93486 93788 94218 95534\n",
" 95597 96341 96789 96994 99506 100623 101509 102441 102442 102443\n",
" 102444 102445 102446 102782 105178 106679 106998 107067 107637 108258\n",
" 108708 111690 112840 114271 116139 116404 118308 119714 119781 120505\n",
" 120837 122479 123141 123201 123238 123270 123301 124036 124087 124115\n",
" 124176 125342 128479 131272 135718 137705 140786 141257 141258 141259\n",
" 141260 142405 142557 143188 143333 143334 143335 143336 143728 143731\n",
" 144104 144108 144754 145800 146790 147548 147605 149145 149357 149522\n",
" 149577 149587 149600 149869 149874 150601 150644 150647 150654 150660\n",
" 150661 150662 150663 150665 150666 150667 150668 150669 150677 150678\n",
" 150679 150680 150684 150687 150692 150697 150715 150925 151006 151007\n",
" 151008 151009 151011 151103 151196 151462 151519 151730 151807 152019\n",
" 152223 152295 153823 153835 153885 154234 154286 154371 154454 154587\n",
" 154633 154668 154670 154676 154684 154693 154694 154697 154718 154719\n",
" 154720 154960 156988 156990 157585 157868 157871 157918 163149 163586\n",
" 167184 167305 172787 176049 177195 178208 181966 182992 183106 184379\n",
" 189587 189701 189878 190368 191074 191267 191359 191544 191690 192382\n",
" 192529 192584 192687 195383 197586 198868 199896 201098 201601 203324\n",
" 203328 203700 204064 204079 204503 208651 212516 212644 213092 213116\n",
" 214662 214775 215132 215953 215984 218442 219025 219892 220725 221018\n",
" 221041 222133 222419 223366 223572 223578 223618 226814 226877 229712\n",
" 229730 230076 230476 231978 233258 234574 234632 234633 234705 235616\n",
" 235634 235644 237107 237426 238222 238366 238466 239499 239501 240222\n",
" 241254 241445 243393 243547 243699 243749 243848 244004 244333 245347\n",
" 245556 247673 247995 248296 248971 249167 249239 249607 249828 249963\n",
" 250761 251477 251866 251881 251891 251904 252124 252774 254344 254395\n",
" 255403 255556 258403 261056 261473 261925 262560 262826 263080 263274\n",
" 263324 263877 268375 272521 274382 274475 275992 276071 276864 279863\n",
" 280143 280149 281144 281674]\n"
]
}
],
"source": [
"# print the removed indices\n",
"print(\"Removed indices: \", id_rus)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above indices are removed from the original dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, I will demonstrate random oversampling. The process will be the same as random undersampling."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"from imblearn.over_sampling import RandomOverSampler\n",
"\n",
"ros = RandomOverSampler()\n",
"\n",
"X_ros, y_ros = ros.fit_sample(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"283823 new random points generated\n"
]
}
],
"source": [
"print(X_ros.shape[0] - X.shape[0], 'new random points generated')"
]
},
{
"attachments": {
"Tomek%20links.jpg": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wgARCAHfAn4DASIAAhEBAxEB/8QAHQABAAMBAQEBAQEAAAAAAAAAAAYHCAUJBAMCAf/EABwBAQACAwEBAQAAAAAAAAAAAAAEBQMGBwIBCP/aAAwDAQACEAMQAAAB38AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA+CqJcS5lMyTJisIQLAAAARXJjlQx5D5OPkxSMir7KhjyCK5McqGPIAAAAAAAAAAAAAAAAAAAAAAAAAAB+XF773jyvoiiPr2rVO3a39UBX2NuRKDTCXD7cq5dXxpN5VhXVlSI9w0AqbPg1XNKKvahv41WVm1lMh3lXNjZ4jSbghsRuGRG/Cpfk4dnWaH6maborLSJ2JSti549nDXdkAAAAAAAAAAAAAAAAAAAAAAAAAAzR9ly9u/1/mULLLT8+qQklhfT59fHTl/8aPnz5b8nrSfB7Vd6N4kWTnzUnA62LNxKR0VWubDIalnMpyY6h0PHpFBm54+S8FjX0jZE94keREqnn9hTYMqGtbOAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRdKBF0oEXSgRf6e+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACDzc/2P/J5y7Hrfpx9uFd1QZwVVsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFKd0bFjqYdu3RVvT5j04RZQQpz8/wBM0TYN1S3yz2vf0F6jU9tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVRa+d7Orztsbz+2jumk2NV1gQTmHQO38WT+DuWj7zs6lrp03dFSW2n5vPnYtgrapCivkY+Dzt2XWfTv7cQ7errEKu1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAf5/ojHyzGrcULKHA/KsdQ/Tugf519nDpX5OuC5eH3KK3DPKyHB/hg3WOR+mv7c7o8p61xsQ73W1RnDR5DmBDmgAAAAAAAAAAAAAAAAAAAAAAAAAACnscuUS/yF1NTdK28L3lSHTGMsED+OG6Dy1sEqbWNGxLG8n55ys8OkVYWfgz/D8vYY8gefeP4Xz4V1nkfpn+3w/dyfrQfPQAAAAAAAAAAAAAAAAAAAAAAAAAAhvnLMoh2et89ebWodCIG3hZaSy/qDhWFd5oa9XxtOqdejbyonRt8nHnj6l0Bt2n0Lvqv7ArLMKS9A+H5+s9eA8++L8uFIhvOienz4fu0feQ+egAAAAAAAAAAAAAAAAAAAAAAAI55Tet3n/Q9Z5vpBkvWMqh/tXNiWOl/0PeBwu7VeCN88F5uI6Hpm/wCOUboXZObW7OKEnXmFYJE80+WMtaKmwu2K+x4vxYZhm86H6gOf0NH3rJkX222HXfz/AENd2MH0AAAAAAAAAAAAAAAAAAAAAAAABlvUlVRL/wA0N0522tR9Ws4bPwrj1xlqvt20DVWddzwrmHX87VfqWOb1y2GamgWhK7P0Mqar4P2580dh/wAX1s+rdIaPveUojttsWufn+hruxA+gAAAAAAAAAAAAAAAAAAAAAAAAAAAADg+fOfarumhp9fuKpu9n6pl7X/epbU9v2hn9ec+16/6a9TCm64FgFTbgAcLB+3POvfNB1Rp3Fu0qO9DX9hAAAAAAAAAAAAAAAAAAAAAAAAAGdsNholi3ZniX+x8cij+yrPpiMOvr3Gt6cKl7XHbA0bkHrHAtIWpnnQlLAszFO6Hy7zlo0iy1LXTm+RGzptHz+3Fuel3SOcdJcrqvvn8/0PnoAAAAAAAAAAAAAAAAAAAAAAAAAB5memcMgbZ5Xeq8Pt2PbxOve9FMvPaM7uTOpU909Hc/Q2v+q/j3b368y9tF3Wp7P+lIsQy5ylona1OlRVW3I6568qRu7NdnV542n5+bm3HTLlHOekgAAAAAAAAAAAAAAAAAAAAAAAAAAAfxT9xsMfLnUu3Okad06q0709l0uO3CUmz48gH2V717lPpZ9nM6fKOtecsO9BI90TnNn9k530dBf687Nl1j036mGNz11lyesVdqHn0AAAAAAAAAAAAAAAAAAAAAAAAAAAAOT589aju3nrDG22q78/Pm1UckciV+Pz/c9/Q+eikolbVGmhU2/MxfuNa1OfdBEWWESWAAAAAAAAAAAAAAAAAAAAAAAAAAAAB8Oc74pSu9ZBsSiLkov0NZMI/GK/of8N7Muqir2470v/RJsQPOaG+g8f6Lzey+yc86MHn0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgM+Y8VMxPSVe4MVfSD9Lf9Y+N2TPM4HKxPBd+0L1GcXtaLvoefQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADldUVZaYAZrgmz1/r/wCX6lBsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD4Pv4pzpXG5IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAf//EADEQAAICAQQBAwIDBwUAAAAAAAUGBAcDAAECFggQIFA2YBMXNxEUFTAxcIAnNDU4kP/aAAgBAQABBQL/ACUKEowcb+fNe6/PmvdK1nKriR/kq7iEb8fpPm4BsFXaRDeN00uIROwevFxCc2r5jNhwyMXVFfVBCBJLXOIsLODN5BIeOSAYgzQPl2SpDyEG+USbPZLlS1qcpOq+6w2JnCKkCJf6FJkxZUabH8fv9kYf10Ka05fSHj59CasWYkQobO2hlGOxXOlrk9Tdl50ix/8AsQ0uAFOhwy0aWLnX6hw5Sq5L7nD+W8ddXjnkFi8FfCDRafH4pN0jAUA7ft+joedGrRfGBUmsouAXbTaYXy1xmXmnDwnx6KZcwhRsLOnBaRChcwbTl9IePn0JryH/AOG8jN9+KyqrQ9ZAoUKMJu9oaYaddVbfgWG72mOJlUKsLHr4SuJqWrjDny3jrq8Y8kUVgs6+RFpsjZ3udS/X29/08TfpBC/Wg5hDqt1F8FYhBVVlo7AFoEdBybRuXOmbH/rpy+kKQbFkKm/mGi68gc2GSv8Akd9L6VP1+5wYc/yGtgLOUWA7YeGAiwsdd2UGQ8MEJcnyy6nrinqdBhkomagkDLJBgA61AiKgCCfPL4hmHQ4kcfDgKgAYaYlgG1QYtBoEaTGjR4UdfVACtwYlYC2RIUOOOhzIkchD/JWs9fkrWeiySsHBzGqAG2LqIqAIJ/iqAOLFOgxCcMOnLgEVLoJBk51dLXE6P/dTrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusa6xrrGusahgf3SR/g2Ra1kRIgEYBSP8AZbfYyiiS9t9t9ngxJAKUmTImSKRYSAl2+y3RLXn9fCsAjx/FlxcJgEHqaeg8+oKkLBinsyZMeHHjfErLn+xnJNAPq+Ha1+jMnu8ijxGHE1QbAQMKn2HbNgZUQLxs5/4zKteuT0vP6Ev2QtQrQVKtLknxvyRhJBxYMo5uYhBj0tmusj2LxVLYeWVWyPwRF71JuimGkwCMArG+e8hl0iUD6oBbnh1tnbp+xNm8fALYJenzI4uUCfMFzLFIbmKj99jHJa2k5s2WRlodgnj3X57fbbltxSk/hJeGTIrgXewzKfIULyfFsndVVR8JVKqFna+eXHHbm71ti3Tv8bVLhcV8jHkYZcfTAEhsYUzTL4KnU9U5Ncn/AGA1fsz2RdqiTWXwSJInSO6cPkV7TOayN0pdXhiuJ9bXWyIB0Eip5wiIgfwoV8+SstCETYc2IRjerkrc2WHMYzvOKKmzoXLorEz7r+DGoeVHsIDBxbAMXwQXf0te3D/I4pXC3rxHBnwysHzV6MpRYr3XjAyEsbD6tbWIThHHyUj/AI62xi2oRos/nId/ZcuLBiM+RguHPr+0gz77rWXCQB1DhyB4kMhbDRvzT0nwXpZJUTZw6bSNQy0Lj6+SnHP+768cOEvYFqD0A3fNo8ZXOvtUxwk87G9s8aOK4BoIIG9TTEDXMANvWWTn8qdsFKWpYU6HYoPuYlwQ1C+HjgB2lgwYxcGaqxBLAnbfbbfY549rRGcj1uvomL+TZpkgZeBhKaHIDJe88b8m3k5ARUly5U+TRLGRBWJ7mNiGKwv/AFIY8HCRVkqVYijJUkWl1ayUCtV19EnZnvNsYJbjgnNWZcvpaFMnJJxRo1rJlMWPHhx/Jkh8YsOZ/Hx+Dk6Vo8uAM5MmPDj2tytd5fHlx58fVk48JVneUZ8kPDaULbdcWI15GD4hAMzKN1REQ6Ul+ji4iEkRw8lOP4y6wDGgR6WeanmngeQmCpombuSFfNeULESGg9eMTFPJrXoaPhl2IZJLFkYHYfGt8Tjoi0ectHAgJAPVN45OSxlbPjO2JryT4Z/xteOWOTxWvS0aZNTDSnRjYTJ4sWPBi+atqtsdjr+1D2lvLqmu8VcrfpbB4gceYsqRCkcAoSw1RlwNVagaZDuNdJ52pmAwQUKRMD8YsUOCQNMa2IaxeLxxXuMoSJHgh32Ja1NlCpcPU5jkSFGFfaNfaa12IFixsEKN6GrCS16YINCT0P3sx6KrgDNoPRqdS9pFzJP5xlYYCsGbJZCKBg22hwZC6Cqd9AU3XxZmZu3tCdzjyMErBYBmYvpuTJkzZKPNzxr3739fztKfKiyYMihFcjOavnLT34x4/lbik8omgdkNa8CLWc9lpNP2eWJlV/BuiupgVDOizlGvQ6fUdRT1sh6XJYcxLG4rDesMmrHfm8LfrPBhCvPHjx4cfy1tXrjRSC15RF+ROPnwysGps+CNwZrervFlc7sqiIvORa0BY9rV1YBCT6yL8ajnj5ouZSKwQItg+bjebC9vkGrEycLbbflvRquQXVT5u5xs4bZWkMfLEpTmy7qwScsAV0Tm8nGjBlMlpvkeIdVmrYsWMIBwSQHA7FRc5SsU1viq/edjhw4g+N7sQAFgl+lzWJNToGGwnrBIq525vC18u4ICs9RlekK/UyOrU4ZY4vyAi5z1XaUY+XNZ9rHJ5t6jyM8OQqyOYpr9ti3GPS5at5CxiBL2+QaoTJR+PHlz5UgrEFtS+ayY8ebHuEY6+3h1xRLqQeDeWpDtq00SNF4VRmIGVSi5mhh1bdrnch9btd0AEh83ASgasTBKjvW22/Lddwy46/6GrLRl6YINCT8PWIEDwSvnTqktM3CzUWr1wt+WkrnoXWKcMlelvKRFdbwwUmwEQo7iHD6d6vW3nksUapLpH0skzMX0fly5c+VFmZo97+wLqQSzvy9mePgl4YQ0cN29xgTDOijVEPcCbUtRy1SZ88aOh12Dxdm1g15LjLE3SN120cOtz1mANLjUDa4vtse5ISZMVfIXHOJfYRIhGEjjRPkGVptmWFPlbeQ7yMriyrfOkpy9aToBJOn7rDGcd9uW3ssSPKjPe22/Lddwy4y/9gtwjIfV2SDmtSoJEeREzqCQIiojwtch2YMFKMBAxjxnIf8AT2utZLTzpYpBRWyP2GwoeAmRkcHqHktY4ZcK6STbngQ46U4GNgS+GWR+j7QAVowCw0xok/ZNtQ+c+rqD2kcae9bSNTTb3HkZ4mdcIZSy99kHeRLiEpoo9GUD1tKl554ssUC0TZ+HDij4f/T6dLzRtYWnDKlfZxkbKJ45AabJ2/tL/8QAQREAAQIDBQQHBQUGBwEAAAAAAQIDBAURAAYSITETQVFhEBQicYGhsQcyQJHwFVDB0eEgIzOCkvE2QlJgcHKygP/aAAgBAwEBPwH/AGlER8FCKAecCa8TZuay15YQh5JJ5/sqdbS4EE5nTnToLzQdDZPaO6zrrbDZW4aAdDjrbVMZpU0Hf8apptfvCtpa+3A3bREBAKs6d5WQLGImcC811lSVJWcOQphJ03mosqYTZcK9EoKQG1Kyp7wSeNcrJi5k0+wp0pwu5UA93KozrnzysubPvvObJ5tsIJACtVEeIoOFnpswtcHHOZCjnzoBQeOlpa/ERUPt3KUVmANw5nebP/4hZ/6K9RacvGHljjgANOOY1sqImMRMXYZkhISEmpFdd1Ki0THqcgmXYmgKHqE7uziztCzGIioV2MSOwAcKf8xpx7+FpdHR0UpCw6hYV7wGRT55+I+Mak7okYglq7Q3jSuLELdWmcc811oJSlBxZGuIjTcKCwlj4lr8PUVWVkfzaVys/AuuGGoR+7NT/SRlbqEdCOuBhCFpWSQVapJ13GosJa9toZaiDs8Vd1SRuAFLS+CdgFuNgjZE1TxFdR3cLR0JHqmCImGw9kEdqu/uBtEw02joFxh7AK6UKuO+osxCONTB586LCafy1/O32Q+W0pJGT208Knlr9Vt9nxTES6qGUEpcFe5fEDTPfYS+PiYlpbzaElBqVJ1Vy0GR31/4kn0/+xylCUYlKtJpqmbwm1AoQaEfc82kkJNwNrUEbxaXy+HlkMGWdPXomMaiXQS4hQrh/taSXn+1IrYOIwndT7inLsSxK3Vw/vAf3+QtdSOmT0ywKUVIoa1Nac/nbbqccKGhpqdwP4nj62jbwrhZgWXUqIB/1FJ7wBTwraFcO0U3ixCgIPI1/K0VDMxkOpl0VSbSu70DKnS6ipVz3dE+vD9kLS2hGJRz8LSeaJm0HtgKGtCOf3BEgMMHZChJA/qIFfO0gu9dRN1WkpaQWyipUaV07RKtQQflacySUvR+2USrclOXaFct1e/PLkLQzKm0lS/eOv5Du/XojZlOUzdXbUFBWQ9BT6rZOIpGLW01kkHN6F2oI3i0BAQ8uhgyzp8bAQbkwjW4VvVagkeJpa8Ps3RKZQuLYeKijMgilRvp0RGx2Ctr7u+0DM0RyVMQ0YojeNPUZ99LNQzTJqNeJzPztGzGClyQqIXhr9brQ8QxFtB1lVUm2zQV4qZ9EZMZymcK7SgsKyHoKcPWycWEV1+MZYeiF4GklR5Ctm3IiCiQtNUrQa8wRacX9nc6l/U3cKUn3sIzV5ny6JtBKmEvch0mhP51td27sfAx/WIjKld+v6dF55JFzNaHYfMjKlrvyx6VwGzdPaJr3dOBBXipn0P3y2UcUBvsA0599gQoVHxXsxm0ogC+3ErCFqpQqyqBur9V8Le0OYyyZT3HBkKokBShoT+NBQV/Kz8LFQ1NsgprxBHr0RTq2mqp1JA+Zpa6nsvhJ9KRHRjyu1Wm/Q0qRpqNABa8sHA3UjXIOIdwYVaiufA0z3HMbrQccHgntBSVe6ob+RG4/WVnHG2WytZoBaCm0vmKimHXUjvHr0P3yLUcUBvsA0599kqCk1FnbrSt2M6wQda03fGXXi4GBn8O/GD92k58ssj4Gh8Le0OeyGLkOwacS44ogpwmtOJy0yy8eiPvaBFLY2VWwac8t44crXev7eK7kB1doB1o5pJGYryqPxz3C18kRt4YzroxKcNa9lW87sqWu9LIqEhkodFO1i8aUp3cefK02glTGXLh0mhP4Gtru3djoGO6xEZUrlXX9Oh260rejOsEHjTd9wLWltJUrQWgfZvM7zuqj4GEJRXOqgkE9xz8K15WWiYQrim3mxVOVAc6jdQgetmYht6oGRG46/XPS14J+qUKS22mqjnnpS0lmgm8HtqUIND+xfKJj2nG0oJDZG7ja7D0Y/KgqIzzNCd4+q/FyuVR05ixDQiMSj6cSdwtOrvTW77qURiKYtDqD/axIAqbPxEJEtlpDqcR0zGtrje0KRS+SCCijRaK6Z1qa/jThpnb2ozuMmc4MU1VDbh3cshXnQV762kT0U9L2lPe/i7PEpyxeFK+XK00ksHNgNtUEbxraBgYeXQ4ZZGVp25FNSp1UP71Pxz8rXRipg5MCnESimdfLx6FJSsUUK/GXNvE1dual91OJCk4TTUZg1Hytfm+EJeMNMwqSEIzqdSf087SqBhpteBqFij2CpA8FHM8K7hwtfS6N3pfdtyIhmcCm8NCCd6gM66699pnPocx5SlkUSddFd/D5i0KwzFsdonn2lU4jU7wQeVmYZljNAz+Z+Zz6HJ5Kmorq63Ri8fXToSlKPdFp45FNSp1UP73Lvz8rXRiZg5HlJUSimdfLx+OeYS7RVaKGhH15WiJzeGLaSzFulxCdO0f/Jr62fu7DxcZt9lQnM1Vl8gK+YtDs7BFK1JzNphGzlM5V2lBeLIelBw9bIxYBi1tFXRj3JgooIwE1r+nGyE4EBPC14LwLlC0ttJqo556UtJZoJtBbalDWh77JSlOg+OUpKE1UcrIi0Oq7CSRxpl508rdcw/xG1Dwr/5JshaHE4kGothTirTPocnsqZiurrc7Wm/106JpJYKbAbatRvGtoKCh5fDhlkUA+PlzcDEXhhmo7+DUV4e8K17h613WvzL5A1dhayhCSKbOgANeApu48s91pjFzhM5V2lBeLIelBw9bQdS8sjTKvDFnip5dMVdCPcmCihQwE1rv+XGyE4EhPD7hdZbeThWLFmJoAHlUHGh/CxgtoaurKvkPQA+dkpShOFIoLRN8nWY4oQ2C2DTmfrushQWgKG/7nfuzKoiL6wpJqcyK5H/4E//EAD8RAAECBAMDCQYFAQkBAAAAAAECAwAEBRESITEGE0EQFCIyUWFxofBAgZGxwdEVI1Dh8SA2QmBicHKAgqKS/9oACAECAQE/Af8ACUvIzk2kllsq8BDlKqTKCtbSgB3f0pacUgrAyGvv5A06Wi4B0Rxhppx5wIbFyeRDTjt8AvYXPh7alxxHVNoqTC57aNbBWQnK/gEAmBL02eZd5slSVIGLM3xAa8MjCZClJmmZZYUS4E536pUPDOFSlOdYfS0FYms7k9bOxytl3ZwilMMMt71lxwrAJKdEg+43PbDNJfSickm8zdv4Xvc+7WKixLyr+4bvdORJ4nuHAQx/Z97/AHp+RijsiYqTbZJF+zI6QmXp8vT2pl4FRUVC17ad9jEvIJbnHmpa5C2bjt6VsomadLy0y3KKPTJGJX90X7PDtioyUjKpWgtrQR1Scwry+R9sdq7X42Z1CeieB1thwmOcU2SZd5sVKUsYcxbCDrxNzBqTBqLExY2QEX/66wzOtNiYuD+YLD/6Bzjn0lNNtl9a0KQADh0UBpxFjBqLW5mUpBG8w242APEk3ifnGp5CHCPzQLK7DbQ+PbElNyCZBctM4ukQejbh4kRLTNKkZ1t9nGba3CezhnD8027T2WBqjFf32+0fizAcUoXza3fvsO/T1aOfyr0s0mYSVKbNvFHYTrlwg1CQl5Z1DLi1BYsEnRPfqcxwt/pJQqD+LhS1LwpEVilqpM3uibg5g/o9KrU3SSd3Yg8DFQn5ipTBee1+XJT5JdQnES6TbF/MVrZr8Mld+2vEOP6FR2pZ6ptImOqT/HnG1UjTmadjSkJVcWsLX7vhG+K1lDfx4D7xJbFInaeHzMEqI4aeGVvXbCA40+thZvht9cvKJWZek30vNGyhFT2gnao0Gl2Ce7jyULZ/8XQpxa8KRl74rFMVSpzck3GoPd+gTjzxa62eQHdc2h16bZmsKSRY5D16MS2087TJLcoAur437s7ermGULTdS+sdeSSp1HVSE9FJSU5nj3m/q0KwhRtpFKrc3SbhqxB4GJ6emKjMF57X211wNNlZ4RKVgvvhtSbX5NwZr8oC94nKDVJABcxcDt6J84bYbaNxr28Yk6dO1BRTLoxW9cYmJd+VdLbqbKEY1hGG+XJJ06jqpCeinAU5n5m/b8oVbFl7YpSUC5MEIcRY5gxL0uWl3d4OSlTiafUEPqFwP4jaHaGRnZHcMZ3tw0/fk2ZrcpTULafyBzvFfqTVUnt40OiBbx5ca8GG+XIxsdvZILLnTIv3eEEFJsfaq0xMO4SgXAiksvMytnIStC+qb8kw4ptvo65D4m0Ts0iXf3ZGMjW5+mkUulv1dm8mg9uRtb45Q61NyEzzeaFj69ZZGG23HnAhAuTE5SqhT0hUwiwPh9ORjY4OyQWXOmRfuhSSlVjDW1FTak+bi3Zfj7ZOoddlVJb1iky003NYiLDkZ2Mbm6cFOOdJQv3d0TlNJe/OBJGVxa/vH1HwjZbaKm0dlTDqVAcDgUfpFfqQrlSS62nChGl9Trw4DPxikzqafUETChcD7WjaHaGSnpLcS+d+7Tka2nqjMnzcEdl+P6ApSUJxHSBtxUpOVEsMIyyKr4reAv5wlyYUMRAPgfv8AeGnkO5DXs4xQKCmrJU44qyRllrFZpZpM5ur3BFx/RsfLyLrbilgFy/HsjaZmUYqhTL5ZC4HA+re1vPtS7eNw5RLTbE0LtmCQBnDz0u+gtpWLnv4xOSa5yZxtEG+ovp+0bCSMkiXUlyylp7ezujbFqRYqyRLWCsrgf+v/ADme+KZWZykk7qxB4GJ6emKjMF545xRW5V2qNJmOrf8AjzjayVp7cgFYQF3yt5+7kSpSDdJ9sqEoqcYwpOYimU9cndSzmYmM1OFQvhFwPrFNn5pybCFG4MN7ASU7IpdUrC6RcW0Hj2wywpClJKlBSTY2UfvCGW2zcDPt1Px5G6JVHZXnCW+j7vlryKUpfWMURuVdqjaZjq9/hl5xtbLU9uRCsIC75W8/d7c6yHM9COMNMrYXcIT4jI/D94Y2przMnzdtIHAE6j4Q03uxmbk5kxISdHVR09FJRhzPzue35QvDjOHSJXa2Qbp6QsHGBa3rhC1Y1FUUCgIqyFOOqskZZa3is0w0mc3N7i1x4QpSlan25SgkXMJmUuHoAnvtl5/SOc4eugjz+RMJWlacSTcRiVa3I3Q6o7K84S30deHy15KZWZ2lE7m1jwOkTs7MVCYLzxufb5zEpdv8pI8f24RTHZozoFye2KdKUhVHT0UlGHM/O57flCN3zt3ddT6538rX7+WV2tkW5BIWk4wLW9cIWrGoq/QXG0OpsoQJdxPVcPl9o3ClCy1kjs0HlaEpSkWES2x7TsiFrcOMi/cPXjC0lCik8P0dnaWqMSnN0qFhkDbMf8BP/8QAVxAAAQMCAgQICAkFDAoDAQAAAQIDBAURABIGEyExFCI2QVFhlOEHEBUjMnGBoSAzQlBSYGKRwRY0dLHwMFNyc3WCkqKys8LRFyRDRHCAg6TS8ZDD0/L/2gAIAQEABj8C/wCZR+qzSQ1HaLjhSL7Bj87k9mOPzuT2Y4VSqI+8p1LRcIWyU7AQPx/cn3KK4tQjOZHc7eXb43qjKJ1bDSnHLDmAucGq0Va1MhwoutGXaP8A34mZFbccSl9zI3q28234CtDg4vhqW85Tq+La19/zypiQ0laFiy0LFwRjk3A7Gj/LFZ8o0uO/kebya5kKy+nuvhysGFChIbR5x9LKUWT6xjUIbnuJv8ciOMvvVf3YFToc5L7RNiRvSegjmxOps6eWXKcgKklaNnNu6TtwmCvhccKNg/IZAR7iSMKprrr8p1v40Q2woI9ZJAwqXQ5JVqzZ1pxNlo9Yx5Rrs5LLd7J5ys9AHPjg7vDWE3+OdjjL/VJPuwiXDfS404nM24g3Chisfp/4YZ0ceecdmvejHjNZ1D19Hiqv8mv/AN2cL/T3P1J8UVem8PXNF+0cZCbLt1YZk1lxaUvvapvIjNxsLprjkiU60bPcDbCgg9BJIwqVQpebV/GtLTZaPWMP/oH/ANacJm12ZqwtWVtCRdS/UMJrCgthlTes/wBZTkKU9JvuwY7XDJABtrWGBl/rEHCplCmZ8hs62pNlI9Y+d65/Htf48ULQlp5SG50oF63WoJH3XVgUWHTGUxQjLqdWLK9fTibopT7phTWc7bXMOLnH3cYYqaakwl1EVvXhtYuCoBtI/tY4athOtjyUapdtovsIxAbixUBUmIh2QvLtcUpNzfp340ipsJAbYSklLadw44/zwWNNJiUUylt2bbcSSlarA2sOs/cnCqLUKvGLCk2SBGXxOtPF2HFRohkaxqFJBYV9ld//ABv7cVaBSIqnajKlqU1xLhpABus4VphwozKnKcUJb73pNm/o/jfn8VV/k1/+7OF/p7n6k+Kmfp/+HEBSTYif/gVhmkRI6RZsa9VtriucnpxXabT2w2wIalBpO4XU0fxOJlcmtqWEQAENo3rUW02GJlc01dK58KxjU1xFkNC/Qfonm6TfFRhUhClvqbSQhG9QCgSPuvhjR+o5YEltOWTrmNjqunMPxxK0u0Xq2tampKSywtJZTcg7Lftt+d65/Htf48UPTdlkrbgygHrfwgpP6jgVqLWI5jFGYul0AJ9fRibpdTwTChM5G3bb+LkH38Y4r36Af1s4f/SGv7WKV/JrH92MaSfxZ/tJw5O0rgsuU6qsebcktBSEKsnbt+0n2BWFVidSqOGUozJKYzRz9Sdm3Dtbi6JRqUh17IgR0jzwTz7EjnJH34rE9yIhT3CtVrFJ25OjBiOEih1c8Qnc1/8AyT/ROLjFV/k1/wDuzhUOr16LGd4atWreeCTayduOV1P7UnFJkR3AtDk3MhaTsIKd+IP6f/gV4q7+gf8A44WmZHS4G4iXEBY3KDYscRvCno43xm3Ame2Nyua56iOKfZhGnVGgcNZVkJb1uXKCbG+w7jsw1WZ9NgOOONAvgkaxpXOkqFjh+i6FzVPUwsK4QA5mSOL089lWF+v53f8AIFO1HCVAvedWrNa/0ienC4FQjIeZdTZxtwXBGNelExtN/iUSOL7xf34TTKJBQwynblTznpJ5zh/SiLAyzpKMj7+tVxhs5r2+SMGlVuJrmFKBKM6k7R/BOGoERvK0w2G2k3vZIFhiTpDBgZJksWkPa1Rzewmw3Y8nV2Al9u9032FJ6QebHCHG5bwB+KdkcX3AHCIkRhLbTacqG0JsEjDzdBgagSHM7vnVKuf5xOEwdIKeJDaF50jOU2PrSQcNwIiSGmUBDYUsqsB1nbh2BLbzNPtlt1N7XSRY45Nf949/545Nf949/wCeI1JqlM1seHbgzeuWMlhYbjt2YRD0ggcIbbXnQnWqTY7vkkeJ/SiLAyzpLeR9/Wq4w2c17fJGDpWIH+vqbyF/Wq3WtuvbDtPnsBxl5BQ62rnBw7Q6ZTcsR4nWsOOKcSbix9InBebTMYBPxbUji/1gThTFBp4bK/jHVHMtfrP/ABV5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192OUFU7X3Y5QVTtfdjlBVO192EyPLM9zL8h6TdJ/5HOCVXSCHHd/e3pKUn7sCZTZrUho7nGXAoe76mU+JpTVBFNTfLUZaknLcc6juSN23rxcYn1iF8cxHJaJ5lbgcLly3lOOurKnHFm5UTz4YpjDp1E66H2r7NxIPr+pj2jWksIPR3hv8AlNq5lpPMRijaAad6VTJvDZDiIVSej5WYzQ9BCl/d02vzJAw/SZvGYlNFCig8x5xhUaNSVzWr+bkRtoUPVzYRpTpO0GVNoPBYt7qBPyldGzm+Cp55wJSkXUpRsAMcGb0qgFZNgnhSd/1Hf0Z0lhB6M+P5yFcyknmIxQfBLpNV6hJEllXB61NbAZSc3FavzdH2eLzHZ8KDQor5QxKzrkBJ9PLaw9XidgT3M/AHtWysnbktsT7PqI35PbBmTFKTHK9yAN6uveNnXjh35Vy8+fNYucT+jut1YMuWhCJcdzVyUo3HoV7fwOHtGdIo+ZtwXadHpsr5lp68RvBDpDIqhch0xsxKlMaz8NP0U5d/QPURzbeGxKFCo0RR4kvSKTlUr/pJ239uCmkeFmivPAXMZinAj+1fDFA09psdoy15IVShKOodX9AhW1Cv1+NpynOpRNiElkL9FwHenq5sCL+TLySVWzLUkJHtvgU1bqXJLq9ZKdRuKugdQHwDCqmkURl0b21vC4wJtNmNvtK3ONLzD5/hVmCypxEBTmvSgXslWXjeoZff4n6rPbU35QcSpltSbcQDYr23wNENDo6JFVWnM6tz4qEj6bn4JwjytXZhrSZKHxXQrzqVA7kjclPQObZ0YqMB2VIIpMlUVtqSLGyOLntzFRFzsw3UKfIU080rM24g2Iw3WS3lkyEQ34wG9LqloIt9/wC4VCsQPjm2gGz9EqUE39l74VIkOqWtaipa1G5UenDVEbdUY85Kw43m2BQSVBXr4tvb8/5VC4xwxOi8AO5r6zgqb3+7CpUNnWzH3Ex6ex++PK2JH4+zEjQzRaqqQ+ly9aqqPjZUn5dlfJSk7NnRhMidWXqjGKhr40xzNcfZUdqTj/Tho5U2GabJYS9UUE2vntZQ6cxI674jzHWBGp7yEuiWtQOdB2jKBvuMRNGqWgeSNG1oclKT6K5CRZtofwd5+74EjRvRqaqKxFXq3nmvTcWN+35IB2bMJdn1R6dGUoa9iUvNs+yTtBwiVGcC23EBTaxzg8/ik0OePNSW8pPR0H2HbhUWPSFS27+bfjkWUPwx+VGkiENvBspjRt6kX+UT6v1/UHRiJI+KAlupB+U4lAt91ycTZclpRj1GQuRGfy8VWY3KfWCbfd04apNJiLekPryttoGE6CVlQcj+Skw319Qby5urpx/oyhFCeATHWVV9DoWhuNe4CLekvabdAt7GqNSWsrTfOo3UtXOonnJ+BNelsq1U2St+O9l2LCjcj2XthqlUuMp195VkISP22YjUsOZ+DR0NZ7b8otf6gLp1R0rhtvNmzjZd2pPQcImwJTbzLguh1peZKvb8BlyBN4LPhPa6BKtfIvoPSk7jg0rTnwVSJo+UqnpRJZc67KII9uFfkJ4F+Aqc2KelhqKPaE3Jxn8IWkWsYvfyTTLtsH+Er0l4m0KEgNQ6po2hxmMjYhBQQNg5vQX9/wAHgtUgMyG/3t5sKHvwo0ejRYpV6RjsJTf7vHI0c0cmrhx4rhaddaNluLB27d6QDs2YS5UKq9OiqWNezKXnNvsk7QcIkx1hSHEhSFDnB+e5EqjvFp591LGuTvQFbyOuwt7fFK0WU8pUV2MX0tlWxC0kC49d/cPgLq9XeskbG2x6TivojGVWh69Xm9MTttvVk/HDVapD2ZpwbiNqDzpPX4qX4OYzUfgMuiKlSFlrzmYF21lX3cUYU++4lCEJutajYAdOFxqPo8uYyhVg+uTq8/WBlOzC4zTCosttOZUZxd7jpSef4U52YwrVTJK347ttiwpV/dfDVJpccuvPLyoSB7/ViPTg5m1DCW81t9hb57kaOzllGtF2nQNraxuV+3NhURGjxkhJ2PRnUlCvvth6uaQKbM+S2EJabN9QjeRfnJ2fd8CjqTm1ed/P0X4lvx8VRcXm1BlJ1e3Zmy8b/D4pj7Cpx0ho9KS09e3Bw0qyhb7XnP14qohhWfgu3J9G/G91/FT+Dg7M5ct9HIfhcFqcBmQ3+9vthQ9+FeSKPFi5vS4OwlF/u8YkVyqsxkq9DWr9L1Dnwpqh1piQtAupCFbQOm3ztwCuaSxY79trKl3UPWBuwKlQ6k1KYJtrGV329HUfhrpFaihxpe0dKD0g8xxnXpFLLGb4vVpzW/hd2G6RR4waYaHFT+J8WlWnNalxJHlyalUByI9nswnNYHZvsU/0cWIwqXS6k9CSs3UwEBaR6ujClU5KnZLibOynvSI6B0Dq/cqiqdIUsR5bjDKSdiEJUQAMNVSnPlt5heZtY5jiPPKMuuYSvL0XF/nSpViJbWxYDrrd/pJQSMLmzpC3nnVZnHXFXUo9JOIcWI4dTPXqJLV9igdx9YP7bfhrq1VdIQk5UoQLqcUdyUjnJwqozqoxo1By5tUGw7IydK1K4qPwxwOb4ZqpJcK7XcrKkov60gJxVK/o5pZpKp+PDUY0dqep0KWdg4tjsucUwUmPAmtSWOFvU98Fh5KnONYL3E2tvwaPJYep9SQLrp85GVfrTzLHq/cBKrlUZjIV6OsVtV6hz4UxQ62zIcQLqbSbKt02Pje0g0WZ4S3LcLj0fMApCzv37xhs6RQuBQ0Lu9nWMyx0JA/XhLLKAlKRZKQNw+dH6XNRmZksqadHSkixwtijU/yjGJ8y+0tINvtAnYcI0t0wZbbWyn/U4mbMoKPylc2zm7sF11YSlIupSjsAxwL8soWfPlvrOLf+Fu9uAtCgQRsI+Bo9EnfEojyXo6TuU+An3gbcUyhxJCkMznHVSQk2z5Mlknq43uHi0e8HNCrPBnF1nM7LWEq1cJKbrSc4tlAzK/m4cjUbR9UplCrB9yRq8/WBlOzCqVPhLhVGONZHWlzzjX2219R5sS9GdI1BVTpTgQ+6kWD7ZF0O+0e/xGqVVdydjDCfSdV0DFnNELN32kTdtv6OGq3SHszLw596TzpPX46iZz5UI8txhlN9iEIURs+6+GqjAfU28yvM2tJ3HEaoqRlL8dDhT0XF/nunUGG8UNVBx0ySk2KgjLxfVx/d4ptDmLK0095GoWpV7JWDxfUMvv8AHw6t1JqM1ewU4reegDnwhnRLShgVaA7winubQQsc1jvSRsOEaN1ECk6S04lTcOV6Ll9+Q/KQbDaOjHBDo0U8a2tU+jJ69+NK4sPRhLtU0cCmFV5p8nXFWx1CR9EBKh1g32X8VP4MFcUrLlvo5Dis6RU45obEZqCHk+i64klSrdNr28VJcsrV5XR1X4vinOLCtUqb5u+6+UX/AA8b2kWirQkJlLK342YJUlZ3kX33wj8oYfAoiVXeK1gqUOgAYSwy2EoQmyUpGwD57TEZfSzNiqK4bq07L22oPQDs+7HBfya+VbW8IRl/Xjya48h2W+vWTHkJ+Vb0R1D9t/jnJmOcSJIVHYbvsSlJt79+ES4j6m3W1Zm3EKsUnFPm6SUttxb0Rt5KxsU0pSQeKobRiRXY/hEbVTYqOOK7HLhaubDjt8Y7SOY4ESk6HorLVQfM3yk1VkIL+sAymywPk2wZVI8Hb1OCzdTS6ywW0+q1yPVhbdYrjcVp5OWQ3Tb615P0C6dw6QkbcN0ulREMR2U2bbQNg8SqRWousaUbjpQr6QPMcBx6vy1s5vi8qQSOi/dhqk0qMGY7KbNtjm+ormkuibSXDIN5MS4Sc30k9N+fr9exLWlcuLSY6T55yTLbzW6AL78N0+kVmEptlAQ2hmSk2A3DYcU3Q/R5COBv1RpVad1wSptgHoO/p/mjDcOK0ENNICG0D5KRuHj8n1fSBlp4DjNi6in15RswKhRqg1IZPy2lX9h6D+4Sq9M2ojNZsv0lbkp9psMKmr0klR7+izDfU0hI6LA/r24/JPSOTwhS2yYkhfp8UbUnp2bb9XP8+vVuo5sjQ2IQOMtR2BI6ycK0r8KVekQo6/zWgUt7IVHmQtY2qPTzDFk+B2AprNvef1q7fz0nbhqvUrROAWXBtSmGlC0K50nLz4r3hGps2oaOtCWYdFjlJJS2jfnQ7vG7Z05sJa8IVPbdhk2Fap6TkT/Go3o9Y2YRJivJcbcTmQtBuFDpxUKvT1APtMeaUR6JJtf34LrqypSjdSlHaTiPT2HlamaFNvtX2GySQfWLe8/uE6hxV2deaBa61JIUB7SLYXEmR1tOtqsttxNik4TpMWVJiwkL86RsWtSSnKOneT/7+faNU5X5pEr8dyWTuSnaAo9QURiiPoCtSlx8LI3ZiG8v6leJ3QnQqY6xNqExEhcpH+7Mti6jbrsPYm3PjhDmk8xrZYIjPFof1LYGh+lMgzGpSClh1/jKBt6KvpAjpx+RzCj5LqbK5FMbJ/N3E/GNj7O3Nh+j1BGZmQ0ULt+2/C2KbBTOYzHVvNOpFx1hR2HH5S6TBCX0otGjJVfJfeo9f+fjZg0fZMmZsrx/2SRbb68CUnS6oFQVeypain+idmOGS0BMqOvVSbDYo29L2/ADtUo8WSpIslT8dKyPvwGWWwlKRZKUiwA+d/yeoEJqVPSLyFP3yM3FwNnpG3XhDWllFicFWoBbsJKwpvrsVG+ESY7gW24kKQtO4g+IyqjNaYaG9x5wJT95wWG9IQ+sc0WO47/ZScPo0ikyXI71mlx/J7oUsKNtmYDdv382FRolZpWl+i5/N2HEp10dv5OcpCVoWB8o92KemkU+t1ap1SAl+PBpwQ8y2s7NWt1O02PQnbbmw3Q9JINParaqctoPMRwC2lSs4bUrn2gZiOfp3lyn1GMpl5pWVbaxtBw1pAWy3DgXW6+rYCbEAft0YYrVK49NoTLraJY9F+Q5YKCekBI3/CiaQwGS6iGFJkJTvSDbjYypFzhciptltyc9rUtK3pRawv8APlUE1go18jXMk/KQrcR+3N4qVTZ6MrzMBtLqeg5cGexF18l11LEKP++vKNkj9ujDunXhWl+VZLCMzmtTmZav/s2m92+w2+7BboejNKjRR8UyttaiketKkj3YpFHgtxYrtNrbMqrRX3jZxkXF29nG3nZjylpPCiQjfiSWfNOk9WTao/fhc7QPSuTPaG15uC6Y89KelNwEv9ey+G6tov4T2J0Vz4sTqWnMOpRSQb4T5Xqujy8vouKo+sUn1ZzhDOl2lUyost7oLSUxo1ujVt/54RCgxkMstps222mwSPh8PYosRD+YnXJjJCr9N7eNmmUU5ZkwE6796QOcdZwJKNLqiVA3suWpSfuJtgTpaAmUwvVSbblG3pe354TH0ipocUgeafQcrjfqP4bsCqw6e4++ggsrlu59WekDdfxU/SJLRW3SKszLkoSLnVC6VH2Zr+zCptHGvabebkqU2b3aseN6toPi0UWhslKa8xmIG7zicT+GOcWLIVHYRfYlKTb37/bhEqK8ptxtQU24g2KT04pU5hIQxpXTNfIjp9FMpCErKx0XB+F5HgQxMmgXdSV2S16+k4TB0jpKYrbhsmQ0skJPWPhRdJIDRdRFQW5KEjakHaFerASkXJ3AYLlUaLbs17W6pW9CbWF+v57LTqApKhZSVDYRhadHIJq1DcJK6ST56Nffqr+kn7OFLp7jsOST5ym6/UrQrnGrVt+7ZjQzRHQeFGiwKxWODzm9VmKhdsXudt9u/C9JNFEtrW/tkxCQnjfSTzbefvwJmnUmPR4CDd116SgqUOhASTc4j6WCnLiUqmxDHojLqbLczWCnbcwsLDxP6O6OVFyJHhrLbjrCilbjg9LbzAHZswiY9W5M1rN52PLfUsKHt3YZqMVV2pDSXGz9ki48VXRLSoKNQdUM30Sq6fdbFgMQWKhm16IbSX8yrnPlF/f4zT6tpC0h4ek2hCnMvUcoNj1YFQo1QakMq+W0q9uo9B6vFw5ijRUPXvrkx0hV/X8/ZK7RI8noU43xh6lbxjRzyxP0hYceq6G6OIM3Ohh8lNlecvYbt3RjK/4SNJFJ6Ez0p94TgVBVNMuUP96qDqn1/wBbxy5zrS1RZz6n2X8uwlRuU+sG+G6VSYi3nnDYJSN3WegdeIlJSvMIsZDQUefKm3iTJnJWxJSLCSxbMR0HpwmpvuPTXGzdpMi2RJ6bDf46jVaerK8hkJbV9HMoJv6xfBUo3J3k4ZprDp1M5C0PN32GySoH17PefqDo8/SZsNk0quNynTMdKLoHMmwO34JjymEONq9JDibg4UmnU9lgK9LUtBN/u+G/R6g3mZkNFC/8/XhTNKiNzmbnI8h5CNnWFEWODpFpEpHCsmVhhBvqr7yT083z+qpVuotxmU/LcVv6h0nF9DNCl6g+jOq7moQesI9IjFPqdeqkFwMVxlaI9Ohq4isq+NdRucZ2PCPFe+w/RUpH9VWM9e0UjVNgek9RXTrB/wBNe/2HCpVGmZ9WcrzKxlcaV0KSdo+F5FpkIS5oALuZVkNc+3pOEwtJ6UiO04bJksKJCD9oHm+ob9UmKs1GZU44eoC5wrww6Xwky6g/l8i097a1ESv0Nn0rcYnqtswqW/ppUgpZuQ1LU2n+imwGKgluchdSgqaXHmPNBZU0VhCgrr4w24FJ0fqDkSOwka12OopW6uwvt5gDswiaa7JlIB85HlvqWlY9u7EXwwaOp1b7KGnJITs4VGXa6VdJsbg9WMydx+DV0S0KCjUHVDN9Equn3EYypFydwxBj1C+vbhtJezKuc+UX9/1CqFFYVZcmIttsn6RGzHkuk8Wp07Jr4KtikvNiymz0X229mFxZbC2nG1WW24mxSegjFa068JdHnKoYitoDUbiuP3eQcybkbAQnb14Y0ipkZ/yTVY7cmA66NoStIVkV9oYRS6PDW+84diUDd1noHXiF4H6E9whEdtlNalp9Fllu3Ev9NRG7Fh8FL9RbWzIQLCTHsFEdB6cJqi1PTXGzdoSrZUnpsBt+on5Q0Kpu0qqWsZkcXDo6HEHYsY1mkHg/pNcKdnC4biUOW/guD9RxVdFm/B9WY70qPlS46yjVIIIO1QVu2YpOjTHg1dlmNTGY7j0uW22yvIgJvtuSNmFIrdXiUiI58ZBoDOVTg+06dv3YTTKHARHZT8lPOeknnPiEuv1RuMhXo5tpV6gNpxwOiV5t17maUlSFH1BQF/Z9StIorTZUtVFk5EpG0nVqtigty2VtuIhZFIcTYiyiPgVFyYv83krjspG5KEEpH+frJwiVGdU242sKbWk2KSNxxAqj4AXJhtOrA6VJB+pMxVH/ADsRXOC7L+cynL78RKl4RorjNUcW5rG3Y2pUE5zlunm2fAc0k0WU1rHheREWcuZX0kndt57/AI4SdJdXDipWNYEuha1joTbYPWffhMdhAShCQlCRzD/5P0JjwlvLWbAA2A2c55sMRo7KPPNIc87ISlXGvsA+Vu+p7bTEtDaAq7jbjRUlzqNlDZgNOzGNWdXrQiLY8U34u3Z7b2/4Tf/EAC0QAQABAwMDBAICAgMBAQAAAAERACExQVFhEHHhIIGRoVBgscEw8HCA0ZDx/9oACAEBAAE/If8AspNhtSClg16rrhQKETC7yP8AFcWxqHGc9SeHqZHQGtilahVqIWz2dJ2p5JRm8Y9GYP8AmF9CfmVatIrZEcnR1hB+3zdYYwfFLpXp8WQG7FtWKRBMIlcwFV8ZQTcK64aY/FOxsHrRsXpgSd2nCojlO9OzqFl2kTssVCvqUVmO5DCSWdmn1qijsquqJgiHynCUBpoGphEzX1f8qi5fOTMWas30vii5j0cVbkYMbsXGlCusjyQt4xito9GTIFJrExjNI5bCerjuQ3JLV9DVc7OF2sF0NXB7lJXamwMw4YvfSrsXHxWU+KhByTPcSb3uSMN7fmCpWIdcv+R8bVO50SFG/N1XNLajk3cb4fDWfwXjIHMXdwqYdoLVcbM44NqjRB5ApmjAnQKMeGUXcA0CYUP1M0zkDm/uA015JcuLaG0Sn7aTmxnAq9yo22C4zbWCWMWVsXKF8rX5OYmyykdvTxV/7/KnoSIjijBtsZZf6ln+MUbRDYawmgMW00E3frES6X10N8UVlaQSiTiCW4lbE1gO3e3U2a41oKFkxeMsHPbGMFbWorcXtCWJsL8wU42jpj25h+N6j9AEEYm3mo4pFPlWPG+Hx6QOnHPcQwdALggkC7htTb6IS2wF10CoefFdxuQA5KanqACFb9FzvBtUiFCr02+o8LmhBJI4enFPMFLuKC4s1/v390K2koloHUToQYr7v+azm9GFkm5k5vUM6wpJZ5kSyzUFF3JvULl0RmjsqJRsQYdbTnWl4zWLz+BMriW8/lu+KOrGiyxvXuKAVYpQ4S+iuUOVTJXeF+RrLq8OznDwhjvWRdjxJGQfur8LZwoJbtgzU+SrBZHIuDAVA1Lk7MrrtRGRz/cjvRwtChMAGCrSBHu/A9qCrVao0kwB3pxGRIUBKq27V+Vs5UElyy466afMqMJjdC61trnDWUowvTIY4OznDwhjvUC7WNihxQ0rEU+A0JQQGmiyEQQwVLWpHhx/aVHkWZgYl04Lcf8AKsfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaR82kfNpHzaDfWdpIuRf/AKORY4FITuZpOcwsff8ATEG0qaEupKZanQUBJI4SofAMBmwGzCzGsVM9AiqVOqtMYOpcSJ7Ezspr+mSHiAsf+Cz3GRRNiy5gJZsAjNxQT4AkTBkuSZO1SJCiHISl8PsuamK2IFRatBMZS3iIfQ/D8cK6q4KERay5sz+j6VAFhfwmfZkUoGNtocu7CdTBk+sHVyAL03uVjW23RypCdSzT5BxBp+iRWcRJpb1EBuleIYYfUyUzH9ZHFaqEBkSJoCfegjsMZDGfoJ7Io2Wm+uh0Vsq4ImTMqHIjPbLZ8Glqwqi7tvmpFxlcovgMTZWOrdTRgIldVkOLQ5kPM4eYWKOfik7YlJliUGAF8stpj0DQeT3fNPesVv7X3Pz9yiHIv9p92nSRwKSnZdJ44B1oszpaTrGvuPwOshvTTOspgyleFQ8EMbTskhsL2gAKIrtGnuUBpkchI97jtP8AghCHvoz3/SpPDg0WVLlXWkiHlRMD/wDB+fPQIQiZrROi75NOcnQP2BdcKkJB5qhCZAkG68N5HV07+4hTi2403CYKyvTmFpkIHQerZAYjJaSYOcSQ6BPYjW714UegTZD0NWkZA3GG6MVOJaFutwltbcaxTRJNIO49JEKTF3keQByUVl3O30LK4fsvRWWSRtiKw3AE2cxj9BHDC8D94JVo119kzyozErQoziJGrvwGVwArUgJyAkDE5EQdEKHZmNOCo0Uo+LRGJVZgX1JXX+vRj8ZxiR3UhnDqUMhbt7rGASroC0EMgFC1taTH6Ak35BHZgmE1MlQVkoBuCz6EgnniAhHQ2P5iKxlrweyHi4qRGxNjf2iKQz3wuOa3wxcq0INwi0hcg1b6UrGyy32FZ8oi90DrwshHPcAsV4ZUYJrmGprcNcw6jWMK1xpH4/NglPpnuJIlkuLh0Rw+JhM6Qv34PR9aAOh3+gu0oCWkDzYp4+VJ2v0IuDoPjUUR6OisKSAMCVuNXenUkJhEqXAFL5hFcHUMtJh4KJijI38AiObCSeoADLzZg7qCZ11KL/ZExuowCVdAWmbEtwt5Rpj82AdCfLTFreyapJJmrQ0mLuKXyDxURMhGSRiq6LGKz6GBQgjVL/MEffofMGnh4DeGXtt0HzeXZ4+DOynCZEZQD7M3E9EKK5vFzPEx6nKmy/BYSjx/7HmJPV5npYKInUhJMYmvl/qZCGLl8X/LZUggTAlwyESYmlxRiACJXcLN7nrcowmC40xvsphSrqyMQ2zRPPwpfqAFVW6jdV16JNnALpxEQCEmhpEZEhEzT5+IieaI4l4ii1zjnMsa94ZglYP8VtUE0SNMS7qutCr6zg/k0TUYoUSbDNnD7/KY726T+QClMxVt5RdauLS1iPwBHuavWPbnunWLB/QtEB44uAl0FvYnVQmLSAcQhzMc1r8aKOJmAsRYWiV83ZYkwLSIjSry2LM3Ni3WLwf4EBpDjS06kJMFqlVwD2AFOS1+ozQATkskDMu5PvUCAFcXWW7iUBdukIsXxoIgA2/KRTAHUg+Gp0SuxKDexJzSMzIkNDFZBsluzaEoP/EBXVXBUF7xM0abe6Oan4OdIm/oPscbYgxqtG2aSG6OA2DMmxidjpI+KHYjIETaGQFLjZjpuSWkwxoYqHA7UhFkDgmBpZ0EtcsCkaRXNA7x0caiZP6hu4D2FxVqWPZEn3pfEyBg2DoGz9SQ9XuWdwBBpNzlWiLqiiFFpKVkZw+/zahuJg4uMpknHo725tWRHCJP/PXQ+AT7wXAND/xZKMoxnIRsi7NtBbEIOaQkSEiXoOA0bd2n2qJvLSTEyJSZwEp0eVSp4vZ4uUfDIUZVwp/HTcQ7l+8dDnjTVwtf7f8AnXfZjQIABS7i61AYeFtrrdxLEXbxCeLEcEQAaH5ubBDBKCMw5JhDDEVdMGX7+b49qxMauMAI3b4LEysFnVVcClOrTdEuXaKC9tEJhEuNSLlhcFiXdtqA6FFwGCngA6Wahdi9CZeiGebsE0gpyhOIB8NoLVc6hRJmFHbCwlYGrT8Fg/tcq3W70P8AGOMEmNES32UZFKLhJJ0ks/f0qe/KVhKt26qqrlf0W2A5qiXG2RDDleg890MtizcwH0xes6+DsFb/AGJMQa2ZiXmoYS62BAdg6upVBduJLWGGrDhSwOxngb/4DDkBdeyAYlk0mWghCbgJQCmJzLlRVeqOSL4kK90siPziEI18Uh9UAVab480LcHNxFPFEcrp5vWRR2dAtWGSzhZyKI07iW6WuO5JDOlRaleNj+0JMUCSg9UkCZKy9lBCGLZS69rUg/wDPFXVXLV1EwcYuJLHMBq/4LryBEX10gDLSZpawHBmRG41GZ8gkGTQIjEE4fnu0MeBpZM9ahLmIPfo2zgZkBq0xV1RYVZ/IJJyQLzFWhmAA1RuAmTPDSy6bKWMvUAGny05gJ5AmTYZHRCrqMGMtohgyXOXNDGGkFiElrJAJy6xVFhJzxEhUwbXaiDQztyiziIofhFYIQmkHGiOkeiEwQQbCWCj0fjhYAMH5eBcbluLEKKbACMzYLUibObVmxDSp4skaROI6Y+2+vcCnKzf7gT7o/TrBAmDsnWMJYKG+xNRa2OiRNNWlw8VDUBJPOFuyRVbZ+CzPQBuwoZDCFA/3NGmPG1iuW6uxdpKpQkLAp54NXZH1Hyr+U0Q1JIdpHEwdQjABmo+5CgQhaLdjZNZPzk76ALXytTJwo0rNinjLZkRIzUbiLY+COrwqeyReigE7wIu3cqWLHZQKD2FG9eS1BmBriGNyv7JqO4knEdkVrANPpgIl0M++tGjS5lMxcaiTRVEEYI5hDQfCCKDCgkNBpAalfbALHr19YdWYZSy356nkmaTMCBIUUNoeKgCjSZyxHERVriEwEINoONGdI/MNyUMyNnTlPCrY/wCs2QQXIuikWjpgZYACDcfIoSkXMgIRke0l06PAhQS5v8VfWBKn1s6oVyqef9MVkCYRpVCbaJNu4xlJ9UDwFB0EFFwcGNXSg9b0u6YmOTG2p6ZT9olrHWUjtbmEDogEq1BoxKIKFoomNBNZPzaD/wAwVZE1KL0W9DCrc999srWSq/bQMO1rSparRJb9YXdLY2q7NEakCRhagxe8uABzjlZmBx74cVDEt/AiruRt0vbpCQKc4StQkGULKJBaSN6eC1eyaxMlqs7peJEnw9AJsLuuTtOHFAHKsAGaK7RxApurdfqTV71OY+WSGsZhLkiYM8iE6X0LvZZYEyy/P54IvCDI+D6GgS4tI0ThM5k0eSYxfk1c36SRveg8gdc9UZ5TywgRlAdaIG+XQmJMBqrFIXzOCE/rpYhmNslEO2pvpXwBUHCuNJtxMJ0GNqwqB+5OQpA6JRKtX7uFxd7GA7bn6COZmpetxNCC3f0lMWDD8jZpaGibXxMCfXFIJwSCYTgNx0QaukTWMrY8aEnNPzzw+BsDHRYFuzb89YBUwpbGeAlosj32ZiF2ZtWKCrH5KATiLyU3JfIslbaOYO+d8UTkEqe5Y2eGLT6lRJF5oAgSw4IiS+lDJ2zBynuG2zp+hRD8bR30FR54TlEjdDLhbEqh3EHOyi4CgukTFibIYcr3xQaQ0UtMQyADN1m0Fu7vLS9lyUmUS27E+gCLnsgekhI+kgzmrrm7LDijqUQBdaO7TSApurdf9CHI/CATdxMVGzeasRsT37Lw7EjkzKLjxUHm3R9zSMCKFhp8CwVWMKEkZZCb1G9JhExJgJurFSIi7OUBaBNhMmwAggMHpYmTNyREO040y1NIIz4yCNJxnIJ+hwAUkxMfMK9i9qitQDscLncQbUOa8PBVDuBUxIfEMdxMTTxEOIwEcEMI04UJhl1lX5lXo1SEM4cTGYZJgYmnhtsKrelozdH6VciboNgGrIVJZUXO4blj0SWh1MLBLaYXSkPmWFIOiN6xVyhGse7+kk/erfCGzjmomAdiBZk4GhaPRD7oZFg3DIQvLLYSXqaze52JLTMYUSlYbAQB7f8A0/HSWCCJVrC0cqUVpHRIQyZBO25+nsFmLWIATuiYdaDanXSjSnBg+B/xN//aAAwDAQACAAMAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGAAAAEADAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAjEM27ZxRoR/wBSawAAAAAAAAAAAAAAAAAAAAAAAAAADDAI2N+YgM/EkQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABGKgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMMggERAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEZyHAk0ABIgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAF9OqwKiMQQAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwFUdwSwgGIwAAAAAAAAAAAAAAAAAAAAAAAAAABg0ABjxAwwAMwDAQAAAAAAAAAAAAAAAAAAAAAAAAPiSwA8vAAygDw84wAAAAAAAAAAAAAAAAAAAAAAAAAAgQJJ5qhXwAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAABgqQg4gAADKAAAAAAAAAAAAAAAAAAAAAAAAAABxBhJyCEYFoAEoAAAAAAAAAAAAAAAAAAAAAAAAAAAQzIh6oABQAADQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQywJ4YQFSIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzTyUwBQIgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJSxvgADgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwkgqyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/EACsRAQEAAgICAgICAQIHAAAAAAERITEAQVFhEHFAgVCRIGCxcICh0eHw8f/aAAgBAwEBPxD/AEkyaFACnnPHrFAAq+Av+I7y8LnYw7hv4BKApWUNoeDgxd5WB1l+MNYqMq0HlZr81umvYP8AvwjDgQrkDcqX0ci09XYnC0EioOn1wjtomqFsNMEGouBIto1MFLlbrEj0eLGlSOAV3bhAu3mWXoGcEeSieWOBou1K6yRNsAHHwMVCgwaYGTv/AL8uHsVkvQNTdwEitF2AmuwIGpQ1/Xjgg2DZS6usUQJQ3XhfOo8suK2HCA+PzAqFkogJMgpoceZxgl6shOFgLUa6PfGtTCsBL2CXMH1eJmA0XIdBlr3McULChMiEwDkBHrmIt1QggYDN8Q8t4lJA1rlGTPMNKid8rtVRHYaeh64ejEEwoF6JDEHPGvJILSA3EzEi+5wRbRZYtJ+wxrgQkkOWcsFGEqZKeOHOa8shoMXSTjR/wkxFBcsAGdZVfqczWxRQQHDiiI+tdX+Hp7RoWOxoif8AUdO6UTDVcqdr7x0B4PhQwBg7VAespXozxFKgqSjMo0xit1iT+CvFqTKFKPdE+uMG5LBGgqxYPY+sNtMhehgDqZAgGw44fiwqgeBETIpSVFw2gKzfScFmYpYg1KxSGJr2I+RBPfHZUQWYO4AFe3cwQW8xeGqoFIa2qPiT3wvLLYgBwwoiP7nX8ADhaACP2JMs7TgnWiKskNgrRZMTggcKCIJEjK2gDsMSQrY0Yh6ATqtgs+JWiArJcGiJOnK5t40hIKeHvhjMGIWPTRE8eHTloJIc1yq7V8v6OgD81PAdHQkF7hayvjgt3BAwCQ4TcbTA2XhdIAqWQM29TY7Epw2jyIJkygkYIXWbxkavaKfarPAQOg4JIYGFX6ArO2Y4ZDUT/wBonY5OIjM4GFDxd/E4YAVkuDRROnK5t40hkF++/wAzsxMRTzAXibjAyJUY6ROzkuQKQgiCoBSoC61R5DmSLqgA+mR9PMACQAFSTqkCuW2Y+IzBUkGWiLjzanUvJKKGDSgIP6z1bKZ+UxiMWFni7+FDVya2DEGvYP0p0Z1JT9/lJ+9qAJocBFsUyxdA/WJDWRMUeRJkHPSnZj6gu+vjAWAXQkF8hbMXVLzJgQfBmhmpMAW5xU4gLoKB1QUETHIc3YTQEZeggvhBwicMMKq4AO3gWcxIk8wCmdmPg165NbBiOvYP7ToNqSn08T5SlltrSWLsGfRjgAQ/Lt0poolCnY6yvg65iVTDG1VJVy5XTCnJFIJqOowwhL2wNHWY+PINfJ2ghpLNMohFOtKtRqUAAPHI8omLlUAdAymLgGyUoJF1QFnTI/fMSCQAKklZgBe7Zj4a5Ropba0li7Bn0cACH56VQFV6DbzLAjtyYVXaCLkLOGuXSAJGEJnYiT6Vq2gg+zs8CrpeDpK1UBnWVW9kndxvLgbKA0fCI+tZ/wAGaoakttFPBEHyv0hyoWLhirlzA+A/LSoJZQAbRAMZXaBVBxFpSSCWJ3RRiUZEVEkDjPmAWwaCDXIU3OOutE4265ooKDCNoe6RmGYFj0EWKRZhq1Et+yLlOxxTsceLwoEOzIifZjrk+xznKrtXz/8ADHLJyJMoRR7Kbs2cTa6bKD2rpOPZfGOPSB6SnNflp3QwoIAUGIEUouacZOChBAAALAWq1OiVkWgGQgi5VURIoZaLLvIUyjkVXBCMok/NJyIxw/oJjsdKDFYKLAgIgALtOrxJhWFVQ8KKPS/ElFxJAvTEPmuO58DIBc4JnzykchHYIo9lNMmzJy+zmyg4oXScey7mPzVQXuAuzNEe0IwdgnclRhhjQvV1NExxw6kAtdqpVygb54KKhSSuDXQAAdAFd8AYCCWS4aFE8ga2vE0Mgs1Zmfvj8T6uQWoxUesPkuBAVAGd48++FNPTUVDAiqj2Sd3BJToNkA4fCI+tVl4OhFzgmfP5yggMqsD7eABt0j7XIe64rMHygfuk+0h3w2RNI0f3xQMR3M/38SNLokD4Yh81w4Z8QeNKAB2ZET7GdS8z9k5aq7V7X/wQA/PY0E4WLA1+70IZCCJSlpSRFRoUz6EM4CCWS46FE6Sm2vGLAjF0/wCwlTsRyPyiA+rgLUYyOph8lwIjQBnePP8AA5uDZsR8iRE6RHlJWgjDwLX9rwR9KwfWYPIoezh4AYAIB6ONYErcTFGwvQr2+NOYE+nJ/DrahBDbVJcuwQfHNf8AIH//xAAsEQEBAAICAgICAAUDBQAAAAABESExAEFRYXGBEEBQkaGx8CBgwXCA0eHx/9oACAECAQE/EP8AaQxoYqMfGOETKqoA9v8ApT+zqYKhXquvwqVQGMC6F8vEy6wFXvB+M1aoFg2vgLv90+KPSn9uJEVKsMAasGe3lKHiZUZQQGgKbPfG0noIASFbZakEMo0KQoIgDaXegXt8yEClMgBqzIUMhzAnwXGSHgivhnjGrpSO8FDRVXf4GERU5cMnD1ydGxBbM5BB1MraSJeIBZoYkgxd/wA/PGflG0TAwoGrYuocwnfXhsbCFMih5/cUwZDASAYUHaZ8XhfZxIKMpQCCQ2+uYjGCFUBGZmYqe5wlyGAYXsMEOrngfIULgUOQMNEwPM86jWiZQYnmvgnHdFBCOFbccURgj1yTkFQtEj7HHsJqCMUGGbXNTHDFbNMjaJm4q0PV4gsxGrIX6HO+EKyUYLXMA5WDhnGSNLCyRObsAzt/6SZqiGCqpe8AHzeYrYQRRUyZiInvfr+Dwfs6SmkiI/0ezBFYKwBgBoPXyr+CZJOXoBT7gMO3HBBoIBAS4Ej5hPdv8Cl1uXAsYfTA/PApeEiraAKFPpPeY4sorZMHtdggG0ccsrsChfK6jhcx9KvwVKylQpMMVnwlu2hA0d+kfSKPrh8iagcjVVcHRq5ahOZnDEBWBd6ATzb64mpg0KlMmYiJ9Xv+AXalVLcDC+LQ1xQwkCyXGO7tW5Vt4r6iwtgGOiGZDY5eGLq3/gHoMXtr3+LySKhlMmwjezCYk4QnaY+TriZMm0lOyIj58m9EcoViGADQHg+3tb+7o/Cv0XgMhgI2PV/w/Fg1sALVuJM26mR5ZzYIb4gQZcIPGA1bSq+3M9aOjjMFtkA+VAXrOeK23B/zI9Jh4OF1lKxfjX4tTAVC2ZNgN7MJiTgKHS4+Ov3IhB7Zx1AE9I8ytU1XB/Q/r+KzC07iKT2Wns5k+2lUAG9xyAwSXP4pikAFzIiGfEg92cvYAKkYVqfeO5LNfkNFVmVl8zX4BdDAhKlE79KfIPbncYf2hHKKGYvc/wAn3yUEqoOw/wCK1n/nl+EeEf7fihikLoQp+LZ3rvjpMFNtBwIMeD64UFwKFMkVHlBwyzHPVeslxYmSzIpAUknHDKgGVXjIcQai+KmPpj+GfhgQlSg9+lPoe024Y/JwOuBEskhmyhpS/LxVa/tudmY95KfZT75ccxGkvgzvOfr8PyDgAilC7pjIkdWZW3SlCmM2FNVz4cQd0Q0iRl+I5zbN8WqIEQc2hdjcwKGeJ6S0NxSnstPjmazYqoANhcrZ1Jc/gxhCGMkhGyhpRfviq1/fcLAVfAcno4YxoYAQ1iwycmYpRSt9AM7rwo0BtEH148JR6eLpaRFMveAD029TOocJpipk8iJ73/oPKgQDJIg+WingPkLAUgmWgGDFJ5X9tlAP7+A7eI6Js0nKBQ4vfqIcGhB8mTj54DAsAffsbzwRoQYMVVD1WWUIYuRM6IQVwhowDWDuc85xBSmnCI/DnvlADxjABoDx/wDXPJBWrdLGH0wTTrhPHBIFOkNgz6Z5zytg+Rjzf7ZiRKXThI/z40c4IaD58/24y6TXkcLR20nqYi8yexSGILiSa+OK3CYlEoYX2b8aqnZ2J0mRpBEYOLOPs5tKnylX7fxbNmjaTyVR4hnq/hYUZjLceOSOtaaLUPpgjh074L44kCmYQ2DPpmrn91MC6Q2XZnCPYiPyDx0hd5D7Cb395wYSCnS0TIYYLPnjrkCjte/RAA6Ay7440AsFsy2AfhRCQ4CDtMu51frgIR4mFCCNgveTwzKKEVXGs+ONNbBFRXKIAJ029TLnLBNNKZPIie9ws4uLMxluP3k7gG1wHCoI+j+aKe64vIPzDilgJpMnAGVj11/L8WzZhtJ5Ko8QyZL+KKW0FSacIj8Je7OUBOMEANAdB/7at/fqkqA9J7naIj3ZjA4A3JUnlvfh8474t0AsFsz2Ab2MEkOFo7RHyGnzhjomx/I8BomFCCNwveTwzKKEVXGs+P4Dl2N+EfImR9iPEWQ9j/rC/avKr9uhPkKfhU4DMA0GA+uH4GEmYoJKztB6PO3lI/Jh/g5eIUUWgGzBpRTzzf8A2B//xAApEAEBAQACAQMEAgEFAQAAAAABESEAMUEQUWEgUGBxMJGBcIChwfCQ/9oACAEBAAE/EP8Acok8tCu4aoOeef8Aov8Avn/ov++FQd2WDovT/ft/EITrMILB0Qd9Vpusd0GqkPPJp85ETsk09H3IQAYDonn6FGStgVp1Xre/3kl2B1IORDESPp9/TR8wsa6BLHtx9LTiyliASbgBZyHZMJXpFfuvji3DECwUkOIwMRKIoRwgzGZVghqrmPHZ/ZyogjTGBqAXmlxiIxbQujFjGnHFq+2rYkCISFoDVjmhKFsm4QNUK8C7suD5A/AXvYUquQJ5PSS9PyvHWcsWEQ3IVIKhTp8fRtVPa3qiLDjDrTiAU1gKDoi3i9hCO2htMDoKgTjeO42uy5gZKBohxYF3rFpeiBHChWgy9QOpVVh8aaQ4f7umoxUM+TfZTk4lUPKNwAhogSg+7iWqjOtGOkwWlrgxoM4koD5FKlVV4hL5tGIF9qGvdXeS6kWq0yFsjlXxzKezA+J2CC9FaUSoqQCaMUKLAThnwc4h2QIYDDhOS4Ha3bmiNEziaOmKQo9wgk8lOMlUdAFh1Fz58y8Q+lsLEDZB0ZcOPf8AUlWZUGxBsAH0bVX/AD3IAIY5QwidPDOp1Y7nafawQQBzMy0D2HAIGCCGc8HPjzyIXs+hQUSnAFARiugAgLyFLzcR5zQINeoWOUpoDz3J1pWZoKa98n/JpNEgKGT7uIDcq6yC+KtYVPAzHbHRheFGoRCcaBV8eBopKBiEsc9MH/oez127wh7geEOlyIUE1ZqDJpQHp0W+1ePd6kRS+gkGQTa7pSARe1JEHuLEL9tt0KV6aK1QoAOgBQaJ6bXbdFAME0gPw+iMT/MmkViAidjz/geXX+vSyPdiL3uxVo2AGgm4AmiAQeRTxFoKAxWJcEuMQrYXjdNa5LrtoJAhIOXe/P5SGhegjyR93f2KkifE9cD2sIvkhFLwnzEexBIg8yJV1JejP/t88Ys3UABdXAC6wCwDliI7DrbeujvftewP/aCnHYQ+bzzrolAKwJUrKq7zsZgRnndR9t8vE1JaB5TJMaiYiZyQkkt7Qbh8O/N4N3NCUHAAQDkDsZxibVhcg+OEecVyAhUEIfIw4e2VLRUlAVHO+edNEoBShKhLRHfVpqD4DDi6Jgdp213n7I58ssxRU3q8ACHKxQ7h3t+7O9+1Ae1pHTzzUvLbvL90AuEIiY4iIxEQeNJtuZBjgGA9y7xGQLinhCOdxepmtFYqkiqgV/1USiXv2/CEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSB+mUXqJ8Wm9g+P9jiZEWuAijAiJTRHi+iKGAUGSlMt38MaaoodDkSqgIZy4BwCiiPSPIfxUIEgwthMHHi6MEJirUFV7XjRvZzS9CIdvwxmZNSkEZFUqIiIOIHBL2JmjuAuywBURBHA4BIyJQomcr/RjHi2OglKMJV+oJJnQxGjtr9MMB+MApgACq4BeL6XQaSHNv4P5eDQec9VNGmoINMjOOdSMARhAciJR+qhyVwBaFQrUQkYnHTZuATDqMZcZgH8E3A8YBdiW4giofIEKPVH2J3fDMIzlt1qRBdquCVk2BxRqJswib18Iky0LgwqmU5GdEh4WiJeV8PxaCzUAeSgQJh3KZ8gZl74wakmNoRZlNHTvp8kuHByVZ0KAOdsrYQFrWOLC33UHAyXBhiRBA4eL1CtyoopNVpGQxHp5V/16TsqlLp2efv4PeGV7QaJCjCkE+jFBKSd1Gi4ieoYeqpETnYMRLtFI8N+gCTsTSjHG7LmmVGjKn0rR4CHXwUvkQYiidIo0XmMXJZ/Ao1AP8AuKh0kbRFJCRSOLx7BziSsqCqaqrzxQyvh6waYqbgff0EirgOIj2cIGKI5ffvN74AUFJpRMqRUpqKcommYdAF85YJebBvQTUgkKF2xOBxgwwsYpgQCw6bwvoIcyJAExNSxxrRUFBM0ZJAAIF9Yk/8A3dAOpGK6WJp1A8xEiSlWzIRV1s0655AJ8Po/ajhjHc2Jkbx/anwiUXxk8RjBGDeRxvK9qloIfwG0x5fu6PfloLfHF909k2wlajsQBx/CHHnVDFUxQALwf8ID10iPfItLxCpsk68UgLkvLaPUguOGgj9AAD1JGbjq6SO6IiAWfrFBXuIwl8UQF4sizDPbD5MrL2/gCc/uoMKHCKiCIPNrIvRSqhRMexPoI6jFs8lF8kRmjIFnshL4thodreDIMeD2TOZ5n3zjfYIsGjXitCAmZw8yyyzH5GAHtlv0DRMig6bC/PBqRCFzE5MO/b1xOSLxQQggDaB5bC/B5ECBKBUHQjP3hwB/hQn7+9mlRQUqUaAGaoQKrVq9vGy1DOCaOKZobH0OG2yLRWO1KrgUA4pBwC35go3oue7kEao2jVa2IKsQDhDN0HYAdqvsvgnZWR5WABVWAK8KYKxCHhwqaCVsCiq0ugLQAqwSN+phSPxTRSwxBBAUJxYYrUhlL4gweXEOggt061Ky/ew2A1k2XQ7IQyQQDAuBQMoywwSjhfOuRwDA9sgVp6tBXIloeHkWwnz6GL9+zTtkVTQFenAxdZAYGCoF32Trlr0RR2WbnLpk56LGrkiOv2DPdPqHLZT0aO4z24IBd9gv8EO/b1SUtA8ht4wMBYcz+f3aGNSgkQgWp92DEAumQPhAwaU4G/QZ5DNARIAiRPrDewHVZ1uSO0BQMjFn2q68FxdguMMMeoqSsVT58ABz3I8qLrWHAUNX7OrgPYnk4gLGS1nnfsSyAHDYL4SaMAxFprrH8LToQPX0ymYKyqWUxnyjpPIVGIQinJY3fmkWFDF8z7oooTTBGRESGIjxvoMsahVLVVXjyRUo0MadNLNw+sesWtdB9yA9lUABCbK0QZpu2hCUeky7Yisa0LISLHgEHWpBp8TtFgCgovxnGyQn13HbJ7uE23OgHYoE/gYWM2EwKvQjQ1hvJZt2FAgyqCCECinrsOydWMHxREIhwQRa1wViPKAoIOQ9LzCQTAAAHQfdFYIyBDRRBU8PLNtlNaBMYlCKLodF0tanbqBW7vmMQA8gwgAFVwDlX7hI+x7n+nuN4AcBNIoExE0T6DbtPXALEd7aiO8YJgsMZF1XpSKJyWQxQoIgjjKC2FH1i0kEgqaQqKKWzANRbVLU0gXglY91LoyUF8FhxZoMCdDehRfymrwNqsh3OxsjxB88ioSfPArvAKMqRPRqOzjFzPEJXKq8Rs2pZRzsek6RRovFLijQ8qFBi+Z97HwhEVsdsnb8vorLpI77RPM2QBfR8rG4LwrPWoAsgvGudjo4BaBQAw4K4ciwTrpjbIKVqmggLop69iX255P/ANCKffHdAvCDE6gge/IE8qHNHoDDUsF4YrpN9G4LostJ7Yf0egokEIwj7xLGwK9PQiNTVgvQ1SiGBplcYlZXb3UC6ByC07gmGYAABgH3vP0JTIWhunsaWGn6D+P/ACdfHDJ2sHsDOAlL9D0VQdcrROkX8qdAJyxknVwBKI3isGQpWWhjoGKITjqYlNMXtIjpSKU66jI9DJQppI4p3xDU2/lWkWSHG+EW1LX2gDzSchnkbwlWtUKkoUVX0hP0Bwd0YAdgEDoJ2+WhRehAuwoRuAhQiSihIVCqr+Cpw04HBZCYgCglHHYdZABWr0C6KrDgEpT2WgnQDjtR6NdDUnqzgayByCxMjwAD4PUX7r0AQsYQVBEIjxViwomL0EJTBdD+BZo4oUAKEqhFkBeGO2aowJ1BmAFzndBGA00d8x376im6She+F4XRawFFggxC88IGFVQcFI7tOLh0u1dfPA43Z3LAHYwUKB7jcbMUk6QQQBy8nV4UGo0wckB5AyptRKyIIiMR4ioUYhQDgoFIESnHHkDYKQqKquq8BeXRWO7oEYAQJ+s78xqHCBiNhcMiwDz5UHgHwnBL5Xj5dizeWNPvk9DsSl31QN9x8HEhwohALFWvY9+8iUt1v2AqQl+wRDyEhAGr5VaVkMIGN+DqLXBAAaktUnXa0LphGoGotk1glroiJI0PHKYwd7lUpEJIgURfa/t3Q01YBFoB6AVJhoBkU2ZCDCp9uyaVmiPeXtnIfxRDDWChxFA09Wczvp2xU2GXgmOBgAEAAABgH3eB34hBDbWJW0jl5PSFootguUMVhyF9ZJD3lIR9n0WCkSn5EdPnkVIBZ+RyfJHO3jq0MiREBNIox8xQ5grka1oeG43EM4HJmZQXgAx69QX0lokgXI2C9rRE9kzBRNFG8AaPNTAyyRjlCnhEvtt2xfDFOVD6YAxOumNNI1wNOAEirlOAB28CFh6ohgZ+1LYg+90dQtMOkZLO16gBQCq4HGACyx2IIjiPSPGoXkV5tJV2HJs5XzUsn7Q2E2gxF7vwFEInsvjhCdCIDaxeKgVGWDM0zglyMKIFWDzNsDMiBSm+gQPC6IuCZULoGHvxxgu6+ghADBTzDj6Y7OMgpwQIaN4LcBb3Q4H6PrGlYdx8h3Cq1e/UtskbQkCAGy0qhTAWk1msnZfDOLNUc1NmAa+ATyfd/PllSju6jFEFU5O8YBCIwCGAVq8Tr9tJn9/ponQvJmBUSjdIxMCuJ9Gq+lACoIY++IarhB5KZ7i5IQDjFBVKwkCI0TgRzXjwUI1BOtA+qf4pmTSaYlCPQkFGPVnbFhZ9yLwESj9LxeSnoBoUfc4lcGH+8SYAGqvjifi55MlSpvQmQfejGENIKTEFEcR5cR4gdRXKnzomqRbW+K0aixVyIOXC9NK+2pV21cc7VRdeBolRe9o512AGOigYHSEUpCTSfmgknB/B4yEbcgAxlilCeO1gA6yEmqsggSkWowlLR0osfRq82EDzXb/YgyTh13VynoDy8L8IJeubkqmta+rVJIBEocE7YxkRRnoQKB4BCUaog+hs/EKVJ3uDa1e/v36IQzhAX6OZkJUk6glMBYwOCVD+rCMn6T98a95DnoU/dFh31dVu4YIjRQkLDhDQywgXmEKANUOCFtogpwCpf+fQL9nReBoPdOgAqJRo2GiMI47ULcPULzEKldYkBP1uEooH+8SaquqvnheMJSvnVVQUEMQ/gCK202VtOTD7Pp3lBjAYNCgxPHBgQFFQjYKy9V+sl16SKFCQpsZocuo4e3tIkKpNB4pRiG50fqwG6EH38rT/AA6qBqgYaTB4KRQ2B6CticnWnLQtzYKKdFkSKTSxn54/r+TBjk585r/zvjh5u1CgSgKGmigX6gNPKNiIBNgEa5UNiqSGgS7dHUrwIlGj0/gT8DfVXQeWkPLy2vlyQAREIiJJ44Tqols+JBh7cOCygHTu9yhFNPcgrBsKWuEKKee1I4SiKpKjiILSjHKqHmzgWCubOQnQReRKP01SoeB21wp4QZJw1wEShgAdrwUkDIo5rkqmu1/Ao0bSIreO18XimuAgho6qhgoUXwuaj30AgnFAjxeq5lqFS0UeWgEpBajW7DRlA0wZMKygTJEUgOpxNOm0yhEajQkLwdACAQD6WfEzCepOdFtoAfa5WhnzAlYoD8EBCc4+QYaATB8AAR7zECr73s5Xpx2rWjGbA7pmZxHdpZBSZZGx4OQpScTY24oFAp3xA1GheVayFcQLAPRv4CtIhZV4ymG8I7FH2jDioQKyn4VRshpOpqEA1U4w2CqQwZgpTpPoFcoDKhamSQUADBmYvx0toAE0QeFGuxkbWwWfhINk2ag39UcPTnFIPSinbjAWUkrX1RNkSUsRACNm9wRm14RpGxDAyEnibwwh4bwAAPj/AOn/AInq5mCsB2gBKi9A5RHKlitEgo38Pr1k3mm4LTUQCUSy3K823MAKwq2H+kv/2Q=="
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11.\tUnder-sampling : Tomek links\n",
"\n",
"\n",
"Tomek links are defined as the two observations of different classes which are nearest neighbours of each other.\n",
"\n",
"\n",
"The figure below illustrate the concept of Tomek links-\n",
"\n",
"\n",
"\n",
"![Tomek%20links.jpg](attachment:Tomek%20links.jpg)\n",
"\n",
"\n",
"\n",
"We can see in the above image that the Tomek links (circled in green) are given by the pairs of red and blue data points that are nearest neighbors. Most of the classification algorithms face difficulty due to these points. So, I will remove these \n",
"points and increase the separation gap between two classes. Now, the algorithms produce more reliable output.\n",
"\n",
"This technique will not produce a balanced dataset. It will simply clean the dataset by removing the Tomek links. It may result in an easier classification problem. Thus, by removing the Tomek links, we can improve the performance of the classifier even if we don’t have a balanced dataset.\n",
"\n",
"\n",
"So, removing the Tomek links increases the gap between the two classes and thus facilitate the classification process.\n",
"\n",
"\n",
"In the following code, I will use `ratio=majority` to resample the majority class.\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"from imblearn.under_sampling import TomekLinks\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"tl = TomekLinks(return_indices=True, ratio='majority')\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"X_tl, y_tl, id_tl = tl.fit_sample(X, y)\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removed indexes: [ 0 1 2 ... 284804 284805 284806]\n"
]
}
],
"source": [
"print('Removed indexes:', id_tl)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Under-sampling : Cluster Centroids\n",
"\n",
"\n",
"In this technique, we perform under-sampling by generating centroids based on clustering methods. The dataset will be grouped\n",
"by similarity, in order to preserve information.\n",
"\n",
"In this example, I have passed the {0: 10} dict for the parameter ratio. It preserves 10 elements from the majority class (0), and all minority class (1) ."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"from imblearn.under_sampling import ClusterCentroids"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"cc = ClusterCentroids(ratio={0: 10})"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"X_cc, y_cc = cc.fit_sample(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"284305 New points undersampled under Cluster Centroids\n"
]
}
],
"source": [
"print(X.shape[0] - X_cc.shape[0], 'New points undersampled under Cluster Centroids')"
]
},
{
"attachments": {
"smote.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAt4AAADhCAIAAAEHQhXPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4QsNEhgnrMXRZwAAIABJREFUeNrtvX1cE2e68H9H8SygDUnpC7a8BOwPu1BLeLDPikUJBd+2tWLXyq8vK4m7aLvblVDOsdbdCtntaWm7lLhtt1JPSejT7Vmt+wOX7vENlqAIPusLSS1spaeQCCtYQZIcF2ilze+PwXGYJJOZyUwyk1zfjx8/YTKTebnu+5rrvu7rvi6Jy+VCwExmietyNRpNXl5eY2MjQuj73/8+T2eRULWUihiEENI5eLzLihh+f5/djbhYsEvq4htPpxgfH3e5XOnp6UuXLl24cOGyZcv4OIvL5YoQSL/4tM98f4qSep+oqCiE0Geffcb+NNZ2pMhh1lIStzfx3QK8nWLVCzkBOAvNNj5D0a5Mj+O7RSxJifW4/XDVCW+HdHR0cHUWmmpFIuRXcktLS35+fnBeyUkvfELclPTCJ9NbDA8jw8M3FTj+mTnup5j+RPxN7B2BUFlZ2eodyxBCTJ+I+1lKPjjj4Ubczsjk7bNLerMT7pK66n7I61uma8tc0pcmk4mzsxBvxBeIp9cnffR6PdX1ITTjlvij9VX8LMG3aEtLS2022+odyz7tM3tsyBxbcd66DMdmPm1L8VTfqGddkJTkvnH1jmWYZkE6B49WL/6YVDtunoX4br9v1yHfzcwPzULLDur6g5/KK3F7U82xC/7cCGKsMty7t58dnqQyCH0bY90qFTc6hfaNIL71KOOHEtjxFFVLITXsxO1NG/d08Grg8zSkcD9L55cjTH9k+qGwOJIp3ZccxD89nnHVCzkVb++c8T5mCK0b8dU2kV+tsfVVbh+cWq3GP9vt9mB1K0TdAme0Ru4u0fMp6n7I7VNgfSMeHgqt95l/HPlsKAAtgPWNIP+FgP35/L4uDu/nvl2HgqWYB66O03IydX45QnrqJK3J+lZ5umdvv7ZxT4dj4jpxC+lP3y0Ffxbdlxw8NXj86nl1+t236xB28zXHLnh8ClSeNxLSqDlp86UIobT5Uo9OOW9jGRbYXnvE4/ai2k4O3H0LbsM+aAtSpZEe3NJ17f1+e/P9M424Hy5xfRbEaO+aYxc4v0p3VcfHg2B0FmZTHNqCVG1BKq/uFW/9KJBnEbTjOljAQxHzBDvmnXvjjTfeeOMNbEtCQgJfJxPgnHHQz8K8+3AaiiCXy8fGxjycIu9FpNpB3Jabmzt79uzR0dHc3NyFCxf+/Oc/5y8UIcjdZ2xsTCKRfNpnxlz5eXl53vZsa2v761//arFYEEIsn8ikA1nbmXUfx8R1vi0lby7lVS/k+JxgNxgMv/3tb+mc5fl9XZ5vpP8EzX40Q6cMXB13BQOlUknx7cWLF10uF80nwgnIFWy8zY1euXIF//zOO+8E8pIQleeCODPix2uCZFPPOCN+iht+U4TQqhdy/veK/0XsOCzO4vtGvHtqab+SeXp3+rrEkydPcvxKDsQE+y6pq/8Etw9KoVAQtUngbRmhWLTT08YIIYT6+/txmzUzM5P7MEn6r+QATP1Q4P5KZh0FR3Uj9OZ8ETN3Tt0PWc9++7SDVCoVWSny55eivBHku6eRtJT7NLX/E+ye/rz5quZqdp04dUd5IwwfiqjHhLTnM1FgJlkStzcR3eh8uDV9mCr+W7TBUrqrXsjx05dO60ZYTJsKpdkHL24FUbfAm5+5u0TS7884BaddaemrzR5efDRuJAg+2qPdwwGIdw+0jzbphU8GxyYoQp2pofNESD9Yvt/M6BTsIrRvbiG2N2/RkVy9KSh+h/SVP5o+cXvT+yf66JzF27wyonMP/HnkAjPBzvQstLoPr7Pf1Kc42j0cgIlB8o3wPsrwg+5LDhFMsHN+iY6J66T4n4BFLPnbffib/db9+bPyfWaYYIcJ9rDk1ltvvXr1Kv7nuXPn0tPTv/e974Vl0AG2Soamx1Ow8HMLWCspKCjQaDQnTpz4xS9+4XK5amtr8R0WLlx4zz338HIvvEkkAhQATzQ3N2MfTp48iRDaunUr/tWFCxfEdz8BcuEEzF3E09QyvuKVhpf3888/n56WuAGmVN5++23sz5GRkSDfD3OJgI3iYYbytSfebmxsrKio8PMHt27dSnzjBBRr+/SCfY7izTw3lPL95gNnBwM2COEJ3KcWsLvYuXPnK6+8ghCqrq4uLy8X9b0E49UTumzZsoW0JZABaIFE4nK5BscmHqxqoWitWHPeu2mxkOcksIvctzXbY34dn/dIxGNYL/Zierpg89MFGo9HGY1GtVrNjX+pqRtbe0QtEfbaBR8c0X4xzUIIxcujEEKHtMu97XRet5o8T+NtMIZt9CMXD2uwi/SWhcnzPXq5hbFt36GKmLa2NpqnxrJFcdVKEEIVa9Op28HeTYvJS9uoJQKvHj/RarXuGysrK73GV81kOkIxDBDAqAdr75q/0MrQxw+ZmZldXV3T5r2E1jMZHx+Pjo5mrcmFi6kKtb7qfi8wPCY8C4kEi596fctbpOyV69evb2hoII2ioy7chm8UqN/ZZ/Ol3cQjhDgS85uj3cNY/j9Gt+ByuYiqhQjeIBwOh/vGwNjpafOlFHakJ5OYS/XGs0bhtFHziseRjkajqVl3p6zrXdLlaTQag8GAfa6vry/OXcCtd0uAEvG7ofgv5qA3FP8vgGs3KAe347ZA188HAjaKD4jWSTgTQd80EYXtQu12K6rtxLIB0bmFaQ/bzzcHqwHRlEjZChr5WvzWmsHQKP7rxoCw/b1tCKEVWWtWZK1BCFksFmwNuSjHw7gLVPMXrxKhvJcI/5s8ub2L6iF67rWGh5G1/fWZt5CRkSHw13TPkHON/jhPEvGtURi/cQTZUJhNjhgens4gIsi2HhSJgDEL0MJzzOypvlEOV6jg/SDphU8oMktiO0wXyuCCotpObhfBYFdIsRyt5IMz1Duwpq69nyeJ+NyhZ8jpVaNQjx1EMSbC2zqj6/F3+l4gr04eJMLy1RMCbn4hNwsBSiRoNgp2Y9LICCyORIwsqjjsnJwKsaZWsTZ9c05yoBtKCHRZLNgsNJoCJpElKbH7tmYzPZADPwojU0Z0GhsLSewZcmLp34WMvrm35livz57JLlM9DI8BaCgAr04UgDW//OUvn3zySW/fnjt3jrh4HVpJmHL16tWioqJVq1ZpNJpvv/0W3/7SSy+9//775eXlYlTebG1YTrPiBwd+pqLOnj37gx/84MiRI6TtExMT8+bN++abb3i5Fy+B0MFuJYAX/va3vyGEVCqVyWQibv/tb3+LECoqKgK7BLjRt2c2EdETuJQKXBc19R+73c5HVohly5b96U9/ysnJqa2tPXfunMvlwiJXsK+WL18+Pj4e5DtnmOEirN84MTG0llXW19cXM/nZhoaGRYsWrVmz5vr16x0dHadPnzabzRMTE/fee6/NZrtw4cIPf/jD1tZW8Jd4shMFHOPocDhiYmKw4kkL7rpnbuQ8bKNMJhNrjCNeosnjOkuGgw/wqt0EC5l+fctbuVl5HiqRMYTzPChBHMF5tl6xmJTy/WbxinxwbIJ1lJDHJkJRoM29fSCEOE+V454WXBBjHCynkkjRNXUzPeRw1YnDVSdIi4rp09TUND4+znn7YHlTOsf0P17HOBv3dIg9C8NP60/TKQiOYTabpxP2Mae0tJTvexm4Ov7T+tNhnM9CJOTl5ZEGJhcuXBgdHV26dGnI3zt41RCp2FxycrLPQ3bu3IkQWrhwYTg0kekxDnXWNefk1KKKw0jYsWc0M8uxDpvC05wcrjoRgNuhDvbzmbSN7giIUU42DG/rHkrqT3vYanjYQ+K14OUxX6OnlT9txgoGw8MSicTrXZiqcFeKRCLBv9RoNAG7KX1zr8ftWBOZgfkjXiUyC93IeeetYe7bmi2NjCDnY7G20y2tHRCwW/CZfHDGPVrbXbukPu8iJiYGM92uXnQghPAcJ7yyd9PieHmUt3Xk2F3MuBf7RV4lEoEQkkZGUOsuUYS8U9+Cz3v0OUgmmTIeUyxxxcr0OOqcq/6+/RmOkMN6jFNfX19cTJ6isVgsGRkZdA6fkbYv5K3XYJ5ftDkRx8fHr1+/Pj1fGAIxWZQSgZEwslgsTA8ZGBiIjo6mOaUM/pJQAH+/rN6x7H8/eT8+6CUil8uJfyYkJOAjIIHelbWdw/FmsONLBKalb030/Fjx+b8Pmw0fNtdh9uzNxFrifde4v1883YsHXXKqbxRbCitq2K1jY0TAkvcF4F6YWa/s8u8KDdbr8T0OcIjj3tvjbstSfx/TJTQzmPvJGv1xLK0ILxKhN3qIcB+pc3YFw+fRoR0IeUkJxz9YDQx2NopOp5uu2WV4OMPVjQwPY3dxZXgE3zkww8NV98VhrYQD3JP00XtX8tkbaGbVFcYwEgtqdDdax7Z9R/1Ad+/eXXp1lzjG83Q0h6d0c7P8PWuoFKCNeTORwmglQSy7U1paKrR74VwifI5xFDkh4GhK/uDW/n7y7Fpubq4ob4atOCLEe+ncW/KerFH3JuJhz9DwupJMlgDZJeLHo7GCPMWthTY3Wwk2ejyvW00uJndjyM4iCaLQBsB0krsT9URgxrrUt+PtXgKZ4Y1svfZccgjTscMhnV9S3QvWLFbvWLZ6xzKsWpcwYeb59M+kndFXqHOIneobFX4WNZ8XSfMu8NmcC/sG9Xr9unXrgtUOPKp2xhLxz90w4wqo08xx00R4DhXweZE07+L+lOmQ6cP90/FHMyKPArKmlaJ9MJYIRb1NGhKB/CWe+WXRb0h2K95ENBqNITEURi6BGAl7NhV5TsoTIAux4kYD8XQXBoMBVfx/ojHeuVDeESyfo6i8I9wn4w5qHwi8ROjqEufkFOk1GRp5lUV3FxSlO/i7F9/+gFN9o4NjExuy4kXdGvTNvb4L4ImEuvb+lelxLGa8eWwlABCgMY5PR2H5fjOWCEP4vt2TO/K99WNshw1Z8dUblQIXPKOysJ4jBzgvXeXTUch5DOXR7mHOK1ZhDI6NB/heMIkEK8WN5zcOuwkCn4Z3gF23LGrxYHNV1LM8QSlxwW6MxqFEPOiSQ1r2wRMU0z1B8e7vLX6AUX9FCGElRLwRlCooHksc+bTWfe7DQCLskuMkbm9K3N5EvUXg1By7kLi9qfPLEXxL9yVH4vamI58NiTHxE68S8auMX3jG2QtznE+nihJrgra2j3MDmfHLaNNiRJnMgqYdI4RWwokriMJADo6/JDRqJIZAsUqamjU4umTfM6GTjoyRgSxYqNdh8ahLQqOMZtILn4SGtkAIPVjVMjg2wUIi7HWJc3IKr3nuESz9Fx813rltBBRXiJkdYlk1feDsIPXTxswOFhJh30qwxI1r9MfpGIkihUUB3iCCZYT32QhY6BL28zibc5Lr2vspvEwQWhBgpJERzskpiikk1vcCc8IAn28cIHwAXQIAALxzgPDg7bffRggNDnqt14SVuJqcnLx27Rq2ZWpqivgVhnCTJYI2AYDA8NxzzyGEZs+effr06WeeeWZkZATdKB1x4sSM7O/z5s2rra2VSqWrV68uKCgYGBjAtmOljMInR294jHRIi9918K4IEmITxNDQ0Pz587HPr7/++vbt26n3n5iYiIoix7FfuXLl9ttvF/R94gtkxdlHwDYBRACuShBCPlUJQshdlSCEhK5KxE9gc1QoeEsyRCo4JUtEskSQLrUgVLojpopVISmId9555+c//7n45CJLDFwf4eFEoTKnQzLdec5YFDI4HA6FQuEtEbJwBJGZmfnXv/5VLpdfv359zpw55eXl1dXVr7zyys6dO998883nn3++tra2u7v7d7/7HULo2rVr9fX1er3+5MmTs2fPnjt3bnZ2NnY4DG95HUPBSCesiYmJwVSJzWbz53c0Gs3Bgwf5u065XL5x48bz588/+eST69evT0tLwy4eITR79myEEFbb9957733ooYfmzZuHEPriiy9UKtXSpUsjIyPxw0HivALxJsCMTmu1WmNiYkhl6l7f8tb9KTcDJ202W2VlJUWp5d27dwurqEPIgNeN4dPECJBtcrR7GFusgAVwA8Girr2fD0GMjY15nEbNWJC5e/du/M+kpCTqqu08qRKz2bx169YLFy4YjUbBiqZ8vxkTDbb2KKxgoE2KajuxMn+IxqIhgD+Kajt1Td2BFITlyy5GCoIYM+YPIyMjWq22o6MD+1OpVKampi5cuFCtVgtTNEkvfIIlg0MI6Zq6BZKAQKAjHWLD3btpMZcFIgGGrRb/fEi7nPO8TQ6HQ61WNzQ0JCcne6wuRQ3rkc74+PjLL7+ckZFRVFRE+qq6urq8vFzgcsEL604PCUNiOSP4TQD2ekSr1eIDGYvF0tjYOF1Mlx9+//vfR0dHU5gbotAjAGgT4CYajYbaG8JopENR8XDfvn29vb3l5eXR0dHUvwN6RKza5FTfKJYD8PHFCSzqFvQMOXV/7mZ9OICDC+InOcksxpIsDs/MzGxsbExKSqJQNJWVlRQ74GCTQdbT/1A8cDdC6OmCzU8XaFpaWpqamn71q1/ddtttNO/CaDQK0DlS195/pHsYIVS9UcmikoSfh/sL//Em09oET6nIbshXVNtJyinNwYiRNBMWtwitqQp5VeKnIJgeLpfLfYeuGR5GCKl3HzOWrkAIocynkPJJam2Cg2kTRk9AmHoEuaWgos5T7/PwlelxHCQRI/UR6qp8AYteI+WmZ3qfpGyXXg/Hqt7i/6ixts/4NxwWoUckQTBN/Eb38BsiGNv2HU1BGNdFTQvCfvHgwYOZmZkSicRisfi8JIvFkpxMK88+Nncj2PkaUrUAppmaSYd77iOmKr/6SHj5TRhpR9LOihw/C6IC3AiCRrC8xWLJyMhw3+5wODzGs3R0dCxduhTEwngNscDWggu4MjXkK+C76VqnVIoImoKQSOi+eBwOh0dVghCKiYnB8pJg7t7x8fHR0dGEhARQJaHRR2BOB+C6SdHQO+Pj47W1tWVlZfC4Qkr04a5NSLai5i88LgkXKvX19Y2NjQ0NDR6/5XDmGCE0MDAQGxuLzw3fVD2QSUv8gy9YQwyg4uJib6pEp9MxcotiAxkKEhISiGEmmCrBVgADYici3B9A3osz/gz7HEt4xhN8rrfzUPOrh8jLiD1isViUSiWLk7pcLmSqgt5IiwAvI1bkkPsIaBOvQFKlmeAZT1iQkZHh7n/9sNnwYXMdccuR19rtdjt5cgcEEfRhvkfFpMihP/anpU2IMVGw2C+I8L3Yj8j9KZkIofrf/qH4X59CCM2Lmke9f1tbm0KhoBMv63K5MjMzu7q6QlU0/C72i4wRrGvPtxdW39yLRWoH6GEBwhBEXl6eXq/HbY28vDyz2YzlUvLcktymcpKTkxUKBb5gxz3ullYkrthUSTj3EVpzOs7JKazisIAqe3M4eiSN2AVscvcMObGiz2nzpYe0y/nTIxRr9nAXiUwmS0pKIvn87ZMuTW++N58uFfaLyPyReMc+5fvNWGaTDVnxFAV+BTeQoYP5I2S/SEc0op0h5lCbwNwkwVigsD48W0zPrNDe+Tc6D1Cn03nLbKDRaIpzF6j63wBBCFGbGB4mh+17+Z2IULhbgCNBjG1D6M1ERoLQ/r/5qPVvtE5SUeFNoRgMBmRtRyRtAiCRFWMIqm0C2kSoLzHZbondbqd5NLv8bEDoaZOg2ibhF3UqUNwEYbczWGBJX5XQX+wDeBaNsOOhPEsX91FvzkmuWJvO9EfxdCfx8qiTO/KhSbAGF0T1RiWLHFScCIJCBTCd6A2Zyhi4OxwhdF63WhrJ+K2MR12w62Ki0SZ+ZoXhJXNSeKsSdoLgdtrSbDZjca7EfEgTjsmSH/2MaT6k0BMN0wdLOnxJSizTRDa0hkUB9xt4WKdDyuPCNCsM6bkIZUZZhBAnGqWREUwFQWrifup0TJWQFtRExUSeOMosSU9oLMkhxg2ysBlJs/vcqBLBjnRCClI4CZNIYcAdn7kavQ5nTFUIocIdexurSkAQvMDaNuEo5CoMtAkUPOf8FcTEmXrw4EGTyVRTUwOCCEE1NJPgzOnUtffj1eq4SbcLBEoQycnJjN5A69atW7duHTxqpjxY1TI4NoGPjHhdkyWskQ7uVaLp3yZ5ofZtzV6SEuvVBmOUKTO88VMQVL4V2gGRgDv4HBDNtSnM1mQJJm5rFoctGCG0qOIwaULHI8QHenJHvldVAvghCDqHkAQBz5AP9M29+HSyc3KKTt1obUEq0TUrlllRDkY6afOlPUNO7LM0MoKOapBGRtB9QEyStYQ5JEHQOYSBIDKfQsnL4CGzQFuQSjQ0ylbQmptbkhJLVzSC6SAQmwgAgGBsE8Ea/KyX7Qcu8014jLz8F4SAUmGIn6PdwyUfnME+s05+5jFcflaotmCEUM+QU9/c688vIE8RpUDgBUHT3QDQAVclpM/sRFPX3o8Ll+5IB0+YhHFyR34QyjLTgLiAgrVlIWTbZHBs4sGqFuELgvgC9F8QorBNSBMxgrVq1+iP4/41dmYjsZcRlwXQ1SZ+VttmbRrAQIP6+YAgBCsapuuq2L1OBCUaun6T87rVuHkilliakMT22iP4mwEEITTRYBZZ2C6dD+icDrzuhPkWBUEIc5wuOtFArT8AAERomwQdsfiSQRBAsPDHlxxetsmDrzYT/1yjb4PWA4IAiJDWB9W1M8j4G3axsODCBEEA1Oiauuva+1nMyotSm5DsZAiUDBan+kaLajtBEMIfsPA0XU0iOJH1fs4p9FxykJQLtJ6gCIK0XhwEwdP7koVoOr8cJf2pLeD9skW5TgdbXtkz5HROXIdsBkFEW5CqLUjFdAoIgsth4CV/E5Rg8akBFk0wRzqn+kbj5dHgzA/6a7DnkiPtrhgWZRwAfnWK2N6XkJEAAADQJgAACAmIhQUAALQJEB7ccccdkZGR165do9jn2Wef9fbVli1b4BmCNgEAtH79+q+++mpycvLhhx8mfWW32xFCHR0dCCGDwUAcs//P//wP/tV7770HjxG0CQCghoaGyMjIkpKStra2P/3pTwihP/zhDwihgoICmUy2fv36iIgIhNDXX38tkUgWLlyIGSO33HLLypUrsa9iY2MRQk1NTZOTkxaLBR4pf8CkICBozGbz5OQkulFg8K233tLpdE899dQTTzyBEPrqq68mJiYQQomJiehGseS+vr6CgoLvvvsO+wohdOTIkbVr1yKEMjIy4JGGhDYRTA2hcEdUgjhw4MDExER2dnZCQgJCqK6u7tixY8QdZs+ejRC6fPkyQkgmkyGEFArFf/zHf6SmpmJfIYRWrVrV0tKiUqnMZnNWVpZoRCO24qoBnCEGbQLahC2jo6PYgGXXrl2//vWvSd9+++23uOLAGBkZue222zx+JSbRiE2bgN8EEAGYKmlsbExJSXH/1l1fYKrE41dASIx0AMA/CgsL4SGAbQIAANgmHKLI4euX7ReR/WKAzhUCKHIQQna73To8prw3OfQE8dxzz7399tviFc1NZImc/bK1PQD9MYDaRPMXvn7Z/BFqfXXGFvDv+hKEDCFNZqbxyVe4nDTlTRC5ubkREREtLS3f//73586d29HR8fzzz//Xf/3XnXfe+dVXX8XFxZ08efLtt99ubGx89913sYmFkpKS3t7e/fv333nnnfjhohANLxjIsX989BEY6YQvXV1darVa+AFdBoPh2WefxXTB3//+93379tXW1kokkr6+vsWLF3/55Zf3338/tmdzc/OqVauwz//6r//68ccf33PPPcTDgVAZ6QCCVCic/I7D4VCr1Y2Nja5d3Gd4feyxx2JiYn7zm9/86Ec/+t73vrdgwQKXy4UZIMT4BtKWr7/++o477vjiiy+ioqKww1966SWQOK+AbQIghNDu3bv9OVytVjc0NLhIwxyOmDt37rJlyz7++OO0tDS1Wr1ixYovvvjCfbeRkZHc3Nz29nYsBPb1119fsWJFX18ffjhIGWwTIBBga14aGhoQQn1D/31tYsaC3ftTlKT96+vr1Wp1YWEhdgj2P19tNCLixIkT2Gf8AwbmbX333XcRQvPmzWtra0MI1dXVIYQ+/PBDfDfSUSImIM5U1kC2JGAai8WiVqu7urq2v7ft074ZI6DDVTN64/r169Vq9bp167z9lE6nq6iogEfKPcKOY4aRDjBNRkYGhRtFo9FIJBLsc0NDA4UqQQjxqko6Ojq0Wi2WbQAQ8Uhnjf54z5ATQUWlYPNgVcvg2IQ0MuLkiwV8Z4f+x/nLmZmZjY2NBoMhuHf9+9//Pjo6Wq1WI4SWLl0qQLngJcfi5VEnd+SHW7NkYJssqjiMqRJMrUCXDhZJL3wyODaBPFVd4YO7F93Z1dWVlJRE/5C8vDwOL+DChQtbt27t6Oj42c9+tnHjxoGBAWGqEmK/GBybINUqAm0yA1LtJVJlJiBYhLAg9u3b95vf/CYhIaG2tnbp0qXj4+Ojo6NYagIQhOhHOkSkUXOgJwuBeHk0tz/4+pbfBfeORkZG3nzzzfz8/KKiInyjwFVJgHqEsIO8GdgmG7LiiX+C3yRYEJ+8NDKCj/JmmZmZCKHkZJareFQqFbsDW1patFotQuiVV17Jz88XkSpx7xFh2EGYzRCf6hutOdabdpe0Ym069OogcuDs4MdnBrIXxPJRqloul5vN5qSkJIfDodVqA+N8ra6ujo2NxTysJAYGBqKiovCUJQJH19Tdc8n5+OIE0tsXtAkQdkgkErvdHhMTg//Jawsxm83vvvvu888/jyWI9qhKYmNjo6OjQTSgTYBwJC8vr7W1lXofo9E4OjpaXl5OsQ+oEnEBkfXAzQHO2NgY32cZGRl5+eWXN27c6HFQA6pE1EAsLIAcDgeFKrHZbDabzf+zNDU17dy5Mzo6Wq/X+wwY6ejoAFUi4pFO+X7z4NiENGrO3k2LWfyQn4cDXD1JFodnZmZSpyag6T35sNnwYXMdQui/2y/ek5OIEDpcdWJ8fPzll1/OyMggTvf6VCUCjE9zTk6V1J9GCLFzfuMPMzJKAAAgAElEQVSHB2cSIyALfKZbyaKKw8TgNNtrjzD6FT8PB7h6kjwJIi8vr7GxEXfN+tQmOKTlguJVJQghYmzrkpTYfVuzWR+eNl96SLs89LRJBK44iVt7hpyMZsv9PNwD5o9Q1x9mbOEvyZ3AXoABO1yj0fie/TV/hLr+0LopurIotbJoMd+CEKwqqWvvJ/7JNOyVdDi+QsUv3JMzBruPBNALy+jm7Rc9pHIAuBOEfdKVO35uWig0BFH5A+RNIsnJyVarVaVS3XXfHWgumyES7ljB6nuGuWgIQ9CnkPJJrzsLr4NMe2FJ61CZWha0Dre2k/8Bvp4kL4db25G1XTZ8Un3vBAtBOByOsrIyiUSCJyjo7+93uVytra1r1q0m7exyueRyOZ3Us0ajUciqZHPOjLDgJSmx/hzutX+ROgipAIDgmdYm53WrsdC9JSmxLAbbfh4OuD/Jlelx/giC3eE0qaysxDOzeuRy782BwNjYmM+0+Eaj0eeEcdCxvfYItoihbEUqU6cJ6fBAO00ChVDrEJuqoKiFeAXhcDjcXbYWi8WjWhGFKgmOaKjrEAsvDxvEmwC0MA9/S39nb7M/crkcVEkII9RYWNUOcVWHD1lUO5BqR319PUpByuJiOkd4s0EyMjLGxsaIftnq6mrqyHqACuFZ67BOJ3yhP+HCaGqGziIdUCUhCYx0whf6CoL+nhaLBUtQ4lOVFBQUgAhC7f0U1raJsOsJBAabzcYo5ys1dCLiMKtk/fr1hYWFxdjoCQQhlj5C6RgG2yTcSUpKkkgkDofX3uvuOqWApipBCDU0NFitVo1GAyIIGSAjATAdY2a1Wt3nYjQajdls5upEJF8JVPAKMcA2ARBCaGxszOO0rtFopD8OojBwMDy6XXU6XeG+cRAB2CZAqIG5UfDiocufWbx6xzJEbzUwVtuc0ekkEonZbK747k148nQRsI8pvLVJ3ovQOEkUFhbq9Xr8z6iYSPrHNjY2shhkIYTQGAhCJH2Esog6xJsAZDQazd+dFtn/M2M5MB3bxGMCN8y0wXm6YPPaBx6TyWTEXNZAcGwT0k/5beOA3wQgYzAYlq/KYXEgnbSyu57XqdVql8sFqiT0oDvSwdLDMF2IDXCLc3Kq55JD1IL49ZsVTxeE4Kxwz5DTOXFdGjUnnKvWRdB5TMQa5pBwIFgc7R4u+eAM9lkaGXFet5q/c6XMvwchZLfb//rntsc2raNzCKPoe1pp38TDGv1xPJ0a71kaFTki1iZEVYK16ZXpcdC3Aw+uSjAjhVdBPLN2m8PhUCgU9GtiKJVK9406nc7jzjKZrL6+vpjeMkJRGCYeP/OCgFOazvLnwQECacG8vAIJqsThcEgkEm+qASF08ODByspK4pb6+nqJRKJSqQ5XnThcdSI7qgD7gA1zampqjEYjnZxsAI/oHDP+BcA2iZdHDY5N4H9uzkkRxIMgZdCMW4TWVIW26KWREcQk0nxUICZCtEpiYmJcLldbW5uH4YzhYYRQa/1pffEDyPAenspUoVAQ93QPe6Wzzhjgso+sqUJxi3g9Ia2xLj4sPKRdLhQnE2lyS5HD0gI0uekgAedVwQtcnNet9jODrDcsFotMJvMZ/3rT8TFTEJJfO1nGHIhKEO7gBS4E5Fgk9RHNX1j6XEiiUeR4+x1aLTJU01gihMhpCoXdiHn1vGKqRK1WU1fqwqisrJRIJJWVlSSTw+X+PG+wfv16j5Gy0/aOqAThTijPTpBEk/eiN20C8SbAjf5gs9FUJQihpKQkl8ulUqno/763SFmXy4WnvwdETVAj67kOxQP8EUQSQl2PIlQRQ18Qubm56K8cnN9utzdumV947xwQBXmIIapc62CbAP5iNH9DZzeZTObtq5iYGFAlYJsEDzBkeMNqtSoUCvqC0CUnqw/2+9yRfugKINI+EgFPHFodCWwZMU2fSF5eXn9/PwgC+giMdAAPmM1mo9FoNBrp7Ew/bMRms0G4Gox0eEPAKw7CCzdB0FQljEhKSjIajT6riAI3kSWKq49AfhOAPR4TmlAPiyAENoTxoE10Td117dMjYRYxOc7JqUUVh7HPAoqdFeNw9YYg2K0Y5kQQarVapVJ5K+6ZnJzMyGnCaJ2xkHmwqgVbbrIkJZZFhfMDZwfL95tZdzHBMsu9CeKqBCGEN0f6EA8hrT8G6HOqbxQXBFEvBFgQRqPRZDJ5HPiUlZWZTCZGvxYaqkTf3IuvXDvVN6pv7mWq5XFVgggh+SEA2W9S195HunP/ewXkWGJBzbFebgXBGqPRiGeKxbNPI4SuXnb8/d0zdDI8hrZoao71MlqBSepinCGA7NNk24Tzng+qhB3ZCwT03DwWA701kXEqxtCoxUVq0kxbeAj3CB/aZO+mxUx/cXNOMv4ZnCbsO/DM1x2LhZcbsuK5FYTJZBocHCRttNls9H+BQVCcgCE5Spj6TZakxBLXf1dvVIZMo/XsFcOSXPqjRE/1jcbLo+PlUaAX/Bwn+vM2w/LIpt0Vw1X6AuJIB+P7Xy/W6/UKhYKOO9Zms5lMptBIueanaPzvYgIc6YT6DLG1HVnbZ2wR1Tp3oeGuTUh+k7KyMrVa7SGo5IYgVNr3TPotIAgB+U24yywT6pH11nbyKkxoxH6AZZ+moKamZvfu3UqlUiaTzQhFuSEI07ob+TJAENzCOs6Nu8wyUDkUYMAza7e1tbWZzebS0lJv+5SWlhK/TU5O1uv166B4Dt8IIPs0rNMBmKFSqShUiTv9/f12u71wx154dCFPcGwTYsQOFOgJIiwE4bHSBTXFxcXFSUPIe5JHgAQpXlEsfSQI2oQUO7io4jBV2DjJt5T3Ioy3uYJYoAchVFTbSTXZeUMQ0/nZQBB8Qgp9rmvvJwZe+OgjwcvtEMGJdsCiA1emx9GJT+n8cpSkhvm0y3eET6NnKoij3cPEP7EpTxAEH+BVH/ZuWsyiptqR7mEqbeIn3Gkff/0mR7uH8UDjo93DuqZun4c8vjiB+CcEy3ICSRAHzg76PKRsxYwAOWK0G8CHKnG3B71Big8iSUqw+KtNSE+HuGLQGxuy4vGGmzZfymIJJuBTEMR1Zd7QFqQSBRFKQZmCglSVkc4qQWK9pIq16WJ54/o70kmbL2VRwrJ6oxLaLreQBEEzlB4EERRJ0dmN79pJfMBBLCxxXuDkjnyOo+lp1xkDiILgvhggCMKPQSjRcuR+goYkmuD5p7iJrMeMt805KTyVswQYCYLvEsUAUwbHJg6cHZBGzuHRmRoatgkAAACCWFgAALgi1AYmfia1JY1y2UUHAGjmtKj/guDeHxfG+B+J7k24IWWb+J/UFs2caqUZHQC4uwmIE0z+C+LBqhZ4qly9bin+ZCFcom4KKW3CeVJbgB0Hzg6AIITaR/op/mQhXJYjnfL95sGxCWnUHBbpHQNDmITVlnxwxjlxHQQhQNO4pP40Qih7Qaxgp9WWpMQSl1CwkNTK9DhSnm3GtsmiisMHzg6e6hs92j0s2Jz9pIyb7DobcQ5PgPN5SS98crR7WPiCIP7pvyBEEWK3qOLwqb7RU32jNcd66cQiBwU/k9oihNLmS4k+LGKUHd0ZYlLD5W+JtP8uIu4zbgpMm4AghDmCIPkgAiAadmXbkN9Jbb0xS2giIf7JzveWNl8KKwn9hPRqZeeNBkHwreWdk1N0lnd6NB75EA1dbRKYINcjM9fI4xXVgABDaqOk3AUA9bgMBdBz9PGZAeE8BLra5LxuNbbedGV6HH8m3KqZwR0QYuCO7bVHsBAYXgVByk4AQTc0RYO12LIVqQFbGU/K7xFcAhpZT2cojgfGsB4TAiAIkUJM4Ejx2HHx0UyLFYLapHy/mWg/C+1BhA8lH5whjlw2ZMVDUgIBanmE0CHtcnFVywycFxaG4gKB9OTZufGAwEsKtAkMxQUH6clDAkexSEr4BNRvIoShuPAjegPAoorDWLR72nwpi3rpIAieIPpNguUK8CeiN7zym+C9CANK+YAgABJE3w1Tn1p45TeB5WcgCIACUvgoU58aZEsCAIAbQJsAADCNnxG94aVNbK89gk3gp82XwlgdBAF4FA22jGZzTjLTiF6xemFxX9HmnOSKtenQCIIuiOqNSphsFg49Q841+uPYZ+7LoYSSbULMDFjX3g8uvaCrEoRQ+X4zCEI44KoEsc2kyYLgZJkuqu3EMizEy6NO7shnejip1da190EFmaAIggQIgttXJtbORWR9B8E2wfJTYZ8HxyaKajv9/EFIosEOfXMvURB06uOCIAKsSjDrm5h7EbTJDEhJJVk8KVKMIDRiTgThLdknBcTQJmlkBAiCK0jWNwvREJcLBsyfFYSRTvaCWD91LZba41TfqDRqjrgWWQoK/xMOb8iK35AVD4IIQJdhesgh7XLn5FTPJUfaXTEBq+cbnDkdfutvA6wEATO1woE4IyMi0QRthpinPLcACAJEE3baBACAEAMi6wEAAG0CAACMdAAAAAAAAGCgAwAAwBdXrlz59ttv4TkAAJgmAAAAweT9999fvny51Wq9/fbb//nPf27evHnjxo2MfuHEiRN5eXnY56SkpMnJSaZHFRUVXb16FWQBgGkCAAAAoKtXr05OTmKWgVQqraur279/P0Jow4YN77//PrbPt99+m5WV5XK5NBpNU1NTRkbGokWL7r777uPHj1+5cmXXrl2XLl3Kzp5Oc/fmm28mJiYmJSU99thj+FlKSkruvffe3Nzc5cuXX758mXSUxWLBpuZPnjyZlpaWl5eXmZn57//+7yAdQKy4AAAAAP+4cuXKRx999NOf/hQh9OMf/xjbGBMTc/nyZZfLtW7duiNHjrhcLrVavXv3buzbtra2pUuXulwuk8mkUqmwjXfeeeeVK1ewz48//viePXtcLtc999zz+eefX758+fLlyyMjI/PmzZuYmCAedeutt46MjJw7dy4pKQm/pN7eXpALIFJCNAuAtd3HDoocsEqBMAJ6BG/8y7/8S2tr64MPPvjEE0888cQTe/fuTU9Pb2lpyc/P/+STT370ox898cQTycnJK1euxPafO3cu9kEikbj/mt1unzdvHvZZLpdjH7755puvv/76jjvuwP784osvIiMj3Y+dmpqaNeumI7yxsfHee+9du3YtyIgN9ovIfpFqB1kikiXCc+KJEDVNDA/72EHnANkDYQT0CN745ptvtm/fXlRUZLfbp6amlEplbW1tTk4OQignJ+ehhx7as2fPp59+SvELixYtslqtd9999z/+8Y+UlBR8+7Vr17C4WpvN9vTTT587d+7uu+++ePGiwWCIi4sjHiWVShFCDzzwgMFgSE1NVSgUV65ceeKJJ8AuYY/5I9T6KtUOeS8i1Q54TjwRoouHK2JAEQMA9Igg0tra+u233z733HMnTpy4/fbb4YGIDFMVmCZBBNI6AwAAcA+2fObzzz+HRwEATIEVOgAAAAAAgGkCAAAA0Ka0tPSdd96B5wCECSE6oSPGiXOY2gQC1SMyMzOtVqvJZMrIyIAewSH//Oc/Z82aFRUVRdzodDrnzJlD2ogQGhsbw9fgED/j2O12mUyGfZ6amlHdfmpqamJi4pZbbvF5doAlqh2i1LehElUGXhMACDu6urqsVqtarZbL5RaLBR6I/1RXV7/00ktz586Niop65pln3njjDYTQ7bfffu7cOalUGhUV9eijj9bV1SGEfvGLXxQUFHz33XdyuXz16tWPP/44Qkgul69Zs+aPf/wjQmjbtm2ZmZlTU1Mymayuri4rK4t4opdeemnLli0RERG33HLL+++/j61J9nh2ABAvYJoAQDgSExODGSiVlZUHDx4U2uXZbDbj4bMiep7l5eW33XZbdnb2rFmzrFbrQw89hBB6+eWXf/KTnyxcuPCxxx5TKBTXrl1DCLlcrvXr12MJSO655x6VSoX9gkKhcDqd2OedO3dGREQghDZv3vz1119brVb8RLt37+7r6ysoKCgoKPjP//zPqampgYEBj2cHADBNAAAQpYHS0NCwbt06IVxMWVmZXC6XSCRyubyyslKpuFVET3Lnzp0XL17s7Oz87rvvfvzjH7/00kvHjx+vrq7u6uq6cOGCXq//+OOPr1+/7n6gx/QNr7zyCvZhz549crlcoVDgX23ZskWhUDQ3Nzc3N5eUlAwODiYkJLifHdo2IGpCNK+JGIFYEyDYtLW1qVQqtVptMBj4doo0Njba7faKiopQ6hFOp/P69euxsbH4FrvdPnv2bFJQCDXbtm1LSEj4t3/7t5GRkdtuu83jPhMTE//85z9J37qfHQg7QiXWBPKaCAZZoo9k4ZAUGeCZ3Nxcl8t18OBBbKRuMpliYqY13fb3tn3a10Vx7Otb3ro/RentW4fDoVarGxsbEUIymUytVqvVah9BuDN7hP7gWfukq7JosZB7BJaVdcZN3Ihjpc8tt9yChbJ6s0sQQlFRUe7hru5nB4KJtT0IWZhDpeIEeE0AAPCAxWLRarVGozEpKYmFaXLw4EGFQiHoFUABZ2BgoKWlxWw2/+pXv6IwOwAwTQDwmgAA4IGMjIzW1laaO1+fnHpVV3Xq+P/FAjYLCwsLCwuJERLhycjISEtLS2dnZ3R0dH5+/oULF5RKpVqthtYFAMExTXqGnGv0x923x8ujDmlzpZFgEgHhRV17v66p2337yvS4vZsWi/3u5kRGvFixg2JCx3/a2trsdrtAIna9MT4+3tnZ2dLSghDKzs7Ozs4uKipau3at0WiMior62c9+Bh2BPs7JqTX6tsGxCfevKtamb85JhkcUwvAyoaNv7q051kuxw3ndarBOgPBB19Rd195PsYPttUcEfgt+xpr4D1aShr4jJ2C2iNlsbmlpGR0dzc7Ozs/PJ87UjI+Pv/vuuxs3bkxISIBewNQuWVRxmGKHzTnJFWvTBX0PMKHjB7zYB51fjlLv0HPJsSQFwsiBcKHnkpN6h1N9owLvESnz76HeYV7UvDCRZkdHR2dnZ29vL+YXWbp06dKlS91NFswoKS8vh/bPqss4/OxTwScyJmSCUkPENClbkVpU2+nt23h5FNglQFjx+OKEU31e7fUlKbHC7xHPrN1G2pKZmVlZWSnwGRZOuHDhQmdnZ2dnZ0ZGxtIbeNsZjBJOWJISGy+P8jibg/cpod9D3CKk+QuIkh08rtBxn1yXRkbsLX4A7BIgDHFOTun+/NmBs4PEjWnzpXuLH4iXi6DoicViKSwstNvteOWd3bt3a7Vak8mUm5sbYsLCl9IkJCRgfhE6R42MjNTX18P0DYec6hstqT/tnJxRP2hDVnzFo/dBSACYJgAAhC/19fVarZaU5gRDo9EYjUar1YotMBYvpKU02dnZ0dHRjEyZ/fv3P/vss4yOAgAATBMAANjQ1tZG4RfRaDQKhYIqqSt3l8HhCh33pTTsEo2AUQIAYJoAABCm+LlCh3opDRglAACmCQAAQsThcBQWFppMJqPRWFxcLHbThLSUZuHChZxcCRglAACmCQAAvOMe5coIm82mUCh4rQtIxzQhLaVRKjnOs9LR0dHb2wvpXAEg0KbJqb5Rj+t+y1akagtS+b6a4J4dAITWJgN2dovFolAoSFGuLKwTrVZbU1PDySV92Gz4sLmOYoenCzY/XaDBLAaz2axUKmkupQGjhG+KajvdV8tLIyNOvlgQgJU1wT17kAmVysMzTJOSD84c7R72tqs0MuK8bjV/l+Itk3dgzg4A7lD3CL5zzAf37Cxoa2tTqVSVlZWcRMXSN014BYwSRngrUYJzSLs8bb40WGcP/Qz3oWKazDAhKfQgQsg5OcVrzsojQT07XUxVqPVVqh0UOZBmJ2Sg7hHU3wr57NiiX24CSgg9Ihch1y4p+u5NVPHmzR3yXkSqHXR+yeFwmM1mk8lkMpnMZrPdbs9WK6V3ek4ye31y6scrNo/qnaWlpXwbJfw5Y8Kty2A78Gea+Dz7ke5hQZgmoWJABMg02ZAVT0oJRYTvLK6r0uMoMmYyOzsIHuAC6h6xMj1ONGef2SMMiciwS4r6tqGKbQHuETabDTM+KisridlQdDqdTCZTKpUVFRWY04XaazInMuL/HKv79h+zJBKJUql0z7niD0ajMTU1lTrrK+Cl0SZQF1Djtdf4PDuDHLI+R6G0bW6ABbOIf1RvVHr0tkkjI/ZuWnxyRz6vl7I5JzmIZwcAd6h7BN/zKcE9O4esX79eIpFIJJLMzEy9Xq9QKAwGAylLW0VFRWlpKdPEssXFxS6Xq7GxUaVSrV+/nhOjpKOjAzwlrImXR9lee6RsRapHo+S8bjV/LhM6Z9+QFQ8yEgUhukKHP68JTOgA0COCMbjMzMxUKpX8rQDCPSXQWADeGzb49X0BZQgAABABXV1djOwYzNqgsxAaM0og0BUAwDQRLconfdS5joyBhwSEbY9QLFmrvDe50fi7mzvIEoNix/hMH4eVCM7PzwejBAg04Fn3RYhO6FjbfexAbV4AQOj2CLvdLs9cW7zqfxn37Oa2R+Tl5ZnN5rGxMW6vHStxzFpT1dfXNzY2NjQ0kIwSKBEMUGG/iOwXqXaQJQbF7AbTBACAkOXgwYMIIa6q5aEbS3+VSiWHK2WmlZREwlWuFDBKAABME0DYmKp87KB8EoYFQHCpr69Xq9VWq5W0ooe1UbJp0yaz2Ww0Gj2YZT57BCwWBQAi1nYfcxSKHHYeWYg1CWOo48+xVgWmSRiAZXFlXQHH4XCYTCYOHTBE7HY7KQMKU7BqfBs3biwvL0c3qvBoNJrCwkKyM8ZnjwDTBABIponPdUysTJNZ8GwBIMzJzc11uVyFhYVyuTwzM9PhYLBw0WazyWSyxsZGnq6ttLSU9VTOwMBAdXV1bGxseXk5aQbHYDC4XC6FQsFJNhQAALgFTBMAABBCaN26dWNjY0ajsbCw0Gaz0TnE4XAolcrCwkL+0o2wY3x8HDdKoqOjve1WXFyMh8fabDar/TtoBgAgBGBCBwCAm2RkZGBTHh7pG/rvaxPX8D9zs/LiEu/UVVd82mfGtiy46565kfM4uRJsYY7ZbKaTm4REdHQ0Nn3jE4vFolarzWZzZWVlhQyGagDXTDrQ8Hkf+8CKUTBNAACgT1lZmdFoNJlMmH2wp+l3n/bdTH32g633IYS2v/cLfMvrW966P0XJyakrKysLCwtZ2CWMsNvtN5O5EasSAgAnDJ9Hhod97AM13dyAFToAAFBBzF3W/fVZomniDlemCX2XCXUVQITQ0wWbny7QYJ8PHjyo1+uNRqOf630AgC7WdiGaJoLPlA8OTAAAqIiJiWltbXW5XCaTqWbb7wNzUpVKxW4qxyMHDx6Uy+VyuRwh1NraCnYJAAgcmNABAIAWBoPh9uxbqL0mXMGVUXJ9curfy1+r+vVrnOeoBQBABKbJgbODuj9/5pycwrdIIyM2LE6oWJsOTxkIQ3RN3QfODJB6RMWj90FZdp/YbDauHBtzIiN+Wf3CuoJ18FSFT8+Qs3yfuWfISdyYNl9aXaRMmy+F5xNWcBNrskZ/nNSecOLlUYe0udJIcM8A4YJzcmqNvm1wbMLjt2nzpYe0y8V+jw6HQ6VSWa1WlUpFLE/jP2VlZXq93m6300x4Tz/WBEssq9frS0tLoZUKjbr2fl1Tt7dvK9amb85JhqfEGeEQa6Jv7vVmlyCEBscm6tr7oCUA4aRk+7zZJdjQUN/cK2qjJDMzU6FQGI3GsbExol1is9mSk5N1Oh2jpG2kH9fr9Vqt1qddsnv3brlcLpFIdv78JeL2qxcdl3tHPR5SXFzscrnMZrNEImlra4OGKigo7BKf3wKhBwfODJ+utnh5NDxoIHzw2eBF7Z1ubGw0Go0eY0GSkpL6+/vb2tqwFT0ymUyr1dKxM3AqKyvx/71ZJFqtVqFQVFZWegwfcTgcVquVIlTFYDDo9Xq1Wm2323lKrg+w6jVRFAZ9vDwKHhGnlqDQlytzM6HTM+Qs2tNBnFbHqd6ohMn1m/hcSKbIQZq/wHMSOwfODpbvN7tvl0ZG7HtmafhMnFssFoVCQWWamKqoa3BYk5+Q5L0Ia2pCHufkVNGeDo8O+LT50n3PLIWogJv4nI7R/EXsady4EXbafOl53WpMIw+OjWMDx5XpcdCYQqGVQzog5mzIit+QFe+cnDraPYz3CDHa6BaLRaVSKRQKk8lE3/mBQ/JeHDx4sLCwECGExXzQ+UGr/TsVz3ZJZmamUqmczrUvzCwUYYA0MgKLweoZch7tHsY2rkyPgwBYEWB42Ef94bwXmZbG5Nh0AAcJABC1rXh7BG6UWK1WFkaJR9atW4e5aS0WCzbpE/M9ZH+B6t2jUqYwPYtGozGZTP39/TT37+rqKisrk0gkRqOxOHcBtNvgkjZfCuYIEK5eDWoTD0FRAyCse4TJZNLr9f2t/0cmk6Gx82iM4x5xs1KPrwkdFqhUKqPRyGgFck1NTU1NTV5eXulWk6l4rjJuNrQIwAf2i8h+kWoHWSKSJcJzAtOECeCzBQDvPUKFkGodQg1PibFHqFQqzLoqLi5mdGBrayutCR0AQAiZP/JhVTOfxQDC3jQJFnGLfES5RsbAQwLCCOWTSJFj/GOD5sU3uxreUiqVHoaeDElKSjKbzQqFAp4uEJr4XCoRtwhME4AJkTGCmyrKexHEAvhEJpOZTCYPpoO/v5tYf7BV8+KbRqNRWVjM1a+yz3MvS4QeAQgdob1EMp9Cycu4vWAwTcIecDkCNLBarYWFhW1tbQaDQa1Wc/jLxcXFTGde+DTBEqFHAAAzlE9y/pNQeRgAABqvbJnMZDJh9YdlMpnAr9Zms0kkkvr6ehAcAIgRblKuAQAAMKW+vl4mk/GUkjU5OVmlUk1nKwEAQFQwmNDpGXKe+nLUOXkdISSNnLNkQWyAV5+f6hs91TddHQNSugFBh5gbSho5Z2V6XIDTaR/tHsazZ6bNly5ZcFuAe4TdbpfL5eymeLBie42NjTxdm0qlMplM0ONMLIsAAAtWSURBVEoFhXNy6tSXI8RGuzI9LpAXMDg2cbR7GHuLIUjpJmBoeU28Zd1GgcpDX77ffODsoPt2SGAMBIWg56EXVI9Qq9X19fWMDBSbzaZQKLRabU1NDU9X5bOYDhBgo8RbHvoNWfHVG5V8XwBUUwk104TCLgmMXL1pYZyTO/Kh+BMQMHw2SL57xINVLdSF0E7uyA/8Y8EMlIaGBiwVPUbf0H9fm7jmvnNuVl5c4p37Gv54pzzuTnkcNKqQJ7iN1udbLDDmEV0gIyidCR2sAog/O/gJRYPGLwBMEyBg0GmQQbwAn5fHE0aj0Wg0kjbuafrdp31d7jv/YOt9CKHt7/3i6YLNTxdooFGFea/hu9HSeItNCOhhQUZQOit0tAWpFC/+eHmUtiCV10ukNmY35yQvSYmFng8EjOqNSooZk7T5Ur57xN5Ni1n3l8AN/KxWvV4f3GvQaDTJycnQYgXSa1g3af/RFqRSTLNKIyME5DIBEEL0V+gMjk2U1J8mzhSmzZfuLX4gYO6KU32jJfWniTOFwnLBAWFGz5CzpP40cbAV4DgPdx91xdr0zTkCehNrtdr/O9oWc/c8bzs4L1/rNJoRQkqlsrCwsLCwkNvQkLa2NiwYNjc3F1qsEKhr79c1dZNMloDFeTgnp0rqT+NrKVAAg8OYAdXgYfEwAAD8sf29bR4ndHDwCR2bzdbY2NjY2KjVajlcTiyRSCorKysqKkAWgGgA0wSywYoen4VbocQUIAaSkpJKS0tLS0tJ2zMzM81mM0JIoVBgnhUfLpCZPaJ/2zx09beo4k3oEQAgImMITBMAAPji9S2/wz6UlZVptdqkpCSmv9DVddPpYrPZTCaT2Wx2t2C8oZBBwmtAbEDde0hUDwAA32g0Gr1e7396+6SkpOLiYpJdYrFYMjMzJTco3LEXHjgAiJ3w8prUtfe/396Hhy5KIyNWpsdVPHofJG0DwhN9c+/HZwaIPWLD4oSyFQs57BE6nc5oNJpMppiYGD5uISMjg+hZ8T3FCQB+0DPkrDnWi6eBRgilzZeWrUgNcFpbME1CBOfk1IOvNpNSATonpw6cHTxwdpD90gZru4/0OIqccEiPA4hRw67RH3fvJnXt/XXt/fu2ZrNck2/+CNkv3rQTzH2VlXsNpStyXZ3I1Ak9AhA1JR+cIRoleFcq+eBMvDzqkDaXpU1vqvKxQ/hFRwlohY5zcqrn0vQcW7w8mttlyUW1ncQ1Y+6w1MVBj0IF2yik7Wm8R6TdFcOtby/phU+od7C99gib3zU87KNBQo8AeLa5nRPXEULSqDncrgrWN/fWHOul2GFJSuy+rdlsfjroUajCs40E4TVxX+yOsTI9jqtUPNKoOdQ7xMujRdkRQc+GIgHoEfHyKIoMmCKe4oQeEa52/Bp9m8cmfUi7nBMbxec7QsRJyYXnlQl+GOwa/XGPWhghdLR7OOmFTzwWZGLK3k2LvbVOaWTEIe1ySHUPCATqHuE+C8OOQ9pcyh4BOcoA0XC0e3hRxWFvpvYa/XH3WRgWUCf5hBSg3BL8sZHHWpQzdrjk4CQV/SHtcoTQgbODR7qHMY9f9oLYzTkpEAMLCArqHuGzv9AEs8ixHvHxmQFs46r0uA2LE6BHAKHUZbAdOAlT3ZAVvyErfnBsoubYBcwSkkbN+QkUSwlJ06RsRSrFBN6SlFhupY61LRA8IFioe0TZCo4L9ECPAMTO5pyU90/0efOvSyMjNuekcHi6eHkUOEj4RhBhsM7JKd2fPyMVmg9wjR4AEA4ee8TK9LiKtenQIwDAI+511qSRERWP3geWN5gmAAAAAAAAfgHZYAEAAAAAEBAQ7yYCeoacxAhzzuNvaF7Dx2cGei5Nh5ul3SV9fHGC4IqJA+HBqb5RYpoigfQIbrPoAgCHOCenDpwZcE5ex/6URs4JSsB7XXt/Z98onnjm8ax4b+HJMKEjaI52D5d8cMbjV2UrUrUFqQG4hsGxiTX6No8hZtgqU4h+AAIGRdorIfSItPnSfc8sBQMFEJRR4p4JHVfgJ18sCExz1TV117X3e/yqeqPSPR4IJnSEy4Gzg97sEoRQzbFeUpgkT3jTwuhGmiOQFBCwHkGxdkkIPaJnyPngq80gKUA4CEGBY+UvvH1bvt/s/i2YJsJlcGzczx0AAHoEAIR3r5lg/W0Q4X5C51TfaFFtp/v2gLlbQ8vgPe4tm1C8POrkjvwAXIM3gWKwrwMXNnibg4AewYIHq1q8KVOB9Aiu0qKHOR6rngVyAiJkOHB2sHy/2du3HidTAiZQDI/1Nzg2Tcr3myl8qhxWAAkkPUPO90/03SwcHzVnVXpcwNbKOyenivZ0EA2UoGR86Rlylu8z45eRNl9aXaQEFewTj6VKxd4jTvWNfnxmgNgjKMLZ+BgFltSfhh4RwqP8B6taKHYQo/GHRaEeIaiCtLukP8lJCVijdS/LVbE2fXNOcoCfg66p+8CZAWyCSRoZsWFxQsXa9EB4TfgqZyrIURo4DIBw6xEUIXUIoZM78iEmGvATnwV+RedupLgj9sWKQx2OY02ofQkBG1dxOOSlmIorqu3kqqAJEKpQt3nR9QgKuwQhBDHRABcvkQR/+pTQqGvvp7C0TvWNUqx1ANOEM6o3Kj1626SREXs3LRad7xpbfg0ArMGavfvsuEh7hI/+wkWRcCDMiZdH2V57xGOtqJXpced1q2HWLBwI5bwm/s+IU7uv925aLLpRLxDOHO0e/vjsIG5wx8ujHl+cwGhSkiKlB4IpTiDkGBybOHB2oPPLm/Gb2QtiN2QlMJq4pEjpkTZfihUAB8LCNOF2RhyrtXa0exj7wSUpsRWPpoPlDohLw3oLLWSx6oFUfXBlelzZilToEUCIQbGqY0NWPNPiw0e7h3VN3dhQWRoZ8ZNlKbBAL+xMk0UVhyl8y9LIiPO61SB7IHyAHgEAjHBf0kIiKCtcwodwTLkGM+IAtHkAACi7zHU/dwDANCFzSJtL4aCGxVpAuEHR5rFCSPCIAICItiCVInBqSUosTMfwSiiHwcKMOGTmBYj0DDlrjvXiKeA2ZMVXPHpfWOXWhB4BMEXf3Isvp8Aix8OtqQQlMy9UHg5ZQjIzLwBAjwCAwBDEzLxQ3i9koa7CSpE9HQCgRwAAcODsAPUO/PUaME1ClhDLQwoA0CMAQDhdhtdeAxM6IT5M1P35M/fVGZtzkr0VVQIA6BEAAGA4J6dK6k+7x5rEy6P2Fj/AX+wmmCaBpmfIeerLUWzhmTRyzpIFsZCrCgjzHoG7haWRc1amx0GNQACgNhdOfTlCrHodej4/ME0COmIr32/2+FX1RiV1ZUQACJ8eIY2M2PfMUjDZAcDdKCna0+GxrCyLBLVgmgiIU32juG8qXh69Mj0uMIsnKewSsE6AIHK0e5g4/Fqy4LbA9Ajq9TLQIwDBMjg2cbR7GE+5tjI9LmBm9INVLXhVOHfi5VEnd+SHxkMOo5QG3lRh2nzpvmeW8q2OB8fG/dwBAEKrR0xAjwDERc+Qs2hPBylWqeZYb8Asaepe47NPiYhwWaFDMUTrGXIuqjjMt1C1BakUM+jx8ijI+AQEkgerWih6xBp9G98XUL1RSWH9pM2XQo8ABMWBs4Nr9Me91Xwo32+m9otz1Wsovg2lxDzh4jWhM0TjO/ju5I78wbGJkvrTxJnCtPnSvcUPQNwfIKgeEYDhV7w86rxudc+Qs6T+NPF0gfHZAADzLjPuT5/ihA1Z8Ruy4t1LD4be7Ge4xJpQZ7WDpYNAuHG0e7jkgzMUgzOI8wAAEmv0xz2GoKIbtahgkAmmCRtO9Y2W1J8meuRCLKoZABjhHp0Npd4BgAL3PB+woAxMEwAAAAAAQhxIVA8AAAAAAJgmAAAAAAAAnvj/AUUdpJDvk2gFAAAAAElFTkSuQmCC"
}
},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 13.\tOver-sampling : SMOTE\n",
"\n",
"\n",
"\n",
"In the context of synthetic data generation, there is a powerful and widely used method known as **synthetic minority oversampling technique** or **SMOTE**. Under this technique, artificial data is created based on feature space. \n",
"Artificial data is generated with bootstrapping and k-nearest neighbours algorithm. It works as follows:-\n",
"\n",
"\n",
"1.\tFirst of all, we take the difference between the feature vector (sample) under consideration and its nearest neighbour.\n",
"\n",
"\n",
"2.\tThen we multiply this difference by a random number between 0 and 1.\n",
"\n",
"\n",
"3.\tThen we add this number to the feature vector under consideration.\n",
"\n",
"\n",
"4.\tThus we select a random point along the line segment between two specific features.\n",
"\n",
"\n",
"The concept of **SMOTE** can best be illustrated with the following figure:-\n",
"\n",
"\n",
"![smote.png](attachment:smote.png)\n",
"\n",
"\n",
"So, **SMOTE** generates new observations by interpolation between existing observations in the dataset.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"from imblearn.over_sampling import SMOTE"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"smote = SMOTE(ratio='minority')"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [],
"source": [
"X_sm, y_sm = smote.fit_sample(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"283823 New points created under SMOTE\n"
]
}
],
"source": [
"print(X_sm.shape[0] - X.shape[0], 'New points created under SMOTE')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 14. Conclusion\n",
"\n",
"\n",
"In this jupyter notebook, I have discussed various approaches to deal with the problem of imbalanced classes. These are `random oversampling`, `random undersampling`, `tree-based algorithms`, `resampling with imbalanced learn library`, `under-sampling : Tomek links`, `under-sampling : Cluster Centroids` and `over-sampling : SMOTE`.\n",
"\n",
"\n",
"Some combination of these approaches will help us to create a better classifier. Simple sampling techniques may handle slight imbalance whereas more advanced methods like ensemble methods are required for extreme imbalances. The most effective technique will vary according to the dataset.\n",
"\n",
"\n",
"So, based on the above discussion, we can conclude that there is no one solution to deal with the imbalanced classes problem. \n",
"We should try out multiple methods to select the best-suited sampling techniques for the dataset in hand. The most effective technique will vary according to the characteristics of the dataset.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment