Skip to content

Instantly share code, notes, and snippets.

@apetenchea

apetenchea/practical.ipynb

Created March 20, 2019 01:05
Show Gist options
  • Save apetenchea/0ef0e5fe6af48b8ea0e1913ce256dc2d to your computer and use it in GitHub Desktop.
Save apetenchea/0ef0e5fe6af48b8ea0e1913ce256dc2d to your computer and use it in GitHub Desktop.
Download ZIP
For meetup
Display the source blob
Display the rendered blob
Raw
practical.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
rf_xgb.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Trees"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./media/nice_tree.jpg\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Binary Search Tree\n",
"\n",
"<img src=\"./media/bstree.jpg\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## YES/NO `questions`\n",
"\n",
"A node asks the following question: `Is the number you're looking for greater than me?`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Simple questions\n",
"- Space partitioning\n",
"<img src=\"./media/kdtree.png\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Decision Trees\n",
"Start with observations, reach a conclusion.\n",
"<img src=\"./media/decision_tree.jpg\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Further reading:\n",
"- KD Trees: same concept, generalized to K Dimensions\n",
"- K Nearest Neighbors - supervized learning using KD Trees\n",
"- K Means - unsupervized learning, used for clustering \n",
"\n",
"Video introduction to identification trees:\n",
"- https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/lecture-11-learning-identification-trees-disorder/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Regressing using trees - predicting a real number\n",
"\n",
"https://www.kaggle.com/c/house-prices-advanced-regression-techniques"
]
},
{
"cell_type": "code",
"execution_count": 325,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"from IPython.display import Image\n",
"from sklearn.metrics import mean_squared_error, mean_squared_log_error\n",
"plt.rcParams['figure.figsize'] = 16, 9"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1460, 81)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_csv('./data/train.csv')\n",
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simplest model - predicting the mean"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"180921.19589041095"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.SalePrice.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding more information"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>YrSold</th>\n",
" <th>SalePrice</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2008</td>\n",
" <td>208500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2007</td>\n",
" <td>181500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2008</td>\n",
" <td>223500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006</td>\n",
" <td>140000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2008</td>\n",
" <td>250000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" YrSold SalePrice\n",
"0 2008 208500\n",
"1 2007 181500\n",
"2 2008 223500\n",
"3 2006 140000\n",
"4 2008 250000"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df[['YrSold', 'SalePrice']].head()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0x16484dc9da0>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA8kAAAIaCAYAAAATEQ+sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X+U3Wd9H/j3M9bwu9Sg65jYohVbiSS0x1CiJW5aCA7M2EMDZnvKluwP7lJaezlUbg/tNqSHhYQ45yTdbrOVNmXNBpbr3W4pSZuDw/HEGhxTSJakiEBEgMSapAoMBlsj2wT/IIw8z/4xj8xIzIxkWXe+9169Xufcc+/3ud8fnzFfRvP+Pt/v85RaawAAAIBkqusCAAAAYFQIyQAAANAIyQAAANAIyQAAANAIyQAAANAIyQAAANAIyQAAANAIyQAAANAIyQAAANAIyQAAANDs6LqAUdHr9eru3bu7LgMAAIAh+MxnPrNca73sbOsJyc3u3btz+PDhrssAAABgCEopf3Iu67ndGgAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGeAJWF5ezv79+3PixImuSwEAYAiEZIAnYDAY5MiRIxkMBl2XAgDAEAjJAOdoeXk58/PzqbVmfn5ebzIAwAQSkgHO0WAwSK01SbK6uqo3GQBgAgnJAOdoYWEhKysrSZKVlZUcOnSo44oAALjQhGSAczQzM5Pp6ekkyfT0dGZnZzuuCACAC01IBjhH/X7/8c+llNOWAQCYDEIywDnq9Xq58sorkyRXXHFFdu7c2XFFAABcaEIywDlaXl7OPffckyS55557jG4NADCBhGSAc7R+dOtaq9GtAQAmkJAMcI6Mbg0AMPmEZIBzZHRrAIDJJyQDnKN+v59SSpJkamrK6NYAABNISAY4R71eL3NzcymlZG5uzujWAAATaEfXBQCMk36/n2PHjulFBgCYUEIywBPQ6/Vy8ODBrssAAGBI3G4NAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAzdBCcinl+0opn1v3+tNSyj8qpTy3lLJQSjna3p/T1i+llAOllMVSypFSykvX7avf1j9aSumva//BUsrn2zYHSimltW94DAAAANjK0EJyrfUPa60vqbW+JMkPJnkkya8meUeSO2ute5Pc2ZaTZC7J3va6Icl7k7XAm+TdSX4oycuSvHtd6H1vW/fUdte19s2OAQAAAJvartutX5Xkj2qtf5Lk+iSD1j5I8vr2+fokt9Y1v53k0lLK9ya5NslCrfX+WusDSRaSXNe+e3at9VO11prk1jP2tdExAAAAYFPbFZLfmOTfts+X11q/liTt/Xta+5VJvrJum6XWtlX70gbtWx0DAAAANjX0kFxKeUqS1yX55bOtukFbPY/2J1LbDaWUw6WUw8ePH38imwIAADCBtqMneS7J79Za723L97ZbpdPe72vtS0mev267XUnuOUv7rg3atzrGaWqt76u17qu17rvsssvO88cDAABgUmxHSP7xfOdW6yS5LcmpEar7ST6yrv1NbZTrq5N8o90qfUeS2VLKc9qAXbNJ7mjffbOUcnUb1fpNZ+xro2MAAADApnYMc+ellGckmUly47rmn0vy4VLKW5J8OckbWvvtSV6TZDFrI2G/OUlqrfeXUn4myafbeu+ptd7fPr81yQeTPD3JfHttdQwAAADYVFkbGJp9+/bVw4cPd10GAAAAQ1BK+Uytdd/Z1tuu0a0BAABg5AnJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJAAAA0AjJbIvl5eXs378/J06c6LoUAACATQnJbIvBYJAjR45kMBh0XQoAAMCmhGSGbnl5OfPz86m1Zn5+Xm8yAAAwsoRkhm4wGKTWmiRZXV3VmwwAAIwsIZmhW1hYyMrKSpJkZWUlhw4d6rgiAACAjQnJDN3MzEymp6eTJNPT05mdne24IgAAgI0JyQxdv99PKSVJMjU1lX6/33FFAAAAGxOSGbper5e5ubmUUjI3N5edO3d2XRIAAMCGhGS2Rb/fz1VXXaUXGQAA1lleXs7+/fvNADNChGS2Ra/Xy8GDB/UiAwDAOoPBIEeOHDEDzAgRkgEAADqwvLyc+fn51FozPz+vN3lECMkAAAAdGAwGqbUmSVZXV/UmjwghGQAAoAMLCwtZWVlJkqysrOTQoUMdV0QiJAMAAHRiZmYm09PTSZLp6enMzs52XBGJkAwAANCJfr+fUkqSZGpqykwwI0JIBgAA6ECv18vc3FxKKZmbmzMTzIgYakgupVxaSvmVUsoflFK+VEr5a6WU55ZSFkopR9v7c9q6pZRyoJSyWEo5Ukp56br99Nv6R0sp/XXtP1hK+Xzb5kBpl2E2OwYAAMAo6ff7ueqqq/Qij5Bh9yT/qyS/Xmv9/iQvTvKlJO9IcmetdW+SO9tykswl2dteNyR5b7IWeJO8O8kPJXlZknevC73vbeue2u661r7ZMQAAAEZGr9fLwYMH9SKPkKGF5FLKs5O8Isn7k6TW+u1a64NJrk9yamzzQZLXt8/XJ7m1rvntJJeWUr43ybVJFmqt99daH0iykOS69t2za62fqmvjpt96xr42OgYAAABsapg9yf9FkuNJ/q9SymdLKb9USnlmkstrrV9Lkvb+PW39K5N8Zd32S61tq/alDdqzxTEAAABgU8MMyTuSvDTJe2utfzXJw9n6tueyQVs9j/ZzVkq5oZRyuJRy+Pjx409kUwAAACbQMEPyUpKlWuvvtOVfyVpovrfdKp32ft+69Z+/bvtdSe45S/uuDdqzxTFOU2t9X611X61132WXXXZePyQAAACTY2ghudb69SRfKaV8X2t6VZIvJrktyamh2/pJPtI+35bkTW2U66uTfKPdKn1HktlSynPagF2zSe5o332zlHJ1G9X6TWfsa6NjAAAAwKZ2DHn/+5P8m1LKU5L8cZI3Zy2Yf7iU8pYkX07yhrbu7Ulek2QxySNt3dRa7y+l/EyST7f13lNrvb99fmuSDyZ5epL59kqSn9vkGAAAALCpsjYwNPv27auHDx/uugwAAACGoJTymVrrvrOtN+x5kgEAAGBsCMkAAIyl5eXl7N+/PydOnOi6FGCCCMkAAIylwWCQI0eOZDAYdF0KMEGEZAAAxs7y8nLm5+dTa838/LzeZOCCEZIBABg7g8EgpwagXV1d1ZsMXDBCMgAAY2dhYSErKytJkpWVlRw6dKjjioBJISQDADB2ZmZmMj09nSSZnp7O7OxsxxUBk0JIBgBg7PT7/ZRSkiRTU1Pp9/sdVwRMCiEZAICx0+v1Mjc3l1JK5ubmsnPnzq5LAibEjq4LAACA89Hv93Ps2DG9yMAFJSQDADCWer1eDh482HUZwIRxuzUAAAA0QjIAAAA0QjIAAAA0QjIAAAA0QjIAAAA0QjIAAAA0QjLbYnl5Ofv378+JEye6LgUAAGBTQjLbYjAY5MiRIxkMBl2XAgBMCBfhgWEQkhm65eXlzM/Pp9aa+fl5/5ABABeEi/DAMAjJDN1gMEitNUmyurrqHzIA4ElzER4YFiGZoVtYWMjKykqSZGVlJYcOHeq4IgBg3LkIDwyLkMzQzczMZHp6OkkyPT2d2dnZjisCAMadi/DAsAjJDF2/308pJUkyNTWVfr/fcUUAwLhzER4YFiGZoev1epmbm0spJXNzc9m5c2fXJQEAY85FeGBYhGS2Rb/fz1VXXeUfMADggnARHhiWHV0XwMWh1+vl4MGDXZcBAEyQfr+fY8eOuQgPXFBCMgAAY8lFeGAY3G4NAAAAjZAMAADQkeXl5ezfvz8nTpzouhQaIRkAAKAjg8EgR44cyWAw6LoUGiEZAACgA8vLy5mfn0+tNfPz83qTR4SQDAAA0IHBYJBaa5JkdXVVb/KIEJIBAAA6sLCwkJWVlSTJyspKDh061HFFJEIyAABAJ2ZmZjI9PZ0kmZ6ezuzsbMcVkQjJAAAAnej3+ymlJEmmpqbS7/c7rohESAYAAOhEr9fL3NxcSimZm5vLzp07uy6JJDu6LgAAAOBi1e/3c+zYMb3II0RIBgAA6Eiv18vBgwe7LoN13G4NAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAAAAjZAMAMBYWl5ezv79+3PixImuSwEmyFBDcinlWCnl86WUz5VSDre255ZSFkopR9v7c1p7KaUcKKUsllKOlFJeum4//bb+0VJKf137D7b9L7Zty1bHAABgcgwGgxw5ciSDwaDrUoAJsh09ydfUWl9Sa93Xlt+R5M5a694kd7blJJlLsre9bkjy3mQt8CZ5d5IfSvKyJO9eF3rf29Y9td11ZzkGAAATYHl5OfPz86m1Zn5+Xm8ycMF0cbv19UlOXe4bJHn9uvZb65rfTnJpKeV7k1ybZKHWen+t9YEkC0mua989u9b6qVprTXLrGfva6BgAAEyAwWCQtT8Bk9XVVb3JwAUz7JBckxwqpXymlHJDa7u81vq1JGnv39Par0zylXXbLrW2rdqXNmjf6hgAAEyAhYWFrKysJElWVlZy6NChjisCJsWwQ/Jfr7W+NGu3Ur+tlPKKLdYtG7TV82g/Z6WUG0oph0sph48fP/5ENgUAoEMzMzOZnp5OkkxPT2d2drbjioBJMdSQXGu9p73fl+RXs/ZM8b3tVum09/va6ktJnr9u811J7jlL+64N2rPFMc6s73211n211n2XXXbZ+f6YAABss36/nzZma6amptLv98+yBcC5GVpILqU8s5Ty5059TjKb5PeT3Jbk1G+xfpKPtM+3JXlTG+X66iTfaLdK35FktpTynDZg12ySO9p33yylXN1GtX7TGfva6BgAAEyAXq+Xubm5lFIyNzeXnTt3dl0SMCF2DHHflyf51XaFb0eS/7fW+uullE8n+XAp5S1JvpzkDW3925O8JslikkeSvDlJaq33l1J+Jsmn23rvqbXe3z6/NckHkzw9yXx7JcnPbXIMAAAmRL/fz7Fjx/QiAxdUOTUq4MVu37599fDhw12XAQAAwBCUUj6zbmriTXUxBRQAAACMJCEZAICxtLy8nP379+fEiRNdlwJMECEZAICxNBgMcuTIkQwGg65LASaIkAwAwNhZXl7O/Px8aq2Zn5/XmwxcMEIyAABjZzAY5NQAtKurq3qTgQtGSAYAYOwsLCxkZWUlSbKyspJDhw51XBEwKYRkAADGzszMTKanp5Mk09PTmZ2d7bgiYFIIyWwLo08CABdSv99PKSVJMjU1lX6/33FFwKQQktkWRp8EAC6kXq+Xubm5lFIyNzeXnTt3dl0SMCGEZIbO6JMAwDD0+/1cddVVepGBC0pIZuiMPgkADEOv18vBgwf1IgMXlJDM0Bl9EgAAGBdCMkNn9EkmiUHoAAAmm5DM0K1/TqiU4rkhxppB6AAAJpuQzND1er1ceeWVSZIrrrjCc0OMLYPQAQBMPiGZoVteXs4999yTJLnnnnsEC8aWQegAACafkMzQrQ8WtVbBgrFlEDoAgMknJDN0ggWTwiB0AACTT0hm6AQLJkW/308pJUkyNTVlEDoAgAkkJDN0ggWTotfrZW5uLqWUzM3NGYQOAGACCckMnWDBJOn3+7nqqqtc7AEAmFBCMttCsGBS9Hq9HDx40MUegBGwvLyc/fv3mzkDuKCEZLaFYAEAXGiDwSBHjhwxcwZwQQnJAACMneXl5czPz6fWmvn5eb3JwAUjJAPARcYtqkyCwWCQWmuSZHV1VW8ycMEIyQBwkXGLKpNgYWEhKysrSZKVlZUcOnSo44qASSEkA8BFxC2qTIqZmZlMT08nSaanpzM7O9txRcCkEJIB4CLiFlUmRb/fTyklSTI1NWUGDeCCEZIBngDPcjLu3KLKpOj1epmbm0spJXNzc2bQAC4YIZltIVgwKTzLybhziyqTpN/v56qrrtKLDFxQQjLb4pZbbsnv/d7v5ZZbbum6FDhvnuVkErhFlUnS6/Vy8OBBvcjABSUkM3TLy8tZWFhIkhw6dEiwYGx5lpNJ4BZVANiakMzQ3XLLLVldXU2yFiz0JjOuPMvJpHCLKgBsTkhm6D72sY+dtnyqVxnGjWc5mRRuUQWAzQnJDN2pZ982W4Zx4VlOAIDJJyQzdK961atOW371q1/dUSXw5PR6vVxzzTVJkmuuuUYvHADABBKSGbobb7wxU1Nrp9rU1FRuvPHGjisCAADYmJDM0PV6vczMzCRJZmdn9b4xtpaXl3PXXXclSe666y4jtTO2zF0PAJsTktkWN954Y1784hfrRWasDQaDx0dqf+yxx0wBxdgaDAY5cuSIcxgANnDOIbmU8jdKKW9uny8rpbxgeGUBjJ6FhYWcPHkySXLy5ElTQDGWlpeXMz8/n1pr5ufn9SYDwBnOKSSXUt6d5CeS/GRrmk7y/wyrKCaPXgsmwctf/vLTll/xild0VAmcv8FgkFprkrW56/1eBoDTnWtP8n+V5HVJHk6SWus9Sf7csIpisui1ABgdCwsLWVlZSZKsrKy4IwIAznCuIfnbde2yc02SUsozh1cSk0avBZPik5/85GnLn/jEJzqqBM7fzMxMpqenkyTT09OZnZ3tuCIAGC3nGpI/XEq5JcmlpZS/n+RjSf7P4ZXFJNFrwaSYmZnJjh07kiQ7duwQLhhL/X4/pZQka9Py9fv9jisCgNFyTiG51vovkvxKkn+f5PuSvKvWenCYhTE59FowKfr9/mlzfgsXjKNer5e5ubmUUjI3N2daPgA4w7kO3PWCJJ+stf5PtdZ/kuQ3Sym7h1kYk0OvBZOi1+vliiuuSJJcccUVwgVj67WvfW2e8Yxn5HWve13XpQDAyDnX261/OcnquuXHWhuclV4LJsXy8nK++tWvJknuueceg9Axtn7t134tjzzySG677bauSwGAkXOuIXlHrfXbpxba56cMpyQmUb/fz1VXXaUXmbG2ftC5WqtB6BhLZhwAGC3Ly8vZv3+/38cj5FxD8vFSyuP3ZJVSrk+yPJySmES9Xi8HDx7Ui8xYMwgdk8CMAwCjZTAY5MiRI34fj5BzDcn/Y5J/Vkr5cinlK0l+IsmNwysLYPQYhI5J4GIPwOhwd89oOtfRrf+o1np1khcleVGt9YdrrYvDLQ1gtBiEjkngYg/A6HB3z2jaMiSXUv679v72Usrbk9yQ5O+vWwa4aBiEjkngYg+T5O67787c3FwWF/XdMJ7c3TOaztaT/Mz2/uc2eQFcVAxCx7hzsYdJ8lM/9VN5+OGH8653vavrUuC8uLtnNG0Zkmutt5RSLknyp7XWnz7zdS4HKKVcUkr5bCnlo235BaWU3ymlHC2l/LtSylNa+1Pb8mL7fve6ffxka//DUsq169qva22LpZR3rGvf8BgAT5ZB6JgELvYwCe6+++4sLS0lSZaWlvQmM5bc3TOazvpMcq31sSSvO9t6W/iHSb60bvnnk/xCrXVvkgeSvKW1vyXJA7XWPUl+oa2XUsqLkrwxyV9Ocl2Sf92C9yVJfjHJXNaelf7xtu5WxwCAi56LPUyCn/qpnzptWW8y48jdPaPpXEe3/v9KKf97KeXlpZSXnnqdbaNSyq4kfzPJL7XlkuRHk/xKW2WQ5PXt8/VtOe37V7X1r0/yoVrrn9Va/3OSxSQva6/FWusft3mbP5Tk+rMcAwCACXCqF3mzZRgX7u4ZPecakn84az2570nyv7bXvziH7f63JP80yWpb3pnkwVrryba8lOTK9vnKJF9Jkvb9N9r6j7efsc1m7VsdA+BJWV5ezv79+03RwFhzHjMJTt2iutkyjIv7778/i4uLeeCBB7ouheZcp4C6ZoPXj261TSnlx5LcV2v9zPrmjXZ/lu8uVPtGNd5QSjlcSjl8/PjxjVYBOM1gMMiRI0dM0cBYcx4zCX7kR37ktOVXvvKV3RQCT9LNN9+chx9+OO95z3u6LoXmbFNA/VAp5fdKKQ+VUj5VSvmBJ7Dvv57kdaWUY1m7FfpHs9azfGkpZUdbZ1eSe9rnpSTPb8fdkeTPJ7l/ffsZ22zWvrzFMU5Ta31frXVfrXXfZZdd9gR+NOBitLy8nPn5+dRac/vtt+uFYyw5j5kUN9100+O9x6WU3HTTTR1XBE/c3XffnWPHjiVJjh07ZgC6EXG2nuRfTPJPsnYL87/MWsg9J7XWn6y17qq17s7awFu/UWv9b5PcleRvt9X6ST7SPt/WltO+/426NrP2bUne2Ea/fkGSvUn+U5JPJ9nbRrJ+SjvGbW2bzY4BcN4Gg8FpcxnqhWMcOY+ZFL1e7/He5Fe+8pUGPGIs3Xzzzact600eDWcLyVO11oU2aNYvJ7kQ3a0/keTtpZTFrIXv97f29yfZ2drfnuQdSVJr/UKSDyf5YpJfT/K2Wutj7Znjf5DkjqyNnv3htu5WxwA4b4cOHcradbik1po77rij44rgiXMeM0luuummvPjFL9aLzNg61Yu82TLd2HGW7y8tpfytzZZrrf/hXA5Sa/14ko+3z3+ctZGpz1znW0nesMn2P5vkZzdovz3J7Ru0b3gMgCfj8ssvP+0fr8svv7y7YuA8OY+ZJKemM4NxtXv37tN+J+/evbuzWviOs/Uk/8ckr133Wr/8Y8MtjUliJFUmwb333rvlMowD5zGTxN8XjLt3vvOdpy2b73s0bBmSa61v3uL1d7erSMafkVSZBLOzs6cNEnPttdd2XBE8cbOzs6ctO48ZZ/6+YNy98IUvfLz3ePfu3dmzZ0+3BZHkHKeAKqVcXkp5fyllvi2/qJTyluGWxqRYP5Lq/Py8q72MrX6/v+UyjIPXvva1py2/7nWv66gSeHL8fcGkeOc735lnPvOZepFHyDmF5CQfzNoAWVe05buT/KNhFMTkGQwGjw8Ss7q66movE+FUjzKMm1/7tV87bfm2227rqBJ4cvx9waR44QtfmPn5eb3II+RcQ3Kv1vrhJKtJ0kaWfmxoVTFRFhYWTptu5NChQx1XBOfHH2RMgoWFhdOW/U5mXPn7gknh2frRc64h+eFSys4kNUlKKVcn+cbQqmKizMzMZHp6OkkyPT39Xc/Dwbg48w8wU+cwjl7+8peftvyKV7yio0rgyZmZmcmOHWsTtezYscPfF4wtz9aPnnMNyW9PcluSv1RK+a0ktybZP7SqmCj9fv/xW1OnpqY8x8nYOnOqHFPnAHSn3+9ndXU1ydrdPf6+YBx5tn40nVNIrrX+bpIfSfLDSW5M8pdrrUeGWRiTo9fr5ZprrkmSXHPNNdm5c2fHFcH5+frXv77lMoyDT37yk6ctf+ITn+ioEgA8yjWatgzJpZS/deqV5HVJvi/JC5O8trUBXDSe97znbbkM42BmZiaXXHJJkuSSSy5xiypja324qLUKF4wlz9aPprP1JL92i9ePDbc0JsXy8nLuuuuuJMldd93lNhLG1r333rvlMoyDfr9/WrBwiyrj6tChQ6edy8aJYBwZu2c0bRmSa61v3uL1d7erSMbbYDB4/Jmhxx57zJVextbs7Ozjz9eXUnLttdd2XBHAxavX6225DOPA2D2j6VwH7kop5W+WUv5pKeVdp17DLIzJsbCwkJMnTyZJTp486TYSxla/3398JNXp6Wn/kDGWBoPBaRd7XLhkXH31q1/dchnGQa/Xy9zcXEopmZubM3bPiDinkFxK+T+S/J2sjWhdkrwhyV8cYl1MENONMCl6vV5e85rXpJSS17zmNf4hYywtLCzkscceS7J2d48Ll4yrU7dab7YM46Lf7+eqq65y8X2EnGtP8g/XWt+U5IFa608n+WtJnj+8sgBGk3/IGHcuXDIppqamtlyGcdHr9XLw4EEX30fIuf42ebS9P1JKuSLJySQvGE5JTBrTjTBJ/EMGMBpe/epXn7Y8MzPTUSXApDnXkPzRUsqlSf55ks8k+c9JPjS0qpgophsBGB0uXDIpbrzxxsd7j6empnLjjTd2XBEwKc42T/J/WUp5Xq31Z2qtDyZ5VpLPJ/nlJL+wHQUy/kw3wiRZXl7O/v37TWXG2JqZmXl8ALodO3a4cMnY6vV6j/cez87OusMHuGDO1pN8S5JvJ0kp5RVJfq61fSPJ+4ZbGpNkfUiGcTYYDHLkyBEjAjO2+v3+471vl1xyiQuXjLUbb7wxL37xi/UiAxfU2ULyJbXW+9vnv5PkfbXWf19r/Z+T7BluaUyKW2655bSQfMstt3RcEZyf5eXlzM/Pp9aa+fl5vcmMJdONMEmMEwEMw1lDcillR/v8qiS/se67HRusD9/lzjvvPG35Yx/7WEeVwJMzGAyyurqaZG3qHL3JjCujtAPA5s4Wkv9tkv9YSvlI1ka4/mSSlFL2ZO2Wazgr8xgyKRYWFnLy5MkkycmTJ80vy9jS+wYAm9syJNdafzbJP07ywSR/o34n3Uwl2T/c0pgUpmhgUphfFgBg8p11Cqha62/XWn+11vrwura7a62/O9zSmBSmaAAYLUZpZ1I4l4FhONd5kuG8maKBSWF+WSaFUdqZFM5lYBiEZLaFKRqYBOaXZRIYpZ1J4VwGhkVIZlsYJIZJYH5ZJsFgMHh8AMXV1VU9cIwtMw4Aw2Iapwl14MCBLC4udl3G45aWlpIku3bt6riS79izZ09uuummrstgjJyaX/a2224zvyxja2FhISvYbzD/AAAY30lEQVQrK0mSlZWVHDp0KG9/+9s7rgqeuI1mHHAuAxeCnmS2xaOPPppHH3206zLgSTO/LONuZmYm09PTSZLp6WmPDTC2zDgADIue5Ak1aj2kp+o5cOBAx5XAk3Pq0QEYV/1+P7fffvtpywDAd+hJBoCLSK/Xy1Of+tQkyVOf+lSPDTC2zDgADIuQDAAXkbvvvjsPPfRQkuShhx4aqfEr4Ikw4wAwLG63BkaaQei2ZgA6nqibb775tOX3vOc9ufXWWzuqhnEzSr+TV1ZWHh+467HHHsvRo0c7/33odzJMBiEZ4AkwAB3j7tixY1suw7iYnp7Ojh07cvLkyTz3uc99fEA6gCdLSAZG2qhdkTcIHeNu9+7dpwXj3bt3d1YL42fUfie/9a1vzbFjx/JLv/RLnq8HLhghGQAuIu985zvz9/7e33t8+V3veleH1cCTMz09nb179wrIPGGj9OjAqD3KlXh0wMBdAHAReeELX/j4H2K7du3Knj17Oq4I4OL26KOPepxrxOhJBoCLzJ49e7K0tJS9e/d2XQpAJ0apl9SjXKNHTzIAXESWl5fzW7/1W0mS3/zN38yJEyc6rggARouQDAAXkcFgkJWVlSRrU+gMBoOOKwKA0SIkA8BF5I477jht+dd//dc7qgQARpOQDAAXkUsuuWTLZQC42AnJAHARefjhh7dcBoCLnZAMAAAAjZAMABeRV77ylactX3PNNd0UAgAjSkgGgIvImXODjtJcoQAwCoRkALiI9Hq9x3uTr7nmmuzcubPbggBgxOzougAAYHvddNNNeeCBB/QiA8AGhGQA2AYHDhzI4uJi12UkSZaWlpIkP/3TP91xJd+xZ88eoR2AkSAkA8BF5tFHH+26BAAYWUIyAGyDUeolPVXLgQMHOq4EAEaPgbsAAACgEZIBAACgEZIBAACgGVpILqU8rZTyn0opv1dK+UIp5adb+wtKKb9TSjlaSvl3pZSntPantuXF9v3udfv6ydb+h6WUa9e1X9faFksp71jXvuExAAAAYCvD7En+syQ/Wmt9cZKXJLmulHJ1kp9P8gu11r1JHkjylrb+W5I8UGvdk+QX2noppbwoyRuT/OUk1yX516WUS0oplyT5xSRzSV6U5MfbutniGAAAALCpoYXkuuahtjjdXjXJjyb5ldY+SPL69vn6tpz2/atKKaW1f6jW+me11v+cZDHJy9prsdb6x7XWbyf5UJLr2zabHQMAAAA2NdRnkluP7+eS3JdkIckfJXmw1nqyrbKU5Mr2+cokX0mS9v03kuxc337GNpu179ziGAAAALCpoYbkWutjtdaXJNmVtZ7fH9hotfZeNvnuQrV/l1LKDaWUw6WUw8ePH99oFQAAAC4i2zK6da31wSQfT3J1kktLKTvaV7uS3NM+LyV5fpK07/98kvvXt5+xzWbty1sc48y63ldr3Vdr3XfZZZc9mR8RAACACTDM0a0vK6Vc2j4/Pcmrk3wpyV1J/nZbrZ/kI+3zbW057fvfqLXW1v7GNvr1C5LsTfKfknw6yd42kvVTsja4121tm82OAQAAAJvacfZVztv3Jhm0Uainkny41vrRUsoXk3yolHJzks8meX9b//1J/u9SymLWepDfmCS11i+UUj6c5ItJTiZ5W631sSQppfyDJHckuSTJB2qtX2j7+olNjgEAAACbGlpIrrUeSfJXN2j/46w9n3xm+7eSvGGTff1skp/doP32JLef6zEAAABgK9vyTDIAAACMAyEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAmh1dFzBJDhw4kMXFxa7LGElHjx5Nktx0000dVzK69uzZMxL/fZzHW3Mub21UzmMAgPMlJF9Ai4uL+eznv5jVZzy361JGTvl2TZJ85o++3nElo2nqkfu7LuFxi4uLufv3fzd/4VmPdV3KSHrKytoNON869umOKxk9X37okq5LAAB40oTkC2z1Gc/Nt170Y12XwZh52hc/2nUJp/kLz3os79z3UNdlMGZuPvysrksAAHjShGQAAGCoPM61OY9ynd12P84lJAMAjCjBYmvCxdZGaZyIxcXFfOHzX8qlz/ierksZOavfLkmSr/7RiY4rGU0PPnLfth9TSAYAGFGLi4v5g899Ls/rupARdWqalgc/97lO6xhFozgKzKXP+J5c8/1v7LoMxsxdf/ChbT+mkAwAMMKel+QtKV2XwZh5f2rXJcDYEpIBmEhuU92cW1TPbpRuUwVgewnJAEykxcXFfPYLn00u7bqSEbS69vbZr3622zpG1YNdFwBAl4YWkkspz09ya9buElpN8r5a678qpTw3yb9LsjvJsST/da31gVJKSfKvkrwmySNJ/oda6++2ffWTvLPt+uZa66C1/2CSDyZ5epLbk/zDWmvd7BjD+lkBGFGXJquvXO26CsbM1Menzr4SABNrmP8KnEzyj2utP5Dk6iRvK6W8KMk7ktxZa92b5M62nCRzSfa21w1J3pskLfC+O8kPJXlZkneXUp7TtnlvW/fUdte19s2OAQAAAJsaWkiutX7tVE9wrfWbSb6U5Mok1ycZtNUGSV7fPl+f5Na65reTXFpK+d4k1yZZqLXe33qDF5Jc1757dq31U7XWmrVe6/X72ugYAAAAsKltuZ+olLI7yV9N8jtJLq+1fi1ZC9JJTk2WdmWSr6zbbKm1bdW+tEF7tjgGAAAAbGroIbmU8qwk/z7JP6q1/ulWq27QVs+j/YnUdkMp5XAp5fDx48efyKYAAABMoKGG5FLKdNYC8r+ptf6H1nxvu1U67f2+1r6U5PnrNt+V5J6ztO/aoH2rY5ym1vq+Wuu+Wuu+yy677Px+SAAAACbG0EJyG636/Um+VGv9l+u+ui1Jv33uJ/nIuvY3lTVXJ/lGu1X6jiSzpZTntAG7ZpPc0b77Zinl6nasN52xr42OAQAAAJsa5jzJfz3Jf5/k86WUz7W2f5bk55J8uJTyliRfTvKG9t3tWZv+aTFrU0C9OUlqrfeXUn4myafbeu+ptd7fPr8135kCar69ssUxAAAAYFNDC8m11t/Mxs8NJ8mrNli/JnnbJvv6QJIPbNB+OMlf2aD9xEbHAAAAgK1sy+jWAAAAMA6EZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGiEZAAAAGh2dF0AMFqWlpby8Dcvyc2Hn9V1KYyZP/nmJXnm0lLXZQAAPCl6kgEAAKDRk3wBLS0tZeqRb+RpX/xo16UwZqYeOZGlpZNdl5Ek2bVrV7518mt5576Hui6FMXPz4Wflabt2dV3G45aWlpJvJFMfdz2YJ+jBZKm6KwLgYiUkAwCMqKWlpXwzyftTuy6FMfO1JA95BAbOi5B8Ae3atSv3/tmOfOtFP9Z1KYyZp33xo9m163ldlwETZdeuXTlejmf1latdl8KYmfr4VHZdOTp3RQCwvYRkAIARtWvXrjy4vJy3pHRdCmPm/am5dIQegYFx4kEtAAAAaPQkAwAAQ7W0tJRvPPLN3PUHH+q6FMbMg4/cl7r06LYeU08yAAAANHqSAQCAodq1a1fKn53INd//xq5LYczc9QcfypW7dm7rMfUkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQLOj6wImzdQj9+dpX/xo12WMnPKtP02S1Kc9u+NKRtPUI/cneV7XZcDkeTCZ+rjrwd/lofb+rE6rGF0PJrmy6yIA6IqQfAHt2bOn6xJG1tGj30yS7P1LguDGnjdS58+XH7okNx/21/NG7n1kLXBd/ozVjisZPV9+6JK8sOsi1hml/0+NmqNHjyZJ9l65t+NKRtSVzh+Ai5mQfAHddNNNXZcwsk79tzlw4EDHlXA2/jDc2rdbuHjabuHiTC/MaJ0/fidvzu9koAsPPnJf7vqDD3Vdxsh56FsPJEme9bTndFzJaHrwkftyZXZu6zGFZOA0gsXWhAsAeOJG6SLqqDl69P4kyZV/aXuD4Li4Mju3/fwRkgEARtjXk7w/tesyRtKJ9i5afLevJ7m06yLWcRF+cy7Ajx4hGQBgROl929rx9gjMpXs9AnOmS+P8gfMlJAMAjCi9b1vTAwcMg3kxAAAAoBGSAQAAoBGSAQAAoBGSAQAAoBGSAQAAoBGSAQAAoBGSAQAAoBGSAQAAoBGSAQAAoBlaSC6lfKCUcl8p5ffXtT23lLJQSjna3p/T2ksp5UApZbGUcqSU8tJ12/Tb+kdLKf117T9YSvl82+ZAKaVsdQwAAAA4m2H2JH8wyXVntL0jyZ211r1J7mzLSTKXZG973ZDkvcla4E3y7iQ/lORlSd69LvS+t617arvrznIMAAAA2NLQQnKt9RNJ7j+j+fokg/Z5kOT169pvrWt+O8mlpZTvTXJtkoVa6/211geSLCS5rn337Frrp2qtNcmtZ+xro2MAAADAlrb7meTLa61fS5L2/j2t/cokX1m33lJr26p9aYP2rY4BAAAAWxqVgbvKBm31PNqf2EFLuaGUcriUcvj48eNPdHMAAAAmzHaH5HvbrdJp7/e19qUkz1+33q4k95ylfdcG7Vsd47vUWt9Xa91Xa9132WWXnfcPBQAAwGTY7pB8W5JTI1T3k3xkXfub2ijXVyf5RrtV+o4ks6WU57QBu2aT3NG++2Yp5eo2qvWbztjXRscAAACALe0Y1o5LKf82ySuT9EopS1kbpfrnkny4lPKWJF9O8oa2+u1JXpNkMckjSd6cJLXW+0spP5Pk022999RaTw0G9tasjaD99CTz7ZUtjgEAAABbGlpIrrX++CZfvWqDdWuSt22ynw8k+cAG7YeT/JUN2k9sdAwAAAA4m1EZuAsAAAA6JyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAIyQDAABAs6PrAgAAALbTgQMHsri42HUZSZKjR48mSW666aaOK/mOPXv2jFQ9201IBgAA6MjTn/70rkvgDEIyAABwUbmYe0k5O88kAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQCMkAwAAQGMKKADYBgcOHMji4mLXZSRJjh49mmS0pkDZs2fPSNUDwMVLSAZG2igFi2T0woVgwfl4+tOf3nUJADCyhGSAJ0C44Hy5mAEA40FInlB6385OD9x48L8RAADbSUhmW+h9A4Dx5yL81lyAh8kgJE8ov6ABgEnnIjwwDEIyAADnxEV44GJgnmQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABohGQAAABoJjYkl1KuK6X8YSllsZTyjq7rAQAAYPRNZEgupVyS5BeTzCV5UZIfL6W8qNuqAAAAGHUTGZKTvCzJYq31j2ut307yoSTXd1wTAAAAI25SQ/KVSb6ybnmptQEAAMCmJjUklw3a6netVMoNpZTDpZTDx48f34ayAAAAGGU7ui5gSJaSPH/d8q4k95y5Uq31fUnelySllOOllD/ZnvIuWr0ky10XAReAc5lJ4DxmUjiXmQTO4+3xF89lpVLrd3Wwjr1Syo4kdyd5VZKvJvl0kv+m1vqFTgu7yJVSDtda93VdBzxZzmUmgfOYSeFcZhI4j0fLRPYk11pPllL+QZI7klyS5AMCMgAAAGczkSE5SWqttye5ves6AAAAGB+TOnAXo+l9XRcAF4hzmUngPGZSOJeZBM7jETKRzyQDAADA+dCTDAAAAI2QzHkrpTy/lHJXKeVLpZQvlFL+YWt/billoZRytL0/p7WXUsqBUspiKeVIKeWl6/b1F0oph9q+vlhK2d3NT8XF6EKdy6WUa0opn1v3+lYp5fVd/mxcPC7w7+R/3vbxpbZO6ern4uJzgc/lny+l/H57/Z2ufiYuPudxHn9/KeVTpZQ/K6X8kzP2dV0p5Q/bOf6OLn6ei42QzJNxMsk/rrX+QJKrk7ytlPKiJO9IcmetdW+SO9tykswl2dteNyR577p93Zrkf2n7elmS+7bnR4AkF+hcrrXeVWt9Sa31JUl+NPn/27u3ELuuOo7j3x9JaoilVrTaW8R6N9SQBrybRAuFRilBjKURNLYPJU9SRCnog/oQBK3aB5WCeElUvGHVWu3FhxCsNpqaWlOMWG1RiyFaWpNOQtU2fx/2GjwZMk1nZp89nc73A4c5s/be66wF/7PZ/33WWptjwO2D9kSLWS9xnORNwJuB1cCFwGuBDQP2Q+orlt8BrAXWAK8HPpzkjCE7okVtpnH8MPAB4LrRSpIsAb5AF+ergC2tHo2RSbJmraoOVtW+9v5R4ABwHrAJ2NF22wFM/pK2CdhZnT3AmUnOaV/0pVX1s1bXRFUdG7IvWtz6iuUp1W4GbjGWNZQe47iA5cBpwLOAZcChwTqiRa/HWF4F7K6qx6vqKHAPcOmAXdEiNtM4rqp/VNVe4L9Tqnod8Kequr+q/gN8u9WhMTJJVi/a8OiLgF8BL6yqg9CdIIAXtN3OA/42ctiDrewVwL+S3Jjk7iSfbnfNpMHNMZZHXQF8a5xtlaYzlziuqjuBXcDB9rqtqg4M03LpRHM8J98DbEyyIsnzgbcBK4dpufR/TzGOp/NUrjnUM5NkzVmS04HvA9dU1ZEn2/UkZUX3vO51wIfohvW9BHh/z82UTqmHWJ6s5xzgNcBt/bZQOrW5xnGSlwGvBs6nuxC7OMn6/lsqPbm5xnJV3Q78FPgl3U3LO+mGwEqDmUEcT1vFScp8PNGYmSRrTpIso/vif7OqbmzFhyaHnra/k/OLH+TEO7jnA39v5Xe3YSSPAz+km0MkDaanWJ50OfCDqpo6ZEoaq57i+J3Anjb1ZQK4hW4+nTSYvs7JVbW9rRVxCV2ycd8Q7ZdgxnE8nVNdc2gMTJI1a2210y8DB6rqsyObbgK2tvdbgR+NlL+vrUL5BuBwG2ayF3hukrPafhcDvx97B6Smx1ietAWHWmtgPcbxX4ENSZa2C7wNdHPppEH0FctJliR5XqtzNd1idC6mqEHMIo6nsxd4eZILkpxGN53rpr7bqxOlyl/rNTtJ3gL8HNgPHG/FH6Gbb/Fd4EV0F1vvrqqH28ni83SLZhwDrqyqu1pdlwCfobvL+xvg6rY4gTR2Pcfyi4FfACur6jjSQPqK47YmxBeB9XRD+m6tqg8O2hktaj3G8nJgXzv+CLCtqn47XE+0mM0ijs8G7gLOaPtPAKuq6kiStwPXA0uAr1TV9kE7swiZJEuSJEmS1DjcWpIkSZKkxiRZkiRJkqTGJFmSJEmSpMYkWZIkSZKkxiRZkiRJkqTGJFmSpAWoPRP2jiQbR8ouT3LrSfa9Ksn+JL9Lcm+STaeo+2tJNp+k/K1Jbu6nB5IkPT0tne8GSJKkmauqSrIN+F6SXXTPz9xO96xYoEukgZXAR4G1VXU4yenAWfPRZkmSFgKTZEmSFqiqujfJj4FrgWcDO4EnkhwAdgFvBK4BHgUm2jETk++TrAFuAFYAfwauqqpHRj8jyaXA9cBDwL4BuiVJ0rxyuLUkSQvbJ4D3ABuBT7WyVwI7q+oi4A7gEPBAkq8muWzk2J3AtVW1GtgPfGy04iTLgS8BlwHrgLPH2RFJkp4OTJIlSVrAquoo8B3g61X171b8l6ra07Y/QTcEezPwR+BzST6e5DnAmVW1ux2zA1g/pfpXAQ9U1X1VVcA3xtwdSZLmnUmyJEkL3/H2mnR0dGN1fl1VnwSuAN41g7qrh/ZJkrRgmCRLkvQMluTcJGtHitbQ/dJ8GHgkybpW/l5g95TD/wBckOSl7f8t422tJEnzz4W7JEl6ZlsGXJfkXOAx4J/AtrZtK3BDkhXA/cCVowdW1WNJrgZ+kuQhuvnNFw7WckmS5kG6KUaSJEmSJMnh1pIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1/wM//QZxM/T2yQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 1152x648 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"sns.boxplot('YrSold', 'SalePrice', data=df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simplest regression tree"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.ensemble import RandomForestRegressor\n",
"import graphviz\n",
"from sklearn.tree import export_graphviz\n",
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": 296,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
"<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
" -->\r\n",
"<!-- Title: Tree Pages: 1 -->\r\n",
"<svg width=\"669pt\" height=\"373pt\"\r\n",
" viewBox=\"0.00 0.00 669.00 373.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 369)\">\r\n",
"<title>Tree</title>\r\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-369 665,-369 665,4 -4,4\"/>\r\n",
"<!-- 0 -->\r\n",
"<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.407843\" stroke=\"black\" points=\"359,-365 226,-365 226,-297 359,-297 359,-365\"/>\r\n",
"<text text-anchor=\"middle\" x=\"292.5\" y=\"-349.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YrSold &lt;= 2007.5</text>\r\n",
"<text text-anchor=\"middle\" x=\"292.5\" y=\"-334.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6306788585.3</text>\r\n",
"<text text-anchor=\"middle\" x=\"292.5\" y=\"-319.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 1460</text>\r\n",
"<text text-anchor=\"middle\" x=\"292.5\" y=\"-304.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 180921.2</text>\r\n",
"</g>\r\n",
"<!-- 1 -->\r\n",
"<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.803922\" stroke=\"black\" points=\"284,-261 151,-261 151,-193 284,-193 284,-261\"/>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-245.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YrSold &lt;= 2006.5</text>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-230.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6826452950.6</text>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-215.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 643</text>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-200.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 184347.3</text>\r\n",
"</g>\r\n",
"<!-- 0&#45;&gt;1 -->\r\n",
"<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M268.15,-296.884C261.731,-288.154 254.724,-278.625 248.025,-269.514\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"250.729,-267.283 241.985,-261.299 245.089,-271.429 250.729,-267.283\"/>\r\n",
"<text text-anchor=\"middle\" x=\"238.215\" y=\"-282.313\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">True</text>\r\n",
"</g>\r\n",
"<!-- 4 -->\r\n",
"<g id=\"node5\" class=\"node\"><title>4</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.098039\" stroke=\"black\" points=\"435,-261 302,-261 302,-193 435,-193 435,-261\"/>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-245.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YrSold &lt;= 2008.5</text>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-230.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 5881290473.7</text>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-215.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 817</text>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-200.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 178224.8</text>\r\n",
"</g>\r\n",
"<!-- 0&#45;&gt;4 -->\r\n",
"<g id=\"edge4\" class=\"edge\"><title>0&#45;&gt;4</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M317.175,-296.884C323.679,-288.154 330.78,-278.625 337.568,-269.514\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"340.52,-271.409 343.689,-261.299 334.907,-267.227 340.52,-271.409\"/>\r\n",
"<text text-anchor=\"middle\" x=\"347.302\" y=\"-282.337\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">False</text>\r\n",
"</g>\r\n",
"<!-- 2 -->\r\n",
"<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.596078\" stroke=\"black\" points=\"133,-149.5 0,-149.5 0,-96.5 133,-96.5 133,-149.5\"/>\r\n",
"<text text-anchor=\"middle\" x=\"66.5\" y=\"-134.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6288531575.3</text>\r\n",
"<text text-anchor=\"middle\" x=\"66.5\" y=\"-119.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 314</text>\r\n",
"<text text-anchor=\"middle\" x=\"66.5\" y=\"-104.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 182549.5</text>\r\n",
"</g>\r\n",
"<!-- 1&#45;&gt;2 -->\r\n",
"<g id=\"edge2\" class=\"edge\"><title>1&#45;&gt;2</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.475,-192.884C150.565,-180.786 130.38,-167.151 112.723,-155.224\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"114.525,-152.217 104.28,-149.52 110.607,-158.018 114.525,-152.217\"/>\r\n",
"</g>\r\n",
"<!-- 3 -->\r\n",
"<g id=\"node4\" class=\"node\"><title>3</title>\r\n",
"<polygon fill=\"#e58139\" stroke=\"black\" points=\"284,-149.5 151,-149.5 151,-96.5 284,-96.5 284,-149.5\"/>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-134.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 7333820019.7</text>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-119.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 329</text>\r\n",
"<text text-anchor=\"middle\" x=\"217.5\" y=\"-104.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 186063.2</text>\r\n",
"</g>\r\n",
"<!-- 1&#45;&gt;3 -->\r\n",
"<g id=\"edge3\" class=\"edge\"><title>1&#45;&gt;3</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M217.5,-192.884C217.5,-182.326 217.5,-170.597 217.5,-159.854\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"221,-159.52 217.5,-149.52 214,-159.52 221,-159.52\"/>\r\n",
"</g>\r\n",
"<!-- 5 -->\r\n",
"<g id=\"node6\" class=\"node\"><title>5</title>\r\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"435,-149.5 302,-149.5 302,-96.5 435,-96.5 435,-149.5\"/>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-134.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 4847058504.8</text>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-119.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 304</text>\r\n",
"<text text-anchor=\"middle\" x=\"368.5\" y=\"-104.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 177360.8</text>\r\n",
"</g>\r\n",
"<!-- 4&#45;&gt;5 -->\r\n",
"<g id=\"edge5\" class=\"edge\"><title>4&#45;&gt;5</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M368.5,-192.884C368.5,-182.326 368.5,-170.597 368.5,-159.854\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"372,-159.52 368.5,-149.52 365,-159.52 372,-159.52\"/>\r\n",
"</g>\r\n",
"<!-- 6 -->\r\n",
"<g id=\"node7\" class=\"node\"><title>6</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.156863\" stroke=\"black\" points=\"586,-157 453,-157 453,-89 586,-89 586,-157\"/>\r\n",
"<text text-anchor=\"middle\" x=\"519.5\" y=\"-141.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YrSold &lt;= 2009.5</text>\r\n",
"<text text-anchor=\"middle\" x=\"519.5\" y=\"-126.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6493464273.9</text>\r\n",
"<text text-anchor=\"middle\" x=\"519.5\" y=\"-111.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 513</text>\r\n",
"<text text-anchor=\"middle\" x=\"519.5\" y=\"-96.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 178736.7</text>\r\n",
"</g>\r\n",
"<!-- 4&#45;&gt;6 -->\r\n",
"<g id=\"edge6\" class=\"edge\"><title>4&#45;&gt;6</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M417.525,-192.884C431.707,-183.304 447.315,-172.761 461.965,-162.864\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"464.135,-165.623 470.462,-157.125 460.217,-159.822 464.135,-165.623\"/>\r\n",
"</g>\r\n",
"<!-- 7 -->\r\n",
"<g id=\"node8\" class=\"node\"><title>7</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.239216\" stroke=\"black\" points=\"510,-53 377,-53 377,-0 510,-0 510,-53\"/>\r\n",
"<text text-anchor=\"middle\" x=\"443.5\" y=\"-37.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6522097622.7</text>\r\n",
"<text text-anchor=\"middle\" x=\"443.5\" y=\"-22.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 338</text>\r\n",
"<text text-anchor=\"middle\" x=\"443.5\" y=\"-7.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 179432.1</text>\r\n",
"</g>\r\n",
"<!-- 6&#45;&gt;7 -->\r\n",
"<g id=\"edge7\" class=\"edge\"><title>6&#45;&gt;7</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M492.919,-88.9485C485.709,-79.9834 477.894,-70.2666 470.653,-61.2629\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"473.193,-58.8358 464.198,-53.2367 467.738,-63.2228 473.193,-58.8358\"/>\r\n",
"</g>\r\n",
"<!-- 8 -->\r\n",
"<g id=\"node9\" class=\"node\"><title>8</title>\r\n",
"<polygon fill=\"#e58139\" fill-opacity=\"0.003922\" stroke=\"black\" points=\"661,-53 528,-53 528,-0 661,-0 661,-53\"/>\r\n",
"<text text-anchor=\"middle\" x=\"594.5\" y=\"-37.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 6435423276.0</text>\r\n",
"<text text-anchor=\"middle\" x=\"594.5\" y=\"-22.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 175</text>\r\n",
"<text text-anchor=\"middle\" x=\"594.5\" y=\"-7.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 177393.7</text>\r\n",
"</g>\r\n",
"<!-- 6&#45;&gt;8 -->\r\n",
"<g id=\"edge8\" class=\"edge\"><title>6&#45;&gt;8</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M545.731,-88.9485C552.846,-79.9834 560.558,-70.2666 567.704,-61.2629\"/>\r\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"570.599,-63.2454 574.074,-53.2367 565.116,-58.8938 570.599,-63.2454\"/>\r\n",
"</g>\r\n",
"</g>\r\n",
"</svg>\r\n"
],
"text/plain": [
"<graphviz.files.Source at 0x1648bdd72e8>"
]
},
"execution_count": 296,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rf = RandomForestRegressor(n_estimators=1, bootstrap=False)\n",
"rf.fit(df.YrSold.values.reshape(-1, 1), df.SalePrice)\n",
"s = export_graphviz(rf.estimators_[0], feature_names=['YrSold'], out_file='simple.dot', filled=True, precision=True)\n",
"path = 'simple.dot'\n",
"graphviz.Source.from_file(path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding another column"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'tree.dot.pdf'"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rf = RandomForestRegressor(n_estimators=1, bootstrap=False)\n",
"rf.fit(df[['YrSold', 'LotArea']], df.SalePrice)\n",
"s = export_graphviz(rf.estimators_[0], feature_names=['YrSold', 'LotArea'], out_file='tree.dot', filled=True, precision=True)\n",
"path = 'tree.dot'\n",
"s = graphviz.Source.from_file(path)\n",
"s.view()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Asking the right questions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\begin{align}\n",
"MSE = \\frac{\\sum{(\\text{expected} - \\text{predicted})}^2}{\\text{number of samples}}\n",
"\\\\\n",
"\\\\\n",
"LOSS = \\frac{MSE(\\text{left}) * len(\\text{left}) + MSE(\\text{right}) * len(\\text{right})}{\\text{number of samples}}\n",
"\\end{align}"
]
},
{
"cell_type": "code",
"execution_count": 199,
"metadata": {},
"outputs": [],
"source": [
"chosen_columns = ['YrSold', 'LotArea', 'YearBuilt', 'OverallCond', 'OverallQual']\n",
"X_raw = df[chosen_columns].values\n",
"Y_raw = df.SalePrice.values\n",
"\n",
"X_train, X_test, Y_train, Y_test = train_test_split(X_raw, Y_raw, test_size=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Creating our own tree"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cost functions"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"def mse(expected, predicted):\n",
" return np.mean(np.square(expected - predicted))"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"def loss(left, right):\n",
" return (mse(left, np.mean(left)) * left.size + mse(right, np.mean(right)) * right.size) / (left.size + right.size)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./media/rfdepth.png\">"
]
},
{
"cell_type": "code",
"execution_count": 263,
"metadata": {},
"outputs": [],
"source": [
"def create_tree(X, Y, samples, max_depth=2, min_samples=3, col_proc=0.5, depth=1):\n",
" \"\"\"\n",
" :param samples: training data\n",
" :param depth: current depth\n",
" :param max_depth: don't go bellow this level\n",
" :param min_samples: make a leaf for fewer samples than this\n",
" :param col_proc: percentage of columns to be used as features\n",
" \"\"\"\n",
" \n",
" # Default is the mean\n",
" node = dict(value=np.mean(Y[samples]), split=(), samples=len(samples), left=None, right=None)\n",
" \n",
" # Stop if the maximum depth was reached\n",
" if depth > max_depth:\n",
" return node\n",
"\n",
" # Create a leaf if there are less than min_samples samples in this node\n",
" if len(samples) < min_samples:\n",
" return node\n",
" \n",
" best_cost = None\n",
" best_left = []\n",
" best_right = []\n",
" best_split = ()\n",
" \n",
" # Choose columns\n",
" columns = np.arange(X.shape[1])\n",
" np.random.shuffle(columns)\n",
" columns = columns[:int(X.shape[1] * col_proc)]\n",
" \n",
" # Try each column\n",
" for col in columns:\n",
" values = X[samples, col]\n",
" \n",
" # Try every possible value as a split point\n",
" for split_value in values:\n",
" left = []\n",
" right = []\n",
" \n",
" # Partition samples\n",
" for s in samples:\n",
" if X[s][col] < split_value:\n",
" left.append(s)\n",
" else:\n",
" right.append(s)\n",
"\n",
" if len(left) == 0 or len(right) == 0:\n",
" continue\n",
"\n",
" # Update best loss\n",
" cost = loss(Y[left], Y[right])\n",
" if best_cost is None or cost < best_cost:\n",
" best_cost = cost\n",
" best_left = left\n",
" best_right = right\n",
" best_split = (split_value, chosen_columns[col])\n",
" \n",
" # Expand subtrees\n",
" if len(best_left) > 0 and len(best_right) > 0:\n",
" node['split'] = best_split\n",
" node['left'] = create_tree(X, Y, best_left, max_depth=max_depth, min_samples=min_samples, col_proc=col_proc, depth=depth+1)\n",
" node['right'] = create_tree(X, Y, best_right, max_depth=max_depth, min_samples=min_samples, col_proc=col_proc, depth=depth+1)\n",
" \n",
" return node"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./media/rf_partitioning.gif\">"
]
},
{
"cell_type": "code",
"execution_count": 264,
"metadata": {},
"outputs": [],
"source": [
"from graphviz import Graph\n",
"\n",
"def plot_tree(tree, g=None):\n",
" g = g or Graph('G', filename='tree.gv')\n",
" node_name = str(tree['samples']) + ' ' + str(tree['split']) + ' ' + str(tree['value'])\n",
" for child in ('left', 'right'):\n",
" if tree[child] is not None:\n",
" child_name = str(tree[child]['samples']) + ' ' + str(tree[child]['split']) + ' ' + str(tree[child]['value'])\n",
" g.edge(node_name, child_name)\n",
" for child in ('left', 'right'):\n",
" if tree[child] is not None:\n",
" plot_tree(tree[child], g)\n",
" return g"
]
},
{
"cell_type": "code",
"execution_count": 288,
"metadata": {},
"outputs": [],
"source": [
"tree = create_tree(X_train, Y_train, np.arange(X_train.shape[0]), max_depth=3)"
]
},
{
"cell_type": "code",
"execution_count": 297,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'tree.gv.pdf'"
]
},
"execution_count": 297,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plot_tree(tree).view()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation"
]
},
{
"cell_type": "code",
"execution_count": 274,
"metadata": {},
"outputs": [],
"source": [
"def predict_tree(tree, sample):\n",
" if tree['left'] is None or tree['right'] is None:\n",
" return tree['value']\n",
" value, col = tree['split']\n",
" col = chosen_columns.index(col)\n",
" if sample[col] < value:\n",
" return predict_tree(tree['left'], sample)\n",
" return predict_tree(tree['right'], sample)"
]
},
{
"cell_type": "code",
"execution_count": 275,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2169115062.780037 6191585914.697112\n"
]
}
],
"source": [
"predictions = np.array([predict_tree(tree, sample) for sample in X_test])\n",
"print(mse(Y_test, predictions), mse(Y_test, np.mean(Y_train)))"
]
},
{
"cell_type": "code",
"execution_count": 276,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x164890fbfd0>"
]
},
"execution_count": 276,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7sAAAIMCAYAAAA0IruPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3W+QpdddH/jvmVbLbonglkFQqGVFCnENwau1xurFyqoqFcTGI+MFzwocQ3nXKpYqp9gkhZ3ULKNUamUDWU12amPi2sQVJyTYCxvkGDHIyGRQWeaNC4FbGdmKsKckbLDVcrBAGgWsttWaOfuinzu603P/dt/ue/vpz6eqq7vPfe7znPvckWa+95zzO6XWGgAAAGiTA9PuAAAAAEyasAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA61w27Q5M2rd/+7fX66+/ftrdAAAAYAc88sgjf1prvXrYca0Lu9dff31WVlam3Q0AAAB2QCnlj0c5zjRmAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaZ2jYLaUcLKU82vX1X0sp7y6lvLqU8mAp5Ynm+1XN8aWU8oFSypOllM+VUt7Qda47m+OfKKXc2dV+cynlseY5HyillKa95zUAAABgkKFht9Z6ptZ6U631piQ3J3khya8nOZbkk7XW1yb5ZPN7krw5yWubr3cl+WCyEVyT3J3kjUm+L8ndXeH1g82xnefd3rT3uwYAAAD0Ne405h9I8oe11j9O8tYkH27aP5zkSPPzW5N8pG54OMliKeW7khxO8mCt9dla63NJHkxye/PYt9Zaf7fWWpN8ZNO5el0DAAAA+ho37P5Ykn/f/PydtdavJknz/Tua9qUkX+l6zlNN26D2p3q0D7oGAAAA9DVy2C2lXJ7kh5P8h2GH9mirW2gfWSnlXaWUlVLKyjPPPDPOUwEAAGihcUZ235zkP9Va/6T5/U+aKchpvn+taX8qyWu6nndtkqeHtF/bo33QNS5Sa/1QrXW51rp89dVXj/GSAAAAaKNxwu6P5+UpzElyf5JOReU7k/xGV/s7m6rMtyR5vpmCfCrJm0opVzWFqd6U5FTz2J+XUm5pqjC/c9O5el0DAAAA+rpslINKKVck+VtJ/k5X8/EkHy2l/GSSLyd5W9P+iSQ/mOTJbFRu/okkqbU+W0r5uSSfaY772Vrrs83PP5Xkl5IsJPmt5mvQNQAAAKCvslEAuT2Wl5frysrKtLsBAADADiilPFJrXR523LjVmAEAAGDmCbsAAAC0jrALAABA64xUoAoA2FknT6/mxKkzefrsWq5ZXMjRwwdz5NDStLsFwD7Rxr+HhF0AmLKTp1dz132PZW39XJJk9exa7rrvsSTZ8//QAGD2tfXvIdOYAWDKTpw6c+EfGB1r6+dy4tSZKfUIgP2krX8PCbsAMGVPn10bqx0AJqmtfw8JuwAwZdcsLozVDgCT1Na/h4RdAJiyo4cPZmF+7qK2hfm5HD18cEo9AmA/aevfQwpUAcCUdYp/tK0KJgB7Q1v/Hiq11mn3YaKWl5frysrKtLsBAADADiilPFJrXR52nGnMAAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOuMFHZLKYullI+VUr5QSvl8KeWvl1JeXUp5sJTyRPP9qubYUkr5QCnlyVLK50opb+g6z53N8U+UUu7sar+5lPJY85wPlFJK097zGgAAADDIqCO7/zzJf6y1fk+S1yf5fJJjST5Za31tkk82vyfJm5O8tvl6V5IPJhvBNcndSd6Y5PuS3N0VXj/YHNt53u1Ne79rAAAAQF9Dw24p5VuT/I0kv5gktdYXa61nk7w1yYebwz6c5Ejz81uTfKRueDjJYinlu5IcTvJgrfXZWutzSR5Mcnvz2LfWWn+31lqTfGTTuXpdAwAAAPoaZWT3ryR5Jsm/K6WcLqX8m1LKlUm+s9b61SRpvn9Hc/xSkq90Pf+ppm1Q+1M92jPgGgAAANDXKGH3siRvSPLBWuuhJF/P4OnEpUdb3UL7yEop7yqlrJRSVp555plxngoAAEALjRJ2n0ryVK3195rfP5aN8PsnzRTkNN+/1nX8a7qef22Sp4e0X9ujPQOucZFa64dqrcu11uWrr756hJcEAABAmw0Nu7XW/5LkK6WUg03TDyT5gyT3J+lUVL4zyW80P9+f5J1NVeZbkjzfTEE+leRNpZSrmsJUb0pyqnnsz0sptzRVmN+56Vy9rgEAAAB9XTbicX8/ya+UUi5P8sUkP5GNoPzRUspPJvlykrc1x34iyQ8meTLJC82xqbU+W0r5uSSfaY772Vrrs83PP5Xkl5IsJPmt5itJjve5BgAAAPRVNgogt8fy8nJdWVmZdjcAAADYAaWUR2qty8OOG3WfXQAAANgzhF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1hF0AAABaR9gFAACgdYRdAAAAWkfYBQAAoHWEXQAAAFpH2AUAAKB1Rgq7pZQ/KqU8Vkp5tJSy0rS9upTyYCnlieb7VU17KaV8oJTyZCnlc6WUN3Sd587m+CdKKXd2td/cnP/J5rll0DUAAABgkHFGdr+/1npTrXW5+f1Ykk/WWl+b5JPN70ny5iSvbb7eleSDyUZwTXJ3kjcm+b4kd3eF1w82x3aed/uQawAAAEBf25nG/NYkH25+/nCSI13tH6kbHk6yWEr5riSHkzxYa3221vpckgeT3N489q211t+ttdYkH9l0rl7XAAAAgL5GDbs1yW+XUh4ppbyrafvOWutXk6T5/h1N+1KSr3Q996mmbVD7Uz3aB10DAAAA+rpsxONurbU+XUr5jiQPllK+MODY0qOtbqF9ZE0Af1eSXHfddeM8FQAAgBYaaWS31vp08/1rSX49G2tu/6SZgpzm+9eaw59K8pqup1+b5Okh7df2aM+Aa2zu34dqrcu11uWrr756lJcEAABAiw0Nu6WUK0spf6nzc5I3JfnPSe5P0qmofGeS32h+vj/JO5uqzLckeb6ZgnwqyZtKKVc1hanelORU89ifl1Juaaowv3PTuXpdAwAAAPoaZRrzdyb59WY3oMuS/H+11v9YSvlMko+WUn4yyZeTvK05/hNJfjDJk0leSPITSVJrfbaU8nNJPtMc97O11mebn38qyS8lWUjyW81Xkhzvcw0AAADoq2wUQG6P5eXlurKyMu1uAAAAsANKKY90bYnb13a2HgIAAICZJOwCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6l027AwDA3nHy9GpOnDqTp8+u5ZrFhRw9fDBHDi1Nu1sAcAlhFwAYycnTq7nrvseytn4uSbJ6di133fdYkgi8AMwc05gBgJGcOHXmQtDtWFs/lxOnzkypRwDQn7ALAIzk6bNrY7UDwDQJuwDASK5ZXBirHQCmSdgFAEZy9PDBLMzPXdS2MD+Xo4cPTqlHANCfAlUAwEg6RahUYwZgLxB2AYCRHTm0JNwCsCeYxgwAAEDrCLsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALSOsAsAAEDr2GcXAGCbTp5ezYlTZ/L02bVcs7iQo4cP2o8YYMqEXQCAbTh5ejV33fdY1tbPJUlWz67lrvseSxKBF2CKTGMGANiGE6fOXAi6HWvr53Li1Jkp9QiARNgFANiWp8+ujdUOwO4QdgEAtuGaxYWx2gHYHcIuAMA2HD18MAvzcxe1LczP5ejhg1PqEQCJAlUAANvSKUKlGjPAbBF2AQC26cihJeEWYMaYxgwAAEDrCLsAAAC0zshht5QyV0o5XUr5zeb3G0opv1dKeaKUcm8p5fKm/RXN7082j1/fdY67mvYzpZTDXe23N21PllKOdbX3vAYAAAAMMs7I7k8n+XzX7/80yftrra9N8lySn2zafzLJc7XWv5rk/c1xKaV8b5IfS/K6JLcn+ZdNgJ5L8i+SvDnJ9yb58ebYQdcAAACAvkYKu6WUa5O8Jcm/aX4vSW5L8rHmkA8nOdL8/Nbm9zSP/0Bz/FuT/Gqt9Zu11i8leTLJ9zVfT9Zav1hrfTHJryZ565BrAAAAQF+jjuz+QpL/Pcn55vdvS3K21vpS8/tTSTolCJeSfCVJmsefb46/0L7pOf3aB10DAAAA+hoadksp/2OSr9VaH+lu7nFoHfLYpNp79fFdpZSVUsrKM8880+sQAAAA9pFRRnZvTfLDpZQ/ysYU49uyMdK7WErp7NN7bZKnm5+fSvKaJGkef1WSZ7vbNz2nX/ufDrjGRWqtH6q1Ltdal6+++uoRXhIAAABtNjTs1lrvqrVeW2u9PhsFph6qtb4jyaeS/Ghz2J1JfqP5+f7m9zSPP1RrrU37jzXVmm9I8tokv5/kM0le21Revry5xv3Nc/pdAwAAAPrazj67P5PkH5RSnszG+tpfbNp/Mcm3Ne3/IMmxJKm1Pp7ko0n+IMl/TPJ3a63nmjW5fy/JqWxUe/5oc+ygawAAAEBfZWMAtT2Wl5frysrKtLsBAADADiilPFJrXR523HZGdgEAAGAmCbsAAAC0jrALAABA6wi7AAAAtI6wCwAAQOsIuwAAALTOZdPuAAAwG06eXs2JU2fy9Nm1XLO4kKOHD+bIoaVpdwsAtkTYBQBy8vRq7rrvsaytn0uSrJ5dy133PZYkAi8Ae5JpzABATpw6cyHodqytn8uJU2em1CMA2B4juwA7xJRQ9pKnz66N1Q4As07YBdgBuzUlVKBmUq5ZXMhqj2B7zeLCFHoDANtnGjPADtiNKaGdQL16di01Lwfqk6dXJ3YN9o+jhw9mYX7uoraF+bkcPXxwSj0CgO0RdgF2wG5MCbXGkkk6cmgp99xxY5YWF1KSLC0u5J47bjRTAIA9yzRmgB2wG1NCrbFk0o4cWhJuAWgNI7sAO2A3poT2C87WWAIACLsAO2I3poRaYwkA0J9pzAA7ZKenhHbOrRozAMClhF2APcwaSwCA3oRdAOhi7+LRuVcAzDJhFwAanb2LO1s6dfYuTiLEbeJeATDrFKgCgIa9i0fnXgEw64RdAGjYu3h07hUAs07YBYCGvYtH514BMOuEXQBo2Lt4dO4VALNOgSoAaNi7eHTuFQCzrtRap92HiVpeXq4rKyvT7gYAAAA7oJTySK11edhxpjEDAADQOsIuAAAArSPsAgAA0DrCLgAAAK2jGjMwESdPr6rKOib3DABg5wi7wLadPL2au+57LGvr55Ikq2fXctd9jyWJ8NbHVu9ZJyCvnl3LXCk5V2uWBGUAgEuYxgxs24lTZy6Eto619XM5cerMlHo0+7ZyzzoBefXsWpLkXLN1XCconzy9unMdBgDYY4RdYNuebsLXqO1s7Z71CsgdPlwAALiYsAts2zWLC2O1s7V7NuzDAx8uAAC8TNgFtu3o4YNZmJ+7qK0k+f7vuXo6HdoDet2zhfm5HD18sO9zhn144MMFAICXCbvAth05tJQfuXkppautJvm1R1atI+3jyKGl3HPHjVlaXEhJsrS4kHvuuHFgkaleAbljWFAGANhvVGMGJuJTX3gmdVNbZx2pKsG9HTm0NNa96RyrGjMAwHDCLjAR+7FI1TT2yR03IE+bvYQBgGkRdoGJuGZx4cKWOJvb28jewsO5RwDANFmzC0zEVgou7WX2Fh7OPQIApsnILjAR3etJ98OU1f04bXtc7hEAME3CLjAxe2096Xbst2nbW+EeAeOwxh+YNNOYAbZgv03b3gr3CBhVZ43/6tm11Ly8xt/2dcB2CLsAW7CVfXL3G/cIGJU1/sBOMI0ZYIv207TtrXKPgFFY4w/sBCO7AABMVb+1/Nb4A9sh7AJjOXl6Nbcefyg3HHsgtx5/yHoqALbNGn9gJ5jGDIysU0Cks66qU0AkiamqAGzZftu+Dtgdwi4wskEFRPyDBIDtsMYfmDTTmIGRKSACAMBeYWQXGNk1iwtZ7RFs21BA5OTp1ZmcPrcb/ZrV1w4AsB1GdoGRtbWASGct8urZtdS8vBZ52sW3dqNfs/raAQC2S9iFFpt05eQjh5Zyzx03ZmlxISXJ0uJC7rnjxj0/CjhoLfI07Ua/RrmGCtwAwF5kGjO01E5VTm5jAZFZXYu8G/0adg0VuAGAvcrILrTUrI5WzqJ+a46nvRZ5N/o17Br+HAEAe5WwCy01q6OVs2hW1yLvRr+GXcOfIwBgrzKNGVqqzZWTJ60zHXfWKhLvRr+GXcOfIwBgryq11mn3YaKWl5frysrKtLsBU7d5rWWyMWLXhoJS7B5/jgCAWVNKeaTWujzsOCO70FKzOlrJ3uLPEQCwVxnZBQAAYM8YdWR3aIGqUsorSym/X0r5bCnl8VLK+5r2G0opv1dKeaKUcm8p5fKm/RXN7082j1/fda67mvYzpZTDXe23N21PllKOdbX3vAYAAAAMMko15m8mua3W+vokNyW5vZRyS5J/muT9tdbXJnkuyU82x/9kkudqrX81yfub41JK+d4kP5bkdUluT/IvSylzpZS5JP8iyZuTfG+SH2+OzYBrAAAAQF9Dw27d8BfNr/PNV01yW5KPNe0fTnKk+fmtze9pHv+BUkpp2n+11vrNWuuXkjyZ5PuarydrrV+stb6Y5FeTvLV5Tr9rAAAAQF8j7bPbjMA+muRrSR5M8odJztZaX2oOeSpJp1rJUpKvJEnz+PNJvq27fdNz+rV/24BrbO7fu0opK6WUlWeeeWaUlwQAAECLjVSNudZ6LslNpZTFJL+e5K/1Oqz5Xvo81q+9V+AedHyv/n0oyYeSjQJVvY4BYPJOnl5VqRkAmEljbT1Uaz1bSvmdJLckWSylXNaMvF6b5OnmsKeSvCbJU6WUy5K8KsmzXe0d3c/p1f6nA64BwJRt3oN39exa7rrvsSQReAGAqRulGvPVzYhuSikLSf6HJJ9P8qkkP9ocdmeS32h+vr/5Pc3jD9WN/Y3uT/JjTbXmG5K8NsnvJ/lMktc2lZcvz0YRq/ub5/S7BgBTduLUmQtBt2Nt/VxOnDozpR4BALxslJHd70ry4aZq8oEkH621/mYp5Q+S/Gop5eeTnE7yi83xv5jk/y2lPJmNEd0fS5Ja6+OllI8m+YMkLyX5u8306JRS/l6SU0nmkvzbWuvjzbl+ps81AJiyp8+ujdUOALCbhobdWuvnkhzq0f7FbFRS3tz+jSRv63Ouf5Lkn/Ro/0SST4x6DQCm75rFhaz2CLbXLC5MoTcAABcbqRozAGx29PDBLMzPXdS2MD+Xo4cPTqlHAAAvG6tAFbA3tblibptf26zr3OdJ3H/vIwAwacIutFybK+a2+bXtFUcOLW37XnsfAYCdYBoztFybK+a2+bXtJ95HAGAnCLvQcm2umNvm17afeB8BgJ1gGjO0XJsr5m7ltVkbuvNGucfdxxwoJedqveQ8bfgzCgBMj5FdaLk2V8wd97V11oaunl1LzctrQ0+eXt2F3u4Po9zjzcf0Crpt+TMKAEyPsAstd+TQUu6548YsLS6kJFlaXMg9d9zYitHMcV+btaE7b5R73OuYJJkrpXV/RgGA6TGNGfaBSVTMnVXjvDZrQ3feKPe43zHna82Xjr9lR/oFAOw/RnaBfaPfGlBrQydnlHvsfQAAdoOwC+wbbV6/PCtGucfeBwBgN5jGDOwbnenO41Rj3unqzTt1/kmcdyvnGOUeb+V9YOeoUA5AW5XaowrmXra8vFxXVlam3Q1ghmz1H/OdqsHdxZQW5ucmVjxpp84/ifPu9GtnNnifAdiLSimP1FqXhx1nGjPQatvZbminqzfv1PkncV6Vq/cH7zMAbWYaM0xJW6YOzvrrGPSP+WH93OnqzTt1/kmcV+Xq/cH7DECbGdmFKdjOaOMs2QuvYzv/mN/pqsE7df5JnFfF5P3B+wxAmwm7MAVtmTq4F17Hdv4xv9NVg8c5/8nTq7n1+EO54dgDufX4QwM/UJhEv1VM3h+8zwC0mWnMMAVtmTq4k69jUtOjjx4+2LMAzyj/mN/pqsGjnn9zEaHOCHr3OSbdbxWT9wfvMwBtphozTMGtxx/Kao9AuLS4kE8fu20KPdqanXodk64QO+vriodpy58XAIBJGLUas5FdmILtjDbOkp16HdspKtXLkUNLeyrcbtaWmQAAALtJ2IUpaMvUwZ16HcLdxa5ZXOg5squIEABAf8IuTMleH23s2InXIdxdrC0zAQAAdpNqzMDMUSH2YkcOLeWeO27M0uJCSjbW6m51/TIAwH6hQBUwdb0KSCV7f5o3AACTp0AVsCf021bnnjtuVGkYAIAtM40ZmKpBlZcBAGCrhF1gqlReBgBgJwi7wFT1q7C8XysvAwAwGcIuMFUqLwMAsBMUqAKmqlNhWeVlAAAmSdgFpu7IoaWJhtteWxkJzwAA+4uwC7RKv62Mkgi8AAD7iDW7QKvYyggAgETYBVrGVkYAACTCLtAytjICACARdoGWsZURAACJAlVAC2yuvvwjNy/lU194ZurVmFWFBgCYHmEX2NN6VV/+tUdWc88dN041WKoKDQAwXaYxA3varFZfntV+AQDsF8IusKfNavXlWe0XAMB+IewCe9qsVl+e1X4BAOwXwi5MwcnTq7n1+EO54dgDufX4Qzl5enXaXdqzZrX68qz2CwBgv1CgCnbZrBQumvVKwaP2r9O2269lWP+m1S8AADaUWuu0+zBRy8vLdWVlZdrdgL5uPf5QVnus21xaXMinj922K33YHLiTjVHHaVcw7tA/AAD6KaU8UmtdHnacaczsa9OYTjwLhYu2Uil4N+/VrFcynvX+AQBgGjP72LSmE1+zuNBzZHcnCxdtnnLb6/pJ/8C9k/eq13TgWfhAYJBZ7x8AAEZ22cemNTq324WLOkF19exaajaCaulzbL/AvVP3qlff7rrvsbxqYX6s/u02lZYBAGafsMu+Na3RuSOHlnLPHTdmaXEhJRtrdXdyrWevoFqTSwLvoMC9U/eqX4guJTNdyVilZQCA2WcaM/vWJKcTj1vZ+MihpV0rZNQvkNZsBO1R+rxTU6/79e3sC+t5/9tvmtlKxiotAwDMPmGXfevo4YM9K+qOOzo3K1sJ9dMvqI5T/XlS92rUvl2zuLCrHwhsxaz3DwBgvzONmX1rUtOJZ70y7ySm3O7U1GvTgQEA2Cn22YVtuuHYA+n1X1FJ8qXjb9nt7vQ07jTr3TrXOOfrd9yk+7PbBvV/r782AICdMOo+u8IubNOtxx/a9jThvWLzlO1kYyR2Jwts9btuklx5+VxefOl81s+//P+x3ejPpAy6n0mmcq8BAGbdqGHXNGbYpv00FXdaU7Z7XTdJvv7iuYuC7m71Z1IG3c9Znx4PADDrFKiCbdorlXm3MiV283N6jWAnO79d07jn3+n+TMpWtnTaK68NAGDahF2YgFmvzLuVitG9nlOSnuuTt7sF0TCDgna/4/uZpXWww7Z02ontngAA9gvTmGEf2MqU2F7PqdkovNVtN6Zs95oq3s+g/nQC/OrZtdS8HPpPnl6dYG9HN2gK/H6aHg8AsBOM7MI+MMnpsjUbxbdGrZ68enYtc6XkXK1Z2uJIauf4997/eM6urV/02PxcyZWXX5bn19aHjtQOCv3TGN0dZQr8+z7+eJ57YeM1v+Ky/f355CyNygMAs0/YhX1g2HTZcZ4zSpXpzVOgzzVV37unTyfjrXPuTBXfTuDZSujfacOmwH9j/fyFn8+urQ+dft5WW5mKDwDsb8Iu7ANHDx/suY3NoCmxW3lOR7/qycnGSOr7Pv54vrF+fkvBZTvro/sF+MUr5rd0vp02ayPR0+ReAADjGjonrpTymlLKp0opny+lPF5K+emm/dWllAdLKU80369q2ksp5QOllCdLKZ8rpbyh61x3Nsc/UUq5s6v95lLKY81zPlBKKYOuAYznyKGl3HPHjVlaXEjJxujssP1at/KcjmEjpc+9sD6VbXWOHj6Y+bnNq46Tv/jGS1NbtzvILI5ET4t7AQCMa5QFYC8l+Ye11r+W5JYkf7eG+hxgAAAgAElEQVSU8r1JjiX5ZK31tUk+2fyeJG9O8trm611JPphsBNckdyd5Y5LvS3J3V3j9YHNs53m3N+39rgF73snTq7n1+EO54dgDufX4QzsatgZN/R2lHzXJf3n+G3n3vY+O1NetVgzeTnDpvI7rjz2Q777rE7m+x+s5cmgpV15+6YSW9fN1Jvev7Xcf92NFZvcCABjX0GnMtdavJvlq8/Ofl1I+n2QpyVuT/M3msA8n+Z0kP9O0f6TWWpM8XEpZLKV8V3Psg7XWZ5OklPJgkttLKb+T5Ftrrb/btH8kyZEkvzXgGuxjbShSs1vrDzeu87msda373Lxutl8/Nj/Wa91tv772mgLdsTA/l1dcduCSQlPJ1oPLKGuEO319vsd1k9kcIdzOVPK2cS8AgHGNtWa3lHJ9kkNJfi/JdzZBOLXWr5ZSvqM5bCnJV7qe9lTTNqj9qR7tGXAN9qm9WqRmc0D/+jdfmvj6w83X+P7vuTr3/v5Xsn7+0p1xu6cMD5pOPGjd7aC+dlcZ7lWNOcnA4DLuBxrD1gh393XxivkL1Y27HSglJ0+vztSfo1GqNe8X7gUAMK6Rw24p5VuS/FqSd9da/2uzrLbnoT3aem3POax9ZKWUd2VjGnSuu+66cZ7KHrMXi9T0Cuj9dEYXxw17va7xKw9/eeB/SKP0Y5Bhx4xSSKp7K6FXzm+squj1Wt5z76NZ+eNn8/NHbtxSX7rva7+R3XO1zuQHJ9spyNU27gUAMI6RNm0spcxnI+j+Sq31vqb5T5rpyWm+f61pfyrJa7qefm2Sp4e0X9ujfdA1LlJr/VCtdbnWunz11VeP8pLYo/ZikZpBo46bXbO4cCHsrZ5dS83LYa/XGtRB1xjrE6Me/Rg2pXgSayW/+dLL06ufe2E977n30fyj+z7X87X8ysNf7rtWeNS+vu/jj6fHQPcFu1EkCwCA3TFKNeaS5BeTfL7W+s+6Hro/Saei8p1JfqOr/Z1NVeZbkjzfTEU+leRNpZSrmsJUb0pyqnnsz0sptzTXeuemc/W6BvvUXixSM2oQ70zjHRRcO9O2N4e+SYb9ko31kUcPH8z8gd4zOOYPlG2vlez3Ol/oWl+8+bF+QfTo4YNZmJ/r+Vj39Ohe05c326kPTnazIBkAAKON7N6a5H9Jclsp5dHm6weTHE/yt0opTyT5W83vSfKJJF9M8mSSf53kf0uSpjDVzyX5TPP1s51iVUl+Ksm/aZ7zh9koTpUB12Cf6hVqZr1ITb8gftUV85ds65MMnl6c9B59nGTYr3l5uui3vLL3Sof5uZITp870DG6jhrqthMp+z+neJilJ5pplFuNsl9SxEx+c9Bqt7/WhBQAAk1Nq3c5kx9mzvLxcV1ZWpt0NdtBeq8a8eQ1qshHQN4ewXscNsrS4kOu/bSEPf/G5C9WHuy3Mz+VHbl7Kb372qxfWxV51xXxqTc9KyB1XXTGfKy6/LE83wWwUndeT9C481Stw3nr8oaHBfrOlxYV8+thtYz2n203v++2Br71fX7er32vd7usBANiPSimP1FqXhx4n7MLOGyWgbyX89bPUVGP+1Beeuag686e+8MzErtHrmknvkelONebuqszf/z1X55cf/nLPc3UmT3f/32kSQfTk6dUc/Q+f7VmhemkHPzi54dgDPT84KEm+dPwtE78eAECbjRp2x9p6CPaKWRv9HaWK7KTWis6VcsmepKtn1/oGy0kZFKI7I8/d+9/+2iP9p/DWJL/w9psm/h5Oa/uaaxYXet6fWV5rDgCw1xnZpXVGnTY8a/qN7C4uzOfKV1w21ojsUp9wtR0lG+HshRdfGqnQ0yg6I72bbZ7eO2sfXoxrr/6ZBACYRaOO7I609RDsJYP24h3FJKrmbuUc/YpvvfeHX5dPH7vtwjThYeZKmXhF4aXFhXzp+Fvy6WO35e4fel3fysfjOlfr0IJjbSju1F1Aq7sgmaALALBzTGOmdbazF+/mEbhOsEoycjDZ6jmGTbHdPDW5nx9/42tGXps7V0rO1zpwxLazFVGvfm539LizTnbQqO2gDy/2UlgcZSo7AACTI+zSOttZHzmJYLWdcwwKRJtDZkkuKXp063e/Oj9/5Mb845OP5Vce/vLAasqbp9H2mmpbkrzjlusu6VOnn9spqtUZwR0WArfz4QUAAPuXacy0znb24p1EsNqtcNYryP7Rn63l5OnV/NojqwOD7uLC/CXTaHtNtX3/22/Kzx+5se95et3r+bmS+QPlorbOb732vx025bvfhxSKOwEAMIiRXVpnOxV3J1E1d6cq746yD+/TZ9d6jixvduUrNv7Tv/X4Q5fco3Gm2va7173aep13lCnfvaZvj/rhBQAA+5dqzNBlElVzd6ry7ihThpcWF/J0U8hpmIX5ualXB+73mtpWjRkAgMmxzy5swST2YR3nHOOEuGHToDujnaMWjpqFok+jTvlW3AkAgHEJu7DJJILVKOcYt2pzv+nRyctVjZPkhRdf2nK/R11XvJ2R1u7nHuizz27NxqivEVwAALZK2GVkppJO1qhVmzv3vVcF5u6px6Os6R1mlHXF29me6eTp1Rz92Gezfm7jVfQKuh1b2fYJAAA6hF1GMon9Z2fVtEL8KFN4N9/3mlwIvEub+jpKYapBRi36tJ2tld738ccvBN1uB0pyvkfu3c7Uah/OAADsb8IuI5nE/rOzaJohfpSqzb3ueyfodhdwSra3tdFcKSMXpxp3a6Xu0NlvHPd8Tc99gwedd5A2fzgDAMBo7LPLSHZr79jd9r6PP943xO+0UfYDHue+95uCfNUV85dcZ/M1/++//fqRQ+A4+952QufqCBWiJ7mf7qAPZwAA2B+EXUYyySAyK06eXs1zL6z3fGw3QvyRQ0u5544bs7S4kJKN0drNo6vj3Pd+4fnuH3rdRddZXJjPVVfM973mMKOE9I5Rp1YvLsyPdd5h2vrhDAAAozONmZEcPXyw596xWwkis2LQKN/iFfPbPv8oa0aHVW0e574P2/JoUtN3x9laaZRwOX+g5L0//LqJbPvU0W+K+IFScsOxB8Y69zTW/lpvDACwfcIuI5lkEJkVg4LYX3zjpZw8vbrl1zepNaM7tWfvdo26PVO/0DlXSs7X2jOQT6LPvT4kSF6u/jzq+zGNtb/WGwMATEapA7b+2IuWl5frysrKtLvBHnDr8Yf67lub9C4Ctd1zd8456WDaa9uh7m2JpmWa/RplP99h7/Gw93EnTOOaAAB7SSnlkVrr8rDjjOyyb/Ub/esYZQpuv9A6aM3oTozcbbda9k6NCk9zRkD3KPENxx7oecyw93gaa3+tNwYAmAxhl32rE4Tec++jPSsFv2ph8LrdQaF10LZCO7GN03YC0k5Pm53U1OTtGGWbp0k+bzumcU0AgDZSjZl97cihpb7FqEoZ/NxBoXVQZeFRg+nJ06u59fhDuf7YA/nuuz6R6489kFuPP5STp1cveW6/1zBKoa2d3qan8zpuGND/nbbVSs+TrBA9qmlcEwCgjYzssu/1237obJ/2ZCPA9Vvvu3p2LSt//GxecdmBCyHyqivmc/cPbVQcPnHqTM/nXnH5XG49/lCePruWVy3M5+svvpT1cxtjzsMKK/Vbej/KkvydnDY7K8WWtjqdehrTsNtYDA4AYBqEXfaMnVhXOmiUsd+00U6AG+SXH/7yRb9/Y/38hZ+PHj6Yox/77IUg2/H1F8/l6y9uBMyza/2Ddq8pz8/3Ob5fe7ednDa7E1O2t2qr06mnMQ17FqZ+AwDsdaYxsyd0Aubq2bXUvDxC2C+sjjp19n0ff7zvNftNG33v/Y/3LWrVT/e04COHlnLZgSFzpIfYPOraL5iOElh3ctqsYksAAEyLsMueMM660lGD8cnTq32nMCe9p9mePL06cNR1kE7AO3l6NWtdI71bsTnEbiewHjm0lHvuuDFLiwsp2djiZlJbA20nhE/CLKwXBgBgOkxjZk8YZ4RwlKmzo0xF7mU7RZsOlJIbjj2QA8MqXw3RK8Rud53nTk2b7bW9024VW5qV9cIAAEyHsMueMM660n7BePXs2oUCUAdKuVD0qZeF+QM91wj3K0o1is71Bl2324EkKcn5rsOXBoTYWVznOc1iS7O0XhgAgN0n7LInjDNC2C8Yl+RC+7DAeaCUnqOCBzaFz15Kkv/+u1+dP/qztZGCdccV8wdy1ZWv6FmNOXn59XYHtZ0o2jVp0wrh1gsDAOxv1uyyJ4yzrrTX+tWSZLTx1A1ff/Fcz1HBYUE3zXX+6M/WcvTwwVyzuDBS0F2Yn8v/ecd/m08fuy1fOv6WXPmKyy6p1ry2fu6iglrjFu3ab6a9XrgXa4gBAHaPkV32jFFHCHtNnd3O9OOtWD27lnff++jAY+ZKyflae47I9ht9fO6F9Zw8vXphv17TdPub5nrhXqwhBgDYXUZ2aaUjh5by6WO35f1vv2ns5w4qH7Uwf+CSUeOt+kuvvCzvf/tN+fSx2y4JO4NGHztFskzTHWwnq0xvxTgVxQEA2D4ju7TW5pG0QTrTnIdNd37pfM3b/7tr86kvPJOnz65l8Yr5/MU3Xsr6KPObNzm7tt53ZO/o4YN9R4Y7YXacol371SwV7fLhBADA7jKyy0zrXuN40/t+O4d+9rdHXu/YayStn5qNacXDIuv6uZpffvjL+S/PfyPvuOW6nP4/3pQTb3t9FhfmR3tBm6ytn8s/uu9zl7QfObTU95ydMNtvbXKn6rT1oLNlFtcQAwC0mZFdZtY/PvlYfuXhL18IoGfX1i88Nsp6x3FHzEbdEqhz7C8//OX8+n9azf/0hqV886XzY12r2wvr5/Pddz2Qb12Yz9kX1i+s4X3vD79u6JrTV84fuOjxziuwHnT2zNoaYgCAtit1jH/g7wXLy8t1ZWVl2t1gm06eXs177n106Ejr4sJ8rnzFZT233rn1+EO7XphqkPkDyfqImXhhfi733HFjkkv3qE2S93388Tz3wvqgUyTZWKf66WO3bbnPTNZe2CoKAGDWlVIeqbUuDz1O2GUWbTWodkLikUNLY63Z3Q2/8PabhlZo7tYrqI77mkqSLx1/yzjdBACAmTZq2DWNmZm01aI93VvvdEbM/uFHPzvWFOWdsLS4kCOHlkYare7orL19+uxaXrUwn1Iy0mhuN+tBAQDYrxSoYiZtJ6R1B+Ujh5ZyfgtB96or5jN/YNAmRKPrrMs8eXo1ZYxTdopN1WysVx436FoPCgDAfibsMpN6VRpOBu+B27E5KI8bnEuSu3/odTnxttePdL1B5+ne2/XEqTMZdYeiYVsg9X1eufS6AACwH5nGzEzqhLTNxXzeM2TN6/xcuWQ0s1cV3H5Kknfcct2F6w+73iCb18oOmpp9xfyBvGJ+7kI15q2sV54/UHLiba8XcAEAIMIuUzSsMm33utvOsUNHO+ul5168Yj7d46RXXj6XF186n/VNw6yLC/N57w+/7qI+bDV4XnXFpXvk9jtXr0JU4xbo6tV3AADYz0xjZio6VYU7a1I7+8KePL068Nhh1s/XvO/jj1907udeWM9a154/X3/xXObnyoVAOtcspL3yFZd+9tNvOvUg83Mld//Q6y55DV//5kuXHNtvXe24193OPr8AANBGwi5TceLUmUumFXcqKY9y7CAb4Xbw8S+sn8/zL6xnfq5cqNTcK3AfObSUe+64MUt91v0uzM/lf77luiwtLlxYK3viRy+eStwJ62fXLi4wddUV833X1XZft2Rj5PaqK+ZT8nI479bv3gEAwH5lGjNT0W/9ame7nc5o54lTZ7Y0jXgU55OcP3fxVOburYs6Oj9vXvdbkvzIzUv5+SM3DrxOv7B+xeWXDZx23D2Nu9sNxx7oefxWt2sCAIA2EnbZMYPW5A5aC7t6di1HP/bZpOaSdbXDbLWKcbfu0Nh5Db36WpN86gvPjHW+UdqH6Xfv7KkLAAAvM42ZHTFsTe6wNanr5+rIQbd0fd9u0E1eDo2jrBUeJbD2C6FbDae97p09dQEA4GLCLjti2JrcYWthBzlQkoX5AxfWyL7/7TdlaXGhb9C9ojl2FCW5aAr1sLW/owTWSYfTzet5h+2pe/L0am49/lBuOPZAbj3+UM8iYAAA0DamMbMjRpm621mTOu42OxsDviXvf/tNSYav660pecct1+Xez3wl6+cGj/3WvLxGd5RR21ECa789g7ezTVC/9bybdUanO6G9M8Le3S8AAGgjYZcdMWxdafd63lctzGd+rlwUROfnysA1u2vr5/K+jz+eb6yfHzr6urZ+Lp/6wjM58aOvv3DNA+XlKszdFhde3h932B67V10xP3JgHDWcTtqgEXZhFwCANjONmR0xaOru5vW8Z9fWk5oLW+tcdcV8rrz8sqyfrz232ekYZYuhjtWzazlyaCmfPnZbvnT8Lfm///brM3/g0nN//cWXRlpXvDA/d8leusNMYzrxpItjAQDAXiHsMlGdQPeeex/NK+cPZHFh/pJ1pe/7+OOXhNT18zW1JotXzOe5F9Yv7El7rtaR19sOUpq+dRw5tJRveeWlExvWz9W89/7HLxzTva64E7yHrZHtpVfBrvfc+2iu3+HgO+niWAAAsFeYxszEbF4f+twL61mYn8v7337ThWB48vRqnnthvefzOwF3s5pLKy1vjLjWrK2fH6lvNblk6u6gfpw8vXph6vEkpvv2mk7ceT3jrKMdtJ1TL0cPH7xkf2CVmwEA2A+M7DIxwyowd47ZippcUn34lQO2Lupl89TdQVOkT5w6M9Fpx8OmDW++T70M286p1/Gd92Q7o9IAALAXGdllYkZZH7rVtaJLiwv59LHbLmp7z72PjnWOzVN3exWo6ugEyUlVMR5W7CoZfm/GKTa1eZT9XK0XRnQFXQAA9gMju0zMKOtD+x0zYJD1kmm3nRHXflG1R92plGwE1u4R2kF7/M6VMnSUehyDil11DFtHO06xqVFG2QEAoM2EXSZmUAXmYce8443X9QyDiwvzF0277Z7K28vC/Fz+2d++Kb/w9psuhNnu9b7dU3+PHj7YsyLz/FzvbYk6z9/KlOYjh5byIzcv9S22Nco62nGKTanCDADAfmcaM32NWwyp89ig5ww6Zvkvv3ro9XqNWHYsbXrOkUNLufX4Q5cE484IZ2da9Hvvf/xCcayrrpjP3T/0upw4daZvoN7qlOZPfeGZnqPRc6XknjtuTJLcevyhvq9/nGJTw/Y5BgCAthN26Wnzms9RA94o1Yv7HTPKc/uNTJbkkjW9g47vtA+65uZg2a3fWtlB+vXlfDOKPOx+j/JhQocqzAAA7HfCLj2NUwxpN407YrnVEc7uYNlvhHfcKcGD+jLq/R51K6RxgjEAALSRNbv0NKtrPkdZF7yd47sdObSUTx+7rW8hq3GnBA/qy07c707/v3T8Lfn0sdsEXQAA9pWhYbeU8m9LKV8rpfznrrZXl1IeLKU80Xy/qmkvpZQPlFKeLKV8rpTyhq7n3Nkc/0Qp5c6u9ptLKY81z/lAKRt1eftdg8nrtZ/sOMWQdtORQ0u5544bL+y5u7gwn1fOH8h77n20Z+GozcdvZZ/ZUQLzKHvyDurLrN5vAADYq0odsNdokpRS/kaSv0jykVrrf9O0/V9Jnq21Hi+lHEtyVa31Z0opP5jk7yf5wSRvTPLPa61vLKW8OslKkuVsFMZ9JMnNtdbnSim/n+Snkzyc5BNJPlBr/a1+1xj2gpaXl+vKyspW7sW+tHltbrIR5H7k5qX82iOrl0yt7RRwmoVRwn59HzfMjnqtflOCe/WjJHnHLdfl54/cOHOvBQAA9rJSyiO11uWhxw0Lu83Jrk/ym11h90ySv1lr/Wop5buS/E6t9WAp5V81P//77uM6X7XWv9O0/6skv9N8farW+j1N+493jut3jWF9FXbH06tacfJyZePuSsXdFhfm87pr/lIe/uJzOVdr5krJLX/lqvzRn63l6bNredXCfEpJzr6wvqX1oqNUgh7U917FqnZKv36UJO9/+00jv+5xq18DAMB+NGrY3WqBqu+stX41SZow+h1N+1KSr3Qd91TTNqj9qR7tg67BBA1aK3rk0FJOnDrTM+yeXVvPp//w2Qu/n6v1ot+7nzPuVj2jVoIeZ51rryCZTKaAU79+1Ob8o55z1OJTAADAcJMuUFV6tNUttI930VLeVUpZKaWsPPPMM+M+fV8btlZ0UgWpOpWFR9GvMvG7N63LHXWdayc8r55dS81GeD76sc/m6H/47EVtd933WM+1tsMMWlc77YJeAACwX2017P5JM7U4zfevNe1PJXlN13HXJnl6SPu1PdoHXeMStdYP1VqXa63LV1999RZf0v40rPjSJAskjRr8Bh3XHUpHrbTcKzyvn6tZP3/x5yrjBPJuRw8f7PmpTaLAFAAATMtWw+79SToVle9M8htd7e9sqjLfkuT5ZiryqSRvKqVc1VRVflOSU81jf15KuaWpwvzOTefqdQ0maFi14lG26BnVqMFv2HHd+89u7vuP3Lwx9bq7KvI4o6tbGYk9cmgp77jluksC76hbHM2aUSpLAwDArBulGvO/z0aBqW9P8idJ7k5yMslHk1yX5MtJ3lZrfbYJrP9PktuTvJDkJ2qtK815/tck/6g57T+ptf67pn05yS8lWUjyW0n+fq21llK+rdc1hr0gBaq2rl+BpJve99s91+2OY5zKwu/417970frfXkqSLx1/y0Vt/Soav3L+QJ57YbT+b6e4VRsKTKkKDQDArJtoNea9RNjdmkEhJ0nPrXX+6ndcmS8+88JEqzGfPL2a99z76NCF271Cab+qyIsL8/nmS+cvmcq8mVA3OxWu94o2fMABALDX7HQ1ZlqmX1GoE6fOXAg5u/GP+hOnzgwNut3Tg7vDRr/nPb+2nve//aa8+95H+55zaRuvqU2BZ5wK1/vdqFXDAQCYDmGXJMNDzm5tizMsVC0tLuT7v+fqnDh1Ju++99GUDC/f/aqF+YGFp7Y7dblNgeeaxYWeI7sKbV1q0AdEe/G9BwBom0lvPcQeNeo2PtPqR0nyC2+/KUcPH8yvPbJ6IZANC7rzB0q+/uJLPQNcsv0iUoMCz140aoVrjIIDAMw6YZcksxNyevWjJHnHLdflyKGlnuGyl0515m955WVZP9c7Em+uPL0VbQs8w6pz87JZ+YAIAIDeTGMmyctTbqe99nRYP0YJkd3Tkm849kDPY0rSd+ryOGtw2zjtd7emrO91Rw8f7FnUzSg4AMBsEHa5YJIhZztFmwb1o1+47NgcNsYNo+OuwRV49q9Z+YAIAIDebD3ExPXaxmh+ruTKyy/L82vjbUU0yrk7Rap6VVQed9/YrWy906ZqzAAAMOtsPcTU9FpXu36u5uzaepLtVSwedzRt3OO3sgbXtF8AAJg9wi4TN8q62u1s0TJuuBzn+DauwQUAgP1INWYmbtRgOIsVi2elKjUAALA9wi4T1ysw9jKLo6W23gEAgHYwjZmJ27xOdvGK+fzFN17K+vmXi6HN8mipNbgAALD3CbvsiM2BUcViAABgNwm77AqjpQAAwG6yZhcAAIDWMbK7S0zjBQAA2D3C7i44eXo1d933WNbWzyVJVs+u5a77HksSgRcAAGAHmMa8C06cOnMh6HasrZ/LiVNnptQjAACAdhN2d8HTZ9fGagcAAGB7hN1dcM3iwljtAAAAbI+wuwuOHj6Yhfm5i9oW5udy9PDBKfVo+06eXs2txx/KDcceyK3HH8rJ06vT7hIAAMAFClTtgk4RqrZUY1ZwCwAAmHXC7i45cmipNUFwUMGttrxGAABgbzONmbEpuAUAAMw6YZexKbgFAADMOmGXsbWx4BYAANAu1uwytrYV3AIAANpH2GVL2lRwCwAAaB/TmAEAAGgdI7vMtJOnV02XBgAAxibsMrNOnl7NXfc9dmFP39Wza7nrvseSROAFAAAGMo2ZmXXi1JkLQbdjbf1cTpw6M6UeAQAAe4Wwy8x6+uzaWO0AAAAdwi4z65rFhbHaAQAAOoRdZtbRwwezMD93UdvC/FyOHj44pR4BAAB7hQJVzKxOESrVmAEAgHEJu8y0I4eWhFsAAGBspjEDAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6wi4AAACtI+wCAADQOsIuAAAArSPsAgAA0DrCLgAAAK0j7AIAANA6pdY67T5MVCnlmSR/PO1+MJZvT/Kn0+4E2+Z93Pu8h3uf97AdvI97n/ewHbyPs+sv11qvHnZQ68Iue08pZaXWujztfrA93se9z3u493kP28H7uPd5D9vB+7j3mcYMAABA6wi7AAAAtI6wyyz40LQ7wER4H/c+7+He5z1sB+/j3uc9bAfv4x5nzS4AAACtY2QXAACA1hF22ZZSyr8tpXytlPKfu9peXUp5sJTyRPP9qqa9lFI+UEp5spTyuVLKG7qec2dz/BOllDu72m8upTzWPOcDpZQy6BqMr5TymlLKp0opny+lPF5K+emm3fu4R5RSXllK+f1Symeb9/B9TfsNpZTfa+7vvaWUy5v2VzS/P9k8fn3Xue5q2s+UUg53td/etD1ZSjnW1d7zGmxNKWWulHK6lPKbze/ewz2mlPJHzf/vHi2lrDRt/n+6h5RSFkspHyulfKH5u/Gvew/3llLKwea/wc7Xfy2lvNv7uA/VWn39/+3cS2hdVRTG8f/CNFXb2qT1QTBiGyhiBtKGogmVIj4CCdJRBwlCiw8EdSIOpKHgXAfSgWIDijjw/S6FEqXViYOora221miihYamjVVaxZGP5WCvpMdwc8u9TczdN98PFmffdfa5u5wFh+7cfbai6gA2Ax3A0ULuGWBHtHcAT0e7F9gHGNAJDEd+FfBjHJuj3RznPge64pp9QE+5MRRV1bAF6Ij2CuB7oF11zCfivi6P9hJgOGrzFpRdU7EAAAQLSURBVNAX+d3AI9F+FNgd7T7gzWi3A0eApcBaYAy4LGIMaAMao097XFNyDEXVtXwCeA3YW+7+qoa1G8AJ4OoZOT1PMwrgFeChaDcCTaphvhHPv9PAjarj4osF/wco8g9gDf+d7I4ALdFuAUaiPQj0z+wH9AODhfxg5FqA7wr56X6zjaGYk3p+CNyjOuYZwJXAIeA24CzQEPkuYCjaQ0BXtBuinwEDwEDhu4biuulrIz8QYbONoaiqdq3AfuBOYG+5+6sa1m5QerKr52kmAVwF/ETsa6Ma5h9AN/CZ6rg4Q8uYZT5c5+4TAHG8NvLXAycL/cYjVy4/XiJfbgy5BLEUcgPpl0HVMSOx/PUwMAl8TPoV75y7/xVdivd9ulZx/jywmspru7rMGFK5XcCTwD/xudz9VQ1rlwMfmdlBM3s4cnqe5qMN+Bl42dIrBS+a2TJUw5z1Aa9HW3VcZDTZlf+Tlch5FXmZB2a2HHgXeNzdfyvXtUROdVxg7v63u68n/Tp4K3BzqW5xnKsaqrZzxMzuBSbd/WAxXaKralj7Nrl7B9ADPGZmm8v0Vb1qTwPp9awX3H0D8AdpKepsVMMaFnsQbAHevljXEjnVsQ5osivz4YyZtQDEcTLy48ANhX6twKmL5FtL5MuNIVUwsyWkie6r7v5epFXHDLn7OeBT0jtHTWbWEKeK9326VnF+JfArldf2bJkxpDKbgC1mdgJ4g7SUeReqYXbc/VQcJ4H3SX980vM0H+PAuLsPx+d3SJNf1TBPPcAhdz8Tn1XHRUaTXZkPe4Dt0d5Oegd0Kr8tdrzrBM7H8o4hoNvMmmPHum7SO2MTwO9m1hk73G2b8V2lxpAKxb19CTju7s8WTqmOmTCza8ysKdpXAHcDx4FPgK3RbWYNp+77VuCAu3vk+yzt9LsWWEfagOMLYJ2lXXsbSUvC9sQ1s40hFXD3AXdvdfc1pPt7wN3vQzXMipktM7MVU23Sc/Aoep5mw91PAyfN7KZI3QV8i2qYq34uLGEG1XHxWeiXhhV5B+kBMgH8Sfor14Okd8D2Az/EcVX0NeB50ruE3wAbC9/zADAacX8hv5H0H4Ux4Dliw4jZxlBUVcPbSUtvvgYOR/SqjvkEcAvwVdTwKPBU5NtIE51R0hKupZG/PD6Pxvm2wnftjDqNEDtLRr6XtFP3GLCzkC85huKS6nkHF3ZjVg0ziriXRyKOTd1nPU/zCmA98GU8Uz8g7cKrGmYWpA0bfwFWFnKq4yKLqaKIiIiIiIiI1A0tYxYREREREZG6o8muiIiIiIiI1B1NdkVERERERKTuaLIrIiIiIiIidUeTXREREREREak7muyKiIiIiIhI3dFkV0REREREROqOJrsiIiIiIiJSd/4Fxrt+K+1jQ1QAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 1152x648 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(Y_test, predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Ensembles\n",
"<img src=\"./media/forest.jpg\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bagging - independent learners\n",
"https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Boosting - sequential learners, each one learns from previous mistakes\n",
"https://xgboost.readthedocs.io/en/latest/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gradient Boosting\n",
"\n",
"Best ever explanation of gradient boosting https://explained.ai/gradient-boosting \n",
"\n",
"TL;DR http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/\n",
"\n",
"<img src=\"./media/boosting.png\">"
]
},
{
"cell_type": "code",
"execution_count": 421,
"metadata": {},
"outputs": [],
"source": [
"df_simple = df[['YrSold', 'YearBuilt', 'SalePrice']].head(n=10).copy()\n",
"X = df_simple.drop('SalePrice', axis='columns').values\n",
"Y = df_simple.SalePrice.values"
]
},
{
"cell_type": "code",
"execution_count": 422,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>YrSold</th>\n",
" <th>YearBuilt</th>\n",
" <th>SalePrice</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2008</td>\n",
" <td>2003</td>\n",
" <td>208500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2007</td>\n",
" <td>1976</td>\n",
" <td>181500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2008</td>\n",
" <td>2001</td>\n",
" <td>223500</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006</td>\n",
" <td>1915</td>\n",
" <td>140000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2008</td>\n",
" <td>2000</td>\n",
" <td>250000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>2009</td>\n",
" <td>1993</td>\n",
" <td>143000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>2007</td>\n",
" <td>2004</td>\n",
" <td>307000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>2009</td>\n",
" <td>1973</td>\n",
" <td>200000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>2008</td>\n",
" <td>1931</td>\n",
" <td>129900</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>2008</td>\n",
" <td>1939</td>\n",
" <td>118000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" YrSold YearBuilt SalePrice\n",
"0 2008 2003 208500\n",
"1 2007 1976 181500\n",
"2 2008 2001 223500\n",
"3 2006 1915 140000\n",
"4 2008 2000 250000\n",
"5 2009 1993 143000\n",
"6 2007 2004 307000\n",
"7 2009 1973 200000\n",
"8 2008 1931 129900\n",
"9 2008 1939 118000"
]
},
"execution_count": 422,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_simple"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Fit a model - F(x)"
]
},
{
"cell_type": "code",
"execution_count": 423,
"metadata": {},
"outputs": [],
"source": [
"ensemble = []"
]
},
{
"cell_type": "code",
"execution_count": 424,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
"<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
" -->\r\n",
"<!-- Title: G Pages: 1 -->\r\n",
"<svg width=\"350pt\" height=\"116pt\"\r\n",
" viewBox=\"0.00 0.00 349.84 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\r\n",
"<title>G</title>\r\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 345.836,-112 345.836,4 -4,4\"/>\r\n",
"<!-- 10 (2000, &#39;LotArea&#39;) 190140.0 -->\r\n",
"<g id=\"node1\" class=\"node\"><title>10 (2000, &#39;LotArea&#39;) 190140.0</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"192.691\" cy=\"-90\" rx=\"119.679\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"192.691\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">10 (2000, &#39;LotArea&#39;) 190140.0</text>\r\n",
"</g>\r\n",
"<!-- 6 () 152066.66666666666 -->\r\n",
"<g id=\"node2\" class=\"node\"><title>6 () 152066.66666666666</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"102.691\" cy=\"-18\" rx=\"102.882\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"102.691\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">6 () 152066.66666666666</text>\r\n",
"</g>\r\n",
"<!-- 10 (2000, &#39;LotArea&#39;) 190140.0&#45;&#45;6 () 152066.66666666666 -->\r\n",
"<g id=\"edge1\" class=\"edge\"><title>10 (2000, &#39;LotArea&#39;) 190140.0&#45;&#45;6 () 152066.66666666666</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M170.904,-72.055C156.754,-61.0492 138.387,-46.7636 124.277,-35.789\"/>\r\n",
"</g>\r\n",
"<!-- 4 () 247250.0 -->\r\n",
"<g id=\"node3\" class=\"node\"><title>4 () 247250.0</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"282.691\" cy=\"-18\" rx=\"59.2899\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"282.691\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">4 () 247250.0</text>\r\n",
"</g>\r\n",
"<!-- 10 (2000, &#39;LotArea&#39;) 190140.0&#45;&#45;4 () 247250.0 -->\r\n",
"<g id=\"edge2\" class=\"edge\"><title>10 (2000, &#39;LotArea&#39;) 190140.0&#45;&#45;4 () 247250.0</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M214.477,-72.055C228.94,-60.8064 247.807,-46.1317 262.032,-35.0676\"/>\r\n",
"</g>\r\n",
"</g>\r\n",
"</svg>\r\n"
],
"text/plain": [
"<graphviz.dot.Graph at 0x1648bcedba8>"
]
},
"execution_count": 424,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ensemble.append(create_tree(X, Y, np.arange(X.shape[0]), max_depth=1, col_proc=1))\n",
"plot_tree(ensemble[-1])"
]
},
{
"cell_type": "code",
"execution_count": 439,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x1648c0af6d8>"
]
},
"execution_count": 439,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7sAAAIMCAYAAAA0IruPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3W2MpWd93/Hfv34glijY4IWCbWqTOAininiYOm7TtClpbYPS2G1J5SiKV8SqFUoqSBOKHaSQpxcQJ0G1mhBRYWFHNIaAA34B2qwITd5gwxgDxnEcL+SBtV1sapu4wgVMrr4415LjZWd2Z7zemfnv5yMdzZnr3OfMfe5r7jn7nXPPvTXGCAAAAHTy97Z6BQAAAOBoE7sAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtHPiVq/A0Xb66aePs88+e6tXAwAAgKfAbbfd9uUxxq7DLdcuds8+++ysrq5u9WoAAADwFKiqvzqS5RzGDAAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAOydu9QoAwPHmg7ffm2v23J37Hnkszz/1lLzxohfl0peesdWrdUzZBjufOYReOu7TYhcAjqEP3n5vrr7pjjz2jW8mSe595LFcfdMdSbLj/1FxpGyDnc8cQi9d92mHMQPAMXTNnru/9Y+JAx77xjdzzZ67t2iNjj3bYOczh9BL131a7ALAMXTfI49taLwj22DnM4fQS9d9WuwCwDH0/FNP2dB4R7bBzmcOoZeu+7TYBYBj6I0XvSinnHTCE8ZOOemEvPGiF23RGh17tsHOZw6hl677tBNUAcAxdOBEH93OeLkRtsHOZw6hl677dI0xtnodjqqVlZWxurq61asBAADAU6CqbhtjrBxuOYcxAwAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0M5hY7eqzqqqj1XVXVV1Z1W9/qDbf66qRlWdPj+vqrq2qvZV1Wer6mVLy+6uqnvmZffS+Mur6o55n2urqub4s6pq71x+b1WddvSeOgAAAF0dyTu7jyf52THGi5NckOR1VXVesgjhJP86yV8vLf/KJOfOy5VJ3jGXfVaStyT5viTnJ3nLUry+Yy574H4Xz/Grknx0jHFuko/OzwEAAGBdh43dMcb9Y4xPzeuPJrkryRnz5rcn+a9JxtJdLklyw1i4JcmpVfW8JBcl2TvGeGiM8XCSvUkunrc9Y4zx8THGSHJDkkuXHuv6ef36pXEAAABY04b+Zreqzk7y0iS3VtWPJLl3jPGZgxY7I8kXlz7fP8fWG99/iPEkee4Y4/5kEd1JnrOR9QUAAOD4dOKRLlhVT0/ygSRvyOLQ5jcnufBQix5ibGxi/IhV1ZVZHAadF7zgBRu5KwAAAA0d0Tu7VXVSFqH7njHGTUm+M8k5ST5TVX+Z5Mwkn6qqf5DFO7NnLd39zCT3HWb8zEOMJ8mX5mHOmR8fONT6jTHeOcZYGWOs7Nq160ieEgAAAI0dydmYK8m7ktw1xvjNJBlj3DHGeM4Y4+wxxtlZBOvLxhj/O8nNSS6fZ2W+IMlX5iHIe5JcWFWnzRNTXZhkz7zt0aq6YH6ty5N8aH75m5McOGvz7qVxAAAAWNORHMb8/Ul+IskdVfXpOfbzY4wPr7H8h5O8Ksm+JF9N8pokGWM8VFW/kuSTc7lfHmM8NK+/Nsm7k5yS5CPzkiRvTfK+qroiizM+/+gRPi8AAACOY7U4AXIfKysrY3V1datXAwAAgKdAVd02xlg53HIbOhszAAAA7ARiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaOWzsVtVZVfWxqrqrqu6sqtfP8Wuq6s+q6rNV9QdVderSfa6uqn1VdXdVXbQ0fvEc21dVVy2Nn1NVt1bVPVX13qo6eY4/bX6+b95+9tF88gAAAPR0JO/sPp7kZ8cYL05yQZLXVdV5SfYm+UdjjO9N8udJrk6SedtlSb4nycVJfruqTqiqE5L8VpJXJjkvyY/NZZPkbUnePsY4N8nDSa6Y41ckeXiM8V1J3j6XAwAAgHUdNnbHGPePMT41rz+a5K4kZ4wx/nCM8fhc7JYkZ87rlyS5cYzxtTHGXyTZl+T8edk3xvjCGOPrSW5McklVVZJXJHn/vP/1SS5deqzr5/X3J/mhuTwAAACsaUN/szsPI35pklsPuuknk3xkXj8jyReXbts/x9Yaf3aSR5bC+cD4Ex5r3v6VuTwAAACs6Yhjt6qenuQDSd4wxvibpfE3Z3Go83sODB3i7mMT4+s91sHrdmVVrVbV6oMPPrj2kwAAAOC4cESxW1UnZRG67xlj3LQ0vjvJDyf58THGgQjdn+SspbufmeS+dca/nOTUqjrxoPEnPNa8/ZlJHjp4/cYY7xxjrIwxVnbt2nUkTwkAAIDGjuRszJXkXUnuGmP85tL4xUnelORHxhhfXbrLzUkum2dSPifJuUk+keSTSc6dZ14+OYuTWN08I/ljSV497787yYeWHmv3vP7qJH+0FNUAAABwSCcefpF8f5KfSHJHVX16jv18kmuTPC3J3nnOqFvGGD81xrizqt6X5E+zOLz5dWOMbyZJVf10kj1JTkhy3Rjjzvl4b0pyY1X9apLbs4jrzI+/W1X7snhH97In9WwBAAA4LlS3N0pXVlbG6urqVq8GAAAAT4Gqum2MsXK45TZ0NmYAAADYCcQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGjnsLFbVWdV1ceq6q6qurOqXj/Hn1VVe6vqnvnxtDleVXVtVe2rqs9W1cuWHmv3XP6eqtq9NP7yqrpj3ufaqqr1vgYAAACs50je2X08yc+OMV6c5IIkr6uq85JcleSjY4xzk3x0fp4kr0xy7rxcmeQdySJck7wlyfclOT/JW5bi9R1z2QP3u3iOr/U1AAAAYE2Hjd0xxv1jjE/N648muSvJGUkuSXL9XOz6JJfO65ckuWEs3JLk1Kp6XpKLkuwdYzw0xng4yd4kF8/bnjHG+PgYYyS54aDHOtTXAAAAgDVt6G92q+rsJC9NcmuS544x7k8WQZzkOXOxM5J8celu++fYeuP7DzGedb4GAAAArOmIY7eqnp7kA0neMMb4m/UWPcTY2MT4EauqK6tqtapWH3zwwY3cFQAAgIaOKHar6qQsQvc9Y4yb5vCX5iHImR8fmOP7k5y1dPczk9x3mPEzDzG+3td4gjHGO8cYK2OMlV27dh3JUwIAAKCxIzkbcyV5V5K7xhi/uXTTzUkOnFF5d5IPLY1fPs/KfEGSr8xDkPckubCqTpsnprowyZ5526NVdcH8Wpcf9FiH+hoAAACwphOPYJnvT/ITSe6oqk/PsZ9P8tYk76uqK5L8dZIfnbd9OMmrkuxL8tUkr0mSMcZDVfUrST45l/vlMcZD8/prk7w7ySlJPjIvWedrAAAAwJpqcQLkPlZWVsbq6upWrwYAAABPgaq6bYyxcrjlNnQ2ZgAAANgJxC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtHPY2K2q66rqgar63NLYS6rqlqr6dFWtVtX5c7yq6tqq2ldVn62qly3dZ3dV3TMvu5fGX15Vd8z7XFtVNcefVVV75/J7q+q0o/vUAQAA6OpI3tl9d5KLDxr7tSS/NMZ4SZJfmJ8nySuTnDsvVyZ5R7II1yRvSfJ9Sc5P8paleH3HXPbA/Q58rauSfHSMcW6Sj87PAQAA4LAOG7tjjD9J8tDBw0meMa8/M8l98/olSW4YC7ckObWqnpfkoiR7xxgPjTEeTrI3ycXztmeMMT4+xhhJbkhy6dJjXT+vX780DgAAAOs6cZP3e0OSPVX161kE8z+d42ck+eLScvvn2Hrj+w8xniTPHWPcnyRjjPur6jmbXFcAAACOM5s9QdVrk/zMGOOsJD+T5F1zvA6x7NjE+IZU1ZXzb4dXH3zwwY3eHQAAgGY2G7u7k9w0r/9+Fn+HmyzemT1rabkzszjEeb3xMw8xniRfmoc5Z358YK2VGWO8c4yxMsZY2bVr16aeEAAAAH1sNnbvS/Iv5vVXJLlnXr85yeXzrMwXJPnKPBR5T5ILq+q0eWKqC5Psmbc9WlUXzLMwX57kQ0uPdeCszbuXxgEAAGBdh/2b3ar6vSQ/mOT0qtqfxVmV/2OS/1ZVJyb5f1mcTTlJPpzkVUn2JflqktckyRjjoar6lSSfnMv98hjjwEmvXpvFGZ9PSfKReUmStyZ5X1VdkeSvk/zopp8lAAAAx5VanAS5j5WVlbG6urrVqwEAAMBToKpuG2OsHG65zR7GDAAAANuW2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7Rw2dqvquqp6oKo+d9D4f66qu6vqzqr6taXxq6tq37ztoqXxi+fYvqq6amn8nKq6taruqar3VtXJc/xp8/N98/azj8YTBgAAoL8jeWf33UkuXh6oqn+Z5JIk3zvG+J4kvz7Hz0tyWZLvmff57ao6oapOSPJbSV6Z5LwkPzaXTZK3JXn7GOPcJA8nuWKOX5Hk4THGdyV5+1wOAAAADuuwsTvG+JMkDx00/Nokbx1jfG0u88AcvyTJjWOMr40x/iLJviTnz8u+McYXxhhfT3JjkkuqqpK8Isn75/2vT3Lp0mNdP6+/P8kPzeUBAABgXZv9m93vTvID8/DiP66qfzzHz0jyxaXl9s+xtcafneSRMcbjB40/4bHm7V+Zy3+bqrqyqlaravXBBx/c5FMCAACgi83G7olJTktyQZI3JnnffNf1UO+8jk2M5zC3PXFwjHeOMVbGGCu7du063LoDAADQ3GZjd3+Sm8bCJ5L8bZLT5/hZS8udmeS+dca/nOTUqjrxoPEs32fe/sx8++HUAAAA8G02G7sfzOJvbVNV353k5CzC9eYkl80zKZ+T5Nwkn0jyySTnzjMvn5zFSaxuHmOMJB9L8ur5uLuTfGhev3l+nnn7H83lAQAAYF0nHm6Bqvq9JD+Y5PSq2p/kLUmuS3Ld/O+Ivp5k9wzRO6vqfUn+NMnjSV43xvjmfJyfTrInyQlJrhtj3Dm/xJuS3FhVv5rk9iTvmuPvSvK7VbUvi3d0LzsKzxcAAIDjQHV7s3RlZWWsrq5u9WoAAADwFKiq28YYK4dbbrOHMQMAAMC2JXYBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGhH7AIAANCO2AUAAKAdsQsAAEA7YhcAAIB2xC4AAADtiF0AAADaEbsAAAC0I3YBAABoR+wCAADQjtgFAACgHbELAABAOydu9QocTz54+725Zs/due+Rx/L8U0/JGy96US596RlbvVqbspOey2bWdSc9P44PviefHNsPAI4/YvcY+eDt9+bqm+7IY9/4ZpLk3kcey9U33ZEkO+4fXDvpuWxmXXfS8+P44HvyybH9AOD45DDmY+SaPXd/6x9aBzz2jW/mmj13b9Eabd5Oei6bWded9Pw4PviefHJsPwA4PondY+S+Rx7b0Ph2tpOey2bWdSc9P44PviefHNsPAI5PYvcYef6pp2xofDvbSc9lM+u6k54fxwffk0+O7QcAxyexe4y88aIX5ZSTTnjC2CknnZA3XvSiLVqjzdtJz2Uz67qTnh/HB9+TT47tBwDHJyeoOkYOnASlw9lAd9Jz2cy67qTnx/HB9+STY/sBwPGpxhhbvQ5H1crKylhdXd3q1QAAAOApUFW3jTFWDrecw5gBAABoR+wCAADQjtgFAACgHbELAABAO2IXAACAdsQuAAAA7YhdAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO2IXQAAANoRuwAAALQjdgEAAGinxhhbvQ5HVVU9mOSvtno9tqnTk3x5q1eCI2Kudg5ztbOYr53DXO0c5mpnMV87h7la2z8cY+w63ELtYpe1VdXqGGNlq9eDwzNXO4e52lnM185hrnYOc7WzmK+dw1w9eQ5jBgAAoB2xCwAAQDti9/jyzq1eAY6Yudo5zNXOYr52DnO1c5irncV87Rzm6knyN7sAAAC0451dAAAA2hG7O0BVXVdVD1TV55bGrqmqP6uqz1bVH1TVqXP87Kp6rKo+PS+/s3Sfl1fVHVW1r6quraqa48+qqr1Vdc/8eNocr7ncvvl1Xnasn/tOs8Zc/WJV3bs0J69auu3quX3vrqqLlsYvnmP7quqqpfFzqurWOVfvraqT5/jT5uf75u1nH5tnvHOtMVfvXZqnv6yqT89x+9UWq6qzqupjVXVXVd1ZVa+f4xvezlW1ey5/T1XtXhrf0FxyaOvMldetbWadufK6tQ2tM19eu7aZqvqOqvpEVX1mztUvzfEN7w9Ha587bo0xXLb5Jck/T/KyJJ9bGrswyYnz+tuSvG1eP3t5uYMe5xNJ/kmSSvKRJK+c47+W5Kp5/aqlx3rVXK6SXJDk1q3eFtv9ssZc/WKSnzvEsucl+UySpyU5J8nnk5wwL59P8sIkJ89lzpv3eV+Sy+b130ny2nn9PyX5nXn9siTv3eptsd0vh5qrg27/jSS/MK/br7Z+vp6X5GXz+t9P8udzH9rQdk7yrCRfmB9Pm9dP28xcumx4rrxubbPLOnP1i/G6te0ua83XQct47doGl7mdnj6vn5Tk1rndNrQ/HM197ni9eGd3Bxhj/EmShw4a+8MxxuPz01uSnLneY1TV85I8Y4zx8bH47r8hyaXz5kuSXD+vX3/Q+A1j4ZYkp87HYQ2Hmqt1XJLkxjHG18YYf5FkX5Lz52XfGOMLY4yvJ7kxySXzt66vSPL+ef+D5+rAHL4/yQ8d+C0th7beXM1t9x+S/N56j2G/OnbGGPePMT41rz+a5K4kZ2Tj2/miJHvHGA+NMR5OsjfJxZucSw5hrbnyurX9rLNfrcXr1hY63Hx57do+5nb6v/PTk+ZlZOP7w9Hc545LYreHn8zit20HnFNVt1fVH1fVD8yxM5LsX1pmf/7uB+Rzxxj3J4sfpEmes3SfL65xHzbmp+dhP9fV3x0Cudb2XWv82UkeWfrH4vJ8fOs+8/avzOXZnB9I8qUxxj1LY/arbWIe3vXSLH5TvtHtvN74RueSwzhorpZ53dpmDjFXXre2sTX2La9d20hVnTAPKX8gi1+sfj4b3x+O5j53XBK7O1xVvTnJ40neM4fuT/KCMcZLk/yXJP+zqp6RxeEUBzvcqbg3cx++3TuSfGeSl2QxP78xx9favhsdX++x2JwfyxN/M26/2iaq6ulJPpDkDWOMv1lv0UOMbXb/YhPWmiuvW9vPIebK69Y2ts7PQa9d28gY45tjjJdkcRTL+UlefKjF5sejtW+Zp4OI3R2sFidW+eEkPz4PQ8k8zOH/zOu3ZfFbpO/O4jc7y4eMnZnkvnn9SwcORZkfH5jj+5OctcZ9OEJjjC/NH3h/m+R/ZPEDL1l7+641/uUsDhs68aDxJzzWvP2ZOfLDqVkyt9+/S/LeA2P2q+2hqk7K4h947xlj3DSHN7qd1xvf6FyyhjXmyuvWNnSoufK6tX2ts2957dqmxhiPJPlfWfzN7kb3h6O5zx2XxO4OVVUXJ3lTkh8ZY3x1aXxXVZ0wr78wyblJvjAPRXm0qi6Yx/NfnuRD8243JzlwRtLdB41fXgsXJPnKgUNbOHIH/U3Lv01y4Oy/Nye5rBZn4Dsni7n6RJJPJjl3nk3v5CxOVHDz/Ifhx5K8et7/4Lk6MIevTvJHB/4hyYb9qyR/Nsb41iFe9qutN7fvu5LcNcb4zaWbNrqd9yS5sKpOm4dmXphkzybnkkNYa668bm0/68yV161taJ2fg4nXrm1lbvsDZ5w/JYv5uSsb3x+O5j53fBrb4CxZLutfsjgk5f4k38jiNzlXZPEH6l9M8ul5OXAGt3+f5M4szsr2qST/ZulxVrJ4wfp8kv+epOb4s5N8NMk98+Oz5ngl+a25/B1JVrZ6W2z3yxpz9btz+302ix9az1ta/s1z+96deSbEOf6qLM6y+Pkkb14af2EWP+T2Jfn9JE+b498xP983b3/hVm+L7X451FzN8Xcn+amDlrVfbf18/bMsDsX67NLPvVdtZjtn8fei++blNZudS5cNz5XXrW12WWeuvG5tw8ta8zVve3e8dm2bS5LvTXL7nKvP5e/OkL3h/eELHuL1AAAARUlEQVRo7XPH6+XANzYAAAC04TBmAAAA2hG7AAAAtCN2AQAAaEfsAgAA0I7YBQAAoB2xCwAAQDtiFwAAgHbELgAAAO38fwjmB8pOP7yLAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1152x648 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(Y, [predict_tree(ensemble[0], sample) for sample in X])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Compute residuals"
]
},
{
"cell_type": "code",
"execution_count": 425,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>YrSold</th>\n",
" <th>YearBuilt</th>\n",
" <th>SalePrice</th>\n",
" <th>p1</th>\n",
" <th>r1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2008</td>\n",
" <td>2003</td>\n",
" <td>208500</td>\n",
" <td>247250.000000</td>\n",
" <td>-38750.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2007</td>\n",
" <td>1976</td>\n",
" <td>181500</td>\n",
" <td>152066.666667</td>\n",
" <td>29433.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2008</td>\n",
" <td>2001</td>\n",
" <td>223500</td>\n",
" <td>247250.000000</td>\n",
" <td>-23750.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006</td>\n",
" <td>1915</td>\n",
" <td>140000</td>\n",
" <td>152066.666667</td>\n",
" <td>-12066.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2008</td>\n",
" <td>2000</td>\n",
" <td>250000</td>\n",
" <td>247250.000000</td>\n",
" <td>2750.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>2009</td>\n",
" <td>1993</td>\n",
" <td>143000</td>\n",
" <td>152066.666667</td>\n",
" <td>-9066.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>2007</td>\n",
" <td>2004</td>\n",
" <td>307000</td>\n",
" <td>247250.000000</td>\n",
" <td>59750.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>2009</td>\n",
" <td>1973</td>\n",
" <td>200000</td>\n",
" <td>152066.666667</td>\n",
" <td>47933.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>2008</td>\n",
" <td>1931</td>\n",
" <td>129900</td>\n",
" <td>152066.666667</td>\n",
" <td>-22166.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>2008</td>\n",
" <td>1939</td>\n",
" <td>118000</td>\n",
" <td>152066.666667</td>\n",
" <td>-34066.666667</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" YrSold YearBuilt SalePrice p1 r1\n",
"0 2008 2003 208500 247250.000000 -38750.000000\n",
"1 2007 1976 181500 152066.666667 29433.333333\n",
"2 2008 2001 223500 247250.000000 -23750.000000\n",
"3 2006 1915 140000 152066.666667 -12066.666667\n",
"4 2008 2000 250000 247250.000000 2750.000000\n",
"5 2009 1993 143000 152066.666667 -9066.666667\n",
"6 2007 2004 307000 247250.000000 59750.000000\n",
"7 2009 1973 200000 152066.666667 47933.333333\n",
"8 2008 1931 129900 152066.666667 -22166.666667\n",
"9 2008 1939 118000 152066.666667 -34066.666667"
]
},
"execution_count": 425,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predictions = np.array([predict_tree(ensemble[-1], sample) for sample in X])\n",
"df_simple['p1'] = predictions\n",
"df_simple['r1'] = Y - predictions\n",
"df_simple"
]
},
{
"cell_type": "code",
"execution_count": 426,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1068688333.3333334, 0.029237410094233335)"
]
},
"execution_count": 426,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mean_squared_error(Y, predictions), mean_squared_log_error(Y, predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fit more learner on residuals - h(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./media/boosted_stumps.gif\">"
]
},
{
"cell_type": "code",
"execution_count": 427,
"metadata": {},
"outputs": [],
"source": [
"next_learner = 2"
]
},
{
"cell_type": "code",
"execution_count": 432,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
"<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
" -->\r\n",
"<!-- Title: G Pages: 1 -->\r\n",
"<svg width=\"426pt\" height=\"116pt\"\r\n",
" viewBox=\"0.00 0.00 426.18 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\r\n",
"<title>G</title>\r\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 422.182,-112 422.182,4 -4,4\"/>\r\n",
"<!-- 10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12 -->\r\n",
"<g id=\"node1\" class=\"node\"><title>10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"210.391\" cy=\"-90\" rx=\"174.369\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"210.391\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12</text>\r\n",
"</g>\r\n",
"<!-- 8 () &#45;6518.055555555562 -->\r\n",
"<g id=\"node2\" class=\"node\"><title>8 () &#45;6518.055555555562</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"101.391\" cy=\"-18\" rx=\"101.282\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"101.391\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">8 () &#45;6518.055555555562</text>\r\n",
"</g>\r\n",
"<!-- 10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12&#45;&#45;8 () &#45;6518.055555555562 -->\r\n",
"<g id=\"edge1\" class=\"edge\"><title>10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12&#45;&#45;8 () &#45;6518.055555555562</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M184.005,-72.055C166.742,-60.9683 144.295,-46.5533 127.158,-35.5475\"/>\r\n",
"</g>\r\n",
"<!-- 2 () 26072.22222222222 -->\r\n",
"<g id=\"node3\" class=\"node\"><title>2 () 26072.22222222222</title>\r\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"319.391\" cy=\"-18\" rx=\"98.5829\" ry=\"18\"/>\r\n",
"<text text-anchor=\"middle\" x=\"319.391\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">2 () 26072.22222222222</text>\r\n",
"</g>\r\n",
"<!-- 10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12&#45;&#45;2 () 26072.22222222222 -->\r\n",
"<g id=\"edge2\" class=\"edge\"><title>10 (2009, &#39;YrSold&#39;) &#45;5.820766091346741e&#45;12&#45;&#45;2 () 26072.22222222222</title>\r\n",
"<path fill=\"none\" stroke=\"black\" d=\"M236.777,-72.055C254.04,-60.9683 276.487,-46.5533 293.624,-35.5475\"/>\r\n",
"</g>\r\n",
"</g>\r\n",
"</svg>\r\n"
],
"text/plain": [
"<graphviz.dot.Graph at 0x1648b6f7668>"
]
},
"execution_count": 432,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ensemble.append(create_tree(X, df_simple[f'r{next_learner - 1}'].values, np.arange(X.shape[0]), max_depth=1, col_proc=1))\n",
"plot_tree(ensemble[-1])"
]
},
{
"cell_type": "code",
"execution_count": 433,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>YrSold</th>\n",
" <th>YearBuilt</th>\n",
" <th>SalePrice</th>\n",
" <th>p1</th>\n",
" <th>r1</th>\n",
" <th>p2</th>\n",
" <th>r2</th>\n",
" <th>p3</th>\n",
" <th>r3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2008</td>\n",
" <td>2003</td>\n",
" <td>208500</td>\n",
" <td>247250.000000</td>\n",
" <td>-38750.000000</td>\n",
" <td>240611.111111</td>\n",
" <td>-32111.111111</td>\n",
" <td>234093.055556</td>\n",
" <td>-25593.055556</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2007</td>\n",
" <td>1976</td>\n",
" <td>181500</td>\n",
" <td>152066.666667</td>\n",
" <td>29433.333333</td>\n",
" <td>145427.777778</td>\n",
" <td>36072.222222</td>\n",
" <td>138909.722222</td>\n",
" <td>42590.277778</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2008</td>\n",
" <td>2001</td>\n",
" <td>223500</td>\n",
" <td>247250.000000</td>\n",
" <td>-23750.000000</td>\n",
" <td>240611.111111</td>\n",
" <td>-17111.111111</td>\n",
" <td>234093.055556</td>\n",
" <td>-10593.055556</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006</td>\n",
" <td>1915</td>\n",
" <td>140000</td>\n",
" <td>152066.666667</td>\n",
" <td>-12066.666667</td>\n",
" <td>145427.777778</td>\n",
" <td>-5427.777778</td>\n",
" <td>138909.722222</td>\n",
" <td>1090.277778</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2008</td>\n",
" <td>2000</td>\n",
" <td>250000</td>\n",
" <td>247250.000000</td>\n",
" <td>2750.000000</td>\n",
" <td>240611.111111</td>\n",
" <td>9388.888889</td>\n",
" <td>234093.055556</td>\n",
" <td>15906.944444</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>2009</td>\n",
" <td>1993</td>\n",
" <td>143000</td>\n",
" <td>152066.666667</td>\n",
" <td>-9066.666667</td>\n",
" <td>145427.777778</td>\n",
" <td>-2427.777778</td>\n",
" <td>171500.000000</td>\n",
" <td>-28500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>2007</td>\n",
" <td>2004</td>\n",
" <td>307000</td>\n",
" <td>247250.000000</td>\n",
" <td>59750.000000</td>\n",
" <td>307000.000000</td>\n",
" <td>0.000000</td>\n",
" <td>300481.944444</td>\n",
" <td>6518.055556</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>2009</td>\n",
" <td>1973</td>\n",
" <td>200000</td>\n",
" <td>152066.666667</td>\n",
" <td>47933.333333</td>\n",
" <td>145427.777778</td>\n",
" <td>54572.222222</td>\n",
" <td>171500.000000</td>\n",
" <td>28500.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>2008</td>\n",
" <td>1931</td>\n",
" <td>129900</td>\n",
" <td>152066.666667</td>\n",
" <td>-22166.666667</td>\n",
" <td>145427.777778</td>\n",
" <td>-15527.777778</td>\n",
" <td>138909.722222</td>\n",
" <td>-9009.722222</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>2008</td>\n",
" <td>1939</td>\n",
" <td>118000</td>\n",
" <td>152066.666667</td>\n",
" <td>-34066.666667</td>\n",
" <td>145427.777778</td>\n",
" <td>-27427.777778</td>\n",
" <td>138909.722222</td>\n",
" <td>-20909.722222</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" YrSold YearBuilt SalePrice p1 r1 p2 \\\n",
"0 2008 2003 208500 247250.000000 -38750.000000 240611.111111 \n",
"1 2007 1976 181500 152066.666667 29433.333333 145427.777778 \n",
"2 2008 2001 223500 247250.000000 -23750.000000 240611.111111 \n",
"3 2006 1915 140000 152066.666667 -12066.666667 145427.777778 \n",
"4 2008 2000 250000 247250.000000 2750.000000 240611.111111 \n",
"5 2009 1993 143000 152066.666667 -9066.666667 145427.777778 \n",
"6 2007 2004 307000 247250.000000 59750.000000 307000.000000 \n",
"7 2009 1973 200000 152066.666667 47933.333333 145427.777778 \n",
"8 2008 1931 129900 152066.666667 -22166.666667 145427.777778 \n",
"9 2008 1939 118000 152066.666667 -34066.666667 145427.777778 \n",
"\n",
" r2 p3 r3 \n",
"0 -32111.111111 234093.055556 -25593.055556 \n",
"1 36072.222222 138909.722222 42590.277778 \n",
"2 -17111.111111 234093.055556 -10593.055556 \n",
"3 -5427.777778 138909.722222 1090.277778 \n",
"4 9388.888889 234093.055556 15906.944444 \n",
"5 -2427.777778 171500.000000 -28500.000000 \n",
"6 0.000000 300481.944444 6518.055556 \n",
"7 54572.222222 171500.000000 28500.000000 \n",
"8 -15527.777778 138909.722222 -9009.722222 \n",
"9 -27427.777778 138909.722222 -20909.722222 "
]
},
"execution_count": 433,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predictions = np.zeros(len(Y))\n",
"for t in ensemble:\n",
" predictions += [predict_tree(t, sample) for sample in X]\n",
"df_simple[f'p{next_learner}'] = predictions\n",
"df_simple[f'r{next_learner}'] = Y - predictions\n",
"df_simple"
]
},
{
"cell_type": "code",
"execution_count": 434,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(502074529.3209877, 0.017968490219020167)"
]
},
"execution_count": 434,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mean_squared_error(Y, predictions), mean_squared_log_error(Y, predictions)"
]
},
{
"cell_type": "code",
"execution_count": 435,
"metadata": {},
"outputs": [],
"source": [
"next_learner += 1"
]
},
{
"cell_type": "code",
"execution_count": 441,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x1648c5d9eb8>"
]
},
"execution_count": 441,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7sAAAIMCAYAAAA0IruPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X+wZnWdH/j3Jw04veVqo7RGG9jGmZYSk1nBO9gbdzaJZujWyg5k1tnCSkmXoULFaEpnHSJo1TCjUxUcJlKh1mGKLJRguYOMEqCyWp0uJDP/CHoRtEWGcNWJNBDBNCApOyrMd/94vj3zcOf+bBvuvd9+vapOPef5nO85zznP95779Pue83y7WmsBAACAkfyNtd4BAAAAONqEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIZz3FrvwNF20kknte3bt6/1bgAAAPA8uPvuu3/QWtu6XLvhwu727dszOzu71rsBAADA86Cq/vNK2rmNGQAAgOEIuwAAAAxH2AUAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhiPsAgAAMBxhFwAAgOEIuwAAAAxH2AUAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhiPsAgAAMBxhFwAAgOEsG3ar6ueq6itV9fWquq+qfqfXT6uqu6rqwar6bFWd0Osv6s/n+vLtU9u6tNcfqKpdU/XdvTZXVZdM1Rd8DQAAAI6eW+55OG++/Es57ZL/L2++/Eu55Z6H13qXfmYrubL74yRvaa39z0nekGR3Ve1M8vEkV7bWdiR5IsmFvf2FSZ5orf1Ckit7u1TVGUnOT/L6JLuT/EFVbaqqTUk+meRtSc5I8s7eNku8BgAAAEfBLfc8nEtv3p+HnzyUluThJw/l0pv3b/jAu2zYbRP/rT89vk8tyVuSfK7Xr09yXp8/tz9PX/7Wqqpev7G19uPW2neTzCU5u09zrbXvtNZ+kuTGJOf2dRZ7DQAAAI6CK/Y+kEM/ffY5tUM/fTZX7H1gjfbo6FjRd3b7Fdh7kzyWZF+Sbyd5srX2TG9yIMm2Pr8tyUNJ0pc/leTl0/V56yxWf/kSrwEAAMBR8MiTh1ZV3yhWFHZba8+21t6Q5ORMrsS+bqFm/bEWWXa06n9NVV1UVbNVNfv4448v1AQAAIAFvHrL5lXVN4pVjcbcWnsyyX9MsjPJlqo6ri86Ockjff5AklOSpC9/aZKD0/V56yxW/8ESrzF/v65prc201ma2bt26mkMCAAA4pl286/RsPn7Tc2qbj9+Ui3edvkZ7dHSsZDTmrVW1pc9vTvIPktyf5I4k7+jN9iS5tc/f1p+nL/9Sa631+vl9tObTkuxI8pUkX02yo4+8fEImg1jd1tdZ7DUAAAA4Cs47c1v+1a/97WzbsjmVZNuWzflXv/a3c96ZG/tbpMct3ySvSnJ9HzX5byS5qbX276vqW0lurKrfTXJPkmt7+2uTfLqq5jK5ont+krTW7quqm5J8K8kzSd7bWns2SarqfUn2JtmU5LrW2n19Wx9a5DUAAAA4Ss47c9uGD7fz1eQC6jhmZmba7OzsWu8GAAAAz4Oquru1NrNcu1V9ZxcAAAA2AmEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABjOsmG3qk6pqjuq6v6quq+q3t/rb6iqO6vq3qqaraqze72q6qqqmquqb1TVWVPb2lNVD/Zpz1T9jVW1v69zVVVVr7+sqvb19vuq6sSj/xYAAAAwmpVc2X0myQdba69LsjPJe6vqjCS/l+R3WmtvSPJb/XmSvC3Jjj5dlOTqZBJck1yW5E1Jzk5y2VR4vbq3Pbze7l6/JMntrbUdSW7vzwEAAGBJy4bd1tqjrbWv9fmnk9yfZFuSluQlvdlLkzzS589NckObuDPJlqp6VZJdSfa11g621p5Isi/J7r7sJa21L7fWWpIbkpw3ta3r+/z1U3UAAABY1HGraVxV25OcmeSuJB9Isreqfj+T0Px3erNtSR6aWu1Ary1VP7BAPUle2Vp7NJmE7qp6xWr2FwAAgGPTigeoqqoXJ/l8kg+01n6Y5D1JfqO1dkqS30hy7eGmC6zejqC+YlV1Uf/e8Ozjjz++mlUBAAAY0IrCblUdn0nQ/Uxr7eZe3pPk8PwfZ/I93GRyZfaUqdVPzuQW56XqJy9QT5Lv99uc0x8fW2j/WmvXtNZmWmszW7duXckhAQAAMLCVjMZcmVy1vb+19ompRY8k+bt9/i1JHuzztyW5oI/KvDPJU/1W5L1JzqmqE/vAVOck2duXPV1VO/trXZDk1qltHR61ec9UHQAAABa1ku/svjnJu5Lsr6p7e+3DSf5pkn9TVccl+e+ZjKacJF9I8vYkc0l+lOTdSdJaO1hVH0vy1d7uo621g33+PUk+lWRzki/2KUkuT3JTVV2Y5HtJfv0IjhEAAIBjTE0GQB7HzMxMm52dXevdAAAA4HlQVXe31maWa7fiAaoAAABgoxB2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhLBt2q+qUqrqjqu6vqvuq6v1Ty/5FVT3Q6783Vb+0qub6sl1T9d29NldVl0zVT6uqu6rqwar6bFWd0Osv6s/n+vLtR+vAAQAAGNdKruw+k+SDrbXXJdmZ5L1VdUZV/f0k5yb5xdba65P8fpJU1RlJzk/y+iS7k/xBVW2qqk1JPpnkbUnOSPLO3jZJPp7kytbajiRPJLmw1y9M8kRr7ReSXNnbAQAAwJKWDbuttUdba1/r808nuT/JtiTvSXJ5a+3HfdljfZVzk9zYWvtxa+27SeaSnN2nudbad1prP0lyY5Jzq6qSvCXJ5/r61yc5b2pb1/f5zyV5a28PAAAAi1rVd3b7bcRnJrkryWuT/HK/vfhPquqXerNtSR6aWu1Ary1Wf3mSJ1trz8yrP2dbfflTvT0AAAAs6riVNqyqFyf5fJIPtNZ+WFXHJTkxk1ubfynJTVX1miQLXXltWThYtyXaZ5ll0/t2UZKLkuTUU09d5kgAAAAY3Yqu7FbV8ZkE3c+01m7u5QNJbm4TX0nyF0lO6vVTplY/OckjS9R/kGRLD8/T9Uyv05e/NMnB+fvXWrumtTbTWpvZunXrSg4JAACAga1kNOZKcm2S+1trn5hadEsm37VNVb02yQmZBNfbkpzfR1I+LcmOJF9J8tUkO/rIyydkMojVba21luSOJO/o292T5NY+f1t/nr78S709AAAALGoltzG/Ocm7kuyvqnt77cNJrktyXVV9M8lPkuzpQfS+qropybcyGcn5va21Z5Okqt6XZG+STUmua63d17f3oSQ3VtXvJrknk3Cd/vjpqprL5Iru+T/T0QIAAHBMqNEulM7MzLTZ2dm13g0AAACeB1V1d2ttZrl2qxqNGQAAADYCYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGM6yYbeqTqmqO6rq/qq6r6reP2/5b1ZVq6qT+vOqqquqaq6qvlFVZ0213VNVD/Zpz1T9jVW1v69zVVVVr7+sqvb19vuq6sSjd+gAAACMaiVXdp9J8sHW2uuS7Ezy3qo6I5kE4SS/kuR7U+3flmRHny5KcnVv+7IklyV5U5Kzk1w2FV6v7m0Pr7e71y9JcntrbUeS2/tzAAAAWNKyYbe19mhr7Wt9/ukk9yfZ1hdfmeRfJmlTq5yb5IY2cWeSLVX1qiS7kuxrrR1srT2RZF+S3X3ZS1prX26ttSQ3JDlvalvX9/nrp+oAAACwqFV9Z7eqtic5M8ldVfWrSR5urX19XrNtSR6aen6g15aqH1igniSvbK09mkxCd5JXrGZ/AQAAODYdt9KGVfXiJJ9P8oFMbm3+SJJzFmq6QK0dQX3FquqiTG6DzqmnnrqaVQEAABjQiq7sVtXxmQTdz7TWbk7y80lOS/L1qvrzJCcn+VpV/c1MrsyeMrX6yUkeWaZ+8gL1JPl+v805/fGxhfavtXZNa22mtTazdevWlRwSAAAAA1vJaMyV5Nok97fWPpEkrbX9rbVXtNa2t9a2ZxJYz2qt/ZcktyW5oI/KvDPJU/0W5L1JzqmqE/vAVOck2duXPV1VO/trXZDk1v7ytyU5PGrznqk6AAAALGoltzG/Ocm7kuyvqnt77cOttS8s0v4LSd6eZC7Jj5K8O0laawer6mNJvtrbfbS1drDPvyfJp5JsTvLFPiXJ5UluqqoLMxnx+ddXeFwAAAAcw2oyAPI4ZmZm2uzs7FrvBgAAAM+Dqrq7tTazXLtVjcYMAAAAG4GwCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGM5xa70DAHCsueWeh3PF3gfyyJOH8uotm3PxrtNz3pnb1nq3XlDeg41PHwLrnbALAC+gW+55OJfevD+HfvpskuThJw/l0pv3J8kxExS8BxufPgQ2ArcxA8AL6Iq9D/xlQDjs0E+fzRV7H1ijPXrheQ82Pn0IbATCLgC8gB558tCq6iPyHmx8+hDYCIRdAHgBvXrL5lXVR+Q92Pj0IbARCLsA8AK6eNfp2Xz8pufUNh+/KRfvOn2N9uiF5z3Y+PQhsBEYoAoAXkCHB+85lkex9R5sfPoQ2AiqtbbW+3BUzczMtNnZ2bXeDQAAAJ4HVXV3a21muXZuYwYAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhiPsAgAAMBxhFwAAgOEIuwAAAAxH2AUAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhiPsAgAAMBxhFwAAgOEIuwAAAAxH2AUAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhiPsAgAAMBxhFwAAgOEIuwAAAAxH2AUAAGA4wi4AAADDEXYBAAAYjrALAADAcIRdAAAAhrNs2K2qU6rqjqq6v6ruq6r39/oVVfVnVfWNqvp3VbVlap1Lq2quqh6oql1T9d29NldVl0zVT6uqu6rqwar6bFWd0Osv6s/n+vLtR/PgAQAAGNNKruw+k+SDrbXXJdmZ5L1VdUaSfUn+VmvtF5P8pySXJklfdn6S1yfZneQPqmpTVW1K8skkb0tyRpJ39rZJ8vEkV7bWdiR5IsmFvX5hkidaa7+Q5MreDgAAAJa0bNhtrT3aWvtan386yf1JtrXW/kNr7Zne7M4kJ/f5c5Pc2Fr7cWvtu0nmkpzdp7nW2ndaaz9JcmOSc6uqkrwlyef6+tcnOW9qW9f3+c8leWtvDwAAAIta1Xd2+23EZya5a96if5Lki31+W5KHppYd6LXF6i9P8uRUcD5cf862+vKnensAAABY1IrDblW9OMnnk3ygtfbDqfpHMrnV+TOHSwus3o6gvtS25u/bRVU1W1Wzjz/++OIHAQAAwDFhRWG3qo7PJOh+prV281R9T5J/mOQft9YOh9ADSU6ZWv3kJI8sUf9Bki1Vddy8+nO21Ze/NMnB+fvXWrumtTbTWpvZunXrSg4JAACAga1kNOZKcm2S+1trn5iq707yoSS/2lr70dQqtyU5v4+kfFqSHUm+kuSrSXb0kZdPyGQQq9t6SL4jyTv6+nuS3Dq1rT19/h1JvjQVqgEAAGBBxy3fJG9O8q4k+6vq3l77cJKrkrwoyb4+ZtSdrbV/1lq7r6puSvKtTG5vfm9r7dkkqar3JdmbZFOS61pr9/XtfSjJjVX1u0nuySRcpz9+uqrmMrmie/7PdLQAAAAcE2q0C6UzMzNtdnZ2rXcDAACA50FV3d1am1mu3apGYwYAAICNQNgFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADCcZcNuVZ1SVXdU1f1VdV9Vvb/XX1ZV+6rqwf54Yq9XVV1VVXNV9Y2qOmtqW3t6+weras9U/Y1Vtb+vc1VV1VKvAQAAAEtZyZXdZ5J8sLX2uiQ7k7y3qs5IckmS21trO5Lc3p8nyduS7OjTRUmuTibBNcllSd6U5Owkl02F16t728Pr7e71xV4DAAAAFrVs2G2tPdpa+1qffzrJ/Um2JTk3yfW92fVJzuvz5ya5oU3cmWRLVb0qya4k+1prB1trTyTZl2R3X/aS1tqXW2styQ3ztrXQawAAAMCiVvWd3aranuTMJHcleWVr7dFkEoiTvKI325bkoanVDvTaUvUDC9SzxGvM36+Lqmq2qmYff/zx1RwSAAAAA1px2K2qFyf5fJIPtNZ+uFTTBWrtCOor1lq7prU201qb2bp162pWBQAAYEArCrtVdXwmQfczrbWbe/n7/Rbk9MfHev1AklOmVj85ySPL1E9eoL7UawAAAMCiVjIacyW5Nsn9rbVPTC26LcnhEZX3JLl1qn5BH5V5Z5Kn+i3Ie5OcU1Un9oGpzkmyty97uqp29te6YN62FnoNAAAAWNRxK2jz5iTvSrK/qu7ttQ8nuTzJTVV1YZLvJfn1vuwLSd6eZC7Jj5K8O0laawer6mNJvtrbfbS1drDPvyfJp5JsTvLFPmWJ1wAAAIBF1WQA5HHMzMy02dnZtd4NAAAAngdVdXdrbWa5dqsajRkAAAA2AmEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABjOsmG3qq6rqseq6ptTtTdU1Z1VdW9VzVbV2b1eVXVVVc1V1Teq6qypdfZU1YN92jNVf2NV7e/rXFVV1esvq6p9vf2+qjrx6B46AAAAo1rJld1PJdk9r/Z7SX6ntfaGJL/VnyfJ25Ls6NNFSa5OJsE1yWVJ3pTk7CSXTYXXq3vbw+sdfq1LktzeWtuR5Pb+HAAAAJa1bNhtrf1pkoPzy0le0udfmuSRPn9ukhvaxJ1JtlTVq5LsSrKvtXawtfZEkn1JdvdlL2mtfbm11pLckOS8qW1d3+evn6oDAADAko47wvU+kGRvVf1+JoH57/T6tiQPTbU70GtL1Q8sUE+SV7bWHk2S1tqjVfWKxXamqi7K5OpwTj311CM8JAAAAEZxpANUvSfJb7TWTknyG0mu7fVaoG07gvqqtNauaa3NtNZmtm7dutrVAQAAGMyRht09SW7u83+cyfdwk8mV2VOm2p2cyS3OS9VPXqCeJN/vtzmnPz52hPsKAADAMeZIw+4jSf5un39Lkgf7/G1JLuijMu9M8lS/FXlvknOq6sQ+MNU5Sfb2ZU9X1c4+CvMFSW6d2tbhUZv3TNUBAABgSct+Z7eq/ijJ30tyUlUdyGRU5X+a5N9U1XFJ/nv692WTfCHJ25PMJflRkncnSWvtYFV9LMlXe7uPttYOD3r1nkxGfN6c5It9SpLLk9xUVRcm+V6SXz/iowQAAOCYUpNBkMcxMzPTZmdn13o3AAAAeB5U1d2ttZnl2h3pbcwAAACwbgm7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGc9xa7wCsR7fc83Cu2PtAHnnyUF69ZXMu3nV6zjtz21rvFnCEnNMAcOwRdmGeW+55OJfevD+HfvpskuThJw/l0pv3J4l/HMMG5JwGgGOT25hhniv2PvCX/yg+7NBPn80Vex9Yoz0CfhbOaQA4Ngm7MM8jTx5aVR1Y35zTAHBsEnZhnldv2byqOrC+OacB4Ngk7MI8F+86PZuP3/Sc2ubjN+XiXaev0R4BPwvnNAAcmwxQBfMcHrDGyK0wBuc0ABybqrW21vtwVM3MzLTZ2dm13g0AAACeB1V1d2ttZrl2bmMGAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAwxF2AQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhLBt2q+q6qnqsqr45r/4vquqBqrqvqn5vqn5pVc31Zbum6rt7ba6qLpmqn1ZVd1XVg1X12ao6oddf1J/P9eXbj8YBAwAAML6VXNn9VJLd04Wq+vtJzk3yi6211yf5/V4/I8n5SV7f1/mDqtpUVZuSfDLJ25KckeSdvW2SfDzJla21HUmeSHJhr1+Y5InW2i8kubK3AwAAgGUtG3Zba3+a5OC88nuSXN5a+3Fv81ivn5vkxtbaj1tr300yl+TsPs211r7TWvtJkhuTnFtVleQtST7X178+yXlT27q+z38uyVt7ewAAAFjSkX5n97VJfrnfXvwnVfVLvb4tyUNT7Q702mL1lyd5srX2zLz6c7bVlz/V2wMAAMCSjvsZ1jsxyc4kv5Tkpqp6TZKFrry2LByq2xLts8yy56iqi5JclCSnnnrqkjsOAADA+I70yu6BJDe3ia8k+YskJ/X6KVPtTk7yyBL1HyTZUlXHzatnep2+/KX567dTJ0laa9e01mZaazNbt249wkMCAABgFEcadm/J5Lu2qarXJjkhk+B6W5Lz+0jKpyXZkeQrSb6aZEcfefmETAaxuq211pLckeQdfbt7ktza52/rz9OXf6m3BwAAgCUtextzVf1Rkr+X5KSqOpDksiTXJbmu/3dEP0mypwfR+6rqpiTfSvJMkve21p7t23lfkr1JNiW5rrV2X3+JDyW5sap+N8k9Sa7t9WuTfLqq5jK5onv+UTheAAAAjgE12sXSmZmZNjs7u9a7AQAAwPOgqu5urc0s1+5Ib2MGAACAdUvYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGs2zYrarrquqxqvrmAst+s6paVZ3Un1dVXVVVc1X1jao6a6rtnqp6sE97pupvrKr9fZ2rqqp6/WVVta+331dVJx6dQwYAAGB0K7my+6kku+cXq+qUJL+S5HtT5bcl2dGni5Jc3du+LMllSd6U5Owkl02F16t728PrHX6tS5Lc3lrbkeT2/hwAAACWtWzYba39aZKDCyy6Msm/TNKmauclwRLNAAAJ5ElEQVQmuaFN3JlkS1W9KsmuJPtaawdba08k2Zdkd1/2ktbal1trLckNSc6b2tb1ff76qToAAAAs6Yi+s1tVv5rk4dba1+ct2pbkoannB3ptqfqBBepJ8srW2qNJ0h9fcST7CgAAwLHnuNWuUFX/Q5KPJDlnocUL1NoR1Fe7Txdlcit0Tj311NWu/oK55Z6Hc8XeB/LIk4fy6i2bc/Gu03PemduWX3Ed2kjHspH2FRbj5xgAYHWO5Mruzyc5LcnXq+rPk5yc5GtV9TczuTJ7ylTbk5M8skz95AXqSfL9fptz+uNji+1Qa+2a1tpMa21m69atR3BIz79b7nk4l968Pw8/eSgtycNPHsqlN+/PLfc8vNa7tmob6Vg20r7CYvwcAwCs3qrDbmttf2vtFa217a217ZkE1rNaa/8lyW1JLuijMu9M8lS/BXlvknOq6sQ+MNU5Sfb2ZU9X1c4+CvMFSW7tL3VbksOjNu+Zqm9IV+x9IId++uxzaod++myu2PvAGu3RkdtIx7KR9hUW4+cYAGD1VvJfD/1Rki8nOb2qDlTVhUs0/0KS7ySZS/Jvk/zzJGmtHUzysSRf7dNHey1J3pPk/+nrfDvJF3v98iS/UlUPZjLq8+WrO7T15ZEnD62qvp5tpGPZSPsKi/FzDACwest+Z7e19s5llm+fmm9J3rtIu+uSXLdAfTbJ31qg/l+TvHW5/dsoXr1lcx5e4B+mr96yeQ325mezkY5lI+0rLMbPMQDA6h3RaMys3sW7Ts/m4zc9p7b5+E25eNfpa7RHR24jHctG2ldYjJ9jAIDVW/VozByZw6OmjjCa6kY6lo20r7AYP8cAAKtXkzuPxzEzM9NmZ2fXejcAAAB4HlTV3a21meXauY0ZAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4Qi7AAAADEfYBQAAYDjCLgAAAMMRdgEAABiOsAsAAMBwhF0AAACGI+wCAAAwHGEXAACA4VRrba334aiqqseT/Oe13o916qQkP1jrnWBF9NXGoa82Fv21ceirjUNfbSz6a+PQV4v7n1prW5drNFzYZXFVNdtam1nr/WB5+mrj0Fcbi/7aOPTVxqGvNhb9tXHoq5+d25gBAAAYjrALAADAcITdY8s1a70DrJi+2jj01caivzYOfbVx6KuNRX9tHPrqZ+Q7uwAAAAzHlV0AAACGI+xuAFV1XVU9VlXfnKpdUVV/VlXfqKp/V1Vben17VR2qqnv79IdT67yxqvZX1VxVXVVV1esvq6p9VfVgfzyx16u3m+uvc9YLfewbzSJ99dtV9fBUn7x9atml/f19oKp2TdV399pcVV0yVT+tqu7qffXZqjqh11/Un8/15dtfmCPeuBbpq89O9dOfV9W9ve68WmNVdUpV3VFV91fVfVX1/l5f9ftcVXt6+weras9UfVV9ycKW6CufW+vMEn3lc2sdWqK/fHatM1X1c1X1lar6eu+r3+n1VZ8PR+ucO2a11kzrfEryvyU5K8k3p2rnJDmuz388ycf7/PbpdvO285Uk/0uSSvLFJG/r9d9Lckmfv2RqW2/v7SrJziR3rfV7sd6nRfrqt5P85gJtz0jy9SQvSnJakm8n2dSnbyd5TZITepsz+jo3JTm/z/9hkvf0+X+e5A/7/PlJPrvW78V6nxbqq3nL/3WS3+rzzqu1769XJTmrz/+PSf5TP4dW9T4neVmS7/THE/v8iUfSl6ZV95XPrXU2LdFXvx2fW+tuWqy/5rXx2bUOpv4+vbjPH5/krv6+rep8OJrn3LE6ubK7AbTW/jTJwXm1/9Bae6Y/vTPJyUtto6peleQlrbUvt8lP/w1JzuuLz01yfZ+/fl79hjZxZ5ItfTssYqG+WsK5SW5srf24tfbdJHNJzu7TXGvtO621nyS5Mcm5/a+ub0nyub7+/L463IefS/LWw3+lZWFL9VV/7/7PJH+01DacVy+c1tqjrbWv9fmnk9yfZFtW/z7vSrKvtXawtfZEkn1Jdh9hX7KAxfrK59b6s8R5tRifW2touf7y2bV+9Pfpv/Wnx/epZfXnw9E8545Jwu4Y/kkmf2077LSquqeq/qSqfrnXtiU5MNXmQP7qF+QrW2uPJpNfpEleMbXOQ4usw+q8r9/2c1391S2Qi72/i9VfnuTJqX8sTvfHX67Tlz/V23NkfjnJ91trD07VnFfrRL+968xM/lK+2vd5qfpq+5JlzOuraT631pkF+srn1jq2yLnls2sdqapN/ZbyxzL5w+q3s/rz4Wiec8ckYXeDq6qPJHkmyWd66dEkp7bWzkzyfyX5f6vqJZncTjHfckNxH8k6/HVXJ/n5JG/IpH/+da8v9v6utr7Utjgy78xz/zLuvFonqurFST6f5AOttR8u1XSB2pGeXxyBxfrK59b6s0Bf+dxax5b4Peizax1prT3bWntDJnexnJ3kdQs1649H69zST/MIuxtYTQZW+YdJ/nG/DSX9Nof/2ufvzuSvSK/N5C8707eMnZzkkT7//cO3ovTHx3r9QJJTFlmHFWqtfb//wvuLJP82k194yeLv72L1H2Ry29Bx8+rP2VZf/tKs/HZqpvT379eSfPZwzXm1PlTV8Zn8A+8zrbWbe3m17/NS9dX2JYtYpK98bq1DC/WVz631a4lzy2fXOtVaezLJf8zkO7urPR+O5jl3TBJ2N6iq2p3kQ0l+tbX2o6n61qra1Odfk2RHku/0W1Gerqqd/X7+C5Lc2le7LcnhEUn3zKtfUBM7kzx1+NYWVm7ed1r+UZLDo//eluT8mozAd1omffWVJF9NsqOPpndCJgMV3Nb/YXhHknf09ef31eE+fEeSLx3+hySr9g+S/Flr7S9v8XJerb3+/l6b5P7W2iemFq32fd6b5JyqOrHfmnlOkr1H2JcsYLG+8rm1/izRVz631qElfg8mPrvWlf7eHx5xfnMm/XN/Vn8+HM1z7tjU1sEoWaalp0xuSXk0yU8z+UvOhZl8Qf2hJPf26fAIbv9HkvsyGZXta0n+96ntzGTygfXtJP93kur1lye5PcmD/fFlvV5JPtnb708ys9bvxXqfFumrT/f37xuZ/NJ61VT7j/T394H0kRB7/e2ZjLL47SQfmaq/JpNfcnNJ/jjJi3r95/rzub78NWv9Xqz3aaG+6vVPJfln89o6r9a+v/7XTG7F+sbU7723H8n7nMn3Ref69O4j7UvTqvvK59Y6m5boK59b63BarL/6sk/FZ9e6mZL8YpJ7el99M381Qvaqz4ejdc4dq9PhH2wAAAAYhtuYAQAAGI6wCwAAwHCEXQAAAIYj7AIAADAcYRcAAIDhCLsAAAAMR9gFAABgOMIuAAAAw/n/AcjHRvIKVEm1AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 1152x648 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(Y, predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You always fit the next learner on **y - y_predicted** \n",
"You can start with any weak learner F(x) - for example the mean \n",
"Then, incrementally, every **step** helps you get in the **right direction**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Does it remind you of Gradient Descent?\n",
"<img src=\"./media/matterhorn.jpg\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"F(x) + h1(x) + ... + hn(X)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Minimize a loss\n",
"\n",
"\\begin{align}\n",
"\\text{MSE}: L(y, \\text{y_pred}) = \\frac{1}{N} * \\sum{(\\text{y} - \\text{y_pred})^2}\n",
"\\end{align}\n",
"\n",
"Gradient shows us 2 things:\n",
"- the direction in which the function increases (we want to go down, so we invert the sign)\n",
"- the steepness of the slope\n",
"\n",
"<img src=\"./media/grad2d.gif\">\n",
"\n",
"\\begin{align}\n",
"\\text{Gradient descent}: x_{i} = x_{i-1} - \\eta * \\nabla f(x_{i - 1})\n",
"\\end{align}\n",
"\n",
"\\begin{align}\n",
"\\text{Gradient of L}: \\nabla L(y, \\text{y_pred}) = -2(y - \\text{y_pred}) \\\\\n",
"\\end{align}\n",
"\n",
"\\begin{align}\n",
"\\text{Gradient boosting}: F_i(x) = F_{i-1}(x) + \\eta * \\nabla_m(x)\n",
"\\end{align}\n",
"\n",
"\\begin{align}\n",
"\\text{Model}: F(x) + \\eta * h_1(x_1) + \\eta * h_2(x_2) * ... * \\eta * h_n(x_n)\n",
"\\end{align}\n",
"\n",
"<img src=\"./media/golf-MSE.png\">\n",
"\n",
"For catching up on math: https://explained.ai/matrix-calculus/index.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# XGBoost - Extreme?\n",
"https://arxiv.org/pdf/1603.02754.pdf\n",
"- regularization\n",
"- scalability\n",
"- 10 times faster\n",
"- lots of optimizations, caching\n",
"- runs on GPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Real life comparison"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"+-------------+------------------+--------------------+\n",
"| 4,865,545 | Predicted YES | Predicted NO |\n",
"| xgb:w-c | | |\n",
"+=============+==================+====================+\n",
"| Actual YES | 83.43% (948,394) | 16.57% (188,356) |\n",
"| (23.36%) | | (fn) |\n",
"+-------------+------------------+--------------------+\n",
"| Actual NO | 0.68% (25,413) | 99.32% (3,702,623) |\n",
"| (76.62%) | (fp) | |\n",
"+-------------+------------------+--------------------+\n",
"\n",
"+-------------+------------------+--------------------+\n",
"| 4,864,131 | Predicted YES | Predicted NO |\n",
"| rf:w-c | | |\n",
"+=============+==================+====================+\n",
"| Actual YES | 86.48% (979,967) | 13.52% (153,217) |\n",
"| (23.30%) | | (fn) |\n",
"+-------------+------------------+--------------------+\n",
"| Actual NO | 0.75% (27,893) | 99.25% (3,697,473) |\n",
"| (76.59%) | (fp) | |\n",
"+-------------+------------------+--------------------+\n",
"\n",
"+-------------+------------------+--------------------+\n",
"| 4,862,386 | Predicted YES | Predicted NO |\n",
"| ultra:w-c | | |\n",
"+=============+==================+====================+\n",
"| Actual YES | 78.33% (889,761) | 21.67% (246,099) |\n",
"| (23.36%) | | (fn) |\n",
"+-------------+------------------+--------------------+\n",
"| Actual NO | 1.01% (37,488) | 98.99% (3,688,280) |\n",
"| (76.62%) | (fp) | |\n",
"+-------------+------------------+--------------------+\n",
"```"
]
}
],
"metadata": {
"celltoolbar": "Raw Cell Format",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment