multi_task_learning.ipynb

multi_task_learning.ipynb

{
  "cells": [
    {
      "metadata": {
        "toc": true
      },
      "cell_type": "markdown",
      "source": "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n<div class=\"toc\" style=\"margin-top: 1em;\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span><ul class=\"toc-item\"><li><span><a href=\"#What-you-will-learn\" data-toc-modified-id=\"What-you-will-learn-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>What you will learn</a></span></li><li><span><a href=\"#Prerequisites\" data-toc-modified-id=\"Prerequisites-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Prerequisites</a></span></li><li><span><a href=\"#The-use-case\" data-toc-modified-id=\"The-use-case-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>The use case</a></span></li></ul></li><li><span><a href=\"#Imports---the-usual-stuff\" data-toc-modified-id=\"Imports---the-usual-stuff-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Imports - the usual stuff</a></span></li><li><span><a href=\"#Generate-Data\" data-toc-modified-id=\"Generate-Data-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Generate Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Generate-training-and-testing-data\" data-toc-modified-id=\"Generate-training-and-testing-data-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Generate training and testing data</a></span></li></ul></li><li><span><a href=\"#Helper-Functions\" data-toc-modified-id=\"Helper-Functions-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Helper Functions</a></span></li><li><span><a href=\"#Preparing-the-data\" data-toc-modified-id=\"Preparing-the-data-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Preparing the data</a></span></li><li><span><a href=\"#Predict-y-from-X\" data-toc-modified-id=\"Predict-y-from-X-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Predict y from X</a></span><ul class=\"toc-item\"><li><span><a href=\"#A-model-for-one-y-value-at-a-time\" data-toc-modified-id=\"A-model-for-one-y-value-at-a-time-6.1\"><span class=\"toc-item-num\">6.1&nbsp;&nbsp;</span>A model for one <em>y</em> value at a time</a></span></li></ul></li><li><span><a href=\"#Multi-Task-Learning-(without-missing-values)\" data-toc-modified-id=\"Multi-Task-Learning-(without-missing-values)-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Multi-Task Learning (without missing values)</a></span><ul class=\"toc-item\"><li><span><a href=\"#Lessons-learned:-MTL-without-missing-values\" data-toc-modified-id=\"Lessons-learned:-MTL-without-missing-values-7.1\"><span class=\"toc-item-num\">7.1&nbsp;&nbsp;</span>Lessons learned: MTL without missing values</a></span></li><li><span><a href=\"#Warning:-scale-the-targets!\" data-toc-modified-id=\"Warning:-scale-the-targets!-7.2\"><span class=\"toc-item-num\">7.2&nbsp;&nbsp;</span>Warning: scale the targets!</a></span></li></ul></li><li><span><a href=\"#Multi-Task-Learning-with-missing-labels\" data-toc-modified-id=\"Multi-Task-Learning-with-missing-labels-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Multi-Task Learning with missing labels</a></span><ul class=\"toc-item\"><li><span><a href=\"#Generate-data\" data-toc-modified-id=\"Generate-data-8.1\"><span class=\"toc-item-num\">8.1&nbsp;&nbsp;</span>Generate data</a></span></li><li><span><a href=\"#Train-model-with-single-output,-filter-missings\" data-toc-modified-id=\"Train-model-with-single-output,-filter-missings-8.2\"><span class=\"toc-item-num\">8.2&nbsp;&nbsp;</span>Train model with single output, filter missings</a></span></li><li><span><a href=\"#MTL-model-that-deals-with-missings-values\" data-toc-modified-id=\"MTL-model-that-deals-with-missings-values-8.3\"><span class=\"toc-item-num\">8.3&nbsp;&nbsp;</span>MTL model that deals with missings values</a></span></li></ul></li><li><span><a href=\"#MTL-vs.-Transfer-Learning\" data-toc-modified-id=\"MTL-vs.-Transfer-Learning-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>MTL vs. Transfer Learning</a></span></li><li><span><a href=\"#Summary-and-assumptions\" data-toc-modified-id=\"Summary-and-assumptions-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>Summary and assumptions</a></span></li><li><span><a href=\"#Further-Reading\" data-toc-modified-id=\"Further-Reading-11\"><span class=\"toc-item-num\">11&nbsp;&nbsp;</span>Further Reading</a></span></li></ul></div>"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Introduction\n\nFor a project I wanted to use Multi-Task Learning with Keras, and I was surprised that I could not find any tutorials online that cover the full implementation beginning to end.\n\nThere are a lot of examples that explain the idea of MTL, or show an implementation. I also saw a lot of questions about how to deal with missing values in the target variables, but I found nothing that plugs it all together. So I decided to publish my own tutorial.\n\n## What you will learn\n- understand why Multi-Task learning can be advantageous over single-output models\n- how to build a Multi-Task Learning model in Keras\n- implement your own loss function that deals with missing values\n\n## Prerequisites\n- you should be familar with creating simple sequential models with Keras\n- you should understand the concept of neural networks\n- for a much more complete review of MTL and related methods, take a look at [An Overview of Multi-Task Learning in Deep Neural Networks](http://ruder.io/multi-task/) by [Sebastian Ruder](https://twitter.com/seb_ruder)\n\n## The use case\n![bread.png](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Korb_mit_Br%C3%B6tchen.JPG/1920px-Korb_mit_Br%C3%B6tchen.JPG)\n\nLet's assume you run a bakery, and you want to design the perfect bread. Since you cannot randomly throw ingredients together, bake the bread and hope for a best, you decide to use machine learning to guide you.\n\nYou ask your customers to fill in a survey about every bread you sell. You hope that you can then train a model using the recipes of the sold breads and the customer feedback as training data, to predict the ratings of new bread recipes.\n\nYou ask the customers to rate three categories:\n- sweetness\n- fluffiness of the inside\n- crunchiness of the crust\n\nUnfortunately, on average every customer answers only one of the questions, so you have a lot of missing labels.",
      "attachments": {}
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Imports - the usual stuff"
    },
    {
      "metadata": {
        "hide_input": false,
        "nbpresent": {
          "id": "76450513-bf90-4c35-bd0c-23c65642a57e"
        },
        "trusted": true
      },
      "cell_type": "code",
      "source": "%pylab inline\nfrom IPython.display import SVG\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom sklearn.metrics import r2_score\n\nfrom tensorflow.contrib.keras.api.keras.layers import Dense, Input\nfrom tensorflow.contrib.keras.api.keras.models import Model\nfrom tensorflow.contrib.keras.api.keras.optimizers import Adam\nfrom tensorflow.contrib.keras.api.keras.callbacks import EarlyStopping\nfrom tensorflow.contrib.keras.python.keras.utils.vis_utils import model_to_dot\nfrom tensorflow.contrib.keras.api.keras import backend as K\nimport tensorflow as tf",
      "execution_count": 71,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Populating the interactive namespace from numpy and matplotlib\n"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Generate Data\n\nThis function creates a dataset with 6 input variables and 3 targets. The targets are our crunchiness, fluffiness and sweetness indicators. The inputs specify how much of each ingredient you put into the dough.\n\nThe targets are linear combinations of ratios of the input variables. In our bakery usecae, 'sweetness' could e.g., be highly influenced by the ratio of sugar and salt. We add some noise to make the data a little more realistic.\n\nFor out neural net this means, that all targets depend on the same latent features (the ratios), and just differ in how those are combined."
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "def generate_data(n):\n    X = rndst.uniform(50, 100, [n, 10]) \n    noisy_X = X * rndst.normal(1, .02, [n, 10])\n\n    lf = noisy_X[:,:5] / noisy_X[:,-5:] * rndst.normal(1, .05, [n,5])  # latent features\n\n    y1 = 2*lf[:,0] + 3*lf[:,1] + lf[:,2]\n    y2 = 5*lf[:,0] + 7*lf[:,1] + lf[:,2]\n    y3 = 3*lf[:,0] + 4*lf[:,1] + 5*lf[:,2]\n        \n    y = np.concatenate([y1.reshape(-1,1), y2.reshape(-1,1), y3.reshape(-1,1)], axis=1)\n\n    return X, y",
      "execution_count": 72,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Generate training and testing data\n\nTo start easy, we generate data without missing values. We'll add some missings further down the road."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "X, y = generate_data(500)\nX_test, y_test = generate_data(500)\nX.shape, y.shape",
      "execution_count": 73,
      "outputs": [
        {
          "data": {
            "text/plain": "((500, 10), (500, 3))"
          },
          "execution_count": 73,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Helper Functions"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "To help us output the learning behavior of our models, let's define a small helper function to plot the learning curve:"
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "def plot_lc(model):\n    \"\"\" Plots the learning curve of a keras model \"\"\"\n    plt.plot(np.log(model.history.history['loss']), label='loss')\n    plt.plot(np.log(model.history.history['val_loss']), label='val_loss')\n    plt.legend()\n    plt.xlabel('epoch')\n    plt.ylabel('log loss')\n    plt.title('Learning curve')",
      "execution_count": 74,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Preparing the data\nAs a best practice with neural networks, we normalize our input data:"
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "from sklearn.preprocessing import StandardScaler\nss = StandardScaler()\nX_std = ss.fit_transform(X)\nX_test_std = ss.transform(X_test)",
      "execution_count": 75,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Predict y from X"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let's also create a function to create a Keras model. For now, we create a normal model with a single output. We use 2 dense hidden layers, the adam optimizer with default settings and the MSE loss function."
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "def build_model1():\n    inputs = Input(shape=(10,))\n    x = inputs\n    x = Dense(20, activation='relu')(x)\n    x = Dense(20, activation='relu')(x)\n    outputs = Dense(1, activation='linear')(x)\n    model = Model(inputs=inputs, outputs=outputs)\n    model.compile(optimizer='adam', loss='mean_squared_error')\n    return model",
      "execution_count": 76,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## A model for one *y* value at a time"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "To warm up, we can build and fit a model that predicts only the first column of our y matrix. We use up to 1000 epochs with early stopping. For the early stopping a value of 10 usually works quite well, i.e., we will stop training if the validation loss does not decrease for 10 epochs."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%time\nmodel = build_model1()\n\nmodel.fit(X_std, y[:,0], epochs=1000, validation_split=.2, verbose=0, \n    callbacks=[EarlyStopping(patience=10)])\nplot_lc(model)\n\npreds = model.predict(X_test_std)\nprint('r2 score:', r2_score((y_test[:,0]), preds))",
      "execution_count": 77,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "r2 score: 0.863290257227\nCPU times: user 9.12 s, sys: 1.06 s, total: 10.2 s\nWall time: 5.27 s\n"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd0XNW59/HvM9KoWb13ucmWe5MLxQZDKKGEUE1waAFM\nCIFwQ3hDQnJDcmHlJrkJt6QRWiCUmBoINRRjYzC2ZWHj3mSrWVbvXTP7/eOMQRhrJMuSzozm+ayl\nhTRzZs4zJ7F+2nufvbcYY1BKKaX64rC7AKWUUr5Ng0IppZRXGhRKKaW80qBQSinllQaFUkoprzQo\nlFJKeaVBoZQXIvKGiFxrdx1K2Ul0HoXyRSJyELjRGPOO3bUoFei0RaEClogE213DiRoNn0H5Pg0K\n5XdE5AIR2SwiDSLykYjM7PXc3SKyX0SaRWSHiFzc67nrRORDEXlARGqBez2PrRWR/xKRehE5ICJf\n7fWa90Xkxl6v93bsOBFZ4zn3OyLyBxF50svnuMjzOZo8NZ/refygiHyl13H3HnkfERkrIkZEbhCR\nEuA9T/fYd4967y0iconn+zwReVtE6kRkt4hcMfirrwKRBoXyKyIyB3gUuBlIAB4EXhGRUM8h+4HF\nQAzwc+BJEUnr9RYLgSIgBbi/12O7gUTg18AjIiJ9lODt2KeBDZ667gWu9vI5FgBPAHcBscAS4GB/\nn7+X04ApwDnAM8A3er33VCAHeE1ExgBve2pLBq4E/ug5RqkB0aBQ/mYF8KAxZr0xxmWMeRzoBBYB\nGGOeM8YcMsa4jTErgb3Agl6vP2SM+T9jTI8xpt3zWLEx5iFjjAt4HEjDCpJjOeaxIpINzAf+3RjT\nZYxZC7zi5XPcADxqjHnbU2u5MWbXcVyHe40xrZ7P8BIwW0RyPM8tB140xnQCFwAHjTGPeT7zJ8AL\nwOXHcS4V4DQolL/JAe70dDs1iEgDkAWkA4jINb26pRqA6Vh//R9Reoz3PHzkG2NMm+fbyD7O39ex\n6UBdr8f6OtcRWVitn8H67L2NMc3Aa1itBbBaF095vs8BFh51vZYDqSdwbhVgdCBM+ZtS4H5jzP1H\nP+H5i/oh4ExgnTHGJSKbgd7dSMN1m18FEC8iEb3CIsvL8aXAhD6eawUiev18rF/qR3+OZ4Cficga\nIAxY1es8q40xZ3krXilvtEWhfJlTRMJ6fQVjBcG3RWShWMaIyPkiEgWMwfoFWg0gItdjtSiGnTGm\nGCjAGiAPEZGTgAu9vOQR4HoROVNEHCKSISJ5nuc2A1eKiFNE8oHLBlDC61ith18AK40xbs/jrwKT\nRORqz/s5RWS+iEwZzOdUgUmDQvmy14H2Xl/3GmMKgJuA3wP1wD7gOgBjzA7gt8A6oBKYAXw4gvUu\nB04CaoH7gJVY4ydfYozZAFwPPAA0AquxftED/BSrtVGPNSD/dH8n9oxHvAh8pffxnm6ps7G6pQ5h\ndZ39Cgg9xtsodUw64U6pYSIiK4Fdxpif2V2LUidCWxRKDRFPl84ET1fSucBFwD/srkupE6WD2UoN\nnVSs7p8EoAy4xXM7qlJ+TbuelFJKeaVdT0oppbzyq66nxMREM3bsWLvLUEopv7Jp06YaY0zSYF/v\nV0ExduxYCgoK7C5DKaX8iogUn8jrtetJKaWUVxoUSimlvNKgUEop5ZVfjVEopQJTd3c3ZWVldHR0\n2F2KTwsLCyMzMxOn0zmk76tBoZTyeWVlZURFRTF27Fj63lMqsBljqK2tpaysjHHjxg3pe2vXk1LK\n53V0dJCQkKAh4YWIkJCQMCytLg0KpZRf0JDo33Bdo4AIilW7qvjj+/vsLkMppfxSQATFR/tr+O93\n9tLV4+7/YKWUOkpkZF874waGgAiKWVmxdPW42X242e5SlFLK7wRGUGTGArC5rMHmSpRS/swYw113\n3cX06dOZMWMGK1euBKCiooIlS5Ywe/Zspk+fzgcffIDL5eK666777NgHHnjA5uoHLyBuj80seZlf\nhb/MhpIfcfWinP5foJTyWT//53Z2HGoa0vecmh7Nzy6c1u9xL774Ips3b2bLli3U1NQwf/58lixZ\nwtNPP80555zDPffcg8vloq2tjc2bN1NeXs62bdsAaGjw3z9UA6JFITV7uNT8ix2lVXaXopTyY2vX\nruUb3/gGQUFBpKSkcNppp7Fx40bmz5/PY489xr333svWrVuJiopi/PjxFBUVcdttt/Hmm28SHR1t\nd/mDFhAtCtJnE4yLkNqdNHWcTnTY0M5aVEqNnIH85T/SlixZwpo1a3jttde47rrr+P73v88111zD\nli1beOutt/jzn//Ms88+y6OPPmp3qYMSEC0K0mYDMEOK2FbWaHMxSil/tXjxYlauXInL5aK6upo1\na9awYMECiouLSUlJ4aabbuLGG2+ksLCQmpoa3G43l156Kffddx+FhYV2lz9ogdGiiM3GHR7P9J4D\nfFLawMkTE+2uSCnlhy6++GLWrVvHrFmzEBF+/etfk5qayuOPP85vfvMbnE4nkZGRPPHEE5SXl3P9\n9dfjdlu35f/yl7+0ufrB86s9s/Pz882gNy7628XsO3CAX499mL9ckz+0hSmlhtXOnTuZMmWK3WX4\nhWNdKxHZZIwZ9C++wOh6AkidwVh3KbsP1dldiVJK+ZXACYqU6QTTQ0jjAVo6e+yuRiml/EbgBEXy\nVACmSInO0FZKqeMQOEGROAnjCCbPUcKuw0M7WUcppUazwAmK4BBInMTUoDJtUSil1HGwLShEJExE\nNojIFhHZLiI/H/ZzJk1mclAF+6pahvtUSik1atjZougEzjDGzAJmA+eKyKJhPWNCLinuSsqq/XfN\nFaWUGmm2BYWxHPnT3un5Gt5JHQkTceDG2VxCe5drWE+llApc3vavOHjwINOnTx/Bak6crWMUIhIk\nIpuBKuBtY8z6YT1hwkQAxksFB2tbh/VUSik1Wti6hIcxxgXMFpFY4CURmW6M2db7GBFZAawAyM7O\nPrETJkwAYJxUcLCmlSlp/ruao1IB64274fDWoX3P1Bnw1f/s8+m7776brKwsbr31VgDuvfdegoOD\nWbVqFfX19XR3d3Pfffdx0UUXHddpOzo6uOWWWygoKCA4OJjf/e53LF26lO3bt3P99dfT1dWF2+3m\nhRdeID09nSuuuIKysjJcLhc//elPWbZs2Ql97IHyibWejDENIrIKOBfYdtRzfwH+AtYSHid0ovBY\nTEQS45sqKKrRFoVSamCWLVvGHXfc8VlQPPvss7z11lvcfvvtREdHU1NTw6JFi/ja176GiAz4ff/w\nhz8gImzdupVdu3Zx9tlns2fPHv785z/zve99j+XLl9PV1YXL5eL1118nPT2d1157DYDGxpFb4NS2\noBCRJKDbExLhwFnAr4b9vPHjmNBeS6F2PSnln7z85T9c5syZQ1VVFYcOHaK6upq4uDhSU1P5t3/7\nN9asWYPD4aC8vJzKykpSU1MH/L5r167ltttuAyAvL4+cnBz27NnDSSedxP33309ZWRmXXHIJubm5\nzJgxgzvvvJMf/vCHXHDBBSxevHi4Pu6X2DlGkQasEpFPgY1YYxSvDvtZY7PJkmrKG9qH/VRKqdHj\n8ssv5/nnn2flypUsW7aMp556iurqajZt2sTmzZtJSUmho6NjSM511VVX8corrxAeHs55553He++9\nx6RJkygsLGTGjBn85Cc/4Re/+MWQnGsgbGtRGGM+BeaM+Iljs0lyV3OoTudSKKUGbtmyZdx0003U\n1NSwevVqnn32WZKTk3E6naxatYri4uLjfs/Fixfz1FNPccYZZ7Bnzx5KSkqYPHkyRUVFjB8/nttv\nv52SkhI+/fRT8vLyiI+P55vf/CaxsbE8/PDDw/Apj80nxihGVGw2QbjoaazA5TYEOQben6iUClzT\npk2jubmZjIwM0tLSWL58ORdeeCEzZswgPz+fvLy8437P73znO9xyyy3MmDGD4OBg/vrXvxIaGsqz\nzz7L3/72N5xOJ6mpqfz4xz9m48aN3HXXXTgcDpxOJ3/605+G4VMeW+DsR3HEvnfhyUu4vPPf+Z8f\nfof02PChKU4pNWx0P4qB0/0ohkJsDgCZUk1ZvY5TKKVUfwKv6ykmEzgSFG0sGBdvc0FKqdFo69at\nXH311V94LDQ0lPXrh3de8XAIvKBwhmHGJJPeWKstCqX8iDHmuOYo2G3GjBls3rx5RM85XEMJgdf1\nBEhUKpnBjRxuGppb2ZRSwyssLIza2tph+0U4GhhjqK2tJSwsbMjfO/BaFABRqaRV76NKg0Ipv5CZ\nmUlZWRnV1dV2l+LTwsLCyMzMHPL3DdigSKRAWxRK+Qmn08m4cePsLiNgBWTXE5GpRLvqqWlss7sS\npZTyeYEZFFGpODCY1mq6XW67q1FKKZ8WsEEBkEw9NS2dNhejlFK+LbCDQuo53KjjFEop5U1gBkWk\nFRQp0kClDmgrpZRXARoUyRiEZKmnskm7npRSypvADIogJ4xJJNXRoLfIKqVUPwIzKDgyO7tJu56U\nUqofARsURKaS5tAxCqWU6k/gBkVUKglGxyiUUqo/AR0U0a56qnV2tlJKeRXQQeHATUhnLa2dPXZX\no5RSPsu2oBCRLBFZJSI7RGS7iHxvRAv4bC5FvY5TKKWUF3a2KHqAO40xU4FFwK0iMnXEzh6VBkCy\n6C2ySinljW1BYYypMMYUer5vBnYCGSNWQFQKoC0KpZTqj0+MUYjIWGAO8KXNZEVkhYgUiEjBkG5a\nEmkFRTINVOmdT0op1Sfbg0JEIoEXgDuMMU1HP2+M+YsxJt8Yk5+UlDR0Jw5yQkQi6bolqlJKeWVr\nUIiIEysknjLGvDjiBUSlkhXcqC0KpZTyws67ngR4BNhpjPmdLUVEppDiaNQxCqWU8sLOFsUpwNXA\nGSKy2fN13ohWEJVGgqnTriellPIi2K4TG2PWAmLX+QGISiG6p56a9naMMViNHKWUUr3ZPphtq8hU\nHLiIdDXS0NZtdzVKKeWTAjsoem2JWtms3U9KKXUsGhTo3tlKKeVNYAfFkUl3opPulFKqL4EdFEda\nFOgGRkop1ZfADorgUAiPI9ups7OVUqovgR0UAJGpZDibdKc7pZTqgwZFVAop0kCV3vWklFLHpEFx\nZHa23vWklFLHpEERmUJ0Tx01LR30uNx2V6OUUj5HgyIqlSDTQ4xppqaly+5qlFLK52hQRB3ZO7uB\nQ43tNhejlFK+R4Mi2tp9NU1qKa1rs7kYpZTyPRoUcWMByJYqyuq1RaGUUkfToBiTBM4IckNqKavX\nFoVSSh1Ng0IEYnOY6KzRFoVSSh2DBgVA3FgyqdIxCqWUOgYNCoC4HJJ6DlPe0IbbbeyuRimlfIoG\nBUDcWELdbUS5mnQDI6WUOooGBUD8eADGymEdp1BKqaPYGhQi8qiIVInINjvrIGEiABMch3ScQiml\njmJ3i+KvwLk21wCxORiHk/FSoS0KpZQ6iq1BYYxZA9TZWQMAQcFIwgSmOCu1RaGUUkexu0XRLxFZ\nISIFIlJQXV09fCdKmMhEh7YolFLqaD4fFMaYvxhj8o0x+UlJScN3osRcUl0VHKprHr5zKKWUH/L5\noBgxSVMIpofw5iI6e1x2V6OUUj5Dg+KItJkA5JmDHKzRcQqllDrC7ttjnwHWAZNFpExEbrCtmIRc\n3EGhTHccYG+Vdj8ppdQRwXae3BjzDTvP/wVBwZAyjWllxXxc2WJ3NUop5TO066kXR9ospjuK2Vep\nLQqllDriuIJCROJEZOZwFWO7tJlE0Upz5T67K1FKKZ/Rb1CIyPsiEi0i8UAh8JCI/G74S7NB6iwA\noup30u1y21yMUkr5hoG0KGKMMU3AJcATxpiFwFeGtyybpEzFLUFM5gDFta12V6OUUj5hIEERLCJp\nwBXAq8Ncj72c4XTFTmCaFLNXB7SVUgoYWFD8AngL2GeM2Sgi44G9w1uWfYIzZjPDcYC9OqCtlFLA\nAILCGPOcMWamMeY7np+LjDGXDn9p9gjOWUiyNFBXPmqzUCmljstABrN/7RnMdorIuyJSLSLfHIni\nbJF9EgDhhzfaXIhSSvmGgXQ9ne0ZzL4AOAhMBO4azqJslTSFjqBIslq20NbVY3c1SilluwENZnv+\nez7wnDGmcRjrsZ/DQUvyPBbKTraVN9ldjVJK2W4gQfGqiOwC5gHvikgS0DG8ZdkrLO8sJjgqKNq7\nw+5SlFLKdgMZzL4bOBnIN8Z0A63ARcNdmJ0ip55jfbP/XXsLUUopH9DvooAi4gS+CSwREYDVwJ+H\nuS57JeZSG5xCRtUajDF4PrdSSgWkgXQ9/Qmr2+mPnq+5nsdGLxGqMs9hofsTSsrL7a5GKaVsNZCg\nmG+MudYY857n63pg/nAXZrfIBcsJEReV6/5udylKKWWrgQSFS0QmHPnBMzN71O8Vmpm3gL2STcbe\np8AYu8tRSinbDCQo7gJWeVaRXQ28B9w5vGXZTxwONqVfTUZXEd273rC7HKWUss1A7np6F8gFbgdu\nAyYbY1YNd2G+IPGkqyh2J9P9xk+hp8vucpRSyhZ9BoWIXHLkC2uy3UTP1/mex0a9Uyalcb+5noim\nffD+L+0uRymlbOHt9tgLvTxngBdP9OQici7wP0AQ8LAx5j9P9D2HUnhIEJEzzuPF7Ru4ZO3vIGsB\nTP6q3WUppdSI6jMoPHc3DRsRCQL+AJwFlAEbReQVY4xPTYdevjCbqwqv5bTkwyS8eDPc+DYkTba7\nLKWUGjHHtWf2EFuAtcdFkTGmC/g7Pjjje252HHlZydzceQfGGQaPXwjlhXaXpZRSI8bOoMgASnv9\nXOZ5zKeICHd8JZeCxihemv5HCAqBR8+FwifsLk0ppUaEnUExICKyQkQKRKSgurralhpOn5TEaZOS\n+MlHPVRc+SbknASv3Ab/+A601dlSk1JKjZSBbFx0yTG+zhSR5BM8dzmQ1evnTM9jX2CM+YsxJt8Y\nk5+UlHSCpxwcEeG+r0/HGLjnrQrM8hdg8Q9gy9/hj4ugdIMtdSml1EgYSIviBuBhYLnn6yHgh8CH\nInL1CZx7I5ArIuNEJAS4EnjlBN5vWGXFR3Dn2ZN4b1cVj60rhTN/CjevBmc4/PV82PAQuN12l6mU\nUkNuoBsXTTHGXOrZK3sq1u2xC7ECY1CMMT3Ad4G3gJ3As8aY7YN9v5HwrVPGcc60FO57bQerdldB\n6gy4aRWMPRVe/wE8di5U+tRNW0opdcIGEhRZxpjKXj9XeR6rA7pP5OTGmNeNMZOMMROMMfefyHuN\nBIdDeGDZbKakRfPdpwopLKmHiHj45ovw9T9B7T54cDG8cy90tdldrlJKDYmBBMX7IvKqiFwrItdi\ndQ+9LyJjgIbhLc/3RIQE8+h180mKCuXaRzawpbQBRGD2VXDrRph5Jax9AH4/H7b/w+5ylVLqhA0k\nKG4FHgNme74eB241xrQaY5YOZ3G+KiU6jKdvWkTsGCdXP7KebeWebcTHJMDX/wDXvwHhcfDctfDK\n7dDZbG/BSil1AgayKKAB1mKtGvsusMbzWEBLjw3nmZsWERXm5KqHPmZTca/bZHNOtga6T/0+FD5u\ntS62Pq/LlSul/NJAbo+9AtgAXAZcAawXkcuGuzB/kBkXwcqbF5EQGcryh9dbA9xHOILgKz+DG96B\nyGR44QZrVnf1HvsKVkqpQRhI19M9fL7L3TVYS2/8dHjL8h+ZcRE8e/NJjE+M5MbHC3h6fckXD8ia\nb90Zdf7v4PBWa7B77QPQ3WFPwUopdZwGEhQOY0yvP5WpHeDrAkZSVCjPfvskFucm8uOXtvKfb+zC\n7e7VzeQIgvk3wHc3woQzrbuifp8P21/S7iillM8byC/8N0XkLRG5TkSuA14DXh/esvxPZGgwD1+T\nz/KF2fx59X6u/+tGals6jzooGb7xNFzzCoTHwnPXwdNXQO1+W2pWSqmBkIGMS4vIpcApnh8/MMa8\nNKxV9SE/P98UFBTYceoBM8bw5PoS/uPVHcRHhPC/35jDgnHxXz7Q1QMb/gKr7oeeTlh4Myy5ywoQ\npZQaQiKyyRiTP+jX+9MNTP4QFEdsK2/ku08XUlrfzvfPmsQtp03A4ZAvH9hcCavug8K/WbfUnnEP\n5N9gzc1QSqkhMGxBISLNWEt1fOkprLtmowd70sHyp6AAaO7o5scvbeOfWw6xODeRB5bNJjEy9NgH\nH94Kb/4IDn4AYxfDybfBxK9Y4xtKKXUCtEXh44wxPLOhlHv/uZ3YcCf/feVsTp6Q2NfBUPAIrP4N\ntByG2Bw44ycw84qRLVopNaqcaFDo3UvDTES4amE2//jOKUSGBnPVQ+v56T+20dLZc6yDYf6N8G/b\n4LLHrHWkXrwJnrteB7yVUrbRFsUIauvq4Tdv7eavHx0kPSacX106k1Nz+2hdgDXg/cFvrS9Xp9UV\nteBmT5eUZrxSamC068kPFRys4/89/ylFNa1cOT+LH58/hegwZ98vaD4Mm/4KBY9CSyUkToaTvwsz\nl0FwH2MeSinloUHhpzq6XTzwzh4eWlNEclQYv7xkBkvz+tk0sKcLdvwDPvpfa/A7NBomLIXFd0La\nrJEpXCnldzQo/Nzm0gbuem4Le6taOHtqCvecP4WchDHeX2QMFL1vhca2l6CzESZ9FWYts7qlQqNG\npHallH/QoBgFOntcPLL2AL9/bx89bsN3l07k5tPGExo8gFtjOxph/YOw7g/Q0QAxWXD63TDxLIhK\nGf7ilVI+T4NiFKls6uA/Xt3Bq59WkBUfzh1nTuKSuRnIQCbf9XRCyTp488dQ5dlRduJZsGAFjD8d\ngkOGs3SllA/ToBiFVu+p5rf/2s2nZY2cND6B//j6dCYmRw7sxcZA+SbY+y8oeAxaqyAkCiadDfnf\ngpxTdNa3UgFGg2KUcrsNz2ws4T/f2EV7l4vrTh7LbWfmEhPu5e6oo/V0wr53Ye9b1rasHQ0QnQm5\nZ8GsKyFzvs78VioA+GVQiMjlwL3AFGCBMWZAv/0DKSiOqGnp5Lf/2s3fN5YSHxHCD86ZzBX5WQQd\na90ob7raYPuLsOct2Ps29LRDcDikTLVaGjOvhKDg4fkQSilb+WtQTAHcwIPADzQo+retvJFf/HMH\nGw7WMTklirvPy+P0SUkDG784Wmcz7H4TDn0CB9ZA5VaISrdusZ14pnXnVNxY7aJSapTwy6D47OQi\n76NBMWDGGN7YdphfvbmL4to2TpmYwJ1nT2ZOVuzgAsN6U9j9Bmx5xpqbUX/AejxhIky7BKZ9HVKm\nDd2HUEqNuFEfFCKyAlgBkJ2dPa+4uHiEqvNdXT1unlpfzP++u5f6tm5mZcbwkwumMn/sMfa9OF41\ne605Glufg7KNVpDEZkFUmjWukXeBtQGTUspv+GxQiMg7QOoxnrrHGPOy55j30RbFoDV3dPPy5kP8\n/r19HG7q4Iy8ZFYsGc/CcfGDb2H01lZnLRtSswcqd1hdVABx4yBjLmTMgzlXQ9iIrzivlDoOPhsU\nAzq5BsWQaOvq4ZEPDvDYRwepa+1iVmYMK5ZM4Nzpqcc/6N0XY6CswJqrUboeKj6FxhIQB6TPgZyT\nrdnhY0/p/72UUiNKg0J9pr3LxQuFZTz8QREHa9vIig/nxlPHc3l+JhEhw3BHU9km2POmdftt1S5r\nhduYLEieYo1rTD4fMvN1UFwpm/llUIjIxcD/AUlAA7DZGHNOf6/ToBgYl9vw9o7DPLimiE9KGoiN\ncHLNohyuOXls3zvsnajudih8whrXqNoF1bvA3Q0x2TDpHIjJgNSZ1u59OktcqRHll0ExWBoUx8cY\nQ0FxPQ+uLuKdnZWEBju4dF4mN546jvFJA5zpPVgdjbDrddj2ApR8DF3Nnz+XNgumXwap0yHnVA0O\npYaZBoUakP3VLTz8QREvFJbT7XJz1pQUViwZz7ycuKEZ+O5PZ4u1rEj1bis8avdaj0elW91TkSmw\n8NuQOHH4a1EqwGhQqONS3dzJE+sO8sS6Yhrbu0mMDOW2MyZyRX4W4SEjuJxHW501MP7JU1ZoNJRa\ns8XjxlkD4nHjICnP+orLgaDjWLpEKfUFGhRqUNq6enh1SwUvfVLOuqJaIkODuTw/kzPzUjh5QgKO\nobpbaqCaK2Hb83DwQyj5CNrrP38uMgUy8q1Z4+NO01aHUsdJg0KdEGMM6/bX8szGUt7YWkGP25AV\nH86V87O5bF4mKdFhI1+U223dQVX8ITRVWF1WvWeNJ02BGZdaM8cTJox8fUr5GQ0KNWTau1y8s7OS\np9YX83FRHUEOYenkZK5amMVpk5KHbk7GYBhjTfw7sMYzQL7OenxMMsSP//xr4pmQNhscDvtqVcrH\naFCoYXGgppVnC0p5rqCMmpZO0mPCOH9mGpfNy2Jyqg9stdpQas3hqNgMdQegrgiaK6zngsOsMY6E\nCVZoLLxZZ4+rgKZBoYZVV4+bd3dW8mxBKWv31dDtMszJjuWyeZlcMCOdmAgfGmRur4ddr0HVTqjd\nD3X7rbWrgkKseRzh8ZC1AE6+HaLT7K5WqRGjQaFGTF1rFy8WlrFyYyl7q1oICXZw1pQULpmbwZJJ\nSTiDfLC7p3wTbH8JGsuhtdoa9zBucI6B8Dhrm9jYbJi1zFpaXalRSINCjThjDNsPNfH8pjJe2XKI\nutYuEsaE8LXZ6Vw6N5Np6dEjMzdjMGr2Wps3NVdAYxnsf8/anwMD8ROsO6wSc63VcqdcaE0KVMrP\naVAoW3W73KzeXc0LhWW8u7OKLpebySlRXDI3g4tmZ5AaY8NdU8er6ZC1H0fFFuv7+oPQVms9lzbL\n2mc8bTZMOAPGJIDbZbVKdG6H8hMaFMpnNLR18eqnFbxQWMYnJQ0A5CZHcum8TC6dm0lS1DCtMzUc\n2urgo/+1uq6KPwJ3j7VSbkSiFSIhY2Dx960lSCKTtNtK+TQNCuWTiqpbeGt7Je/urKSguB6HQP7Y\neC6anc7FczKGZzXb4dLVat2au/tNaCqHMUlQtcO66wqswfLkqdZSJDmnWIsfhozRAXPlMzQolM/b\nW9nMP7cc4s3th9lT2UJMuJOvTk/lvBlpnDIx0d75GSfi4IdWcBS9D/XFUPqx1fI4IiMfwmJg5hXW\nmEfaTOilntYyAAAUuElEQVTusIJlTIJtZavAo0Gh/MaR1WyfWFfM+7uqaO7sITU6jPyxcczOiuXS\nuZnEjfHjlWTb6qwB8orN1n/3vGXdadVU/uVjp10Cp/0Qkibrfh1q2GlQKL/U0W3NAn91SwXbDjVS\nVt9OuDOIZfOzuOHUcWTFR9hd4tDo6bTutGqtsgbLg8Og+TB89H9gXBCRYN1tFZNpdVeJA7JPsnYN\n1BBRQ0SDQo0KeyqbeXB1ES9vLscASycnc/rkJE6fnERm3CgJjd4aSqwuq9L11veN5dZtuj2d0Nlo\nHROb7dns6VSr6yoiHmJzIDTKCpuUqbZ+BOU/NCjUqFLR2M5jHx7ktU8rKG9oB2BSSiSLc5OIi3Cy\nfGGOf3dP9cfVYy1HUvwh7H0bqndaP/cWHAauLlh6D+SeBZGp1u26UanaAlHHpEGhRiVjDEU1raza\nVcV7u6rYcKCOHrchJTqUJblJ3H5m7ujpnupPXRF0tVm35VZsgZrd1qq6+9/94nFjF1uLImYusMKk\nox7aG6zWyNhTrIUVjdEFEwOQBoUKCC63YWt5I//9zh42HKijvdvF9PQYFucmsjQvmTlZsQT74hIi\nw8UYaz2rw1usMGivgzX/BT0dxz5+8vlQtd2aTDjlazD+NKu7K2myNTckY67VtaVGJQ0KFXAONbTz\nXEEZa/dVU1jSgMttiAl3smRSEksnJ3HapCQSIv1oct9Q6Wq1br8t+QgcTgiPhbBY2PI0bH0eojOs\nQfKCRzy38Qpw5N+/QOZ8a8VdccDiO6G7HVoqPQPsi8AZbuOHUyfCL4NCRH4DXAh0AfuB640xDf29\nToNCHa2xvZsP99WwalcVq3ZXU9PSiQjMzIxl6eQklk5OZkZGzMjv2OfLmiqsMY3IZKjdB6011uzz\nT1dag+TGbW1L29uRgXV3jzWg3tlszUZ3dYKr2wqYhImQtRAQcHeDI/jzZU7cLitwdAzFFv4aFGcD\n7xljekTkVwDGmB/29zoNCuWN220tVrhqdxWrdlexubQBYyAxMoRzpqXytVnp5KVG+9bS6L7E7bZC\noqUSdrwMoZGQOMmaC7L+QStQjNu6Sys81lpYUYLAEWQNroMVHo3lVlAEh1sLK7ZWQ+kGiEqxWjUx\nmdbA+5F90Su3WXNN2uutLrKsBZ5AEesW4pBI6xxq0PwyKL5QgMjFwGXGmOX9HatBoY5HXWsXa/ZU\n896uKt7cfpiuHjehwQ4W5yayaHwCS/OSmZAUaXeZ/qvb0+oICrEmGJZ8DB//AdLnQlyOtS/I/lVW\nyyV7kdVqcXVZx7q6rNA5QoKsAfjuVgiJsgKip8M6JiTKCi1xWCGTvQhaqqxAi8mCBSussZfuditw\nIuKt1k1kChxcaw3mTzq7/89jDHS3WXWMsmAaDUHxT2ClMebJPp5fAawAyM7OnldcXDyS5alRoqq5\ng+3lTby/u4rVe6o5WNsGwISkMXxlagrzsuPIToggL1V3wht23e3WGEpdkbVmVvps65e6cVvzSjY/\nY3VbxWSC0zNBsafD6r6q3G7d+RWdbrVKKnd8uZvsWFJnWi2iMQnWsEx7PeScbLVedr7imcPSZQ34\nR6bA/ButMZnYHKjeDUHBMPEsKNtg7WUyYam1Ha8zwnptUp41Iz8j31qmvqvVCsOavZC90GpluTqt\n7XrB+kzd7RA/DopWQ+HjcN5/9X1DQUu1tfjkIPlsUIjIO0DqMZ66xxjzsueYe4B84BIzgEK0RaGG\nSll9G+/urOLtHZV8XFRLj9v6v9+501JZmpfExXMyCQkOoLuo/Inb9flf/K211thKTAYk5Frh0dlk\nBUrdAZh0Dmx9zlqXKzbLas0Eh1lb4+57BzoardZHVBp0tVhdX6UfW88dD3F4WkhivXdHo3UjQUeD\ntVBk2UYrOCafZ3W/FTxiHT/9Mk8dDRCVbn2u7narRRaRaI0NOYKg4DFY/px1t9og+GxQ9HtikeuA\nm4EzjTFtA3mNBoUaDq2dPeytauGNbRW8WFhOdXMn0WHBnJqbyGmTklgyKYm0GL3jZ9Tp7rBaNGmz\nvtzVVH/Q+uVfud1qLbh7YOc/YdK5ny/HkjHP6v5yu6HgUeuXuCPI2s/dGQ6HNlutpYNrrf/G5sCH\n/2O1NuYst1omGx60wmDJ/4Pdb1iD/yFjrBZHSxU0FFvBN+1iOOeXg16R2C+DQkTOBX4HnGaMqR7o\n6zQo1HAzxrB2Xw2vbqlg9Z5qDjdZ8xImpUSyJDeJ/LFxzMmOIyXaDzZkUr6nq80aqD9yq3F7g7XC\nsLe7wbo7rC64E+CvQbEPCAU824jxsTHm2/29ToNCjSRjDHsqW1i9p4o1e2rYcKCOLpc1AJuXGsV3\nz5hIXmoUiZGhxEaM4mVFlN/zy6AYLA0KZaeObhc7KpooLK7n6fUlFNW0AhAS5OCy/Ey+vWQC2QkB\nsqyI8isaFErZoMflZsOBOiqbOyg4WM/KjaX0uA3TM6JZvjCH82emER2m8zWUb9CgUMoHlDe08/qn\nFbz0STk7KpoQgay4CL42K51zp6cyPSPG7hJVANOgUMqHGGMoLGngw301FBTX88HeaoyB3ORI5mTH\nMiMjhhmZsUxPjw6sRQyVrU40KPxoh3ulfJ+IMC8njnk5cQDUt3bxypZDvLerind2VvFsQRkAYxMi\nOGd6Kl+ZkkJ+ThyiayApH6YtCqVGiDGG8oZ2Cg7W8+THxWwpa6DbZchLjWJqejRn5qWwNC+JiBD9\n+00NLe16UspPtXb28MqWQ/zjk3L2VrVQ19rFmJAgTs1NZFZWLOdNTyM7PkJXvlUnTINCqVHA5TZs\nOFDHS5+UUXCw/rNbb4MdwkkTErgiP4vFuYmUN7QzNS1au6rUcdGgUGoUKq5t5cN9teyvbuGNrRUc\navx857pF4+O54yuTGJ84hmSdIa4GQINCqVHO5TZ8tL+GdftriQ538sdV+2jq6CEkyME3FmTR2uUi\nJz6CS+Zlkh4Tpq0N9SUaFEoFmNqWTgqK63lr22Fe/bQCZ5DQ2uUCICchgsvmZnLpvEzSY3UhQ2XR\noFAqgLV3uXA4YPuhJraXN/La1go+LqpDBObnxJMZH87UtGiuPimH0ODRtRmPGjgNCqXUF5TWtfH8\npjLe31NNdVMHhxo7iB8TwpysWGZnxTI7O5b5Y+MJc2pwBAoNCqWUVx/sreYfnxxiS1kD+6paAIiN\ncHLqxETmZluTA6emRxPsuQ1XxzhGH52ZrZTyanFuEotzrW00mzq62VRcz8uflLPhQB2vfloBQGiw\ng+hwJ8bANxdlc9qkJBrbu8lNiSJDxzoCnrYolApgFY3tFBY3UFhST01LJw1t3aze8/leYiFBDmZl\nxfC12RlkxoYzKVWDwx9p15NSakgdbuxga3kjocEOVu2uYt3+WnYdbgasjdi+uTCHaenRnDQhgcKS\nemLCnZyRl4LLbXh7RyXzcuJIigq1+VOo3rTrSSk1pFJjwkiNsSbyLZmUhDGG7Yea6Oxx8fcNpTy5\nvpij/768bF4mRdUtFJY0EO4M4qmbFjI3O86G6tVw0BaFUuq4dPW4KalrY+3eamIjQthc2sAzG0qI\nHxPCiiXjeWhNEZFhwdz7tWnMyowlIiSIzh43ocEOHSi3iXY9KaVs19njwulw4HAIq3ZXseKJArpd\nhiCHMCYkiKaOHvJSo/jB2ZOpaGyn22VIigrlrKkpepvuCPDLoBCR/wAuAtxAFXCdMeZQf6/ToFDK\nPzS2dVNYWk9hcT21rV2kRIXx5Ppiqps7v3BcXmoUZ09LpbmjmwtnpTMuYQxxY0Jsqnr08tegiDbG\nNHm+vx2Yaoz5dn+v06BQyn+1d7nYXNpASnQo8WNCWH+gjvte20FpXTvOIKHbZQh2CIs9y6yv3VvD\nvLFx3HLaBBraumnvdpGXGoWI0NrZQ1uXSwfNB8gvg+ILBYj8CMg2xtzS37EaFEqNLsYYWjp76Oxx\ns+FAHYXF9azaXcX+6lbix4RQ19r1heMvn5fJ2MQxPL2+hOaObl669RQmJEXaVL3/8NugEJH7gWuA\nRmCpMaa6n5doUCgVIJo6ugl3BrGpuJ6tZY3ERjj5tKyRv31cDFiLHzZ39NDS2UO4M4js+AjOmprC\nxORIlk5OxhkkBDlEB889fDYoROQdIPUYT91jjHm513E/AsKMMT/r431WACsAsrOz5xUXFw9HuUop\nP9DQ1kWYM4gwZxD7q1t4Zn0JrV0utpU3srW8EYCo0GCaO3sIczpYND6B6ekxRIcHMz0jhqb2bjLj\nIpiWHlibP/lsUAy4AJFs4HVjzPT+jtUWhVKqL80d3WwpbeT5TaVkxkXQ1NHN2r01n+0W2Fu409py\nNkiEO87KZWJSJF0u96jdr9wvJ9yJSK4xZq/nx4uAXXbUoZQaPaLCnJyam8ipuYlfeNwYQ11rF9sO\nNREREsT+qha2lDXy3q5K2rpcvLurEsFqXaTFhhEREkxIkIAIl8/LpKalk4tmZ1Be305NSyfnz0zD\nGeSw4yPaxq67nl4AJmPdHlsMfNsYU97f67RFoZQaSmX1bfxtXTEiQlNHNxUN7ZTVtwMQHORgZ0UT\nAA4Bt+dXZV5qFHNz4rhgRhrzxsYREuT7Ewn9vuvpeGhQKKVGitttWH+gDoAnPy7m7GkpdLsMT35c\nzN7K5s92FUyMDCUlOpSM2HDSYsJIjQnnUEM7J01IoK61C4cI50xLISHSvlt5NSiUUmqEtXu6rA5U\nt3KgppX6ti6K69qoauqkpdPaz7zL5f7s+DEhQVy5IJu52XG0d7soqWvjtEmJ5KZE8f7uaoqqW3ix\nsJzs+Age/9YCghxD20LRoFBKKR9hjKG2tYsxIcHsqGgiPTaMhrZu/vudPazaVf2F8ACICgumuaMH\ngDnZsXxS0sB3Tp/ASRMSmJQS9dnziZGhJxQeGhRKKeUHOntc7DncQkRoEMlRoazcWMq/tldy05Lx\nZMaFk5caxbf+upFVu60pZc4gweU2uA1EhATxx+VzOX1y8qDOrUGhlFKjRI/Lza7DzTR1dPPezioi\nQoJIig5jf1UL3zplHNkJEYN6X7+8PVYppdSXBQc5mJ4RA8DJExL7OXrkBNbNwEoppY6bBoVSSimv\nNCiUUkp5pUGhlFLKKw0KpZRSXmlQKKWU8kqDQimllFcaFEoppbzyq5nZIlKNtSz5YCQCNUNYzkjR\nukeOP9YMWvdI8seaASYbY6IG+2K/mpltjEka7GtFpOBEprDbReseOf5YM2jdI8kfawar7hN5vXY9\nKaWU8kqDQimllFeBFBR/sbuAQdK6R44/1gxa90jyx5rhBOv2q8FspZRSIy+QWhRKKaUGQYNCKaWU\nVwERFCJyrojsFpF9InK33fX0RUQOishWEdl85HY2EYkXkbdFZK/nv3E+UOejIlIlItt6PdZnnSLy\nI8+13y0i59hTdZ913ysi5Z5rvllEzuv1nO11i0iWiKwSkR0isl1Evud53Kevt5e6ff16h4nIBhHZ\n4qn7557HffZ6e6l56K61MWZUfwFBwH5gPBACbAGm2l1XH7UeBBKPeuzXwN2e7+8GfuUDdS4B5gLb\n+qsTmOq55qHAOM//FkE+VPe9wA+OcaxP1A2kAXM930cBezy1+fT19lK3r19vASI93zuB9cAiX77e\nXmoesmsdCC2KBcA+Y0yRMaYL+Dtwkc01HY+LgMc93z8OfN3GWgAwxqwB6o56uK86LwL+bozpNMYc\nAPZh/W8y4vqouy8+UbcxpsIYU+j5vhnYCWTg49fbS9198ZW6jTGmxfOj0/Nl8OHr7aXmvhx3zYEQ\nFBlAaa+fy/D+f1g7GeAdEdkkIis8j6UYYyo83x8GUuwprV991ekP1/82EfnU0zV1pEvB5+oWkbHA\nHKy/GP3meh9VN/j49RaRIBHZDFQBbxtjfP5691EzDNG1DoSg8CenGmNmA18FbhWRJb2fNFa70efv\nZ/aXOj3+hNUtORuoAH5rbznHJiKRwAvAHcaYpt7P+fL1PkbdPn+9jTEuz7/DTGCBiEw/6nmfu959\n1Dxk1zoQgqIcyOr1c6bnMZ9jjCn3/LcKeAmrOVgpImkAnv9W2VehV33V6dPX3xhT6flH5gYe4vMm\nuM/ULSJOrF+2TxljXvQ87PPX+1h1+8P1PsIY0wCsAs7FD643fLHmobzWgRAUG4FcERknIiHAlcAr\nNtf0JSIyRkSijnwPnA1sw6r1Ws9h1wIv21Nhv/qq8xXgShEJFZFxQC6wwYb6junIP36Pi7GuOfhI\n3SIiwCPATmPM73o95dPXu6+6/eB6J4lIrOf7cOAsYBc+fL37qnlIr/VIjs7b9QWch3XXxX7gHrvr\n6aPG8Vh3ImwBth+pE0gA3gX2Au8A8T5Q6zNYTdlurP7NG7zVCdzjufa7ga/6WN1/A7YCn3r+AaX5\nUt3AqVjdHJ8Cmz1f5/n69fZSt69f75nAJ576tgH/7nncZ6+3l5qH7FrrEh5KKaW8CoSuJ6WUUidA\ng0IppZRXGhRKKaW80qBQSinllQaFUkoprzQolBohInK6iLxqdx1KHS8NCqWUUl5pUCh1FBH5pmd9\n/80i8qBnwbUWEXnAs97/uyKS5Dl2toh87Fl47aUjC6+JyEQRecezR0ChiEzwvH2kiDwvIrtE5CnP\nDGalfJoGhVK9iMgUYBlwirEWWXMBy4ExQIExZhqwGviZ5yVPAD80xszEmgV75PGngD8YY2YBJ2PN\nCAdrFdU7sPYEGA+cMuwfSqkTFGx3AUr5mDOBecBGzx/74VgLwLmBlZ5jngReFJEYINYYs9rz+OPA\nc541uzKMMS8BGGM6ADzvt8EYU+b5eTMwFlg7/B9LqcHToFDqiwR43Bjzoy88KPLTo44b7No3nb2+\nd6H/BpUf0K4npb7oXeAyEUmGz/ZKzsH6t3KZ55irgLXGmEagXkQWex6/GlhtrB3dykTk6573CBWR\niBH9FEoNIf1rRqlejDE7ROQnwL9ExIG10uytQCvWhjA/weqKWuZ5ybXAnz1BUARc73n8auBBEfmF\n5z0uH8GPodSQ0tVjlRoAEWkxxkTaXYdSdtCuJ6WUUl5pi0IppZRX2qJQSinllQaFUkoprzQolFJK\neaVBoZRSyisNCqWUUl79f3ADBTmyMYk5AAAAAElFTkSuQmCC\n",
            "text/plain": "<matplotlib.figure.Figure at 0x7f01f3fb1d68>"
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This model delivers an acceptable r2 score after a bit more than 300 epochs. For the other two y columns the model looks similar, but we will skip through this repetitive task here.\n\nInstead, let's see how we can learn all 3 y values *at the same time*!"
    },
    {
      "metadata": {
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "# Multi-Task Learning (without missing values)\n\nTo learn several outputs at the same time, we can simply define an output layer with more than one unit. In our case we have three *y* values, so we modify the `build_model1()` function from above to generate an output layer with 3 units by using the line `outputs = Dense(3, activation='linear')(x)`. Apart from that, nothing changes.\n\nThis means that all outputs share the hidden layers of the model, and only the output layer trains different weights for every output."
    },
    {
      "metadata": {
        "collapsed": true,
        "trusted": true
      },
      "cell_type": "code",
      "source": "def build_model3(loss='mean_squared_error'):\n    inputs = Input(shape=(10,), name='input')\n    x = Dense(20, activation='relu', name='hidden_1')(inputs)\n    x = Dense(20, activation='relu', name='hidden_2')(x)\n    outputs = Dense(3, activation='linear', name='output')(x)\n    model = Model(inputs=inputs, outputs=outputs)\n    model.compile(optimizer='adam', loss=loss)\n    return model",
      "execution_count": 78,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "This is how it looks:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "SVG(model_to_dot(build_model3(), show_shapes=True).create(prog='dot', format='svg'))",
      "execution_count": 79,
      "outputs": [
        {
          "data": {
            "image/svg+xml": "<svg height=\"304pt\" viewBox=\"0.00 0.00 251.00 304.00\" width=\"251pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 300)\">\n<title>G</title>\n<polygon fill=\"white\" points=\"-4,4 -4,-300 247,-300 247,4 -4,4\" stroke=\"none\"/>\n<!-- 139646358976048 -->\n<g class=\"node\" id=\"node1\"><title>139646358976048</title>\n<polygon fill=\"none\" points=\"0,-249.5 0,-295.5 243,-295.5 243,-249.5 0,-249.5\" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-268.8\">input: InputLayer</text>\n<polyline fill=\"none\" points=\"112,-249.5 112,-295.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"139.5\" y=\"-280.3\">input:</text>\n<polyline fill=\"none\" points=\"112,-272.5 167,-272.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"139.5\" y=\"-257.3\">output:</text>\n<polyline fill=\"none\" points=\"167,-249.5 167,-295.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"205\" y=\"-280.3\">(None, 10)</text>\n<polyline fill=\"none\" points=\"167,-272.5 243,-272.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"205\" y=\"-257.3\">(None, 10)</text>\n</g>\n<!-- 139646436538856 -->\n<g class=\"node\" id=\"node2\"><title>139646436538856</title>\n<polygon fill=\"none\" points=\"2,-166.5 2,-212.5 241,-212.5 241,-166.5 2,-166.5\" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-185.8\">hidden_1: Dense</text>\n<polyline fill=\"none\" points=\"110,-166.5 110,-212.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"137.5\" y=\"-197.3\">input:</text>\n<polyline fill=\"none\" points=\"110,-189.5 165,-189.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"137.5\" y=\"-174.3\">output:</text>\n<polyline fill=\"none\" points=\"165,-166.5 165,-212.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"203\" y=\"-197.3\">(None, 10)</text>\n<polyline fill=\"none\" points=\"165,-189.5 241,-189.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"203\" y=\"-174.3\">(None, 20)</text>\n</g>\n<!-- 139646358976048&#45;&gt;139646436538856 -->\n<g class=\"edge\" id=\"edge1\"><title>139646358976048-&gt;139646436538856</title>\n<path d=\"M121.5,-249.366C121.5,-241.152 121.5,-231.658 121.5,-222.725\" fill=\"none\" stroke=\"black\"/>\n<polygon fill=\"black\" points=\"125,-222.607 121.5,-212.607 118,-222.607 125,-222.607\" stroke=\"black\"/>\n</g>\n<!-- 139646433365296 -->\n<g class=\"node\" id=\"node3\"><title>139646433365296</title>\n<polygon fill=\"none\" points=\"2,-83.5 2,-129.5 241,-129.5 241,-83.5 2,-83.5\" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-102.8\">hidden_2: Dense</text>\n<polyline fill=\"none\" points=\"110,-83.5 110,-129.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"137.5\" y=\"-114.3\">input:</text>\n<polyline fill=\"none\" points=\"110,-106.5 165,-106.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"137.5\" y=\"-91.3\">output:</text>\n<polyline fill=\"none\" points=\"165,-83.5 165,-129.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"203\" y=\"-114.3\">(None, 20)</text>\n<polyline fill=\"none\" points=\"165,-106.5 241,-106.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"203\" y=\"-91.3\">(None, 20)</text>\n</g>\n<!-- 139646436538856&#45;&gt;139646433365296 -->\n<g class=\"edge\" id=\"edge2\"><title>139646436538856-&gt;139646433365296</title>\n<path d=\"M121.5,-166.366C121.5,-158.152 121.5,-148.658 121.5,-139.725\" fill=\"none\" stroke=\"black\"/>\n<polygon fill=\"black\" points=\"125,-139.607 121.5,-129.607 118,-139.607 125,-139.607\" stroke=\"black\"/>\n</g>\n<!-- 139646364110976 -->\n<g class=\"node\" id=\"node4\"><title>139646364110976</title>\n<polygon fill=\"none\" points=\"10,-0.5 10,-46.5 233,-46.5 233,-0.5 10,-0.5\" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"56\" y=\"-19.8\">output: Dense</text>\n<polyline fill=\"none\" points=\"102,-0.5 102,-46.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"129.5\" y=\"-31.3\">input:</text>\n<polyline fill=\"none\" points=\"102,-23.5 157,-23.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"129.5\" y=\"-8.3\">output:</text>\n<polyline fill=\"none\" points=\"157,-0.5 157,-46.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195\" y=\"-31.3\">(None, 20)</text>\n<polyline fill=\"none\" points=\"157,-23.5 233,-23.5 \" stroke=\"black\"/>\n<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195\" y=\"-8.3\">(None, 3)</text>\n</g>\n<!-- 139646433365296&#45;&gt;139646364110976 -->\n<g class=\"edge\" id=\"edge3\"><title>139646433365296-&gt;139646364110976</title>\n<path d=\"M121.5,-83.3664C121.5,-75.1516 121.5,-65.6579 121.5,-56.7252\" fill=\"none\" stroke=\"black\"/>\n<polygon fill=\"black\" points=\"125,-56.6068 121.5,-46.6068 118,-56.6069 125,-56.6068\" stroke=\"black\"/>\n</g>\n</g>\n</svg>",
            "text/plain": "<IPython.core.display.SVG object>"
          },
          "execution_count": 79,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now to fit the model, we can pass a matrix of width 3 to the model, instead of only a single column of *y*.\n\nWhen we predict, the model generates an (n,3) matrix -- one column per *y* value."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%time\nmodel = build_model3()\n\nmodel.fit(X_std, y, epochs=1000, validation_split=.2, verbose=0, \n    callbacks=[EarlyStopping(patience=10)])\nplot_lc(model)\n\npreds = model.predict(X_test_std)\nprint('Shape of predictions: ', preds.shape)\nprint('r2 score:', r2_score((y_test), preds, multioutput='raw_values'))",
      "execution_count": 80,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Shape of predictions:  (500, 3)\nr2 score: [ 0.90316314  0.90613141  0.90201537]\nCPU times: user 12.5 s, sys: 1.47 s, total: 14 s\nWall time: 7.46 s\n"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4XNW57/HvO6NR712yJMu9ygUbmxKbDqb3DsEcAucA\nh0DCJY3khuSGm5yce+Dk5hJaIBBKMKEEgiHEFGMbDLZsZNybbNmSZfXeNbPuH2sMwki2bGu0R5r3\n8zx6LO3Zs/c7O0E/7bX2WkuMMSillFIupwtQSikVHDQQlFJKARoISiml/DQQlFJKARoISiml/DQQ\nlFJKARoISh2SiLwjIjc5XYdSg0F0HIIKRiKyG/iOMeY9p2tRKlToHYIKWSIS5nQNx2o4fAYVPDQQ\n1JAjIheISJGI1IvIJyIyrcdrPxKRnSLSJCKbROTSHq8tFJGPReRhEakBHvBvWyEi/0dE6kRkl4ic\n2+M9S0XkOz3ef6h9R4nIMv+53xORR0Tk+UN8jov9n6PRX/MC//bdInJmj/0eOHAcEckXESMit4jI\nHuADf7PWvx907HUicpn/+4kiskREakVkq4hcdfRXXw1nGghqSBGRmcDTwL8CKcDjwJsiEuHfZScw\nD0gAfgE8LyJZPQ4xFygGMoAHe2zbCqQCvwWeEhHpo4RD7fsisMpf1wPAjYf4HHOAPwP3AYnAfGD3\n4T5/D6cAk4BzgL8A1/Y49mRgJLBYRGKAJf7a0oFrgD/491HqazQQ1FBzG/C4MeYzY4zXGPMs0AGc\nAGCM+asxZp8xxmeMWQRsB+b0eP8+Y8zvjTHdxpg2/7YSY8yTxhgv8CyQhQ2M3vS6r4jkAccD/9MY\n02mMWQG8eYjPcQvwtDFmib/WMmPMliO4Dg8YY1r8n+F1YIaIjPS/dj3wmjGmA7gA2G2M+ZP/M38O\nvApceQTnUiFCA0ENNSOBe/3NRfUiUg/kAtkAIvLtHs1J9cBU7F/zB+zt5Zj7D3xjjGn1fxvbx/n7\n2jcbqO2xra9zHZCLvZs5Wl8e2xjTBCzG/vUP9m7hBf/3I4G5B12v64HMYzi3Gqa0Q0oNNXuBB40x\nDx78gv8v5CeBM4CVxhiviBQBPZt/AvVYXTmQLCLRPUIh9xD77wXG9PFaCxDd4+fefnkf/Dn+Avxc\nRJYBkcCHPc7zkTHmrEMVrxToHYIKbh4RiezxFYb9hf9vIjJXrBgROV9E4oAY7C/KKgARuRl7hxBw\nxpgSoBDbUR0uIicCFx7iLU8BN4vIGSLiEpERIjLR/1oRcI2IeERkNnBFP0p4G3s38EtgkTHG59/+\nFjBeRG70H88jIseLyKSj+ZxqeNNAUMHsbaCtx9cDxphC4Fbg/wF1wA5gIYAxZhPwX8BKoAIoAD4e\nxHqvB04EaoBfAYuw/RvfYIxZBdwMPAw0AB9hf6ED/Ax791CH7Rh/8XAn9vcXvAac2XN/f3PS2djm\npH3YJq//ACJ6OYwKcTowTakAEZFFwBZjzM+drkWp/tA7BKUGiL8pZoy/CWgBcDHwN6frUqq/tFNZ\nqYGTiW22SQFKgdv9j3kqNSRok5FSSilAm4yUUkr5Dakmo9TUVJOfn+90GUopNaSsWbOm2hiTdrj9\nhlQg5OfnU1hY6HQZSik1pIhISX/20yYjpZRSgAaCUkopPw0EpZRSwBDrQ1BKhaauri5KS0tpb293\nupSgFhkZSU5ODh6P56je72gg+NfNbQK8QLcxZraT9SilglNpaSlxcXHk5+fT99pFoc0YQ01NDaWl\npYwaNeqojhEMdwinGWOqnS5CKRW82tvbNQwOQ0RISUmhqqrqqI+hfQhKqSFBw+DwjvUaOR0IBnhP\nRNaIyG297SAit4lIoYgUHm3yrSmp5dGlx7I4lVJKDX9OB8K3jDEzgHOBO0Vk/sE7GGOeMMbMNsbM\nTks77EC7Xv19XTn/7x+fs2K7tkwppY5cbGxfK6oOL44GgjGmzP9vJXah8DmHfsfRuT/8Jd6M/l98\n7/mPWbe3PhCnUEqpIc+xQPAvfRh34Hvsqk4bAnEuz9hTGW328Af3f3HrUx9RuLs2EKdRSg1zxhju\nu+8+pk6dSkFBAYsWLQKgvLyc+fPnM2PGDKZOncry5cvxer0sXLjwy30ffvhhh6s/PCefMsoAXvd3\ngoQBLxpj/hGQM409A7n4D8x+4w6ed/2Cf3nqPn5/6wJm5iUF5HRKqcD5xd83smlf44Aec3J2PD+/\ncMph93vttdcoKipi3bp1VFdXc/zxxzN//nxefPFFzjnnHO6//368Xi+tra0UFRVRVlbGhg3279z6\n+uBvnXDsDsEYU2yMme7/mmKMeTCgJ5xxLXLtIsa5y/lz2IP84IWPaev0BvSUSqnhZcWKFVx77bW4\n3W4yMjI45ZRTWL16Nccffzx/+tOfeOCBB1i/fj1xcXGMHj2a4uJi7rrrLv7xj38QHx/vdPmHFQzj\nEAbP+LORa19i1HOXcEvLk/xx+QTuOmOc01UppY5Af/6SH2zz589n2bJlLF68mIULF/L973+fb3/7\n26xbt453332Xxx57jJdffpmnn37a6VIPyemnjAbf6FOQE+7gqrCPWPbxctq79C5BKdU/8+bNY9Gi\nRXi9Xqqqqli2bBlz5syhpKSEjIwMbr31Vr7zne+wdu1aqqur8fl8XH755fzqV79i7dq1Tpd/WKF1\nh3DAvHsxq5/msvY3Wbr1PBZMzXS6IqXUEHDppZeycuVKpk+fjojw29/+lszMTJ599ln+8z//E4/H\nQ2xsLH/+858pKyvj5ptvxufzAfDrX//a4eoPb0itqTx79mwzUAvk+F6/ndZ1r/M/x77KQzecPCDH\nVEoFxubNm5k0aZLTZQwJvV0rEVnTn7niQq/JyM81/WpiaaN7x1K8vqETikopFSghGwjknURXWAwn\ndBeyoazB6WqUUspxoRsIYeH48udzsmsDK4trnK5GKaUcF7qBAESMOomRrkp27S52uhSllHJcSAcC\nuXMBcJUNTEe1UkoNZaEdCFnT8Ymb7NYt1LV0Ol2NUko5KrQDwRNJW9woJspetlU0OV2NUko5KrQD\nAZDMqUyUPeyoana6FKXUMHGo9RN2797N1KlTB7Ga/gv5QIgcUUCuq4o9+yqcLkUppRwVmlNX9OBK\nnwBAS/lWArQ+j1JqIL3zI9i/fmCPmVkA5/6mz5d/9KMfkZuby5133gnAAw88QFhYGB9++CF1dXV0\ndXXxq1/9iosvvviITtve3s7tt99OYWEhYWFhPPTQQ5x22mls3LiRm2++mc7OTnw+H6+++irZ2dlc\nddVVlJaW4vV6+dnPfsbVV199TB/7YCEfCCSPBiCsfpfDhSilgtXVV1/NPffc82UgvPzyy7z77rt8\n97vfJT4+nurqak444QQuuuiiI1ro/pFHHkFEWL9+PVu2bOHss89m27ZtPPbYY9x9991cf/31dHZ2\n4vV6efvtt8nOzmbx4sUANDQM/IBaDYSkUQDEt+7B6zO4Xf3/H1Mp5YBD/CUfKDNnzqSyspJ9+/ZR\nVVVFUlISmZmZfO9732PZsmW4XC7KysqoqKggM7P/k2WuWLGCu+66C4CJEycycuRItm3bxoknnsiD\nDz5IaWkpl112GePGjaOgoIB7772XH/7wh1xwwQXMmzdvwD9nyPchEB5Na2Q6I2U/5Q1tTlejlApS\nV155Ja+88gqLFi3i6quv5oUXXqCqqoo1a9ZQVFRERkYG7e3tA3Ku6667jjfffJOoqCjOO+88Pvjg\nA8aPH8/atWspKCjgpz/9Kb/85S8H5Fw9aSAAXfGjyJMK9tZqICilenf11Vfz0ksv8corr3DllVfS\n0NBAeno6Ho+HDz/8kJKSkiM+5rx583jhhRcA2LZtG3v27GHChAkUFxczevRovvvd73LxxRfzxRdf\nsG/fPqKjo7nhhhu47777ArK+gjYZAe7kPLIrdvBxXSsnkuJ0OUqpIDRlyhSampoYMWIEWVlZXH/9\n9Vx44YUUFBQwe/ZsJk6ceMTHvOOOO7j99tspKCggLCyMZ555hoiICF5++WWee+45PB4PmZmZ/OQn\nP2H16tXcd999uFwuPB4Pjz766IB/xpBdD6Gn7iW/hBUP8+i8T7jrTJ1zXalgo+sh9J+uh3CMwpJy\nCRMfrTVlTpeilFKOcbzJSETcQCFQZoy5wJEiEnIB8NXtdeT0SqnhZ/369dx4441f2xYREcFnn33m\nUEWH53ggAHcDm4F4xypIGAGAq0nvEJQKVsaYI3rG32kFBQUUFRUN6jmPtQvA0SYjEckBzgf+6GQd\nxGUBEN6m01coFYwiIyOpqak55l94w5kxhpqaGiIjI4/6GE7fIfw38AMgrq8dROQ24DaAvLy8wFQR\nmUC3eIjqrKOz20d4mHatKBVMcnJyKC0tpaqqyulSglpkZCQ5OTlH/X7HAkFELgAqjTFrROTUvvYz\nxjwBPAH2KaMAFUNHRAopXQ3UtnSSmXD0CauUGngej4dRo0Y5Xcaw5+SfwicDF4nIbuAl4HQRed6p\nYrqjUkiVBmpaOpwqQSmlHOVYIBhjfmyMyTHG5APXAB8YY25wrJ7odFKkkVpdOU0pFaK0sdzPFZdG\nqjRoICilQpbTncoAGGOWAkudrMETn0EKjdQ2a5ORUio06R2CX0RCBhHSTUtjrdOlKKWUIzQQ/Fxx\nGQB0NupYBKVUaNJAOCAmFQDTVOlwIUop5QwNhANi0u2/LdXO1qGUUg7RQDggJg0AT7sGglIqNGkg\nHBCdgg8hoqPG6UqUUsoRGggHuMNoD0sgtrsWn08n0FJKhR4NhB46IlJIoZGGti6nS1FKqUGngdBD\nd1QKSdJEjY5WVkqFIA2EHiQqiSSadPoKpVRI0kDowRWTQpI0ayAopUJSUMxlFCw8canE0UydToGt\nlApBeofQQ0R8Gh7x0tZc53QpSik16DQQevDEpgDQ0ahjEZRSoUcDoQeJtoHg0+krlFIhSAOhp+hk\nAEyL3iEopUKPBkJP/jsEadM+BKVU6NFA6CkqCQB3hwaCUir0aCD0FJmIDxfhnfVOV6KUUoNOA6En\nl4v2sDiiujUQlFKhx7FAEJFIEVklIutEZKOI/MKpWnrq8CQS52uis9vndClKKTWonLxD6ABON8ZM\nB2YAC0TkBAfrAaArIolEmnTGU6VUyHEsEIzV7P/R4/9yfCECX2QSSdJMQ5vOZ6SUCi2O9iGIiFtE\nioBKYIkx5rNe9rlNRApFpLCqqirwRUUnkyRN1LfqHYJSKrQ4GgjGGK8xZgaQA8wRkam97POEMWa2\nMWZ2WlpawGtyx6SQRLMGglIq5ATFU0bGmHrgQ2CB07V4YlOJkk4am5ucLkUppQaVk08ZpYlIov/7\nKOAsYItT9RwQHp8KQHvDIDRPKaVUEHFyPYQs4FkRcWOD6WVjzFsO1gNApD8Qupt1gjulVGhxLBCM\nMV8AM506f19cMTYQvM06wZ1SKrQERR9CUPHPeEprrbN1KKXUINNAOFiUDQRp1wnulFKhRQPhYP4Z\nTz0deoeglAotGggHCwunzRWjM54qpUKOBkIv2sISiOpucLoMpZQaVBoIvegMTyDG24jX5/jUSkop\nNWg0EHrRHZFEkjTR1K7TVyilQocGQi98kck6n5FSKuRoIPRCopP9U2BrICilQocGQi/csSnESysN\nLa1Ol6KUUoNGA6EXnlg7fUVrvU5wp5QKHRoIvYhIsOsudDTpBHdKqdChgdCLaH8gdDXpBHdKqdCh\ngdCLsNgDM57qHYJSKnRoIPQmNh0Ad2ulw4UopdTg0UDoTUwaXlxEtGkgKKVChwZCb1xuGlxJRHVo\nk5FSKnRoIPShyZNCXLcGglIqdBxRIIhIkohMC1QxwaQlPI3Ebn3KSCkVOg4bCCKyVETiRSQZWAs8\nKSIPBb40Z3VEpZNiajFGZzxVSoWG/twhJBhjGoHLgD8bY+YCZx7riUUkV0Q+FJFNIrJRRO4+1mMO\nJF9MBqnSSGNzm9OlKKXUoOhPIISJSBZwFfDWAJ67G7jXGDMZOAG4U0QmD+Dxj4k7IQuAusq9Dlei\nlFKDoz+B8EvgXWCHMWa1iIwGth/riY0x5caYtf7vm4DNwIhjPe5A8STaUppqyhyuRCmlBkfY4XYw\nxvwV+GuPn4uByweyCBHJB2YCnw3kcY9FdEo2AO21pQ5XopRSg6M/ncq/9Xcqe0TkfRGpEpEbBqoA\nEYkFXgXu8fdVHPz6bSJSKCKFVVWDN/tofHouAN31+wbtnEop5aT+NBmd7f9FfQGwGxgL3DcQJxcR\nDzYMXjDGvNbbPsaYJ4wxs40xs9PS0gbitP2SmJJNt3FBU8WgnVMppZzUr05l/7/nA381xjQMxIlF\nRICngM3GmKB7jNUdFkaNJOJp3e90KUopNSj6EwhvicgWYBbwvoikAe0DcO6TgRuB00WkyP913gAc\nd8BUuzOIadMmI6VUaOhPp/KPROS3QIMxxisiLcDFx3piY8wKQI71OIFUF5HNuPb1TpehlFKDoj+d\nyh7gBmCRiLwC3AKExJwOLdE5pPqqobvT6VKUUirg+tNk9Ci2uegP/q/j/NuGvY64PNz4MA06OE0p\nNfwdtskION4YM73Hzx+IyLpAFRRMTGI+AO0VO4hKGeNsMUopFWD9uUPwisiXvw39I5W9gSspeLjS\nJgDQVrbR4UqUUirw+nOHcB/woYgUYzuBRwI3B7SqIBGfmkmViYfKzU6XopRSAdefp4zeF5FxwAT/\npq3GmI7AlhUc0mIj2O7LYVL1FqdLUUqpgOszEETksj5eGisi9DWyeDgZkRjFZyaX4xs+Am83uPtz\nQ6WUUkPToX7DXXiI1www7AMhPiqMTa7xeHzvQsUGyJ7hdElKKRUwfQaCMSYk+gkORUQojZsOLcDe\nzzQQlFLD2hGtqRyKwpLzqHSlwa5lTpeilFIBpYFwGNkJUSwzM6F4KXSHRF+6UipEaSAcRl5KNIs7\npkNnM+x43+lylFIqYA772EwfTxs1AOuNMZUDX1JwGZMWw8O+ArqiM/CsfhImBtWErEopNWD6c4dw\nC/BH4Hr/15PAD4GPReTGANYWFEanxdJNGNtG3QA7P4Ct/3C6JKWUCoj+LpAzyRhzuTHmcmAy9rHT\nudhgGNZGpkTjElgSfzmkTYLF90JbvdNlKaXUgOtPIOQaY3quI1np31YLdAWmrOAREeYmPyWGTRVt\ncNH/heYKePFq6Gx1ujSllBpQ/QmEpSLylojcJCI3AW/6t8UAIfGnckFOAl+UNkDuHLj8SShdBU+f\nA9U7nC5NKaUGTH8C4U7gT8AM/9ezwJ3GmBZjzGmBLC5YTMtJZH9jO5WN7TDlUrj2JWgohcfnQ9Ff\nnC5PKaUGxGEDwRhjgBXAB8D7wDL/tpAxIzcRgNW76+yG8efA7R9D9kz427/Ba7dBR5ODFSql1LHr\nzxKaVwGrgCuAq4DPROSKQBcWTKblJBAbEcbHO6u/2hifDTe9Caf+BNb/FR49Gda/AqGVlUqpYaQ/\nTUb3Y1dNu8kY821gDvCzwJYVXDxuFyeMTuGjrVV87ebI5YZTfwgLF0NkPLx6C/x1IbTWOlarUkod\nrf4EguugAWg1/XzfYYnI0yJSKSIbBuJ4gXTu1EzK6ttYtauXX/YjT4LbPoIzfg5bFsOjJ8Fnj9sp\ns5VSaojozy/2f4jIuyKyUEQWAouBtwfo/M8ACwboWAF1XkEWcZFhPPPJ7t53cLlh3vfh1vchIRfe\n+QE8dRZUbR3UOpVS6mj1p1P5PuAJYJr/6wljzIAMSDPGLAOGRPtKVLibm0/K550N+1lTcoiSs6bD\nd5bAFX+Cut22b2HxvdBSM2i1KqXU0RCnHxgSkXzgLWPM1D5evw24DSAvL29WSUnJ4BV3kKb2Ls77\nv8sxBt6+ex7xkZ5Dv6G5Epb+BtY+CxHxcM7/hunXgMjgFKyUUoCIrDHGzD7cfn3eIYhIk4g09vLV\nJCKNA1tu34wxTxhjZhtjZqelpQ3WaXsVF+nhd9fMpLyhne88U0hj+2EGasemwwUPwb8uh5Sx9hHV\nJ0+HDa+Bzzs4RSulVD/1GQjGmDhjTHwvX3HGmPjBLDKYHJeXxO+umcHne+u49olPqW7uxxoJGZPh\nX96Fi/8ALVXwys3wzPlQszPwBSulVD/peghH4YJp2Tz57dnsrGrmqsdWUlrXj3mNXC6YeT3cvc4G\nQ8Um+MOJ8MadsD/oH7JSSoUARwNBRP4CrAQmiEipiNziZD1H4tQJ6Tx/y1yqmzu45JFPen8ctTcu\ntw2GOz+FGdfB+lfhsZPhuUuhbG1gi1ZKqUNwvFP5SMyePdsUFhY6XcbXbK9o4rbn1rC3tpX7z5/E\nwpPykSPpNG6ttZ3On/weWmtg/AKY/wPImRW4opVSIaW/ncoaCAOgsb2L7y9ax3ubK7hkRja/vmwa\nUeHuIztIeyOsehxWPgJtdTD2TBsMeXMDU7RSKmRoIAwyn8/wh6U7+K8l25iYGc/jN8wiLyX6yA/U\n0QSr//jVHUP2cXDinXaWVdcRhoxSSqGB4JilWyu5+6UijDH87tqZnDYh/egO1NkCa561zUlVWyBt\nIpx8D0y+CMJjBrZopdSwpoHgoD01rfzr82vYsr+RO04dwz1njsfjPsr+e58PNv0NPvoPGwxRyTDh\nPDjhdsjsdSyfUkp9jQaCw9o6vfz8zQ28XFjKcXmJ/O6ameQmH0UT0gE+H5R8DGuega3vQFcLjDnD\nPrE06SJwH2bUtFIqZGkgBIk31+3j/tfWA/CLi6dw6cwRR/YUUm9aa6HwaVj1hF3jOTYTplwCx30b\nMqYMQNVKqeFEAyGI7K1t5XuLiigsqeP8giwevHQqidHhx35gnw+2/xM+fw62LwFft+1jmLUQ8ufb\nwXBKqZCngRBkvD7DYx/t5OEl20iJDec/Lp/GqUfb4dyb1lpY/l/w+fPQXg9Jo2DmDTD2DLvUp1Iq\nZGkgBKkNZQ3cs6iIHZXNnD8ti5+dP5nMhMiBO0FXO2x+0z6hVLLCbht3jp1ldfwCCD+Gfgyl1JCk\ngRDE2ru8PP5RMY8s3YHHJXzvrPHcdFL+0T+J1JfGctsJvfZZaCqHsCg7Xca0qyBnjjYpKRUiNBCG\ngJKaFh54cyMfbq1iYmYc/+uSqRyfnzzwJ/J5YfcKWPcXO/W2twMS8+C4m2DmjRCXMfDnVEoFDQ2E\nIcIYwz83VfDLv2+irL6NK2bl8KNzJ5IaGxGYE7Y3wta3bV/D7uXgCoNR82HaNXZt6MTcwJxXKeUY\nDYQhprWzm99/sIM/Li8myuPmvgUTuW5OHm5XAFdXq94BRc9D0Yv28VWA9Ckw9zaYcb2ObVBqmNBA\nGKJ2VDbxs79tZGVxDdNyEvjFRVOYmZcU2JN2tUP1Ntj1EWx4FfZ9bpuUxp5pm5WyZwT2/EqpgNJA\nGMKMMby5bh+/WryZqqYOzivI5H+cPYHRabGDcXI7tuHTR2HvZ9DVCrlz4fhbYfw5EBmyi+UpNWRp\nIAwDzR3d/HF5MU8sK6aj28c1x+dy9xnjSI8fwMdUD6Wt3jYnrXoC6nbZ/oaJ58OUy2D0qRCVODh1\nKKWOiQbCMFLV1MHvP9jOi5/tweN2ceu8Udw6fzRxkYPUxu/zwZ6V/s7o56C9wYZD9nEw6UKYdRNE\nJgxOLUqpI6aBMAztrm7h//xzK299UU5yTDh3nT6W6+eOJDxsEMcTeLuhdDVs+4d9SqlsDYTHwqhT\nYN69dlS0jm9QKqhoIAxjX5TW85t3tvDJzhpyk6P47unjuHTmCMIGemBbf+wrshPtbX7TrvQWmWhH\nRU+7CjKmQliAHp9VSvWbBsIwZ4xh+fZqfvvuFjaUNTIqNYa7zxjHhdOzA/uoal/aG2DTm7DzA9j8\nd/B12XAouNJO0Z01A451llel1FEZEoEgIguA3wFu4I/GmN8can8NhG8yxrBkUwUPLdnGlv1NjEmL\n4Z4zx3N+QRYuJ4IBoKXaPsK65W3Y8hZ0t0P6ZDu2YdKF9pFWDQelBk3QB4KIuIFtwFlAKbAauNYY\ns6mv92gg9M3nM7y7cT8Pv7eNbRXNjEmLYeFJ+Vw5O5dIj4NrMbfVw8bX4PMXoMz/v11SPow5Hc78\nhT7GqtQgGAqBcCLwgDHmHP/PPwYwxvy6r/doIByez2d4a305Ty0vZl1pA2lxESw8KZ/r5uSRFDMA\nazAci6qtdrW3LxZB5SYQN4zwP6mUMwdy54DLwfBSapgaCoFwBbDAGPMd/883AnONMf9+0H63AbcB\n5OXlzSopKRn0WociYwwrd9bw6Ec7Wb69miiPmytm5fAv3xrFqNQYp8uDvath2zu2z2Hf53bbyG/B\nhHMh70TImAyeKGdrVGqYGDaB0JPeIRydLfsbeWr5Lt4o2keXz8eZkzK4dd5ojs9POvblPAdC4z7Y\n9AasePirOZUSR8IJd0DeXMgoAHeYszUqNYQNhUDQJqNBVtnUznMrS3ju0xLqW7uYlpPALd8axXkF\nWQO/FsPRqi2GLYth9VN2dDRAXBbMvgVGnwIjZmmzklJHaCgEQhi2U/kMoAzbqXydMWZjX+/RQBgY\nbZ1eXl1bytMrdlFc3UJqbDiXHZfDVbNzGZs+CPMl9VdDKez51IbDnk/stqRRkP8tyJpuxzroCGml\nDivoAwFARM4D/hv72OnTxpgHD7W/BsLA8vkMS7dV8tKqvXywpZJun2HWyCSunp3LBdOziA4Pomaa\nlhrYscSu47CvCDqb7PQZGVNhzGn239GnQUyK05UqFXSGRCAcKQ2EwKlq6uC1taUsKtxLcVUQ3zWA\nXQGuvMg2Le35FEo+ttvFDZlTIXkMzLwBMgsgNt3ZWpUKAhoI6qgYY1i1q5bHlxWzfHsVXp9h3rg0\nLjtuBOdMyXR2TENfanbawXDb3rFzK+1fb6fRAEjItU8uTbrQTuOtU2moEKSBoI5ZTXMHf/p4N69/\nXkZZfRuJ0R4umWGDYc6oZGemyOiPtjooWWk7pfd8asc++LrAE22XCc2fB+mTIHk0pI5zulqlAk4D\nQQ0Yn8+wsriGF1ft4Z8b99PlNWTER3Du1CwWnpRPfjCMaziU9kbbrLR9CZR8AlWbv3otOsWu7zBq\nvp1eIz7enQmMAAAT9ElEQVQbwqOdq1WpANBAUAHR0tHNB1sqWfxFOe9vqaDLa5g9MomLZmRzfkEW\nKbFDoEmmsdz/BNMnsOE12x9xQHSKbWKKSYfoZDv/ksutTzOpIU0DQQVcZWM7Lxfu5e/rytla0USY\nSzhlfBqXzBzBWZMzgrO/4WDGQP0eaKmyI6Z3vGeXDj3QBwEgLphwHqSMgfQpMPJEiB+h4yHUkKGB\noAbVlv2N/O3zfbxRVEZ5QztxEWGcW5DJJTNHcMKoFOdmXj1anS1QuRmKl9oO68+ft7O2+rrs67EZ\ntokpdbwNipk3QHiQN52pkKWBoBzh9Rk+K67h9c/LeGfDfpo7uhmRGMXCk/K5cHo2mQmDtB50IPh8\ndrW4/V/A3lX2jqJm+1evRyXbx17zToJxZ9mxEZ4h/HnVsKGBoBzX1ullyeYKnv+0hFW7agGYPTKJ\ncwuyOHdqJtmJQ3zyOp/PNjXVl9j1pltrYcOr0NlsXw+LgvSJ9nHXuEw7aV/uXF0LQg06DQQVVHZW\nNfP2F+W8vWE/m8sbATguL5GLZ4zglPFpwf+kUn/V77UhUbnZjoco+diOk+hqsa9HxNs+icwCmHi+\nHTiXNQPCIu2cTboetQoADQQVtIqrmnlnw35eW1vKzir7i3JiZhwXzcjm8uNyyIgfZs0sxthO6q1v\n24Fzxthpv+sPmso9dQKMP8feQeQcb8dLRCU6U7MaVjQQVNAzxrC3to0lmyt4e305a0rqcLuECRlx\nHDcykVu+NTo41m4IlOZKO/X3vrXQ0QSrnrSPw7rc4Ou2+2ROg4wpdoT1+HMhaxp4u3QZUnVENBDU\nkLOruoWXVu9h5c4aNpQ14DMwJTueC6bZMQ55KcN8wJjPC95OO2nfpjfsSOsd79vJ/Lrbvr5v6gQ7\nyvrAmImEHIhKgvgsZ2pXQU0DQQ1p++rbeHt9OW99UU7R3noARqfFcOLoFE4em8rpE9OHxjiHgdDV\nbu8Y9n4KVdsAA5vfsuMnGku/vm/KOBsOcZl2kr+MyXYUtrjs1B16VxGSNBDUsLG3tpW315ezalct\nH++spr3LR7jbxSkT0lgwJZNZI5OGT6f0kfD5oHQVNJXb0dcdjfZuomEvtNV/Myyiku2/KWNgyqU2\nNGLS7CC7xDxwewb/M6hBoYGghiWvz7BiRzVLt9rpMyqbOgDISYrispkjOGF0CrPzkwkP06d1aK2F\n3cuhbrftd6jeZgfPlRbasRQ9RSbap5zcYTB+gX1Syh0BM66zTVHRKRARZNOgq37TQFDDntdn2Lq/\nifc3V/DPTRWsL2sAIDYijDMmpXPC6BTOnJRBWtwQmF9psFVttc1QLdX2aaeiF+0jsS2VdsBdRIId\nT2G8dn93uA2GxDw7Sjtntg2QiRfYoOhstc1RniE+tmSY0kBQIaelo5tPdtbw7sb9LN1aRXWzvXvI\niI9gwZRMzi3IYk5+8tCbRmMwHXhENjIBmivs7LAt1bb5qbXWjq2o222bp8D2TcSPsE1UEXH2sdmE\nERCdCrU77biMix+BxjK7LSbFnkP7MgaVBoIKacYYNpU38smOGlbtrmXZtio6un0kRnuYk5/M3NEp\nzBqZRGZ8JBnxEYj+guo/n9fO61S21o7Mbq+322p22LDoav36/tEpNkxi0mDUPPvk1Jxb7ZoUbXWQ\newKkTfiqD8Pbpf0ZA0wDQake2jq9vLOhnJU7a/h0Vw17a796jDMzPpJTJ6Rx+awcRqfGDI0pvIOV\nMXaep9h0O6iutNA/MWCHbZqqLbZLnR4Yud1T+hQbNHW77OjtuEy7oNGY0+1yqePPgbSJNjB2fWT7\nOnTG2X7RQFDqEPbVt/FFaQPbK5r4W1HZlyOmRWBGbiJnTEzntInpTMqM1yamgeTz2ek52hvsY7Pe\nTruqXeM++4RUazWkjLVNU/inJj+YuMD4bFDEZcP+dTBilp1M0Ndt3x+bbgf16Z0foIGg1BHZXd1C\nSW0rRXvq+WBLBetKbQd1QpSH4/OTOW5kIjNzk5iZlxg64x+CwZ5PYc9KGHcO7FgCXW22s7umGHa+\nbzuxE3K/+dQU2KekwqNtf0jSKDunVNoE2yk+4VzwdtiAik2DhjIbKqnjbOB0NPrnnRoegRLUgSAi\nVwIPAJOAOcaYfv2W10BQg6WysZ3l26v5bFcNhbvrKK62dxAx4W5m5yczZ1Qyo1JjmJmXSFpsBGFu\nfcx10HW126efRKC1xq5h4YmCyk12CpCqrXZba42dbDA23W5rqTz0cd3h9s5lwnmQONLeraSM8Tdf\nnWGbu3Yts3ckWTPsHU93p+33MCYoJygM9kCYBPiAx4H/oYGggl1dSydrSup4b3MFq3fXftnEBLYP\n4uKZ2Zw9OYPMhCjS4yLwaEAEr85W+2htez10NNuBfEn59i6js8U+XVW52YZKd3svBxDA/3szKtmO\n7WjYa0eCIzD9GrstKtHeZXQ02eMWXGH/zZ4Ji79v73oyptintNxhAf3IQR0IX55cZCkaCGoIqmxs\np6y+jdW7a3l/cyVr99TR5bX/LaXEhDMzL5Fr5+QxOTuelJgIHSg3FHX6Q1/cdqW8ik2w+knbRzHx\nAqjYYO8UWmtsk1NMCnhi7Cp7vXWa9yVjKky9DHZ+COGxkDbennPcWVC7y/aXdLXBxPPstCRHYdgE\ngojcBtwGkJeXN6ukpKSvXZVyTGN7Fyu2V1Pd3MF7myv5orSe+la73GZ0uJtx6bGcV5DFSWNSSYkN\nH/qLA6m+ebttIHQ02a/ipXa51bJC2z+x7Z/QvN82Se351G7vj2sXwYQFR1WS44EgIu8Bmb28dL8x\n5g3/PkvROwQ1DLV3eVm9u5btFc1sLm/kwx4D5cDO4nrSmBSOy0ui0+vjvIIsbWYKVS3VdhxH5SYY\nebLt0PZ57TxVyWNs85MnGqKTj/oxW8cDoT80EFSo8PoMlU3trCmpY09tK0u3VlG0p55Orw+A1NgI\nGtu6mJ2fxB2njmVydjxJ0R4dMKcGRH8DIbA9GUopANwuISshigum2aaiO04dS3NHN9sqmthc3sgH\nmytJi4tg8fpybnjqMwCyEiK5anYu6fERzBqZxPj0OB0ToQLKqaeMLgV+D6QB9UCRMeacw71P7xDU\ncNfW6WXZ9ir21rayZFMFn+2q/fK1uIgwIsPdXDcnj3MLMpmQEad3EKpfhkST0ZHSQFChpqy+jc5u\nH2tL6igsqWNbRRNrSuoASI4JZ+6oZMZnxJGdGElOUjQTM+N06g31DdpkpNQwMML/NNKo1Bgun2Uf\nOSxvaGP59mpW7apl+fYq3tmw/8v9XQKzRyaTlRjJtJxEJmTEYTAcn5+sI6zVYekdglJDmDGGbp9h\ne0Uz+xvb+HxP/ZdTf5c3fDWoKiHKw5TseMLcLk4dn8askUmkx0foKOsQoU1GSoW4isZ2dlY1U93c\nyYrtVWzZ30RLR/fXRlnnJEVx1uQM4iLCSImNYFpOAlOyE3Qg3TCjTUZKhbiM+Egy4iMBuGh6NmDv\nKDaXN7Gvvo2y+jZeLtzLK4WlNHV0f/m+MJcwOi2GqSMSiAkPY3xmHLPykpicHe/I51CDRwNBqRAi\nIkzOjv/yl/tNJ+UD4PMZKpraKdpTzxdlDWwub+TjHdW0dni/DItx6bFkJ0bR0tFNWX0bN544kklZ\n8WQnRBEd7mZEYpQ+FjvEaZORUqpPPp+hrL6NJZsq+HhHNZVNHVQ1dbC/8ZuTvqXGhnPtnDx8xpCX\nHE1uUjQRHhc7K1vYuK+Bn184RQPDIdqHoJQKmC37G9lV1YKIUNfaCcA7G/azbFtVn+8Zmx7LWZMz\nSIzy4BLhxDEpTMmO17EUg0ADQSk16KqbO4gOd1O4u46Obh/1/rD4ZGcNa/fUUVLz9fWW85KjGZUa\ng9sldHl9TM6K5+SxqcwdnYzPBwZDdLi2bB8rDQSlVNDx+gwVje2EuYSlW6t4Z0M5Vc0deH22M3t9\nWcM33jM6LYYZuYl0dvs4a3IG88el0djeRU5SNG5tguoXDQSl1JCzq7qFLeWNbN7fREeXFwNs2tfI\nmpI6YiLCvjZj7IGpPEYmR1PV3MHZkzM4aUwqucnRZMRH0OU1xES4iQjTAXkaCEqpYcXrM6zcWcOG\nfQ1EhLnYVtFEc4eXz/fUUdPcSVuX9xvviQhzMTY9lqTocHKTozhhdApj02OJDg9jZHI0Bmju6Kbb\n6xvWU35oICilQkpbp5e1e+qoauqgorEdt0sorWtjc3kjpXVt1LR00N7l+3L/cLeLSI+LTq8Pr88w\nOSueCI+bnKQo5o5KZt64NJrau9lZ1cyCKZm0dXmJiRia/Rk6ME0pFVKiwt2cPDa1z9d9PsN7myto\n7/ZR2dhOZVMHtS2dtHd5iY0Io6y+jY5uH+9tquC1tWVfe29MuJuWTi/pcRHER3mI9LhIiYkgJymK\nk8emEuYSkmLC8foMs0cmDdnpQDQQlFIhweUSzp7S2yKOX9fe5WV3TQsrtlfjM4bmDi87K5sZkxbD\nyuIadte0MiIxntqWTlbvruWFz/Z87f05SVGMz4gjOtxNfkoM2YlRZCVEkpMURWpsBInRHqqaOoiL\n9BAVHlz9GxoISinVQ6THzcTMeCZm9j5Vh89nvhxg19TeRUlNKz5j2Lq/ifYuL+9trqSyqZ3a5k7e\n+qL8G++P8rhp6/KSEhPO9NxEBMhOjCI5JpzTJ6bT0tGNwT6S29LZbUMkyjModx3ah6CUUgFgjKHT\n66O4qoXiqhbau7zUt3Wxt9YGyK7qFkrr2ogOd7Otookub9+/izPiI3j4qhmcdIgmsUPRPgSllHKQ\niBAR5mZSVjyTsg49MWBnt4+Wjm4+2FJJalwEHrdQXNWCS4TG9i4+La4hPzUm8DXrHYJSSg1v/b1D\nGJpd4UoppQacBoJSSinAoUAQkf8UkS0i8oWIvC4iiU7UoZRS6itO3SEsAaYaY6YB24AfO1SHUkop\nP0cCwRjzT2PMgTX7PgVynKhDKaXUV4KhD+FfgHf6elFEbhORQhEprKrqe/ENpZRSxyZg4xBE5D2g\nt3Hi9xtj3vDvcz/QDbzQ13GMMU8AT4B97DQApSqllCKAgWCMOfNQr4vIQuAC4AwzlAZDKKXUMOXI\nwDQRWQA8BJxijOl3O5CIVAElR3naVKD6KN87XOg1sPQ66DWA0LoGI40xaYfbyalA2AFEADX+TZ8a\nY/4twOcs7M9IveFMr4Gl10GvAeg16I0jcxkZY8Y6cV6llFJ9C4anjJRSSgWBUAqEJ5wuIAjoNbD0\nOug1AL0G3zCkZjtVSikVOKF0h6CUUuoQNBCUUkoBIRIIIrJARLaKyA4R+ZHT9QSKiDwtIpUisqHH\ntmQRWSIi2/3/JvV47cf+a7JVRM5xpuqBJSK5IvKhiGwSkY0icrd/e8hcBxGJFJFVIrLOfw1+4d8e\nMtfgABFxi8jnIvKW/+eQuwZHYtgHgoi4gUeAc4HJwLUiMtnZqgLmGWDBQdt+BLxvjBkHvO//Gf81\nuAaY4n/PH/zXaqjrBu41xkwGTgDu9H/WULoOHcDpxpjpwAxggYicQGhdgwPuBjb3+DkUr0G/DftA\nAOYAO4wxxcaYTuAl4GKHawoIY8wyoPagzRcDz/q/fxa4pMf2l4wxHcaYXcAO7LUa0owx5caYtf7v\nm7C/DEYQQtfBWM3+Hz3+L0MIXQMAEckBzgf+2GNzSF2DIxUKgTAC2Nvj51L/tlCRYYwp93+/H8jw\nfz/sr4uI5AMzgc8IsevgbyopAiqBJcaYkLsGwH8DPwB8PbaF2jU4IqEQCMrPP4lgSDxnLCKxwKvA\nPcaYxp6vhcJ1MMZ4jTEzsGuNzBGRqQe9PqyvgYhcAFQaY9b0tc9wvwZHIxQCoQzI7fFzjn9bqKgQ\nkSwA/7+V/u3D9rqIiAcbBi8YY17zbw656wBgjKkHPsS2i4fSNTgZuEhEdmObiU8XkecJrWtwxEIh\nEFYD40RklIiEYzuO3nS4psH0JnCT//ubgDd6bL9GRCJEZBQwDljlQH0DSkQEeArYbIx5qMdLIXMd\nRCTtwDrlIhIFnAVsIYSugTHmx8aYHGNMPva/+Q+MMTcQQtfgaDgyud1gMsZ0i8i/A+8CbuBpY8xG\nh8sKCBH5C3AqkCoipcDPgd8AL4vILdipw68CMMZsFJGXgU3YJ3PuNMZ4HSl8YJ0M3Ais97ehA/yE\n0LoOWcCz/qdkXMDLxpi3RGQloXMN+hJK/z84Yjp1hVJKKSA0moyUUkr1gwaCUkopQANBKaWUnwaC\nUkopQANBKaWUnwaCUoNERE49MOumUsFIA0EppRSggaDUN4jIDf71BIpE5HH/RHHNIvKwf32B90Uk\nzb/vDBH5VES+EJHXD8yvLyJjReQ9/5oEa0VkjP/wsSLyiohsEZEX/COrlQoKGghK9SAik4CrgZP9\nk8N5geuBGKDQGDMF+Ag7Chzgz8APjTHTgPU9tr8APOJfk+Ak4MAMmzOBe7Brc4zGjqxWKigM+6kr\nlDpCZwCzgNX+P96jsBOg+YBF/n2eB14TkQQg0RjzkX/7s8BfRSQOGGGMeR3AGNMO4D/eKmNMqf/n\nIiAfWBH4j6XU4WkgKPV1AjxrjPnx1zaK/Oyg/Y52zpeOHt970f8GVRDRJiOlvu594AoRSYcv1+Ad\nif1v5Qr/PtcBK4wxDUCdiMzzb78R+Mi/UlupiFziP0aEiEQP6qdQ6ijoXydK9WCM2SQiPwX+KSIu\noAu4E2jBLjTzU2wT0tX+t9wEPOb/hV8M3OzffiPwuIj80n+MKwfxYyh1VHS2U6X6QUSajTGxTteh\nVCBpk5FSSilA7xCUUkr56R2CUkopQANBKaWUnwaCUkopQANBKaWUnwaCUkopAP4/s9sASaGxXD4A\nAAAASUVORK5CYII=\n",
            "text/plain": "<matplotlib.figure.Figure at 0x7f01f3893978>"
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Lessons learned: MTL without missing values\n\n- by fitting only a single model we can learn three target variables! Isn't that great?\n- The r2 score is very comparable to the single-output model, and the timing is only slightly slower, even though we have learnt three variables.\n\n**So even if MTL does not improve the performance, it can drastically reduce the training time.**\n\nOn real-world data sets it can also reduce overfitting."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Warning: scale the targets!\nThe loss function simply sums up the quadratic loss of all targets. If your targets are on a different scale, it is mandatory to standardize them! Here, we skipped step for simplicity (and the targets' scale does not differ too badly)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Multi-Task Learning with missing labels\n\nIn the previous section we have seen how MTL can reduce the training time. However, it really shines when you have a lot of data that only has labels for some of the output variables. \n\nNormally, if you have missing values for the label, you would filter out those columns. Not with MTL -- if any of the labels has a value, we can still use that example for our training!\n\nThe loss function will simply compute the loss for the labels with known values and sets the loss for missing labels to zero*. That way the missings do not not contribute to the loss function, and hence do not influence the gradients and subsequently the optimization. \n\n**) Zero works for the mean squared error loss. Other functions may need different treatment.*\n\n## Generate data\nLet us remove 70% of our labels by settings them to NaN. That leaves us with merely ~150 labeled examples per target, out of an original 500. The `mask_data()` function is defined above. It simply sets random examples to NaN."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "def mask_data(y, nan_rate):\n    # randomly set some y values to nan\n    y_wm = y.copy()\n    y_wm[rndst.binomial(1, nan_rate, y_wm.shape) == 1] = np.nan\n    return y_wm\n\n# y with missings\ny_wm = mask_data(y, nan_rate=.7)\ny_wm[:10,:]",
      "execution_count": 81,
      "outputs": [
        {
          "data": {
            "text/plain": "array([[         nan,          nan,  11.48166491],\n       [         nan,  10.83981353,          nan],\n       [         nan,          nan,          nan],\n       [         nan,          nan,          nan],\n       [  4.38298642,   9.63693232,          nan],\n       [         nan,          nan,          nan],\n       [  5.9671087 ,          nan,          nan],\n       [  5.74760774,          nan,  13.6949167 ],\n       [  5.40848614,          nan,          nan],\n       [         nan,          nan,          nan]])"
          },
          "execution_count": 81,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "A lot of NaNs, right? You will see that MTL handles them quite well. Unfortunately, Keras does not support missing values out of the box:"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model.fit(X, y_wm)",
      "execution_count": 82,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Epoch 1/1\n500/500 [==============================] - 0s - loss: nan     \n"
        },
        {
          "data": {
            "text/plain": "<tensorflow.contrib.keras.python.keras.callbacks.History at 0x7f01f34762e8>"
          },
          "execution_count": 82,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "See? The loss is always nan. So we have to define our own loss function that ignores missing values. But let us first see how our model performs if we go for the classical approach:"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Train model with single output, filter missings\n\nThis is the classical approach -- train a model for one target only and filter missing values. For a dataset with 70% missing values this results in a big loss of training data."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%time\nimport tensorflow as tf\n\nmodel = build_model1()\n\n# mask of only finite (=non-missing) values in first column of y\nnotnan = np.isfinite(y_wm[:,0])\n\n# fit on non-missing values\nmodel.fit(X_std[notnan], y_wm[notnan,0], epochs=1000, validation_split=.2, verbose=0, \n    callbacks=[EarlyStopping(monitor='val_loss', min_delta=0, patience=10, verbose=0, mode='auto')])\nplot_lc(model)\n\npreds_test = model.predict(X_test_std)\nprint('r2score:', r2_score(y_test[:,0], preds_test, multioutput='raw_values'))",
      "execution_count": 83,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "r2score: [ 0.54153904]\nCPU times: user 5.03 s, sys: 476 ms, total: 5.51 s\nWall time: 3.16 s\n"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd0XOW1+P3vVu/FqlazLXdbcpVNtYFgSighVIcW4BdC\n2oU0uCGX5EISeHNDErj3JuQSSIOEgKkJoYRmgzHVcpF7x7Il2Vaxqm21mf3+cY5s4diSbEs6o5n9\nWWvWzJxz5pz9aC3Nnqec5xFVxRhjjAnzOgBjjDGBwRKCMcYYwBKCMcYYlyUEY4wxgCUEY4wxLksI\nxhhjAEsIxvRIRF4VkRu8jsOYwSB2H4IJRCKyHbhZVd/0OhZjQoXVEEzIEpEIr2M4UcFQBhM4LCGY\nIUdELhKRlSLSICLvi8iUbvvuFJGtItIsIutE5NJu+24UkfdE5EERqQPucbctEZFfiEi9iHwiIp/t\n9pm3ReTmbp/v6dhRIrLYvfabIvKQiPylh3Jc4pajyY35fHf7dhGZ1+24e7rOIyIjRURF5EsisgNY\n6DZr/dth5y4Tkcvc1xNE5A0R2SsiG0XkquP/65tgZgnBDCkiMh34A/AVIA34LfCiiES7h2wF5gDJ\nwI+Av4jI8G6nOAnYBmQB93XbthFIB+4Hfi8icpQQejr2r8DHblz3ANf3UI7ZwOPAHUAKMBfY3lv5\nuzkDmAicBzwJXN3t3JOAEcDLIhIPvOHGlgl8AfiNe4wxn2IJwQw1twC/VdWPVNWnqo8BbcDJAKr6\njKpWqapfVRcAm4HZ3T5fpaq/UtVOVT3gbitX1UdV1Qc8BgzHSRhHcsRjRaQAmAX8p6q2q+oS4MUe\nyvEl4A+q+oYba6WqbjiGv8M9qrrPLcMLwDQRGeHuuxZ4XlXbgIuA7ar6R7fMK4DngCuP4VomRFhC\nMEPNCOC7bnNRg4g0APlADoCIfLFbc1IDUITza77LziOcc3fXC1Xd775MOMr1j3ZsDrC327ajXatL\nPk5t5ngdPLeqNgMv4/z6B6e28IT7egRw0mF/r2uB7BO4tglS1iFlhpqdwH2qet/hO9xfyI8CZwMf\nqKpPRFYC3Zt/BmpY3S5gmIjEdUsK+T0cvxMYfZR9+4C4bu+P9OV9eDmeBO4WkcVADLCo23XeUdVz\negreGLAagglskSIS0+0RgfOF/1UROUkc8SJyoYgkAvE4X5Q1ACJyE04NYcCpajlQitNRHSUipwAX\n9/CR3wM3icjZIhImIrkiMsHdtxL4gohEikgJcEUfQngFpzbwY2CBqvrd7S8B40Tkevd8kSIyS0Qm\nHk85TXCzhGAC2SvAgW6Pe1S1FPgy8GugHtgC3AigquuAXwIfAHuAYuC9QYz3WuAUoA64F1iA07/x\nL1T1Y+Am4EGgEXgH5wsd4Ic4tYd6nI7xv/Z2Ybe/4HlgXvfj3eakc3Gak6pwmrx+BkQf4TQmxNmN\nacYMEBFZAGxQ1bu9jsWYvrAagjH9xG2KGe02AZ0PXAL8zeu4jOkrzzuVRSQcp+21UlUv8joeY05A\nNk6zTRpQAXzNHeZpzJDgeZORiHwHKAGSLCEYY4x3PG0yEpE84ELgd17GYYwxxvsmo/8G/h1I7MvB\n6enpOnLkyAENyBhjgs2yZctqVTWjt+M8SwgichFQrarLROTMHo67BWe6AgoKCigtLR2kCI0xJjiI\nSHlfjvOyyeg04HPuvPdPAZ850syQqvqIqpaoaklGRq8JzhhjzHHyLCGo6vdVNU9VR+LcNLNQVa/z\nKh5jjAl1dh+CMcYYwPtOZQBU9W3gbY/DMMYEqI6ODioqKmhtbfU6lIAWExNDXl4ekZGRx/X5gEgI\nxhjTk4qKChITExk5ciRHX7sotKkqdXV1VFRUMGrUqOM6hzUZGWMCXmtrK2lpaZYMeiAipKWlnVAt\nyhKCMWZIsGTQuxP9G4VGQvhkMbz7gNdRGGNMQAuJhKCbXkcX/gRqNnkdijFmCEpIONqKqsElJBLC\nvQ3ncECj8L39X16HYowxASskEsKcaRP5U+e5hK19HqrXex2OMWaIUlXuuOMOioqKKC4uZsGCBQDs\n2rWLuXPnMm3aNIqKinj33Xfx+XzceOONB4998MEHPY6+dyEx7PSMcRk8mnk1N+x9g9i3f0bYVX/y\nOiRjzHH60T/Wsq6qqV/POSknibsvntzrcc8//zwrV66krKyM2tpaZs2axdy5c/nrX//Keeedx113\n3YXP52P//v2sXLmSyspK1qxZA0BDQ0O/xjwQQqKGICLcNG8mf+6ch6z7O+z9xOuQjDFD0JIlS7j6\n6qsJDw8nKyuLM844g6VLlzJr1iz++Mc/cs8997B69WoSExMpLCxk27Zt3Hrrrfzzn/8kKSnJ6/B7\nFRI1BICzJ2byx7Qr+FLjq0R88BBy4S+8DskYcxz68kt+sM2dO5fFixfz8ssvc+ONN/Kd73yHL37x\ni5SVlfHaa6/x8MMP8/TTT/OHP/zB61B7FBI1BHBqCZfOLeH5ztPxL/8z7Kv1OiRjzBAzZ84cFixY\ngM/no6amhsWLFzN79mzKy8vJysriy1/+MjfffDPLly+ntrYWv9/P5Zdfzr333svy5cu9Dr9XIVND\nALh46nCufeXzzPe9Dcv+BHNv9zokY8wQcumll/LBBx8wdepURIT777+f7OxsHnvsMX7+858TGRlJ\nQkICjz/+OJWVldx00034/X4AfvrTn3ocfe88X1P5WJSUlOiJLpDz4BubOGnxDcxKaSby22UQFt5P\n0RljBsr69euZOHGi12EMCUf6W4nIMlUt6e2zIdNk1OXakwt4SucR2bwTti70OhxjjAkYIZcQMhNj\naB97AXUk418a2B08xhgzmEIuIQBcVjKKBZ1nIJv/Cc27vQ7HGGMCQkgmhLMmZLIo+ixE/bDu716H\nY4wxASEkE0JkeBhTp5/MBn8+naue9TocY4wJCCGZEAAumZbLi75TiKj8GBp2eh2OMcZ4LmQTQlFu\nEqUJZzpv1r7gaSzGGBMIQjYhiAjTps5glb+QzjXPex2OMSaI9LR+wvbt2ykqKhrEaPrOs4QgIjEi\n8rGIlInIWhH50WDHcN7kbF7zlRCxawU07xnsyxtjTEDxcuqKNuAzqtoiIpHAEhF5VVU/HKwApuWn\ncH9UCejTsOUNmH7dYF3aGHO8Xr0Tdq/u33NmF8Nnj76A1p133kl+fj7f+MY3ALjnnnuIiIhg0aJF\n1NfX09HRwb333ssll1xyTJdtbW3la1/7GqWlpURERPDAAw9w1llnsXbtWm666Sba29vx+/0899xz\n5OTkcNVVV1FRUYHP5+OHP/wh8+fPP6FiH86zhKDOnBkt7ttI9zGo82iEhwmZY2dRvWkYGZteQywh\nGGOOYP78+XzrW986mBCefvppXnvtNW677TaSkpKora3l5JNP5nOf+9wxLXT/0EMPISKsXr2aDRs2\ncO6557Jp0yYefvhhvvnNb3LttdfS3t6Oz+fjlVdeIScnh5dffhmAxsbGfi+np5PbiUg4sAwYAzyk\nqh8d4ZhbgFsACgoK+j2GM8dn8ta6qVy1ZRHhvk4ID6n5/owZenr4JT9Qpk+fTnV1NVVVVdTU1JCa\nmkp2djbf/va3Wbx4MWFhYVRWVrJnzx6ys7P7fN4lS5Zw6623AjBhwgRGjBjBpk2bOOWUU7jvvvuo\nqKjgsssuY+zYsRQXF/Pd736X733ve1x00UXMmTOn38vpaaeyqvpUdRqQB8wWkX/paVHVR1S1RFVL\nMjIy+j2GueMyeN8/mfCOZthd1u/nN8YEhyuvvJJnn32WBQsWMH/+fJ544glqampYtmwZK1euJCsr\ni9bW1n651jXXXMOLL75IbGwsF1xwAQsXLmTcuHEsX76c4uJifvCDH/DjH/+4X67VXUCMMlLVBmAR\ncP5gXzsjMZrqYe4kgNuXDPbljTFDxPz583nqqad49tlnufLKK2lsbCQzM5PIyEgWLVpEeXn5MZ9z\nzpw5PPHEEwBs2rSJHTt2MH78eLZt20ZhYSG33XYbl1xyCatWraKqqoq4uDiuu+467rjjjgFZX8HL\nUUYZIpLivo4FzgE2eBHLuDFj2aY5+D9514vLG2OGgMmTJ9Pc3Exubi7Dhw/n2muvpbS0lOLiYh5/\n/HEmTJhwzOf8+te/jt/vp7i4mPnz5/OnP/2J6Ohonn76aYqKipg2bRpr1qzhi1/8IqtXr2b27NlM\nmzaNH/3oR/zgBz/o9zJ6th6CiEwBHgPCcRLT06raYx2oP9ZDOJJXVu+ifsE3+ELsR4TfWW79CMYE\nGFsPoe+G5HoIqrpKVaer6hRVLeotGQykk0YN40P/RMI7WqwfwRgTsuynMJCWEM2ulJmwH6cfIXem\n1yEZY4a41atXc/31139qW3R0NB999C+DKQOGJQRX/ohCytfnMKL8Azjtm16HY4w5jKoe0xh/rxUX\nF7Ny5cpBveaJdgEExCijQDA1L5lSXyG+ilIYQutMGxMKYmJiqKurO+EvvGCmqtTV1RETE3Pc57Aa\ngmtqfgov+Edz+f4l0FQJyXleh2SMceXl5VFRUUFNTY3XoQS0mJgY8vKO/7vLEoJr4vAk7pMxzpvK\nZZYQjAkgkZGRjBo1yuswgp41GbliIsPRrCI6iYDK/r/hwxhjAp0lhG4m5WeyUQtQSwjGmBBkCaGb\nqfkpLPcV4q9aAX6/1+EYY8ygsoTQTVFuEqu0kPD2Zqjb4nU4xhgzqCwhdFOYnsAa3I7lKms2MsaE\nFksI3URFhKFpY+kgEvas8TocY4wZVJYQDjN2eCrbJB/2rPU6FGOMGVSWEA4zITuRVR15+HdbQjDG\nhBZLCIcZl5XIBi0gbN8eaLG7Io0xocMSwmEmZCeyQfOdN9VWSzDGhA5LCIfJTYllR4R7i/yedd4G\nY4wxg8gSwmHCwoT07DwawlKtY9kYE1IsIRzBhOxE1vvzbeipMSakWEI4gtEZCazuzEdrNoCv0+tw\njDFmUFhCOILCjHg2+vORzlbYu83rcIwxZlBYQjiCUekJbNAC542NNDLGhAjPEoKI5IvIIhFZJyJr\nRSRgFjLOS43lE8nFT7h1LBtjQoaXNYRO4LuqOgk4GfiGiEzyMJ6DIsPDyB6Wwp7IPEsIxpiQ4VlC\nUNVdqrrcfd0MrAdyvYrncKPS49mEzWlkjAkdAdGHICIjgenAR0fYd4uIlIpI6WAusD0qPZ4VrTnQ\nUA5tLYN2XWOM8YrnCUFEEoDngG+patPh+1X1EVUtUdWSjIyMQYurMCOBtb48503NhkG7rjHGeMXT\nhCAikTjJ4AlVfd7LWA43Kj2ejV1zGlmzkTEmBHg5ykiA3wPrVfUBr+I4msKMeHZqBh3hsVBtcxoZ\nY4KflzWE04Drgc+IyEr3cYGH8XxKZmI0sVGR7IkeZQnBGBMSIry6sKouAcSr6/dGRBiVHs/WtgLy\n9pR6HY4xxgw4zzuVA1lhRgJl7TmwvxZaqr0OxxhjBpQlhB4Upsfz0b5s5401GxljgpwlhB50TXIH\n2GI5xpigZwmhB6MzEqglmbboYTbJnTEm6FlC6MGo9HgAamJHWw3BGBP0LCH0ID46guykGLaFFTh3\nK/t9XodkjDEDxhJCLwoz4lnZUQAd+6Fuq9fhGGPMgLGE0IvCjHjeac5x3uwq8zYYY4wZQJYQelGY\nnkBZaxYaHg27LSEYY4KXJYReFGbE00kE+1LGWw3BGBPULCH0YnRGAgC748Y5CUHV44iMMWZgWELo\nRU5KLFERYWwOHw2tjVD/idchGWPMgLCE0IvwMGFUWjwfdox2Nuz4l0XdjDEmKFhC6IPCjHiWNGRA\ndDLs+MDrcIwxZkBYQuiDwox4yutb8efPhh0feh2OMcYMCEsIfVCYnkCnX6lPnwm1G2FfndchGWNM\nv7OE0AeFGc6cRttipzgbdlo/gjEm+FhC6INCd+hpmW8UhEdZP4IxJihZQuiD5NhI0hOi2Ly3E3Jm\nQPn7XodkjDH9zhJCHxWmJ7CttgUKz4Cq5bB/r9chGWNMv7KE0EeFGfFsq9kHY88D9cOWN70OyRhj\n+pWnCUFE/iAi1SKyxss4+qIwI566fe00phZBfAZses3rkIwxpl95XUP4E3C+xzH0SWG607G8tW4/\njDnHqSF0tnsclTHG9B9PE4KqLgaGRGP8waGnNftg8uehtcGajYwxQcXrGkKvROQWESkVkdKamhrP\n4sgfFkdEmLCtpgVGfwbi0mHVU57FY4wx/S3gE4KqPqKqJapakpGR4VkckeFhjEiLY3N1C4RHQtHl\nsPGfcKDes5iMMaY/HVNCEJFUEZkyUMEEuuLcZMp2NqCqMON68LXBsse8DssYY/pFrwlBRN4WkSQR\nGQYsBx4VkQcGPrTAM70glermNnY1tkJ2MYyaCx/9FnwdXodmjDEnrC81hGRVbQIuAx5X1ZOAef1x\ncRF5EvgAGC8iFSLypf4470CZXpACwIodDc6GU/4Nmqtg7d88jMoYY/pHXxJChIgMB64CXurPi6vq\n1ao6XFUjVTVPVX/fn+fvbxOyk4iOCGPFDrffYMw5kD4O3vsf8Pu9Dc4YY05QXxLCj4HXgC2qulRE\nCoHNAxtWYIqKCKM4N5nlXQkhLAzm3A57VsPa570NzhhjTlCvCUFVn1HVKar6dff9NlW9fOBDC0zT\nC1JYU9lEa4fP2VB8JWQVw1s/ho4D3gZnjDEnoC+dyve7ncqRIvKWiNSIyHWDEVwgmjliGO0+P2sq\nG50NYWFw3n3QUA7v3O9tcMYYcwL60mR0rtupfBGwHRgD3DGQQQWykpGpAHywtduqaYVnwNRr4P3/\nhd0BPy2TMcYcUZ86ld3nC4FnVLVxAOMJeOkJ0UzNT+HNDdWf3nHefRCTDC/+G/g6vQnOGGNOQF8S\nwksisgGYCbwlIhlA68CGFdjmTcikbGcD1U3d/gxxw+DCX0LVClgSkrdpGGOGuL50Kt8JnAqUqGoH\nsA+4ZKADC2TzJmUBsPDwWsLkS6HoCnjnZ05iMMaYIaQvncqRwHXAAhF5FvgSUNfzp4LbhOxEclNi\neXP9nn/decHPISEbnroOWqr/db8xxgSovjQZ/R9Oc9Fv3McMd1vIEhHmTcxkyZZaDrT7Pr0zbhh8\n4QnYXwcLroOOkG5dM8YMIX1JCLNU9QZVXeg+bgJmDXRgge7cydm0dvh5Z9MRagE50+Dzv4GdH8OC\na6GzbfADNMaYY9SXhOATkdFdb9w7lX09HB8STho1jLT4KP6+surIBxRdBhf/j7OIzoLrbXU1Y0zA\n60tCuANY5M56+g6wEPjuwIYV+CLCw7h8Zh6vr9tDZcNR7lCeeQNc+ABsfg2euQHa9w9ukMYYcwz6\nMsroLWAscBtwKzBeVRcNdGBDwQ2njgTgsfe3H/2gWV+CC34BG1+FP5wHDTsHJTZjjDlWR00IInJZ\n1wPnprQx7uNCd1vIy02J5bNF2Tz58Q5a2nq4GW32l+GaBVC/HR45E7a8NVghGmNMn/VUQ7i4h8dF\nAx/a0PCl00fR3NrJM6W9/PIfdx7c/BbEp8NfLoNX77QRSMaYgBJxtB3uaCLTi+kFqcwoSOGP723n\ni6eMJDxMjn5wxji45W1442746P9g29tw+e8gu2iQojXGmKM7pjWVzZHdPKeQHXv388a6I9yodrjI\nWLjgfrj2OTiwFx49C959wOY/MsZ4zhJCPzh3UhYj0uJ44I2NdPj6uHLa2Hnwtfdh3Pnw1o/g9+dA\n9fqBDdQYY3pgCaEfRISHcdcFE9m0p4W/fFje9w/Gp8NVj8MVf3TWU/jtXFj8C6stGGM8cdQ+hC5H\nGVHUCKxWVZusx3XOpCzmjE3ngTc2cfHUHNITovv2QRHnJraRc+CV22HhT2Dt3+C8e6HwzIEM2Rhj\nPqUvNYQvAb8DrnUfjwLfA94TkesHMLYhRUS4++JJHGj38V+vbjj2EyRkwFWPwZWPQWsDPH4J/Pky\n2L26/4M1xpgj6OsCORNV9XJ3LeVJgAIn4SSG4yYi54vIRhHZIiJ3nsi5AsGYzES+PLeQZ5dV8P6W\n2uM7yeTPw7+Vwrn3QuUyeHgOvPBVu6HNGDPg+pIQ8lW1+/CZanfbXqDjeC8sIuHAQ8BncZLM1SIy\n6XjPFyhu+8xYCtPjue2pFexqPMqUFr2JjIFTb4VvrnSe1zwPv5oJr/8AGiv6N2BjjHH1JSG8LSIv\nicgNInID8KK7LR5oOIFrzwa2qOo2VW0HniIIFt6JjQrnt9fP5EC7j6/+ZTltnScwD2BsKpz7E7h1\nmdPP8P6v4b+L4eXbobGy/4I2xhj6lhC+AfwRmOY+HgO+oar7VPWsE7h2LtC9HaTC3fYpInKLiJSK\nSGlNTc0JXG7wjM1K5JdXTaVsZwPff241fr+e2AlT8uHSh50aw8yboPQP8D9T4W9fh5pN/RO0MSbk\n9WVyOwWW4Mxy+haw2N02KFT1EVUtUdWSjIyMwbrsCTu/aDjfOWccz6+o5O4X19Ivf7LUkXDRA3Db\nCii5yWlK+s1JbmLYeOLnN8aEtL4soXkV8DFwBXAV8JGIXNEP164E8ru9z3O3BY1bPzOGr8wt5M8f\nlvMfL6yhs683rfUmdYSzVOe318BJX3MSw0Oz4a/zofx9GLx8bYwJItLbL1cRKQPO6brnQEQygDdV\ndeoJXVgkAtgEnI2TCJYC16jq2qN9pqSkREtLS0/ksoNOVbn/tY3839tbmTcxk/+9ejpxUb3e/nFs\n9tXB0kfh40ecpTtzS2D6tTDxc87Nb8aYkCYiy1S1pLfj+tKHEHbYDWh1ffxcj1S1E/g34DVgPfB0\nT8lgqBIRvnf+BH5yyWQWbqjm6kc/oq6ln5fUjE+DM++Eb61x1l44UA8vfRseLHJmVbUpMYwxfdCX\nGsLPgSnAk+6m+cAqVT2hexCOx1CsIXT3+trd3PrkCoYnx/Cnm2YzMj1+YC6kCnvWwoe/gbKnQH2Q\nPt65x2HyZZA5YWCua4wJSH2tIfSaENyTXQ6c5r59V1VfOMH4jstQTwgAy8rrufmxpYgID10zg1NG\npw3sBZt3w/p/ONNhlL8HKORMhxk3wORLITZlYK9vjPFcvyaEQBEMCQHgk9p93PzYUrbX7ec/LpjI\n/zttJCI9rKPQX5r3wNrnYfnjUL0OwqOc2VanzIex50BEH+dfMsYMKSecEESkGWeKin/ZhTMaNenE\nQjx2wZIQAJpbO/ju02W8vm4PFxRn89PLppAcGzk4F1eFquWw6hlY8yzsq4GYFKdJadz5MPozlhyM\nCSJWQxgC/H7lt4u38YvXNzI8OYZfXzODafmD3ITj63RWblv9NKx/CTr2QXQy5EyDvBLIPxnyZ1vT\nkjFDmCWEIWRZeT23PbmCPU2t/L/TR3H7ueOJivBgqYrONtj2Dmx8GapWOB3T/k5AIHMSFJwEBadA\n/kmQUuBM3W2MCXiWEIaYxv0d/PTV9Ty1dCdT81P43y9MY0TaAI1C6qv2fc6Mqzs+dB47P4b2Zmdf\nXDoMnwIjTnMeaaMhPsOShDEByBLCEPXq6l1877lV+PzK3Z+bzJUz8wanw7kv/D6nM3rHh1C10qlF\nVHe7dSQyzkkM6eMhYzykj3MeaaOtT8IYD1lCGMKqGg7wnadX8uG2vXy2KJv/79JiUuOjvA7ryFpq\nnA7q+nKo/wRqNzkT7jXuOHSMhDvzMGWMh/Sxziyu8RkwbDQkZEJcGsQkW+3CmAFiCWGI8/mVR9/d\nxi9f30hybBT3XVrEeZOzvQ6r79r3Qe1mN0FshNqNTqLYu9XtlziMhEPcMCc5xA479Prgc9d2d1t0\nEkQnOmtHGGN6ZAkhSKyrauL2Z8pYt6uJz0/L4Z7PTSYlLkBrC33h9zmd1y27oW6bM+T1wF5nDqb9\ndbB/r/uoO7T9SAmkS3jUoeQQk+S+7va+a39ChlMriYxzmq/i0p15nmJTISx88MpvjAcsIQSRDp+f\nhxZt4dcLt5AaH8VPLy1m3qQsr8MaHKrQ1nwoWRxwE0ZbE7Q2Ovvampzn1qZD71ubnGdfO3Ts7/ka\nkXEQFe8kibhh7vs4N5FkQnym+5zhJJDYFOc5Ms6aucyQYAkhCK2pbOT2Z8rYsLuZy2fk8Z8XTxq8\nm9mGss522F8LLdVOcvB1ODWTriTTse/TSadjP7TvdxLOvhpnLqgjCY9yEkNMilPbSMh0mrWiE5xk\nEZ3oJJPuSSQhy9lvzCCyhBCk2jv9/GrhZn7z9lbSE6L4r8umcNaETK/DCl5+v1Mraal2kkNrgzOb\n7AH3uev9vlpn3qiuWouvhxltY5KdvpDoJKdZKy7NrZ2kHWrG6kogscOc5+gkCPPg3hQTFCwhBLlV\nFQ3c/kwZm/a0cFVJHj+4aBJJMVZbCBh+n9Nk1VJ9KHkcqHf6TpqqnNpIV7PW/jonobT2sES5hEFk\nvFPriE1xkkpMipNQImKcR9wwSMyGhGy35pLkHpcMEbGWUEKYJYQQ0Nbp43/e3MzD72wlKymGn10+\nhbnjhs4yo+Ywvk6nNtI9gRx87HVGbrU1OftbG53ntkank76z1dnWEwl3EkrcsEMjuWJSIDwSohKc\nBBKV4DRpJea4TWCpkJRro7mGOEsIIWTlzga++/RKttbs4+rZ+fzHBRNJtNpC6Olsh33Vzqy2rfVO\ngmh1O9+7kkZb86GO+QN7nX2+TucO9NYmjjyfJRAWCZGx3YYBu01c0YlOAomKh6iu125SOTjiKwHC\no52aSng/rxZo+sQSQohp7fDx4BubePTdbQxPjuVnl0/h9LG2fKY5BqrQccBJGk0VztKs+2uhsdLp\neO84cKh5q6sDvr0Z2lqO3vHenYQ5nerh7rDpiJhuw4UTQf1OjSQ82jm26xEW7gwVjoh2PhMeBWER\nTs1Gwp2mNn8nDCt0ElXnASfJhUcC6pw3yq0ZhYU7yS0s4tD5I2OdQQBB3KRmCSFELSuv545nythW\nu4/rTi7g+5+dSHy0/SozA0jVqX20u6O12lucJNF9SLCv3e14r3JGecGhGkvXkGFwvtx9Hc6XuLpf\n5v4Op4ZztNpLf4mI5WACCY9ym9ISneewCPcR7jzU7ySWqHinRhQV9+kk0/UZcY/vSmxd2yJjnNpT\nV1I7uD915GhxAAATfklEQVTcKXfzbiem7tccey4k5x1X0fqaEOybIsjMHJHKK9+cwy9e28jv3/uE\ntzfWcP8VUzh1tNUWzAARcX9lxzqjpAaCqpMofG1OcvB3Ou/9nU4nu4RB3RanCSwi1vmi7Ww79AXd\n3uzUaPyd7sPnJh2/U/Np3+fUgiQMEOeYriTnaz/0Gb/PrQ2Jk6ja9zm1pfZ9zna/333u/NfP+N3t\nx5vYrnvuuBNCX1kNIYgt3b6XO54pY3vdfq4qyeP2c8eTmWSdg8Z4StW516WtuVttyN8tYeCMFhM5\nlFD8ne7NkMf3/2tNRgaAA+0+HnxzE3987xNiIsL5989O4JrZBYSH2R22xoSKviYET3pRRORKEVkr\nIn4R6TVIc/xio8L5jwsm8vq3z6A4L5kf/m0NF/1qCR9srfM6NGNMgPGqW30NcBmw2KPrh5xR6fE8\ncfNJPHTNDJoOdHD1ox/y9SeWsXNvL/P8GGNChiedyqq6HgichV9ChIhw4ZThnD0xk0cWb+P/3t7K\nm+uruWVOIV87c7SNRjImxAX8wFsRuUVESkWktKamxutwgkJMZDi3nT2WhbefwQVF2fx60RY+88u3\neWFFBX7/0OlTMsb0rwHrVBaRN4Ejrehyl6r+3T3mbeB2Ve1TT7F1Kg+MZeV7+dE/1rGqopHpBSnc\nffFkpuWneB2WMaafeH4fgqrOG6hzm/41c8Qw/vb103hueQU/++dGPv/Qe1xYPJzvnDuO0Rk2VbMx\nocIajQ0AYWHClSX5fLZ4OI8s3sbv3t3GP9fu5vIZuXzljNGWGIwJAZ7chyAilwK/AjKABmClqp7X\n2+esyWjw1La08euFW3jy4x20+/zMm5jFV+YWUjJymNehGWOOkd2YZvpFbUsbj7+/ncc/LKdhfwcz\nR6Ryy9xCzpmYRZjd3GbMkGAJwfSr/e2dPL10J79b8gkV9QcozIjny3MKuXR6LjGRtki9MYHMEoIZ\nEJ0+P6+u2c1vF29lTWUTw+KjuGx6LvNn5TM2K9Hr8IwxR2AJwQwoVeWDrXX8+cNy3ly/hw6fMqMg\nhfmz8rloSo7d5GZMALGEYAZNbUsbLyyvZEHpTrZUtxAXFc7FU3K4alY+MwpS7I50YzxmCcEMOlVl\n+Y56FizdyUurdrG/3cfYzATmz8rn0um5pCVEex2iMSHJEoLxVEtbJy+VVbGgdCcrdjQQESbMGZvO\nRVNyOGdyFkm25rMxg8YSggkYm/Y08+yyCl5etYvKhgNERYRx5rgMLpqaw7yJmcRFWX+DMQPJEoIJ\nOH6/smJnAy+tquLlVbuobm4jJjKMs8Znct7kbM6akElyrNUcjOlvlhBMQPP5laXb9/LSqipeX7uH\n6uY2IsOFU0enc35RNmdPzCQz0Zb7NKY/WEIwQ0ZXzeH1tbt5dc1udriL9hTlJnHW+EzmTcxiSl6y\njVYy5jhZQjBDkqqyflczizZW8/bGapbvaMDnV4Ynx3Dm+EzOHJ/B6WPS7T4HY46BJQQTFBr2t/PW\n+mpeX7eb97bU0dLWSVR4GKeOSWPexCzmTcwiO9malozpiSUEE3TaO/0sK6/nrfV7eGP9HsrrnKal\ncVkJzBmbwelj0zlp1DAbtWTMYSwhmKCmqmyubmHhhmqWbK7l4+17ae/0ExUexswRqcwZl87csRlM\nGp5ks7KakGcJwYSU1g4fH3+yl3c31/Du5lo27G4GYFh8FKeNSWfOmHROH5tOTkqsx5EaM/g8X0LT\nmMEUExnO3HEZzB2XAUB1cyvvbanl3U21vLulln+UVQEwOiOe08akc+roNE4alUZqfJSXYRsTUKyG\nYIKeqrJxT/PB5LD0k70c6PAhAhOzk5g5IpXi3GROHZNGXmqc1+Ea0++syciYo2jv9LOqooEPttbx\n/tY6Vlc20tLWCUBhRjwzClKZmpfM9IJU64MwQcESgjF95PcrW2taeGdTDe9vraNsZwN1+9oBSImL\nZNbIYZSMSHVqEnnJREfYCnFmaLE+BGP6KCxMGJuVyNisRG6eU4iqUtlwgKXb9/LeljpKt+/ljXV7\nAIiKCGNKbjIlI4cxa2QqMwpSrR/CBA2rIRjTB7UtbSwrr2dZeT1Lt+9lTWUjHT7nf2dEWhzFucnM\nKEilZGQqE4cnERke5nHExhwS0DUEEfk5cDHQDmwFblLVBi9iMaYv0hOiOW9yNudNzgacYa5lOxtY\ntqOe1RWNrNjRwEurdgEQGxnOtPwUpuanMCUvmekFKQxPtuGuJvB5UkMQkXOBharaKSI/A1DV7/X2\nOashmEC2q/EApdvrD9YkNuxuOliLyEmOYVJOMsW5yUwrSGFaXgrJcTbVtxkcAV1DUNXXu739ELjC\niziM6U/Dk2O5eGosF0/NAaCt08eGXc2UltdTtrOBtVWNvLVhD12/wQrT45mWn+IkiPwUJmQnERVh\nTU3GO573IYjIP4AFqvqXo+y/BbgFoKCgYGZ5eflghmdMv2pq7WB1RSMrdzYcfNQ0twFOh3VRThJT\n850EMT47kfzUOJvZ1Zwwz4edisibQPYRdt2lqn93j7kLKAEu0z4EYk1GJtioKlWNrazc0cDKnfWs\n3NnA6spGWjv8AIQJjM9OYnpBCtPzU5gxIpVRafF2b4Q5Jp4nhF4vLHIj8BXgbFXd35fPWEIwoaDT\n52fjnma21uxja3ULy3c4iaK51bl5Ljk2kmluh/Wk4UlMzkkmf1isLSBkjiqg+xBE5Hzg34Ez+poM\njAkVEeFhTM5JZnJO8sFtXTfPrdjRwIqd9Swvb2DJllp8fucHXWJMBKeNTqfYTRJT8pJJS4j2qghm\niPJqlNEWIBqoczd9qKpf7e1zVkMw5pDWDh8bdzeztqqJsp0NvLe1lor6Awf356bEMjU/mSl5KQdr\nFLZWRGgK+Caj42EJwZieNbd2sLaqiVUVDZRVNLKqooGde50kESYwLiuRCdmJTByeRMnIVIpybSqO\nUBDQTUbGmIGRGBPJyYVpnFyYdnBbXUsbZRUNTsd1RSMffbKXv610pgMPDxNyUmIoGTGMqXnJTMlP\nYdLwJGIiLUmEIqshGBOCapqdqTjWVDaypbqF0vJ6aluc4a8RYcL47ESm5qc4SSIvhbGZCUTYdBxD\nljUZGWP6TFXZ3dRK2U6nmWmV29zU5I5sio0MZ3JOEsV57t3W+SmMtOGvQ4YlBGPMCfH7lfK9+ynb\n2UCZmyTWVTVxoMMHQGJ0BBOGJzIhO4mTC9OYlJPEyLQ4G/4agCwhGGP6nc+vbKluYeXOetZUNrF+\nVxMbdjcfXGAoITqCKXnJzB41jGn5KUzOSSYj0Ya/es0SgjFmUHT4/KyramJNVSMbdjWzrLye9bub\nDs7ZlJEYzeScpIM30U3KSWLEsDhrbhpENsrIGDMoIsPDnA7o/JSD2xoPdLCuqol1u5pYW+U0NS3Z\nXEuneyNdfFQ447ITGZ/lDIHtuqHORjd5y2oIxphB0dbpY/OeloOJYsPuJjbubqZ+fwfgjG6aODyJ\nolynNjFxeBLjsxNJjLFpwk+U1RCMMQElOiKcotxkinIPTcmhquxqbD04qmnlzgZeXbObJz/eefAY\nZy2JJE4dnc747EQm5ySREmfLlg4ESwjGGM+ICDkpseSkxHJ+kTM5ctcQ2LWVTWyqbmbznhY+/mQv\nb66vPvi53JRYxmUlMDkn2U0ySeSm2AR/J8oSgjEmoIgIw5NjGZ4cy7xJWYCTJGpb2tmwu4k1lU6T\n0+Y9zSzefGiCv2HxUYzJTKAoJ5nc1FhyU2I4Y1wmsVHWL9FXlhCMMQFPRMhIjCYjMYM5YzMObm/t\n8LF+VxNrKhtZXdnI1pp9PPFROW2dh9aTyB8Wx6ThScwoSGV6QQpFucnWeX0UlhCMMUNWTGQ40wtS\nmV6QenBbh89P04EO1u9qZun2vWypbmFVpdM3AU7n9ZjMBCblJDEhO5Hx2c5zZmJ0yDc5WUIwxgSV\nyPAw0hKiOX1sNKePTT+4vbrZWZmurKKBdVVNvLellueXVx7cPyLNqUlML0hhdEYCp4xOC7npwkOr\ntMaYkJWZGMO5k7M5d/KhlX3r97WzcU8z63c1sXhTDWurmg7WJKIiwhidkUBhRjzFucnMHJHK5Jyk\noE4Sdh+CMcZ0U9fSxsY9zSzaUM3m6ha21rR8ak2JsZmJFOclM8Wd6G/iELihzu5DMMaY45CWEM2p\nCdGcOvpQc1NNcxtlOxtY7XZev72xmmeXVQBOn8TYrEQK0+M5qXAYk3OSGJOZSEJ0BOFDbHoOqyEY\nY8wx6rpXYlVFI6srGllV2ciWPc1UNbYePCYzMZqzJ2YdrEmMy0okKsKbNSVscjtjjBlEqkpF/QHW\n73Km5Cgtr2f5jnqa3TUloiLCmJjtNjflOsNfx2YlEDkICw9ZQjDGGI+pKuV1+1lV2ciaSmd6jjWV\nTQenC4+OCGNSThJT3Ck9puSlMDojvt9Xp7OEYIwxAcjvV7bX7WN1ZaPT5FTZyNrKRva1OwsPdV+d\nbnpBKpOGJzI6I+GE7pEI6IQgIj8BLgH8QDVwo6pW9fY5SwjGmGDk8yuf1LYcTBCrKxpZU9VIa4dz\nx3VuSiw/v2IKp45J7+VMRxboo4x+rqo/BBCR24D/BL7qUSzGGOOp8DBhTGYiYzITuWxGHuBMF761\neh+rKhp4c301ealxAx6HJwlBVZu6vY0Hhk67lTHGDILoiHAm5SQxKSeJL8wuGJRrenYfgojcB3wR\naATO6uG4W4BbAAoKBuePYowxoWjA+hBE5E0g+wi77lLVv3c77vtAjKre3ds5rQ/BGGOOned9CKo6\nr4+HPgG8AvSaEIwxxgwcT26bE5Gx3d5eAmzwIg5jjDGHeNWH8F8iMh5n2Gk5NsLIGGM859Uoo8u9\nuK4xxpij82amJWOMMQHHEoIxxhhgiM1lJCI1OH0OxyMdqO3HcAJRKJQRQqOcVsbgEQjlHKGqGb0d\nNKQSwokQkdK+jMMdykKhjBAa5bQyBo+hVE5rMjLGGANYQjDGGOMKpYTwiNcBDIJQKCOERjmtjMFj\nyJQzZPoQjDHG9CyUagjGGGN6YAnBGGMMECIJQUTOF5GNIrJFRO70Op7jJSJ/EJFqEVnTbdswEXlD\nRDa7z6nd9n3fLfNGETnPm6iPjYjki8giEVknImtF5Jvu9qApp4jEiMjHIlLmlvFH7vagKWMXEQkX\nkRUi8pL7PhjLuF1EVovIShEpdbcNzXKqalA/gHBgK1AIRAFlwCSv4zrOsswFZgBrum27H7jTfX0n\n8DP39SS3rNHAKPdvEO51GfpQxuHADPd1IrDJLUvQlBMQIMF9HQl8BJwcTGXsVtbvAH8FXnLfB2MZ\ntwPph20bkuUMhRrCbGCLqm5T1XbgKZwpt4ccVV0M7D1s8yXAY+7rx4DPd9v+lKq2qeonwBacv0VA\nU9Vdqrrcfd0MrAdyCaJyqqPFfRvpPpQgKiOAiOQBFwK/67Y5qMrYgyFZzlBICLnAzm7vK9xtwSJL\nVXe5r3cDWe7rIV9uERkJTMf5BR1U5XSbUlYC1cAbqhp0ZQT+G/h3nGnuuwRbGcFJ5m+KyDJ3yV8Y\nouX0bE1l0/9UVUUkKMYRi0gC8BzwLVVtEpGD+4KhnKrqA6aJSArwgogUHbZ/SJdRRC4CqlV1mYic\neaRjhnoZuzldVStFJBN4Q0Q+teDXUCpnKNQQKoH8bu/z3G3BYo+IDAdwn6vd7UO23CISiZMMnlDV\n593NQVdOAFVtABYB5xNcZTwN+JyIbMdppv2MiPyF4CojAKpa6T5XAy/gNAENyXKGQkJYCowVkVEi\nEgV8AXjR45j604vADe7rG4C/d9v+BRGJFpFRwFjgYw/iOybiVAV+D6xX1Qe67QqacopIhlszQERi\ngXNwlpENmjKq6vdVNU9VR+L8zy1U1esIojICiEi8iCR2vQbOBdYwVMvpda/2YDyAC3BGq2wF7vI6\nnhMox5PALqADp+3xS0Aa8BawGXgTGNbt+LvcMm8EPut1/H0s4+k4bbKrgJXu44JgKicwBVjhlnEN\n8J/u9qAp42HlPZNDo4yCqow4oxfL3Mfaru+XoVpOm7rCGGMMEBpNRsYYY/rAEoIxxhjAEoIxxhiX\nJQRjjDGAJQRjjDEuSwjGDBIRObNr1k9jApElBGOMMYAlBGP+hYhc565XsFJEfutORNciIg+66xe8\nJSIZ7rHTRORDEVklIi90zXsvImNE5E13zYPlIjLaPX2CiDwrIhtE5AnpPkmTMR6zhGBMNyIyEZgP\nnKaq0wAfcC0QD5Sq6mTgHeBu9yOPA99T1SnA6m7bnwAeUtWpwKk4d5iDM3vrt3DmxS/EmfPHmIBg\ns50a82lnAzOBpe6P91icicn8wAL3mL8Az4tIMpCiqu+42x8DnnHntslV1RcAVLUVwD3fx6pa4b5f\nCYwElgx8sYzpnSUEYz5NgMdU9fuf2ijyw8OOO945X9q6vfZh/4MmgFiTkTGf9hZwhTu3fdfauCNw\n/leucI+5Bliiqo1AvYjMcbdfD7yjzkpvFSLyefcc0SISN6ilMOY42K8TY7pR1XUi8gPgdREJw5lZ\n9hvAPmC2u68ap58BnKmNH3a/8LcBN7nbrwd+KyI/ds9x5SAWw5jjYrOdGtMHItKiqglex2HMQLIm\nI2OMMYDVEIwxxrishmCMMQawhGCMMcZlCcEYYwxgCcEYY4zLEoIxxhgA/n/GJd543CeD+QAAAABJ\nRU5ErkJggg==\n",
            "text/plain": "<matplotlib.figure.Figure at 0x7f01f3362748>"
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "That's quite a drop in performance! And it's unstable, if you run the above cell repeatedly. But what do you expect from 150 training examples...\n\nLet's see how MTL performs."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## MTL model that deals with missings values\n\nWe will now create a model that can be fit on targets with missings values.\n\nWe need to implement our own loss function that computes the mean squared error, but ignores missing values. A loss function in Keras gets two tensors `y_true` and `y_pred` as input. Both have shape `(n_examples, n_outputs)`.\n\nOur function contains 3 steps. \n2. replace missing values in `y_true` with the *predicted* value for that example for that target. Let's call the resulting tensor `y_helper`\n3. Compute the sum of squared errors, by computing `y_helper - y_pred` (and square and sum it). Due to our missing-value replacement rule, missing values contribute zero loss.\n4. divide by number of *non-missing* values to get mean squared error"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "def mean_squared_error_with_nan(y_true, y_pred):\n    notnan_mask = tf.is_finite(y_true)\n    \n    # replace missings in y_true[i,j] with y_pred[i,j]\n    y_helper = tf.where(notnan_mask, y_true, y_pred)  \n    \n    # sum of squared errors per output:\n    squared_error = tf.reduce_sum(tf.square(y_helper - y_pred), axis=0, keep_dims=True)\n    \n    # number of non-missing values per output:\n    n_notnan = tf.reduce_sum(tf.cast(notnan_mask, K.floatx()), axis=0, keep_dims=True)\n    \n    # MSE per output:\n    mse_per_output = squared_error / (n_notnan + K.epsilon())\n    return tf.reduce_sum(mse_per_output)",
      "execution_count": 84,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now we can finally fit a model to *all* data, including missing targets!"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "%%time\nmodel = build_model3(loss=mean_squared_error_with_nan)\nmodel.fit(X_std, y_wm, epochs=1000, validation_split=.2, verbose=0, \n    callbacks=[EarlyStopping(monitor='val_loss', min_delta=0, patience=10, verbose=0, mode='auto')])\nplot_lc(model)\n\npreds_test = model.predict(X_test_std)\npreds_test.shape\nprint(r2_score(y_test, preds_test, multioutput='raw_values'))",
      "execution_count": 85,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "[ 0.80896104  0.84156803  0.85085561]\nCPU times: user 9.31 s, sys: 943 ms, total: 10.3 s\nWall time: 5.46 s\n"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8leX9//HXJ8nJ3nsBCXsFAQHBgXujOFHrrtYOq7ZW\nW1tra/3WDttqa2vdqxX36M+BC2WIWmQl7BlWBhmQvXPO9fvjOmCkhARIcp+T83k+Hnlwxn3u+3Ny\nSN65ruu+rluMMSillFJBTheglFLKN2ggKKWUAjQQlFJKeWkgKKWUAjQQlFJKeWkgKKWUAjQQlDoo\nEXlfRK51ug6l+oLoPATli0RkG3CjMWau07UoFSi0haACloiEOF3DkeoP70H5Dg0E5XdEZIaI5ItI\ntYh8ISLjOjx3l4hsEZE6EVkrIhd2eO46EflcRB4Skd3Avd7HFonIn0WkSkS2isjZHV4zX0Ru7PD6\ng22bKyILvceeKyKPiMgLB3kfM73vo9Zb81nex7eJyGkdtrt3735EJEdEjIjcICI7gE+93Vo/3G/f\nBSJykff2SBH5WET2iMgGEZl1+N991Z9pICi/IiITgGeA7wJJwOPA2yIS5t1kC3ACEAf8BnhBRDI6\n7OIYoBBIA+7v8NgGIBl4AHhaRKSTEg627YvAV9667gWuPsj7mAL8C7gTiAemA9u6ev8dnAiMAs4E\nXgKu6LDv0cAg4D0RiQI+9taWClwO/NO7jVLfoIGg/M1NwOPGmMXGGLcx5nmgBZgKYIx5zRhTYozx\nGGNeATYBUzq8vsQY83djTLsxpsn72HZjzJPGGDfwPJCBDYwDOeC2IjIQmAz8yhjTaoxZBLx9kPdx\nA/CMMeZjb63Fxpj1h/B9uNcY0+B9D28B40VkkPe5K4E3jTEtwAxgmzHmWe97XgG8AVx6CMdSAUID\nQfmbQcBPvN1F1SJSDQwAMgFE5JoO3UnVwFjsX/N77TzAPnftvWGMafTejO7k+J1tmwns6fBYZ8fa\nawC2NXO49u3bGFMHvIf96x9sa2G29/Yg4Jj9vl9XAulHcGzVT+mAlPI3O4H7jTH37/+E9y/kJ4FT\ngS+NMW4RyQc6dv/01ml1pUCiiER2CIUBB9l+JzCkk+cagMgO9w/0y3v/9/ES8GsRWQiEA/M6HGeB\nMeb0gxWvFGgLQfk2l4iEd/gKwf7C/56IHCNWlIicKyIxQBT2F2UFgIhcj20h9DpjzHZgKXagOlRE\npgHnHeQlTwPXi8ipIhIkIlkiMtL7XD5wuYi4RGQScEk3SpiDbQ3cB7xijPF4H38XGC4iV3v35xKR\nySIy6nDep+rfNBCUL5sDNHX4utcYsxT4DvAPoArYDFwHYIxZC/wF+BIoA/KAz/uw3iuBacBu4LfA\nK9jxjf9hjPkKuB54CKgBFmB/oQPcg209VGEHxl/s6sDe8YI3gdM6bu/tTjoD251Ugu3y+iMQdoDd\nqACnE9OU6iUi8gqw3hjza6drUao7tIWgVA/xdsUM8XYBnQXMBP7jdF1KdZcOKivVc9Kx3TZJQBHw\nfe9pnkr5Be0yUkopBWiXkVJKKS+/6jJKTk42OTk5TpehlFJ+ZdmyZZXGmJSutvOrQMjJyWHp0qVO\nl6GUUn5FRLZ3ZzvtMlJKKQVoICillPLSQFBKKQU4PIYgIvHAU9j1ZgzwbWPMl07WpJTyPW1tbRQV\nFdHc3Ox0KT4tPDyc7OxsXC7XYb3e6UHlvwEfGGMuEZFQvrnCo1JKAVBUVERMTAw5OTl0fu2iwGaM\nYffu3RQVFZGbm3tY+3Csy0hE4rBXiXoawHtRkWqn6lFK+a7m5maSkpI0DA5CREhKSjqiVpSTYwi5\n2GWKnxWRFSLylPdyf98gIjeJyFIRWVpRUdH3VSqlfIKGQdeO9HvkZCCEABOBR40xE7AXBblr/42M\nMU8YYyYZYyalpHQ5r+KAzM6vaJr3IOgyHUop1SknA6EIKDLGLPbefx0bED1u6duPEbHgNzR/8CsN\nBaXUIYuO7uyKqv2LY4FgjNkF7BSREd6HTgXW9sax2s/8I7PdpxG++GHcX/6zNw6hlFJ+z+l5CLcA\ns0VkJTAe+F1vHGTa0BTCZj7EB+7J8NE9mJ1LeuMwSql+zhjDnXfeydixY8nLy+OVV14BoLS0lOnT\npzN+/HjGjh3LZ599htvt5rrrrtu37UMPPeRw9V1z9LRTY0w+MKkvjnXJpIE8XPZ7xn51OTEvf4e4\n25dCsNNn3SqlDtVv3lnD2pLaHt3n6MxYfn3emC63e/PNN8nPz6egoIDKykomT57M9OnTefHFFznz\nzDO5++67cbvdNDY2kp+fT3FxMatXrwagutr3T6J0uoXQp24+exKvJt9MXMNWGr563ulylFJ+ZtGi\nRVxxxRUEBweTlpbGiSeeyJIlS5g8eTLPPvss9957L6tWrSImJobBgwdTWFjILbfcwgcffEBsbKzT\n5XcpoP5EDg4Szp91I8sfmU3uvD8Tdcz1EBRQmaiU3+vOX/J9bfr06SxcuJD33nuP6667jttvv51r\nrrmGgoICPvzwQx577DFeffVVnnnmGadLPaiA+204NC2W1QOuJKG1hIa1HzpdjlLKj5xwwgm88sor\nuN1uKioqWLhwIVOmTGH79u2kpaXxne98hxtvvJHly5dTWVmJx+Ph4osv5re//S3Lly93uvwuBVQL\nYa9JZ11DxVN/pX7+Y+SOPdvpcpRSfuLCCy/kyy+/5KijjkJEeOCBB0hPT+f555/nT3/6Ey6Xi+jo\naP71r39RXFzM9ddfj8fjAeD3v/+9w9V3za+uqTxp0iTTUxfIefeB6zij8R1C7yqE8Lge2adSqnes\nW7eOUaNGOV2GXzjQ90pElhljujyBJ+C6jPZyjzqfUNqpyn/H6VKUUsonBGwgjJlyKqUmkfoVbzpd\nilJK+YSADYQhqbEsCZlIUsV/wd3udDlKKeW4gA0EEaE24zgiPQ14SvKdLkcppRwXsIEAEDvqFAB2\nr/rI4UqUUsp5AR0I40cOZ51nAK1bPnO6FKWUclxAB8KAxAg2hQwnvmq1LoutlAp4AR0IIkJtYh5R\nnlqo2uZ0OUqpfuJg10/Ytm0bY8eO7cNqui+gAwEgeICdq9G0XZfEVkoFtoBcuqKj9GETaFnuonrT\nYiImzHK6HKVUV96/C3at6tl9pufB2X/o9Om77rqLAQMGcPPNNwNw7733EhISwrx586iqqqKtrY3f\n/va3zJw585AO29zczPe//32WLl1KSEgIDz74ICeffDJr1qzh+uuvp7W1FY/HwxtvvEFmZiazZs2i\nqKgIt9vNPffcw2WXXXZEb3t/AR8IY7KT2WCySdq12ulSlFI+6rLLLuNHP/rRvkB49dVX+fDDD7n1\n1luJjY2lsrKSqVOncv755x/She4feeQRRIRVq1axfv16zjjjDDZu3Mhjjz3GbbfdxpVXXklraytu\nt5s5c+aQmZnJe++9B0BNTU2Pv8+AD4TU2HAWBw9iUK0GglJ+4SB/yfeWCRMmUF5eTklJCRUVFSQk\nJJCens6Pf/xjFi5cSFBQEMXFxZSVlZGent7t/S5atIhbbrkFgJEjRzJo0CA2btzItGnTuP/++ykq\nKuKiiy5i2LBh5OXl8ZOf/ISf/exnzJgxgxNOOKHH32fAjyEA1MUMJa69Ehr3OF2KUspHXXrppbz+\n+uu88sorXHbZZcyePZuKigqWLVtGfn4+aWlpNDc398ixvvWtb/H2228TERHBOeecw6effsrw4cNZ\nvnw5eXl5/PKXv+S+++7rkWN1pIEAuJNHAmDK1zlciVLKV1122WW8/PLLvP7661x66aXU1NSQmpqK\ny+Vi3rx5bN++/ZD3ecIJJzB79mwANm7cyI4dOxgxYgSFhYUMHjyYW2+9lZkzZ7Jy5UpKSkqIjIzk\nqquu4s477+yV6ys42mUkItuAOsANtHdnedbeEJ6dB4VQt3M1sTnHOVGCUsrHjRkzhrq6OrKyssjI\nyODKK6/kvPPOIy8vj0mTJjFy5MhD3ucPfvADvv/975OXl0dISAjPPfccYWFhvPrqq/z73//G5XKR\nnp7OL37xC5YsWcKdd95JUFAQLpeLRx99tMffo6PXQ/AGwiRjTGV3tu/J6yF09PmmCvJeyKN+5CVk\nXvGPHt+/UurI6PUQuk+vh3CEhqbFsNVkQOVmp0tRSinHOH2WkQHmiogbeNwY88T+G4jITcBNAAMH\nDuyVIlJjwlgmGQyo29or+1dKBZ5Vq1Zx9dVXf+OxsLAwFi9e7FBFXXM6EI43xhSLSCrwsYisN8Ys\n7LiBNySeANtl1BtFiAg1kQOJb/oS2lsgJKw3DqOUOgLGmEM6x99peXl55Of37dL6RzoE4GiXkTGm\n2PtvOfAWMMWpWlrjcgnCo2saKeWDwsPD2b179xH/wuvPjDHs3r2b8PDww96HYy0EEYkCgowxdd7b\nZwA9f2JtN4WkDIVd0FaxCVfKCKfKUEodQHZ2NkVFRVRUVDhdik8LDw8nOzv7sF/vZJdRGvCWtwkY\nArxojPnAqWKiM0fAKqgtWk/S6BlOlaGUOgCXy0Vubq7TZfR7jgWCMaYQOMqp4+8vOzOTKhNNc7me\naaSUCkx62qnXoKQoikwypmqH06UopZQjNBC8kqJC2SWphDUUO12KUko5QgPBS0SoCcsgrqVUL6ep\nlApIGggdNEdlE2paoHG306UopVSf00DowBM3wN6oPvRVC5VSyt9pIHTgSsoBoKVym6N1KKWUEzQQ\nOohKtec51+8qdLgSpZTqexoIHaSmpFBtorSFoJQKSBoIHWTFR1BkUnQMQSkVkDQQOkiLDafIpBBW\nX+R0KUop1ec0EDoIDQlijyuNmGadi6CUCjwaCPtpiMgi1DTrXASlVMDRQNiPO1bnIiilApMGwn6C\nEwcB6CJ3SqmAo4Gwn4iUHAAay3UuglIqsGgg7Cc5OZVqE0VzxTanS1FKqT6lgbCfvXMR3FU6hqCU\nCiwaCPvJjLdzEVx1OhdBKRVYNBD2kxgVyi5JJqqpROciKKUCigbCfkTEzkXwNEHjHqfLUUqpPuN4\nIIhIsIisEJF3na5lr7YYnYuglAo8jgcCcBuwzukiOgpOHGhvVOtcBKVU4HA0EEQkGzgXeMrJOvYX\n4b0uQkvlVocrUUqpvuN0C+GvwE8BT2cbiMhNIrJURJZWVFT0SVGpKWnUmEgayzUQlFKBw7FAEJEZ\nQLkxZtnBtjPGPGGMmWSMmZSSktIntWUneOci7NExBKVU4HCyhXAccL6IbANeBk4RkRccrGef7IQI\nik0ywXU7nS5FKaX6jGOBYIz5uTEm2xiTA1wOfGqMucqpejpKjgqjRFKJbNTrIiilAofTYwg+KShI\naAjPJMzTCE1VTpejlFJ9wicCwRgz3xgzw+k6OmqLybY3dC6CUipA+EQg+KKgxBx7Q+ciKKUChAZC\nJyK910Vo3b3N0TqUUqqvhDhdgK9KSUml1kTgKS8k1OlilFKqD2gLoRPZiZEUmxTa92iXkVIqMGgg\ndCI7IZIik0Jwjc5FUEoFBg2ETqREh1EiKUQ2FutcBKVUQNBA6ERQkNAQoXMRlFKBQwPhINyxe+ci\n6DiCUqr/00A4iOCEHHtDA0EpFQA0EA4iMn0oAE1lmx2uRCmlep8GwkGkp6ZSaWJp2rXB6VKUUqrX\naSAcxIDESLaadMxubSEopfo/DYSDGJAYyVZPBuG125wuRSmlep0GwkHEhrvY5coiqrUSWuqcLkcp\npXqVBkIXGqJy7I3dWxytQymlepsGQhc8iUPsjT0aCEqp/k0DoQuhqTYQPJU6sKyU6t80ELqQmZJI\nsUmiuWyj06UopVSv0kDowoCESLZ60vFUaAtBKdW/aSB0IScpim0mHVdNodOlKKVUr3IsEEQkXES+\nEpECEVkjIr9xqpaDyUqIYIdkEtZWC417nC5HKaV6zSEFgogkiMi4Hjp2C3CKMeYoYDxwlohM7aF9\n95jgIKE5NtfeqdzkbDFKKdWLugwEEZkvIrEikggsB54UkQeP9MDGqvfedXm/fPNKNKmj7L/la5yt\nQymlelF3Wghxxpha4CLgX8aYY4DTeuLgIhIsIvlAOfCxMWbxAba5SUSWisjSioqKnjjsIUvMHMIe\nE0170XJHjq+UUn2hO4EQIiIZwCzg3Z48uDHGbYwZD2QDU0Rk7AG2ecIYM8kYMyklJaUnD99tQ1Jj\nWOUZTNtODQSlVP/VnUC4D/gQ2GyMWSIig4Ee7Uw3xlQD84CzenK/PWVoajQrzWDC9qyHtiany1FK\nqV7RZSAYY14zxowzxvzAe7/QGHPxkR5YRFJEJN57OwI4HVh/pPvtDbnJUaw2uQQZN+xa7XQ5SinV\nK7ozqPyAd1DZJSKfiEiFiFzVA8fOAOaJyEpgCXYMoUe7pHpKuCuY3bFj7J2SFc4Wo5RSvaQ7XUZn\neAeVZwDbgKHAnUd6YGPMSmPMBG/rY6wx5r4j3WdviksdxB6J10BQSvVb3RpU9v57LvCaMaamF+vx\nWUPTYihw52I0EJRS/VR3AuFdEVkPHA18IiIpQHPvluV7hqRGU+DJhcoN0FLf9QuUUsrPdGdQ+S7g\nWGCSMaYNaABm9nZhvmZ4Wgz5niGI8UDREqfLUUqpHtedQWUXcBXwioi8DtwA7O7twnzNqIwYlstY\n2oLCYP17TpejlFI9rjtdRo9iu4v+6f2a6H0soISFBDM0K4UVrok2EDwep0tSSqkeFdL1Jkz2LkC3\n16ciUtBbBfmyCQMTeH3xBKa0fGm7jQYe43RJSinVY7rTQnCLyJC9d7wzld29V5LvmjAwnvfajsYT\nEgEFLzldjlJK9ajuBMKd2Alk80VkAfAp8JPeLcs3TRyYQAMRbE05FVa/Ca2NTpeklFI9pjtnGX0C\nDANuBW4BRhhj5vV2Yb4oIy6ctNgw3nWdAS01sOxZp0tSSqke0+kYgohc1MlTQ0UEY8ybvVSTzxIR\nJg5M4I2SYG7LPRE+exDGXQ5RSU6XppRSR+xgg8rnHeQ5AwRcIIAdR3h/9S72XHQPiS+dC69cBdf8\nB0LCnC5NKaWOSKeBYIy5vi8L8RcnjUjld3PW88L2eG694J/wxg3w9q1w4WMg4nR5Sil12A7pmsrK\nzlg+ZWQqz36+laYRF8LJd8PKl2He78Dd7nR5Sil12DQQDsN1x+ZQ1djGF1sqYfqddhxh4QPwpyG2\nC6mlzukSlVLqkGkgHIZjBicS4Qpm4cYK2010waNw2WwYdR6snwOzZ0F7q9NlKqXUIelypnInZxvV\nAKuMMeU9X5LvCwsJZtqQJBZsrLAPBAXBqBn2a/BJdlzhlatg+h0wYIqTpSqlVLd1Z+mKG4Bp2Gse\nA5wELANyReQ+Y8y/e6k2n3bi8BQ+XV/O+l21jEyP/fqJvEugegd89hd49lOY8SBEJEBUqi51oZTy\nad29QM4oY8zF3mspj8aednoM8LPeLM6XnX9UJmEhQTz/xfb/ffKE2+H2tZAxDt6+xbYWnjkDXrgY\nytb2fbFKKdUN3QmEAcaYsg73y72P7QHaeqcs35cQFcqFE7J4a0URO/ccYAmL8Dj49kdw/ftw3Rw4\n/f+geBk8cSJ89WTfF6yUUl3oTiDMF5F3ReRaEbkWeNv7WBRQfbgHFpEBIjJPRNaKyBoRue1w9+WU\nm08eiis4iB++uJzG1gOcchocAoOOhZzj4Lhb4YdLYfDJMOcO+NdM2Lao74tWSqlOiDHm4BuICHAR\ncLz3oc+BN0xXL+zqwCIZQIYxZrmIxGDHJS4wxnTapzJp0iSzdOnSIzlsj/tozS6+98Iypg1J4ulr\nJxPuCj74Czxuu+TFsuegrgSm/xRSR0HudIhM7JOalVKBRUSWGWMmdbVddxa3M8Ai7CqnnwALjzQM\nvPstNcYs996uA9YBWUe63752xph0/nzpUXyxZTc3z15Ol9+aoGA48U64eTHkXQoL/gCvXQtPnQql\nK/umaKWUOoDuXEJzFvAVcAkwC1gsIpf0ZBEikgNMABYf4LmbRGSpiCytqKjoycP2mIsmZvOzs0by\nyfpylm6v6t6LwqLhwsftOMNlL0BLPTxxEsy508563rO1V2tWSqn9dafLqAA4fe+cAxFJAebudxW1\nwy9AJBpYANzf1QqqvthltFdTq5upv/+EgYmR3HhCLjPHH2Jjp3EPfPwrWOE9izc4DCbfAMnDoeBl\nOw5x4l0QEtrzxSul+rUe6zICgvabgLa7m6/rkoi4gDeA2f6+nHZEaDDfO3EI63fV8uNX8lmybQ/V\njYcwWzkyEWb+A362DX68xs5nWPw4vPsjO9bw2V/g9euhubbX3oNSKrB1p4XwJ2AcsPeakZcBK40x\nRzQHwTtY/Tywxxjzo+68xpdbCHtVNbRy4p/mUdvczuDkKD768XRCgg8zP3dvgcL5MOEqOwj9wV0Q\nFgPp4yB7Mky8GhIH92T5Sql+qLsthC4Dwbuzi4HjvHc/M8a8dYT1ISLHA58BqwCP9+FfGGPmdPYa\nfwgEgA9Wl/KfFSV8sGYXf79iAmePTccArsMNhr2KlsLy52HXaigtsAPUx91m11BKy7NLaCil1H56\nNBB8hb8EAoDHYzjtoQW0uT20uw0TByXwyLcm9twBakvh/Tth3Tv2flQKjJwB6WMhIdfOd9CAUErR\nA4EgInXYJSr+5yns2aixB3iuV/lTIAAs31HFt578L81ttgH035+fSnpceM8epK4MCufBpo9g/XvQ\n3mwfTxoGg6ZB5WYYcoqdGKdXdVMqIGkLwUesLallZ1Uj3/33MvKy4rhm2iAunTSgdw7WXGOvxbD9\nC1jyNOzZApFJULEeQsJh4jUw6du2NRESZscjlFL9ngaCj/np6wUs3FjJrtpmbjw+l5+eNZLQkD7q\n0tnyKax+A/JfBLN3uEbg2B/CKb+C3ZshNsOuyqqU6nc0EHxQu9vD/727lue/3M5JI1J44upJfRcK\nADVFULgAWhugNB/yZ0NEIjTtgSAXnHoPZE6AnBP0+tBK9SMaCD5s9uLt3P3Was7Ny+DhKyYQHOTQ\nL9+Vr9p1lSZcBZvn2rEIgAHH2K6nQcfCpBsgfqCdWa2U8ksaCD7uyYWF3D9nHeOy42ht93DHGSM4\nbXSacwW526Fs9dddSxnjYNvn4G6BoBAbEhEJEJ0GJ98NUUlfv3bb55CYC7GZztWvlOqUBoIfeH1Z\nEf/4dBO1ze24PYa/XHoUp4xMJcipFsP+aopg62dQuQE2f2JbDTU7ITLZzn0IdkHcAPjwF5A0BC5/\nCRJydHkNpXyMBoIfKayoZ9bjX1JZ38qsSdn84aJxvhMK+9u1Cl67DmpLwBhob4LYLKjbBcZtxyLG\nXGC7m4adCXF+t4CtUv2OBoKfaW338PAnm/jHvM1cNXUg/zdzLOKrA7vG2Os6GI89gyl1FLTU2rAo\nWQErXoC2RpBge52H3ZvB0w7HfM9OnEsZCXHZTr8LpQKGBoIfMsbwhw/W8/iCQq6ZNohrj81hSIof\nDua2t0L1Dlj2rJ1JnXEUNFXBts++3mbkDDsvIigY0sbas520q0mpXqGB4KeMMdz37lqe/XwbAN89\ncTBpMeFcM23Q4S+S5wuMgYoN9hTXLZ/C4iegpeab27ii7BhETJodvM67FPYU2hZIzvEH3K1Sqmsa\nCH7MGMPKohoembeZj9aWATBrUjbpseEEBwVx66lDfbc7qbuaqu1cCAQqN0JzNTRWQfka25qo3GS7\nnQBckTYUkoaCKwISh8CAKXa2ddpYO7itlOqUBkI/0Ob2sGJHNW8sK+KVpTv3Pf7MdZM4ZaSDp6j2\nhYbdUL7WLrnx+rftgHV9mR20bm/6eruYTGhrsN1SZWttcJz3N7tkR9lqe7qsBoYKcBoI/YjbY9hY\nVkd2QgTnPryIkGDhoVnjOWpAvNOl9Y22JhsEzdW2VVBTBHWlUF8Bq16D8DjY8V/ImgjbFtnBbner\nbWHEZMD4K22LIiLBztROyIERZ+laTipgaCD0Uws2VvCjl1dQ39LOI9+ayCkjU/17bKGnla6EBX+0\nk+SyJ0PBS3bMYn8h4TDibDs2ERptQyc8FgYdBzHpfV+3Ur1IA6Efq2lsY9bjX7KhrI5BSZHccHwu\nqTHhnDE6zXfnLziptcEuD169w57ZVLHRtizWvAmNu/fbWGDkuRAc+nWXkzGQMhwmXmtbIy11NjyU\n8hMaCP1cbXMbH68p448frKe8rgWA00al8uQ1k/x/wLmvuNuhocJ2LQWHQmMlrHsX/vtPO5CdOcGu\n7+SKtPMs9q4G21Rlxy4SBsHYi+0ZUUNPtc+11GkLQ/kcDYQAUd/STnltM++v3sWfPtzA0NRojh2S\nxL3njdHWwuFqa7bzI4Jd9he8KxJ2rYRFD0F4PMQPsGdBlRbYa02AnYQHdvA7e8rX4x0TrrbdU4NP\ntGMXSjlAAyHAeDyGH760nHWldWytbGByTgJ3nT2KowfpNQ56jccDuzdBfbltSUiQncG9/QuISoay\nNVC11W4bGgPn/dWuBdXaaMMmLNaeMZU9BXKOO/ixlDoCGggByhjDs59v44mFhbR7DI9ffTRjMmMJ\ndwU7XVrgcbfZsGiphVevsfMtwAbHvgsVeQ05BcrXQ3SKvT3uckgdaUOncbfdR/wg23IxRq+XrQ6J\nXwSCiDwDzADKjTFju9peA6H71pbUMvORRbS5DVnxEdwzYzSriqu5YHwWw9L0dMs+19Zs13mKSrFX\np5MgOzkvKASWPw9f/N1OssPAzsV27afwODsg7mm3+wiNsWtA7SmEyTfa61SEx9lTbINd0FAJkYmQ\nMsqeUuuKsF1Y7nY73qHzMQKWvwTCdKAe+JcGQs/bWFbH2pJa/jFvM5vL6wFIjQnjvpljOH10unMX\n5lEH11Bpr0tRsQEi4u1cCleEnWtRsd6u+7Tpw0PbZ1SqDaOsCRCfYwfSR82ArKPt8yUrYOmzMP0O\nGzSqX/GLQAAQkRzgXQ2E3tPQ0s4TCwvJTY7iD++vZ1dtM5NzErj73NGMD5TJbf1News010JzjV0U\nsL0VolPtWVPl6+y8itZ628UkwTZAWhvsxYzam77uthp0vA2Zxkq73/B4u5+B02wrw9NuJwEGh8Ko\n8+1zwaGTrldsAAAX1ElEQVS2O8zdameIh8fabiw9u81n9ZtAEJGbgJsABg4cePT27dv7rrh+qN3t\n4T/5Jdz3zhpqm9s576hMTh2ZypjMWO1KCgTtrfZfdwv891G7VHnW0fYXe9ZEWPK0DZKipfaXflCw\nPa22cbddmHB/rkhIHga7t0Dy8K/nabgi7LW5QyNtiyZpiG3pNFTY03n3joVoiPSJfhMIHWkLoefU\nt7TzxIItPLagkFa3HeB84OJxnD8+Uweg1f9qb7HXu2iqhvZm28IICYN1b0NNsR3b2LrAth5iM+1c\njZIVB95Xyij72rLVX88M3/mVbWm01NtxkPoyCA6DkefYwBpxjj1Dq26XvZ5G8TIIjbITB4OC7Zlb\nrgjbouk4VuJuh+CQvvke+TANBNUtLe1udu5p5J7/rOHLwt2EhQRxxxkj+M70wU6Xpvxdban9xd9U\nZVsQtcW2m6rgZRscqSPt2lLVO+yZVe4W203VXGtDobYEipZ4f8mH2i4q+OZZWiERdsyjcqNdDXdP\noR1AD4uF6u22m2zq9yE2254K7G6z4yd1pbaFMuU7thsMbI2t9ZA+zrZcWhvscXdvtlcF9OPZ6RoI\n6pDUNbfx3spS5q4rZ+66Mv52+XjOHJNOWEgQrW4PYSHaalC9xOO2f+V39tymj2H75/YXd8oouzZV\n8jAbHiUr7CTBxMFQvNSeqVVfbsMlfqC9vf7d/92v7D1tV+y+gkLs6rrGA2FxNqxKC2ywNJTbfQ0+\n2QZV8nA7o33XSrsUyrAzbAspLMZ2l0Uld6jfY8dsQqO6/j6UFtiFG3On9/jCi34RCCLyEnASkAyU\nAb82xjzd2fYaCL2v3e1h5iOfs6aklpiwEIakRlNa08QLNxxDakw4wcFCdJg2wZUfqS+3/zZU2Fnj\nrgg7eF5bAvmzbevCeOwv+qSh9jodJfmQmAtV2+zjxcu84yjV4GmzZ20lD4fti/73eMnD7f6GnGLH\nYnZvhszxdkA/aQiMnmm7uJIG2+PUFttAW/qMfV3qGLjwsa/HbNLybEsrJOywTx32i0A4VBoIfWNr\nZQNv55fw/upSCisaiA4PYU+Dba4PTo7i3VuPJzJUQ0EFoLZmOwM9bbQNl7LVNjRqS2xYYKB4ue3m\n2v4FuMLtoHz1Drs4YtFX9vlgl+0Ck2B7DY+94ylTvgP/udle42N/V7359ZpZh0gDQR2xplY3VY2t\neIzhnYJSdte38NSirRw3NIk7zhjBhIEJtLS7cQUF6bpJSu3P4wGM7fZq9Q6Wg70PdkA+KsUuwV5f\nblstIaH29ob3IS7LdmWVrrTbjz7/sNfD0kBQveKpzwp5ZN5mGlvd/Pq8MTz8ySZOGZXK7y7Mc7o0\npVQnuhsIuiCKOiQ3njCYT35yEsPSovnFW6vYVdvMy1/tYHVxjdOlKaWOkLYQ1GFpbffwnxXFxEW6\nuO3lFTS3ebh66iDaPR7eXVnKr88bwyVHZztdplIK7TJSfWhzeR2Pzi/kjeVFBAcJ6bHhNLa289r3\npjE0VWc/K+W07gaCniqijtjQ1Bj+cHEeyTGhHDskmdSYMC545HNOe3Ahg1OiuP7YHMpqW7jh+FwS\nokKdLlcp1QltIaheUVrTxMdry3jpq52sK60FICs+guSYMO49bzQTBuqFe5TqK9plpHxCm9vDkm17\nqGls48GPN1LX3E5VYyvn5mWQkxzFd04YjMcYonSym1K9RgNB+aSKuhb+8P56Pl1fRlVjG+GuIFxB\nQbz4nankZcc5XZ5S/ZIGgvJ5bxeU8HZ+MetK66hvaeem6YMJdwVz4YQsEnWsQakeo4Gg/MbOPY1c\n++xXFFbY6foJkS6+e+IQTh2ZqtdoUKoHaCAov9Lc5qayvoW65nZuf7WAdaW1xEe6mHv7idQ3txMT\nHkJSdJjTZSrllzQQlN/yeAyrimu49LEvyYwPp6iqifjIUO4+dyRnjclABDzG6AJ7SnWTBoLye++v\nKuXpRVsZkBhJQVE1hRUNDE+LprS6mYbWdn57QR7fOkYvCK9UV3RimvJ7Z+dlcHZeBgBuj+GjNbv4\n0Sv5DE+LITQkiN+/v45tuxs4c0w6QQKjM2P1Qj5KHQFtISi/sru+hdgIF1srGzj34c9oc3/9/zc7\nIYLnrp9McnQYn22q5Jy8DIJ1WW6ltMtI9X9ltc0I8NqyIhKjQvnLRxuICA2mrd2wq7aZq6YO5Acn\nDaWx1c2iTRVcMy1Hr9ugApJ2Gal+Ly02HICbTx4KwKDESG5+cTnD02I4ZnAiL/x3By8u3kFshIvq\nxjZKapq544wRhIboqu9KHYi2EFS/ZIwhf2c1z3+xjfkbK5ick8jHa8vIio/gx6cPZ3RGLKEhQllt\nC0cPSiDMGxIi2oJQ/Y9ftBBE5Czgb0Aw8JQx5g9O1qP6DxFhwsAEJgxMwO0xBAnM31jBfe+s5Y7X\nCrzbgDEQH+kiSITJOQk8dtXRGgoqYDnWQhCRYGAjcDpQBCwBrjDGrO3sNdpCUEequc3Nlop6Fm2q\npKHVzbisON4uKKGkuoml26s4Jy+dWZMGsKmsnpqmNi6cmEVKTBgxYSEaFMpv+fygsohMA+41xpzp\nvf9zAGPM7zt7jQaC6i0ej+GO1wuYu7aM2uZ2AIKDhOAgoc3t4YopA/F4DCt2VHPmmDR+eMowgoOE\ngqJqJupS3srH+UOXURaws8P9IuAYh2pRAS4oSHhw1niaWt385aMNjM2KY9qQJP46dyNFVU28uHgH\n4a4g8rLiePjTzYgI0WEh3D9nHY9ffTRnjkl3+i0odcR8/iwjEbkJuAlg4ECdlap6V0RoML+cMXrf\n/d9fNI52t4eXvtrB8cNSyE2O4taXVvDo/C1EhdlJcL+fs471pXUYDN8/aQj5O6rJ31nNDcfn0u4x\nhLt0spzyD04GQjEwoMP9bO9j32CMeQJ4AmyXUd+UptTXQoKDuHpazr7798wYTWlNE8u2V3HrqcN4\ncmEhD83dCMDyHdWs2FFFXXM7c9eVsWFXHeOy48mKj+CPl4xz6B0o1T1OjiGEYAeVT8UGwRLgW8aY\nNZ29RscQlK8wxlDb3E5chAuPx9DS7mH24u088MEGEqJcVDW00er2EBsesm9MYtakbOIjQzn/qEyG\npkZry0H1GZ8fVAYQkXOAv2JPO33GGHP/wbbXQFC+rt3twQB3vlbAJ+vKWfDTkwE4/o+f0tjq3rfd\n0NRo3vjesSzYVMGYzFhykqKYvXg7J49IZUBipEPVq/7KLwLhUGkgKH9R19xGdWPbvl/ubywroqqx\nlamDk1ixo4r/e3cdUWHBVDW2kRgVynFDk3mnoIThadH85vyxpMWGMTgl2uF3ofoLDQSlfNjiwt3c\n+85a8rJi+Xzzboqrmzh1ZCrzN1bg9tifyaumDuQ3548F4OlFhbzw3x387fLxDE2NprCigaMGxDv5\nFpQf0UBQyk94PIa6FjseUVzdxPbKBj5eV8azn2/j1JGpFBTVUFnfQlhIEKEhQaTFhrO5vJ4Z4zKo\naWrjyWsmEe4KxhhDu8fgCta1mtQ3+cM8BKUUdg5EXIQLgKz4CLLiIzh2aDJBIjy9aCsTB8bzm/PH\nMC47jttfzaegqIacpEjeXVkKwFVPLSY7IYJ1pXWIwFs/OI6IUB2wVodOWwhK+SiPx7BiZxVHZccT\n4v2r3xhDfUs7TW1uVhXVMG9DOa8vKyLCFUyQCHsaWzl7bDonj0iluLqJ8QPiOXF4ii67EeC0y0ip\nAGCM2TfmECTCE58V8scP1tPxx/rMMWnkJkezq6aJK6cOIi8rjj9+sJ5Ljx7A6MxYlm7bw9OLtnLX\n2SPZUlFPRlwEja3tHD0o0aF3pXqaBoJSAWrDrjoaW9sZnRnL04u28sAHGwgSiItw0dTm5vihycxd\nV87AxEiuOzaH++esw+0x35gzAfDkNZMYkxnLkm17OP+oTG1l+DENBKUUAB+u2UVmXAQZ8eHc8PxS\nCnZWMyU3kfwd1bS6PUwbnLQvPGZNyuaY3CSeWFhITVMbaXHhFOys5uErJnD+UZlOvxV1mDQQlFL/\nwxjD8h3VjMmMpaapjQ276piSa7uGPllXzumj0wgNCWJ1cQ2zHv+Sxlb3vgHvObedgCtISIoO23et\namMMbxeUkBkfwfDUGKLCgveNdyjfoYGglDoiiwt38+GaMq6YMoDz/rGINrcdrxiSEsXsG6fy/Jfb\nWLChgrWltSRHh+L2GE4YlsLDV0z4xn5217eQGBWqXU4O0kBQSvWY/6woZs6qUiblJPC3uZtwG0Nz\nm4e8rDhGpsfw2rKifdtmxUcwbUgS980cw/wNFfzwxeWcMTqdP14ybl9rY3N5HU99tpWfnzOKuAgX\nxhha3R7CQvR02d6ggaCU6hWri2t47ottZMSFc/vpwxERnlxYSFJ0KPe/t45wVzAlNU1kxkVQVttM\ndkIEO/Y0EhPu4uQRKUzKSeTFxTtYW1rLDcfncs+M0fx17kb+/eV2Fvz0ZKLDQnhvZSmby+u57bRh\nTr/dfkEDQSnV52qb24hwBfPV1j384q1VHJObyN3njKaoupHHFhTy5ZZKKutbARiTGcuGXXXccHwu\nz36xjdZ2D+cflUlqTBhvF5RQXtfC7y7M4+hBCYxIjznocY0xzN9YwbTBSbqK7AFoICilfI4xhqKq\nJlra3aTEhPPT1wv4cE0ZydGhRIQGs3NP075tw11BNLd5iAkP4eWbpvL7OevZUFbHLacM5ZoO16cA\nmLe+nOufW8JPzxrBD04a2sfvyvdpICil/EJVQysRocHM31DBa0t3kh5n12r6xTmjKCiq5g/vr6ex\n1U1oSBDD06LZVFbPyIxYEiNd/OKcUQxOieaCRz5nVXENeVlxvHPL8U6/JZ+jgaCU6hdW7Kji882V\nTB+eQkJkKKf+ZQFhriBCgoTI0BBGZ8by8doyJgyMZ8WOaubfcRKrimt4a0Uxt58+nLFZcQC0tnuo\nbmolNSb8f45RXtfMc59vIyUmjOuOzcEYuP3VfKYOTuLyKf5/6V4NBKVUv7RsexUp0WFUNbZy6eNf\ngoE7zxzB2XnpnPKXBQjQ0u4hJEhwBQdx8dFZfLimjOZWNy3tHl7+7lQmDkzg7YISduxu4MYTBnPu\nw5+xpaIBgAvGZ5KbHM1DczeSERfOop+dsm/ehb/SQFBK9Xtltc3EhIcQGWoXbs7fWc1fPtrAjHEZ\nnDg8lZv+vZSVRTVMG5xETnIkizZXUlHXQlZ8xL4AOH10Gh+vLeO56yezeOsenv18qx27CAuhrqWd\nJ6+ZRJvbw9GDEogMDeZnb6xkQGIkt5wyjOiwrheM3rCrjsKKes7Oy+jV78XBaCAopQJeU6ubpdv3\ncNyQZIKChC0V9Tz3+TYq61tIjQnj9WVFNLS6uWrqQH57QR5gr3aXv7OanKQoLnnsC8pqWwDITY7i\nqOw4/l9BCQA3Hp/L3eeOpr6lnTeWFXHhxCwaW9ykx32zS+rbzy1h4cYKltx9GglRoX37DfDSQFBK\nqS489VkhS7dV8bcrxh9wUlxpTRMvLt5BRGgwf/9kM01tbq6ZNoj65nbmrC7lvVtP4PVlRTw6fwuR\nocE0trp54OJxzJo8ALDjFuPv+4jGVjf3XziWb00ZiIjQ3OYmJEj6bJkPDQSllOpBexpa2Vxez/gB\n8ZRUN3He3xfR0u4BgeFp0YQEBREksHxH9b71oQor6qmsbyU4SHB7DKMzYnnppqlc88xXtLV7eO17\n04jqRrfTkfLpQBCRS4F7gVHAFGNMt37LayAopXxFeW0z/5i3mUWbKnn86qMZlhZDc5ubR+dvYe66\nMqLCQvhq6x4A/nzpUczfUM4Hq3eRGW9nbgOkxISRGhPGJUdnMzojlic/20q7x0NWfAQ3nzyUzPiI\nHqnV1wNhFOABHgfu0EBQSvVH60prKa5q4rTRaQC8U1DCXW+sJCM+gp+dNZI5q0rZUlHPyqIaAJKj\nw8iIC2dDWR3D06KpqGth+rAUfnT6cLKOIBx8OhD2HVxkPhoISqkAsru+BREh0TvA3O728OePNpIc\nHcqVxwwiIjSYJxcWcv+cdSRGhVLf0g4GHr1qIqeOSjusY3Y3EHq/8+oIichNwE0AAwf6/wQRpVRg\nS4oO+8b9kOAg7jp75Dceu+bYQVQ3tXLB+CyiwkJ4dP4WJvXBJU17rYUgInOB9AM8dbcx5v95t5mP\nthCUUqpXOd5CMMac1lv7Vkop1fP0WndKKaUAhwJBRC4UkSJgGvCeiHzoRB1KKaW+5sigsjHmLeAt\nJ46tlFLqwLTLSCmlFKCBoJRSyksDQSmlFKCBoJRSysuvVjsVkQpg+2G+PBmo7MFy+prW7zx/fw9a\nv7OcrH+QMSalq438KhCOhIgs7c5MPV+l9TvP39+D1u8sf6hfu4yUUkoBGghKKaW8AikQnnC6gCOk\n9TvP39+D1u8sn68/YMYQlFJKHVwgtRCUUkodhAaCUkopIEACQUTOEpENIrJZRO5yup7uEJFtIrJK\nRPJFZKn3sUQR+VhENnn/TXC6zr1E5BkRKReR1R0e67ReEfm59/PYICJnOlP11zqp/14RKfZ+Bvki\nck6H53yt/gEiMk9E1orIGhG5zfu4X3wGB6nfLz4DEQkXka9EpMBb/2+8j/vF938fY0y//gKCgS3A\nYCAUKABGO11XN+reBiTv99gDwF3e23cBf3S6zg61TQcmAqu7qhcY7f0cwoBc7+cT7IP134u9ot/+\n2/pi/RnARO/tGGCjt06/+AwOUr9ffAaAANHe2y5gMTDVX77/e78CoYUwBdhsjCk0xrQCLwMzHa7p\ncM0Envfefh64wMFavsEYsxDYs9/DndU7E3jZGNNijNkKbMZ+To7ppP7O+GL9pcaY5d7bdcA6IAs/\n+QwOUn9nfK1+Y4yp9951eb8MfvL93ysQAiEL2NnhfhEH/4/mKwwwV0SWichN3sfSjDGl3tu7gDRn\nSuu2zur1p8/kFhFZ6e1S2tvc9+n6RSQHmID9K9XvPoP96gc/+QxEJFhE8oFy4GNjjN99/wMhEPzV\n8caY8cDZwM0iMr3jk8a2O/3mnGF/q9frUWxX43igFPiLs+V0TUSigTeAHxljajs+5w+fwQHq95vP\nwBjj9v7MZgNTRGTsfs/7/Pc/EAKhGBjQ4X629zGfZowp9v5bjr263BSgTEQyALz/ljtXYbd0Vq9f\nfCbGmDLvD7kHeJKvm/Q+Wb+IuLC/TGcbY970Puw3n8GB6ve3zwDAGFMNzAPOwo++/xAYgbAEGCYi\nuSISClwOvO1wTQclIlEiErP3NnAGsBpb97Xeza4F/p8zFXZbZ/W+DVwuImEikgsMA75yoL6D2vuD\n7HUh9jMAH6xfRAR4GlhnjHmww1N+8Rl0Vr+/fAYikiIi8d7bEcDpwHr85Pu/j9Oj2n3xBZyDPWth\nC3C30/V0o97B2DMQCoA1e2sGkoBPgE3AXCDR6Vo71PwStknfhu0PveFg9QJ3ez+PDcDZPlr/v4FV\nwErsD3CGD9d/PLY7YiWQ7/06x18+g4PU7xefATAOWOGtczXwK+/jfvH93/ulS1copZQCAqPLSCml\nVDdoICillAI0EJRSSnlpICillAI0EJRSSnlpICjVR0TkJBF51+k6lOqMBoJSSilAA0Gp/yEiV3nX\nts8Xkce9i5bVi8hD3rXuPxGRFO+240Xkv97F197au/iaiAwVkbne9fGXi8gQ7+6jReR1EVkvIrO9\nM3SV8gkaCEp1ICKjgMuA44xdqMwNXAlEAUuNMWOABcCvvS/5F/AzY8w47IzavY/PBh4xxhwFHIud\nBQ12Fc8fYdfDHwwc1+tvSqluCnG6AKV8zKnA0cAS7x/vEdgFyTzAK95tXgDeFJE4IN4Ys8D7+PPA\na951qLKMMW8BGGOaAbz7+8oYU+S9nw/kAIt6/20p1TUNBKW+SYDnjTE//8aDIvfst93hrvnS0uG2\nG/0ZVD5Eu4yU+qZPgEtEJBX2XRN3EPZn5RLvNt8CFhljaoAqETnB+/jVwAJjr/hVJCIXePcRJiKR\nffoulDoM+teJUh0YY9aKyC+Bj0QkCLv66c1AA/aiJ7/EdiFd5n3JtcBj3l/4hcD13sevBh4Xkfu8\n+7i0D9+GUodFVztVqhtEpN4YE+10HUr1Ju0yUkopBWgLQSmllJe2EJRSSgEaCEoppbw0EJRSSgEa\nCEoppbw0EJRSSgHw/wHLj+qYd8h3DQAAAABJRU5ErkJggg==\n",
            "text/plain": "<matplotlib.figure.Figure at 0x7f01f2f5be10>"
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Now this is pretty impressive for a dataset with 70% missings, right?"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# MTL vs. Transfer Learning\n\nMTL is far less known and used than transfer learning. Transfer learning is often used when you have one target variable with only very few training data, and another one with much more data. \n\nYou would then train a model for the big data set, cut off the output layer, freeze the weights of the earlier layers and retrain an output layer for the target with less data.\n\nThat way the weights of the bigger part of the network are trained on a big data set, and to retrain the output layer the smaller dataset can be sufficient.\n\nIf the data sets for your targets are roughly of the same size, you are probably better off with Multi-Task Learning.\n\n# Summary and assumptions\n\nIn this notebook we have seen how Multi-Task Learning can speed up model fitting and improve the performance on targets with missing data.\n\nBoth transfer learning and Muli-Task Learning assume that your outputs rely on similar latent features. If you need completely different concepts to describe your targets, you probably won't win a lot by using either technique.\n\nIn our example, the definition of the outputs only differs in linear coefficients. In many cases the relation is more complicated, and a single output unit may not be sufficient to describe the differences.\n\nIn my next tutorial I will show how to share some of the layers between the outputs and then train a few extra layers per output to tackle more complex Multi-Task problems.\n\n# Further Reading\n- [An Overview of Multi-Task Learning in Deep Neural Networks](http://ruder.io/multi-task/) by [Sebastian Ruder](https://twitter.com/seb_ruder). A much more complete overview of MTL."
    }
  ],
  "metadata": {
    "hide_input": false,
    "kernelspec": {
      "name": "conda-env-py36keras-py",
      "display_name": "Python [conda env:py36keras]",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.3",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "nbpresent": {
      "slides": {},
      "themes": {
        "default": "ecdaf271-41f5-49cb-bd42-408ec5cd99c3",
        "theme": {}
      }
    },
    "toc": {
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "toc_cell": true,
      "toc_position": {
        "height": "828px",
        "left": "0px",
        "right": "1626px",
        "top": "106px",
        "width": "212px"
      },
      "toc_section_display": "block",
      "toc_window_display": true
    },
    "gist": {
      "id": "7743fb49560dd2c87b4a3c0a5f9b2f39",
      "data": {
        "description": "multi_task_learning.ipynb",
        "public": true
      }
    },
    "_draft": {
      "nbviewer_url": "https://gist.github.com/7743fb49560dd2c87b4a3c0a5f9b2f39"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}