Using GraphSage for link predictions. So much fun.

GraphSageLinkPrediction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Graph Link Prediction using GraphSAGE"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "This article is based on the paper [\"Inductive Representation Learning on Large Graphs\" by Hamilton, Ying and Leskovec](https://papers.nips.cc/paper/6703-inductive-representation-learning-on-large-graphs.pdf).\n",
        "The [StellarGraph](https://www.stellargraph.io) implementation of the [GraphSAGE](http://snap.stanford.edu/graphsage/) algorithm is used to build a model that predicts citation links of [the Cora dataset](https://graphsandnetworks.com/the-cora-dataset/). \n",
        "\n",
        "The way link prediction is turned into a supervised learning task is actually very savvy. Pairs of nodes are embedded and a binary prediction model is trained where '1' means the nodes are connected and '0' means they are not connected. It's like embedding the adjacency matrix and finding a decision boundary between two types of elements. The entire model is trained end-to-end by minimizing the loss function of choice (e.g., binary cross-entropy between predicted link probabilities and true link labels, with true/false citation links having labels 1/0) using stochastic gradient descent (SGD) updates of the model parameters, with minibatches of 'training' links fed into the model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import networkx as nx\n",
        "import pandas as pd\n",
        "import os\n",
        "\n",
        "import stellargraph as sg\n",
        "from stellargraph.data import EdgeSplitter\n",
        "from stellargraph.mapper import GraphSAGELinkGenerator\n",
        "from stellargraph.layer import GraphSAGE, link_classification\n",
        "\n",
        "import tensorflow.keras as keras # DO NOT USE KERAS DIRECTLY\n",
        "from sklearn import preprocessing, feature_extraction, model_selection\n",
        "\n",
        "from stellargraph import globalvar"
      ],
      "outputs": [],
      "execution_count": 31,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset\n",
        "\n",
        "[The Cora dataset](https://graphsandnetworks.com/the-cora-dataset/) is the hello-world dataset when looking at graph learning. We have described in details in [this article](https://graphsandnetworks.com/the-cora-dataset/) and will not repeat it here. You can also find in the article a direct link to download the data.\n",
        "\n",
        "The construction below recreates the steps outlined in the article.\n",
        "\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "data_dir = os.path.expanduser(\"/Users/swa/Desktop/LargeFiles/Graphs/cora\")\n",
        "cora_location = os.path.expanduser(os.path.join(data_dir, \"cora.cites\"))\n",
        "g_nx = nx.read_edgelist(path=cora_location)\n",
        "\n",
        "cora_data_location = os.path.expanduser(os.path.join(data_dir, \"cora.content\"))\n",
        "node_attr = pd.read_csv(cora_data_location, sep='\\t', header=None)\n",
        "values = { str(row.tolist()[0]): row.tolist()[-1] for _, row in node_attr.iterrows()}\n",
        "nx.set_node_attributes(g_nx, values, 'subject')\n",
        "\n",
        "g_nx_ccs = (g_nx.subgraph(c).copy() for c in nx.connected_components(g_nx))\n",
        "g_nx = max(g_nx_ccs, key=len)\n",
        "print(\"Largest connected component: {} nodes, {} edges\".format(\n",
        "    g_nx.number_of_nodes(), g_nx.number_of_edges()))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Largest connected component: 2485 nodes, 5069 edges\n"
          ]
        }
      ],
      "execution_count": 3,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "The features of the nodes are taken into account in the model:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "feature_names = [\"w_{}\".format(ii) for ii in range(1433)]\n",
        "column_names =  feature_names + [\"subject\"]\n",
        "node_data = pd.read_csv(os.path.join(data_dir, \"cora.content\"), \n",
        "                        sep=\"\\t\",  \n",
        "                        header=None, \n",
        "                        names=column_names)\n",
        "node_data.drop(['subject'], axis=1, inplace=True)\n"
      ],
      "outputs": [],
      "execution_count": 4,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "node_data.index = node_data.index.map(str)\n",
        "node_data = node_data[node_data.index.isin(list(g_nx.nodes()))]"
      ],
      "outputs": [],
      "execution_count": 5,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "node_data.head(2)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 7,
          "data": {
            "text/plain": [
              "         w_0  w_1  w_2  w_3  w_4  w_5  w_6  w_7  w_8  w_9  ...  w_1423  \\\n",
              "31336      0    0    0    0    0    0    0    0    0    0  ...       0   \n",
              "1061127    0    0    0    0    0    0    0    0    0    0  ...       0   \n",
              "\n",
              "         w_1424  w_1425  w_1426  w_1427  w_1428  w_1429  w_1430  w_1431  \\\n",
              "31336         0       0       1       0       0       0       0       0   \n",
              "1061127       0       1       0       0       0       0       0       0   \n",
              "\n",
              "         w_1432  \n",
              "31336         0  \n",
              "1061127       0  \n",
              "\n",
              "[2 rows x 1433 columns]"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>w_0</th>\n",
              "      <th>w_1</th>\n",
              "      <th>w_2</th>\n",
              "      <th>w_3</th>\n",
              "      <th>w_4</th>\n",
              "      <th>w_5</th>\n",
              "      <th>w_6</th>\n",
              "      <th>w_7</th>\n",
              "      <th>w_8</th>\n",
              "      <th>w_9</th>\n",
              "      <th>...</th>\n",
              "      <th>w_1423</th>\n",
              "      <th>w_1424</th>\n",
              "      <th>w_1425</th>\n",
              "      <th>w_1426</th>\n",
              "      <th>w_1427</th>\n",
              "      <th>w_1428</th>\n",
              "      <th>w_1429</th>\n",
              "      <th>w_1430</th>\n",
              "      <th>w_1431</th>\n",
              "      <th>w_1432</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31336</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1061127</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2 rows × 1433 columns</p>\n",
              "</div>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 7,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Define a set of node features that will be used by the model as the difference between the set of all node features and a list of user-defined node attributes to ignore:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "feature_names = sorted(set(node_data.columns))"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "We need to convert node features that will be used by the model to numeric values that are required for GraphSAGE input. Note that all node features in the Cora dataset, except the categorical \"subject\" feature, are already numeric, and don't require the conversion."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "node_features = node_data[feature_names].values"
      ],
      "outputs": [],
      "execution_count": 12,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "node_features.shape"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 13,
          "data": {
            "text/plain": [
              "(2485, 1433)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 13,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Add node data to g_nx:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "for nid, f in zip(node_data.index, node_features):\n",
        "    g_nx.node[nid]['label'] = \"paper\"\n",
        "    g_nx.node[nid][\"feature\"] = f"
      ],
      "outputs": [],
      "execution_count": 17,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Splitting a graph\n",
        "\n",
        "Splitting graph-like data into train and test sets is not as straightforward as in classic (tabular) machine learning. If you take a subset of nodes you also need to ensure that the edges do not have endpoints across the other set. That is, edges should connect only to train or test nodes but not having endpoints in each set. So, this is in general a little tricky but the StellarGraph framework makes it easy by giving us a method to do this in one line of code. Actually the splitting happens in a slightly different fashion. Instead of taking a subset of nodes all the nodes are kept in both training and test but the edges are randomly sampled. Each of these graphs will have the same number of nodes as the input graph, but the number of links will differ (be reduced) as some of the links will be removed during each split and used as the positive samples for training/testing the link prediction classifier."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "From the original graph G, extract a randomly sampled subset of test edges (true and false citation links) and the reduced graph G_test with the positive test edges removed. Define an edge splitter on the original graph `g_nx`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "edge_splitter_test = EdgeSplitter(g_nx)"
      ],
      "outputs": [],
      "execution_count": 18,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Randomly sample a fraction p=0.1 of all positive links, and same number of negative links, from G, and obtain the reduced graph G_test with the sampled links removed:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "G_test, edge_ids_test, edge_labels_test = edge_splitter_test.train_test_split(\n",
        "    p=0.1, method=\"global\", keep_connected=True\n",
        ")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "** Sampled 506 positive and 506 negative edges. **\n"
          ]
        }
      ],
      "execution_count": 19,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The reduced graph G_test, together with the test ground truth set of links (edge_ids_test, edge_labels_test), will be used for testing the model.\n",
        "\n",
        "Now repeat this procedure to obtain the training data for the model. From the reduced graph G_test, extract a randomly sampled subset of train edges (true and false citation links) and the reduced graph G_train with the positive train edges removed:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "edge_splitter_train = EdgeSplitter(G_test)\n",
        "G_train, edge_ids_train, edge_labels_train = edge_splitter_train.train_test_split(\n",
        "    p=0.1, method=\"global\", keep_connected=True\n",
        ")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "** Sampled 456 positive and 456 negative edges. **\n"
          ]
        }
      ],
      "execution_count": 20,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Defining the GraphSage model"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Convert G_train and G_test to StellarGraph objects (undirected, as required by GraphSAGE) for ML:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "G_train = sg.StellarGraph(G_train, node_features=\"feature\")\n",
        "G_test = sg.StellarGraph(G_test, node_features=\"feature\")"
      ],
      "outputs": [],
      "execution_count": 21,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Summary of G_train and G_test - note that they have the same set of nodes, only differing in their edge sets:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "print(G_train.info())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "StellarGraph: Undirected multigraph\n",
            " Nodes: 2485, Edges: 4107\n",
            "\n",
            " Node types:\n",
            "  paper: [2485]\n",
            "        Attributes: {'feature', 'subject'}\n",
            "    Edge types: paper-default->paper\n",
            "\n",
            " Edge types:\n",
            "    paper-default->paper: [4107]\n",
            "\n"
          ]
        }
      ],
      "execution_count": 22,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "print(G_test.info())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "StellarGraph: Undirected multigraph\n",
            " Nodes: 2485, Edges: 4563\n",
            "\n",
            " Node types:\n",
            "  paper: [2485]\n",
            "        Attributes: {'feature', 'subject'}\n",
            "    Edge types: paper-default->paper\n",
            "\n",
            " Edge types:\n",
            "    paper-default->paper: [4563]\n",
            "\n"
          ]
        }
      ],
      "execution_count": 23,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we create the link mappers for sampling and streaming training and testing data to the model. The link mappers essentially \"map\" pairs of nodes `(paper1, paper2)` to the input of GraphSAGE: they take minibatches of node pairs, sample 2-hop subgraphs with `(paper1, paper2)` head nodes extracted from those pairs, and feed them, together with the corresponding binary labels indicating whether those pairs represent true or false citation links, to the input layer of the GraphSAGE model, for SGD updates of the model parameters.\n",
        "\n",
        "Specify the minibatch size (number of node pairs per minibatch) and the number of epochs for training the model:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "batch_size = 20\n",
        "epochs = 20"
      ],
      "outputs": [],
      "execution_count": 24,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specify the sizes of 1- and 2-hop neighbour samples for GraphSAGE:\n",
        "\n",
        "Note that the length of `num_samples` list defines the number of layers/iterations in the GraphSAGE model. In this example, we are defining a 2-layer GraphSAGE model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "num_samples = [20, 10]"
      ],
      "outputs": [],
      "execution_count": 25,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "train_gen = GraphSAGELinkGenerator(G_train, batch_size, num_samples).flow(edge_ids_train,edge_labels_train)\n",
        "test_gen = GraphSAGELinkGenerator(G_test,  batch_size, num_samples).flow(edge_ids_test, edge_labels_test)"
      ],
      "outputs": [],
      "execution_count": 26,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Build the model: a 2-layer GraphSAGE model acting as node representation learner, with a link classification layer on concatenated `(paper1, paper2)` node embeddings.\n",
        "\n",
        "GraphSAGE part of the model, with hidden layer sizes of 50 for both GraphSAGE layers, a bias term, and no dropout. (Dropout can be switched on by specifying a positive dropout rate, 0 < dropout < 1)\n",
        "Note that the length of layer_sizes list must be equal to the length of num_samples, as len(num_samples) defines the number of hops (layers) in the GraphSAGE model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "layer_sizes = [20, 20]\n",
        "assert len(layer_sizes) == len(num_samples)\n",
        "\n",
        "graphsage = GraphSAGE(\n",
        "        layer_sizes=layer_sizes, generator=train_gen, bias=True, dropout=0.3\n",
        "    )"
      ],
      "outputs": [],
      "execution_count": 32,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "x_inp, x_out = graphsage.build()"
      ],
      "outputs": [],
      "execution_count": 33,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Final link classification layer that takes a pair of node embeddings produced by graphsage, applies a binary operator to them to produce the corresponding link embedding ('ip' for inner product; other options for the binary operator can be seen by running a cell with `?link_classification` in it), and passes it through a dense layer:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "prediction = link_classification(\n",
        "        output_dim=1, output_act=\"relu\", edge_embedding_method='ip'\n",
        "    )(x_out)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "link_classification: using 'ip' method to combine node embeddings into edge embeddings\n"
          ]
        }
      ],
      "execution_count": 34,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Stack the GraphSAGE and prediction layers into a Keras model, and specify the loss"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "model = keras.Model(inputs=x_inp, outputs=prediction)\n",
        "\n",
        "model.compile(\n",
        "        optimizer=keras.optimizers.Adam(lr=1e-3),\n",
        "        loss=keras.losses.binary_crossentropy,\n",
        "        metrics=[\"acc\"],\n",
        "    )"
      ],
      "outputs": [],
      "execution_count": 35,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Evaluate the initial (untrained) model on the train and test set:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "init_train_metrics = model.evaluate_generator(train_gen)\n",
        "init_test_metrics = model.evaluate_generator(test_gen)\n",
        "\n",
        "print(\"\\nTrain Set Metrics of the initial (untrained) model:\")\n",
        "for name, val in zip(model.metrics_names, init_train_metrics):\n",
        "    print(\"\\t{}: {:0.4f}\".format(name, val))\n",
        "\n",
        "print(\"\\nTest Set Metrics of the initial (untrained) model:\")\n",
        "for name, val in zip(model.metrics_names, init_test_metrics):\n",
        "    print(\"\\t{}: {:0.4f}\".format(name, val))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Train Set Metrics of the initial (untrained) model:\n",
            "\tloss: 0.6847\n",
            "\tacc: 0.6316\n",
            "\n",
            "Test Set Metrics of the initial (untrained) model:\n",
            "\tloss: 0.6795\n",
            "\tacc: 0.6364\n"
          ]
        }
      ],
      "execution_count": 36,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's go for it:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "history = model.fit_generator(\n",
        "        train_gen,\n",
        "        epochs=epochs,\n",
        "        validation_data=test_gen,\n",
        "        verbose=2\n",
        "    )"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING:tensorflow:From /Users/swa/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "Epoch 1/20\n",
            "51/51 [==============================] - 2s 47ms/step - loss: 0.6117 - acc: 0.6324\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.7215 - acc: 0.6064 - val_loss: 0.6117 - val_acc: 0.6324\n",
            "Epoch 2/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.5301 - acc: 0.7263\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.5407 - acc: 0.7171 - val_loss: 0.5301 - val_acc: 0.7263\n",
            "Epoch 3/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.4952 - acc: 0.7441\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.4658 - acc: 0.7928 - val_loss: 0.4952 - val_acc: 0.7441\n",
            "Epoch 4/20\n",
            "51/51 [==============================] - 3s 56ms/step - loss: 0.4618 - acc: 0.7836\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.3992 - acc: 0.8344 - val_loss: 0.4618 - val_acc: 0.7836\n",
            "Epoch 5/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.5118 - acc: 0.7856\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.3644 - acc: 0.8980 - val_loss: 0.5118 - val_acc: 0.7856\n",
            "Epoch 6/20\n",
            "51/51 [==============================] - 3s 57ms/step - loss: 0.4758 - acc: 0.7964\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 8s - loss: 0.3553 - acc: 0.8936 - val_loss: 0.4758 - val_acc: 0.7964\n",
            "Epoch 7/20\n",
            "51/51 [==============================] - 3s 57ms/step - loss: 0.4806 - acc: 0.8024\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 8s - loss: 0.2726 - acc: 0.9243 - val_loss: 0.4806 - val_acc: 0.8024\n",
            "Epoch 8/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.5067 - acc: 0.8093\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.2734 - acc: 0.9309 - val_loss: 0.5067 - val_acc: 0.8093\n",
            "Epoch 9/20\n",
            "51/51 [==============================] - 3s 55ms/step - loss: 0.5595 - acc: 0.8014\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.2268 - acc: 0.9627 - val_loss: 0.5595 - val_acc: 0.8014\n",
            "Epoch 10/20\n",
            "51/51 [==============================] - 3s 52ms/step - loss: 0.5354 - acc: 0.7984\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.2051 - acc: 0.9616 - val_loss: 0.5354 - val_acc: 0.7984\n",
            "Epoch 11/20\n",
            "51/51 [==============================] - 3s 50ms/step - loss: 0.5476 - acc: 0.8123\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1896 - acc: 0.9627 - val_loss: 0.5476 - val_acc: 0.8123\n",
            "Epoch 12/20\n",
            "51/51 [==============================] - 3s 54ms/step - loss: 0.5359 - acc: 0.8113\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1985 - acc: 0.9594 - val_loss: 0.5359 - val_acc: 0.8113\n",
            "Epoch 13/20\n",
            "51/51 [==============================] - 3s 54ms/step - loss: 0.5721 - acc: 0.8103\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1794 - acc: 0.9737 - val_loss: 0.5721 - val_acc: 0.8103\n",
            "Epoch 14/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.5526 - acc: 0.8152\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1600 - acc: 0.9825 - val_loss: 0.5526 - val_acc: 0.8152\n",
            "Epoch 15/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.6594 - acc: 0.8034\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1576 - acc: 0.9781 - val_loss: 0.6594 - val_acc: 0.8034\n",
            "Epoch 16/20\n",
            "51/51 [==============================] - 3s 54ms/step - loss: 0.5591 - acc: 0.8132\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1426 - acc: 0.9825 - val_loss: 0.5591 - val_acc: 0.8132\n",
            "Epoch 17/20\n",
            "51/51 [==============================] - 3s 52ms/step - loss: 0.5637 - acc: 0.7935\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1359 - acc: 0.9792 - val_loss: 0.5637 - val_acc: 0.7935\n",
            "Epoch 18/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.6060 - acc: 0.8083\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1306 - acc: 0.9912 - val_loss: 0.6060 - val_acc: 0.8083\n",
            "Epoch 19/20\n",
            "51/51 [==============================] - 3s 53ms/step - loss: 0.5586 - acc: 0.7955\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1258 - acc: 0.9857 - val_loss: 0.5586 - val_acc: 0.7955\n",
            "Epoch 20/20\n",
            "51/51 [==============================] - 3s 51ms/step - loss: 0.6495 - acc: 0.7964\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n",
            " - 7s - loss: 0.1193 - acc: 0.9923 - val_loss: 0.6495 - val_acc: 0.7964\n"
          ]
        }
      ],
      "execution_count": 37,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can use tensorboard to see pretty dataviz or you can use a normal Python plot:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "def plot_history(history):\n",
        "    metrics = sorted(history.history.keys())\n",
        "    metrics = metrics[:len(metrics)//2]\n",
        "    \n",
        "    f,axs = plt.subplots(1, len(metrics), figsize=(12,4))\n",
        "\n",
        "    for m,ax in zip(metrics,axs):\n",
        "        # summarize history for metric m\n",
        "        ax.plot(history.history[m])\n",
        "        ax.plot(history.history['val_' + m])\n",
        "        ax.set_title(m)\n",
        "        ax.set_ylabel(m)\n",
        "        ax.set_xlabel('epoch')\n",
        "        ax.legend(['train', 'test'], loc='upper right')\n",
        "\n",
        "plot_history(history)        "
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x288 with 2 Axes>"
            ],
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAtoAAAEWCAYAAABYLDBhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4lMX2wPHvSa8kpNASIKFIEaSF0ERAVMCKDbFjAftVrw1791qv/rwqiogdFBEVFaUoiEgNvZeEFmoIJJCQvvP7YxYNIUCA3ewmOZ/n2Se7b9n3bMTJ2XlnzogxBqWUUkoppZRr+Xg6AKWUUkoppaojTbSVUkoppZRyA020lVJKKaWUcgNNtJVSSimllHIDTbSVUkoppZRyA020lVJKKaWUcgNNtJVSSilVJYjIJhE5x9NxKFVRmmgrpZRSSinlBppoK6WUUkop5QaaaCsFiMhwEUkVkQMiskpELi21b6iIrC61r6Nze0MRmSAiGSKSKSLveO4TKKVUzSEigSLylohsdz7eEpFA574YEflJRLJEZK+I/CkiPs59j4jINmd7vlZE+nr2k6jqzs/TASjlJVKBnsBO4ErgCxFpBpwJPAMMBFKApkCRiPgCPwG/A9cDJUBS5YetlFI10uNAV6A9YIAfgCeAJ4EHgHQg1nlsV8CISAvgbqCzMWa7iCQAvpUbtqpptEdbKcAY840xZrsxxmGM+RpYDyQDtwKvGmMWGGuDMWazc18D4CFjTK4xJt8YM8uDH0EppWqSa4HnjDG7jTEZwLPYTg+AIqA+0NgYU2SM+dMYY7AdIoFAaxHxN8ZsMsakeiR6VWNooq0UICI3iMgS563GLKANEAM0xPZ2l9UQ2GyMKa7MOJVSSgG2o2NzqdebndsAXgM2AFNEJE1EhgMYYzYA92HvUu4Wka9EpAFKuZEm2qrGE5HGwIfYW4rRxphIYAUgwFbscJGytgKNRESHXymlVOXbDjQu9bqRcxvGmAPGmAeMMU2Ai4F/HxqLbYwZY4w503muAV6p3LBVTaOJtlIQim1wMwBE5CZsjzbAKOBBEekkVjNnYj4f2AG8LCKhIhIkIj08EbxSStVAY4EnRCRWRGKAp4AvAETkQmdbLUA2dsiIQ0RaiMjZzkmT+UAe4PBQ/KqG0ERb1XjGmFXAG8AcYBfQFvjLue8b4EVgDHAA+B6IMsaUABcBzYAt2Ik3V1V68EopVTO9gJ2gvgxYDixybgNoDkwDcrDt+nvGmOnY8dkvA3uwE9/rAI9WbtiqphE7P0AppZRSSinlStqjrZRSSimllBtooq2UUkoppZQbaKKtlFJKKaWUG2iirZRSSimllBtUmxrAMTExJiEhwdNhKKXUSVm4cOEeY0zs8Y+sPrTdVkpVVRVts6tNop2QkEBKSoqnw1BKqZMiIpuPf1T1ou22UqqqqmibrUNHlFJKKaWUcgO3JdoiMlpEdovIiqPsFxF5W0Q2iMgyEelYat+NIrLe+bjRXTEqpZRSSinlLu7s0f4E6H+M/QOwqzc1B4YBIwBEJAp4GugCJANPi0htN8aplFJKKaWUy7ltjLYxZqaIJBzjkEuAz4xdmnKuiESKSH2gNzDVGLMXQESmYhP2se6KVSnlWUVFRaSnp5Ofn+/pUNwuKCiI+Ph4/P39PR2KUkqdtJrSbp9qm+3JyZBxwNZSr9Od2462/QgiMgzbG06jRo3cE6VSyu3S09MJDw8nISEBEfF0OG5jjCEzM5P09HQSExM9HY5SSp20mtBuu6LNrtKTIY0xI40xScaYpNjYGlUVS6lqJT8/n+jo6GrbWB8iIkRHR1f7HiClVPVXE9ptV7TZnky0twENS72Od2472nalVDVWnRvr0mrK51RKVX81oT071c/oyUR7InCDs/pIVyDbGLMDmAycJyK1nZMgz3NuU0opr2SMIa+wmD05BezNLfR0ONXW1r0HeeXXNezM1jsCSqmqwZ3l/cYCc4AWIpIuIreIyO0icrvzkElAGrAB+BC4E8A5CfJ5YIHz8dyhiZFKKXU8xQ4HGQcKWLfzAGt3HmDL3oNkHCggJ7+I4hLHUc/Lysrivffeq9A1ShyGnPwidu3Pp/c5/Zi7egvrd+ewPSuPrIOaaLtLdl4RI2akMjct09OhKKU87ETa7NLOP/98srKy3BBR+dxZdeTq4+w3wF1H2TcaGO2OuJRS1VN+UQl7cgrIOliEwxhCAvzw8xFyC4oPS34DfH0IDvAlyN+XYH9fggN88fORvxvtO++887D3LS4uxuBDbmExBwtLyC0oJr/IgcEAMPLL8YQG+BIS6EdogC/+vlV66otXa1W/FuGBfszbuJeBHcqdI6+UqiGO1Wb7+R09vZ00aZK7QztMtVmCXSlV8xhj2J9fxJ6cQnILihERIoP9iQkLIDjgn+atuMRBXlEJeUUl5BeWkFfkIDuv6O/9fj4+PHL/g6SmptL2jHYEBPgTEBBIWEQk69etZeIfKdx3y7Xs3LGN4sICht5xN7ffNoyQAF+aNmlCSkoK2btz6D5gAGeeeSazZ88mLi6OH374geDgYE/8aqolXx8hKaE28zZqj7ZSNd3w4cNJTU2lffv2+Pv7ExQURO3atVmzZg3r1q1j4MCBbN26lfz8fO69916GDRsGQEJCAikpKeTk5DCgEtpsTbSVUl7l2R9Xsmr7/mMeY7DJc1GJwRiDiODvK/j5+lDetJXWDWrx9EWnEx70Tx3UEoch35l85xWWcP9jz7B61Sq+nPQHC+bM4u4br2Li9Dmc1qwpIQF+fPbpxzSoG0tBfj6dO3fm5usGEx4Ufdh11q9fz9ixY/nwww8ZNGgQ3377Ldddd50LfivqkC5Nopm+NoOMAwXEhgd6OhylFBVrt0/UoXb7aF5++WVWrFjBkiVLmDFjBhdccAErVqz4uwzf6NGjiYqKIi8vj86dO3P55ZcTHV35bbYm2krVIKkZOcxJzSQqNICmsWE0jg4hyN/XLddyOAzbs/NIzchlc2YuoQF+1I8MIi4ymHoRQQT6nfh1HcZQVGIodjjA2B5Ofz9ffH1OfFa4r48QGuhHaKBtBkv2hxHk70OzOmFsqRVIcnJnzklu+/eM82feeI/vvvsOgK1bt7J+/fojGu3ExETat28PQKdOndi0adMJx+WNRKQ/8H+ALzDKGPNymf1vAn2cL0OAOsaYSHfEkpwYBcCCTXs5v219d1xCKVUFJScnH1br+u233/aKNlsTbaWqMYfDsHjrPqas2sXUVbtIy8g9bL+PQHztEJrEhtI0NoymsWF/P48JC6hQWaPcgmI27sklNSOH1Az7My0jl417csgvOvrkw5iwQOIig6gfEcyNrf3JOFBAgK/w0Hkt8Pfzwc9HEBEcxrA/zw4POVhYjI8ItUP8iQ4LdMuXhJAAP8IC/QkPC/v788+YMYNp06YxZ84cQkJC6N27d7l1VQMD/+lh9fX1JS8vz+XxVTYR8QXeBc7FLiC2QEQmGmNWHTrGGHN/qePvATq4K562cREE+/syLy1TE22lvMSxep4rS2ho6N/PvanN1kRbqWomv6iEWev3MHXVLn5bs4s9OYX4+Qhdm0RzY7cEep0WS05B8d+JcZrz59y0zMMS4/Agv8MS76axoYQE+JGWkUPann8S6h2lSq0dStybxobSvWn03+cnxoSSV1jC9qw8tmfn25/O5xsycjhYGM6O7MMbuEPDQRwOW0kk0M+XBhHB1A71x9fH9RMOw8PDOXDgQLn7srOzqV27NiEhIaxZs4a5c+e6/PpeLBnYYIxJAxCRr4BLgFVHOf5q4Gl3BePv60OnxrWZt1GLUSlVk1WVNlsTbaWqgb25hfy22vZa/7l+D3lFJYQH+tGrRSzntq5L7xZ1iAj2P+ycNnERh70+NNQjrVSvdGpGDrM3ZDJh0eFrRoUH+tGkThjdmkT/nYg3qcBQlISY0HK3r169mub1a1FUYigqcVBU4qDQOQYbY6gdGkxYoJ9bF0eIjo6mR48etGnThuDgYOrWrfv3vv79+/P+++/TqlUrWrRoQdeuXd0WhxeKA7aWep0OdCnvQBFpDCQCvx/tzURkGDAMoFGjRicVUHJiFG9OW0fWwUIiQwJO6j2UUlVbVWmzxVbZq/qSkpJMSkqKp8NQqtJs2pPLVOeQkJTNe3EYqB8RxDmt6nJu67p0bRJNgJ9ren5zCorZmJFLbmExTWJDiQ0LdGnSu3r1alq1auWy9/N25X1eEVlojEnyUEhHJSJXAP2NMbc6X18PdDHG3F3OsY8A8caYeyry3ifbbs9Ny2TwyLl8eEMS57aue/wTlFIuV5Pa7VNps7VHW6kqxBjDzPV7+L9p61i0xRbcb1kvnLv7NOPc1vVoE1fLLb2+YYF+tI2POP6BqjraBjQs9Treua08gznK+giu1L5hJAG+PszfmKmJtlLKq2mirVQVcCjBfmvaOhZvySIuMpgnLmhFv9Pr0TAqxNPhqeptAdBcRBKxCfZg4JqyB4lIS6A2dkVgtwry96V9w0jm6zhtpZSX00RbKS9mjOFPZ4K9yJlgv3RpW67oFO+yYSFKHYsxplhE7gYmY8v7jTbGrBSR54AUY8xE56GDga9MJY1HTE6MYsQfqeQUFBMWqH/KlFLeSVsnpbxQ2QS7QUQQL17ahis7NdQEW1U6Y8wkYFKZbU+Vef1MZcaUnBjFO9M3sHDzPnqdFluZl1ZKqQrTRFspL2KMYdaGPbw1bT0LN+/7O8G+olP8SS3wolR11alxbXx9hPkbMzXRVkp5LU20VbWyPD2bbVkHCfL3Jdjfl+AA+zOozPMTWUmwqMRBXlEJ+YXO5bqdS3bnFZXYJbwLHYQE+hIXGUz9iKDDlvmuKGMMf23I5K1p60jZvI/6EUG8MLANVyZpgq1UeUID/WgTF6HjtJVSXk0TbVVt/LpiB3d8uYiKjBAN8PUhyN/nsOQ7wM/HJs7O5PnQ8xLHiQ05DQ/y+zvpbhAZ7HwE0SDCPq8XEYS/rx3+UV6C/fzANgzSBLvSZWVlMWbMGO68884TPvett95i2LBhhIToxNTK1CUxik/+2kR+UYlbVglVSnmvqtJma6KtqoX5G/fyr6+W0L5hJM9f0obCEscRPdClk+hDvdH5pfYXljhsL7i/L0F/J+A+R/SIl95/aF9OQRHbsvLZ4VzxcFtWPjuy81iyNYt9B4sOi1UE6oQHUj8imBKHYfm2bE2wvUBWVhbvvffeSTfa1113nSbalaxLYhQjZ6axeEsW3ZpGezocpVQlqipttibaqspbt+sAt366gPjIYD66sTNRoZ5ZKa5T4/K35xWWsD3bJuA7svLZ5kzGd2TncyC/iOcvOZ1BnRtqgu1hw4cPJzU1lfbt23PuuedSp04dxo0bR0FBAZdeeinPPvssubm5DBo0iPT0dEpKSnjyySfZtWsX27dvp0+fPsTExDB9+nRPf5QaI6lxFCL2i7Ym2krVLFWlzdZEW1VpO7LzuHH0fAL9ffn05mSPJdnHEhzgS9PYMJrGhnk6lKrhl+Gwc7lr37NeWxjw8jEPefnll1mxYgVLlixhypQpjB8/nvnz52OM4eKLL2bmzJlkZGTQoEEDfv75ZwCys7OJiIjgv//9L9OnTycmJsa1catjigjxp2W9WszflAk093Q4StVcHmi3q0qb7dY6YSLSX0TWisgGERlezv7GIvKbiCwTkRkiEl9qX4mILHE+JpY9V6nsvCKGjF7AgfxiPrmpsy7colxmypQpTJkyhQ4dOtCxY0fWrFnD+vXradu2LVOnTuWRRx7hzz//JCJCV8v0tC6JUSzcvI/CYoenQ1HKuzhKYNsiT0dRKby5zXZbj7aI+ALvAucC6cACEZlojFlV6rDXgc+MMZ+KyNnAf4DrnfvyjDHt3RWfqtryi0oY9lkKaXty+OSmZE5voAlPtXGcnufKYIzh0Ucf5bbbbjti36JFi5g0aRJPPPEEffv25amnnirnHVRl6ZIYxSezN7F8WzadGtf2dDhKeY/5I+HX4TDkZ0g4073X8nC77c1ttjt7tJOBDcaYNGNMIfAVcEmZY1oDvzufTy9nv1JHcDgMD4xbyryNe3n9ynb0aKa369WpCw8P58CBAwD069eP0aNHk5OTA8C2bdvYvXs327dvJyQkhOuuu46HHnqIRYsWHXGuqlydE6MAtMyfUqU5HDD/Q/t8zruejcVNqkqb7c4x2nHA1lKv04EuZY5ZClwG/B9wKRAuItHGmEwgSERSgGLgZWPM92UvICLDgGEAjRo1cv0nUF7HGMNzP63i5+U7ePz8VlzSPs7TIalqIjo6mh49etCmTRsGDBjANddcQ7du3QAICwvjiy++YMOGDTz00EP4+Pjg7+/PiBEjABg2bBj9+/enQYMGOhmyksWEBdKsThjzNmZyR++mng5HKe+wcQbsTYW6bWDtL5CZCtHV6/8P17TZ9Zn+2zTwcV86LKYiRYdP5o1FrgD6G2Nudb6+HuhijLm71DENgHeARGAmcDnQxhiTJSJxxphtItIE2+vd1xiTerTrJSUlmZSUFLd8FuU93v8jlZd/WcPNPRJ58sJWiFR84RnlvVavXk2rVq08HUalKe/zishCY0ySh0LyCFe12499t5yJS7az9OnzTmgxKqWqra+uhS1zYdh0eLsjJN0E57/m0ktUi3Z7/3bI3QN1Wx8z2T6VNtudQ0e2AQ1LvY53bvubMWa7MeYyY0wH4HHntiznz23On2nADKCDG2NVVcCERem8/MsaLjyjPk9coEm2UsrqkhhFTkExq3fs93QoSnle1lZYOwk63gCRjaDtFbD4S8jL8nRk3qWkGHIzIDDcrT3a7ky0FwDNRSRRRAKAwcBh1UNEJEZEDsXwKDDaub22iAQeOgboAZSeRKlqmJnrMnh4/DK6NYnmjUHt8NFeK6WUU7JznPbctEwPR6KUF1j4CRhje7EBut4JRbmw6FOPhuV1cjPAOCC8nlsv47ZE2xhTDNwNTAZWA+OMMStF5DkRudh5WG9grYisA+oCLzq3twJSRGQpdpLky2WqlagaZMW2bO74YiHN6oTxwQ2ddGGXaspdw9i8TU35nJWpfkQwjaJCdEKkUsWFNqE+rb/tzQaofwYk9IR5I20vrgtV2fbMUWIT7aAI8A8+5qGn+hndumCNMWYSMKnMtqdKPR8PjC/nvNlAW3fGpqqGLZkHGfLxAiJDAvj05mRqBfl7OiTlBkFBQWRmZhIdHV2thwQZY8jMzCQoKMjToVQ7yYlR/LZ6Fw6H0TtequZaPdEmkJ1vPXx71zvhq6th9Q/Q5nKXXKpKt9u5GWBKIOzYvdmuaLN1ZUjltTJzCrjx4/kUlTj4algX6tbS5KS6io+PJz09nYyMDE+H4nZBQUHEx8cf/0B1QpIToxi/MJ0NGTmcVjfc0+Eo5RkLPoLaidD07MO3n9YfoprAnPdclmhX2XbbOGD/DvANgOzNxz38VNtsTbSVVzpYWMzNn6awPSuPMUO70KyO/uGszvz9/UlMTPR0GKoK65oYDcC8tExNtFXNtGslbJkN570APmVGBvv4QJc74JeHYOt8aJh8ypersu327P/BlCfglqnQ0P1VU9y6BLtSJ6O4xMHdYxazPD2L/13dgU6NozwdklLKyzWMCqZerSDm6ThtVVMtGAV+QdD+2vL3t7/GjkmupgvYVEhRHvz1NiT2csmXjYrQHm3lcYXFDlZsz2bhpn0s2LSXRVv2sSenkBcvbcN5p7t3NrBSqnoQEZITo5iblokxpuqNGVXqVOTvh6Vf22EhIUfpnAoMg443wpx3IGvLP5Mla5JFn0PubjhrdKVdUhNtVemyDxaxcMteUjbtI2XTPpamZ1FQ7ACgcXQIZ50Wy3mt69K/TX0PR6qUqkq6NIli4tLtbMo8SGJMqKfDUaryLPvalvDrfMuxj+tym+3RnvcB9Hvx2MdWN8UF8Ndb0KgbJJxZaZfVRFu5lTGGLXsP2qR68z5SNu1l/e4cAPx8hNPjIriua2M6J9SmY+Pa1AnXCY9KqZPTxVlPe/7GTE20Vc1hjB020qADxHU69rER8dD6Elj0GfQebhdrqSmWjIH92+Di/0El3vHSRFu5RU5BMU/9sIKZ6/awJ6cAgPAgPzo1rs3ADnF0alybdvGRBAdoTWyllGs0jQ0jOjSAeRv3clXnGnhbXNVMm2ZBxhq45L2KHd/tLlg5wa4W2fV298bmLUqKYNZ/7ReRshVZ3EwTbeVy+UUlDP00hQWb9nJRuwYkJdQmqXEUzeuEaX1bpdTJ27MBpj0NF74JYXWO2H1onPa8NJ0QqWqQBaMgKBLaXFax4+OTID4Z5o2A5KHgUwM6vJZ/Y8elD3i1UnuzQauOKBcrLnHwr7GLmZOWyetXtuPNq9pzbZfGtKgXrkm2UuoUGVj7C/zxylGPSE6MYltWHun7DlZiXEp5yP4dsOYn6HDdcVc4PEy3O2HfJvv/U3XnKIE/34B6bW098UqmibZyGWMMj05YzpRVu3j24tMZ2CHO0yEppVxARPqLyFoR2SAiw49yzCARWSUiK0VkjFsCiWkOnW6EhZ9AZmq5hyQ7x2kv2KS92qoGWPQZOIoh6eYTO6/lRRDRCOZWcLiJuxljJ2lOftwmxq608jvI3ABnPVTpvdmgibZyEWMML01azTcL07nvnObc2D3B0yEppVxARHyBd4EBQGvgahFpXeaY5sCjQA9jzOnAfW4LqNdw8A2E354td3fLerWoFeTHfK2nraq7kiJY+DE07QvRTU/sXF8/6DIMNv8F25e4J76KMgamPgWTH7OlB3/+t93mCg4HzHwdYlvaLxceoIm2con3ZqTy4Z8bGdI9gXv7Nvd0OEop10kGNhhj0owxhcBXwCVljhkKvGuM2QdgjNnttmjC60L3u2HVD5CecsRuXx+hc4KO01Y1wNpf4MAO6HzryZ3f8QYICPNsr7bDAZMehNlv289x5v32jtVRvkifsDU/QcZq6PngkatlVhJNtNUp+3LeZl6bvJZLO8Tx1IWtdaEIpaqXOGBrqdfpzm2lnQacJiJ/ichcETnqQEgRGSYiKSKSkpGRcXIRdb8HQmNtL1g5PV/JiVGk7cll94H8k3t/paqCBR9CREM4rd/JnR8UAR2uhxXf2rHela2kGH64y07m7P4vOP916Pu0HQYz602Y9dapvb8xMPM1iGpa8YmibqCJtjolPy7dzhPfr6Bvyzq8esUZOuFRqZrJD2gO9AauBj4UkcjyDjTGjDTGJBljkmJjY0/uaoHh0OsRe9t73eQjdndpEg3Ago37Tu79lfJ2GWth40xIuunUqoZ0uc2OiV7woetiq4jiQvj2Flg6Bno/Buc+Z8dPi9iEu83ltsJQyscnf411k2HnMuj5gEcrq2iirU7aH+sy+Pe4JXRuHMW713bE31f/OSlVDW0DGpZ6He/cVlo6MNEYU2SM2Qiswybe7tNpCEQ1gWnPHDF56vQGtQgJ8GXexky3hqCUx6SMBh9/6HDDqb1PVCK0vMC+X2ElVeopyodx18Oq7+G8F6D3I4dPUvTxhUs/gObnwU/32x73E3WoNzuyEZwxyHWxnwTNjNRJWbh5L7d/vpDmdcIZNSSJIP8aUIdTqZppAdBcRBJFJAAYDEwsc8z32N5sRCQGO5Qkza1R+fpD36fs+MulYw/b5e/rQ6fGtXVCpKqeCnPtKoenD4Swk7wrVFq3uyBvHyz76tTf63gKc2HMIFj3K1zwXzsMrDy+/nDlp3a59AnDYP3UE7tO2nTYlmLHfPv6n3rcp8CtifbxSkKJSGMR+U1ElonIDBGJL7XvRhFZ73zc6M441YlZs3M/N328gLq1Avn05mRqBXn2H7FSyn2MMcXA3cBkYDUwzhizUkSeE5GLnYdNBjJFZBUwHXjIGOP+7uTWA+1Kb9NfgqK8w3YlJ0SxZucBsg4Wuj0MpSrV8m+gYP/JT4Isq1E3qN8e5o6wkxPdJT8bPr8MNv0JA9+Hzrcc+/iAELjmK6h7Onx9PWyeXfFr/fEa1IqD9teeWswu4LZEuyIloYDXgc+MMWcAzwH/cZ4bBTwNdMHOeH9aRGq7K1ZVcVsyD3L9R/MJCfDj81u6EBse6OmQlFJuZoyZZIw5zRjT1BjzonPbU8aYic7nxhjzb2NMa2NMW2NMJXSNYW83n/sc7N8G894/bNff47Q36ThtVY0YA/NHQd020LCLa95TxPZq71kHqb+55j3LOrgXPr3Y9jJf8TG0v7pi5wVFwHUTICIexlwFO5Ye/5xNs2DLbOhxL/h5PkdxZ492RUpCtQZ+dz6fXmp/P2CqMWavs1zUVKDyl/NRh9m9P5/rPppHUYmDz29JpmFUiKdDUkrVdAlnQvN+8Oeb9o+50xnxEQT4+TAvTcdpq2pk63zYtdz2ZruywlfrgRBe39axdrUDu+CTC2D3ahjsHPJyIkJj4IbvbdL9+WWwZ/2xj5/5GoTWseULvYA7E+2KlIRaChyquXIpEC4i0RU8V52AtIwc9uWe/C3U7INF3DB6PntyCvjkpmSa1w13YXRKKXUKznkGCg/YZZadgvx9ad8wkvm6QqSqThaMgsBa0PZK176vXwAkD4W0GbBrpeveNzsdPh4A+zbDtd+cfCnCiHi4/nv75eKzgZC1tfzjti6wn6H7PSe2JL0beXoy5INALxFZDPTCzmSv8NqbLqnHWgPM3rCHs9/4gw7PT6XDc1O4YsRsHh6/lA/+SGXaql2kZeRQXHL0cVkHC4u5+dMFpGXk8uENSbRvWG7VLqWU8oy6raHdNTB/pP2D7tQ1MYoV27LJKSj2YHBKAekL4bfnITP15N8jJ8NW6mh3NQSGuS62QzrdBH7BrlvAZm8ajB4AuRlw/XfQpNepvV9MMzuMpOAAfD7Q/j7KmvkqBEed+JL0buTnxvc+bkkoY8x2nD3aIhIGXG6MyRKRbThnsJc6d0bZCxhjRgIjAZKSkly0Xmf18/7MNGLCArm9VxNSM3JJzcjh9zW7GZeS/vcx/r5Co6gQmsaG0SQ2jKaxoTSJDaNRVAgPfrOUxVv28d61HenRLMaDn0QppY6iz2OwYjxMfxEuGwlAcmI0jt83kLJpL71b1PFwgKrGWjIGfrwXSgrtXZcWA6DrHZDQ88S4vA/mAAAgAElEQVSGfyz+3L7H8SYRnqyQKDt2evGX0PeZU6tokrHWjskuKYQbf4QG7V0TY/0z4Npxtlf7i8tgyE92SAnYpeTXT4Gzn3TPF5GT5M5E+++SUNgEezBwTekDnGWg9hpjHMCjwGjnrsnAS6UmQJ7n3K9O0NqdB5i5LoMHzzuNW3s2OWxf9sEiUvfkkOZMvtMyckjNyGX62t0UlRz+veWVy9vSv039ygxdKaUqLiIOutwOf/0fdLsb6p9Bx8aR+PkI8zdqoq08oKTYLroy5x1IPAsGvAorJkDKR7B2EtRtaxPutlccf9Keo8Qu3pLQE2JbuC/mrnfamtopH0HvI4rFVcyOZfD5pbYe9pCf7R0nV2rUFa76AsYOthMkr5tgK5TMfM0m3clDXXu9U+S2RNsYUywih0pC+QKjD5WEAlKcs9V7A/8REQPMBO5ynrtXRJ7HJusAzxljdKDdSfhoVhpB/j5c26XxEfsiQvzp2Kg2HRsdXtCluMTB1n15zsQ7h8SYMM5tXbeyQlZKqZNz5v2w6FOb3Fz/HSEBfrSNj9B62qry5e2D8TdD6u+QfBv0e9HWcz77cej5b1uib+4I+OFO+++18612uEPYUb4Qrp8C2VvgvOfdG3dMc7tQzIJR0OM+8A868pjiAjiwAw7shP3b7fP92+3rAztsZZCgSLhxIkQ3dU+czc+xd67G3wzjboCzn4A1P9kVYw/1cHsJMaZ6jLhISkoyKSkpng7Dq2QcKKDHy78zqHM8Lwxs6+lwlFLHICILjTFJno6jMrml3Z79Dkx53E6catqH//yymtGzNrLs6X4EB+jCWl4nPcVWi2l+rmuraHhSxjrb25q1BS54AzodZSkQY+zEvbkjYP1k8A2AtoOg6+1Qr8zf7C8ut5MU71vu/gVYUqfbMdCdb4WQGGdSvQP274AD2+FgOZV8/IJs1ZJaDSCyMfR51K7K6G4LP7HDcgLCAWN/PyFR7r8uFW+z3Tl0RHnY53M3U+RwcHOPRE+HopRSlSN5KMz7AKY+BYl/0CUxig/+SGPx1n10b6pzTLzKjqXw6UVQdNDWhD73eWjkotrQnrJuCnx7ix0KcuOP0Ljb0Y8VgaZ97GPPelsLfskYWPKFHSLS7S5bujJrE2yYBr0frZxVDpv0hnpn2F5tBEJjoVZ9W/kjPskm0+H17bZw5yO4tme+KHUaYhfCmfqUvaNVSUn2idBEu5rKLyrhi7mb6duyLk1ivWdSgFJKuZVfoL2N/N0wWPEtSacNRATmb9yribY3ObATxl5tK0T0fRpmvQmjz4OWF9pyjTHNPR3hiTHGzg+Y9gzUawODx0Jkw+Oe9reY5rb3++wnYNFnMG+k7RWPamJXOBRf6FhJi2SLwE2TIC8Lwut5fAnz4+pxr/1iUvYugJfwdHk/5SYTFm1jb24ht/bU3mylVA3T9kr7R/f356jl56B1/Vo6TtubFOXBV9fYRO6ar+xQiX8tgj5P2KEU73aBn/4NObs9HWnFFOXBhGF2rHXrS+DmySeWZJcWXNsmjvcusSsohkTbJctbX2x7kCtLYLj9DN6eZB8S19FrY9Ue7WrI4TCMmpVG27gIuiR6320UpZRyKx8fOOdZW/5rwUckJ/Zi7PwtFBY7CPDT/iWPMgZ+uAu2LbKVIw71QgaEQq+H7FCAP16BhR/Dsq+h+7/sEApXlWvLTrcTFPP22VVF67e31TFO1v7t9kvD9sW2N7rng64ZQuHrD20us49dq2xVHVUlaaJdDc1Yt5u0jFz+b3B7pLpMLlFKqRPRrK8dazrzNXqcdw4f/+Vg+bYsOjXWzgeP+uNVWPGtHR7S6sIj94fFwgWv21KNvz0LM176p9RchxvA9wTTlsKDsPkvm1xv+A32rD18f1CEHXbQpDc06WOrZFT07+bWBfD1tVCYa5cWb3nBicVWUa4uj6cqlSba1dCHMzdSPyKI89tq3WulVA12zrMwshfdd34BdGHexr2aaHvSigk2cW53jS0ddywxzeCqz2HrfDvR7af7bXWOc56BFucfPRk2xlbnSP3NJteb50BJga2K0bg7dLwBmp79z5CMtOmQ9octDQdQK96uYNikNyT2gvCjlLY9tAhNrQa2wo0mw+ooNNGuZlZsy2ZOWiaPDmiJv6/eIlVK1WAN2kPbKwlZOJKuMe2Zk5rJnb2beTqqmmnbQvj+DmjUDS56q+K9xg2T4aZf7AIvU5+2wzQadbMVShp2tsfkZNix3YeS65xddntsK1uFpunZNsn2Dz78vdteYR/G2OXC02bYx5qfYcmX/7xHk972kdDDLlFeehGaKz/1ykoXyntool3NfDRrI6EBvgxOroT6lUop5e3OfgJW/cDjId9x0frBpGzaS1KCJkaVKnsbjL3GLsZy1RfHXwWxLBE7LKN5P1j8GUz/D3x0DjQ7F3J32zKBYCcSNuljhw01Pdv2Nlf0/aOb2kfnW+wqjDuX/ZN4L/wY5o2wlT9qxdmFY0ovQqPUMWiiXY3szM7nx6Xbub5bYyKC9X9+pZSidgJ0vpU2896ne3hfnvh+BT/ec6be8asshbm2TF1hLtzwPYSeQolFXz+7emLbQTDnXVvnObqZrVbS7OxTn9h4iI8vNOhgH2feD0X5sHWeTbq3LYReD0PH60/9OqpG0ES7Gvlk9iYcxugCNUopVVrPB5HFX/BO+BeclX4bn/y1iaFnNfF0VNWfwwHf3Qa7VsA146BOK9e8b2AY9H7EPiqDf5Bz3Havyrmeqlb0K301kVtQzJh5m+nfph4No0I8HY5SSnmP0GgY8Cq1MxcxJfw5vp82ne1ZeZ6Oqvqb/gKs/hHOe9Eusa5UDaSJdjXxTcpW9ucXc2tP7aVRSqkjtL8aueEH6vod5Gt5nB/GfuDpiKq3pV/Bn2/Y1Qy73uHpaJTyGE20q4ESh2H0X5vo2CiSjo1qezocpZTyTok98b19Jjm1mnLHrqfZ9PXDduKbcq0t82DiPbY+9QVvuGYBF6WqKE20q4Gpq3ayZe9BhmpvtlJKHVtEHFF3TeMn/34krP6Aks8vg4O6PLvL7NtsS/BFxMOgz7Qqh6rxNNGuBkb9uZGGUcGcd3o9T4eilFJeLyAomNirR/BI0VDMpr/gg16wfYmnw6r6Cg7YCiMlRXbyo9aXVkoT7apu8ZZ9pGzex809EvH10dtzSilVEV2aRFPc7noGFT1NUXERjO5nV/vzBkX5dtXBJWM9HUnFOUrg21shYy0M+hRimns6IqW8gibaVdyoWRsJD/LjyqSGng5FKaWqlMfOb0mqfwvuDH0TE9/Zrlz484NQXOi5oBwlMGEoLPwEvr8dfnnE9hB7u6lPwbpf4fxXoWkfT0ejlNdwa6ItIv1FZK2IbBCR4eXsbyQi00VksYgsE5HzndsTRCRPRJY4H++7M86qauveg/yyfAfXdGlEWKCWRFdKuUcF2vIhIpJRqs2+1RNxnqjosECGD2jJ1C0Ovjv9Heh+Dyz4ED69EPbvqPyAjIHJj8HqiXaJ8a53wbz34QsvHke+f7vtyZ7zDiQPg85V4j+9UpXGbYm2iPgC7wIDgNbA1SLSusxhTwDjjDEdgMHAe6X2pRpj2jsft7srzqrsk9mb8BFhSPcET4eilKqmKtiWA3xdqs0eValBnoKrkhrSsVEkL/y6nqwzn4IrPoadK2BkL9g8p3KDmfOOTay73gU9/gX9X4KBI2wVj5G9YdfKyo3nWIoLYNab8L8kWDURznoY+v3H01Ep5XXc2aOdDGwwxqQZYwqBr4BLyhxjgFrO5xHAdjfGU63szy/i6wVbufCM+tSPCPZ0OEqp6qsibXmV5eMjvDCwLdl5Rbzy61pocxncOg0CwmzP9rwPbE+zuy0fD1OegNMvhfNe+Gd7+2vgpkk2sR11Lqz6wf2xHM/6qfBeN5j2jF0t8a55cPbjdol0pdRh3JloxwFbS71Od24r7RngOhFJByYB95Tal+gcUvKHiPQs7wIiMkxEUkQkJSMjw4Whe7+v528lp0AXqFFKuV1F2nKAy51DAMeLyFEnjXhju926QS1u6p7A2PlbWLh5H9RtDUN/h2bnwi8P22XECw+6L4CNf9rx4Y17wMD3wafMn+b4JBg2wy5hPu4GmP6SXd68su1NgzGD4csrbG3sa7+Fq8dCVGLlx6JUFeHpyZBXA58YY+KB84HPRcQH2AE0cg4p+TcwRkRqlT3ZGDPSGJNkjEmKjY2t1MA9qbjEwcd/baRrkyjaxEV4OhyllPoRSDDGnAFMBT492oHe2m7fd+5p1KsVxBPfr6C4xAHBkTB4DPR5HJaNg1F9IWOd6y+8ayV8dS1ENYHBX4J/UPnH1aoPQ36GdtfAH6/A19fZcnqVofAg/P4CvNsVNs6Ec56FO+ZA83Mq5/pKVWHuTLS3AaV7NeKd20q7BRgHYIyZAwQBMcaYAmNMpnP7QiAVOM2NsVYpk1bsZHt2Preeqb3ZSim3O25bbozJNMYUOF+OAjpVUmwuExboxzMXt2b1jv18MnuT3ejjA70ehuvGQ84uO0562Teuu2j2NvjySggIgWvHQ/BxVvb1D4KB70H/l22Fj1Hn2l5mdzEGVn4H73SGma9B64vhnhQ48z7wC3DfdZWqRtyZaC8AmotIoogEYCc7TixzzBagL4CItMIm2hkiEuucgIOINAGaA25sTaoOYwyj/kyjSUwoZ7es4+lwlFLV33HbchGpX+rlxcDqSozPZfqdXo8+LWJ5c+o6dmTn/bOj2Tlw259Q/wyYcKutcV2Uf2oXy8+2SXb+frj2G4isYIlWEeh6B1z3LRzYASP7QOr0U4ulPLtXw6cXwTdD7BeAm36By0dBrQauv5ZS1ZjbEm1jTDFwNzAZ2+iOM8asFJHnRORi52EPAENFZCkwFhhijDHAWcAyEVkCjAduN8Z4aW2jyrVg0z6WpWdz85mJ+OgCNUopN6tgW/4vEVnpbMv/BQzxTLSnRkR49uI2FDsMz/246vCdEXFw44/Q415b4/qjcyAz9eQuVFxgh4vsWQuDv4B6bU/8PZr2gWHTIby+Lf835z3XTNrMy4JfhsOIHrBzOZz/uh0f3rj7qb+3UjWQmMqYTV0JkpKSTEpKiqfDcLthn6WwYNNeZg/vS3CAr6fDUUq5iIgsNMYkeTqOyuSt7fa70zfw2uS1fHxTZ/q0KOfO4dpf7QRJRwlc8j9bKaSiHA67IM2K8XDpSGh31akFW3AAvrsd1vxkx29f+ObRx3mXVVxge8X3b7ePvRth/geQuwc6DYGzn4TQ6FOLT6lqqqJtttbiqUI27sll6upd3N2nmSbZSinlJkN7NmHConSe+mEFU+/vRZB/mfa2RX+4fRaMv8kOrdg825bk8ws8/ptPe9om2X2fPvUkGyAwHAZ9DjNfhRn/sb3kV30JgWF20Z3922wSfWD7Pwn1ocfBPUe+X8Mudrx4g/anHptSShPtquSjWWn4+/hwfbfGng5FKaWqrQA/H54f2IZrPpzHO79v4MF+LY48KLIhDJlka0nPfRfSF8CVn0DthKO/8bwPYPbbdvXEM+93XcA+PtB7ONQ9HSbcBm+2BlNO+b/gKKgVZ8dZx3X853mtBhDewFY2CdJKVkq5kibaVcSiLfsYM28LVyc3ok54BW8LKqWUOindm8ZwWYc4PpiZysAOcTSrE3bkQX4BdvXGxt3h+zvh/bNsVZBWFx557KqJ8Msj0PJCGPCqndToaq0ugqHNYMkYCIkqk0jXB39d3EypyubpOtqqAvIKS3hw3FLqRwQzfEBLT4ejlFI1wmMXtCLY35cnv1/BMecztboQbvvDLtzy9bXw62NQXPjP/i1z7bjs+M62coePG4f+1WkF5z1ve8zPGAQJZ9oa3ZpkK+URmmhXAa/8uoa0Pbm8duUZhAf5ezocpZSqEWLCAnm4f0vmpGXy7aKyy0CUEZUIt0yB5GF2KMnHAyBrq13kZsxVtnf56q804VWqhtFE28vN3rCHT2ZvYkj3BLo3jfF0OEopVaNck9yIzgm1ee7HlezMPk7tbL9AOP81O1Y7Yy28fyZ8fin4+tu611rBQ6kaRxNtL7Y/v4iHxi+jSUwoj/TXISNKKVXZfHyE165oR2GJg0cnLDv2EJJDTr/UDiWJbAh5++CacbbHWylV42ii7cWe/3EVO7LzeH1QOy3np5RSHpLg7OyYvjaDbxamV+yk6KYwdDrcv8JW+FBK1UiaaHupaat28c3CdO7o3ZSOjWp7OhyllKrRbuyWQHJiFM//uIrtWXnHPwHskJGQKPcGppTyappoe6G9uYUMn7CclvXCubfvaZ4ORymlajwfH+H1K9pRYgyPfFvBISRKqRpPE20vY4zhye9XkJ1XyJtXtSfAT/8TKaWUN2gUHcKjA1ry5/o9fLVgq6fDUUpVAbpgjZeZuHQ7Py/fwUP9WtCqfi1Ph6OqOmPgwE7YvRJ2r4ZdqyBrMyT2gk43Qng9T0eoVJVybZfG/LJiJy/8tIqezWOIrx3i6ZCUUl5ME20vsmt/Pk/9sJIOjSK57awmng5HVdSeDZC91dbH9Qty/gwEv2DwD7I/ff3dsxJcaXlZNpnevcr5WA27VkJ+1j/HhNW1yfWMl2Dmq9DqYkgeCo26uSe+3D2w6nvY8Bs06wtJt7j/96CUG/n4CK9cfgb935rJw+OX8cUtXfDx0X/TSqnyaaLtJYxz3F9BcQlvXNkOP18dMuL1ti+Gma/Dmp+Of6z4lEq8g0ol5KV/BlXgmFI/D+49PKneX2pBjcBadoW40wdCndPt8zqt/6njm5kKCz6CJV/Aygn2mM63wBlXQWA5S02fiPz9sOZnWDEeUqeDKYGQGFg7CdZMgkvehVr1T+0aSnlQw6gQHr+gNY99t5wv52/h+q6NPR2SUspLVSjRFpFLgd+NMdnO15FAb2PM9+4Mrib5asFWZqzN4JmLWtMk9hQTnarIGMjaAptnw+ZZkJsJiT2haV+IbeFdvaBb58Mfr8KGqRAUAb2GQ+JZUJz/z6MoH4rzyvzMh6K8cn4WwME95Z9TUnDsWHwDIKYFNO4BdVvbZLpOa4iIP/bvLLop9H8Jzn4Cln8DCz6En/8N056BdlfbpDu2RcV/J0V5sH4KLB9vfxbnQ0Qj6H4PtL0C6raBBaNgypMwohtc+KatNexOWVshIFSrPpQhIvcCHwMHgFFAB2C4MWaKRwOrYq5ObsgvK3bwn0mr6dU8lkbROoREKXUkqcjMaRFZYoxpX2bbYmNMB7dFdoKSkpJMSkqKp8M4KVv3HqT/WzNp1zCy5tyGNAYyN8Dmv2DTXzbB3u+sTxsUaZOjvWn2da04aHo2NDsHmvSCYA+UOzQGNs2ywy02zoSQaOh2F3S+1Sbb7uJwlEre8w5P5APDIKop+LrgxpQxkL4A5n9oh3qUFNovD52HQovzy79GSRGkzYAV38Lqn6DwAITG2gS6zRXQMPnIZH/PepgwDLYvsr3nA16F4MhTj/+wa2ywQ2NWTLC9/+2vhq53QUwz117HxURkoTEmqRKus9QY005E+gG3AU8CnxtjKr3Yc1VutwG2Z+XR782ZtG5Qi7FDu9aMtlspBVS8za7oX+jyxjHosBMXcDgMD3yzFBHh1SvOqL4NtcNhhzgc6rHePBtyM+y+0DrQuDsk3Gd/xrYCHx/bw73hN0j9DVb9AIs/t0Mw4pLseN9m50CDDuDjxsV8jLHXn/k6bJljxzif9yIk3WR7S93NxwcCQuzDnURsYtwwGfq9BIs/g5SPYdz1EN7Aft6ON9pEesscOyxk1Q9wMBMCI+D0S2xyndDz2Il/THO4ZQr8+Ya9K7DpL7h0hE3qT1XWVvjjFVgyxibYZ95n41v8hf0sLS+wPewNu3jXHZLKd+jDn49NsFeK1OxfyMlqEBnMkxe25uFvl/HZnE0M6aGrPyqlDlfRHu3RQBbwrnPTXUCUMWbIcc7rD/wf4AuMMsa8XGZ/I+BTINJ5zHBjzCTnvkeBW4AS4F/GmMnHulZV7RkZ9WcaL/y8mlevOINBSQ09HY7rFBfAzhWwZbZNprbM+WdSXq14SOhhk+rGZ9phDMf7O19SDNtSYMM0m3xvXwwY27vdpLcdYtKsL9Rq4Jr4jYG1v8DM12zva614m7h1uM6Oj64JHCWw7lc75CP1d/Dxtz35OTvtWPIWA+ywkGbn2MmfJyp9IUwYCntTodvdcPaTdnz6icrZbRP3lNH2ddIt0PPfEFbHvj6wyw6NWTDKLocdl2QT7lYXufdL2gmqxB7tj4E4IBFoh217ZxhjOrn72mVV1Xa7NGMMN32ygLlpmfx671kkxFTCF3CllMdVtM2uaKIdir29eA5ggKnAi8aY3GOc4wusA84F0oEFwNXGmFWljhkJLDbGjBCR1sAkY0yC8/lYIBloAEwDTjPGlBztelWxwd6w+wDnvz2Ls5rH8OENSVTZTiVHiR0SsG2hTUq3LYJdK+zwA7DDGxp3t+OIG3eH2i6YOJSbCWnTbQK44Teb/IHtDa97up1sF97AVtio1QDC69vnx0sIHQ5Y/QPMfAN2LYfIxjZpa3cN+AWcetxV1Z4NNpHdnw4tL7JJ9qlOmgQozIWpT9kkOLYVXDYS6p9RsXPz9sFfb8O89+0Xuw7XwlkPQ+RRvrAW5tre7rnv2WFJkY3t8J/217rms5yiSky0fYD2QJoxJktEooB4Y8wyd1+7rKrYbpdnZ3Y+5775By3qhvP1bd3wra53JpVSf3Npon2SAXQDnjHG9HO+fhTAGPOfUsd8gG3sX3Ee/4YxpnvZY0VksvO95hztelWtwS4qcXD5iNls3XuQyfefRZ3wk+jJ8wRjbCm7bYucifVi+yjMsfsDwqFBezukI64jNOzq/goTxtgydqm/2THDe9Ng/47yJxKGRJdKwMsk4wczYdZbsGctRDeHng9A2ytdMwZaHdv6qfDDXbaSytmPQ/d/Hb23uSAH5o2Av/4HBfuhzeXQ5zF7Z6QiHCW2Asrs/8HWeXZOQNLN0OU2j9YVr8REuwewxBiTKyLXAR2B/zPGbHb3tcuqau32sXy7MJ0HvlnKExe04taeWp5VqerOpWO0RWQqcKUxJsv5ujbw1aEk+ijigNJLZ6UDXcoc8wwwRUTuAUKxPeaHzp1b5ty4cuIaBgwDaNSoUUU+itcYMSOVZenZvHtNR+9OsvP322Rk28J/kuuDe+w+3wCo19ZWqYjrCHGdbILqU8mlCUWgXhv76HGv3WaM7fE8sMM+9u8o83w77Fxmhx1Q6stmndZwxWhoPdCrhhVUe83PhTvnwo/32son6ybDwBEQVWrMa1E+pHwEf/7X/htscT70edz+dz8RPr522Eiri2wFmdn/g1lvwpx3oO0g28tdt/Xx38eY8qvM+AVWPOn3jBFAOxFpBzyArTzyGdDLo1FVcZd1jGPS8h28NnktfVrWoWlNrB6llDpCRbvqYg4l2QDGmH0iUscF178a+MQY84azR/tzEanwX01jzEhgJNieERfEUylWbMvm7d/Wc3G7BlxwhhfWE3Y4bDWQJV/Cyu9tAoFAbEs4rZ9Nqht0tEM0TmZsbmUQsZVLQqJsnEdTUgQ5u+zqiSVFdqJcZX9RUFZIFAz6DJZ9DZMegvfPhP4vQ7vBdkLjzNdsrfDEXtD3KYh3Qedvw2S46nN7F2TOe/bf/JIv7AI+/iHHKMnoTLDL06QP3ODVlU+LjTFGRC4B3jHGfCQit3g6qKpORHjpsrac9+ZMHvxmKeNv765DSJRSFU60HSLSyBizBUBEEjisG7Bc24DSgyXjndtKuwXoD2CMmSMiQUBMBc+tkvKLSvj3uCVEhQbw3CXHSAA9ITsdloy1ica+TXbRk3aDoc1ldihIYLinI3Q9X39bczoi3tORKLBfkNoNtuP5v78DJt4NU56wE2njO9te7iZu6HiNagIXvG6HoKR8ZMsVlhTaSZ8hUcdYPKjMCqD+QXY4knc74Byedz3Q0zlm29/DMVULdWsF8ezFp3Pf10sY9Wcat/Xy6jsbSqlKUNFE+3Fgloj8gS0N1RPnkI1jWAA0F5FEbJI8GLimzDFbgL7AJyLSCggCMoCJwBgR+S92MmRzYH4FY/Vq3yxMZ92uHEYPSSIyxAsm1xUX2FX8Fn9hJxZibKm13o/ZW+vuLiunVHkiG8INE+3ExXW/2qokp/Vzf1m+kCg46yH7qL6uwrbFNxtjdjqrP73m4ZiqjUvaN2DS8h28MXUdZ7esQ/O61bCDQilVYRW6R26M+RVIAtZiq4E8AOQd55xi4G5gMrAaGOes1/qciFzsPOwBYKiILHW+7xBjrQTGAauAX4G7jlVxpCoZn7KVlvXC6dPCFSNvTsGOpfb2/BstYPxNkLEWej0M9y6FG3+Edldpkq08y8cHut8NQ36CFv1reu1rlzHG7AS+BCJE5EIg3xjz2fHOE5H+IrJWRDaIyPBjHHe5iBgRcfvETm8kIrx4aVtCA3x58JulFJc4PB2SUsqDKjoZ8lbgXuwQjiVAV2AOcPaxznPWxJ5UZttTpZ6vAnoc5dwXgRcrEl9VsW7XAZamZ/PEBa08U8rv4F673Pbiz2HncvANhFYX2trQib108p9SNYCIDML2YM/A3qH8n4g8ZIwZf4xzfLHrKPxdrlVEJpYu1+o8Lhz7t2Kem8KvEmLDA3nukjbcM3YxH8xM464+3r0qqVLKfSo6dOReoDMw1xjTR0RaAi+5L6zq6duF6fj5CAM7HFFA5dQ4HFCQDXlZttJG6Ud+lt2+bxOsn2LHndZvB+e/bsuihUS5NhallLd7HOhsjNkNICKx2LUKjppoY9c02GCMSXOe8xVwCfauY2nPA68A1XrsTUVceEZ9flmxg9enrAXgzt5Nq+5aCUqpk1bRRDvfGJMvIohIoDFmjYi0cGtk1UxxiYMJi7fRp2UdYsJOslLHyu/sJK2/E+hDCXUWx5yb6h8KodF2xbwO19qSfEqpmsrnUNQ/61cAACAASURBVJLtlMnxhxEet1yriHQEGhpjfhaRoybaVbks64kQEd64sj1+Pst4bfJaVmzL5rUr2xEWqHX5lapJKvp/fLqIRALfA1NFZB9Q6YsbVGUz12eQcaCAKzqdZHWLLfNg/C0QGmsXVwmuDbUT7c/g2hAcWeq58xEUabd7awk+pZQn/OpcBGys8/VVlBnid6KclUv+Cww53rFVtSzryQgO8OX/BrenbVwE//llNRt25zDyhiQS/7+9O4+Pqjz7P/65sofsIQESQoCwo6yGHXcruBSq1SrWllq1q91sbe3z+Fh/9unzs8tjrZW2WuWnrW3dF6oIbqhFBEHKDoEkICSsAUxIIPv9+2MmOsSEBMjJTDLf9+s1r5y5z33mXDmTubk4c51za5p2kbDRrkTbOXeFf/EuM1sCpOC7SFHa6ZkPSkhPiDm1iyCry+HZm3y3oPvGUohL7vgARSQsOOduM7PP88n1MQ85555vY7O2brmaBJwJvOUvj+gDLDCzWc657jH14ykyM24+J48RWcl85x+rmfXAUu6fMy74F8SLSKc46Zk5nHNvO+cWOOdqvQioOzpcVcvrm/Yze2w2MVEnecidg5d+4Juo4/OPKMkWkdPmnHvWOXer/9FWkg0Bt2s1sxh8t2tdEPB65c65DOfcAOfcAHwz+4Z9kh1o+pAMFtwynX5pPfjqoyuZt6QQ57r1CX0R4RQSbTl5/1y3m9qGRq4+q1/bnZtb+wRseBbO/yn0m9DxwYlIWDCzI2ZW0cLjiJlVnGjbdt6uVdrQL70Hz35zKrPGZPPrxQV88/HVVNbUBzssEfGQrsroBE+vKmFkVjIjs0/ybPTBIlj4I+g/Habf6k1wIhIWnHOnNXNKW7drbdZ+3unsqzuLj4nkvmt8ddv/s3AzV8xT3bZId6Yz2h7bsreC9aXlJ38RZH2try47IgqufFD3uBYR6SbMjJvOzuPxGydRVlnDrAeWsmTL/rY3FJEuR4m2x5runT17bPbJbfjW/8Du1TDrft9FkCIi0q1MHRxQt/3YSh54c5vqtkW6GSXaHqpraOT5f+/mwhG96Hky984ufhuW3gfj58LI2d4FKCIiQdVUtz17TDa/eXWr6rZFuhkl2h56Z+sByipruOpkLoI8egie/zr0HAwz/693wYmISEiIj4nkt9eM5Y7LRvDa5n1cMe9dtpdVBTssEekAuhjSQ0+vKqFnQgznDcts3wbOwYLvQFUZXPckxOjiGBGRcNBUtz0yK5lb/vFvZj2wlDkTcxneJ4lhfZIY3CuR2ChdqyPS1SjR9sihqlre2LKPL08ZQHRkO784+OD/wZaX4OJfQNYYbwMUEZGQ46vbnsZPnl3Ho+/uoLahEYDICCMvI4FhfZL8yXcyw/skkZMWj3+SIBEJQUq0PbJgTSl1Da79dxvZvwUW/QcMugAmf8vb4EREJGTlpPXgbzdNpr6hkR0Hq9iy9wgFe4+wec8R1pZ8xEvr9nzcNzE2iqG9Ez9OvIf1SWJkdjLJcdFB/A1EpIkSbY88s7qEM/smMyKrHffOrquGZ2/0lYp87k8QodJ5EZFwFxUZweBeSQzulcTloz9pr6ypZ+s+X/K9ZU8FW/Ye4ZUNe/jH+zsBSImP5uXvTicnrUeQIheRJkq0PbB5TwUbSiu467Mj27fB63fBvg1w3dOQ1NvT2EREpGtLjI1ifG4a43PTPm5zzrH/SA3rS8r51t9Xc9/r2/jN1SpBFAk2T0+dmtlMMysws0Izu72F9b81szX+x1Yz+yhgXUPAugVextnRnvmghOhIY9bYvm133voqrPgjTPoGDL3Y++BERKTbMTN6J8dx0cjefGXqAJ5bXcLWfUeCHZZI2PMs0TazSGAecAkwEphjZsed4nXO/cA5N9Y5Nxb4PfBcwOpjTeucc7O8irOj1TU08sK/S7lweG/SE2JO3LlyP7z4Leh1Blz0fzonQBER6da+ee4gEmKi+M3igmCHIhL2vDyjPREodM4VO+dqgSeAE82+Mgf4h4fxdIq3Cg5wsKqWq/PbuAiysRGe/wbUHIGrHoHouM4JUEREurW0hBi+fm4er27ax+qdh4MdjkhY8zLR7gvsCnhe4m/7FDPrDwwE3gxojjOzVWa23Mw+18p2X/P3WXXgwIGOivu0PPPBLjISYzlnaBv3zl7xRyh6A2b8AnqN6JzgREQkLNwwbSAZibH88pUtmtZdJIhC5fYW1wLPOOcaAtr6O+fygeuA+8xsUPONnHMPOefynXP5mZntnBTGQwcra3hj836uGJd94ntn71kLr/0Mhl0G+Td2XoAiIhIWEmKj+M4Fg1mx/RDvbCsLdjgiYcvLRLsUCJx7PMff1pJraVY24pwr9f8sBt4CxnV8iB3rxTW7qW90J55yvboCnr0JEjJg1u9BEw2IiIgH5kzMJSctnl8t2kJjo85qiwSDl4n2SmCImQ00sxh8yfSn7h5iZsOBNOC9gLY0M4v1L2cA04BNHsbaIZ75oITROSkM65PUcoctL8MfJsPBQrjiT5DQs3MDFBGRsBETFcEPLx7Kxt0VvLx+T9sbiEiH8yzRds7VA7cAi4HNwFPOuY1mdreZBd5F5FrgCXd8EdkIYJWZrQWWAPc450I60d64u5xNeypangmyvBSe+CI8cR3EpcANiyDvvM4OUUREwsysMX0Z3ieJ/321gDr/dO4i0nk8nbDGObcQWNis7c5mz+9qYbtlwCgvY+toz3xQQkxkBJ8dnf1JY2MDvP9nePPnvuWL7oIpt0CkpsYVERHvRUYYt80Yxo2PreLpVSVcNyk32CGJhBXNDNkBausbeXHNbi4a2Yu0pntn714D//we7FkDgy6Ey/4X0gcGN1AREQk7FwzvRX7/NH73xlauGNeX+JjIYIckEjZC5a4jXdqSgv0cqqrl6rP6QU0lLPoP+PP5ULEbrpoP1z+rJFtERILCzPjJJcPZV1HDY+/tCHY4ImFFiXYHeOaDEjKTYjmn8X2YNwmWz4Pxc+GWlXDm53VnERERCaoJA9I5f1gmf1hSSPnRumCHIxI2lGifprLKGjZv2czjifcT+dQXITYJvvoqfPY+iE8NdngiIiIA3DZjOBXV9Tz4TlGwQxEJG0q0T0djA9sW/JpF0T9iSMVyuPBn8PV3IHdSsCMTERE5zsjsZGaPzWb+u9vZX1Ed7HBEwoIS7VO1dz3u4QuZsvXXbIsZScS3l8PZt0JUTLAjExERadGtnxlKfYPj/je3BTsUkbCgRPtUNNTDX6+g4fAuvlt7Cxsu+H+QnhfsqEREPGNmM82swMwKzez2FtZ/w8zWm9kaM1tqZiODEaecWP+eCcyZmMsT7+9iR1lVsMMR6faUaJ+KXcuh6gDP9fk+i2w6s8b0DXZEIiKeMbNIYB5wCTASmNNCIv1359wo59xY4FfAvZ0cprTTdy4YTHRkBPe+tjXYoYh0e0q0T0XBK7jIGH63I5fPnNGblB6agEZEurWJQKFzrtg5Vws8AcwO7OCcqwh4mgAEzvYrIaRXchxfnT6ABWt3s3F3ebDDEenWlGifioJXOJgxgdJjUS1PuS4i0r30BXYFPC/xtx3HzL5tZkX4zmh/t6UXMrOvmdkqM1t14MABT4KVtn3tnEGkxEfzm8UFwQ5FpFtTon2yyrbBoSIW14+nd3Is5wzJDHZEIiIhwTk3zzk3CPgJcEcrfR5yzuU75/IzMzV+BktKfDTfPG8QSwoOsKL4YLDDEem2lGifrIKFAPx571AuH51NZIQmoxGRbq8U6BfwPMff1pongM95GpGctrlTBtA7OZZfLS7AOVX6iHhBifbJKlhEZdoIdjT0ZPqQjGBHIyLSGVYCQ8xsoJnFANcCCwI7mNmQgKeXAbp/XIiLj4nkexcO5YMPD/PG5v3BDkekW1KifTKOHoJdy1nXYwpREcaEAenBjkhExHPOuXrgFmAxsBl4yjm30czuNrNZ/m63mNlGM1sD3ArMDVK4chKuzs9hYEYCv15cQEOjzmqLdLSoYAfQpWx7FVwjzx8dzZh+qSTG6vCJSHhwzi0EFjZruzNg+XudHpSctujICH548VBu+fu/eXFNKVeO1wX+Ih1JZ7RPRsFCGhN78/y+TKbk9Qx2NCIiIqft0jOzOLNvMve+tpWa+oZghyPSreiUbHvV10Dhm+zNuZT6MmPqICXaIiLS9UVEGLfNGM7c+e/z5UfeJyU+mvpGR11DI3UNjdQ3NC076ht9P5va6xsbqa1vJDE2inuvGctknYQSOY6nibaZzQR+B0QCDzvn7mm2/rfA+f6nPYBezrlU/7q5fHJ7qP92zj3mZaxt2rEUao/wr8h8YqIiGN8/LajhiIiIdJRzhmRw1Vk5rN55mPJjdURFGlEREcRERhATFUGP2Chi/G1RkUZMpO9nVKSvzztbD3DTY6v4+82TGJ2TGuxfRyRkeJZoB0zZ+xl8kxusNLMFzrlNTX2ccz8I6P8dYJx/OR34GZCPb3axD/zbHvYq3jYVvAJR8TxRlsdZuYnERUcGLRQREZGOZGb85uoxp7z93vJqrn5wGV+e/z5PfX0KQ3sndWB0Il2XlzXabU7Z28wc4B/+5RnAa865Q/7k+jVgpoexnphzsHURdQPOZc3eGqaobERERORjfVLi+NuNk4mJjOD6h1ew8+DRYIckEhK8TLTbNWUvgJn1BwYCb57Mtp02le++DVC+i4KU6TiH6rNFRESaye3Zg8dvmkRtQyNffGQ5e8urgx2SSNCFyl1HrgWecc6d1OXOnTaVb8EiAF6pGU2PmEjVn4mIiLRgaO8kHrthIoer6rj+kRUcqqoNdkgiQeVlon0yU/ZeyydlIye7rfcKFkLffF7dCfkD0omJCpX/n4iIiISWMf1SeXhuPrsOHWXu/PepqK4LdkgiQeNlxtjmlL0AZjYcSAPeC2heDFxsZmlmlgZc7G/rfEf2wu7VVA64iG37K1U2IiIi0obJeT354/Xj2byngpseXcWxWt2fW8KTZ4l2O6fsBV8C/oRzzgVsewj4Ob5kfSVwt7+t8231lY2sjJ0CqD5bRESkPS4Y3pvfXjOWlR8e4huPf0BtfWOwQxLpdJ7eR7utKXv9z+9qZdv5wHzPgmuvgkWQksurB9JIiqvhjOyUYEckIiLSJXx2TDaVNfX89Ln1/ODJNdw/ZxyRERbssEQ6jWaGPJHao1C8BMbPZdnGQ0wa2FMDhIiIyEmYMzGXyup6frFwM4mxUdzz+VGY6d9SCQ9KtE9k+9tQX01Z3wv48J2jzJ0yINgRiYiIdDk3n5PHkeo67n+zkMS4KO64bISSbQkLSrRPpGAhxCTxdu1QYDNTB6s+W0RE5FT84DNDqaiu55Gl20mKi+L7Fw0NdkginlOi3ZrGRl999uALeXd7BT0TYhjaS1PKioiInAoz487LR1JZU899r28jKS6aG6cPDHZYIp5Sot2a3f+Gqv24YZfw3sKDTM7rSYTqs0VERE5ZRIRxz5WjqKqp5+cvbSIpNoovTOjX9oYiXZRmXmlNwUKwCHalT2NPeTVTdFs/ERGR0xYVGcF9147l7CEZ3P7cOp5cuZPGRtf2hiJdkBLt1mxdBLlTWLrb9+HX/bNFREQ6RmxUJA9+6SzyB6Tzk2fXM+O+d3hxTSkNSrilm1Gi3ZKPdsK+DTDsEpYVldE7OZaBGQnBjkpERKTb6BETxT9unszvrh2LGXzviTVcdO/bPLVqF3UNmtxGugcl2i0p8M0G6YbOZHnxQaYOytBtiERERDpYZIQxe2xfFn3vHP50/Xh6xETy42fWcd6v3+Lx5R9SU6+p26VrU6LdkoKF0HMI2xr6UFZZq/psERERD0VEGDPPzOKl70xn/lfyyUyK5Y4XNnDOr5Ywf+l2jtUq4ZauSYl2c9UVsGMpDJvJssIyQPXZIiIincHMuGB4b57/1lQev3ESA3omcPdLmzj7V2/yp7eLqKypD3aIIidFiXZzRW9AYx0Mu5RlRQfplx5PTlqPYEclIiISNsyM6UMyePLrU3jq61MYkZXMPa9sYfov3+T+N7ZRfqwu2CGKtIsS7eYKFkF8Gg19J7Bi+yGm5mUEOyIRkaAys5lmVmBmhWZ2ewvrbzWzTWa2zszeMLP+wYhTuqeJA9P5642TeOHb08jvn8a9r21l+j1v8pvFBZQcPhrs8EROSBPWBGqoh22LYcgMNu87SvmxOk27LiJhzcwigXnAZ4ASYKWZLXDObQro9m8g3zl31My+CfwKuKbzo5XubGy/VB6eO4GNu8uZt6SQeW8V8sCSQvL7pzF7bDaXjsqiZ2JssMMUOY4S7UC7VsCxw7767CJfffaUPCXaIhLWJgKFzrliADN7ApgNfJxoO+eWBPRfDlzfqRFKWDkjO4U/fPEsdh06yoK1u3lxTSn/9eJG7vrnJs4eksHssdl8ZmQfEmM7NsVpbHTsLj9Gn+Q4oiJVECDto0Q70NZXICIaBl3Isve3MCgzgV7JccGOSkQkmPoCuwKelwCTTtD/RuAVTyMSAfql9+Db5w/m2+cPZsveCl5cs5sFa3bzgyfXEhe9ngtH9Gb2mGzOHZZJbFTkSb12Q6Nje1kVG0rLWe9/bNpdQWVNPX1T4/nK1AFcM7EfyXHRHv120l14mmib2Uzgd0Ak8LBz7p4W+nwBuAtwwFrn3HX+9gZgvb/bTufcLC9jBaDgFRh4NnXRiazcfogrx+d4vksRke7CzK4H8oFzT9Dna8DXAHJzczspMunuhvdJZvjMZH48Yxirdx7mxTW7eXndHl5et4fkuCguOTOL2WOzmZTXk8iI4+fFaGh0FB+o/Dih3uBPqqv8txSMjYpgRFYyV4zrS15mAos27OUXCzdz3+tbuWZCLjdMG0C/dN00QVrmWaLdnro+MxsC/BSY5pw7bGa9Al7imHNurFfxfUrZNjhYCBO/zrqScqpqG3RbPxERKAX6BTzP8bcdx8wuAv4TONc5V9PaiznnHgIeAsjPz9d829KhzIyz+qdzVv907rx8JO8WHeTFNaW8tG43T67aRa+kWC4fnc3wPkls2lPx8ZnqY3W+pDouOoKRWclcdVYOZ/ZNYVROCoMzE48rFblh2kDWl5TzyNJi/vLeDh5dtp1LzszixrMHMj43LUi/uYQqL89ot1nXB9wMzHPOHQZwzu33MJ4TK/B/0zlsJu+t9tVnT1J9tojISmCImQ3El2BfC1wX2MHMxgEPAjODOo6LBIiKjODcoZmcOzST6isaeHPLfl5cU8rjyz+ktqGR+OhIzshO5poJ/RjlT6rzMhLaVX89KieF+64dx08uGc6jy3bw9xU7eXn9HsbnpnLT2XnMOKPPp86cS3jyMtFuT13fUAAzexdfecldzrlF/nVxZrYKqAfucc694GGssHUR9D4TUnNZVrScEVnJpCfEeLpLEZFQ55yrN7NbgMX4xun5zrmNZnY3sMo5twD4NZAIPG1m0FnlfiLtFBcdyaWjsrh0VBblx+ooq6xhQM+E006Gs1Li+eklI/juBUN4etUu5r+7g2/9bTX90uO5YepAvjChX4dflCldS7Df/ShgCHAevq8j3zGzUc65j4D+zrlSM8sD3jSz9c65osCNO6zW7+gh2PkenP1Dqusa+ODDw1w/WbeBFREBcM4tBBY2a7szYPmiTg9K5BSlxEeTEt+xFzEmxEbxlWkD+dKUAby2aR+PLC3m7pc28dvXt3LdxFzmTh1Admp8h+5TugYvE+321PWVACucc3XAdjPbii/xXumcKwVwzhWb2VvAOOC4RLvDav22vQauEYZewr93fkRNfaPqs0VEROSkREYYM8/sw8wz+7Bm10c8/K9iHl66nYeXbufSUVncMG0A4/ql4v/mR8KAlzeC/Liuz8xi8NX1LWjW5wV8Z7Mxswx8pSTFZpZmZrEB7dM4vra7YxUshMTekD2O94rKiDCYMDDds92JiIhI9za2XyoPXDeet287jxumDuCtLfu58g/L+Ny8d3ludQk19Q3BDlE6gWeJtnOuHmiq69sMPNVU12dmTbV7i4GDZrYJWALc5pw7CIwAVpnZWn/7Pc1mIes49bVQ+AYMnQERESwrOsionFTdG1NEREROW05aD+64fCTL/+NCfj77DCpr6rn1qbVMu+dN7n21gH0V1cEOUTzkaY12O+r6HHCr/xHYZxkwysvYPvbhUqg9AsMu5WhtPWt2fcTN5+R1yq5FREQkPCTERvGlKQO4fnJ/lhaW8ei7O/j9kkL+8FYRl47KYu7UAYzPVVlJdxPsiyGDr+AViIqDgeeycsdh6hudpl0XERERT5gZZw/J5OwhmXx4sIq/vPchT63cxYK1uxmdk8LcKQO4fEzWSc9mKaHJyxrt0OccFCyCvPMhpgfLisqIjjTyB+iG8yIiIuKt/j0T+K+mspLPnUlVTT0/fFplJd1JeJ/R3rcRynfCOT8CYHnRQcb1S6NHTHgfFhEREek8CbFRfGlyf66flMvSwjIeW/ZJWcklo7KYnJdOclw0SXFRJMdHkxwXTbJ/OTYqQuUmISy8M8qt/tkgh86g/Fgd60vL+c4FQ4Ibk4iIiISl5mUlf33vQ55ctYt/rt3d6jYxkREBCXgUSXHRJMdHkRwXTUZiLOcP76Xa7yAK70R7zBxIGwhJfXh/0z4aHUzR/bNFREQkyPr3TOCOy0fy45nDOXy0liPVdZQfq6eiuo6KY3UcqW5arudIdR0V1fX+9jr2VlRTcayOQ1W1PLCkkL6p8Vw6qg+Xjc5mTE6Kku5OFN6JdkoOjLoKgGVFZcRGRTAuNzXIQYmIiIj4xERF0Ds5jt7JcSe9bUV1Ha9v2sfL6/bw6LId/Plf28lJi+ey0VlcPiqbM/smK+n2WHgn2gHeKzrIhAHpuspXREREuoXkuGiuHJ/DleNzKD9Wx6sb9/Ly+j088q/tPPh2MbnpPXxJ9+gsRmYp6faCEm3gYGUNW/Ye4bYZ2cEORURERKTDpcRHc3V+P67O78dHR2tZvHEvL63bw0PvFPPHt4oYmJHAZaOyuGx0FsP7JCnp7iBKtIHlxYcA1WeLiIhI95faI4ZrJuRyzYRcDlXVsmjDXl5ev5s/vFXIA0sKyctMYPrgDPIyEsjLTGRgRgJ9U+OJiFDyfbKUaOOrz06MjWJ035RghyIiIiLSadITYrhuUi7XTcqlrLLGl3Sv28OzH5RQVdvwcb+YqAgG9kxgYEYCeZlNPxPJy0ggLSEmiL9BaFOiDbxXfJCJA9OJigzv+XtEREQkfGUkxnL95P5cP7k/zjkOHKmhuKyK7WVVFB+oZHtZFVv3H+H1zfuob3Qfb5faI5q8jAQGZiSSl5lATlo8fVPjyU6Np3dyHJFhfCY87BPtveXVFB+oYs6E3GCHIiIiIhISzIxeyXH0So5jct7xpbV1DY2UHD7G9rJKig9U+ZLxA1W8W1jGs6tLjusbFWH0SYmjb6ov+e4bkIQ3LcdFd98bUYR9ov1ecRmg+mwRERGR9oiOjGBghq985ILhx687WlvP7o+OUXL4GLs/qqb0o6OUHj5G6UfHWLH9EHvWHCPgZDgAGYkxZKfGk50ST1ZqHFkpcfRJiff99N/aMCaqa1YdhH2ivazwICnx0YzMSg52KCIiIiJdWo+YKAb3SmJwr6QW19c3NLK3oprSw8fYXX7s4yS89KNqCg9U8q9tB46rDW+SkRjrT8Djjv+ZHE92ahzZqfFEh2AJcNgn2u8VH2RKXk9dSSsiIiLisajICHLSepCT1qPVPkeq69hbXs2e8upPflYcY095NTsPHmVF8UEqquuP2yY60ujfM4FBmQkMykz0PXr5asaT46K9/rVaFdaJ9q5DRyk5fIybz84LdigiIiIiAiTFRZMUF82Q3i2fFQeoqqlnb4UvES/96Bjby6oo2l9J4f5K3ti8/7iLNXsnx36SfGcmMKiXbzkrJc7z+4WHdaK9rMhXnz1V9dkiIiIiXUZCbNTHyXNzdQ2N7Dx0lKL9lRQdqKLoQCVFByp5YU0pRwLOhPeIiWRQZiLPfWuqZ2UnnibaZjYT+B0QCTzsnLunhT5fAO4CHLDWOXedv30ucIe/23875x7r6PhSe8Rw8cjeDO716TdJRERERLqe6MiIFpNw5xwHKmso2v9J8n24qtbT2m7PEm0ziwTmAZ8BSoCVZrbAObcpoM8Q4KfANOfcYTPr5W9PB34G5ONLwD/wb3u4I2OccUYfZpzRpyNfUkRERERCkJnRKymOXklxnXa3OS8vz5wIFDrnip1ztcATwOxmfW4G5jUl0M65/f72GcBrzrlD/nWvATM9jFVEREREpEN5mWj3BXYFPC/xtwUaCgw1s3fNbLm/1KS922JmXzOzVWa26sCBAx0YuoiINDGzmWZWYGaFZnZ7C+vPMbPVZlZvZlcFI0YRkVAU7BsORgFDgPOAOcCfzSy1vRs75x5yzuU75/IzMzM9ClFEJHwFlAFeAowE5pjZyGbddgJfAf7eudGJiIQ2LxPtUqBfwPMcf1ugEmCBc67OObcd2Iov8W7PtiIi4r02ywCdczucc+uAxmAEKCISqrxMtFcCQ8xsoJnFANcCC5r1eQHf2WzMLANfKUkxsBi42MzSzCwNuNjfJiIinatdpXztpZI/EQknniXazrl64BZ8CfJm4Cnn3EYzu9vMZvm7LQYOmtkmYAlwm3PuoHPuEPBzfMn6SuBuf5uIiHRhKvkTkXDi6X20nXMLgYXN2u4MWHbArf5H823nA/O9jE9ERNqkUj4RkVMU7IshRUQktLWnDFBERFpgvpPKXZ+ZHQA+PIVNM4CyDg7ndIVaTKEWD4ReTIqnbaEWU6jF0985F5K1FGZ2KXAfvll+5zvnfmFmdwOrnHMLzGwC8DyQBlQDe51zZ7TjdbvLuK142hZqMYVaPBB6MSmeE2vXmN1tEu1TZWarnHP5wY4jUKjFFGrxQOjFpHjaFmoxhVo80n6h9t4pnraFWkyhFg+EKkJHWAAAB9dJREFUXkyKp2OodERERERExANKtEVEREREPKBEGx4KdgAtCLWYQi0eCL2YFE/bQi2mUItH2i/U3jvF07ZQiynU4oHQi0nxdICwr9EWEREREfGCzmiLiIiIiHhAibaIiIiIiAfCJtE2s5lmVmBmhWZ2ewvrY83sSf/6FWY2wMNY+pnZEjPbZGYbzex7LfQ5z8zKzWyN/3FnS6/VwXHtMLP1/v2tamG9mdn9/mO0zszGexjLsIDffY2ZVZjZ95v18fwYmdl8M9tvZhsC2tLN7DUz2+b/mdbKtnP9fbaZ2VwP4/m1mW3xvyfPm1lqK9ue8P3t4JjuMrPSgPfm0la2PeHnsgPjeTIglh1mtqaVbT05RnLyQmnM9u8v5MbtUBqz/fsL+ritMfuUY9KY7RXnXLd/4JtkoQjIA2KAtcDIZn2+BfzJv3wt8KSH8WQB4/3LScDWFuI5D3ipk4/TDiDjBOsvBV4BDJgMrOjE928vvpvDd+oxAs4BxgMbAtp+BdzuX74d+GUL26UDxf6faf7lNI/iuRiI8i//sqV42vP+dnBMdwE/asf7esLPZUfF02z9/wJ3duYx0uOk38OQGrP9+wi5cTtUx+yA97DTx22N2acck8Zsjx7hckZ7IlDonCt2ztUCTwCzm/WZDTzmX34GuNDMzItgnHN7nHOr/ctHgM1AXy/21cFmA39xPsuBVDPL6oT9XggUOedOZQa50+Kcewc41Kw58G/lMeBzLWw6A3jNOXfIOXcYeA2Y6UU8zrlXnXP1/qfLgZzT3c/pxtRO7flcdmg8/s/0F4B/nO5+xFMhNWZDlx23gzVmQ5DGbY3ZpxZTO2nMPgXhkmj3BXYFPC/h0wPkx338H4ByoKfXgfm/7hwHrGhh9RQzW2tmr5hZm9MZdwAHvGpmH5jZ11pY357j6IVraf1D1tnHCKC3c26Pf3kv0LuFPsE6Vl/FdwarJW29vx3tFv9Xo/Nb+ao2GMfobGCfc25bK+s7+xhJy0J2zIaQGrdDdcyG0Bq3NWa3j8ZsD4RLoh2SzCwReBb4vnOuotnq1fi+chsD/B54oRNCmu6cGw9cAnzbzM7phH2ekJnFALOAp1tYHYxjdBzn++4qJO6RaWb/CdQDf2ulS2e+v38EBgFjgT34vvoLBXM48ZmRkPsMSGgJsXE7JP9eQ3nc1pjdKo3ZHgmXRLsU6BfwPMff1mIfM4sCUoCDXgVkZtH4Buu/Oeeea77eOVfhnKv0Ly8Eos0sw6t4/Psp9f/cDzyP72uiQO05jh3tEmC1c25f8xXBOEZ++5q+fvX/3N9Cn049Vmb2FeBy4Iv+f0g+pR3vb4dxzu1zzjU45xqBP7eyr84+RlHAlcCTrfXpzGMkJxRyY7Z/PyE1bofomA2hN25rzG6DxmzvhEuivRIYYmYD/f/TvhZY0KzPAqDpKuOrgDdb++M/Xf6ao0eAzc65e1vp06ep3tDMJuJ7r7xM/BPMLKlpGd/FGhuadVsAfNl8JgPlAV/HeaXV/8129jEKEPi3Mhd4sYU+i4GLzSzN/xXcxf62DmdmM4EfA7Occ0db6dOe97cjYwqsA72ilX2153PZkS4CtjjnSlpa2dnHSE4opMZsCL1xO4THbAi9cVtjdtsxacz2SnuvmuzqD3xXX2/Fd8Xsf/rb7sb3hw4Qh+9rrkLgfSDPw1im4/vqah2wxv+4FPgG8A1/n1uAjfiu6l0OTPX4+OT597XWv9+mYxQYkwHz/MdwPZDvcUwJ+AbglIC2Tj1G+P6x2APU4atHuxFfHegbwDbgdSDd3zcfeDhg26/6/54KgRs8jKcQX91c099S050YsoGFJ3p/PYzpr/6/kXX4BuKs5jH5n3/qc+lFPP72R5v+dgL6dsox0uOU3seQGbP9+wupcbu1v1eCOGb79xnUcbuV8Uhjdtsxacz26KEp2EVEREREPBAupSMiIiIiIp1KibaIiIiIiAeUaIuIiIiIeECJtoiIiIiIB5Roi4iIiIh4QIm2SAcxs/PM7KVgxyEiIm3TmC2dQYm2iIiIiIgHlGhL2DGz683sfTNbY2YPmlmkmVWa2W/NbKOZvWFmmf6+Y81suZmtM7Pn/TOGYWaDzex1M1trZqvNbJD/5RPN7Bkz22Jmf2uaAU1ERE6NxmzpypRoS1gxsxHANcA059xYoAH4Ir7ZzFY5584A3gZ+5t/kL8BPnHOj8c2a1dT+N2Cec24MMBXfrFYA44DvAyPxzVo1zfNfSkSkm9KYLV1dVLADEOlkFwJnASv9Jy7igf1AI/Ckv8/jwHNmlgKkOufe9rc/BjxtZklAX+fc8wDOuWoA/+u975wr8T9fAwwAlnr/a4mIdEsas6VLU6It4caAx5xzPz2u0ey/mvVzp/j6NQHLDegzJiJyOjRmS5em0hEJN28AV5lZLwAzSzez/vg+C1f5+1wHLHXOlQOHzexsf/uXgLedc0eAEjP7nP81Ys2sR6f+FiIi4UFjtnRp+p+bhBXn3CYzuwN41cwigDrg20AVMNG/bj++mkCAucCf/INyMXCDv/1LwINmdrf/Na7uxF9DRCQsaMyWrs6cO9VvW0S6DzOrdM4lBjsOERFpm8Zs6SpUOiIiIiIi4gGd0RYRERER8YDOaIuIiIiIeECJtoiIiIiIB5Roi4iIiIh4QIm2iIiIiIgHlGiLiIiIiHjg/wPNTDTixRQIJgAAAABJRU5ErkJggg==\n"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "execution_count": 39,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "So, how well does our model perform?"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "train_metrics = model.evaluate_generator(train_gen)\n",
        "test_metrics = model.evaluate_generator(test_gen)\n",
        "\n",
        "print(\"\\nTrain Set Metrics of the trained model:\")\n",
        "for name, val in zip(model.metrics_names, train_metrics):\n",
        "    print(\"\\t{}: {:0.4f}\".format(name, val))\n",
        "\n",
        "print(\"\\nTest Set Metrics of the trained model:\")\n",
        "for name, val in zip(model.metrics_names, test_metrics):\n",
        "    print(\"\\t{}: {:0.4f}\".format(name, val))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Train Set Metrics of the trained model:\n",
            "\tloss: 0.0549\n",
            "\tacc: 0.9978\n",
            "\n",
            "Test Set Metrics of the trained model:\n",
            "\tloss: 0.6798\n",
            "\tacc: 0.7925\n"
          ]
        }
      ],
      "execution_count": 40,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "There is space for improvements but this article is in the first place a conceptual invitation not a way to accuracy paradise."
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python",
      "version": "3.7.2",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernel_info": {
      "name": "python3"
    },
    "nteract": {
      "version": "0.15.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}