xiupos · June 18, 2024 11:35
diff --git a/nlp-ml-2.ipynb b/nlp-ml-2.ipynb
 {
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP0uBH7xX29UysYHkjbHFb3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/xiupos/1f5ec6153f93cc09da73ba0f6ca35da0/nlp-ml-2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# NLP/ML 勉強会 #2"
      ],
      "metadata": {
        "id": "h33YdTr0Cisa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "とにかく動く言語モデルを作ります。動けばいいので、精度は求めません。単純な構成を採用して「それっぽく」動くものをまずは作ってみましょう。\n",
        "\n",
        "[勉強会 #1](https://gist.githubusercontent.com/xiupos/bf8adde430538f1befc7a6f2b7f9add8/raw/7b931d8dad262c66da6dc24e7e2d2a5ad3dfcff9/Drawing_2023-05-02_22.46.29.excalidraw.png) を前提とした記述をしています。"
      ],
      "metadata": {
        "id": "UlPyWRiRcLhb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 準備"
      ],
      "metadata": {
        "id": "yPEcpnXUCrN6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "まずは使うライブラリを用意します。今回は [PyTorch](https://pytorch.org/) を中心に使います。"
      ],
      "metadata": {
        "id": "DyuV_PX95kXu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UnKGkrykcKD7",
        "outputId": "0ce2b5c8-05c5-4951-9c43-4a9debc1c6be"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<torch._C.Generator at 0x7d0dd8236270>"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "from torch.nn import functional as F\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "import pandas as pd\n",
        "# 乱数を固定\n",
        "torch.manual_seed(1)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次に教師データを用意します。教師データは何でもいいですが、ここでは「シャーロックホームズの冒険 (The Adventures of Sherlock Holmes)」の[原文](https://sherlock-holm.es/stories/plain-text/advs.txt)にしました。有名な教師データには [Tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) などがあります。"
      ],
      "metadata": {
        "id": "lz9mjU5ljFGe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 教師データをダウンロード (https://sherlock-holm.es/stories/plain-text/advs.txt を基に作成)\n",
        "!wget https://gist.githubusercontent.com/xiupos/b7914ea1e3ab35465c34de45146b15d8/raw/b7623310de06160876ffba6299994b0409c85c81/advs.txt\n",
        "lines: str = open('./advs.txt', 'r').read().replace('\\n', ' ')"
      ],
      "metadata": {
        "id": "F85C-xgrgYgg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cf7848da-9951-4b9b-9109-93065cb73669"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-06-17 05:52:18--  https://gist.githubusercontent.com/xiupos/b7914ea1e3ab35465c34de45146b15d8/raw/b7623310de06160876ffba6299994b0409c85c81/advs.txt\n",
            "Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 562263 (549K) [text/plain]\n",
            "Saving to: ‘advs.txt.4’\n",
            "\n",
            "advs.txt.4          100%[===================>] 549.08K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2024-06-17 05:52:19 (11.2 MB/s) - ‘advs.txt.4’ saved [562263/562263]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "教師データは文字列です。文字列を機械学習で直接扱うのは難しいので、まずは単語を整数と対応させます。ただし、英単語を単位に扱うのは高度なので、簡単のためここでは文字を単位に考えます。以降、「単語」は文字のことを指します。言語モデルの文脈では「トークン」と呼ぶのが正確です。\n",
        "\n",
        "単純に文字順の番号を文字と対応する整数として用います。たとえば、文字順で $μ$ 番目の文字には整数 $μ$ が対応します。以降、その文字のことを単語 $μ$ と呼びます。"
      ],
      "metadata": {
        "id": "a4_gSeCj36vC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 文字一覧\n",
        "vocab: list[str] = sorted(list(set(lines)))\n",
        "# {数(番号): 文字} の辞書\n",
        "itos: dict[int, str] = {i:s for i,s in enumerate(vocab)}\n",
        "# {文字: 数(番号)} の辞書\n",
        "stoi: dict[str, int] = {s:i for i,s in enumerate(vocab)}\n",
        "\n",
        "len(vocab)"
      ],
      "metadata": {
        "id": "_Gl30hk9ghRk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3880b09e-f4f0-43b0-ceab-e532d53ae710"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "83"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次に上の対応を用いて、文字列を整数列に変換する関数を定義しましょう。また教師データを整数列に変換して教師データ `dataset` とします。"
      ],
      "metadata": {
        "id": "naUorL6u4Ayw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 文字列 -> 整数列 の関数\n",
        "def encode(s: str) -> list[int]:\n",
        "    return [stoi[s] for s in s]\n",
        "\n",
        "# 整数列 -> 文字列 の関数\n",
        "def decode(l: list[int]) -> str:\n",
        "    return ''.join([itos[i] for i in l])\n",
        "\n",
        "# 教師データを整数列に変換\n",
        "dataset = encode(lines)\n",
        "\n",
        "encode(\"Do you know, Watson,\"), decode(encode(\"Do you know, Watson,\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvkJQt4jhCbY",
        "outputId": "7afd4d65-e349-4194-b2cd-a14b41df0b5a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([27, 65, 0, 75, 65, 71, 0, 61, 64, 65, 73, 7, 0, 46, 51, 70, 69, 65, 64, 7],\n",
              " 'Do you know, Watson,')"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "以降の便利のため、`dataset` から学習用のバッチを作成する関数を定義します。例えば \"Do you know, Watson,\" という文章に対し, バッチは以下の2つの文字列の組から構成されます。\n",
        "\n",
        "- x: `Do you know, Wat`\n",
        "- y: `o you know, Wats`\n",
        "\n",
        "後述しますが、今回は「1単語から次の1単語を予測する」モデルを作ります。そのため、パッチは1文字ずらしたタプルになっています。"
      ],
      "metadata": {
        "id": "ixBGfeS87g69"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# バッチを作成する関数\n",
        "# 幅 xnum で1文字ずれた列を生成する\n",
        "# ex) ' so if you ar', 'so if you are'\n",
        "#     |<-----x----->|  |<-----y----->|\n",
        "def get_batches(xnum: int, data: list[int] = dataset, split=\"train\") -> tuple[list[int], list[int]]:\n",
        "    # 教師データを 8:2 で分割してバッチとする\n",
        "    if split == \"train\":\n",
        "        # 学習データ\n",
        "        batchdata = data[:int(len(data)*0.8)]\n",
        "    elif split == \"val\":\n",
        "        # 検証データ\n",
        "        batchdata = data[int(len(data)*0.8):]\n",
        "\n",
        "    # パッチの開始位置をランダムに決める\n",
        "    idx = torch.randint(0, len(batchdata)-xnum-1, (1,))\n",
        "    # パッチを取り出す\n",
        "    x = batchdata[idx:idx+xnum]\n",
        "    y = batchdata[idx+1:idx+xnum+1]\n",
        "    return x, y\n",
        "\n",
        "[decode(l) for l in get_batches(16)]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uxSYnZjodkAy",
        "outputId": "98b1c075-535a-4819-b44a-edc526cc7277"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['rip has been upo', 'ip has been upon']"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "これで準備が整いました。"
      ],
      "metadata": {
        "id": "nhGrIeL-9K7S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## はじめてのモデル"
      ],
      "metadata": {
        "id": "1mnExgl0Cvew"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "さて、はじめてのモデルを作っていきます。ここでは「1単語から次の1単語を予測する」モデルを作ります。これを繰り返し行うことで文章を生成することができます。\n",
        "\n",
        "構成は大きく三段階になっています。\n",
        "\n",
        "1. 単語のベクトル化\n",
        "2. ニューラルネットワーク\n",
        "3. ベクトルを単語に直す\n",
        "\n",
        "まずは単語のベクトル化について考えます。最も簡単なベクトル化は「ワンホットベクトル」と呼ばれる方法です。ワンホットベクトルとは、対応する単語の要素は $1$ で、それ以外は $0$ である列ベクトルです。つまり、単語 $μ$ に対応するワンホットベクトル $\\vec{x} = (x_1,⋯,x_{\\mathtt{len(vocab)}})^{\\top}$ の $ν$ 行目の要素 $x_ν$ は\n",
        "$$\n",
        "x_ν = δ_{μν} ≡ \\begin{cases} 1, & μ=ν \\\\ 0 & μ≠ν \\end{cases}\n",
        "$$\n",
        "を満たし、ベクトル表記すれば\n",
        "$$\n",
        "\\vec{x} = \\left(\\begin{array}{c} ⋮ \\\\ x_{μ-1} \\\\ x_μ \\\\ x_{μ+1} \\\\ ⋮ \\end{array}\\right) = \\left(\\begin{array}{c} ⋮ \\\\ 0 \\\\ 1 \\\\ 0 \\\\ ⋮ \\end{array}\\right)\n",
        "$$\n",
        "となります。\n",
        "\n",
        "次に、単語ベクトルと確率分布を対応させます。これによって単語ベクトルを単語に戻すことができます。単語ベクトル $\\vec{y} = (y_1,⋯,y_{\\mathtt{len(vocab)}})^{\\top}$ に対し, $ν$ 番目の単語である確率 $p_ν$ を\n",
        "$$\n",
        "p_ν ≡ \\operatorname{softmax}(y_ν) ≡ \\frac{e^{y_ν}}{\\sum_{μ=1}^{\\mathtt{len(vocab)}} e^{y_μ}}\n",
        "$$\n",
        "と定義します。一般に、指数による上記のような正規化を softmax 関数といいます。以下、確率分布 $p = \\{p_1,…,p_{\\mathtt{len(vocab)}}\\}$ とベクトル\n",
        "$$\n",
        "p(\\vec{y})≡\\operatorname{softmax}(\\vec{y})≡(p(y_1),⋯,p(y_{\\mathtt{len(vocab)}}))^{\\top}\n",
        "$$\n",
        "を同一視して扱います。\n",
        "\n",
        "単語ベクトルの確率分布から単語を選択する方法は、「最も確率の高い単語を選べばよい」と思われるかもしれません。このような方法を「貪欲法」といいます。ですが、せっかく確率分布が得られるので、それに従って確率的に選択するようにしましょう。\n",
        "\n",
        "以上の議論に基づいて、ベクトル化の関数を定義しましょう。"
      ],
      "metadata": {
        "id": "XWbnir1y4HPl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 数リスト -> 単語ベクトル列 の関数\n",
        "def ltov(l: list[int]) -> torch.Tensor:\n",
        "    # ワンホットベクトル列を生成\n",
        "    return torch.eye(len(vocab))[l]\n",
        "\n",
        "# 単語ベクトル列 -> 数リスト の関数 (貪欲法)\n",
        "def vtol(v: torch.Tensor) -> list[int]:\n",
        "    # 値が最大(⇔確率が最大)の単語を選択\n",
        "    return torch.argmax(v, dim=-1).tolist()\n",
        "\n",
        "# 単語ベクトル列 -> 数リスト の関数 (確率分布に基づく)\n",
        "def vtol_prob(v: torch.Tensor) -> list[int]:\n",
        "    # 確率分布を計算\n",
        "    p = F.softmax(v, dim=-1)\n",
        "    # 確率分布を基に\n",
        "    return torch.multinomial(p, num_samples=1).view(-1).tolist()\n",
        "\n",
        "xs, _ = get_batches(16)\n",
        "decode(xs), decode(vtol(ltov(xs))), decode(vtol_prob(ltov(xs)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b8wyKO8K9qZG",
        "outputId": "d7d24e47-eeb8-4b2f-cb13-dab02d09b5cd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('bited. The bedro', 'bited. The bedro', '½Qz,eKvHs0:6SdLQ')"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次にモデルの中心となるニューラルネットワークを定義します。簡単のため ReLU を活性化関数に用いた中間層1層の順伝播型ニューラルネットワークとします: 式に書くと\n",
        "$$\n",
        "\\vec{y} = W_2 \\operatorname*{ReLU}(W_1 \\vec{x} + \\vec{b}_1) + \\vec{b}_2\n",
        "$$\n",
        "です。ここで、行列 $W_1$, $W_2$ とベクトル $\\vec{b}_1$, $\\vec{b}_2$ はモデルのパラメータです。成分表示すれば,\n",
        "$$\n",
        "\\begin{aligned}\n",
        "y_μ\n",
        "  &= \\sum_ν \\left[ w_{2μν} \\operatorname*{ReLU}\\left(\\sum_λ w_{1νλ} x_λ + b_{1λ}\\right) + b_{2ν} \\right] \\\\\n",
        "  &= \\sum_ν \\left[ w_{2μν} \\max\\left(\\sum_λ w_{1νλ} x_λ + b_{1λ}, 0\\right) + b_{2ν} \\right]\n",
        "\\end{aligned}\n",
        "$$\n",
        "となります。\n",
        "PyTorch の `nn.Sequential` を使ってモデルを定義しましょう。 `nn.Linear` は $\\vec{x} ↦ W \\vec{x} + \\vec{b}$, `nn.ReLU` は $\\vec{x} ↦ \\operatorname*{ReLU}(\\vec{x})$ を意味しています。"
      ],
      "metadata": {
        "id": "KjrBDikMruW0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# モデルの隠れ層の要素数\n",
        "hidden_layer_dim = 128\n",
        "# モデルの定義\n",
        "model: nn.Module = nn.Sequential(\n",
        "    nn.Linear(len(vocab), hidden_layer_dim),\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(hidden_layer_dim, len(vocab)),\n",
        ")\n",
        "\n",
        "model, [m.numel() for m in model.parameters()]"
      ],
      "metadata": {
        "id": "JUZIrlQsA7uD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb204fed-f73c-4a55-de0f-a3e0ae89a29d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(Sequential(\n",
              "   (0): Linear(in_features=83, out_features=128, bias=True)\n",
              "   (1): ReLU()\n",
              "   (2): Linear(in_features=128, out_features=83, bias=True)\n",
              " ),\n",
              " [10624, 128, 10624, 83])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ここで、隠れ層の要素数は `hidden_layer_dim` としました。さっそくモデルを使ってみましょう。適当にバッチを取得して次の単語を予想させてみます。"
      ],
      "metadata": {
        "id": "Jca-Fql-ncdu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "xs, ys = get_batches(16)\n",
        "decode(xs), decode(vtol(model(ltov(xs)))), decode(vtol_prob(model(ltov(xs))))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i3wJcrV34gf4",
        "outputId": "a48312ac-4da0-404b-c673-fbaa857361ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('ression upon me,', 'Vlgggg(gMgg(ggll', \"uTCf'G)zewtKLHr!\")"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "滅茶苦茶ですね。学習も何もしてないので当然の結果です。ついでに長い文章を生成する関数も定義します。"
      ],
      "metadata": {
        "id": "kZ4l65Y7niU8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 文章を生成する関数\n",
        "def generate(model: nn.Module, xnum=100):\n",
        "    # 文章を格納する配列\n",
        "    out: list[int] = []\n",
        "    # 初期状態\n",
        "    x = torch.zeros(1, len(vocab))\n",
        "    # 文章の長さ\n",
        "    for i in range(xnum):\n",
        "        # モデルを適用\n",
        "        ys = vtol_prob(model(x))\n",
        "        # 結果に追加\n",
        "        out += [ys[-1]]\n",
        "        # 次の単語\n",
        "        x = ltov(ys)\n",
        "    return decode(out)\n",
        "\n",
        "print(generate(model))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "utMCQ2f0rFeq",
        "outputId": "803ff351-a8ce-4022-e3f3-cfa546f1c98a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I6)TfOcQKfélo7h]è(0o?eNSRB7U1vh½m;\"l)£JHB9!h£s&Kgm7TfhDy58b:?2m5B'n\" fS£gW[-VeFPK3P9:léiVEVLQtC?V7Ge\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "見るからに出鱈目です。ですが、どのくらい出鱈目でしょうか？モデルを学習させるためにはこの「出鱈目さ」を計算する必要があります。"
      ],
      "metadata": {
        "id": "2srvCmyEUizZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## モデルの評価"
      ],
      "metadata": {
        "id": "qoQ6OdN7DPqs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "作ったモデルを定量的に評価することを考えます。機械学習においては「損失関数」と呼ばれる関数を用いて、教師データとモデルのずれを評価します。ここでは「交差エントロピー」という関数を使います。まずは必要な知識をまとめます。復習ですが、単語ベクトル $\\vec{y}$ に対する確率分布 $p = \\{p_ν\\}$ は\n",
        "$$\n",
        "p_ν ≡ \\operatorname{softmax}(y_ν) ≡ \\frac{e^{y_ν}}{\\sum_{μ=1}^{\\mathtt{len(vocab)}} e^{y_μ}}\n",
        "$$\n",
        "です。\n",
        "\n",
        "- **期待値**:\n",
        "  $\\vec{y}$ の各成分に対する関数\n",
        "  $$\n",
        "  f(\\vec{y}) = (f(y_1),⋯,f(y_{\\mathtt{len(vocab)}}))^{\\top}\n",
        "  $$\n",
        "  に対し、期待値は\n",
        "  $$\n",
        "  ⟨f(\\vec{y})⟩ ≡ \\sum_{ν=1}^{\\mathtt{len(vocab)}} f(y_ν) p_ν\n",
        "  $$\n",
        "  で定義されます。\n",
        "\n",
        "- **エントロピー**:\n",
        "  単語 $ν$ の情報量\n",
        "  $$\n",
        "  \\log \\frac1{p_ν} \\quad (= - \\log p_ν)\n",
        "  $$\n",
        "  は単語 $v$ の確率が低いほど大きくなる量であり、情報の「不確かさ」を表す指標です。この情報量の期待値\n",
        "  $$\n",
        "  H[p] = ⟨- \\log p⟩ = -\\sum_ν p_ν \\log p_ν\n",
        "  $$\n",
        "  はエントロピーと呼ばれています。\n",
        "\n",
        "- **交差エントロピー**:\n",
        "  教師データとなる単語ベクトルの組 $(\\vec{x},\\vec{y})$ に対し、モデルによって予測値 $y(\\vec{x})$ が得られたとし、この教師データ $\\vec{y}$ と予測値 $y(\\vec{x})$ を比較することを考えます。$\\vec{y}$, $y(\\vec{x})$ に対応する確率分布がそれぞれ $p=\\{p_ν\\}$, $q=\\{q_ν\\}$ であるとき, 単語 $ν$ の情報量はそれぞれ $- \\log p_ν$, $- \\log q_ν$ であり、この差\n",
        "  $$\n",
        "  [- \\log q_ν] - [- \\log p_ν] = \\log \\frac{p_ν}{q_ν}\n",
        "  $$\n",
        "  の期待値は\n",
        "  $$\n",
        "  \\begin{aligned}\n",
        "    D_{\\mathrm{KL}}(p\\|q)\n",
        "      &≡ \\left\\langle \\log \\frac{p}{q} \\right\\rangle \\\\\n",
        "      &= \\sum_ν p_ν \\log p_ν - \\sum_ν p_ν \\log q_ν \\\\\n",
        "      &= - H(p) +  H(p,q)\n",
        "  \\end{aligned}\n",
        "  $$\n",
        "  で、これら $D_{\\mathrm{KL}}(p\\|q) ≡ \\left\\langle \\log \\frac{p}{q} \\right\\rangle$ と $H(p,q) ≡ ⟨- \\log q⟩$ はそれぞれ Kullback–Leibler 情報量と交差エントロピーと呼ばれます。もし教師データの確率分布 $p$ が既知であるとき、確率分布 $p$, $q$ が一致するならば Kullback–Leibler 情報量と交差エントロピーの値は最小になります。つまり、$q$ を $p$ に近づける問題は交差エントロピー $H(p,q)$ の最小化問題と等価です。\n",
        "\n",
        "  PyTorch の `F.cross_entropy` や `nn.CrossEntropyLoss()` は確率分布の組 $(p,q)$ ではなく元のベクトル $(y(\\vec{x}),\\vec{y})$ を引数に取ることに注意しましょう。\n",
        "\n",
        "この交差エントロピーを損失関数として採用します。さっそく今回のモデルの損失関数を計算してみましょう。"
      ],
      "metadata": {
        "id": "Wv-EQh70nuTZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 損失関数に交差エントロピーを採用する\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "\n",
        "xs, ys = get_batches(16)\n",
        "loss = loss_fn(model(ltov(xs)), ltov(ys))\n",
        "loss.item()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OsGfl9doGV-d",
        "outputId": "dc9fa87b-6dfc-4d5e-f5dc-05c253810518"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4.411202907562256"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "この値はどう解釈すればいいのでしょうか? 今回の教師データ $\\vec{y}$ はワンホットベクトルで, ここでは $\\vec{y}$ が単語 $μ$ を表しているときの確率分布が $p = \\{p_ν\\} = \\{δ_{μν}\\}$ としてみましょう。(実際、`nn.CrossEntropyLoss()` ではこのような扱いになります。) このとき1単語のみの損失関数は\n",
        "$$\n",
        "\\begin{aligned}\n",
        "\\mathtt{loss}\n",
        "  &= ⟨- \\log q⟩ \\\\\n",
        "  &= - \\sum_ν p_ν \\log q_ν \\\\\n",
        "  &= - \\sum_ν δ_{μν} \\log q_ν \\\\\n",
        "  &= - \\log q_μ \\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "であるので、損失関数の値 `loss` から単語 $μ$ の生成確率がわかります:\n",
        "$$\n",
        "q_μ = e^{-\\mathtt{loss}}\n",
        "$$\n",
        "実際には損失関数は複数の単語についての平均値になるので、正しい単語の生成確率が\n",
        "$$\n",
        "q = e^{-\\mathtt{loss}}\n",
        "$$\n",
        "になるくらいの目安です。"
      ],
      "metadata": {
        "id": "NXygYyddKFyH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 単語の生成確率\n",
        "q = torch.exp(-loss)\n",
        "q.item()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jVAZ0LItUlph",
        "outputId": "89c59c16-469e-4653-8598-da0c85d86805"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.012140565551817417"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "これでもまだよくわかりませんね。ところで、$N$ 個の単語から無作為に単語を選ぶとき、正しい単語が選ばれる確率は $\\displaystyle q = \\frac1N$ となります。つまり, $1/q\\ (=e^{\\mathtt{loss}})$ は無作為に単語を選び出したとするときの「母数」を表しています。これを計算してみましょう。"
      ],
      "metadata": {
        "id": "HrTgVnCeYNqa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 単語生成の\"母数\"\n",
        "1/q.item(), len(vocab)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mB2S_EfHcQWT",
        "outputId": "35e581f5-21b9-4beb-e426-5b9ba1870a26"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(82.36848569631108, 83)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "つまり $1/q≃\\mathtt{len(vocab)}$ であることがわかりました。モデルはほとんど無作為に単語を選んでいるということです。学習も何もしていないので当然の結果ですね。それではこれを改善すべく、モデルを学習させていきましょう。"
      ],
      "metadata": {
        "id": "9gdbURiQc1xP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## モデルの学習"
      ],
      "metadata": {
        "id": "T2qsjzR5fZgI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "モデルの学習をします。学習とは損失関数の値が小さくなるようパラメータの値を変えていく作業で、代表的な学習のアルゴリズムは以下の通りです。\n",
        "\n",
        "1. パラメータ関して損失関数の勾配を計算する。\n",
        "2. 1 の値をパラメータから引いて新しいパラメータとする。\n",
        "3. `epochs` の回数だけ 1~2 に戻る。\n",
        "\n",
        "具体的には、パラメータ $W_1$, $W_2$, $\\vec{b}_1$, $\\vec{b}_2$ の要素 $X$ に対し、適当なパッチに関する損失関数 `loss` の勾配は\n",
        "$$\n",
        "\\frac{∂\\ \\mathtt{loss}}{∂X}\n",
        "$$\n",
        "で与えられます。これを既存のパラメータ値 $X_{\\mathsf{old}}$ から引いて新しいパラメータ値 $X_{\\mathsf{new}}$ とします:\n",
        "$$\n",
        "X_{\\mathsf{old}} \\quad ⟼ \\quad X_{\\mathsf{new}} = X_{\\mathsf{old}} - η \\left.\\frac{∂\\ \\mathtt{loss}}{∂X}\\right|_{X=X_{\\mathsf{old}}}.\n",
        "$$\n",
        "ただし $η$ は学習率と呼ばれる定数です。この手法を確率的勾配降下法 (SGD) といい、この操作を繰り返せば原理的に `loss` の値が小さくなることがわかると思います。ここでは [Adam](https://arxiv.org/abs/1412.6980) という確率的勾配降下法の改良を利用して学習を実行します。"
      ],
      "metadata": {
        "id": "yzZ3fA-crwz2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 最適化手法に Adam を採用する\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "\n",
        "# 学習\n",
        "def train(model: nn.Module, epochs=100):\n",
        "    # 学習過程を記録\n",
        "    logs: dict[str, list[float]] = {\"train\": [], \"val\": []}\n",
        "\n",
        "    # epochs の回数だけ学習を繰り返す\n",
        "    for epoch in range(epochs):\n",
        "        log_temp: dict[str, list[float]] = {\"train\": [], \"val\": []}\n",
        "\n",
        "        # 学習と検証を10回ずつ行う\n",
        "        for _ in range(10):\n",
        "            for split in [\"train\", \"val\"]:\n",
        "              # バッチを取得\n",
        "              xs, ys = get_batches(16, split=split)\n",
        "              # 損失関数を計算\n",
        "              loss = loss_fn(model(ltov(xs)), ltov(ys))\n",
        "              # 「母数」の値を記録\n",
        "              log_temp[split] += [torch.exp(loss).item()]\n",
        "              # 学習を実行\n",
        "              if split == \"train\":\n",
        "                optimizer.zero_grad()\n",
        "                loss.backward()\n",
        "                optimizer.step()\n",
        "\n",
        "        # 10回の学習・検証の値を平均して記録\n",
        "        for split in [\"train\", \"val\"]:\n",
        "          logs[split] += [np.mean(log_temp[split])]\n",
        "\n",
        "    # 学習過程のグラフを返す\n",
        "    return pd.DataFrame(logs).plot()\n",
        "\n",
        "train(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "uO7evbn_ffcS",
        "outputId": "0d359a2f-d676-40b0-e3b5-c942aa9d38d7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "metadata": {},
          "execution_count": 13
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABz7klEQVR4nO3dd3hUZdrH8e/MpHcSUkkCofeOEMCOYkMQdC3ogrq2xf6qq2vZteK6rh3rKuoquroqdpGuKL3X0EkglfTe5rx/nMkkgQBpZELy+1zXXElmzpy5c4zMPc9zP/djMQzDQERERKSFWF0dgIiIiLQvSj5ERESkRSn5EBERkRal5ENERERalJIPERERaVFKPkRERKRFKfkQERGRFqXkQ0RERFqUm6sDOJLdbic5ORl/f38sFourwxEREZF6MAyD/Px8oqKisFqPP7bR6pKP5ORkYmJiXB2GiIiINEJSUhLR0dHHPabVJR/+/v6AGXxAQICLoxEREZH6yMvLIyYmxvk+fjytLvmommoJCAhQ8iEiInKKqU/JhApORUREpEUp+RAREZEWpeRDREREWlSrq/kQERE5WQzDoKKigsrKSleHckpyd3fHZrM1+TxKPkREpF0oKysjJSWFoqIiV4dyyrJYLERHR+Pn59ek8yj5EBGRNs9ut7Nv3z5sNhtRUVF4eHiokWUDGYZBRkYGBw8epEePHk0aAVHyISIibV5ZWRl2u52YmBh8fHxcHc4pKzQ0lP3791NeXt6k5EMFpyIi0m6cqO23HF9zjRbpv4KIiIi0KCUfIiIi0qKUfIiIiLQTXbp04aWXXnJ1GCo4FRERac3OOussBg8e3CxJw+rVq/H19W16UE3UbpKP5Jxi5qxMpLzSzkMX9XF1OCIiIs3CMAwqKytxczvxW3poaGgLRHRi7WbapbC0gtcW7+ajFQew2w1XhyMiIi5kGAZFZRUuuRlG/d+Dpk+fztKlS3n55ZexWCxYLBbef/99LBYLP/74I8OGDcPT05Nly5axZ88eJk6cSHh4OH5+fowYMYIFCxbUOt+R0y4Wi4V///vfXHbZZfj4+NCjRw+++eab5rrMx9RuRj66dPTFw2alsKySQznFxARrnbeISHtVXF5J38fmueS1tz0xHh+P+r39vvzyy+zcuZP+/fvzxBNPALB161YAHnzwQZ5//nm6du1Khw4dSEpK4qKLLuLpp5/G09OTDz/8kAkTJpCQkEBsbOwxX+Pxxx/nueee45///CevvvoqU6dO5cCBAwQHBzf9lz2GdjPy4V5RyFUdtnOp9Td2pOa7OhwREZETCgwMxMPDAx8fHyIiIoiIiHA293riiSc477zz6NatG8HBwQwaNIhbbrmF/v3706NHD5588km6det2wpGM6dOnc/XVV9O9e3eeeeYZCgoKWLVq1Un9vdrNyAeHd/JEweMcdg/g05TpnNc33NURiYiIi3i729j2xHiXvXZzGD58eK2fCwoK+Pvf/873339PSkoKFRUVFBcXk5iYeNzzDBw40Pm9r68vAQEBpKenN0uMx9J+ko+wvtgtNjqSR8rBfUBPV0ckIiIuYrFY6j310VoduWrlvvvuY/78+Tz//PN0794db29vLr/8csrKyo57Hnd391o/WywW7HZ7s8dbU4OmXSorK3n00UeJi4vD29ubbt268eSTT9YqnjEMg8cee4zIyEi8vb0ZN24cu3btavbAG8zdm6KArub3qZtdG4uIiEg9eXh4UFlZecLjfvvtN6ZPn85ll13GgAEDiIiIYP/+/Sc/wEZoUPLxj3/8gzfeeIPXXnuN7du3849//IPnnnuOV1991XnMc889xyuvvMKbb77JypUr8fX1Zfz48ZSUlDR78A1lizSHlkLyd1BaceL/kCIiIq7WpUsXVq5cyf79+zl8+PAxRyV69OjBl19+yYYNG9i4cSPXXHPNSR/BaKwGJR+///47EydO5OKLL6ZLly5cfvnlnH/++c7CFMMweOmll3jkkUeYOHEiAwcO5MMPPyQ5OZm5c+eejPgbxCtmMAB9LPvZk17o2mBERETq4b777sNms9G3b19CQ0OPWcPxwgsv0KFDB0aPHs2ECRMYP348Q4cObeFo66dBE16jR4/m7bffZufOnfTs2ZONGzeybNkyXnjhBQD27dtHamoq48aNcz4nMDCQkSNHsnz5cq666qqjzllaWkppaanz57y8vMb+LidkiRwEQF/LAdal5dE3KuCkvZaIiEhz6NmzJ8uXL6913/Tp0486rkuXLixatKjWfTNmzKj185HTMHX1HMnJyWlUnA3RoOTjwQcfJC8vj969e2Oz2aisrOTpp59m6tSpAKSmpgIQHl57JUl4eLjzsSPNnDmTxx9/vDGxN1zEAAA6W9P54mAKDIlumdcVERERpwZNu3z22Wd8/PHHzJkzh3Xr1vHBBx/w/PPP88EHHzQ6gIceeojc3FznLSkpqdHnOiGfYAq8IgEoObjx5L2OiIiIHFODRj7uv/9+HnzwQef0yYABAzhw4AAzZ85k2rRpREREAJCWlkZkZKTzeWlpaQwePLjOc3p6euLp6dnI8BuuPLQfJKXgfXhri72miIiIVGvQyEdRURFWa+2n2Gw2ZzVtXFwcERERLFy40Pl4Xl4eK1euJD4+vhnCbTqf2CEAxJTtJreo3MXRiIiItD8NGvmYMGECTz/9NLGxsfTr14/169fzwgsvcMMNNwBmY5K7776bp556ih49ehAXF8ejjz5KVFQUkyZNOhnxN5hn9GDALDpNSMvntLiT17teREREjtag5OPVV1/l0Ucf5c9//jPp6elERUVxyy238NhjjzmPeeCBBygsLOTmm28mJyeHsWPH8tNPP+Hl5dXswTeKo9dHD8tBPks+rORDRESkhVmMhuzt2wLy8vIIDAwkNzeXgICTsBTWMCh+Khbvyjxe6/Eet0+d0vyvISIirUpJSQn79u0jLi6u9XwYPgUd7zo25P273exq62SxUNChDwDWtE0uDkZERKT9aX/JB2CLMqdeAvN21NlgRUREpK3o0qULL730kqvDqKVdJh8BXcx2sz3s+0jOdf2eMyIiIu1Ju0w+3DoNBhwrXlJyXBqLiIhIe9Mukw869qDc4o6fpYTkfTtcHY2IiEid3n77baKioo7anXbixInccMMN7Nmzh4kTJxIeHo6fnx8jRoxgwYIFLoq2/tpn8mFzJ8uvBwDlBze4NhYREWl5hgFlha65NaDW8IorriAzM5PFixc778vKyuKnn35i6tSpFBQUcNFFF7Fw4ULWr1/PBRdcwIQJE465821r0aA+H21JRWh/yN+GV6barIuItDvlRfBMlGte+6/J4OFbr0M7dOjAhRdeyJw5czj33HMB+N///kfHjh05++yzsVqtDBo0yHn8k08+yVdffcU333zD7bffflLCbw7tc+SD6jbrkcW7KK+0n+BoERER15g6dSpffPEFpaWlAHz88cdcddVVWK1WCgoKuO++++jTpw9BQUH4+fmxfft2jXy0VkFdh8ES6GvZz96MQnpF+Ls6JBERaSnuPuYIhKteuwEmTJiAYRh8//33jBgxgl9//ZUXX3wRgPvuu4/58+fz/PPP0717d7y9vbn88sspKys7GZE3m3abfFjC+2HHQpglh3X799ArYrCrQxIRkZZisdR76sPVvLy8mDx5Mh9//DG7d++mV69eDB1qtoz47bffmD59OpdddhkABQUF7N+/34XR1k+7nXbB049MzxgA8vevd3EwIiIixzZ16lS+//573nvvPaZOneq8v0ePHnz55Zds2LCBjRs3cs011xy1MqY1ar/JB9Ros77ZxZGIiIgc2znnnENwcDAJCQlcc801zvtfeOEFOnTowOjRo5kwYQLjx493joq0Zu122gXAFjkQUucRkLfT1aGIiIgck9VqJTn56BqVLl26sGjRolr3zZgxo9bPrXEapl2PfHToaq54iS3fR1FZhYujERERaR/adfLhHzsYgG6WZHYnZ7o2GBERkXaiXScfBERRYPHDzWIndc8mV0cjIiLSLrTv5MNi4bCv2Wa9JGmDa2MRERFpJ9p38gGUhZgrXtwOb3dxJCIiIu1Du08+PKMHAhBSuMvFkYiIyMlmNGBTNzlac12/dp98dOxmrofuat9PTlHrbkcrIiKN4+7uDkBRUZGLIzm1VbVtt9lsTTpPu+7zAeAb3R87FkIteazbv4+hfXu5OiQREWlmNpuNoKAg0tPTAfDx8cFisbg4qlOL3W4nIyMDHx8f3Nyalj60++QDD1/S3aKIqDhE1t71oORDRKRNioiIAHAmINJwVquV2NjYJiduSj6AHP+eRGQfojx5M3CVq8MREZGTwGKxEBkZSVhYGOXl5a4O55Tk4eGB1dr0ig0lH4A9vC9kL8Yne4erQxERkZPMZrM1uWZBmqbdF5wC+MaYbdbDi3erElpEROQkU/IBRPQcBkCccZCM3AIXRyMiItK2KfkAPEO6UIg3npYKDu7a7OpwRERE2jQlHwBWK6leXQHIPbDBtbGIiIi0cUo+HAqCepvfpG5xbSAiIiJtnJIPB2tEfwD883a6OBIREZG2TcmHQ4e4wQB0Kt2L3a4VLyIiIieLkg+H8O7mcttISyYpqckujkZERKTtUvLh4O7bgVRrGACpu9a6OBoREZG2S8lHDRk+PQAoStro4khERETaLiUfNZQEmyte3DK2uTgSERGRtkvJRw0enQYCEFywy8WRiIiItF0NSj66dOmCxWI56jZjxgwASkpKmDFjBiEhIfj5+TFlyhTS0tJOSuAnQ2i3oQDEVhygQjseioiInBQNSj5Wr15NSkqK8zZ//nwArrjiCgDuuecevv32Wz7//HOWLl1KcnIykydPbv6oT5KILn0pM9zwtpSRlrTb1eGIiIi0SW4NOTg0NLTWz88++yzdunXjzDPPJDc3l3fffZc5c+ZwzjnnADB79mz69OnDihUrGDVqVPNFfZJY3dxIsUXS2Z5EZtJ2OnXt4+qQRERE2pxG13yUlZXx0UcfccMNN2CxWFi7di3l5eWMGzfOeUzv3r2JjY1l+fLlxzxPaWkpeXl5tW6ulO0VA0BRijqdioiInAyNTj7mzp1LTk4O06dPByA1NRUPDw+CgoJqHRceHk5qauoxzzNz5kwCAwOdt5iYmMaG1CxKAuLMbzI17SIiInIyNDr5ePfdd7nwwguJiopqUgAPPfQQubm5zltSUlKTztdUlpBuAPjk73dpHCIiIm1Vg2o+qhw4cIAFCxbw5ZdfOu+LiIigrKyMnJycWqMfaWlpREREHPNcnp6eeHp6NiaMk8I3qhdshZBS1yZBIiIibVWjRj5mz55NWFgYF198sfO+YcOG4e7uzsKFC533JSQkkJiYSHx8fNMjbSEdO/cFINyeTkVZiYujERERaXsaPPJht9uZPXs206ZNw82t+umBgYHceOON3HvvvQQHBxMQEMAdd9xBfHz8KbHSpUpYZBeKDE98LKUcPJBAdI9Brg5JRESkTWnwyMeCBQtITEzkhhtuOOqxF198kUsuuYQpU6ZwxhlnEBERUWtq5lRgtVlJsZl1LNlJ210cjYiISNvT4JGP888/H8Mw6nzMy8uLWbNmMWvWrCYH5krZ3rFQuI/iVC23FRERaW7a26UOpQFdALBk7XVtICIiIm2Qko862EJ7AOBbsM/FkYiIiLQ9Sj7q4BvZC4CQ0oMujkRERKTtUfJRh9A4x3Jb4zDlJQUujkZERKRtUfJRh7DQKHINXwDS9u9wcTQiIiJti5KPOlhtVlLcOgGQc3Cbi6MRERFpW5R8HEOudywAJam7XByJiIhI26Lk4xhKA83dba1Ze1wciYiISNui5OMY3EK7A+BbeMDFkYiIiLQtSj6Owb9TbwBCtbutiIhIs1LycQxhnfsBEEwu5YXZLo5GRESk7VDycQxhoR3JMAIBSNuvFS8iIiLNRcnHMVgsFtIcy21zD6rXh4iISHNR8nEcuT6dAShN0+62IiIizUXJx3GUB5nLbW3Z2t1WRESkuSj5OA43x+62flpuKyIi0myUfBxHQCdzd9uwsoNgGC6ORkREpG1Q8nEc4V3M3W39KaQsL8PF0YiIiLQNSj6OIyw4iBQjBID0A1tdHI2IiEjboOTjOCwWC2nu5nLbPC23FRERaRZKPk4gz9dcbluWruW2IiIizUHJxwlUVO1um6MVLyIiIs1ByccJeId1BcCn8KCLIxEREWkblHycQEi02esjuDzFxZGIiIi0DUo+TiA6rjcAweSRla3dbUVERJpKyccJ+AR2JB8fAA7u04oXERGRplLyUQ9Z7pEAZB7c5eJIRERETn1KPuqhxC8agOJ0bTAnIiLSVEo+6qNDFwAMLbcVERFpMiUf9eAbZvb68Ck85OJIRERETn1KPuohJLonAGGVqeQUlbk4GhERkVObko96qGo0FmPJYFd6gYujERERObUp+aiPoFgAAixF7D+Y7OJgRERETm1KPurDw5dCtw4AZB3SBnMiIiJNoeSjnqqX2+5zcSQiIiKnNiUf9WTp0Nn8quW2IiIiTdLg5OPQoUNce+21hISE4O3tzYABA1izZo3zccMweOyxx4iMjMTb25tx48axa9ep3xnUN7wbAIGlKeQWl7s4GhERkVNXg5KP7OxsxowZg7u7Oz/++CPbtm3jX//6Fx06dHAe89xzz/HKK6/w5ptvsnLlSnx9fRk/fjwlJSXNHnxL8uxo9vqIsWSwWyteREREGs2tIQf/4x//ICYmhtmzZzvvi4uLc35vGAYvvfQSjzzyCBMnTgTgww8/JDw8nLlz53LVVVc1U9gu4FjxEmNJZ11aPsM6dzjBE0RERKQuDRr5+Oabbxg+fDhXXHEFYWFhDBkyhHfeecf5+L59+0hNTWXcuHHO+wIDAxk5ciTLly+v85ylpaXk5eXVurVKjhbr0ZbD7ErLd20sIiIip7AGJR979+7ljTfeoEePHsybN4/bbruNO++8kw8++ACA1NRUAMLDw2s9Lzw83PnYkWbOnElgYKDzFhMT05jf4+QLjMbAgo+llNTUg66ORkRE5JTVoOTDbrczdOhQnnnmGYYMGcLNN9/MTTfdxJtvvtnoAB566CFyc3Odt6SkpEaf66Ry86TcJwKAEu1uKyIi0mgNSj4iIyPp27dvrfv69OlDYmIiABER5ptzWlparWPS0tKcjx3J09OTgICAWrfWyhpsLrf1KTxIfolWvIiIiDRGg5KPMWPGkJCQUOu+nTt30rmz+aYcFxdHREQECxcudD6el5fHypUriY+Pb4ZwXcstuAugFS8iIiJN0aDk45577mHFihU888wz7N69mzlz5vD2228zY8YMACwWC3fffTdPPfUU33zzDZs3b+aPf/wjUVFRTJo06WTE37IcjcaiLRnsSlPyISIi0hgNWmo7YsQIvvrqKx566CGeeOIJ4uLieOmll5g6darzmAceeIDCwkJuvvlmcnJyGDt2LD/99BNeXl7NHnyLC6pOPn5N14oXERGRxrAYhmG4Ooia8vLyCAwMJDc3t/XVf+xfBu9fzD57OI/HfcT715/m6ohERERahYa8f2tvl4ZwNBrrZDnMntRcFwcjIiJyalLy0RABnTCsbnhYKinPTaGgtMLVEYmIiJxylHw0hNWGJTAaMFe87MsodHFAIiIipx4lHw3lKDqNsaSTmFXk4mBEREROPUo+Gsq5wVyGkg8REZFGUPLRUDV6fSj5EBERaTglHw0V1AWAGGsGiVmq+RAREWkoJR8NpZEPERGRJlHy0VCOmo9IMknPKaC80u7igERERE4tSj4ayi8cw80Lm8Ug3DhMck6xqyMSERE5pSj5aCiLBUtgDGB2Oj2QqakXERGRhlDy0RiORmNRZKruQ0REpIGUfDRGUPXIh5IPERGRhlHy0RiOaZcoy2ESNe0iIiLSIEo+GsMx7aKRDxERkYZT8tEYzpEPs+bDMAwXByQiInLqUPLRGDVGPgpKy8kuKndxQCIiIqcOJR+NEdAJsOBlKSeEPA5kqs26iIhIfSn5aAw3D/CPAKqnXkRERKR+lHw0Vo1GY1rxIiIiUn9KPhpLK15EREQaRclHY9VoNHZAyYeIiEi9KflorBrLbZOUfIiIiNSbko/Gck67ZJCaV0JJeaWLAxIRETk1KPloLGfBaSaGAQezi10ckIiIyKlByUdjOUY+gi35eFOiqRcREZF6UvLRWN5B4BkAmHUfajQmIiJSP0o+mqLWcltNu4iIiNSHko+mqLXBnEY+RERE6kPJR1Oo0ZiIiEiDKfloiqCqkQ8z+TAMw8UBiYiItH5KPpqixnLbknI7GfmlLg5IRESk9VPy0RSO5CPWlgmgNusiIiL1oOSjKRw1H2FGJlbs2t1WRESkHpR8NIV/BFjdcKOSMLJVdCoiIlIPSj6awmqDgCigarmtkg8REZETaVDy8fe//x2LxVLr1rt3b+fjJSUlzJgxg5CQEPz8/JgyZQppaWnNHnSr4qj7iLYcVot1ERGRemjwyEe/fv1ISUlx3pYtW+Z87J577uHbb7/l888/Z+nSpSQnJzN58uRmDbjVCaxebpuUreRDRETkRNwa/AQ3NyIiIo66Pzc3l3fffZc5c+ZwzjnnADB79mz69OnDihUrGDVqVNOjbY1qNBpLyyultKISTzebi4MSERFpvRo88rFr1y6ioqLo2rUrU6dOJTExEYC1a9dSXl7OuHHjnMf27t2b2NhYli9f3nwRtzaORmMxtiwADmVrjxcREZHjadDIx8iRI3n//ffp1asXKSkpPP7445x++uls2bKF1NRUPDw8CAoKqvWc8PBwUlNTj3nO0tJSSkurm3Pl5eU17DdwNcfIR2dHr4+D2cV0DfVzZUQiIiKtWoOSjwsvvND5/cCBAxk5ciSdO3fms88+w9vbu1EBzJw5k8cff7xRz20VAmMBCDcyADP5EBERkWNr0lLboKAgevbsye7du4mIiKCsrIycnJxax6SlpdVZI1LloYceIjc313lLSkpqSkgtL7ATAD5GEf4UqehURETkBJqUfBQUFLBnzx4iIyMZNmwY7u7uLFy40Pl4QkICiYmJxMfHH/Mcnp6eBAQE1LqdUjx8wTsYMItONfIhIiJyfA2adrnvvvuYMGECnTt3Jjk5mb/97W/YbDauvvpqAgMDufHGG7n33nsJDg4mICCAO+64g/j4+La70qVKUAwUZ5nLbdXrQ0RE5LgalHwcPHiQq6++mszMTEJDQxk7diwrVqwgNDQUgBdffBGr1cqUKVMoLS1l/PjxvP766ycl8FYlMAZSNtLJcphNGvkQERE5rgYlH59++ulxH/fy8mLWrFnMmjWrSUGdchyNxjpZMjlcUEpxWSXeHur1ISIiUhft7dIcjlhueyhHUy8iIiLHouSjOXToDECc22EAkrI09SIiInIsSj6aQ4c4AKKNFAAOarmtiIjIMSn5aA7BZvLhZ88ngAKSVHQqIiJyTEo+moOHL/iFA9DZkq6RDxERkeNQ8tFcgrsC0MWSqkZjIiIix6Hko7k46j5iLelqNCYiInIcSj6aS42Rj+yicgpKK1wckIiISOuk5KO5OIpOu7pV7W6r0Q8REZG6KPloLo7ko4slDYCD6vUhIiJSJyUfzcVR8xFiZOFNCUka+RAREamTko/m4hMMXkGAWXSqFS8iIiJ1U/LRnBxTL50taVrxIiIicgxKPpqTY8VLZ0uaRj5ERESOQclHc+pQY+RDNR8iIiJ1UvLRnGqMfOSXVJBbXO7igERERFofJR/NyVHzEWdLB9TrQ0REpC5KPpqTY+QjksO4U0GSen2IiIgcRclHc/ILB3cfbNjpZMnQyIeIiEgdlHw0J4vFWXTaRSteRERE6qTko7kFV+1um6aRDxERkToo+WhuNfZ4Uc2HiIjI0ZR8NLcOtUc+DMNwcUAiIiKti5KP5uZY8dLFkkZhWSXZRer1ISIiUpOSj+ZWVfNhTceKnUTt8SIiIlKLko/mFhANVnc8qCCCLJbtynB1RCIiIq2Kko/mZnODoFgAOlvT+HZjiosDEhERaV2UfJwMjrqPrtY0EtLySUjNd3FAIiIirYeSj5PBUfcxNsRMOr7blOzKaERERFoVJR8ng2PkY5BvNgDfbkzWklsREREHJR8ng6PXR3hFMl7uVvZnFrHlUJ6LgxIREWkdlHycDI6RD1vOfs7tFQbAt5p6ERERAZR8nBwdOgMWKCtgSm8PAL7flILdrqkXERERJR8ng5snBEYDMCY4D18PG4dyilmflO3iwERERFxPycfJEtoLAM/MHZzfLwJAPT9ERERQ8nHyRAwwv6ZuZsKgSAC+25RCpaZeRESknWtS8vHss89isVi4++67nfeVlJQwY8YMQkJC8PPzY8qUKaSlpTU1zlOPM/nYxNjuoQR6u3O4oJSVezNdG5eIiIiLNTr5WL16NW+99RYDBw6sdf8999zDt99+y+eff87SpUtJTk5m8uTJTQ70lBPhuC5pW/Gw2Lmwvzn18s1GrXoREZH2rVHJR0FBAVOnTuWdd96hQ4cOzvtzc3N59913eeGFFzjnnHMYNmwYs2fP5vfff2fFihXNFvQpIbgruPtCRQlk7WG8I/lYuS/LxYGJiIi4VqOSjxkzZnDxxRczbty4WvevXbuW8vLyWvf37t2b2NhYli9f3rRITzVWG4T3M79P2UT3UD8ADmUXa8mtiIi0a24NfcKnn37KunXrWL169VGPpaam4uHhQVBQUK37w8PDSU1NrfN8paWllJaWOn/Oy2tDnUAjBsDBVZC6ich+U7BZLZRV2knLLyEy0NvV0YmIiLhEg0Y+kpKSuOuuu/j444/x8vJqlgBmzpxJYGCg8xYTE9Ms520VIh11H6mbcbNZiQoyr1lSVrELgxIREXGtBiUfa9euJT09naFDh+Lm5oabmxtLly7llVdewc3NjfDwcMrKysjJyan1vLS0NCIiIuo850MPPURubq7zlpSU1OhfptWpsdwWwyCmgw8ASVlFLgxKRETEtRo07XLuueeyefPmWvddf/319O7dm7/85S/ExMTg7u7OwoULmTJlCgAJCQkkJiYSHx9f5zk9PT3x9PRsZPitXFhfsFih6DDkpziSj0ySspV8iIhI+9Wg5MPf35/+/fvXus/X15eQkBDn/TfeeCP33nsvwcHBBAQEcMcddxAfH8+oUaOaL+pThbs3dOwJGTsgdTMxjg3nNO0iIiLtWYMLTk/kxRdfxGq1MmXKFEpLSxk/fjyvv/56c7/MqSNioCP52ERMsJmgaeRDRETasyYnH0uWLKn1s5eXF7NmzWLWrFlNPXXbEDEANn8GqZuJ7mzWfBxUzYeIiLRj2tvlZKsqOk3ZREywubw2Ja+Esgq7C4MSERFxHSUfJ1tVm/XsfYS6l+LlbsUwIDlHdR8iItI+Kfk42XxDIKATAJa0bURXLbdV3YeIiLRTSj5aQo1+HzEdzKkXrXgREZH2SslHS3AmHxuJCdbIh4iItG9KPlpCRHWbdXU5FRGR9k7JR0uoGvlI305skLm6OSlb0y4iItI+KfloCUGdwTMAKsvoSjKgXh8iItJ+KfloCVarc/SjU+luADILyygsrXBlVCIiIi6h5KOlOJIPn8xtBHiZUy8HNfUiIiLtkJKPluJc8bKpesWLpl5ERKQdUvLRUsL6ml8zEqpXvGi5rYiItENKPlpKSHfza2E63QMqATUaExGR9knJR0vxCgC/cAD6eGYAGvkQEZH2SclHS3KMfsQ5ltuq5kNERNojJR8tKaQbABEVBwFztYthGK6MSEREpMUp+WhJIT0ACCxKBKCgtIKconJXRiQiItLilHy0JMe0iy1rN6H+noDqPkREpP1R8tGSOpojH2TuISbIC9CKFxERaX+UfLSkoM5gsUF5If0DzBEPjXyIiEh7o+SjJbl5QIfOAPSrWm6rFS8iItLOKPloaY6i027WFACStL+LiIi0M0o+Wpqj6DSq4hAABzXyISIi7YySj5bm6PXRocRcbnswuxi7Xb0+RESk/VDy0dIcK1688vZhs1ooq7STnl/q4qBERERajpKPluaYdrFk7ycmwAZoxYuIiLQvSj5amn8kuPuCUcnQgDwAEjOVfIiISPuh5KOlWSzOuo/BPocB2Hu4wJURiYiItCglH67gmHrp7Z4GwO50JR8iItJ+KPlwBUfRaYw9GVDyISIi7YuSD1dwjHwElxwA4EBmEeWVdldGJCIi0mKUfLiCo+bDI3cfPh42KuwGBzILXRyUiIhIy1Dy4QpVy20L0ugfYv4n2J2u5ENERNoHJR+u4BUIvmEAjArMAmBPhuo+RESkfVDy4SqO0Y/+Xubutio6FRGR9kLJh6t0NJOPrpZUQMmHiIi0H0o+XMUx8hFengSY0y7aYE5ERNqDBiUfb7zxBgMHDiQgIICAgADi4+P58ccfnY+XlJQwY8YMQkJC8PPzY8qUKaSlpTV70G1CiNnrw69gP25WC0VllaTmlbg4KBERkZOvQclHdHQ0zz77LGvXrmXNmjWcc845TJw4ka1btwJwzz338O233/L555+zdOlSkpOTmTx58kkJ/JRXteIlaw+dg70BTb2IiEj74NaQgydMmFDr56effpo33niDFStWEB0dzbvvvsucOXM455xzAJg9ezZ9+vRhxYoVjBo1qvmibgs6dAGLDcoKGBZZyp7DZvJxRs9QV0cmIiJyUjW65qOyspJPP/2UwsJC4uPjWbt2LeXl5YwbN855TO/evYmNjWX58uXHPE9paSl5eXm1bu2Cmwd06AzAUN9MAHZrua2IiLQDDU4+Nm/ejJ+fH56entx666189dVX9O3bl9TUVDw8PAgKCqp1fHh4OKmpqcc838yZMwkMDHTeYmJiGvxLnLIcdR99bAcBTbuIiEj70ODko1evXmzYsIGVK1dy2223MW3aNLZt29boAB566CFyc3Odt6SkpEaf65QTNQSA2JIEAPZq5ENERNqBBtV8AHh4eNC9u1ksOWzYMFavXs3LL7/MlVdeSVlZGTk5ObVGP9LS0oiIiDjm+Tw9PfH09Gx45G1Bp2EABGZtAqZwuKCMnKIygnw8XBuXiIjISdTkPh92u53S0lKGDRuGu7s7CxcudD6WkJBAYmIi8fHxTX2ZtqnTUACsmbvoEVAJaOpFRETavgaNfDz00ENceOGFxMbGkp+fz5w5c1iyZAnz5s0jMDCQG2+8kXvvvZfg4GACAgK44447iI+P10qXY/HtCEGdIecA5wQmsysvht3pBQzvEuzqyERERE6aBiUf6enp/PGPfyQlJYXAwEAGDhzIvHnzOO+88wB48cUXsVqtTJkyhdLSUsaPH8/rr79+UgJvMzoNg5wDnOa+j7eI0ciHiIi0eQ1KPt59993jPu7l5cWsWbOYNWtWk4JqVzoNg61f0rNyJ3CGdrcVEZE2T3u7uJqj6DQ8z+wSq14fIiLS1in5cLXIgWCx4VGcRjhZHMwupqS80tVRiYiInDRKPlzNwxfC+gIwxns/hoGmXkREpE1T8tEaOJbcnu59ANByWxERaduUfLQGjrqPgZY9AOzJKHRlNCIiIieVko/WwJF8xJQkYMXOHo18iIhIG6bkozUI7Q3uPnhUFtLVkqxpFxERadOUfLQGNjeIHAzAYOsedmcUkFtc7tqYREREThIlH62Fo+j0DJ9EKu0Gv+8+7OKARERETg4lH62Fo+5juPs+AJbuzHBlNCIiIieNko/WwpF8RBTvxpMyftmZgWEYLg5KRESk+Sn5aC2CYsGnI1ajgkFuSSTnlqjwVERE2iQlH62FxeIc/ZgQmgJo6kVERNomJR+tiSP5iPfcDyj5EBGRtknJR2viSD46F20BDFbuy6K4TJvMiYhI26LkozWJOQ3cfXDPS+SigH2UVdhZsS/T1VGJiIg0KyUfrYlXAAy4AoBbvBcDsDRBUy8iItK2KPlobUb8CYABeUsJJZtfdin5EBGRtkXJR2sTORBiRmI1KrjGbTF7MwpJyipydVQiIiLNRslHazTiJgCmeSzGjQqtehERkTZFyUdr1PdS8A0l2J7JOOu6WsmHYRhsSMphV1p+019n5zxY8Hewa0WNiIi0HCUfrZGbJwz9IwB/tP3M8j2ZpOWV8M4vexn3wlImzfqNi19Z1rTN54qy4H83wLIXYdfPzRR48/ty3UHu+e8GSiuUIImItBVKPlqrYddjWKyMtm0jomw/o2Yu5OkftrMnoxCLBcoq7dz8n7VsOZTbuPOvfBPKHO3bD61tvrib2XM/JfDV+kP8vkdLjkVE2golH61VUAyWXhcBcK1tAYYBg6IDeeayAax95Dziu4ZQUFrB9NmrOJBZ2LBzl+TCijerf26lyUdeSTmpeSUA7Mto4O8oIiKtlpKP1mzEjQBc4/kbP906hK9vH8s1I2MJ9vXgrT8Oo29kAIcLyrju3VVk5JcCkFlQyn9WHOCad1Zw84drKKuwH33eVW9DaS54BZk/H1oHrXAH3Zob6+07rORDRKStcHN1AHIccWdBSHc8MnfT+8DH0OV+50MBXu68f8MILn9jOYlZRVz37kpC/T35fU8mlfbqRGL1/izGdO9Yfc7SAlj+uvn9BTPh27uhJAey9kJIt5b4reptd1p18rG/oaM7IiLSamnkozWzWuH0+8zvl8yE/b/VejjM34sPbziNjn4e7EjN59ddh6m0GwyMDqRPZAAAK/ceUSux5l0ozoLgrjDgD2ZfETBHP1qZ3Rka+RARaYuUfLR2g66CgVeCUWmuTilIr/Vwl46+fHjDSMb3C+e+83uy5L6z+Ob2sfwxvjMAK/ZlVR9cXgy/v2p+f/r/gc0NooaaPye3vuSj5nLiQznFlJRrxYuISFug5KO1s1jgkhchtDcUpMIXNx7Vl6Nv0WreCpjN7X1L6dLRF4CRccEAbEjKqX7TXvsBFGZAUKyZ0IBzJ93WWHS6q0bNh2GgTq8iIm2Eko9TgYcv/OFDcPeFfb+YUzAAqZvhP5fBR5Nh/Ufw6TVmTQcQ19GXUH9PyirsbEjKgYpS+O1l83lj7wGbu/l9J8fIR8omqCxv2d/rOIrKKjiYXQxApyBvQFMvIiJthZKPU0VoL5jgSB5++Sd8cjW8eTrsWQRWd/DuADkHYP6jAFgsFufox8q9WbDoKchPBv8oGDy1+rzB3cAzECqKIX17S/9Wx7Qn3Uw0Qnw9GNq5A6DkQ0SkrVDycSoZeAUMN5ffkvADYEC/yXD7KrjiA/P+Ne+ZCQkwsmsIAN5bPoLfXzEfH/+U2UG1itUKUYPN71vR1MuudLPeo3uYH3EhPoBWvIiItBVKPk41F8yEvpOg+3nwp4VwxWxz5UrXM+G0m81jvr4dSnIZFRdMvHUr12c7ikzP/Av0n3L0OavqPlpR0WlVvUePcD/iQs06Fo18iIi0Derzcapx84Q/fFD3Y+P+DrsXmD07fvor3cfcyVseL+FOJZlxEwg566G6n1dV99Gcy223zjXrSnpf3KinVzUY6xHmT5cQJR8iIm2JRj7aEg9fmPQGYIENH2F5/2ICKGStvQefdXrQXDlTl6qRj/TtUNYMb/DJG+DzafDpVMje36hTVCcffsQ5VvCk5ZVSVFbR9PhERMSllHy0NbGjYPTt5veFGeR7RXFz2b38fuA4SUVAFPhFmL1EUjbVfmzvUvj2LijOrn8MS59zfGPAug8bFD5ASXmlc7+a7uF+BPl40MHHXJ2z/7CW24qInOoalHzMnDmTESNG4O/vT1hYGJMmTSIhIaHWMSUlJcyYMYOQkBD8/PyYMmUKaWlpzRq0nMDZj0D0aeAfSfqED8kkkLUHsimvrL3Py+70An7akoJhGHXXfeQlw3+vg7Xvw5J/nPBlk3OKmfvjj5DwvfM++7qPGryEd9/hQuwGBHq7E+pnFsdW9S9p7NRLTlEZn65KpLSiiY3KCjLgs2lmUiYiIo3SoORj6dKlzJgxgxUrVjB//nzKy8s5//zzKSysfkO45557+Pbbb/n8889ZunQpycnJTJ48udkDl+Nw94Ib5sHdm4nrM5wgH3eKyirZfCjXeUhyTjFXvPk7t360ju83p0CnIeYDVSteDAO+udPcgA5g7WzITz3qpUrKK/nv6kQmzfqN0c8uwuv3fwHwQ+VpZBiBWAvTHCtz6m9XjSkXi2OqKM5R99HYFS8Pz93Cg19uZvZv+xv1fKfV/4Ztc82i3kpNAYmINEaDko+ffvqJ6dOn069fPwYNGsT7779PYmIia9eab1i5ubm8++67vPDCC5xzzjkMGzaM2bNn8/vvv7NixYqT8gvIMVitYHPHarUwokuNfh9AeaWdOz5ZT3aROSLx4vydVEYe0el0/Uewez7YPCG0D1SUwLKXnKc/XFDKSwt2MubZRfzli81sSMqhjzWRC2yrMbBQNPoBPqs8E4CKVe81KPTdjrbqPcL9nPfFNWHkI6eojPlbzdG3X3ZmNPj5texdYn7NTYTt3zTtXCIi7VSTaj5yc81PxcHB5pvb2rVrKS8vZ9y4cc5jevfuTWxsLMuXL6/zHKWlpeTl5dW6SfNyNhvbZ24y9/zPCaw9kI2/pxuB3u7sySjkh8xw8+Ds/Wbdx7y/mj+f87C5vBdg7WwqcpL5+zdbGf3sIl5asIvMwjKiAr3460W9+arfMgAs/SZx2fhx/BZ4CXbDgtv+JeYKnHqqGvnoFlqdfDRl2uW7TSmUOaac1h7IbvzUS0keHFxd/fPy18wRIhERaZBGJx92u527776bMWPG0L9/fwBSU1Px8PAgKCio1rHh4eGkph49ZA9mHUlgYKDzFhMT09iQ5BhGOZqNrdmfzfxtaby11EwEnrt8ILec2RWAf/6SjhHczXzCnCuhNA+iR0D87dD1LIgZBRUlbPj077z/+37KKuwMig7k1auH8MsDZ3NzrxK8dn1nPv+MB7BZLVx53lh+tQ8AoHTV7HrH61zpEu7vvK9q5GN/I5KPL9YddH5fWmFnQ2JOg88BwIHfzaJc/0hzROjQWkjUiJ6ISEM1OvmYMWMGW7Zs4dNPP21SAA899BC5ubnOW1JSUpPOJ0frExmAv5cbBaUVzJhjFpROH92FCwdEMi2+CyG+HiRmFXHAq5f5hPxkcPMyl+1abeYS3bMeBKB/ypeEks2/rhjE3BljmDAoCjeb1Wz5DtB3IoT3BeCSgVEs8jP7fFSu/Qgqyk4Ya3ml3Tm60SPs6JGPzMIycovrX8C6N6OA9Yk52KwWRnU1R4CW782s9/Nrn2yJ+bXnBeZuw2COfoiISIM0Kvm4/fbb+e6771i8eDHR0dHO+yMiIigrKyMnJ6fW8WlpaURERNR5Lk9PTwICAmrdpHnZatR9VI1YPHRRbwB8Pd247SxzxGNueo3/Ruc8Ch17OH882OE01tMLL0s5r8YsZcqwaGcxKCkbzaZiYHZRrfG6w8+/mjQjCJ/yLIo2f33CWA9kFlJhN/D1sBEZ6OW838/TjVB/c+VLvUY/SnKhopQv1x0C4IweHZkwKAqAFU1NPrqeZY4IAez4HjL3NO58IiLtVIOSD8MwuP322/nqq69YtGgRcXFxtR4fNmwY7u7uLFy40HlfQkICiYmJxMfHN0/E0ihVn/r9vdx47ZqheLrZnI9dO6ozYf6efFPYl0qLG8SdCaNucz5eVmHn9k828HyZuWppZPY3kJNkvvHOuRLePgswoM8ECO9X63UvGhTLfM/zAchc8tYJ49yVZk65dA/3r05uHOq94iUnEV4aiPGfy/hqvZl8TB4aTbxj+mldYg4l5Q2s+8hPhYztgAXizoDQnuYICAYsn9Wwc4mItHMNSj5mzJjBRx99xJw5c/D39yc1NZXU1FSKi82tzwMDA7nxxhu59957Wbx4MWvXruX6668nPj6eUaNGnZRfQOrn6tNiuW5UZ2ZPH0FMsE+tx7zcbdx+Tnf2GlGMt/2bkqs+N6dbHJ79cQcbknLY7DGY0sgRWCpK4JUh8Ok1sPMnMOzQeQxc8OxRr2uzWog4+2bshoWY3NVkJx1/59yay2yPVO8VLyvfgpIcLAd+oywnGX8vN87rG05cR1/C/D0pq7CzLrEBTdOguq9H5CDwMRM55+jHhjlQlNWw84mItGMN2tvljTfeAOCss86qdf/s2bOZPn06AC+++CJWq5UpU6ZQWlrK+PHjef3115slWGk8fy93npzU/5iPXzkihjeX7GF3Ltw6ZyPh/l5U2A2Kyyv4YbNZLPyvPwzB0/Nh+M8ksJeDT0cYfDUM+aM5EnAM54wczuqFwxhZsYbP33qSWe7TiQz0IiLQiyExHbj1rK7OkZjjJR/1WvFSWgDr/uP8caR1B/4DB+Llbp4/vlsIX29IZsXeLEZ363js8xyp5pSLM6CxEDkYUjbA6nfhzPvrfz4RkXasQcmHUY9lhV5eXsyaNYtZszQUfSrxdLNx57k9ePDLzSxJOLoXxs1ndOW8vuFAOFw1xxzt6DEe3DxOeG6r1YLv2FtgyRqutC3mxeIp7CguZ0dqPksSMvhlVwZvXDuUMH+vGitd6hr5MEdsjlvzsfGT6sZowEjrdnoPra5Liu/qSD72ZMJ5JwzdZBh1Jx8WC4y+A764EVa9ZX7v7lXXGVh7wFxp9OezuxHg5V7PFxYRaZu0q604/WF4DOV2g8P5pbjbLLjZrLhZLUR38Oa8vjWKURuxU23/My7H2PgMgdn7WHJ+OjuiL2dfRgEvzN/J2gPZTHztN964dhh7Mhw1H6H+R50jrqOZkOw7XIhhGEfVhGC3wwpzdC49NJ6wjOWMcd9JXOcOzkOqlh2vT8qmuKwSbw8bJ3R4l7kCyOZp7p1TU9+JsODvkJsES56B85446umGYXD/5xvZe7iQvRkFvHXdsKNjF2nlyirsXPn2ckL9PHn7j8NdHY6c4rSxnDhZrRauG9WZe87rye3n9ODWM7vxp9O7ckH/SGzWJr5ZWq1YRt4CQPj29zmzR0emj4lj7owxdA31JSW3hMvf+J2yCjte7lY6dfA+6hSdQ8yRj7ySCrIK61i2u3sBZO0Bz0BmupkFs12NRCw16jE6h/gQGehFeaVR/7qPqlGP2FHgfkRcNne40LHvzW+vwIGjm+mtPZDNXsdozc/b0nj/9/31e12RVmRHah7rE3P4eVsa6Xklrg5HTnFKPqTlDL4GPPwgYwfsMws4u4b6MXfGGM7uFUqF3ZzW6xbqV2ey4+VuI8qx/LbOFS8rzNqi/H5XM3e/Gwl2x3TLgd+ch1gsFufox/I99VxyW9eUS029L4bB1wIGfHULlObXevjzNWaTs6rYn/lhO5sO5tTvtduKuX+G/1xWr14v0jpVrUQDau0TJdIYSj6k5XgFmgkImCtSHAK83Pn3tBHOfiNjezgKQcuKIHElJK1yHhsXWlV0WlT73OnbYe9isFh5t+w8DAMS/R2b5R34vdahVUtu69Xvo7IC9v9qfn+s5APMFvSBsZBzAOY97Ly7qKzC3LgPePHKwYzvF055pcHtc9aTV9Kw3X5PWYWZsOFj2LMIEn8/8fHSKlUVgwNsOqjkQ5pGyYe0rNNuNr8m/AhZ+5x326wW/jKuK1snZfBg2Sx4YwzMjIb3zod3z4MtXwLQJaQq+Siofd6VbwJQ1v1C3tpo7jYbPdixx9CBZbUOje9mJh8bD+ZQVHb8nWl/WfozlOZheAWZy2yPxSsAJjlWda37ABJ+AuCnLakUlFbQOcSH0+KCeW7KIKI7eJOYVcSDX2yqVxH3KS91Y/X3exad/NcrzoFtX4O9kXv4SJ12p1eP6G3RyIc0kZIPaVkde0C3cwHD3J6+SmEm/GcSvj/dhWX9fyBti7mPioej8PS7eyAvucYeLzVGPoqyYKPZ5n+u56UUl1fSv1MAvUddYD6eugWKq+s7ojt40ynIm/JKgzX7j133kVNUxppFZtKTETqyVu+TOsWdDqNmmN9/cwekbGLrb98xxfoL/+j4E5bfXibQWsxr1wzF3Wbhh82pfLTiwImv2akuZVP193sWN/vpv9+Uwuaan8R/fgQ++yOs/8+xnyQNVmvkQ8mHNJGSD2l5I281v677j9mXI20bvHOWWZvhGQBj74UrP4Z7tsFf9kHUECjJgbl/pnuoWXT6y64MkrIcCciKN6CihMrwgTyz1VzZctuZ3bH4R0BId8CotQFczbqP4029zN+WxmjrFgC+zT92H5Nazn0MQntDYTq8dTqPZv6Ff3m8yagDb8KCv8H7FzO4Qxl/ucBsb//PeQmUVdjrd+5TVeqm2t8XHm62U689kM2MOeuYPntV9XWsqtFJXNlsr9MUuUXlXPvvlbwwf6dLXr/SbvD+b/vYmZZ//AMPrYO8lDofKimvJDGrOuHPyC8lTUWn0gRKPqTldR8HwV3Nfhzf3WNOq+QkQoc4uHE+jPsb9LkEAjuZq0kmvwNu3rB3Madnf8WQ2CDySyq4fc46Khc+Db88B8CvHa8kp7iCLiE+XNDfsTS48xjza42iU6ieejneJnOHV33GKKvZkfWDtK71G2p294LL3gLPQMqtnuyxR7LZcygMuRZ8Q80333fP5/o+0NHPk7ySisbvNVNPRWUV3PXpei56+VcOZhed+AnNrWrkw+pY2V+VHDSD+dvSAHPDwUU70iH3kLnsGSB1c7O9TlO8umgXy3Yf5vXFu8kpavmC2y/XHeTv327j+tmrj53oJq+Hd86B9y8y65yOsDu9AMOADj7u9HLsNq26D2kKJR/S8qxWOM1cdsvmz6CsALqcDjctgrDeRx/fsQec/yQAtoV/543zfejoZef6tKex/WomHpWj7+ahXeZzbzmzW/VqmarkY3/t5GNU12CGWnZiPbSG9PyjP8EVJW1kWpq5hHau1yQSjXDe+XVv/X6/qMHY79/LWe6fcG7Zv9h74UcwcRbcMA86dIHsfdhmj2danPmP97ytqfU7b31VVsDBNVB4mMyCUq5+ZyVfb0hmW0oe9/53I5X2FqwzKS2AzN3m9/0vN78ea+rl0DpY855ZaFxPi3ekO7//39qDkFRjtCNjh8tX1yRlFfHhcnNqrcJuOJOllvTjFvPv61BOMZ+tOcau4Rv/CxiQtRe2H70BpLP5X5g/A6IDAa14kaZR8iGuMfgac/ULwPAb4bqvqvdMqcuIP5kjJhUlRCy4g/kdX2SS7XfKDRvbRzzNl8F/IiWvlDB/TyYP7VT9vC6O5CNlY60lsNGZy/mf5+N84f4Yhz69p/abVGEmfHI1PpZS1tgG0X3qCwB8tymFQznF9fr1VuzP4VBuCf5ebozv5xiFCekGN/wM4QOgMJ3b9t3BKOs25m1Na56EwDBg13x4YzT8+1zsL/Rj+UvXUnhwC0E+7vh62Fi1P4u3fjnGLrwluZB7sOlx1JS2FTDALwIG/sG8b+9iM9aayorgoynmSNjrI2HXghOe+lBOMQlp+VT1a1uSkE7x3hqraezlZgLiQs/NS6Cs0o6Hzfyn9ofNdU9rnCwFpRUs21U9zfXaot1Hb6por4StX1X//NsrR/332eUoNu0e7seATo7ko70tF5dmpeRDXMMrwHwjnv49XPKCOb1yPBaLOXrgHQypm+lweC0lNj+mlz/A1Wt78tpi89P1jWPjau3YS2A0BHU2i1erPhXnJMIXf8KK+Q/skENzMGZfaN5fWQ6fT8On6BAH7GH8NuSf9I8JYXS3ECrtBrOX7Tsysjp9vtZ8E58wKMq5rwwA/uFw/ffQ5XTcKgqZ5fEK+QX5rG/oRndHStsGH02Gjy+HwwnYbZ5YK0u4pOJnFng+wG+dZvHv4YmMtW5myfxv2b1xmZkYbPocvv8/eGMsPNsZXuxv7lbcXKrqPSIHQufRZpfYvENm19iaNv0Xih3N4HIS4eMp8Pn1kH/skYKqUY9hsR0YGB1Ihd2gYLdjhMtiq/36LrAxKYdvNyZjscDzfzBXSi3bfZjc4pZbYr0kIZ2ySjuxwWZzvdS8EuasTKx9UOJyKEg1663cvMy9io6Ypqzq8dEjzK/WyEejV2sdWmsWiR9vRVLWXnPVWHtYEdYOKfkQ1wnrbW7OVl/+EXDpK2CxQlAs1j/9TF7kWHKKyjmQWYS/lxvXjIw9+nk1p17KS8yVEMVZ2COH8KDlXnINHyyH1sCbp5uP7f+VQsOLP5Xfx1mDegFw0xldAfhkVeJRbx65ReUsSUjny3UH+feve/nnvB38uMX8hHvFsGiO4hUIU/8HgbGEkMeltt/5acsxpl4yEiDzGCMVVZb8A94cYy5jtbqT0u8mRpe/weWlj/GbezwGFnyTlhC/7n4+8pjJZ+5/p/tXF5sjJF/+yVx1lLYZMMzbt3c33y69KeYy2x8Ph3HBrNVkBDt6r9RccmsYzrb4nP2IuVuwxQpbv4TXRsDOn+s8dVXycXbvMKYMjcabEoLzHCMdfS4xv7qo7sMwDJ7+wawXmjwkmksHRdEz3I/ySoMFLTj1UvV3dWH/CO44pwcAry/ZQ3FZjTf9LV+YX/teCoOuNr///dVa56k57dI3MgCb1cLhgjJSG1N0enAtvHeh2ZDv48vr/lvb9Bm8Pho+uVKrltooJR9yaukzAe5YBzNW4RHZj9euGYKfp1nI+Mf4zvjXtWlblxpFpz/9xSyu8+6A9coPCR55JReXPcMe957mipqEHwC4p/w2CgK6M9DxKe+snqH0CPOjsKyST1eZnxzLK+28u2wfY59bxPTZq7n3s4089f12Zi3eQ0m5nZ7hfgyOCar793D3gtP+BMD1tnn8tCXl6E+RWXvNhOids6Ekr+7z5Dj2lDHs0OdS9ly5iPO3nkdqmRceXccw8P++xXLnOhh5G3QaTkXHviQSQYoRTJEtAKKGwqg/wxUfwF2bqlfq/PiXev8nOS7HyMPXaR3ZkZrPv5PjAMjeMq/6mD2L4HCCuax65C0w/mm4abG5yqk0F36476hPvyXllfy2x5xOOKd3GJcOimKY215s2Cn3jYSeFzpe30w+knOKycgvbVjslRXmp/MTrM5JSM0/6twLtqezal8Wnm5W/u98c6XUhf0jAZyJ6clWWlHp3CRyfP8IrhgeTUywN4cLSvlw+X7zoMpysycKQP8pED8DsMDOn8zE13Geqo7CPcL98HK3OXedbnDRae4h+PRqqHRcrz2L4O0zq4uSK8rg+/vgy5ugwjHFueRZ80PDKWJnWj7ZdW3/ILUo+ZBTT3Ccc4+VziG+vDttONeP6cJtZ3Wv+/iqkY+klbD2fcACU/4NQbFcMzKWZMK4IP8RcgbeCG5efBd2Kz/bRzC+X4RzAziLxeIc/Zj9234WbEvjgpd+4cnvtpFfUkFMsDdju3fk0kFRTB/dhXvP68mb155gA7kh12G4edPXeoCo3PVsTT4iwVj0lPmPdElu9RvEkTZ/bn7tcjpJ573F1f9LJ7+kguGdO/DutBFmMhbcFS58Fm5aiNvty9l/zTLiS1+jb+GbLD7jv2Z31n6ToENnmPi6Oeqw+TOzEdwRft6aysWv/MquEy3bBPONLd389L/V6Ez3MD/W2MzpB/ek37l59nLzPFWjHkOuNafjAKIGm1NyNk+za+wRtRvL92ZSUm4nMtCL3hH+dPD14IqwZAB2evSDiAHmgambWb77MGc9v4RLXv2V0ooGNB5bMcv8dP7dPcc8ZOH2NMa/9AujZi7khvdX892mZApLK3j2R/P3vmFsHFFB5t/qRQPM5OOXnYfJb4Hutr/vzqSgtIIwf08GRwfhbrNyp2P0482leygorTC3OSjKBJ+O0OUMs7i710XmCZabO5PvO1yI3QB/LzfC/D0BnEl5g5qNlRXBp9dAQRqE9a0uwM5JhHfPh1XvmKttVr9jHn/6fRAQbU7T1ewJ1Iq9u2wf57/4C1e/swJ7A+q41uzP4g9vLmdrcvsp4lXyIae8kV1D+NuEfs4RkKN06AL+UdU/n/WQWbwKRHfw4Zze4ZTjxsvuN1L+lyQezjgXMIeqa5o4OIpQf09S80r404dr2JNRSIivB89OHsCS+87moz+N5JWrh/D3S/tx57k96Brqd/zAfYKxDLoSgOlu82qvekleXz0cDs4marUYhlkrART0msy091aRnl9Kr3B/3p024pg79p7RM5Trx3QB4JG5W2q/IUcPM6c9wJx+qdGczW43ePL7bWxNzuM/9WmOlrEDKsvIx4ckI4xHL+nLW/dPp8AtCD9LCdk7f+fPL38Cu+djYIGRN9d+vodvdUt7x4hUlaopl7N6hTkTvDEeZt3PD7kxlAd3B5sHlObx9//8SFmFnbS8UudIwAkZBqz/yPx+18/mqp0j2O0G/5xnjg5U2g0W7Ujn9jnrGfLkfPZkFBLs6+HcMgCgZ7gf3UJ9Kau0s3B7+lHna25VUy7j+0Vgdaz+umxIJ7p29CW7qNysX3J0DqbfJLA5/v8ZfYf5deOnUJBeq96j6lpXFZ3We+TDMODrP5v1JD4hcPUn5kaNNy12FJIXmyNcB1eb05LXfAbnPgpnOUbgfv3XsUf/WolZi3fz5HfbANiRms9P9VzFZhgGD3+1hVX7s3hzaT1X1LUBSj6k7bNYqt/EepwPZ9xf6+Hr4jsD5lLNxTuzyC0uJ8TXg+Fdaq++8XSzccMYc9rAzWrhptPjWHz/WVx1Wmzjd/11NFwbb13N+s01iiMXPG5+7XoWYDFbxGfvr/3c1M2QsQPD5slNq6PYe7iQTkHefHDDaQT6HL+A94HxvQkP8ORQTjEfrziiAPHsv0JID7MI8ae/Ou9euiuDpCxzKHzZ7no0CnMMpW+1dybM34sx3ULo6O+NX28zuZsWvpdpVrMN/SJjGK+srzy63X0vx/RJjVEYwzDf6MGccgHAbickx6wvWVrcjV/25FIabE53dCnfg6eb+U/dNxuTTxw3mMt+DzuaglWUwO75Rx3y09ZUdqTm4+fpxpd/Hs3tZ3enU5C3s5fGned0J6DGNKDFYnGOfpzsVS+VdoMF283aEudqK8DNZuWucebox+xfEzC2fWM+0H9K9ZNjR0Gn4eao26p3nJ1Ne4T5Ow8ZEB0EmCMf9So6/eWf5ooaqzv84T/mBwIwV7hd81n1/5MRA+DmpdBzvPnzoGvMv8XiLOdITGtjGAYv/JzgTET7Rpqjd68u2l2va7M4IZ0Ex0ji0oR0yivbeNNBByUf0j6M+ztc8iJcPtvsM1LD6d070iXEh/ySCh6ea3Y0Pb9feJ0Jxc1ndOXlqwbz8z1n8PDFfWu9uTRKWB8qOp+BzWIwNvtr9mQUmPPgexeb/1BPeBnizjCP3fjf2s/d/BkAv9tGsDy5kmBfDz688TQiHLvnHo+3h427x5lvzq8t3l17GsDd21xZhAU2zjGX7wIf1xjt2JtRSPKJlh2nViUfXZg0pBNujuWmdDsHgEvc13C1p7mq4p3yC3hh/k7Ofn4J32xMrv5Hu6ejRf7BNVBgJhy70ws4mF2Mh5uVMd3NZnEcTsBSkkuZ1YvtRmfeXbaPBVlmYnJGQCrvTR8BwIJtaeZ0wxEq7QZXvb2ccS8sNQuKN84xH6hqjLb921rH2+0GLy0wk5MbxnRhaGwH7hvfi18fOJtPbx7Fy1cN5o/xXY56naq6jyU7M+qMo7ms2Z9FZmEZgd7ujOxaO4meMNAsfh1WthZLWb45KhgzqvoAi6V69GP1OxxINUeLeoRXj+T1jvDHzWohs7CM5NwT1GMk/AiLnza/v+SF6hqsKlYbnPMI3LfbTDyC46ofs7nBOY6NGpe/1qzdcZuDYRjM/HEHrywyR90evLA3c24aia+Hje0pefUa4Xp9cXVBeV5JBWsPNHHlW0XZKbFCSMmHtA/+4TD8BvA8eirEarVw7Shz9KOqcPACx5vEkWxWCxMHdzrxlEoDuMXfBsBVtkUs3LQPFvzdfGDEjeYnxKqdgDd+4vxHZevBLLJXmm+Q7xechq+Hjfemj6BbA+K6Ylg0XTv6klVYxju/HrGEOHakWYgKMPfPpCTuZqFjtCHKkdz8doLRj4pDG8xY7Z1r917perb5NWMHtsoSjIgBTP3DNcQEe5OWV8qdn6zn2ndXmissAiLNwlMM2GkWqVaNeozqGoKPhyM5cLTPL48YSiU2ft+TyaqSGAAu75TF6G4hxHX0pbTCzvxtjuHwGs3Mft6ayoq9WexOL+CVeVtg8//MB850DPvvnFer6PH7zSnsTCvA38uNG8d2dd5vtZqt+ycO7uSc6qipT6Q/cR19KauwO3+Pk6FqyP/cPmG422r/M2+1Wpg6sjMTbMvNO/pPPiohp88Ec4l6cTZ/2n8fcZYUuodV/215udvo6eh0uvl4Uy/5qTDX8Xd02i0w9I/HPtYvtO79k/pMNDd1LCuAX1849vMd0vJKGlXwWXHkiMPhXWatyfJZ5usueRYWPwPp1fVH//gpgbd/MadKHr+0H7ee2Y0gHw+ucySery7addzRj9X7s1hzIBsPm5XTHbt5N+nv4tBac0POeX898bEupuRDBLh8WLRzaN7fy414x94vLaLnePK9owmyFDJ0xd3m8lQP/+qh6D4TwMMPsveRuHERMz5ex9Ovv0OHykyyDT9CB1/CvHvOOPbKmmNws1m5b7y5lPjfv+49ejXIOY9AWD8oTMfy36l4GGWM6R7CZY5E4vc9x2kLb7djOKZdSjr2p3dEQPVjgZ2gY/VeOZZRf2bC4E7Mv+dM7j2vJ55uVn7bncmFL//CP37aQVk3xxC8Y+plcYJjiW2v0OpzJq0CwLf7GOew9yEvswDZM2MrFouFSweZdT9fb0iG9R/DM5Gw/HUMw+DNpdWfPlNWzzVXPvlHmfsM+UeZb3yOtvCVdoOXF5p9Sm4cG3fCKa6aLBaLs5box5M09WIYBj9vPXrKpaYJfQIZZ10HwIGoC44+wGqDi57HcPdlQOU2fvR4kCFJH9Zqve5sNnYop+5A7HaYe5s5ZRIx0Nml+Fgy8kvrnnKwWs09k8BMBo7TCC81t4Rx/1rKxFm/NWj64sPl++n/93k89OUm83nF2fDeBWYPnHl/hYWPw5KZsPQf8O9xkLiSX3ZmOP9uZk4ewLTRXZzn+9PpcXi5W9l4MJdfdx07SX9jifn8KcOiuXKEmSwv3N6EpdjLZ5nTZav/bTZLPIb3lu078cjlSabkQwQI8vFg4mDzzem8PuF4uLXg/xpWGxZHseXw8jXmfaPvAF/zkxAevuTGmSsQfv/iVb7fnMJltmUA2PpP4pk/DCO6g0+jXvrC/hEMig6kqKySWY5GbU4ePnD1HAzvYCIKd/BP97e49rRYxnQ341q2+/CxP9Vl78O9sohSw51hw047+nHH1Au+oc56Ay93G3ee24P595zJOb3DKK80eGPJHqb/biaCxt7F5OXnOXcidtZ7ACQ5Ng6MGcnDF/fh3N5h3H/dZPO+vINQlMWljv++y3alU7nUbMvPgr+xYd0KNh7MxdPN/PR5mfUX8/UG/sEc9q/qGeKYevluUzK70wsI8HLjhrE1pgiqFGaaIydrP4CVb8Gyl8xPzfP/Bj88wJ+yX+Ql99c4Z+eT7D1wgIRUs8nc77sPk96Ivhn7DxeSV2PabGtyHodyivF2t3FGj9A6nxN8aBE+llIO2MP4JKlj3SfueT4HrlzAL5UD8LKUE7jsSXMfpsQVUFlRo9lYdSGoYRh8vynFTKxWvmlOIbp5m6vL3DyP+TvM25rKqJkLufu/G+o+oNu50Hms+cb67d3HbD738coD5JdWkJhVVO/i4neX7eOxr7dSUm7nk1VJ3PD+asrmPwVFhyGgEwy4AgZfa46cdhoGZfkY/7mMjz/7BDCX+F99Wu3+Qh39PLnmNHM09bVFu496TYDtKXks2pGO1QK3nNGV03uE4ma1sCejkP2HC+sVey1FWdXTg5Vl1VOHR/h5aypPfLeNi1751SV7DVU5xvIAkfbn4Yv6EtPBhytPi2nx1/YbOY2Spc/gZZSQYQTwx9UD6Zu+kcGxQazcm8nhLT351AMusq5gde87mXxoHZRDwGnXNul1LRYLf7mgN9f8eyUfrzzADWPiiA2pkch06MLvw1/ktF9u4FLbciqz51De5x683K1k5JeyK73AOfxeU/rOVYQBCUYME4Z2PvqFh11vvjGd/n9HvSnFhvjw7rThzN+WxpPfb+P3rCgOeYbQqTyTjz/5DxX2LnQN9aVziK/5hIJ0sycKFogewRjvIGeCZO6lsx9SN9Ot65n07xRAx5RfsOXsNx+vLMN/3t1YeYQ/DO/MjNMCCHnLLFxd6Hku4wD6XAqr3oaE76kof5GXF5ijHjed3rXump/Pp8H+X495zYOBSY7ZhVnvPMI/K65yPhbm78kvD5xduyvucXy94RB3fboBiwW6h/oxNLYDmYXmCNaZPUOPXvFUnG2u4ln+OgDf2uOZuyGZ+y/oXWeN0/biDtxW/iD3Bq7izvL3IXkdvDcePPy5NGwYB20RbE0aiGEfRnpBOQ/8bxNLd2bQ25LIeO+/mZ9uxz8Nob2O+Tuk5pbwly82UWk3E5c/jc1mSGyH2gdZLGbd1nvjzeLfVwab04Jj7nRu01BSXlmre+v/1iZxXt/w416/d37Z62wGN2FQFAu2pXF491psSe+ZB0x6A7qeWf2EsiL49Gose5fwovEUDwc9ykMX1jFyhFkf9tGKA6zan8XK3amM7F57FKpq1OTCAZF06Wj+LY/oEszyvZks2pFed2J7PJv+ayYdVndza4G17zsa9lX/dz1cUMpDX5q9b64cHkOQj0fDXqMZaeRDxCHQx507zu1BmP+JCzabnXcQh/tOA+C5iqvYnmnni3UHeXTuFr7blMJKe28y3cIJsBTzL693sZUXQFAsxIxs8kuP7t6R03t0pLzS4IX5CUc9/vKucP5WMR0A2+Kn8NozjxGOlUDLjjGkvG+LWU+QFdC77usZ1htuXw2Drjr6Mcyk6Px+ESz6v7N4bsogVrmboycBiQsBOKdXzVGPVY5z9gHvoNonqtHvA+DSQVFcZ3OsXOk3mUp3f7qX7eBGtx+56fSuRBz4DndLJRvsXXno1zKzEDc23lweWpzN8kXfsPdwIUE+7kx3LFeuJXOPmXhYrGaxbL/LzBUbw2+AUTPM3hXnPMLOaHO05yLbakJ83Inu4I2Xu5X0/NJ6bz5nGIazWNEwYFd6Af9dk4h7wrc85vYhd3h8Dev+YxYM719mjhi80Bd+fgTykzH8I/nWbTypeSUsP8YUmrnSxcKB2CkwYwUM+AN4BkJZPgEHl/Cg+6f8x/grRc/24ucXbiB71wq8KOUV91ex2svMZm/Dbzjm72C3G9z3+UZyisqpyn1emL+z7oNjRsD078yVOOVF8Ovz8PJgWPYiHPidRSvXkV1YQoCX+Zl64fZ0MguO3VjuzaV7nInHHed055WrBvPfm0fytOeH2LCz0Dqanb5Daz/Jw4dv+73AL5UD8LGU8q/yp/A+WHeiGRHoxRXDo5lhm8uQjwbUWjqfmFnEt46VV7edWb0c+9w+5t91g+s+DMMcaQNzutTDz9zQsUYSbBgGD36xmczCMnpH+HPv+T2PcbKWoZEPkVYiespMOO8OHvGI4KKkbNbuz2ZDUg4hfh7cckY3QnZMg1+egx3fmU8Y8IejCwUb6S8X9ObXXcv4emMykUHe3DAmjlB/TxJS81m1P4u11nE8PMiO78bZ8OVNXDL4fX7dBb/vOXzUJzS73XDWewR3G96kuNxtVv4wIoaKwBthzo+c776eJ8BZdwLUmHKpY3onYpA5FO1IPiZ1KaejdQMAqcPvY/Hhrlyd9jz3u3+OB//nHKpe4jWOjLxSXlm4i79e1If0yHMJ3/MZ+5d9ClzPTad3rbub7oaPza/dzoVr/nv04w49R+ZjPPcdcZUprL0lGsL78q+fE3h10W4+X3uQCYOijvncKsv3ZJKQlo+Ph43v7hjLwaQDRP76ID2yzWkjtv8E2+t4Ynh/OO1mLAOuYMQPe0hYkcgX6w4ytsfR0y/OZbbhfhAQBVPeMfdjSdsC+39j+aKv6Ve2iYCyDK7je67z/J5yd3/cy/PJMAJJHPwEw47TaO+93/axbPdhvNytvHntMP70wRp+3XWYlXszGVlX3VXn0fCnBeb/AwufMJdDOwq0LwISPG0UeUUw33s092VfxtcbkuscQXhz6R6e/dEsHL17XA/nyq+B2QuA7RTjyaNFV5H12jIuGhDJFcNiGBkXTGpeCQ9/u5vS8v9jXse36ZL9O8y5Em773dw48gi3ntEVtw0L8KCMsv/dwvubyvGOG8WKvVnYDbPnTn9H7QyYU4lPfb+dlfsyyS8pr/tvrAa73aCs0o5X6jrI2G5OcQ2bbo72rZ0Na2Y7V8t9tiaJBdvT8LBZefHKwbX3wHIBjXyItBZWGwTFEOjjztm9wrhvfC8++tNIXr5qCH2jAo4eJajaJbYZ9O8UyNWnxWAYZhHcmH8s4uGvNvPqInOK4bw+4fhe+k/ocjqUFTBx18N4UsaKvVlHFfat2pdJt0rzE3mvIQ3Yu+c43LqeAR5+hBpZbLwpnH5R1f9gO0c+ai4XrXLEyEfYjo+xWgx+qRzAqxvsPJw4hGWV/fAwyuCTq8zjrO4Mu8RsfT/7t/2Me2Epf9neBYDzras5u2eIs0lbLfZK2GDWATBk6vF/IU9/LFV1L9vNXhuXO/YB+nVXRr2KAd/7bT8AU4ZG0zVjIWcsmGAmHlZ38w1o8LXQ/Tyz2DOgk1m4PP17uHUZDJsGHj5MHmq+5k9bUutc+lvVybZHjZUuWG3m6pP4P/Nt3+cZXvoGN5f/Hzs6no/h7oN7ufmc+8pv5blfM49ZF7Q9JY/nfjJH2h65uC9n9QpzFl3+6+edx64nsljM3+W25TDhFYg7k1L/WMoMG+6WSgJLDnF58ef0siTyv7VHF6euT8zmHz+Zice95/V0Jh6U5pujQgCn/x/RcT0pKbfz5bpDXP3OCs58fjE3vL+avJIKeseEEX3rl+bfXEWJs9nfkWLK9xFpMfeu8aCcyxLu561vlvC9o9j4z2fVTli6hvoR19GX8kqDTasWQ+LKuq+Bw7XvrmTEUws4tOhN845+k8zRv+HXOy7yt1CQQWJmEU98azZA+7/ze9InMqDO87UkJR8ip4qQbtVvsJGDjjuP3hhPTxrA29cNY0hsEGUVdj5emch3m8x/JK8d1dncefjy98A3DK/sBB73+oSC0go21dha3W43mP3TCkItedix4hk1oHmCc/N0Fql67a2x0VxestkNFo4x8uF4/cMJUJzj3KTsw8rz+XhlInbDwpfRD4C7b3VTsV4XcPrAXpzXN5wKu8GejEI2uA2kxOpLuCWH2edZq5f41rR3MeQng1dQdYvy4+kzwfzqKBLsHOLLyLhgDAO+qONNs6YDmYUs2ZHMUMtOHij8p7khYlEmhA+Am5eY/WEmzYJr/we3/gr3boMrPzI3cqwxEjEkJoiuHX0pLq88anPDiko7ex2FjzUbjNV0yxlduXJUd/586530vv1zLPfvhis+IHPSJyy3DGHlvqw6V0WVlFdy16frKau0M65PGFMdG0Lefk53PNysrNqfdeJGdjY3M4ma9g33R31I79IPeLLn/5z7+tzg/jPbUvJqtSyvqLTzyNwtGAZMHtKJO8/tUX2+pc+ZjfWCu+J95t389+ZRfHFbPFefFoOfpxtJWcXsSM3Hy93Ki38YhJunY5QBjr0T9C5zeXhB5CiyA3oTasnjU78X6RtsLnUfGRd81FPO6R3G+dbVxC+60qxxOfB7nadOSM3n9z2Z2EvzCdpr/g0ZQ64zH4wcZO7bZC/Hvv5j7v1sA4VllZzWJZg/nd61zvO1NCUfIqeSM+4H7w5m7UAzs1rNOosvbxvNf28exVmOpaz9OwUwuptjCNwvDC4z92K5inmMs65l2a7qN5cPl++nzNHfozK4u7liprlUvaHv+AH2/WK+4b7Y3yyy84sw97A5UkAUeAeDvcLs0VCcTWVADL9QPZd/+bljzWLGKoPMvipPX9afq0+L4YmJ/fj1rxfg1c/x+sfaZ2eDY3XBgCuOu7Kj+ve50GxilrbFuXPxH4abn/w/X3uw7r1BirJg/UcUfHQdazxu5UvPv+O/a65ZY3L6fXDTIojof+LXdrBYLM4eLF+uq53wJGUXU1Zhx8vdSnQH7zqf3znElycn9a9e5u3hC/0mETL4IucO0y/Mrz2KUVpRySNzt7AzrYCOfh48O2Wgs217ZKC3MxF5/ojRj6zCMt5ause51LpKel4JP2xOwY6Vy848zSxCBS6z/UYgBXyx9pDz2P+sOMDW5DwCvd3568V9qk9yeBesMItwueBZcPPEYrEwrHMwMycPZPXD43jpysFcMjCSV64aUt3np+d4sNjM/4ZZR/TKAeeOzH5Dr6DDjV+CXwTR5fv5IWo2/5zcr869nyaFJPGK+2tYsQMGfD2jVk+aKt9sNH+vP3itwtdSyh57JA+s8nV22LU7EqPMX95i7YFM/Dzd+NcfBjW+G3MzU/IhcirpMQ7+st/c/vwksVgsjOwawvvXn8byh87h05vjazfM6j7Ouf/Lc+5vsS3BHMJOzCxi7k/zuMvNLKxz7zS4eQPrcb75Jpu2GT6YYCYBRqVZEDrlnVqf6Gv8MtWjH44Ny2wjbmBsT3MVxMDoQOK7hcCIP8HgqdD7EuhxHgBh/l7MnDyQP8Z3Mefe+ziu+fZvj+4gWZwN2x21OCeacqniE2xOY4Fz6uXCARH4ebqRmFXEqv1HbDVfkAFvjIGvZ9AveyFBlkLKPQKh32S4cYG5F4pbw1cvTBpiJh/L92ZyqMZ0T9WUS/cwvzobpp3In8/qhqeblbUHslm601z2ui05j4mv/eacDvnn5YPo6Fc7UbvtrG54u9vYmJTDwu3ppOeV8PT32xjz7CJm/riD62evZsacdRx2FJN+vDKRCrvBsM4dzPqJ2HgIH4CHUcqVtsXM3XDIsbdPCf/62RzdeuCCXrVf99cXzAS1x/jq1u41eHvYmDSkE69dM5Tza/ZO8Qmu7th65OhHURYcdEwJ9hhv9re5+hOzLmP3fPjmDvOYmjJ20v+XW/CylLO4chBlvpHmSq5FtXukGIbh3Crgzg5mcfd/7Wfz+bpDXPvvlfzlf5s4/ftg8gxvQsuTGW3dyt8m9CUmuBk/DDSRkg8ROabIQO+6N+w79zFKQ/sTbClgetozFORksPW9W/nC+iCDrXsx3H3MDq3NyTekutW8hx8Mv9Es9Lvhp+r761KVfBh2c7O5oX/k3vN6Et81hCcm9jc/fVqtMOl1uOpjc3qpLt3PNd84cg5UF5ZW2fKF2YMirB9EDq7/73TE1IuPhxsTBpnddT9bk1R9nGGYb1b5yRR4RfJSxWRm+DyH7YG9cMVsc0PARoru4MOoruZ0z6zFu1mwLY35jhsce8rlRMICvPijY9+kF+bvZNbi3UyctYwdqfmE+Hrw1nXDOLtmr5aq5/l7ORt2PfjlZsY+t5h3ft1HcXkl3UJ9sVktfL8phXEvLOXzNUnMWWUur51e1eTLYoGRt5j3uS8gp7CExQnpPPX9dgpKKxgUE8TVI2r05ShIhy1VHW0faPgv2tvRB6aqELzK7oXm31xYXwhyLN/vNBQuc9RnbJwDLw+CxTPNnavzUuCjyViKs9nn1Yc/l9/Ft7GODrsr3qg1/bI+KYekrGKGeBykQ/YmsLpz5hV34u/pxqr9Wfx3TRKHiqz8YDH/v3ip23quGN7yLQSOR6tdRKTh3Dzx+MNsimeNJd66jbKX+3GhUQoWKOw+Ad8Jz0JgdPO/7mVvw6E1ZrLhWc83xchB1d/3mwy+HenvC5/cXEeB6vF4+MLo281N0r6502yQVvUpeb0jGRkyte4RmGPpfYnZRfPQWrNzZ2A0VwyP4ZNVSfywOYXHL+1njrqs+wB2/ohh8+Auy4MsrAjlidP7YXVrnn/CJw+NZsXeLOasTKzVKwOo1Va9oW45sxsfr0xk08Fc5w645/cN55nJA44a8aj1PEePjKrRjaGxQdxxbg/O6hnK1uQ87v/fJranmF8BwgM8uaDmLtQDLof5jxFVnME461pm/uDH/swirBZ4elL/2iM5a94zp+6iR0B0I1Zn9b4YfnzAbL5WkGG2iQdnvQc9zq99fL9J4PGFuUonbTMsfdZsyOYTDLlJENKdrcPfofjrRN5NDWPKkGvN3ixfz4BbfwMPH77ZkIyNSh7ssBhygd4XMWZQH76MiGbmjzuI7uDN+H4RnOYTA2/PI/TgfHjvQnM60M3L/BoYbfZgcRGNfIhIo1hCe/Jdp3sA8DDMOed5Q9/E99qPTk7iAeYePb0vrn/iAdUjHwCn3dS01z/rrzDwKnO657Np5mqE9O1m8y2rm7n8uSH8w81pAnCOfgyJCaJbqC8l5Xaz4DdzD/z0EAC7B9zDwuxQ/L3cmDK0+a7xpYOiuHhgJIOiAxkUE8TgmCCGxAZxdq/Q2vvyNFBHP0/niIS/pxv/umIQb1037LiJB0AHXw+ev2IQk4d0Ys5NI/nittGc3SsMi8VC/06BfHP7GO47vycejn1rrhvVufYeNu7eZjEqMN02j/2ZZs3EH+O71FraSoWjFTnAqNsa90sGRlfvP5Twg3lfZYVzQ8a6pnHoMQ5u+QWu+AA69jLb+WftBb9wuPYLRg/shdUC21LyWNv7fnO1UtZeWPgEFdlJRK1/kd8872RkrmO3Z8eeOT3C/Xlv+giemNifMd074h410OwMa1RC4u9mUfTOH2Hb3Or4XMRi1Gs/5JaTl5dHYGAgubm5BAS4fjmQiBzbNxsOseLzf2HFzp7oyXx8y+mNqg84qex2+O4uc6pm/DMNG5moS2U5fHoN7PrZXNnSZaw55N7rYri67pbWx7X8dZj3EMSOhhvMN5O3lu5h5o87GBbjzxcej8OhNWSHjeK68r+yJaWAP42N45FL+jbt92ghFZV2ftqayrDOHYgMrLtwtbH2ZBSw7kA2l9XcNblKTpI5rWFUcn7pP8j2687C/zuzdlfaDXPM/WcCOsFdG4895XYiv/wTFj1l1nZM/QwOLIfZF5h/H/fvMVfmHIu90py22/UzjL0HwvsB8NCXm/hkVRI9w/34/qJS3D+5AgDDYsVimEWlhk9HLKNvhzF3H/vvujgHklZCebGZbFWWml89/GDw1Y37fY+hIe/fmnYRkUYb070jD1jNAs0frxjW+hIPMOs5Ln21+c5nc4cr3ocPJ8LB1dVz/fUtND1Snwlm8pG43Kw/8AvjsqGdeG5eAmOT3wf3NeQZPlyUOJUUCvBws9baxKy1c7NZuWTgiZumNUa3UL9j7+QcFGOOkm3/hhk+CwmYMrl24mEY1StcTrup8YkHQO8JZvKxd7HZL6RqyqX7uOMnHmD2TRn4h6P69vzlgt7M25rGzrQC/p3Sm9sc0y8Ww84Kex/2x13FVdf9+cRFxt5BdY++uJimXUSk0UL8PPnytjF8d8dY4hz7U7QLHr5wzWfmkDmY9R9Hzu3XV1CM2ZMBw0xkyksIO7yal8K+5w63rwB4pPwGCOzE1JGxfHHr6Fa1aqFVG3krABMtv3J27BFJwIHfzKZybt4wdFrTXie0FwR3M2tHdi9wLrFtypt+kI8HD19kLgd+eeFOkuKfouzSN7jUeIGryh6l29l/bNTqptZCIx8i0iR9o9rp9KhPMFz3pVmP0W9S0z4595lg1o0seBx+fBAqS5kAYIGE0Au45bIHeDkyoM6+EHIcnUeb7eTTtsD/rjebr3XoYj62wuxXw6CrzP+WTWGxmLsf//YyrHwb0reay8K7j2vSaScP7cRna5JYuS+Lv/2wm8uHnc2m0nV0CvJm2JGb751iNPIhItJYgdFw5X+g/5SmnafvRMBiFh5WlpqFh/2nwKWv0uuW/9AvKlCJR2NYLHDu38wl1nuXwOvx8NsrcHh3dV8Ox+hIk1UtuU10LImNHtHkpMZisfD0Zf1xt1lYtCOdp7+v3oG3VU5xNoBGPkREXC2kG1z9qdmevfNY6Nij6YWxYup5vtkP5tu74cAymP8oLHkWMMwNAMN6N8/rdBpuJo0Fjl2JGzsNd4TuYf7cfEZXZi3e42wCd2k9Nh5s7Ro88vHLL78wYcIEoqKisFgszJ07t9bjhmHw2GOPERkZibe3N+PGjWPXrl3NFa+ISNvU6wJz+/nQnko8mlvHHjD9O7j0NXMFSrm5Zw2j/tx8r2G11t7Tp+cFzXbq28/uQUywuVKoR5gffSIb1/itNWlw8lFYWMigQYOYNWtWnY8/99xzvPLKK7z55pusXLkSX19fxo8fT0lJSZODFRERaRSLBYZeB7evgRE3mVsEVO0s3Fz6TjS/BsU6l8w2B28PG/+8fBCdQ3y449webWIKrkl9PiwWC1999RWTJk0CzFGPqKgo/u///o/77jM3vsrNzSU8PJz333+fq6666jhnM6nPh4iInLI2/89c/VKzuV070ZD372YtON23bx+pqamMG1dd4RsYGMjIkSNZvnx5nc8pLS0lLy+v1k1EROSUNODydpl4NFSzJh+pqakAhIeH17o/PDzc+diRZs6cSWBgoPMWE9O6Nr8RERGR5uXypbYPPfQQubm5zltSUtKJnyQiIiKnrGZNPiIizF0F09LSat2flpbmfOxInp6eBAQE1LqJiIhI29WsyUdcXBwREREsXLjQeV9eXh4rV64kPj6+OV9KRERETlENbjJWUFDA7t27nT/v27ePDRs2EBwcTGxsLHfffTdPPfUUPXr0IC4ujkcffZSoqCjnihgRERFp3xqcfKxZs4azzz7b+fO9994LwLRp03j//fd54IEHKCws5OabbyYnJ4exY8fy008/4eXl1XxRi4iIyCmrSX0+Tgb1+RARETn1uKzPh4iIiMiJKPkQERGRFqXkQ0RERFqUkg8RERFpUUo+REREpEUp+RAREZEW1eA+Hydb1cpf7W4rIiJy6qh6365PB49Wl3zk5+cDaHdbERGRU1B+fj6BgYHHPabVNRmz2+0kJyfj7++PxWJp1nPn5eURExNDUlKSGpidZLrWLUfXuuXoWrccXeuW01zX2jAM8vPziYqKwmo9flVHqxv5sFqtREdHn9TX0O65LUfXuuXoWrccXeuWo2vdcprjWp9oxKOKCk5FRESkRSn5EBERkRbVrpIPT09P/va3v+Hp6enqUNo8XeuWo2vdcnStW46udctxxbVudQWnIiIi0ra1q5EPERERcT0lHyIiItKilHyIiIhIi1LyISIiIi2q3SQfs2bNokuXLnh5eTFy5EhWrVrl6pBOeTNnzmTEiBH4+/sTFhbGpEmTSEhIqHVMSUkJM2bMICQkBD8/P6ZMmUJaWpqLIm47nn32WSwWC3fffbfzPl3r5nPo0CGuvfZaQkJC8Pb2ZsCAAaxZs8b5uGEYPPbYY0RGRuLt7c24cePYtWuXCyM+NVVWVvLoo48SFxeHt7c33bp148knn6y1N4iudeP98ssvTJgwgaioKCwWC3Pnzq31eH2ubVZWFlOnTiUgIICgoCBuvPFGCgoKmh6c0Q58+umnhoeHh/Hee+8ZW7duNW666SYjKCjISEtLc3Vop7Tx48cbs2fPNrZs2WJs2LDBuOiii4zY2FijoKDAecytt95qxMTEGAsXLjTWrFljjBo1yhg9erQLoz71rVq1yujSpYsxcOBA46677nLer2vdPLKysozOnTsb06dPN1auXGns3bvXmDdvnrF7927nMc8++6wRGBhozJ0719i4caNx6aWXGnFxcUZxcbELIz/1PP3000ZISIjx3XffGfv27TM+//xzw8/Pz3j55Zedx+haN94PP/xgPPzww8aXX35pAMZXX31V6/H6XNsLLrjAGDRokLFixQrj119/Nbp3725cffXVTY6tXSQfp512mjFjxgznz5WVlUZUVJQxc+ZMF0bV9qSnpxuAsXTpUsMwDCMnJ8dwd3c3Pv/8c+cx27dvNwBj+fLlrgrzlJafn2/06NHDmD9/vnHmmWc6kw9d6+bzl7/8xRg7duwxH7fb7UZERITxz3/+03lfTk6O4enpaXzyySctEWKbcfHFFxs33HBDrfsmT55sTJ061TAMXevmdGTyUZ9ru23bNgMwVq9e7Tzmxx9/NCwWi3Ho0KEmxdPmp13KyspYu3Yt48aNc95ntVoZN24cy5cvd2FkbU9ubi4AwcHBAKxdu5by8vJa1753797Exsbq2jfSjBkzuPjii2tdU9C1bk7ffPMNw4cP54orriAsLIwhQ4bwzjvvOB/ft28fqampta51YGAgI0eO1LVuoNGjR7Nw4UJ27twJwMaNG1m2bBkXXnghoGt9MtXn2i5fvpygoCCGDx/uPGbcuHFYrVZWrlzZpNdvdRvLNbfDhw9TWVlJeHh4rfvDw8PZsWOHi6Jqe+x2O3fffTdjxoyhf//+AKSmpuLh4UFQUFCtY8PDw0lNTXVBlKe2Tz/9lHXr1rF69eqjHtO1bj579+7ljTfe4N577+Wvf/0rq1ev5s4778TDw4Np06Y5r2dd/6boWjfMgw8+SF5eHr1798Zms1FZWcnTTz/N1KlTAXStT6L6XNvU1FTCwsJqPe7m5kZwcHCTr3+bTz6kZcyYMYMtW7awbNkyV4fSJiUlJXHXXXcxf/58vLy8XB1Om2a32xk+fDjPPPMMAEOGDGHLli28+eabTJs2zcXRtS2fffYZH3/8MXPmzKFfv35s2LCBu+++m6ioKF3rNq7NT7t07NgRm812VNV/WloaERERLoqqbbn99tv57rvvWLx4MdHR0c77IyIiKCsrIycnp9bxuvYNt3btWtLT0xk6dChubm64ubmxdOlSXnnlFdzc3AgPD9e1biaRkZH07du31n19+vQhMTERwHk99W9K091///08+OCDXHXVVQwYMIDrrruOe+65h5kzZwK61idTfa5tREQE6enptR6vqKggKyuryde/zScfHh4eDBs2jIULFzrvs9vtLFy4kPj4eBdGduozDIPbb7+dr776ikWLFhEXF1fr8WHDhuHu7l7r2ickJJCYmKhr30DnnnsumzdvZsOGDc7b8OHDmTp1qvN7XevmMWbMmKOWjO/cuZPOnTsDEBcXR0RERK1rnZeXx8qVK3WtG6ioqAirtfbbkM1mw263A7rWJ1N9rm18fDw5OTmsXbvWecyiRYuw2+2MHDmyaQE0qVz1FPHpp58anp6exvvvv29s27bNuPnmm42goCAjNTXV1aGd0m677TYjMDDQWLJkiZGSkuK8FRUVOY+59dZbjdjYWGPRokXGmjVrjPj4eCM+Pt6FUbcdNVe7GIaudXNZtWqV4ebmZjz99NPGrl27jI8//tjw8fExPvroI+cxzz77rBEUFGR8/fXXxqZNm4yJEydq+WcjTJs2zejUqZNzqe2XX35pdOzY0XjggQecx+haN15+fr6xfv16Y/369QZgvPDCC8b69euNAwcOGIZRv2t7wQUXGEOGDDFWrlxpLFu2zOjRo4eW2jbEq6++asTGxhoeHh7GaaedZqxYscLVIZ3ygDpvs2fPdh5TXFxs/PnPfzY6dOhg+Pj4GJdddpmRkpLiuqDbkCOTD13r5vPtt98a/fv3Nzw9PY3evXsbb7/9dq3H7Xa78eijjxrh4eGGp6ence655xoJCQkuivbUlZeXZ9x1111GbGys4eXlZXTt2tV4+OGHjdLSUucxutaNt3jx4jr/jZ42bZphGPW7tpmZmcbVV19t+Pn5GQEBAcb1119v5OfnNzk2i2HUaCUnIiIicpK1+ZoPERERaV2UfIiIiEiLUvIhIiIiLUrJh4iIiLQoJR8iIiLSopR8iIiISItS8iEiIiItSsmHiIiItCglHyIiItKilHyIiIhIi1LyISIiIi1KyYeIiIi0qP8Hctpjk7QgYZMAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "上記のグラフの縦軸は前述の「無作為に単語を選び出したとするときの『母数』」で、横軸は `epoch` 数です。学習が繰り返されるごとに「母数」は減っていき、最終的に15語程度になっていることがわかります。これはすごい進化です！もはや無作為な単語の羅列では無いはずです。さっそく文章を生成してみましょう。"
      ],
      "metadata": {
        "id": "utcBZIaR7-qA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(generate(model))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KEVrFHxJp_2d",
        "outputId": "abc5fb17-f9fe-4162-f67e-13c52d80095b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "k d, apine t ue. Pd my itie, Mrhent woncaad  he as. an syurthase, thal ucoad, manel, or biche psthhi\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "学習前は出鱈目な羅列だったのが、学習によって「薄目で見れば英語かもしれない」程度の文章を生成するようになりました。現実に存在する単語もいくつかあります。当初の目標だった、「それっぽく」動く言語モデルの完成です。"
      ],
      "metadata": {
        "id": "F8lHDleV9Mvy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## まとめ"
      ],
      "metadata": {
        "id": "abS3tsMS-BHD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "今回作ったモデルの特徴は以下の通りです。\n",
        "\n",
        "- 「単語」の単位  \n",
        "  → 文字\n",
        "- 単語のベクトル化  \n",
        "  → ワンホットベクトル\n",
        "- モデル  \n",
        "  → 順伝播型ニューラルネットワーク (中間層1層, ReLU)\n",
        "\n",
        "とにかく単純な構成を目指してこのような構成にしましたが、それでも「それっぽく」動くものになりました。次回の勉強会では「脱それっぽく」を目指してこれらの構成を改良していきます。"
      ],
      "metadata": {
        "id": "9fvbpYHH-FMW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 参考文献"
      ],
      "metadata": {
        "id": "_7VywL9-_sog"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "理論は以下を参考にしました:\n",
        "\n",
        "- 岡﨑 直観, 荒瀬 由紀, 鈴木 潤, 鶴岡 慶雅, 宮尾 祐介.『IT Text 自然言語処理の基礎』(オーム社, 2022)\n",
        "\n",
        "実装は以下を参考にしました:\n",
        "\n",
        "- [Llama from scratch (or how to implement a paper without crying) | Brian Kitano](https://blog.briankitano.com/llama-from-scratch/)\n",
        "- [CrossEntropyLoss — PyTorch 2.3 documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)"
      ],
      "metadata": {
        "id": "0CuwmNmo_uEV"
      }
    }
  ]
 }