nlp-ml-huit.ipynb

nlp-ml-huit.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNIzz4jlceAnBWNB3hONj4T",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/xiupos/f14c3250f306c79cd14ad9e7ce85f36a/nlp-ml-huit.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# NLP/ML 勉強会 @HUIT\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "h33YdTr0Cisa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "とにかく動く言語モデルを作ります。動けばいいので、精度は求めません。単純な構成を採用して「それっぽく」動くものをまずは作ってみましょう。\n",
        "\n",
        "[発表スライド](https://speakerdeck.com/xiupos/ml-mian-qiang-hui-at-huit) を前提とした記述をしています。"
      ],
      "metadata": {
        "id": "UlPyWRiRcLhb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 準備"
      ],
      "metadata": {
        "id": "yPEcpnXUCrN6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "まずは使うライブラリを用意します。今回は [PyTorch](https://pytorch.org/) を中心に使います。"
      ],
      "metadata": {
        "id": "DyuV_PX95kXu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UnKGkrykcKD7",
        "outputId": "b697417d-7438-4550-fb02-17a9a67bd1cd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<torch._C.Generator at 0x7b62253402d0>"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "from torch.nn import functional as F\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "import pandas as pd\n",
        "# 乱数を固定\n",
        "torch.manual_seed(1)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次に教師データを用意します。教師データは何でもいいですが、ここでは「シャーロックホームズの冒険 (The Adventures of Sherlock Holmes)」の[原文](https://sherlock-holm.es/stories/plain-text/advs.txt)にしました。有名な教師データには [Tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) などがあります。"
      ],
      "metadata": {
        "id": "lz9mjU5ljFGe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 教師データをダウンロード (https://sherlock-holm.es/stories/plain-text/advs.txt を基に作成)\n",
        "!wget https://gist.githubusercontent.com/xiupos/b7914ea1e3ab35465c34de45146b15d8/raw/b7623310de06160876ffba6299994b0409c85c81/advs.txt\n",
        "lines: str = open('./advs.txt', 'r').read().replace('\\n', ' ')"
      ],
      "metadata": {
        "id": "F85C-xgrgYgg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "22ad09fe-0429-4065-ee76-f98727cc362c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-08-16 08:23:44--  https://gist.githubusercontent.com/xiupos/b7914ea1e3ab35465c34de45146b15d8/raw/b7623310de06160876ffba6299994b0409c85c81/advs.txt\n",
            "Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 562263 (549K) [text/plain]\n",
            "Saving to: ‘advs.txt’\n",
            "\n",
            "advs.txt            100%[===================>] 549.08K  --.-KB/s    in 0.03s   \n",
            "\n",
            "2024-08-16 08:23:44 (17.8 MB/s) - ‘advs.txt’ saved [562263/562263]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "教師データは文字列です。文字列を機械学習で直接扱うのは難しいので、まずは単語を整数と対応させます。ただし、英単語を単位に扱うのは高度なので、簡単のためここでは文字を単位に考えます。以降、「単語」は文字のことを指します。言語モデルの文脈では「トークン」と呼ぶのが正確だと思います。\n",
        "\n",
        "単純に文字順の番号を文字と対応する整数として用います。たとえば、文字順で $μ$ 番目の文字には整数 $μ$ が対応します。以降、その文字のことを単語 $μ$ と呼びます。"
      ],
      "metadata": {
        "id": "a4_gSeCj36vC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 文字一覧\n",
        "vocab: list[str] = sorted(list(set(lines)))\n",
        "# {整数(番号): 文字} の辞書\n",
        "itos: dict[int, str] = {i:s for i,s in enumerate(vocab)}\n",
        "# {文字: 整数(番号)} の辞書\n",
        "stoi: dict[str, int] = {s:i for i,s in enumerate(vocab)}\n",
        "\n",
        "# 文字の一覧と数\n",
        "\"\".join(vocab), len(vocab)"
      ],
      "metadata": {
        "id": "_Gl30hk9ghRk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc390000-472d-4307-fed8-b9bbaf73cae6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(' !\"&\\'(),-./0123456789:;?ABCDEFGHIJKLMNOPQRSTUVWYZ[]abcdefghijklmnopqrstuvwxyz£½àâèé',\n",
              " 83)"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次に上の対応を用いて、文字列を整数列に変換する関数を定義しましょう。また教師データを整数列に変換して教師データ `dataset` とします。"
      ],
      "metadata": {
        "id": "naUorL6u4Ayw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# エンコード (文字列 -> 数リスト の関数)\n",
        "def encode(s: str) -> list[int]:\n",
        "    return [stoi[s] for s in s]\n",
        "\n",
        "# デコード (数リスト -> 文字列 の関数)\n",
        "def decode(l: list[int]) -> str:\n",
        "    return ''.join([itos[i] for i in l])\n",
        "\n",
        "# 教師データを整数列に変換\n",
        "dataset = encode(lines)\n",
        "\n",
        "# エンコードとデコードのテスト\n",
        "encode(\"Do you know, Watson,\"), decode(encode(\"Do you know, Watson,\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvkJQt4jhCbY",
        "outputId": "f9e32854-e931-40cd-91a1-16fb8db0115e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([27, 65, 0, 75, 65, 71, 0, 61, 64, 65, 73, 7, 0, 46, 51, 70, 69, 65, 64, 7],\n",
              " 'Do you know, Watson,')"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "以降の便利のため、`dataset` から学習用のバッチを作成する関数を定義します。例えば \"Do you know, Watson,\" という文章に対し, バッチは以下の2つの文字列の組から構成されます。\n",
        "\n",
        "- x: `Do you know, Wat`\n",
        "- y: `o you know, Wats`\n",
        "\n",
        "後述しますが、今回は「1単語から次の1単語を予測する」モデルを作ります。そのため、パッチは1文字ずらしたタプルになっています。"
      ],
      "metadata": {
        "id": "ixBGfeS87g69"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# バッチを作成する関数\n",
        "# 幅 xnum で1文字ずれた列を生成する\n",
        "# ex) ['rip has been upo', 'ip has been upon']\n",
        "#      |<-----xnum----->|  |<-----xnum----->|\n",
        "def get_batches(xnum: int, data: list[int] = dataset, split=\"train\") -> tuple[list[int], list[int]]:\n",
        "    # 教師データを 8:2 で分割してバッチとする\n",
        "    if split == \"train\":\n",
        "        # 学習データ\n",
        "        batchdata = data[:int(len(data)*0.8)]\n",
        "    else:\n",
        "        # 検証データ\n",
        "        batchdata = data[int(len(data)*0.8):]\n",
        "\n",
        "    # パッチの開始位置をランダムに決める\n",
        "    idx = torch.randint(0, len(batchdata)-xnum-1, (1,))\n",
        "    # パッチを取り出す\n",
        "    x = batchdata[idx:idx+xnum]\n",
        "    y = batchdata[idx+1:idx+xnum+1]\n",
        "    return x, y\n",
        "\n",
        "# バッチのテスト\n",
        "[decode(l) for l in get_batches(16)]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uxSYnZjodkAy",
        "outputId": "a584d0f1-12a7-4bf9-e3af-7816953c7f75"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['rip has been upo', 'ip has been upon']"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "これで準備が整いました。"
      ],
      "metadata": {
        "id": "nhGrIeL-9K7S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## はじめてのモデル"
      ],
      "metadata": {
        "id": "1mnExgl0Cvew"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "さて、はじめてのモデルを作っていきます。ここでは「1単語から次の1単語を予測する」モデルを作ります。これを繰り返し行うことで文章を生成することができます。\n",
        "\n",
        "まずは単語のベクトル化の関数を定義しましょう。詳細は発表スライドを参照してください。"
      ],
      "metadata": {
        "id": "XWbnir1y4HPl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 数リスト -> 文字ベクトル の関数\n",
        "def ltov(l: list[int]) -> torch.Tensor:\n",
        "    # ワンホットベクトルに変換\n",
        "    return torch.eye(len(vocab))[l]\n",
        "\n",
        "# logits -> 数リスト の関数 (確率分布に基づく)\n",
        "def vtol(v: torch.Tensor) -> list[int]:\n",
        "    # 確率分布を計算\n",
        "    p = F.softmax(v, dim=-1)\n",
        "    # 確率分布を基に\n",
        "    return torch.multinomial(p, num_samples=1).view(-1).tolist()\n",
        "\n",
        "# logits -> 数リスト の関数 (貪欲法)\n",
        "def vtol_greedy(v: torch.Tensor) -> list[int]:\n",
        "    # 値が最大(⇔確率が最大)の文字を選択\n",
        "    return torch.argmax(v, dim=-1).tolist()\n",
        "\n",
        "# 試しに文字ベクトルを logits として文字に戻す\n",
        "xs, _ = get_batches(16)\n",
        "logits = ltov(xs)\n",
        "decode(xs), decode(vtol(logits)), decode(vtol_greedy(logits))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b8wyKO8K9qZG",
        "outputId": "b3f6da4e-ed59-43d4-921f-7245d008cb72"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('bited. The bedro', '½Qz,eKvHs0:6SdLQ', 'bited. The bedro')"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "次にモデルの中心となるニューラルネットワークを定義します。簡単のため ReLU を活性化関数に用いた中間層1層の順伝播型ニューラルネットワークとします: 式に書くと\n",
        "$$\n",
        "\\vec{y} = W_2 \\operatorname*{ReLU}(W_1 \\vec{x} + \\vec{b}_1) + \\vec{b}_2\n",
        "$$\n",
        "です。ここで、行列 $W_1$, $W_2$ とベクトル $\\vec{b}_1$, $\\vec{b}_2$ はモデルのパラメータです。成分表示すれば,\n",
        "$$\n",
        "\\begin{aligned}\n",
        "y_μ\n",
        "  &= \\sum_ν \\left[ w_{2μν} \\operatorname*{ReLU}\\left(\\sum_λ w_{1νλ} x_λ + b_{1λ}\\right) + b_{2ν} \\right] \\\\\n",
        "  &= \\sum_ν \\left[ w_{2μν} \\max\\left(\\sum_λ w_{1νλ} x_λ + b_{1λ}, 0\\right) + b_{2ν} \\right]\n",
        "\\end{aligned}\n",
        "$$\n",
        "となります。\n",
        "PyTorch の `nn.Sequential` を使ってモデルを定義しましょう。 `nn.Linear` は $\\vec{x} ↦ W \\vec{x} + \\vec{b}$, `nn.ReLU` は $\\vec{x} ↦ \\operatorname*{ReLU}(\\vec{x})$ を意味しています。"
      ],
      "metadata": {
        "id": "KjrBDikMruW0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# モデルの隠れ層の要素数\n",
        "d_model = 128\n",
        "\n",
        "# モデルの定義\n",
        "model = nn.Sequential(\n",
        "    nn.Linear(len(vocab), d_model),\n",
        "    nn.ReLU(),\n",
        "    nn.Linear(d_model, len(vocab)),\n",
        ")\n",
        "\n",
        "# モデルのパラメータのサイズ\n",
        "model, [m.numel() for m in model.parameters()]"
      ],
      "metadata": {
        "id": "JUZIrlQsA7uD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d4188c8d-d4c4-4f67-e8f4-f01c0a6b1e32"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(Sequential(\n",
              "   (0): Linear(in_features=83, out_features=128, bias=True)\n",
              "   (1): ReLU()\n",
              "   (2): Linear(in_features=128, out_features=83, bias=True)\n",
              " ),\n",
              " [10624, 128, 10624, 83])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ここで、隠れ層の要素数は `hidden_layer_dim` としました。さっそくモデルを使ってみましょう。適当にバッチを取得して次の単語を予想させてみます。"
      ],
      "metadata": {
        "id": "Jca-Fql-ncdu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# モデルのテスト\n",
        "xs, _ = get_batches(16)\n",
        "logits = model(ltov(xs))\n",
        "decode(xs), decode(vtol(logits)), decode(vtol_greedy(logits))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i3wJcrV34gf4",
        "outputId": "12c6eff0-4032-451a-9d6d-6c1accbcf72f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('ression upon me,', \"uTCf'G)zewtKLHr!\", 'Vlgggg(gMgg(ggll')"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "滅茶苦茶ですね。学習も何もしてないので当然の結果です。ついでに長い文章を生成する関数も定義します。"
      ],
      "metadata": {
        "id": "kZ4l65Y7niU8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 文章を生成する関数\n",
        "def generate(model: nn.Module, xnum=100) -> str:\n",
        "    # 文章(数リスト)を格納する配列\n",
        "    out: list[int] = []\n",
        "    # 初期状態(1文字目をランダムに選択)\n",
        "    out += torch.randint(0, len(vocab), (1,)).tolist()\n",
        "\n",
        "    # 文章の長さだけ繰り返し\n",
        "    for i in range(xnum):\n",
        "        # 結果の最後の文字に対しモデルを適用\n",
        "        ys = vtol(model(ltov(out[-1:])))\n",
        "        # 結果に追加\n",
        "        out += [ys[-1]]\n",
        "\n",
        "    return decode(out)\n",
        "\n",
        "# 文章を生成してみる\n",
        "print(generate(model))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "utMCQ2f0rFeq",
        "outputId": "6084b85f-2281-426e-8aff-34d045c8ef0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "W6)TfOcQKfélo7h]è(0o?eNSRB7U1vh½m;\"l)£JHB9!h£s&Kgm7TfhDy58b:?2m5B'n\" fS£gW[-VeFPK3P9:léiVEVLQtC?V7Geo\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "見るからに出鱈目です。ですが、どのくらい出鱈目でしょうか？モデルを学習させるためにはこの「出鱈目さ」を計算する必要があります。"
      ],
      "metadata": {
        "id": "2srvCmyEUizZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## モデルの評価"
      ],
      "metadata": {
        "id": "qoQ6OdN7DPqs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "作ったモデルを定量的に評価することを考えます。機械学習においては「損失関数」と呼ばれる関数を用いて、教師データとモデルのずれを評価します。ここでは「交差エントロピー」という関数を使います。詳細は発表スライドを参照してください。"
      ],
      "metadata": {
        "id": "Wv-EQh70nuTZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 損失関数に交差エントロピーを採用する\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "\n",
        "# バッチに対して損失関数を計算してみる\n",
        "xs, ys = get_batches(16)\n",
        "loss = loss_fn(model(ltov(xs)), ltov(ys))\n",
        "loss.item()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OsGfl9doGV-d",
        "outputId": "b96f9d5d-bda3-482c-d812-29acdaf0d22e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4.4361042976379395"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "この値はどう解釈すればいいのでしょうか? 今回の教師データ $\\vec{y}$ はワンホットベクトルで, ここでは $\\vec{y}$ が単語 $μ$ を表しているときの確率分布が $p = \\{p_ν\\} = \\{δ_{μν}\\}$ としてみましょう。(実際、`nn.CrossEntropyLoss()` ではこのような扱いになります。) このとき1単語のみの損失関数は\n",
        "$$\n",
        "\\begin{aligned}\n",
        "\\mathtt{loss}\n",
        "  &= ⟨- \\log q⟩ \\\\\n",
        "  &= - \\sum_ν p_ν \\log q_ν \\\\\n",
        "  &= - \\sum_ν δ_{μν} \\log q_ν \\\\\n",
        "  &= - \\log q_μ \\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "であるので、損失関数の値 `loss` から単語 $μ$ の生成確率がわかります:\n",
        "$$\n",
        "q_μ = e^{-\\mathtt{loss}}\n",
        "$$\n",
        "実際には損失関数は複数の単語についての平均値になるので、正しい単語の生成確率が\n",
        "$$\n",
        "q = e^{-\\mathtt{loss}}\n",
        "$$\n",
        "になるくらいの目安です。"
      ],
      "metadata": {
        "id": "NXygYyddKFyH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 単語の生成確率\n",
        "q = torch.exp(-loss)\n",
        "q.item()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jVAZ0LItUlph",
        "outputId": "85f4ad5c-c9f2-4ac2-8435-c97c62905624"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.011841981671750546"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "これでもまだよくわかりませんね。ところで、$N$ 個の単語から無作為に単語を選ぶとき、正しい単語が選ばれる確率は $\\displaystyle q = \\frac1N$ となります。つまり, $1/q\\ (=e^{\\mathtt{loss}})$ は無作為に単語を選び出したとするときの「母数」を表しています。これを計算してみましょう。"
      ],
      "metadata": {
        "id": "HrTgVnCeYNqa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 単語生成の\"母数\"\n",
        "1/q.item(), len(vocab)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mB2S_EfHcQWT",
        "outputId": "a3a9ade0-2a92-40ca-a7ec-8270b4434cce"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(84.44532576718426, 83)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "つまり $1/q≃\\mathtt{len(vocab)}$ であることがわかりました。モデルはほとんど無作為に単語を選んでいるということです。学習も何もしていないので当然の結果ですね。それではこれを改善すべく、モデルを学習させていきましょう。"
      ],
      "metadata": {
        "id": "9gdbURiQc1xP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## モデルの学習"
      ],
      "metadata": {
        "id": "T2qsjzR5fZgI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "モデルの学習をします。学習とは損失関数の値が小さくなるようパラメータの値を変えていく作業です。詳細は発表スライドを参照してください。ここでは [Adam](https://arxiv.org/abs/1412.6980) という確率的勾配降下法の改良を利用して学習を実行します。"
      ],
      "metadata": {
        "id": "yzZ3fA-crwz2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 学習 (最適化手法に Adam を採用する)\n",
        "def train(model: nn.Module, optim=torch.optim.Adam, epochs=100):\n",
        "    # optimizer を初期化\n",
        "    optimizer = optim(model.parameters())\n",
        "    # 学習過程を記録する配列\n",
        "    logs: dict[str, list[float]] = {\"train\": [], \"val\": []}\n",
        "\n",
        "    # epochs の回数だけ学習を繰り返す\n",
        "    for epoch in range(epochs):\n",
        "        # 一時的に学習過程を記録する配列(後で平均を取って log に加える)\n",
        "        log_temp: dict[str, list[float]] = {\"train\": [], \"val\": []}\n",
        "\n",
        "        # 学習と検証を10回ずつ実行\n",
        "        for _ in range(10):\n",
        "            for split in [\"train\", \"val\"]:\n",
        "                if split == \"train\":\n",
        "                    # 学習用\n",
        "                    model.train()\n",
        "                else:\n",
        "                    # 検証用\n",
        "                    model.eval()\n",
        "\n",
        "                # バッチを取得\n",
        "                xs, ys = get_batches(16, split=split)\n",
        "                # 損失関数を計算\n",
        "                loss = loss_fn(model(ltov(xs)), torch.tensor(ys))\n",
        "                # 「母数」の値を記録\n",
        "                log_temp[split] += [torch.exp(loss).item()]\n",
        "\n",
        "                # 学習を実行\n",
        "                if split == \"train\":\n",
        "                    # 勾配を初期化\n",
        "                    optimizer.zero_grad()\n",
        "                    # 誤差逆伝播\n",
        "                    loss.backward()\n",
        "                    # 学習\n",
        "                    optimizer.step()\n",
        "\n",
        "        # 10回の学習・検証の値を平均して記録\n",
        "        for split in [\"train\", \"val\"]:\n",
        "          logs[split] += [float(np.mean(log_temp[split]))]\n",
        "\n",
        "    # 学習過程のグラフを返す\n",
        "    return pd.DataFrame(logs).plot()\n",
        "\n",
        "# 学習を実行\n",
        "train(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "uO7evbn_ffcS",
        "outputId": "ff1669cb-aa5c-4519-8a99-88c4e1ca634f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Axes: >"
            ]
          },
          "metadata": {},
          "execution_count": 13
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8lElEQVR4nO3dd3iT5frA8e+b0b0H6Yay90Yo4EIUURGV40TF8XOiR8Vx9Hgcx4XrOI/iQlygxwFucaAgIHvvTVtaumm6aNMm7++PJ0mb0kJ3Ctyf68rVNHmTPH0pzZ3nue/70XRd1xFCCCGEaCMGbw9ACCGEECcXCT6EEEII0aYk+BBCCCFEm5LgQwghhBBtSoIPIYQQQrQpCT6EEEII0aYk+BBCCCFEm5LgQwghhBBtyuTtAdTmcDjIzMwkODgYTdO8PRwhhBBCNICu6xQXFxMXF4fBcPS5jXYXfGRmZpKYmOjtYQghhBCiCdLT00lISDjqMe0u+AgODgbU4ENCQrw8GiGEEEI0RFFREYmJie738aNpd8GHa6klJCREgg8hhBDiONOQlAlJOBVCCCFEm5LgQwghhBBtSoIPIYQQQrSpdpfzIYQQQrQWXdepqqrCbrd7eyjHJbPZjNFobPbzSPAhhBDipGCz2Th48CBlZWXeHspxS9M0EhISCAoKatbzSPAhhBDihOdwONi3bx9Go5G4uDh8fHykkWUj6bpObm4uBw4coFu3bs2aAZHgQwghxAnPZrPhcDhITEwkICDA28M5bkVHR7N//34qKyubFXxIwqkQQoiTxrHafouja6nZIvlXEEIIIUSbkuBDCCGEEG1Kgg8hhBDiJNGpUydeeeUVbw9DEk6FEEKI9uyMM85g4MCBLRI0rFq1isDAwOYPqplOmuAjs/Awc1akUWl38NB5vbw9HCGEEKJF6LqO3W7HZDr2W3p0dHQbjOjYTppll3JrDll/zqRq+Tvouu7t4QghhPAiXdcps1V55dKY96DrrruORYsW8eqrr6JpGpqm8cEHH6BpGj/99BNDhgzB19eXJUuWsGfPHiZOnIjFYiEoKIhhw4bx22+/eTxf7WUXTdN47733uPjiiwkICKBbt258++23LXWa63XSzHwkmq28aH6bQj2QjMKnSQiXOm8hhDhZHa600/vRn73y2lufGEeAT8Pefl999VV27txJ3759eeKJJwDYsmULAA8++CAvvvginTt3Jjw8nPT0dM477zyefvppfH19+eijj5gwYQI7duwgKSmp3tf497//zfPPP88LL7zA66+/zuTJk0lNTSUiIqL5P2w9TpqZD3NUFwDCtFL2pR/w8miEEEKIYwsNDcXHx4eAgABiYmKIiYlxN/d64oknOPvss+nSpQsREREMGDCAW265hb59+9KtWzeefPJJunTpcsyZjOuuu44rr7ySrl278swzz1BSUsLKlStb9ec6aWY+8Amk0BhFmD2PvNSt0L+7t0ckhBDCS/zNRrY+Mc5rr90Shg4d6vF9SUkJjz/+OD/88AMHDx6kqqqKw4cPk5aWdtTn6d+/v/t6YGAgISEh5OTktMgY63PyBB9ASWASYUV5lGft8vZQhBBCeJGmaQ1e+mivalet3Hffffz666+8+OKLdO3aFX9/f/72t79hs9mO+jxms9nje03TcDgcLT7emo7vM99IjoguULQWDu319lCEEEKIBvHx8cFutx/zuKVLl3Lddddx8cUXA2omZP/+/a08uqZpVM6H3W7nkUceITk5GX9/f7p06cKTTz7pkbmr6zqPPvoosbGx+Pv7M3bsWHbtah8zDQExXQEILk2VihchhBDHhU6dOrFixQr2799PXl5evbMS3bp1Y+7cuaxfv54NGzZw1VVXtfoMRlM1Kvh47rnnmDFjBv/973/Ztm0bzz33HM8//zyvv/66+5jnn3+e1157jbfeeosVK1YQGBjIuHHjKC8vb/HBN1ZYgurvkaAfJNPq/fEIIYQQx3LfffdhNBrp3bs30dHR9eZwvPTSS4SHhzNy5EgmTJjAuHHjGDx4cBuPtmE0vRFTABdccAEWi4WZM2e6b5s0aRL+/v588skn6LpOXFwc9957L/fddx8AVqsVi8XCBx98wBVXXHHM1ygqKiI0NBSr1UpISEgTfqSjyN4CM0Zi1QNYe+V6zuxpadnnF0II0S6Vl5ezb98+kpOT8fPz8/ZwjltHO4+Nef9u1MzHyJEjWbBgATt37gRgw4YNLFmyhPHjxwOwb98+srKyGDt2rPsxoaGhDB8+nGXLltX5nBUVFRQVFXlcWk14shqTVkZ6RkbrvY4QQggh6tWohNMHH3yQoqIievbsidFoxG638/TTTzN58mQAsrKyALBYPGcULBaL+77apk+fzr///e+mjL3xfAIo9ulAsC0H64HtQPucjhJCCCFOZI2a+fj888+ZPXs2c+bMYe3atXz44Ye8+OKLfPjhh00ewEMPPYTVanVf0tPTm/xcDWEL6QRAVd7uVn0dIYQQQtStUTMf999/Pw8++KA7d6Nfv36kpqYyffp0pkyZQkxMDADZ2dnExsa6H5ednc3AgQPrfE5fX198fX2bOPzGM0V3hbyV+BXtR9d1NE1rs9cWQgghRCNnPsrKyjAYPB9iNBrdpTzJycnExMSwYMEC9/1FRUWsWLGClJSUFhhu8wXFqc6mcQ6peBFCCCG8oVEzHxMmTODpp58mKSmJPn36sG7dOl566SVuuOEGQHVFu/vuu3nqqafo1q0bycnJPPLII8TFxXHRRRe1xvgbzRilen100rLYlV1MfJi/l0ckhBBCnFwaFXy8/vrrPPLII9x+++3k5OQQFxfHLbfcwqOPPuo+5oEHHqC0tJSbb76ZwsJCRo8ezfz589tPaVOE2mAuWctiZVYxZ/To4OUBCSGEECeXRvX5aAut2ucDwFYGz6h8lMd6fMu/rzy95V9DCCFEuyJ9PlqGV/p8nBB8AjjsrxJjy7PbR9t3IYQQorV06tSJV155xdvD8HDyBR+AHt4ZAOOhvbLHixBCCNHGTsrgw9fSDYAYewYHpeJFCCGEaFMnZfBhjFJJp520bHZmF3t5NEIIIUTd3nnnHeLi4o7YnXbixInccMMN7Nmzh4kTJ2KxWAgKCmLYsGH89ttvXhptw52UwYer4kWV25Z4eTBCCCHanK6DrdQ7l0Ys91966aXk5+fzxx9/uG8rKChg/vz5TJ48mZKSEs477zwWLFjAunXrOPfcc5kwYUK9O9+2F40qtT1hRFYHH59kt+JGdkIIIdqnyjJ4Js47r/3PTPAJbNCh4eHhjB8/njlz5nDWWWcB8OWXXxIVFcWZZ56JwWBgwIAB7uOffPJJ5s2bx7fffssdd9zRKsNvCSfnzEd4JwBCtMNkZ8nutkIIIdqvyZMn89VXX1FRUQHA7NmzueKKKzAYDJSUlHDffffRq1cvwsLCCAoKYtu2bTLz0S6Z/akMisNckok9b4/s8SKEECcbc4CagfDWazfChAkT0HWdH374gWHDhrF48WJefvllAO677z5+/fVXXnzxRbp27Yq/vz9/+9vfsNlsrTHyFnNyBh8426yXZGKpVBUvcdJmXQghTh6a1uClD2/z8/PjkksuYfbs2ezevZsePXowePBgAJYuXcp1113HxRdfDEBJSQn79+/34mgb5uRcdgEMrrwPQ5ZUvAghhGjXJk+ezA8//MD777/P5MmT3bd369aNuXPnsn79ejZs2MBVV111RGVMe3TSBh9EqEZjUvEihBCivRszZgwRERHs2LGDq666yn37Sy+9RHh4OCNHjmTChAmMGzfOPSvSnp20yy41K17+yJKKFyGEEO2XwWAgM/PIHJVOnTrx+++/e9w2depUj+/b4zLMSTzzUd1obFumBB9CCCFEWzl5g4/wTuhoBGuHOZR7AFtV+18jE0IIIU4EJ2/wYfaD0HgA4h1Z7MmVvA8hhBCiLZy8wQegOZdekg1ZbDsoSy9CCCFEWzipgw8iuwLQVcuU4EMIIYRoIyd38BHTF4De2n62HZReH0IIcaLTG7GpmzhSS52/kzz4UJvx9Daksi3TKr+UQghxgjKbzQCUlZV5eSTHN1fbdqPR2KznOXn7fAB06IWuGYiiCFNZNjnFFVhC/Lw9KiGEEC3MaDQSFhZGTk4OAAEBAbKnVyM5HA5yc3MJCAjAZGpe+HByBx8+AWhR3SF3O30M+9l6sEiCDyGEOEHFxMQAuAMQ0XgGg4GkpKRmB24nd/ABENNfBR/afrYdLOLMHh28PSIhhBCtQNM0YmNj6dChA5WVld4eznHJx8cHg6H5GRsSfMT2h02f09uQyo+SdCqEECc8o9HY7JwF0Twnd8IpqJkPcM98CCGEEKJ1SfAR0w+AJEMuubnZlFfavTwgIYQQ4sQmwUdABHpoIgC9tFR2ZsvSixBCCNGaJPgANFl6EUIIIdqMBB+gkk5xNhuTpFMhhBCiVUnwAe6k096a6vUhhBBCiNYjwQe4Zz66aRnsOZgnbdaFEEKIViTBB0BIPLp/BCbNQVzFfg4cOuztEQkhhBAnLAk+ADQNzTn70ccgSadCCCFEa5Lgw8Wj4kWSToUQQojWIsGHS0zNiheZ+RBCCCFaiwQfLs5ll15aGjsOHvLyYIQQQogTlwQfLpFd0c0BBGgVGA7tpbSiytsjEkIIIU5IjQo+OnXqhKZpR1ymTp0KQHl5OVOnTiUyMpKgoCAmTZpEdnZ2qwy8xRmMaJY+APTRUtmXV+rlAQkhhBAnpkYFH6tWreLgwYPuy6+//grApZdeCsA999zDd999xxdffMGiRYvIzMzkkksuaflRtxZ33sd+9uSWeHkwQgghxInJ1JiDo6OjPb5/9tln6dKlC6effjpWq5WZM2cyZ84cxowZA8CsWbPo1asXy5cvZ8SIES036tbiarOupbImV2Y+hBBCiNbQ5JwPm83GJ598wg033ICmaaxZs4bKykrGjh3rPqZnz54kJSWxbNmyep+noqKCoqIij4vXxFT3+tiTI+W2QgghRGtocvDx9ddfU1hYyHXXXQdAVlYWPj4+hIWFeRxnsVjIysqq93mmT59OaGio+5KYmNjUITVfh94ARGrF5OUc9N44hBBCiBNYk4OPmTNnMn78eOLi4po1gIceegir1eq+pKenN+v5msXsR1WABYCqglQcDtnjRQghhGhpjcr5cElNTeW3335j7ty57ttiYmKw2WwUFhZ6zH5kZ2cTExNT73P5+vri6+vblGG0CmN4EpRlE2XP4WBROfFh/t4ekhBCCHFCadLMx6xZs+jQoQPnn3+++7YhQ4ZgNptZsGCB+7YdO3aQlpZGSkpK80faRrTwJADitVz2SsWLEEII0eIaPfPhcDiYNWsWU6ZMwWSqfnhoaCg33ngj06ZNIyIigpCQEO68805SUlKOj0oXl1CVc5Kg5bE3t5RTu0Uf4wFCCCGEaIxGBx+//fYbaWlp3HDDDUfc9/LLL2MwGJg0aRIVFRWMGzeON998s0UG2mbCqoOPxTLzIYQQQrS4Rgcf55xzDrpedyKmn58fb7zxBm+88UazB+Y1oa5lFzXzIYQQQoiWJXu71BYmOR9CCCFEa5LgozbnskuoVkaxtYAym2wwJ4QQQrQkCT5q8wkE/whAll6EEEKI1iDBR12csx/xWi57ZXdbIYQQokVJ8FGXsOqk0z05kvchhBBCtCQJPurirHhJ0PJk5kMIIYRoYRJ81KXmsotUvAghhBAtSoKPuoS6gg+VcCobzAkhhBAtR4KPutTocnq40k5WUbmXBySEEEKcOCT4qIsz4TRKK8IXm5TbCiGEEC1Igo+6+IWBTzAACVoueyTvQwghhGgxEnzURdNqJJ3mSdKpEEII0YIk+KhPzaRTKbcVQgghWowEH/WRRmNCCCFEq5Dgoz7uipdcMq3lssGcEEII0UIk+KiPc9mlozEfgH2y9CKEEEK0CAk+6uNcdkk0qOBjj5TbCiGEEC1Cgo/6OIOPCEcBJqqk4kUIIYRoIRJ81CcwGkx+GHAQoxXIzIcQQgjRQiT4qI+mQWgCAIlaLrul4kUIIYRoERJ8HE2oZ6Mx2WBOCCGEaD4JPo7GmfeRZMinospBRuFhLw9ICCGEOP5J8HE0zl4fPfwKAWTpRQghhGgBEnwcTaia+ehkygOQDeaEEEKIFiDBx9E4Zz4sjlxAZj6EEEKIliDBx9E4E05DbNloOCT4EEIIIVqAydsDaNeCY8FgwuCoogOF7M71Rdd1NE3z9siEEEKI45bMfByN0QQhcQAkGnIpLKukoNTm5UEJIYQQxzcJPo7FmXTaL7AIkLwPIYQQorkk+DgWZ9Jp7wArALul4kUIIYRoFgk+jsXZaKyL2VlumyN7vAghhBDNIcHHsUT3BCCpch8gMx9CCCFEc0nwcSwx/QAIL9mFAQd7JOdDCCGEaBYJPo4lojOY/DHay+mkZZFReJgyW5W3RyWEEEIctyT4OBaDESx9ABjmlwnA3lzJ+xBCCCGaSoKPhojpC8DwgAxAym2FEEKI5mh08JGRkcHVV19NZGQk/v7+9OvXj9WrV7vv13WdRx99lNjYWPz9/Rk7diy7du1q0UG3OYsKPnobUgHZYE4IIYRojkYFH4cOHWLUqFGYzWZ++ukntm7dyn/+8x/Cw8Pdxzz//PO89tprvPXWW6xYsYLAwEDGjRtHeXl5iw++zTiTThNtewGZ+RBCCCGao1F7uzz33HMkJiYya9Ys923Jycnu67qu88orr/Cvf/2LiRMnAvDRRx9hsVj4+uuvueKKK1po2G3MmfMRWJFDGMXszgny8oCEEEKI41ejZj6+/fZbhg4dyqWXXkqHDh0YNGgQ7777rvv+ffv2kZWVxdixY923hYaGMnz4cJYtW9Zyo25rvsEQroKsXoY09ueXUmV3eHlQQgghxPGpUcHH3r17mTFjBt26dePnn3/mtttu4+9//zsffvghAFlZWQBYLBaPx1ksFvd9tVVUVFBUVORxaZecSacDTGlU2nXSCsq8PCAhhBDi+NSo4MPhcDB48GCeeeYZBg0axM0338xNN93EW2+91eQBTJ8+ndDQUPclMTGxyc/Vqiwq72Oon1S8CCGEEM3RqOAjNjaW3r17e9zWq1cv0tLSAIiJiQEgOzvb45js7Gz3fbU99NBDWK1W9yU9Pb0xQ2o7zpmPnpqqeJE260IIIUTTNCr4GDVqFDt27PC4befOnXTs2BFQyacxMTEsWLDAfX9RURErVqwgJSWlzuf09fUlJCTE49IuOSteYm2pmKmSDeaEEEKIJmpUtcs999zDyJEjeeaZZ7jssstYuXIl77zzDu+88w4AmqZx991389RTT9GtWzeSk5N55JFHiIuL46KLLmqN8bed0ETwC8VYbqWrlsHu3Chvj0gIIYQ4LjUq+Bg2bBjz5s3joYce4oknniA5OZlXXnmFyZMnu4954IEHKC0t5eabb6awsJDRo0czf/58/Pz8WnzwbUrTVLOx1KX00lL5JacLuq6jaZq3RyaEEEIcVzRd13VvD6KmoqIiQkNDsVqt7W8J5scHYOXbvFd1Hk9VXc3yh84iJvQ4D6qEEEKIFtCY92/Z26UxnEmnA30OAEi5rRBCCNEEEnw0hjPptDupgM6BQxJ8CCGEEI0lwUdjRPcCzUiIw4qFQxw4dNjbIxJCCCGOOxJ8NIbZD6K6AdDLkEq6LLsIIYQQjSbBR2NZVN5Hby1NZj6EEEKIJpDgo7GcSae9DKkcKJSZDyGEEKKxJPhoLGfSaW8tlczCctndVgghhGgkCT4ay7nBXCctC7OjnKyici8PSAghhDi+SPDRWMEWCIjEqOl00Q5K3ocQQgjRSBJ8NEV4JwDitVypeBFCCCEaSYKPpghLAiBRy5WZDyGEEKKRJPhoCmfwkSDBhxBCCNFoEnw0RVhHwBV8yLKLEEII0RgSfDSFR/AhMx9CCCFEY0jw0RTuZZc8DlrLqJReH0IIIUSDSfDRFGGJAARrhwnRS8iySq8PIYQQoqEk+GgKsz8EWQC19CLltkIIIUTDSfDRVDWWXiTvQwghhGg4CT6aypl0mqjlSMWLEEII0QgSfDRVjV4f6TLzIYQQQjSYBB9N5bHsIjMfQgghRENJ8NFU4TWXXWTmQwghhGgoCT6ayt1oLI+sosNUVNm9PCAhhBDi+CDBR1OFJqCjEaBVEKEXcbBQen0IIYQQDSHBR1OZfNGCYwFpsy6EEEI0hgQfzVEj6TRdkk6FEEKIBpHgozmcwYf0+hBCCCEaToKP5giX3W2FEEKIxpLgozlqLrvI/i5CCCFEg0jw0Rweyy4y8yGEEEI0hAQfzeHs9RGv5ZFTXE55pfT6EEIIIY5Fgo/mCE1A1wz4aZVEYyWjUGY/hBBCiGOR4KM5jGa0kHhAll6EEEKIhpLgo7lkgzkhhBCiUST4aC538CEzH0IIIURDSPDRXGHVvT6k3FYIIYQ4Ngk+mqvGsstBq2wuJ4QQQhxLo4KPxx9/HE3TPC49e/Z0319eXs7UqVOJjIwkKCiISZMmkZ2d3eKDbldq9PrIkuBDCCGEOKZGz3z06dOHgwcPui9Llixx33fPPffw3Xff8cUXX7Bo0SIyMzO55JJLWnTA7U54da+P3OIyHA7dywMSQggh2jdTox9gMhETE3PE7VarlZkzZzJnzhzGjBkDwKxZs+jVqxfLly9nxIgRzR9texQch64Z8cFOuP0QBWU2ooJ8vT0qIYQQot1q9MzHrl27iIuLo3PnzkyePJm0tDQA1qxZQ2VlJWPHjnUf27NnT5KSkli2bFm9z1dRUUFRUZHH5bhiNKGFVvf6kKUXIYQQ4ugaFXwMHz6cDz74gPnz5zNjxgz27dvHqaeeSnFxMVlZWfj4+BAWFubxGIvFQlZWVr3POX36dEJDQ92XxMTEJv0gXuWueMkju0iCDyGEEOJoGrXsMn78ePf1/v37M3z4cDp27Mjnn3+Ov79/kwbw0EMPMW3aNPf3RUVFx18AEtYRWEyClkuWBB9CCCHEUTWr1DYsLIzu3buze/duYmJisNlsFBYWehyTnZ1dZ46Ii6+vLyEhIR6X44674iWXbFl2EUIIIY6qWcFHSUkJe/bsITY2liFDhmA2m1mwYIH7/h07dpCWlkZKSkqzB9quOSteOhqyZeZDCCGEOIZGLbvcd999TJgwgY4dO5KZmcljjz2G0WjkyiuvJDQ0lBtvvJFp06YRERFBSEgId955JykpKSdupYtLZFcAkrWDZBVVeHkwQgghRPvWqODjwIEDXHnlleTn5xMdHc3o0aNZvnw50dHRALz88ssYDAYmTZpERUUF48aN480332yVgbcrkV0AsGiFFBUWeHkwQgghRPum6brerrpiFRUVERoaitVqPa7yP6qe7YypPJ/LeY7/PX6rt4cjhBBCtKnGvH/L3i4tJaobABZbOuWVdi8PRgghhGi/JPhoIcaoGnkfUvEihBBC1EuCjxaiOYOPzoaDUvEihBBCHIUEHy0lUi27JGsHpcupEEIIcRQSfLQUd7ltFlmFh708GCGEEKL9kuCjpUQk48BAsHaYkoIMb49GCCGEaLck+GgpJl9K/OMAMOTv8fJghBBCiPZLgo8WVB7SGQC/or1eHokQQgjRfknw0ZKcnU5Dy9K8PBAhhBCi/ZLgowX5WLoDYKlMx+FoV41jhRBCiHZDgo8WFBTfC4BOHCS/1Obl0QghhBDtkwQfLcgUrXp9JGk5ZBcWe3k0QgghRPskwUdLCo6jHF/Mmp2izN3eHo0QQgjRLknw0ZIMBnJ8EgCoyN7p5cEIIYQQ7ZMEHy3MGtARAK1AZj6EEEKIukjw0cLKQ5IB8LNKrw8hhBCiLhJ8tDBHhOr1ESK9PoQQQog6SfDRwnwsPQCw2NK9PBIhhBCifZLgo4WFxPcEIFIvgAoptxVCCCFqk+CjhUV3sJCrhwBQnr3Ly6MRQggh2h8JPlpYsK+JNNTutkUHtnp5NEIIIUT7I8FHC9M0jSyzs9dHlvT6EEIIIWqT4KMVWP1Vrw/ypdeHEEIIUZsEH63gcGhnAHyLpNeHEEIIUZsEH63A1esjtDQVdN3LoxFCCCHaFwk+WoFvdGfsuoavowxKsr09HCGEEKJdkeCjFXQIDyFd76C+yZNyWyGEEKImCT5agSXEj716rPomX4IPIYQQoiYJPlpBTGh18KHnSvAhhBBC1CTBRyuIDvJln64ajdlypNeHEEIIUZMEH63AZDSQ75ekvpFlFyGEEMKDBB+txBamym19itKhqsLLoxFCCCHaDwk+WkmEJZEi3R8NBxRIszEhhBDCRYKPVtKlQ3B1xYuU2wohhBBuEny0kq4dgtjrTDqVvA8hhBCimgQfraRLdCB7Hc5yW5n5EEIIIdyaFXw8++yzaJrG3Xff7b6tvLycqVOnEhkZSVBQEJMmTSI7++RrMZ4UEUCqFg+ALXuHl0cjhBBCtB9NDj5WrVrF22+/Tf/+/T1uv+eee/juu+/44osvWLRoEZmZmVxyySXNHujxxmQ0UO7c3daQv1s2mBNCCCGcmhR8lJSUMHnyZN59913Cw8Pdt1utVmbOnMlLL73EmDFjGDJkCLNmzeKvv/5i+fLlLTbo44W/pRsOXcNcWQSled4ejhBCCNEuNCn4mDp1Kueffz5jx471uH3NmjVUVlZ63N6zZ0+SkpJYtmxZnc9VUVFBUVGRx+VEkWSJJEOPUt9I0qkQQggBNCH4+Oyzz1i7di3Tp08/4r6srCx8fHwICwvzuN1isZCVlVXn802fPp3Q0FD3JTExsbFDardUxYur3FbarAshhBDQyOAjPT2du+66i9mzZ+Pn59ciA3jooYewWq3uS3p6eos8b3vQJTqIPa5yW6l4EUIIIYBGBh9r1qwhJyeHwYMHYzKZMJlMLFq0iNdeew2TyYTFYsFms1FYWOjxuOzsbGJiYup8Tl9fX0JCQjwuJ4rO0YHumQ/ZYE4IIYRQTI05+KyzzmLTpk0et11//fX07NmTf/zjHyQmJmI2m1mwYAGTJk0CYMeOHaSlpZGSktJyoz5OBPqasAZ0gkpw5ErwIYQQQkAjg4/g4GD69u3rcVtgYCCRkZHu22+88UamTZtGREQEISEh3HnnnaSkpDBixIiWG/VxRIvqBgfBpzgdqmxg8vH2kIQQQgivalTw0RAvv/wyBoOBSZMmUVFRwbhx43jzzTdb+mWOGxExHSnJ9COIcji0D6J7eHtIQgghhFdput6+ul8VFRURGhqK1Wo9IfI/PlmeSv8fJ9LfsA8unw29LvD2kIQQQogW15j3b9nbpZV1ia5Rbiu9PoQQQggJPlpb1w5B7HWoctuqHNnjRQghhJDgo5VFBfmQZU4AwJYtFS9CCCGEBB+tTNM0qsK7AGAq2O3l0QghhBDeJ8FHG/CPVRUuPpVWKM338miEEEII75Lgow0kWaI44NpgTvZ4EUIIcZKT4KMNqKRTqXgRQgghQIKPNlGz3NaRK8GHEEKIk5sEH20gMSKAVC0egIosKbcVQghxcpPgow0YDRrlwckAOPKl4kUIIcTJTYKPNuIbrYIPn5IMaF8d7YUQQog2JcFHGwmP7QSA2VEOZQXeHYwQQgjhRRJ8tJFOMZHk6qHqG2u6dwcjhBBCeJEEH22kS3QQGa5eHxJ8CCGEOIlJ8NFGukQHkalHAlCas9+7gxFCCCG8SIKPNuLvY8Tqq3p9FGXt9fJohBBCCO+R4KMN2YNVr4/KglQvj0QIIYTwHgk+2pA5oiMAxuIML49ECCGE8B4JPtpQSGxnAILLs7w8EiGEEMJ7JPhoQ5bErgCEOAqh8rB3ByOEEEJ4iQQfbSg5Pp4S3Q+Astz93h2MEEII4SUSfLSh8CBfsjXV6yM7TfZ4EUIIcXKS4KONWX1Uue2hrD1eHokQQgjhHRJ8tDFbkCq3rciVclshhBAnJwk+2pgxPFFdsR7w7kCEEEIIL5Hgo40FWZIBCDic6eWRCCGEEN4hwUcbi4rvAkB4VQ6VdoeXRyOEEEK0PQk+2lhknAo+YsknNbfYy6MRQggh2p4EH23MEBqHHQNmzU562j5vD0cIIYRocxJ8tDWDEaspGoCCTCm3FUIIcfKR4MMLDgeqctuynLad+ThoPYzDobfpawohhBC1SfDhDaGq3NZemN5mL7l8bz4p03/nyR+2ttlrCiGEEHWR4MML/KM7AuBXmoGut81MxOYMKwBbMora5PWEEEKI+kjw4QUhls4AdHDkctBa3iavmVdiU19LK9rk9YQQQoj6SPDhBaYItewSp+WzJ7ekTV4zv6TC+dXWJq8nhBBC1EeCD28ITQIgXstjd3bb9PrIL1VBh/VwpTQ3E0II4VUSfHhDaAIAwdphDmRltclLumY+AA6VyuyHEEII72lU8DFjxgz69+9PSEgIISEhpKSk8NNPP7nvLy8vZ+rUqURGRhIUFMSkSZPIzs5u8UEf93wCqPAJB6A4q23KbfNqLLfkydKLEEIIL2pU8JGQkMCzzz7LmjVrWL16NWPGjGHixIls2bIFgHvuuYfvvvuOL774gkWLFpGZmckll1zSKgM/3tlD1OxHVUFqq7+Wruvk10g0zZekUyGEEF5kaszBEyZM8Pj+6aefZsaMGSxfvpyEhARmzpzJnDlzGDNmDACzZs2iV69eLF++nBEjRrTcqE8APhFJkLeJoIosCstshAX4tNprldnslFdW53lI0qkQQghvanLOh91u57PPPqO0tJSUlBTWrFlDZWUlY8eOdR/Ts2dPkpKSWLZsWb3PU1FRQVFRkcflZGCKUL0+4rW8Vq94qR1s5JXIzIcQQgjvaXTwsWnTJoKCgvD19eXWW29l3rx59O7dm6ysLHx8fAgLC/M43mKxkHWUpMrp06cTGhrqviQmJjb6hzguObucxmv5/LipdZNOa/f2KJCEUyGEEF7U6OCjR48erF+/nhUrVnDbbbcxZcoUtm5tesvuhx56CKvV6r6kp7ddy3Gvcla8xGt5fPjXfnbntF7Jbe2ZD1l2EUII4U2NyvkA8PHxoWvXrgAMGTKEVatW8eqrr3L55Zdjs9koLCz0mP3Izs4mJiam3ufz9fXF19e38SM/3oWpmY9k8yGqbDqPf7uVj288BU3TWvyl8msts0jCqRBCCG9qdp8Ph8NBRUUFQ4YMwWw2s2DBAvd9O3bsIC0tjZSUlOa+zInHuewSZs8n0GRnye485m9uneUXV4OxsAAzIKW2QgghvKtRMx8PPfQQ48ePJykpieLiYubMmcPChQv5+eefCQ0N5cYbb2TatGlEREQQEhLCnXfeSUpKilS61CUgEkz+UHWYe4cF8MSyCp76YRtn9OiAv4+xRV/KlWDa3RLMyn0FMvMhhBDCqxo185GTk8O1115Ljx49OOuss1i1ahU///wzZ599NgAvv/wyF1xwAZMmTeK0004jJiaGuXPntsrAj3ua5l56mdzLSHyYPxmFh5mxcHeLv5Qrx6OHJRiAApn5EEII4UWNmvmYOXPmUe/38/PjjTfe4I033mjWoE4aYR0hbye+Oev51/lXcNvstbz1514mDUmgY2Rgi72Ma6ajuyUIgFKbncM2e4vPsAghhBANIXu7eFPvC9XXNR9wbp8OjO4aha3KwVM/bGvRl3HNfHSMDMTHqP7JZelFCCGEt0jw4U19J4FvKBzaj7bnDx6d0BuABduyKSxruaURV4JpVJAvkUGqk6qU2wohhPAWCT68yScQBl6lrq96j+6WYLpbgnDosGR3Xou8hMOhU+Cc5YgK8qkOPmTmQwghhJdI8OFtw25UX3f9DIVpnN49GoBFO3Jb5OkLD1fi0NX18EAfIgNVTxWZ+RBCCOEtEnx4W1Q3SD4ddAes+YDTu3cAYNHOXHRdb/bTu2Y9wgLMmI0GIgNdMx8SfAghhPAOCT7aA9fsx9qPGJoQgL/ZSE5xBduzmt9y3ZXv4Qo6qnM+ZNlFCCGEd0jw0R70OA+CY6E0F7/dP5LSJRJQsx/Nle8OPtRyS2SQLLsIIYTwLgk+2gOjGYZcp66vmtn4vI/cHfDBBZC+8oi7XImlrhkP1wxIniy7CCGE8BIJPtqLwdeCZoS0vxgboSpdVqcWUFJRdezHLnsD9i9WX2txL7s4g48o98yHLLsIIYTwDgk+2ouQOOh5PgDxu+fQMTKASrvOsj35x37s/iXqa/aWI+5yBRmuZZcI58xHgcx8CCGE8BIJPtqTYf+nvq6fw4Wd7AAs2plz9McUHYSCPep6wR6wlXncne9uMFY74dTWItU0QgghRGNJ8NGeJJ8GHUdB1WGuLnoPgIU7jlFym7q0+rrugFzP1uzVOR/OhFPnDIjN7qC4IUs6QgghRAuT4KM90TQY/xxoBizpP3GqaRsHDh1mX15p/Y9xLbm4ZG32+Da/Vqmtv4+RQOeGclLxIoQQwhsk+GhvYvrBUNX34xm/jzFiP3rJrSv4iOyqvtbK+8gr8Zz5qHldkk6FEEJ4gwQf7dGZ/wT/CBKr9nO18bf6g4/ibMjfBWjV+SLZ1TMftioHReVqacWV8wE18j4k6VQIIYQXSPDRHgVEwFmPADDN9AU79u6jvNJ+5HGpzlmPmL7QabS6nr0ZnDkirooWk0EjxM/sfpi7xbosuwghhPACCT7aq8FT0GP6E6qVcaf+Kcv31lFyu9+ZbNrpVIjqDgYTlFvBegCoXnKJCPTBYNDcD6veXE6WXYQQQrQ9CT7aK4MR7bwXALjCuJA58745sjeHK9+j4ygw+UJUD/W9M+/DtaxSM99DfS/LLkIIIbxHgo/2LGkE5b0mYdB0biibyU0frqpefinJhbwdgAYdR6rbLH3U1+xNQPXMRs18D6gORvJk5kMIIYQXSPDRzvmd+wQOoy8jDNsIO7CAez/fgMOhV+d7WPqoHBFQuR9QPfNRq8zWxRWMSJdTIYQQ3iDBR3sXmoBhxG0A/NP8KfM3HeDZ+dtr5HuMrj7WNfPh7PWRV3pkmS1Ut1iXhFMhhBDeIMHH8eDUaeAfQRctk8uMC3nnz70c2vaHuq/jqOrjLP3UV2eb9fxam8q5uBNOS1to2WXzXHixO6SvapnnE0IIcUKT4ON44BcKp/8DgEcCviZByyW8ZLe6r2bwEdQBAqLcbdbdOR+BnjMfNZdd7I4W2N9l7UdQkg0b5jT/uYQQQpzwJPg4Xgy9AcKTCajM5x2/1wEoC+sOgZHVx2iaR95HdbWL58xHuHPZxaFDYVkzl150HbI2qusZa+o9LKeonPmbsyizyX4yQghxspPg43hh8oGxjwHQW1ezHnsDBx55nMUZfGRtrrHs4jnzYTYaCAtQTceanXRalAFlzh4k2Vugstx9l67r/LU7j9tnr2Hks79z6ydreG3B7ua9nhBCiOOeydsDEI3Q+yJIGAYHVG7FX/Ze9K19jDP40LM3k1dyJnBktYvrtsKySvJKbHSzNGNMBzdWX3dUQdYmSBzGV2sO8MbC3ezN9dwUr85maUIIIU4qMvNxPNE0OOcpAOy6xjcFHY88xr3sspmKKtUTpPayC7Rg0unBDZ7fZ64lv6SC+77cwN7cUgJ9jFw9Iom3rxkCwNbMImxVjua9phBCiOOazHwcb5JGUD5hBvfP3cqWCj9yisrpEOJXfb+zzbpWbiWOfA6ZLQT4mFRuRvYWdb/Jp7rLaXPLbV35HoEdoDQHMtawKexidB2SIgL48a5TCfI1oes6of5mrIcr2ZFVTL+E0Oa9rhBCiOOWzHwch/yGXMXOqHMA2HDA6nlnjTbrPQ1pKsjQdfjpAXhrFMw6F2ylNYKPFpr5GHS1+pqxlk3OMQ1KCiPIV8W3mqbR3xlwbDhQ2LzXFEIIcVyT4OM4NSDR+UaeXnjknc5mY720NKICzfDDvbDyHXVfxhr48kaiAoxAM/d3Kc1TCacAQ6aor/m72J2mbusX7zm7MSAhDICNEnwIIcRJTYKP49SAxDAA1tcVfDjzPnob9nNP+Zuweiagwai7wOQHO39ifPrLgF697KLrsGUevHMGLHmlYYNwzXpEdIbwTuoC6JlrAehbK/hwz3yk15qtESen4mz1eyeEOOlI8HGcGugMPjYcKFR7vdTknPkYb1jF6SU/AhpcNAPOfgIueRfQ6JH+ObcZv1MJpwV7Yfbf4IvrIHMdLHwWKoqPPQhXvkfsAPU1bjAACWXb0TToExficbgrYNqVUyz9Pk52uxfAf7rDgie8PRIhhBdI8HGc6m4Jxs9soLi8in35nuWsrjbrBk3HgQEueQcGXqnu630hnDsdgH+YP+OmvOfgzRTY/RsYfcA/HKoOw7bvjz0IV5ltTH/1NV5VtAww7CE5KpBgP7PnsEL8iAnxw6HD5oyipv3g4sSQvlJ93fend8chhPAKCT6OU2ajgb5x9eR9BHUg25yIXddY2Ocp6H+Z5/0jbqNwwE0AnFO1EKrKofMZcNsyGK42sWPT58cehGvZxTXzEa9mPvob9tI/vu5qFtfSi+R9nOSsB9TX3B2y9CLESUiCj+NYvXkfmsbjUS9ypu0lirpeVOdj9bOfYk7VGPY6Yqi66F245muI6gr9/qYO2LtQrcnXp7xIbWAH1cFH7AAcGIjVCjglqu4qmqPmqoiTR5Ez+LAVQ/FB745FCNHmJPg4jrneyOuqeNl3OIA03VJngzGA0ABfHnHcxBjbS+R3vlA1MAOI7ALxQ9XmdFvm1v/i2ZvV15B4CIxS130C2aslAjDUtL/Oh1XPfLRy0ml5EVS10K69ouVZM6qv5+7w3jiEEF7RqOBj+vTpDBs2jODgYDp06MBFF13Ejh2efzjKy8uZOnUqkZGRBAUFMWnSJLKzj/IJWjTZQGfp6taDRe5upgC2KgcHrWqPlchaO9q6GAwaEc6263m1e324lmk2HmXppXa+B5BbXMGaymQAOtm21/mw/vFqzGkFZRyqo8x3d06xx8/SJIXp8Eo/+Pji5j2PaB26Xr3sApC303tjEUJ4RaOCj0WLFjF16lSWL1/Or7/+SmVlJeeccw6lpdUJj/fccw/fffcdX3zxBYsWLSIzM5NLLrmkxQcuIDHCn4hAHyrtOtsOVlenvLVoD9bDlUQG+tA5OrDex1tCVGCSll/meUefS0AzQuZayN9T94Nr53sAmzOsbNC7AOCTtb7Oh4UGmEmOUmPamOE5+/H1ugzGvvQnD361qd4xN8jqmVBeCKlL4VBq855LtLzDh1RSs4vMfAhx0mlU8DF//nyuu+46+vTpw4ABA/jggw9IS0tjzRq1lbrVamXmzJm89NJLjBkzhiFDhjBr1iz++usvli9f3io/wMlM0zQGJHgmne7OKeG/v6udYx+d0Bs/s7Hex/dzzkKsOyJhNRq6qE3p6p39cJfZVs98bMqwssHRWX2TubbeRML+CUcmylZU2XnhZ/Um9PX6DHbnlNQ77qOqLIe1H1V/v+f3pj2PaD01Zz1AZj6EOAk1K+fDalWfXCMiIgBYs2YNlZWVjB071n1Mz549SUpKYtmyZXU+R0VFBUVFRR4X0XA1EzgdDp1/zt2Eze7gjB7RXDgg7qiPHZykHrs29dCRd/ZzLr1s+vzIIKKyHHKdyyo1Zj42ZVjZoSdSZfCBcqvqH1KH/nV0Ov1sZToZherTsK7D24vqmXE5lq3fQFmNnXP3LGja84jW4wo+TM49iXLrXqITor3LLDzMgm3Z6FKx1WhNDj4cDgd33303o0aNom9f1VEzKysLHx8fwsLCPI61WCxkZWXV+TzTp08nNDTUfUlMTGzqkE5KNZNO/7c6nZX7C/A3G3lyYl80VxJpPQZ3DAfU8scRO832PB/MASqAyFjreV/OVnBUgX+ESjh12nTAShUmyiNVkzMy1tT5ugNdreEPWNF1nTJbFa87Z2suHZIAwLx1Ge5gpFFWvau+dh+vvu79E+zS0KxdcbXkT0pRX0tzoazAe+MRTbfnD3i5H+xb7O2ReMW0z9dz44erWbW/jg9w4qiaHHxMnTqVzZs389lnnzVrAA899BBWq9V9SU9Pb9bznWxc+6XszSvlmR+2AXDvOd1JjAg45mM7RwUSFmDGVuVg68FaM06+QdDjPHW9ds+Pmp1NnQFObnEFWUXlaBr4dhym7q8dtDj1jg3FaNDcj/nwr1TySipIjPDn6Yv7kdI5kiqHzrt/1j1zUq/M9XBgFRjMcMHLqmFahbXeIEh4idX5fzy6B4SoYFOWXo5Ta2aBNQ3WfeLtkbS5KruDdWmFgOraLBqnScHHHXfcwffff88ff/xBQkKC+/aYmBhsNhuFhYUex2dnZxMTE1Pnc/n6+hISEuJxEQ0XEehDkjPQKK6ool98KNeN7NSgx2qaxuAkNfuxpq6lF2fVi2PTV56zB+5k0+p8j83O5NEu0UGYE4eqG+t50/f3MdLdEgzA4l15vOVcYrn7rO74mAzcfqZKWv1sVVrjdt1d9Z762nsihMSqxmnQvKWXtR/BwudOyEZYy/fmN39X46ZwldmGJkB0d3Vdkk6PTxnr1FdX6f1JZG9eKRXOGePMpszSnuQaFXzous4dd9zBvHnz+P3330lOTva4f8iQIZjNZhYsqP5jv2PHDtLS0khJSWmZEYsjuJZejAaN6Zf0w2Rs+D+rO+8jrY7go8sYSk1hGMpyyX39LFjzocrlOFhj5sPJ1bejX3you806WRtVfkhdY3YmnT7z4zashyvp2iGIiwapJZzRXaPonxBKeaWDWUv3N+wHOXwINn2prp9yk3P8Z6mvu5sYfNjK4Pt7YOEzJ1xewur9BVzxznLu/WJD27+4a9klJB6ieqjrMvNx/CnJVbMeoP5/nGR9dbZkVlfrHSys+++cqF+jgo+pU6fyySefMGfOHIKDg8nKyiIrK4vDh1XUFxoayo033si0adP4448/WLNmDddffz0pKSmMGDGiVX4AAef2UbNKd5zZ9YidZI/FNfOxro6Zj3KHgf/YJuHQNaIL18N3f4cXu6vN5wBiPJNNwbmTbWQXCI5Vbdv3170W7AqYCssqAZh2dneMBrWEo2kat5+hZj8+XLaf4vLKY/8g62ar8k1LP0gcrm7rMkZ9zVzbtJyCg+tVbguccJ/MXR1mt2Z6IcHblXAamqCWXuCEO78nhcway6qOqhMuQD+WLTX2p2pSftpJrlHBx4wZM7BarZxxxhnExsa6L//73//cx7z88stccMEFTJo0idNOO42YmBjmzj1Kp0zRbOf3j2XDY+dwz9ndG/3YAYlhGDTItJaTZfWM3hfvyuN921mMsr3Os5VXsMsRrwIK3Q6+oRDR2X2sa9mlf0KoygPpfq66Y8ePdb6uq9wW1O63rgDK5ZzeMXSJDqS4vIpPlqcd/YdwOKqXXIbdWN2tNTQeonuqbq37Fh3rVBzpwKrq63m7Gv/4dsxVypxTXMFhWzObujWGww5Fmep6zeAjr2HBx0fL9nPB64s5cKjs2AeL1lU7pyvr5Fp6qZknl2mV4KOxGr3sUtfluuuucx/j5+fHG2+8QUFBAaWlpcydO7fefA/RckL9zcc+qA6BviZ6xKg8m9pLLz9uUntujEsZTMGg2znb9jwXVT7Fvl63qp1yDerXJ6e43J1s2jvWmbPT83z1dcdPKjiopbslmEAf1YPkvnN6YDB4VuYYDBq3ndEVgJlL9vLX7jwcjnryLvb+Dof2qYCo9iZ6jVx6qbLXGKtH8HFiLQvU7KPSpm/kJdkqeDWYIMhSvexSmAa20qM/Fnhv8T42ZxTxQUOX40TrceV0+To/SGQ1szngcUTXdbbUmDXMspbX//dJ1En2dhF19vuoqLLz21bVFv+C/rFMv6Q/Fw6IZ729M+M2nsHHh3qxJvUQOUXlbDpQnWwa6GtST9DpVDAHqk3DDq4/4jXNRgNfjc7g+4ErOKNbRJ3jmjgwjsQIf/JKbFz13gpGPfc7z83fzq7sGpnlDgcsfkldH3gV+NTq6NrVufSy5/djJo3+sPEgPR6Zz9y1B9Sx6TWDj+YtCxSVV7Jib/6xD2wDuq6zO7c6+EgraMPgw7XkEhwLBiMERkJApLrtGLNLh0pt7rF+vT6DSvuRQa1oI7pevezS/1L19SQKPjIKD2M9XInJoGHQoNKuH7lNhTgqCT6EO++j5szH0t15FFdUYQnxZXBSOEaDxn8uG8C4PhZsdgePfL2ZSTP+4pRnFnDjh6sB6F8z38TsB12dsw51Lb1YD9Bz2f303f4q2h9P1Tkus9HAnP8bwVXDkwjxM3HQWs6MhXs4++U/efL7reqglW+rNurmABhx25FP0nEUGH1VkuNR8gp0Xee1BbuwO3Tmb85Sx5fU6E2Tt6vOGZyGKLNVcdlby7j8neXugM6b8ktt7lwbgNTa7fVbU818D5cGJp1uqNGULq/ExqIduS08ONFghamqmZ/BDAOuUrdlbzohq8Lq4pr16GYJJiZENcuTvI/GkeBDuJuNbc6o3qDux03qjXd831j3kojZaOC1Kwdx2xldOKVTBPFh/tRcLTmte7TnE7uWXrbXEXyseFtNvwMsebm6UqWWxIgAnrm4HysfHsuMyYMZ07MDAJ+tTMOWtQ1+e1wdeM6TEN7xyCcw+0PHker6UVqtr0k9xA7njMqunJLqJRdLX7VEUFkGxZn1Pr4+uq7z8LzNbM9Sz/3dxsY/R0ur3bq+TWc+ala6uDQw6XRDuhXQ8dFUEvBXaw8c9XhRQxMD53q58j1i+kJMPxWElFure7ic4FzBR5+4EGLD/AHcm3mKhjF5ewDC+zpFBhAR6ENBqY0tmUX0jQvlly2u4MMzX8fXZOQf5/Z0f19pd5BlLae80k7XDkGeT9ztHLVBXc4WOLQfwjup2ytKVNkuqOWZ/Yvhm6mqSiZuUJ1j9DMbGd8vlnF9Yhg+fQEFxWVUfHELPlXlqqpl6I31/4Bdz4K9f6h+Hym313nI7BXVSa2p+aVUpa1U/zmSUsBuU5/K83Z6fmJvgNkr0pi3rnr7+D+251Bpd2BuRDl0S6sdfKR7Y9kltI7g4xhLWxvTC3jV/AbnmVZzUfnj/LZN41CpjXDn7syiHtlbYOY5MOQ6GPd0yzyna8klbjCYfFRid/YmtfQSltQyr9GObXWW2faJC6GiysGa1EPS66ORZOZDoGkag5ylr2tTD7Fsbz5F5VVEBfkytFPd+RguZqOBxIgAulmCj2znHhBR3UJ7x0/Vt6+frTqPRnaFa75WQUpVOXw2GYqPvixhMGic2SOa24zfEpy/QSW7Xfjf6gqXuriSTvcvrbPvSEGpjR82quRak0HDoUPF/hXqzoRhEOVqhNW4pNMN6YU88Z1aHvrHuT2JDPShqLyKlfsaVvZrd+jc9dk6Hpq7sUX3jtjjzPfoGaMavXkl5yO0xjYKDTi/uq4zJH0WE41/YdZtPBA8n0q7zrcbvD+T1O6teBtsJaoLqaOFKptcMx/xg9XXmH7q60lS8eIqUe8dG0JcmCy7NIUEHwKoXnpZl1bIT84ql3P7Wty9N5qsh3OPle0/qK8OOyx/U10fcRsYTTDpPYjspqbkP7/mmM2KJlryuMvkLN8+7wXPT9F1OBTYhTLfDlB1mNKfHoXUvzyCkC/XpGOzO+gbH8LgjuGYqcIv15k8lzAUorqp63k7SS8o48L/LuGb9Rl1vFKN1yy1cfvstdjsDs7pbeHW0ztzVi+1ZPRrA/M+1qcX8s36TD5dma6WglqIa+bDtYSVVlDWdhtjOYOPyqBY3lu8V1XauGY+CvaAve6eLvnrv+dWe/VWDqdW/kUHDvHlGll6OarKw7BlnrpeXli9NUJzOOxqKwOobigYo/b3apHnb+cOldrIdC6x9I4LIS7UuezSDhqNZVnLufjNpXyxuv0vf0nwIQAY5Kx4WZ1awM/OJZfz+sY2/4l7OveHSf1LdSHd8ZNagvEPhwFXqvv8QuHKz9QsRvoKmHM5FNbzn6ckhxEbH8as2fnJPozU+PPrPqyiinnrDnD9rJUMe2YBX5eqP46Ba9+GWeNhegK8dzaO9Z8yx7nkcvXwjnS3BNFLS8XosKmN8yI6eyREfrnmABsPWHnk680U1dP8zOHQuefz9WQUHqZTZAAvXjYATdMY28sCqOCjIW/2f2zPcV9vaMDSEHucwcdp3aMxaFBR5SC3uI0y9Z05H9/vN/DUD9t46vttKv/DJ0g1qqprJ+SCvYT+eDsGTecn3/GQlIJBr+Ia8wK1k3JW2+2rUVpRRXaR999kGmz7D1BRo5Hc3oXNf87cHVBZqqrZXLNW7pmPE7/ixZXv0TEygGA/M3HOnI/20Ovjq7UHWJdWyMwl+7w9lGOS4EMAaoM6o0Eju6iCQ2WVRAT6cEry0ZdcGiSiM0T3Usmlu36FZW+o24dc71kWG9UVLn1fVabs/QPeHAGrZlYnylWUwMJn4dWBGHO3UWgI4+HKG/mjjoqHnOJyxv5nEff8bwN/7MilyqEzN+oWXtCu5wf7KRSZIsBRCQdWYvj6VnwLthPsa2LCgDi6W4IZZFA77JIwVC3nuP7A5u1yV1wUlVcxc3Hd/8Fnr0xj4Y5c/MwGZlw9hBA/1YPl1G7R+JkNZBQeZtvBY79h/rGjOvj4bVvLBB+lFVXuT209Y4LdfzjbZOmlslztYAv8mqHyNFanHkKHGksvtfI+bKXw2dWYK4tY6+jKsu73wyk3A3Ctzx+YqWrTxNPrZ63i9Bf+cC9dtXvr56ivrg389jah2V5t7nyPgapcGlRiNqgqmHJrnQ87UWypke8BuJdd2kPOx6r9akl3d04J5ZVt2DywCST4EIBqNubKAQAY18fSqD1ijsq19LL4JUj7S2XGO99APHQdC7cuUe3RbSXwwzT4cAIsexNeGwQLp6tPXHGDWTDkTQoIqTP4mLlkH1lF5VhCfLnrrG78Nu00vrz7XEZd/S/+br+H/iWv89XoH9TrAbeYvufiwfEE+pqcwYez30SCc3feKNXsjJIs9qRVL7e8v2QfhWU2WPAkvDsGSvPJK6nghfmqzfSD5/akl6vpGmpDvdFdVUXQsWYysovK2ZJZhL9WgS821qcXklPc/E/crjfNqCAfwgKqNyVsVvBhr2xYLoFz1kM3+bMwXc0a5ZVUqCqBupJOdR2+vRNytlBoCOdW2z307dgBek2A4FhC7Yc4z7CcuWszPJvDtZK0/DJW7i+gvNLB560wrT1/cxZLduW13BMWZapAHuD8/6ivacvq3W+pwWrne4DK73IFONlbmvf87Vx1pYtqLeBadskrsXn1Dd/u0FmzX7VLqHLoRySWtzcSfAg3V78PUCW2LcZVcpu7TX3tO0ntOluX6O5w/U9w7nOqd0fqEvj5ISjNgfBkuPQDuOl3+g89DYBle/M92oNbD1cy29mO/emL+nHP2d3p2kEFVSO7RPHQ+J6Axj9+L2Jx4i0AXGj4iym9VG5Ld0swgzQ181Ee46y88QuFIFX1E1mehtmo0d0SRHFFFR8s3AZ/va66Pa6fzfQft1NUXkWfuBCuSel0xI93Tm/n0su2rCPuq2nhjhxCKGGx//38EfBP/PVyft+Wc9THNITrD1KX6CAot9IpwhdoRvBhK4X/DoN3Tj92AOIMPioCYiizVQcLGw8UHpl0Wm6F/10Nm79CN5i4o+pucghnYGIYGM0w9AYA/s/nV/JKKvhzV/N7fuzPK2V7Vv173dScffp6XQb2FuxouSe3hFs/WcMNH65q2F5GDbHxf2prgaQU6D5O/Q5XlcOBlc17Xldn07jBnrefJEsvrpmP3s6Zj7AAM/5mNQNUe4uKtrQ9q4jiiurdx72yb1MjSPAh3AZ3DANUq/aULpEt98Rxg1UrbZd6yl3dDEYYcSvc9peanQiOVcHI1JXQ52LQNLp2CCI+zB9blYO/9lR/Wpy9IpWSiiq6dQhyJ1TWdOPoZCYMiKPKoXPNT5UstvfFpDnosluV/kboVjoacnDoGrtMPaof6Nz6vYuWSa/YEO49R923bfmPYFf5EmWrPuGrteloGjx1Ud86k3XH9OqApqmeKkebpv19ew43mX4kypFHnCOTm4w/tMjSiyv4ODNwP7zYnWtzXgDUp/qmPeFvqrV91qY6O9l6sKrgI8fg2Q9mfbrVc+bj4EZ4+3TY/j0YzGSd/iJLbN0I9DGqoAlU2ajRh37sor+2h89WHn0morzSftTZkYoqO5Nm/MWF/11ab7v5Bdurz392UQVLd7fcLMUvW9Rz26ocLTP7oeuw/lN1fcCVavmw8+nq++bkfVSWV89suJJNXVo7+LCVwid/U32BvKTMVsXePLUNgGvZRdM0YtvB0kvtKrqae8+0RxJ8CLdz+8Ry8aB4npjYp2X7UBgM1RvNdToVYgcc/XiXiGS4+iu4d7sKRkzV/Rw0TXMHF67ciPJKO+8v2Q/Arad3OWK/GNfjnpvUz73E9JZ9grpj7UdQmgcZqlvrbj2O7YdqPN75ybyLIZMBCWGc09tCn7gQRjjWuQ8JKNxBHy2VK4YlMajGLFJNUUG+DHHeV18wYatysHnXXq43znffdovpe7bv2tXsTeBU8KFzcd5bUFVOt9xfCORw02c+tn1X48mPsX+Os9Jlb0UYUL254Ib0wuqk3uytMPNsFdCEJsINP7M4QC2P9Y0PrQ7ogjqoQBSYYvqFX7Zm8+lKZ68WhwPKq//w7sgqZvRzf3Dea4vrna1Ytief/FIbtioH89YeWcmk2uOrP+6ndosCWrbJ2a9bq2fCft/e/BkuMteqQM7kB30uUrclu4KPZuR9ZG9W+VIBkUf283BXvLRS8LHrV9j9Kyx6AexVxz6+FWzPKkbXITrYlw7Bfu7b491Jp96b+XDle7j6LcnMhzhu+PsYefnygUwcePTS1SY540EYPAUuaLlPLWf2VJ+g/9iei67rfLX2AHklFcSF+nHhwLh6HxfgY+Lta4YQFeRDWugwHDEDoeqw6ofg7Gy6ztHNs7zVGXx01TIYkBiGpmlMO7s7pxlUaWGpSQUUV/ou5YFxPTjCjvkw92Yozefs3tVVL3VZtb+AaxzfEKSVo8cORE8YRoBWwVQ+Z3Ezlxd255YwxrAOS+F6AIyOSk41bGpa8FFVATt/rvHkvx39+CL1Zr25VAV+t57eBYBNGVYcYR3B6KPe2KrKVe+XW/6EhCGsdyb5DnT2onE7RS2bTTQtJxIrb369iLR5j8NrA+HZJNgxn/15pVw9cwV5JRXszC5x/4GurWYgOHddhqpGyt2hcnk2fMafO1XicufoQPes189bslpkiSSnqJx16YXu7xfuzG3+JmWuRNNeE9SyIVTPfGSuhcOFdT7smDJqNBer3VvHNfORs611ggPnBwMqS1XjQi+o2dm0Jlfeh7dmPnRdZ+U+le8xZWQnQM18tOfN7iT4EG0jJA4ufK26Z0YLSOkchY9JVY/syC7m3T9VmeaNp3Y+5sxNx8hAFt5/Jj/fczqGU+9RN658xz0lvU7v6lHCaY9Q4+6iZTIwUf0xH2Mpo4vhIJW6kYfK1P4Wf/NZRrhfrT/KZQUw72a1Br/g3+7gY/ne/DrLdVdu2sYU4y8AaGc+jHaO6kp5mXEhm9Yta+DZOZKtykF6fgkPmP6nbnC+KZ1tXE1OcUXjZ1X2LlJlnK43twOrVDl1fZzLLmn2CCwhvpzT24K/2UhJRRV7C8ohaQRoBjjrUbjyfyqJEWdOCDCgdvCRMATih2DSK/kp+CkWme8kacPLquICnYpFLzH5vRUeZcSunZpr0nWdBTXyafbllap9jhZOV/kNX99O+mq1RcDYXhYGJITSOTqQ8koHP206eu5OQ/y2LQddVw2rAn2M5BZXqDe59FXw7lmQtqJRz7cnK5/KDV+ob1zl7KC680Z2VXkgqUubNlhXpUvtJReAsE7gE6yWIfOPvklgkxxYU309vZl5K020tVali4u3l1325ZWSV1KBj9HAJYPi8TEZKKmo4sAh71fg1EeCD3Hc8vcxktJZ5aY8PG8z+/PLCAswc8WwxGM8UgnyNRHgY1KfDiO6qCZMzmS6dY6uHrvn7kXNBiVpOXQOV0ma2h61zLBW78aPjuEUamH42QqO3ENm0fPV5YfrPqYzGXSJDqTSrrOwjmqdpK1v46/ZOBQxALqdDUnDyUs6F6OmM3z3q01OdEzNL+V8ltLTkI7uFwoXzQDgLON6jNjrzXWo17Zv1dd+l6n22rrj6PkEzmWXg3okI7tEYTIa6OfcjHB9uhWu+gLu2Qqn3quW6lBLadudZcmuZRoPzqqpDpUZGDSdZfbevOZ3K7pmxDdzBYHWnSRHBfL8pP4A/LQ564jztyWziIPWcgJ8jJzfXyVC/7JsHWx1/ny6nclpj5KsHeSsnh3QNI1Jg1VlR0ssvbiWXM7vH8uormpJ548dObBihvq0/+N9Dd6wTdd1Zr3/FmablXJ/C3Q+w/MA1/dNzftwJZvGDz7yPoMBLH3U9ZZeerFXeeYUpTcuIGsptStdXOK8vOzimtEbmBhGoK+JHhY1u7j1YPste5bgQxzXXHkfa1LVJ+5rUzoR6NvILYsMRhj1d/e3uk8gu/QEMq3l7mn1NQV+lOq+mDU7hsL96kBnjkNqeAq+Pr7Y+0xSt2/4tPq58/fAqnfV9aju6g36t39zdm9VPePaQ8flQOouLrCpXA+fcx51T22HTniaSoyMZh17ln9HU+zJKuBek/pErI2+B7qNA/9wwihhqLazcUsv9qrqrrW9JrjLlo+69OKsdsnUI90JzQMSa+R9mP2OqILakllElUMnKsjHva7uoe/fYPQ9cNr95F2/nHv8n+KlwtP4xa4+md8csIhP/m84Fw2KJ9jPRG5xBatrLb24llxO7RbF5FNUHkPk9k9Ub5rE4RRHDyaEUmb5vsiQDurf4+JB8WgapO/bSeGvLza5vLSkooqlu/MBVQl1pvP3+fftOdUzHlkb1b5EdbFmwBfXw4cXwsxzKP/vaKZVqA7Cc6tGUVE7x7apeR8Oh+q7k+ec0ahd6eLiTjptQqfTo3U2ztmqNnd08ULwUWl3uDeI7B3rOfPhzvnw0syHa8llWLJa/nWNrz3nfUjwIY5rZ/aormjxMxu4zrne2WgDrnSX02pxg4kOUX9MdmarvI8NGVb26M48krwdUGVz/wG/5NLrWPnwWCJHTVH3b/+xevnht8dU586uZ8Pln6hlhR0/cHGUqs74fuNBPl623z2Mkl+fw1erZKtPPwJ7nOW+3RzdlaXhFwEQtuTfqqxV17Hbyvlh1Xae+3Ezr/62i/cW72X2ilS+3ZB5xJJOwMaPSDTkYjVFqnwJo0kFIMBY45rGBR9pf8HhAtUFtuMotXkfqICsrk/p5VZ3p00186GCj/4JYUD10kptG5y5EAMSwo7cOwjUzzD2cRjzL6I69uK9KUPxMxv4uEqN5xLjYuIDHPiYDO7lrp82ewZ8ruBjbC8LIzpHkhxqZJL+q7ozZSrvJzzJAT2KThzENPd6sFcSV5nGh+GzWOR7D2FLn1RlwU3YOfbPnbnY7A6SowLp2iHI/fucc2CPO0cGgCWvHPlgXYfv74Ytc2HfIkhfgX/+ZiK0Eip0E++WjOLDv/Z7Pib5VEBTv8NFDdwXJ3+P6rfzwzRAh14XQlB03cc2dY+Xbd/Dsx3hpwfrvt+1y3T8UPV/qDANio5cQiuvtHP1eyuY9r/1R+8inLuj0QHjntwSbFUOgnxN7v44LrGh1csubbZVQQ2umY9hzr24XGXA7bniRYIPcVxLigygc7TqlHrFsCQimrrDqckXTn9AXe81ge7OaUvX0sv6dCu7dWcibt5OSF+uEt8CO2CK669mW2L6Q4feas17y9eQukxVg2gGOOdJVU466BoAemx4nhucgdIj32zhrUV7oDCNrgfUPhy7+/z9iIS+wyn3UqQH0KFsNzwdA/8Ow/iMhfN/GM7kFRN597f1PPXDNh6et5m/f7qO2z9ZW/2HsKKYQfvfA2Bd51vAx/nH09n+/mzDGtLySxt+vlxVLj3PUwFA0kgw+UPxQfUptTZnvkehHkhURAQJ4er1XUmkWw8WUVF1ZM5Jvfke9egbH8q71w4lvM9YbCEdMdiKYfNXAJzfT82q/LT5oDsR76D1MJszitA0NYtmMGg8kLiFSK2YfGM09Difb3ZXcZPtXqqMAWq54q3R8MZwTiv7FbNmx45BtYWvvdzWAK6Zr7N7W9A0jZhQP3rHhjBEczZbC+uomvLtX6xyQGra/j3s+kXdP+E1uOxj/h3yb66w/Yv7LO+zT4/l9QW7ySupMaPgH646kwLs+/Pog3PYVUfiGaNUvx1zgCp5v/SD+h/jqng5uEFVjzXE3oXw5fUq6Xv97Lr393Et93Q+Azo4l3bq6FfyxZoDLNmdx9x1GaxOrSf/aMvX6md6dwwUNzxnxzVD1Tsu5IhKOteyS4XNRsUvT6gPIG0ky1pOWkEZBg2GxPvDgicY6rMfkJkPIVrVv87vxYQBcdw5pmvznmjYjXDvThh2kzv42JFdzGGbnZ3ZxexxuGY+dqmyP1Cf+J35CWgaDLhCXV8/B355WF0ffC106KWun/GQ+iN+YCWPdN3DHWeqMS+Y/zV5b12AiSoW2/vSc8T4I4Y3sl93XrZfpr6x2zzuS9DyeLzrbi4aGMc5vS34mAws2Z3H987delk+g2B7IXsdMdj6XVX9wC5jsGtmOhmyqcza3rDz5HBUBx+9LlRfzX7OT9XUvfTiXHKpOesBkBDuT0SgD5V2vc6W8xsOqDXrOvM96nFqt2henzwUn+E3qhtWvw/A6G5RBPuayC6qYE2aemP6zZloOiQpnMggX9B1xljVpoXvVZzFilQre3NL2aV1wjbxLUCD3O2ATlX387jc8TQfVp2jXse1vNZAlXaHu6zW1XwOVBXXEIOz2VqP8dD/cnW9Zn8LW2n1LMGou2DIFHITxzErpxvLHb15ePI4+sWHUlxRxX9+qdWyvqF5H/Mfgp//qYKC5NNU350Rt1a3VK9Lh96qaulwAfynp5oR2jG//uqXjDVqN2vX73NFkerCWtsBZ6VLwlBIdHYerpV0anfo7qRzgFlL69j+YO1HKtBxVVW5qoKOweHQmb08FYALBxxZSednNhIZ6MN4w0r8lr0EX/2fSjRvAyudsx6940II3voZLP4Pvf6cihm1lcKhUtsxnsE7JPgQx70xPS28fuUg9ebRXMEWMBjoblG18ruyS9iSacXu0Mn1c/Y1yNtZ3dPClevg0u8yNdNxYKX6w+oTBGf8s/r+kFhImQqAtuAJ7hsdxQ/JX/KF7xNElaeSq4fytv//0c1Zq19TqL+ZXR2vZET564wsf41TbG8xfeCvHB6t3oT+ZlzMK1cM4p1rhzL1DBXUPPn9VoqLrejOPXVeqrqULjE1epD4BmONSQGgU75nHkCl3cFT32/lwa828vGy/axJPUSZrUr9XMUHVWVDzYRG57mwbf+F9xbv9Vz/tqplpowa+R6g+q549PuoobDMxj5nQ6cBzuWZRhk4Wb0RZq6DzHX4mozupZcfnEHZb85y57GuN//0FfjmbqYCHz6tOpMHvlK5C8M7RxDQf6LagXn4bXD7CkxXfUpC31P52H42APrOn9WmiQ20cl8BReVVRAb6ePSFGdOzA0OdwYc9/hQVXKDBjh8gxxkgLnpeLcuEJqkEXar73fRPCCUm1I9HJ/QG4LNV6e6unIBn3kd9SwQ526uDqfNehGu/VX13jsXsr85R3CD1Br/tO/j0cni5N/z8sJoRcb1mznbVNMxWosbU92/q9prl26CW7PKcwVj8ULX9AhwRfPy0+SBpBWUE+qjg6Oct2Z7b3C99VbXr1x3Ve9Gs+6RBybx/7clnb14pQb4mLhpUdyuC2DA/Jhr/Ut9UlqrS/eY4tF8teR3Dqn01llx2qnwxQ9EBbg1RY9nWTpdeJPgQog6umY+d2cWsd74p+sY4Zy+yNjv7DGjQ+UzPB4bEet426m4V0NQ08u+qSVP+LnilL30Oqk/ac6rO5KyKF+jUe2jd+Q3AdaOSOWSKpl+fPvxv2oU8dNEp+A+7Vo0ldan7ze+W0zvTKTKAnOIKFn45A628kDRHNL9qI+hYa71a76GWXoaUL/NYr/5+YybvLdnHZ6vSeeSbLUya8Rd9H/uZLz5503mSxqnlKhdn8KGlL+elH9ZxwetLWL5XTVWX56sGYAdrBR9QHVhsqJX38aez02fHyADCm7KcFhgFvSeq66tnAXBejaWX4vJKlu1R43PtOOx6w0hPuIBCgkl1dn49q6fz/n5/g/HPQoeeAFw1PIl9eix/2vuhoVO14r0GD8+15DK2l8WjG+7ADiZ6GdSn7I2GXqq7bq8L1J1LX1Vv2sv+q74f/5x7Cc3Vft+VhD2sUwQX9I9F1+GJ77ZW/9smjVAbOBZnqp4cdfntMfUm3fMCOOWmI3t6HE3viXDzQudMyVT1u16Srcb89mnwxnC1SeTHF6sZkvgh2C+fzUr/0erxO+d7Pl/GWkBXS1BB0ZB4irr94Hr3PjW6rqulS+Cm0zozskskdofOx8tSVXDx2+Pw66PqcaPuhhvmq115C/bUPdNSy8fL9wMwaXA8QfUktHcNquR0w/rqG1a8BRVN23HZsfYT7K8Nwf7fU9CPsTzmyvdISfCDfYvdt1/v+AofKuvO+1g1070U6i0SfAhRh27O4COnuIJFO1U5bGxyL9CM7nbqxA+GwDra0A90LmsEx7lnOTz4hcDp/1DXK8sgshtc9yMBk96gZ3LSUZNmx/a2sOOp8bx9zVCSo5y7AofGVzeQ2qB6ePiZjfx7Yl9AJ3mfqr75xD6WpMjgIzYMDBmgurwOYDf5WWqGQtd1Zi3dD8Bp3aM5s0c00cG+OHSdUw4vAeDLskHYqqqTLJcXhpKOBTNVjDZupaDUxtXvreCjZfvJz1TT4RWBcR6dIaE676PmzEdpRRXP/qjeGCf0r79h3DE594Bh05dQbuXU7tVLL6/8tgub3UGnyAC6RAeqP8ZbvwHAcvbf8TVVnyd3cFLLkI7hvHrFQGbraunl8IoPsFrr+GNvrwJbmTspVdd1d5O5c/p4Prcxcw1GdA7oUfyc7hzDKGcvmk2fw7xbVBJz93PdOTsVVXZ3Azp3oAQ8OL4nviYDK/YVVCfamv2rf1++v/vIHIt9i1UAYDDB2H/X+XM3iKUPnPsMTNsOl89WQYnRVyW7Lpyugp/onjD5S15fcpAbFgdh042Qv5tfFy+t/t2queQCao+nwGi1VHNwA6DyMTZnFOFvNjIlpZP7/9Bnq9KwLX2jeslq7L/h7H+DbzD0VR1yWfvxUX+MzMLD7n+rq0d0rPe4MfoKfDQ7Of5dVD+V8kJ30FtTeaWdWz9ew7T/rcd6uNa5d9jRf/4Xhm+nYtSrMOpV2D+dfORuz06FZTZ3Bc5ww1b1tykkAYLjiKjK5TLjwiPzPjbPVcnD75zeZktDdZHgQ4g6BPma3OVzS5x7ePTr2AHCO1UfVHvJxaXvJLjwdbhmbnViZ21Db1DT6WMfVzv5dhrFRYPi+fyWFPdGeI3iaia14VP3NPLp3aOZ2rWAvob9lOtmPref4W69XJM5PIFtWlcMmk7Jpu8BWJt2iI0HrPiYDLx82QBmXX8Kqx4ey9qbYuloyKFcN/Po1lj+9tZf7M8r5fuNmVz7/ioWVqlqh5eH5jNxoNpD59FvtnAwTW3WFxbT6YjXdy277MktdVfovLpgF5nWchLC/Zl6ZjNyeZJS1BtcZSls/Bxfk9G9xOLKCRjbSyV7svp9VV7bcTTBHQdxTh9V/dTdEkRSZD3/jsDEgfFMmXILGXo0wXoxb8940XO6P225WnZ4JhaeCIcno3FMT+KTw7cz0Jzu7u3h5iwjXe3ozkLnUgoJQ1TehcPZ78Lkp2Y9nFbuK6DUZqdDsK9HA6yE8ABuPq0zAA9+tbF647zxz4FvqHqtXx+rfm2HA359RF0fcn31js7NYfJRMzeXfQT374KJb6rZwaQUuGYemwqM/Pf33ZQQwEpdzS4unz+Hkc/+zht/7EavWekCahbGvfSiztXbf6pZj8uHJRIe6MNZvSwkRvhTWnaYqsWvqmPPfhJG3109rkHXqq9bv/Zox1/bpyvTcOgwonOE+0NJXYYUqVyn5YFjVPk3qNmeWrsIf7E6nflbspi7LoOL31zKXudO05QXoX96Jdqy1wF4veoiVju6Y7IVwey/QcmRbfdXO3ex7RwdSGi6M+G5+zg4dRoAU03fsCuzRuJv9hb4xvmBaMCV7kZ+3iDBhxD1cOV9uGar+8eHVe++CvUHH5rmmWRaF6MZzn5C/ZEy+9V/XEP1mqCmkQ/t8+iBcEeQ2lL9W/tICgmuM/gA2Bw0EgDfPWq93TXrMXFAnEcuTcR+lcVvjT8dn4BgNh6wct5ri7ljzjpsdgdFCeoTdWDq77xy2QAePq8XgVo5MQ71yTGx05EdbiODfEkIV4He5gNWtmcVMXOJCgyemNgHf5+jJDgei6ZVz36smgkOh3vpxdVrbGxvi5oeX+P8lDpcNS677fQudI4K5LYzuhzzZUZ2s2AeoRJcx5V9x8VvLFU7nO5dpJYXSmq00rfbMNqK6GzI4tGQH/Az1/r50pYDsNbRg+1ZxdWBzOhp1cecep9HILygxpJL7UqMqWd2ZUjHcIrKq7h25kq1iWBEZ7jIuXy2/A1VAQKqbDdzncrncc3OtSS/UBg0Ga79Gm6YT0WAhXu/WE+VQ+e8fjEMPEslbJ/rs4G8kgpe+Hk7FanO4MM18wHVSy/pK9icYWXxrjyMBo0bR6u8FKNBY0pKJ8YbVhBQkYMeZIHht3qOJfEU9f+5ssxdEVWbrcrBp85NC6+tY5dqt6KDxBWqipyfGKVyv0IS1L/7+tnuwyrtDt5apGYBfU0G9uaWMvGNpSxftwH9/XFou36mXDdzZ+Wd5A67n5ts00glRpUWz7lczZ7V4FpyOaVjOOxUXZHpPg4GX4s9KI5YrYCh+d9RXmlXsxyfXaV+3s5nwFmP4U0SfAhRj+41PuUkRwUSGmB2726LX1jdLaa9xSewOr/B1eSsJAf/naoq5UO7WhaoL/jIjFF5KtG5yziYl++eor9+VI0kw3IrrFI5DZaRV/Hj30/llE4RlDnbsl83shO3TrlOlX4WpqKtnslN2U+wMWAqCZr69NWrV/86X99VSrsuvZCH523G7tA5t08MY3rWvdzRKP0vV2+mudtg0+ec2i3KvW4f6m9maMdwWPoalOWrTrc9zgdU9cDv953BxYMSGvQyHU67Cd3owwDDXmJLtvDnj7Nh9qXqj32XMXD/Xrh/D/a7NnGX+XEABpYtdXd+BVR5q3OZodSi3mx/d+070/kMFdT2vMCzKZ6uu3fcrWsnZz+zkfenDKNnTDA5xRVc8/4KcorL1WzESPU8trm3c/XT71PwrbNCa/Rd9ffyOIr8kgoe/3YLD83d1KAqi5d/3cXO7BKignx4cmJfgvqpcz9U28bfR3UgQcvFz1aAbjCrUnaXGkmnby1Us2oT+seSWCOf6dIhCdxk/kkd1uUqj40pARWYOkvfWVf30sv8LVnklVTQIdjXnaxcpy1z0dBZ5ejOxpJQ9Vquf6Olr7irfeatyyCj8DBRQb4suPd0hnQMx1ZeRtC8a9FytpKth3FF5aOMvfQ2Hp/Qh4AwC1Mq7qfCJ0y1tp97k/odcXLlVI2NylcJyCY/tXmnyRfDaSoR+RbjN+zJyFEVOIf2Q1gS5RPfQz9a1VIbkOBDiHrUnGId4Cr17DhKfe098eglh94w0Ln0snkeVB6GNR+CoxJH/DC0uAH4GA0Mrme3XXNcPw7oUZgcFaz/cSZ2h87w5Ah3syIAls9Q69hRPaD3RcSF+TPnpuE8eVFfXr58AI9N6I3RP0QlNIJqC75lHkb7YexhnSg+7TFC6lh2ARjoTDp958+9rEk9RICP0V2t0Wz+YeD8Q8xvj+OnlzO2l3qTPrNHNKbS7OoEzrP/rfqWNEVgFFpf1eX2KfP7XLzjAbUG3+M8uPIzlR8UGMWfOX58U9ydVfTBoNs98wKyt4CtGHxD6DVAvcHO+mu/agmvaWo574rZHom+u3NKSC84jI/JcOQSjlNogJmPbjiFpIgAUvPLuHbmSgpKbXweej3rtN742Et52/YgEZVZVAbGqETRRrA7dGavSGXMfxbxwV/7+XRlGue++idLd9ff62NNagHvOJdLnr64n5phi+gMUd3RHFXckpDKcLOaJSgJ7+U5Qxg7UAW5pTls3Kwqkm453XOGKjRvHf20vVToZl46NLruQQy4UuW2ZKxROyrX8skylfh75SlJR98vatOXgJphzCoqp8ruUIFNQJSatdj8FXaHzoyF6ue9+bRkEsIDmHPTcN6P+4a+hv0U6EFMsv2b6y+bxMSB8RgMGpcOTWC/HsuzoY+pfJnt38OfLwDw1+48NhywYjRonFLlzItJPs291KsNvoY8YzQx2iHiv7pQdck1+XPowg+47OMd/Pf33fX/PG1Agg8h6tGjZvDhanLVfZzabbXGenu70XG02oa+wqo6RjqXEQyn3MQXt4xk8T/O9PhkWFNSZCCfVKllpBF7XyOcIs9Zj8OHVMMpUDsUOwMvk9HANSM6cvGghOoKnUFXq6+hSeqT9U1/YLxrPcFjaiwb1OLK+3Al4E07u7u7cVOLGH6bqpYoPghLX+WBc3syeXgS943rAQufUbMTicPVrEJzDLsJgH6G/arPQvy5KtehRrAwZ4Wq/NnX2ZmYvOaD6rwA15JZwlAuH96JUH8ze3NVTk19Fjh7haR0jjzq1gIdQvz45MbhRAf7sj2rmJTpC3hg3jZuPjyVfMII1FQi9ZchU+rPVarDxgOFXPLmUh6etxnr4Up6xYbQOTqQ7KIKrp65guk/bvNITAYos1Vx7+cbcOhwyeB4xjnzawD1fwwI3P8bf7OokuiVtlplvmY/iB0AwCBtJ2f0iKZXrZbnLFfLSvPso/hmVwWbDtSxz0lQtErcBVV2W8P2rCJW7i/AaNC4anhS/Scgfw9krkXXjPysj8Du0MkprlDnMOV2dcySl/hhYwb78koJCzAzebhKXPXd8S2jClRjwRcC7uXBK8/x2FXctYfQBwcs5I95wflcL2M/lMZTP6iE7KuHJxGSppZX6XZO9bhMvqxMuB6AsGJVqpx/1n+4ZG4xGw9YmfXXfgq82ANEgg8h6tG1Q5C7wtCjw2bsAFUx0N4YDNUNqeY/qBp7BURBn4vw9zFiCak/tyQpIoD37Oex3ZFIOMU8HfA/z2nmv/6rGkBZ+kLvi44+jgFXwD/2w90bVWfX+Dq2X6+lb3worlSFXrEhTW+TXx+znxoLwNJXiSOPpy/uR0JlavWbztlPNq6ktC4JQ9yJkV/aT+N+/e8qv8cpy1rubiw2+JzJEBIPZXkq6RHc+R4kjiDYz8z/OXMYXv99d73bo7tKbF2zOUeTFBnAxzeeQoifiYoqB2EBZm45fyTBV3+Ew+DDBkdnHkvt36A9SnbnlHD3Z+uY+MZSNhywEuxr4rEJvfnujlH8cOepXDU8CV2Ht//cy0VvLOXxb7dw/xcbuH32Gi59axn788uICfHjsQl9PJ/YFQzs+oVBBvXp/Lv8+OrETKfMYLUMM8Sw68gGg4Vp7o0PNyVcha7DhP8u4eyXFvH8/O2sSztEdlE5i3bm8pOP6tNiXfExV7y5iGvfX8ltn6zh3s9VJc24Ppaj/t9x5Ytonc/AHKL+zxy0Os/fsP8D3xDI3Y7/D3cSTSE3jEpWQWL+HvjmTnXc6HuY/o9pXFCrsisxIoBRXSPRdfi4bIT6gFFVzoEvHmTrwSKC/UzcPbpDddBaM/gAKvpdyQFdzYbl97+Zc3+3sC+vlPgwf768NaXpHaFbgAQfQtTD38fI1DO6ctHAOPrHN7zDple5OqyWOae7h0zx7MVRj6SIAKow8c/KG3HoGuc5/sC439l0rDRf9SwA1aHV0IA/G/7hjXojD/Q1MaJzJD4mA09f3PeIcuAW0etCtWxWVQ4LnCWkvzr7WfSaAEnDW+Z1Lv+EvIkf84+qm1m6t9DjTfPz1enYHTqndIqga0x4dTKsqyGVK/hwjmXKqE6E+JnYnVNyxJ40oEotV6eqpMMz68j3qEvPmBDm3j6Spy7qy58PnMn/ndoZn66nY7hnMy/FvYTNofFB7T1hatidU8xdn63j7JcX8fX6THRdbbS34L7TuX5UMiajAX8fI89c3I93rhlCeICZrQeL+OCv/Xyx5gA/bspiS6Zqaf/c3/oT6m/2fIHE4SoxtSwfv+x1AKzXu3iMqcxWxYy96k11bPB+hnSsVbWx8l3175p8OrdeNoFTu0VhMmjsyinhzYV7uPjNvxj+zAKmvL+SO1ZGkKWHE6oX0TPjSxbvzOanzVnuHWyvGdGp/pOp67Dxc3W936XuCrmMQudMll8onKnyaM6u/J0/fO/lJuP3qrrmi+vUEltSCpz5r3pf4tIhapfuL9dm4DjnaXQ0Omb+wEBtN3eO6Ur4wcWqSiu6J4R7lgL3SojiOtsDPOS4lbM2nElucQU9Y4KZe/tIOkfXnf/VVpq4uCnEyeG+cT28PYTGieoGCcPURlyaQZVLNkCov5lgPxNry7vzmX42V2m/wPf3qEZRS19RXShjB0LP81tt6O9NGUpxedXRP2U2h6bBuGfgnTNg0xeqFfiun9Wa/1mPt9zrhMQSNehCTt+wit+35/DpyjQePr83dofOZyvVksuVw9UbCoOnwKLnVDLh1m9U0qBmdM+ehPiZuWF0Mq/8tovXFuxifN8Yj2qWD/9KxaFDz5hg9345DdG1Q/CRJd3BFqacAYs+WM2nK9K4c0xXgv2qA4OKKjv/mreZL9cecFeAnd3bwl1ndaNvPcH5OX1iGJAYxpwVaVTaHQT6mgj2MxHoY6JfQqhHUreb0awqyZwzCpU+YewvjyF79QHuPbsHoQFm/vPLTn4u6siTfhBbvkdVK/k6n6uiBNZ+qK6PuJ3EiAA+vnE41sOVLNyRwy9bs1m0I5cyWxXJUYH0jAkhvfxiYtLf53HzR9wfsoCdCRezOXoCoZYk1RSvqkIt2RVnq8A6LEnNpmVtVM0CTX7Q83xit6ucjpozR/rwW/jHMgOTD81ggGEvLHwc/npR/Z8KiIRJM4+aZ3Ru3xiCvzFx4NBhlh9OIKTDBfTN+Y4n/WfTPWUqfPe8OrDWrAdAl+gg0gxJ7LYlADqnJEfw7rVDjwz4vECCDyFONEOuU8FH74kQltigh2iaRlJEAFsyi9jdfxrs36g2S5v/oLtxGWc+3PxliaMI8DER4NPKf5LiBqq26+s/qZ79aKl+FrVcdUoSv2/P4cs1B7j3nB78tSePTGs5YQFmxvdV5b4ERUOfS2DjZ/DDfeq2mL7gW/2p9PpRycxcvI8d2cX8sjWLc52P/e/vu3j5N7WWf9SchEY4o3sHunYIYndOCf9blc7/nap6hFTaHdw5Zx2/OJttjetj4e9ndaNP3LFnBC0hftxzdvdjHueh+7nu4MOUNIyeeSFszyrmf6vTGNopgveX7kMnnMOB8fiXZqg9aLqPV+WzW+apyqyILh5vyKH+ZiYOjGfiwHiq7A7suo6vyZk0XtkbFvjB+tkEHs5g0K7/Mmj3DFUu/0sOlB7ZY4PgOBW4gspT8Qtx5ykddAYflXYH89Zm8Hl2PN+bn2blubkELX6q+vkufkc1CTwKP7ORCwfEMXtFGjMW7SH14Hh+MvxCP8cO2DYPdjv3maoj+DAbDfRLCGVN6iHO7RPDK1cMPLK020tk2UWIE83AyXDdj3Dhfxv1sOtGdmJox3BuHDsQznMmt635QG0sFj8Uup3d4kP1irMeUT1RQO290xr9LIAzekQTG+rHobJKft6S5U40nTQ4wfMNwNlXxP2GlDjC43lC/c1cP6oTAK8uULkfL/y8nRd/UYHHtLO7c81ROm82hsGgcdOpKs/k/SX7qLQ7cDh07v9iA79szcbHZOCjG07h7WuGNijwaLKuY9XMHaAlDOUGZ/Lzh3+l8o8vN6LrcMmgePy7j1HHr/0IPrsSXuiiNsQDGHFbvUuEJqOhOvAANYtx7nS4dwdc/LZaCtHtkL25+t/F6KuSqH2cgWFxJljVvyn91XKnK/hYsa+AOz9dx+Anf3XvD3Tl8E4EjZgCd65RPX7+Ngu61dMrqJZLh6oPEYt35ZFWGca3Qc4NJr+/R5WI+4ZWV5nV8p9LB/DalYN4Y/LgdhN4gMx8CHHi0TToNKrRD7t0aKL7jxyhE1SJ6A7n1uBjWnfWo00Fx6gAZP6DcOY/m9TPoiFMRgNXDEvi5d928sYfu9mdo3I/rjyl1ixF/BB1cW0bX0fuyQ2jk3l/6X62HSzimvdXuLd3/+d5Pbn5tGM3QWuMiQPjeeHnnWRay/lh40FW7Cvg6/WZmAwaMyYP5rTurXO+PAREqABk1y/Q5SwujInj2fnb3Q3XooJ8eOSC3mB+TgUK6SvURnO529SGdgGR1V1/G8Psr/KmBlwBuTvV7F9IrGoYFhCh/g/oumrYdWi/aupnNKvdh4G4ULVkuD2r2N32PDLQh3P7xlTP/viFODcLbLgBCaF0twSxM1v9DvWa9E/4+nf3btF0OdMjsbmmTlGBdHJtxdCOaLregC392lBRURGhoaFYrVZCQkKO/QAhROuwHoBZ56mliks/PHGCD5fS/Lr35mlBWdZyRj33u+rTAQxPjuB/t6QceeCGz9SeLQD3bK1zKv6Fn7fzxh/Vu5w+MbHP0btuNsPrC3bxn1934m82crjSjkGDV68YxIQ6tpNvNYcL1e9gjNqB9qVfdvCaszfFG1cN5vz+sXU85pDq0BreSfUMaWNF5ZVc9tYy7A6dsb0tjO3VgYGJ4R4bBzbVx8v288g3W7h4UDwvXz5QLYfOc86aTXxTdY71ssa8f0vwIYQQreimj1a7NyZ79YqBHn0c3KoqVOvrwA5w8Yw6n+dQqY3Tnv+DElsVz17Sj8uHtUyeR32vlfLsAsorVX+O5yf157JhDcsfai35JRVcM3Mlg5LCeOqivvXu/Hyi0nWdtWmF9E8IVQ3PHA6Ycxnkble9h7y4T4uLBB9CCNFOLNqZy5T3VxIR6MOyh8Z45ho00u6cEg7b7PRLaP3S7+fnb+etRXt49ILeXFez4ZxoP1xv3+0kEGvM+3ejE07//PNPJkyYQFxcHJqm8fXXX3vcr+s6jz76KLGxsfj7+zN27Fh27drV2JcRQogTwmndonjtykF8eP0pzQo8QDW+a4vAA+D+cT3Y8Ng5Eni0Z5rWbgKPxmp08FFaWsqAAQN444036rz/+eef57XXXuOtt95ixYoVBAYGMm7cOMrLy+s8XgghTmSapnHhgLg2CxpaiqZpHn0+hGhJja52GT9+POPHj6/zPl3XeeWVV/jXv/7FxIlqh82PPvoIi8XC119/zRVXXNG80QohhBDiuNeifT727dtHVlYWY8dW1y6HhoYyfPhwli1bVudjKioqKCoq8rgIIYQQ4sTVosFHVpbae8BisXjcbrFY3PfVNn36dEJDQ92XxETvZlQLIYQQonV5vcPpQw89hNVqdV/S09O9PSQhhBBCtKIWDT5iYmIAyM7O9rg9OzvbfV9tvr6+hISEeFyEEEIIceJq0eAjOTmZmJgYFixY4L6tqKiIFStWkJJSR1c/IYQQQpx0Gl3tUlJSwu7du93f79u3j/Xr1xMREUFSUhJ33303Tz31FN26dSM5OZlHHnmEuLg4LrroopYctxBCCCGOU40OPlavXs2ZZ57p/n7atGkATJkyhQ8++IAHHniA0tJSbr75ZgoLCxk9ejTz58/Hz8+v5UYthBBCiOOWtFcXQgghRLO1ant1IYQQQojmkOBDCCGEEG1Kgg8hhBBCtCkJPoQQQgjRphpd7dLaXPmvsseLEEIIcfxwvW83pI6l3QUfxcXFALLHixBCCHEcKi4uJjQ09KjHtLtSW4fDQWZmJsHBwWia1qLPXVRURGJiIunp6VLG28rkXLcdOddtR85125Fz3XZa6lzruk5xcTFxcXEYDEfP6mh3Mx8Gg4GEhIRWfQ3ZQ6btyLluO3Ku246c67Yj57rttMS5PtaMh4sknAohhBCiTUnwIYQQQog2dVIFH76+vjz22GP4+vp6eygnPDnXbUfOdduRc9125Fy3HW+c63aXcCqEEEKIE9tJNfMhhBBCCO+T4EMIIYQQbUqCDyGEEEK0KQk+hBBCCNGmTprg44033qBTp074+fkxfPhwVq5c6e0hHfemT5/OsGHDCA4OpkOHDlx00UXs2LHD45jy8nKmTp1KZGQkQUFBTJo0iezsbC+N+MTx7LPPomkad999t/s2OdctJyMjg6uvvprIyEj8/f3p168fq1evdt+v6zqPPvoosbGx+Pv7M3bsWHbt2uXFER+f7HY7jzzyCMnJyfj7+9OlSxeefPJJj71B5Fw33Z9//smECROIi4tD0zS+/vprj/sbcm4LCgqYPHkyISEhhIWFceONN1JSUtL8wekngc8++0z38fHR33//fX3Lli36TTfdpIeFhenZ2dneHtpxbdy4cfqsWbP0zZs36+vXr9fPO+88PSkpSS8pKXEfc+utt+qJiYn6ggUL9NWrV+sjRozQR44c6cVRH/9Wrlypd+rUSe/fv79+1113uW+Xc90yCgoK9I4dO+rXXXedvmLFCn3v3r36zz//rO/evdt9zLPPPquHhobqX3/9tb5hwwb9wgsv1JOTk/XDhw97ceTHn6efflqPjIzUv//+e33fvn36F198oQcFBemvvvqq+xg51033448/6g8//LA+d+5cHdDnzZvncX9Dzu25556rDxgwQF++fLm+ePFivWvXrvqVV17Z7LGdFMHHKaecok+dOtX9vd1u1+Pi4vTp06d7cVQnnpycHB3QFy1apOu6rhcWFupms1n/4osv3Mds27ZNB/Rly5Z5a5jHteLiYr1bt276r7/+qp9++unu4EPOdcv5xz/+oY8ePbre+x0Ohx4TE6O/8MIL7tsKCwt1X19f/dNPP22LIZ4wzj//fP2GG27wuO2SSy7RJ0+erOu6nOuWVDv4aMi53bp1qw7oq1atch/z008/6Zqm6RkZGc0azwm/7GKz2VizZg1jx45132YwGBg7dizLli3z4shOPFarFYCIiAgA1qxZQ2Vlpce579mzJ0lJSXLum2jq1Kmcf/75HucU5Fy3pG+//ZahQ4dy6aWX0qFDBwYNGsS7777rvn/fvn1kZWV5nOvQ0FCGDx8u57qRRo4cyYIFC9i5cycAGzZsYMmSJYwfPx6Qc92aGnJuly1bRlhYGEOHDnUfM3bsWAwGAytWrGjW67e7jeVaWl5eHna7HYvF4nG7xWJh+/btXhrVicfhcHD33XczatQo+vbtC0BWVhY+Pj6EhYV5HGuxWMjKyvLCKI9vn332GWvXrmXVqlVH3CfnuuXs3buXGTNmMG3aNP75z3+yatUq/v73v+Pj48OUKVPc57OuvylyrhvnwQcfpKioiJ49e2I0GrHb7Tz99NNMnjwZQM51K2rIuc3KyqJDhw4e95tMJiIiIpp9/k/44EO0jalTp7J582aWLFni7aGckNLT07nrrrv49ddf8fPz8/ZwTmgOh4OhQ4fyzDPPADBo0CA2b97MW2+9xZQpU7w8uhPL559/zuzZs5kzZw59+vRh/fr13H333cTFxcm5PsGd8MsuUVFRGI3GI7L+s7OziYmJ8dKoTix33HEH33//PX/88QcJCQnu22NiYrDZbBQWFnocL+e+8dasWUNOTg6DBw/GZDJhMplYtGgRr732GiaTCYvFIue6hcTGxtK7d2+P23r16kVaWhqA+3zK35Tmu//++3nwwQe54oor6NevH9dccw333HMP06dPB+Rct6aGnNuYmBhycnI87q+qqqKgoKDZ5/+EDz58fHwYMmQICxYscN/mcDhYsGABKSkpXhzZ8U/Xde644w7mzZvH77//TnJyssf9Q4YMwWw2e5z7HTt2kJaWJue+kc466yw2bdrE+vXr3ZehQ4cyefJk93U51y1j1KhRR5SM79y5k44dOwKQnJxMTEyMx7kuKipixYoVcq4bqaysDIPB823IaDTicDgAOdetqSHnNiUlhcLCQtasWeM+5vfff8fhcDB8+PDmDaBZ6arHic8++0z39fXVP/jgA33r1q36zTffrIeFhelZWVneHtpx7bbbbtNDQ0P1hQsX6gcPHnRfysrK3MfceuutelJSkv7777/rq1ev1lNSUvSUlBQvjvrEUbPaRdflXLeUlStX6iaTSX/66af1Xbt26bNnz9YDAgL0Tz75xH3Ms88+q4eFhenffPONvnHjRn3ixIlS/tkEU6ZM0ePj492ltnPnztWjoqL0Bx54wH2MnOumKy4u1tetW6evW7dOB/SXXnpJX7dunZ6amqrresPO7bnnnqsPGjRIX7Fihb5kyRK9W7duUmrbGK+//rqelJSk+/j46Keccoq+fPlybw/puAfUeZk1a5b7mMOHD+u33367Hh4ergcEBOgXX3yxfvDgQe8N+gRSO/iQc91yvvvuO71v3766r6+v3rNnT/2dd97xuN/hcOiPPPKIbrFYdF9fX/2ss87Sd+zY4aXRHr+Kior0u+66S09KStL9/Pz0zp076w8//LBeUVHhPkbOddP98ccfdf6NnjJliq7rDTu3+fn5+pVXXqkHBQXpISEh+vXXX68XFxc3e2yartdoJSeEEEII0cpO+JwPIYQQQrQvEnwIIYQQok1J8CGEEEKINiXBhxBCCCHalAQfQgghhGhTEnwIIYQQok1J8CGEEEKINiXBhxBCCCHalAQfQgghhGhTEnwIIYQQok1J8CGEEEKINiXBhxBCCCHa1P8D2gYsL1dDiNEAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "上記のグラフの縦軸は前述の「無作為に単語を選び出したとするときの『母数』」で、横軸は `epoch` 数です。学習が繰り返されるごとに「母数」は減っていき、最終的に15語程度になっていることがわかります。これはすごい進化です！もはや無作為な単語の羅列では無いはずです。さっそく文章を生成してみましょう。"
      ],
      "metadata": {
        "id": "utcBZIaR7-qA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(generate(model))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KEVrFHxJp_2d",
        "outputId": "ff845009-57bc-4ff4-c685-2db4cc32ac46"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pdN.apine t ul. Md my itie, Mrd, Tgooncaad  he weny t syurthes a BW. \"ucofe, mounl, ot b cpan stohiva\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "学習前は出鱈目な羅列だったのが、学習によって「薄目で見れば英語かもしれない」程度の文章を生成するようになりました。現実に存在する単語もいくつかあります。当初の目標だった、「それっぽく」動く言語モデルの完成です。"
      ],
      "metadata": {
        "id": "F8lHDleV9Mvy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 参考文献"
      ],
      "metadata": {
        "id": "_7VywL9-_sog"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "理論は以下を参考にしました:\n",
        "\n",
        "- 岡﨑 直観, 荒瀬 由紀, 鈴木 潤, 鶴岡 慶雅, 宮尾 祐介.『IT Text 自然言語処理の基礎』(オーム社, 2022)\n",
        "\n",
        "実装は以下を参考にしました:\n",
        "\n",
        "- [Llama from scratch (or how to implement a paper without crying) | Brian Kitano](https://blog.briankitano.com/llama-from-scratch/)\n",
        "- [CrossEntropyLoss — PyTorch 2.3 documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)"
      ],
      "metadata": {
        "id": "0CuwmNmo_uEV"
      }
    }
  ]
}