word2vec with MXNet Gluon

word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WordEmbedding 학습 "
   ]
  },
  {
   "cell_type": "markdown",  
   "metadata": {},
   "source": [
    "- 목적 \n",
    "  - Neural Network를 이용해 직접 한글 임베딩 학습을 해보고 평가해본다. \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 데이터 확보  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/haven-jeon/KoWordSpacing\n",
    "!bunzip KoWordSpacing/input.txt.bz2\n",
    "#임베딩 평가 데이터 \n",
    "!git clone https://github.com/SungjoonPark/KoreanWordVectors "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import warnings\n",
    "import logging\n",
    "import random\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import mxnet as mx\n",
    "import gluonnlp as nlp\n",
    "from mxnet.gluon import nn\n",
    "import numpy as np\n",
    "import mxnet as mx\n",
    "from mxnet import nd, gluon, autograd\n",
    "import itertools\n",
    "\n",
    "from konlpy.tag import Mecab\n",
    "import re\n",
    "mecab = Mecab()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "라인단위 형태소 분석을 통해 토큰을 추출여 이를 list 형태로 리턴함 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `vocab` 과 `dataset` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sejong_dataset = nlp.data.dataset.CorpusDataset('KoWordSpacing/input.txt', \n",
    "                                tokenizer=lambda x:mecab.morphs(x.strip()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['옷', '을', '만드', '느라', '늘', '대하', '는', '천', '을', '실내']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sejong_dataset[0][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- index to token, token to index 그리고 미등록어 및 문장 시작, 종료, 패딩 등의 작업을 해줄 `vocab`객체 생성\n",
    "- `vocab`은 빈도수 정보로 단어사전의 크기를 결정함으로 '단어:빈도'의 정보를 `counter`로 받아 생성한다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = nlp.data.count_tokens(itertools.chain.from_iterable(sejong_dataset))\n",
    "\n",
    "vocab = nlp.Vocab(counter, unknown_token='<unk>', padding_token=None,\n",
    "                  bos_token=None, eos_token=None, min_freq=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "단어는 인덱스로 접근되며, 인덱스 번호로 토큰에 접근할 수 있다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<unk>\n",
      ".\n",
      "이\n",
      "는\n",
      "을\n",
      "다\n",
      "의\n",
      "에\n",
      "하\n",
      "은\n"
     ]
    }
   ],
   "source": [
    "for word in vocab.idx_to_token[:10]:\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "토큰을 기준으로 인덱스 번호도 출력 할 수 있다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0\n",
      "631 631\n"
     ]
    }
   ],
   "source": [
    "print(vocab.token_to_idx[\"<unk>\"], vocab['<unk>'])\n",
    "print(vocab.token_to_idx[\"아침\"], vocab['아침'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative Sampling 과 subsampling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ P({ w }_{ i })=1-\\sqrt { \\frac { t }{ f({ w }_{ i }) } } $$\n",
    "\n",
    "자주 출현하는 단어들에 대해서 학습셋에 포함되는 확률을 줄여주기 위한 서브샘플링(subsampling)기법을 통해 학습의 효율을 획기적으로 올리는 기법도 사용된다. 이를 통해 학습 데이터 자체를 줄여주어 학습속도를 올릴 수 있게 된다. \n",
    "\n",
    "여기서 $f(w_i)$는 단어의 출현 확률을 의미한다. $P(w_i)$가 작아야 학습셋에 들어갈 확률이 높아지는데, 따라서 출현 확률이 낮은 단어일 수록 학습셋에 포함될 확률이 높아진다. 고빈도 단어의 학습셋 출현 확률을 줄여주고, 저빈도 단어의 학습셋 출현확률을 높여주는 것이다. \n",
    "$t$는  subsampling constant로 일반적으로 $10^{-5}$로 한다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "frequent_token_subsampling = 1E-5\n",
    "idx_to_counts = np.array([counter[w] for w in vocab.idx_to_token])\n",
    "f = idx_to_counts / np.sum(idx_to_counts)\n",
    "idx_to_pdiscard = 1 - np.sqrt(frequent_token_subsampling / f)\n",
    "coded_dataset = [[vocab[token] for token in sentence\n",
    "                  if token in vocab\n",
    "                  and random.uniform(0, 1) > idx_to_pdiscard[vocab[token]]] for sentence in sejong_dataset]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Center 토큰을 기준으로 좌우 `windows_size` 만큼의 Context 토큰을 샘플링해준다. 같은 비율의 negative sample도 추가해준다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "batchify = nlp.data.batchify.EmbeddingCenterContextBatchify(batch_size=2048, window_size=5)\n",
    "context_sampler = batchify(coded_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "빈도수 기준으로 `negative sample`을 추출할 수 있게 샘플러 구성 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "negatives_weights = nd.array([counter[w] ** 0.75  for w in vocab.idx_to_token])\n",
    "negatives_sampler = nlp.data.UnigramCandidateSampler(negatives_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class embedding_model(nn.Block):\n",
    "    def __init__(self, input_dim, output_dim, neg_weight, num_neg=5):\n",
    "        super(embedding_model, self).__init__()\n",
    "        self.num_neg = num_neg\n",
    "        self.negatives_sampler = nlp.data.UnigramCandidateSampler(neg_weight)\n",
    "        with self.name_scope():\n",
    "            #center word embedding \n",
    "            self.w  = nn.Embedding(input_dim, output_dim)\n",
    "            #context words embedding \n",
    "            self.w_ = nn.Embedding(input_dim, output_dim)\n",
    "    \n",
    "    def forward(self, center, context, context_mask):\n",
    "        #이렇게 해주면 \n",
    "        #nd.array를 선언시 디바이스를 지정하지 않아도 된다. \n",
    "        #멀티 GPU 학습시 필수 \n",
    "        with center.context:\n",
    "            #주변단어의 self.num_neg 배수 만큼 비 주변단어를 생성한다.  \n",
    "            negs = self.negatives_sampler((context.shape[0], context.shape[1] * self.num_neg))\n",
    "            negs = negs.as_in_context(center.context)\n",
    "            context_negs = nd.concat(context, negs, dim=1)\n",
    "            embed_c = self.w(center)\n",
    "            #(n_batch, context_length, embedding_vector)\n",
    "            embed_u = self.w_(context_negs)\n",
    "\n",
    "            #컨텍스트 마스크의 크기를 self.num_neg 만큼 복제해 값이 있는 영역을 표현한다.\n",
    "            #결국 주어진 주변단어 수 * self.num_neg 만큼만 학습을 하게 된다. \n",
    "            context_neg_mask = context_mask.tile((1, 1 + self.num_neg))\n",
    "\n",
    "            #(n_batch, 1 , embedding_vector) * (n_batch, embedding_vector, context_length)\n",
    "            #(n_batch, 1, context_length)\n",
    "            pred = nd.batch_dot(embed_c, embed_u.transpose((0,2,1)))\n",
    "            pred = pred.squeeze() * context_neg_mask\n",
    "            \n",
    "            #네거티브 샘플들은 레이블이 모두 0이다. \n",
    "            label = nd.concat(context_mask, nd.zeros_like(negs), dim=1)\n",
    "        return pred, label\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = len(vocab.idx_to_token)\n",
    "vec_size = 300\n",
    "embed = embedding_model(vocab_size, vec_size, negatives_weights, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctx = mx.gpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed.initialize(mx.init.Xavier(), ctx=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\n",
       " [[ 0.00023127 -0.00471653 -0.         -0.         -0.         -0.\n",
       "   -0.         -0.         -0.         -0.         -0.00119895  0.00133953\n",
       "   -0.          0.         -0.         -0.          0.          0.\n",
       "    0.          0.         -0.00092426  0.00187225  0.         -0.\n",
       "    0.         -0.         -0.         -0.          0.          0.\n",
       "    0.00105074 -0.00181746  0.          0.          0.         -0.\n",
       "   -0.          0.         -0.          0.         -0.00184016  0.0013353\n",
       "    0.         -0.          0.          0.          0.          0.\n",
       "   -0.         -0.          0.00023259 -0.00101983 -0.          0.\n",
       "   -0.          0.          0.          0.         -0.          0.        ]]\n",
       " <NDArray 1x60 @gpu(0)>, \n",
       " [[1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
       "   0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
       "   0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
       " <NDArray 1x60 @gpu(0)>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embed(nd.array([[3],],ctx=ctx), nd.array([[10, 20, 0,0,0,0,0,0,0,0],], ctx=ctx), \n",
    "      nd.array([[1,1,0,0,0,0,0,0,0,0],], ctx=ctx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "wv_golden = pd.read_csv('KoreanWordVectors/WS353_korean.csv')\n",
    "\n",
    "word1 = wv_golden['word 1']\n",
    "\n",
    "word2 = wv_golden['word 2']\n",
    "\n",
    "score = wv_golden['kor_score']\n",
    "\n",
    "res = [[vocab.token_to_idx[i],vocab.token_to_idx[j],k] for i,j,k in zip(word1, word2, score) \n",
    "       if vocab.token_to_idx[i] != 0 and vocab.token_to_idx[j] != 0]\n",
    "\n",
    "word12score = nd.array(res, ctx=ctx)\n",
    "\n",
    "word1, word2, scores = (word12score[:,0], word12score[:,1], word12score[:,2])\n",
    "\n",
    "\n",
    "def pearson_correlation(w2v, word1, word2, scores):\n",
    "    from scipy import stats\n",
    "    evaluator = nlp.embedding.evaluation.WordEmbeddingSimilarity(\n",
    "        idx_to_vec=w2v,\n",
    "        similarity_function=\"CosineSimilarity\")\n",
    "    evaluator.initialize(ctx=ctx)\n",
    "    evaluator.hybridize()\n",
    "    pred = evaluator(word1, word2)\n",
    "    scorr = stats.spearmanr(pred.asnumpy(), scores.asnumpy())\n",
    "    return(scorr)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word 1</th>\n",
       "      <th>word 2</th>\n",
       "      <th>kor_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>사랑</td>\n",
       "      <td>섹스</td>\n",
       "      <td>6.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>호랑이</td>\n",
       "      <td>고양이</td>\n",
       "      <td>7.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>호랑이</td>\n",
       "      <td>호랑이</td>\n",
       "      <td>10.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>책</td>\n",
       "      <td>종이</td>\n",
       "      <td>6.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>컴퓨터</td>\n",
       "      <td>키보드</td>\n",
       "      <td>6.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>컴퓨터</td>\n",
       "      <td>인터넷</td>\n",
       "      <td>5.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>비행기</td>\n",
       "      <td>자동차</td>\n",
       "      <td>6.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>기차</td>\n",
       "      <td>자동차</td>\n",
       "      <td>6.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>전화기</td>\n",
       "      <td>의사소통</td>\n",
       "      <td>4.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>텔레비전</td>\n",
       "      <td>라디오</td>\n",
       "      <td>5.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  word 1 word 2  kor_score\n",
       "0     사랑     섹스       6.42\n",
       "1    호랑이    고양이       7.17\n",
       "2    호랑이    호랑이      10.00\n",
       "3      책     종이       6.17\n",
       "4    컴퓨터    키보드       6.67\n",
       "5    컴퓨터    인터넷       5.92\n",
       "6    비행기    자동차       6.00\n",
       "7     기차    자동차       6.75\n",
       "8    전화기   의사소통       4.83\n",
       "9   텔레비전    라디오       5.58"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wv_golden.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "206it [00:31,  6.59it/s]\n",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 epoch, loss 0.6093261241912842, corr -0.0398154996422957\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "206it [00:30,  6.84it/s]\n",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 epoch, loss 0.5968636870384216, corr -0.02563234379998689\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "206it [00:30,  6.84it/s]\n",
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 epoch, loss 0.6014129519462585, corr 0.0046576298785832955\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16it [00:02,  6.36it/s]"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "ctx = mx.gpu()\n",
    "\n",
    "num_negs = 5\n",
    "vocab_size = len(vocab.idx_to_token)\n",
    "vec_size = 100\n",
    "\n",
    "embed = embedding_model(vocab_size, vec_size, negatives_weights, 5)\n",
    "embed.initialize(mx.init.Xavier(), ctx=ctx)\n",
    "\n",
    "loss = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n",
    "optimizer = gluon.Trainer(embed.collect_params(), 'adam', {'learning_rate':0.001})\n",
    "\n",
    "avg_loss = []\n",
    "corrs = []\n",
    "interval = 50\n",
    "\n",
    "epoch = 70\n",
    "\n",
    "for e in range(epoch):    \n",
    "    for i, batch in enumerate(tqdm(context_sampler)):\n",
    "        center, context, context_mask  = [d.as_in_context(ctx) for d in batch]\n",
    "        with autograd.record():\n",
    "            pred, label = embed(center, context, context_mask)\n",
    "            loss_val = loss(pred, label)\n",
    "        loss_val.backward()\n",
    "        optimizer.step(center.shape[0])\n",
    "        avg_loss.append(loss_val.mean().asscalar())\n",
    "    \n",
    "    corr = pearson_correlation(embed.w.weight.data(), word1, word2, scores)\n",
    "    corrs.append(corr.correlation)\n",
    "    print(\"{} epoch, loss {}, corr\".format(e + 1, loss_val.mean().asscalar()), corr.correlation)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\n",
       " [5700.]\n",
       " <NDArray 1 @gpu(0)>, \n",
       " [ 713.  857. 1659.   21. 8299.    0.    0.    0.    0.    0.]\n",
       " <NDArray 10 @gpu(0)>, \n",
       " [1. 1. 1. 1. 1. 0. 0. 0. 0. 0.]\n",
       " <NDArray 10 @gpu(0)>)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "center[20,], context[20,], context_mask[20,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs_pd = pd.DataFrame({'epoch':list(range(1, 71)), 'corr':corrs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEWCAYAAACdaNcBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4lOW9//H3N5M9JIGQsGVh35FFUNxQStWKe0+1dal1a61t7WZPWz1d9Fjb0732nFqPtrXVn1ZarVqO0rrjAsq+hy1AgATIBgkkIdvk/v0xD3SICSSQZLbP67rmyjzLPPOdZyafued+NnPOISIi0SUu1AWIiEj3U7iLiEQhhbuISBRSuIuIRCGFu4hIFFK4i4hEIYW7SBeY2UIz+2yo6+gsMys2swtP8rGzzGxzd9ckvUPhHkXM7DwzW2xmNWa238wWmdkZoa6rN5nZMDNzZlbr3YrN7J4Q1nODmS33atlrZv8ws/NCVc/xeOtt1JFh59y7zrmxoaxJTl58qAuQ7mFmGcBLwBeAvwKJwCygsZfriHfOtfTmc3agr3OuxcxmAG+b2Qrn3Gu9WYCZ3Q3cA9wJvAI0AZcAVwHvdXFZH1qvYbSuJQyp5R49xgA4555xzvmdc4edc68659YCmNktXkv+N17LfpOZffTIg80s08z+4LUuS83sQTPzedNGmtmbZlZlZpVm9rSZ9Q16bLGZfdvM1gJ1Zhbvjfumma01szpv2QO9lushM3vdzPoFLeNZM9vn1faOmU0MmvYnM3vYzF72HrvEzEZ2ZqU455YDG4CpQcu7x8y2ecsqNLOPB027xczeM7Ofm9kBM9thZnPbW7aZDfZe3zfbmZYJPAB8yTn3vHOuzjnX7Jz7P+fcN715kszsITPb490eMrMkb9psMyvx1us+4I/tjfPmvdzMVptZtffLbXIH9Z5pZu978+31PguJ3rR3vNnWeL8yPnXk+YIeP97rlqo2sw1mdmV3vEfSQ5xzukXBDcgAqoAngLlAvzbTbwFagK8DCcCngBogy5v+AvAokAYMAJYCn/emjQIuApKAHOAd4KGgZRcDq4F8ICVo3AfAQCAXKAdWAtOAZOBN4L6gZdwGpHvP8RCwOmjan7zXdiaBX5tPA/M6WA/DAAfEe8NnAfXAx4PmuRYYQqBx8ymgDhgctJ6agc8BPgK/hPYA5k1fCHwWGA5sAe7ooI5LvPUdf5z37AFvHQ3w1uti4AfetNne43/irZOUDsZN89btTK/em711nxT0Plzo3Z/urY94bz1tBL4WVI8DRgUNzwZKvPsJQBHwHwR+Fc4BDgFju/oe6dZLmRDqAnTrxjcTxnv/ZCVeCMwHBnrTbgkOKW/cUuAmAgHciBfM3rTrgbc6eJ6rgVVBw8XAbW3mKQZuDBr+G/BI0PCXgRc7WH5fL2gyveE/Ab8Pmn4psKmDxw7zHlsNHPbu/zz4dbfzmNXAVUHrqShoWqq3jEHe8ELgl97ru/44y7wR2HeC92sbcGnQ8MeAYu/+bALdOMlB09sb9wjeF0LQuM3ABUHvw4UdPP/XgBeCho8X7rOAfUBc0PRngPu7+h7p1js3dctEEefcRufcLc65PGASgdbpQ0GzlDrvP8+z05tnKIGW2V7vJ3c1gVb8AACvO2We111zEHgKyG7z9LvbKaks6P7hdob7eMv3mdmPva6SgwQCiTbPsS/ofv2Rxx5HtjfPNwiEVMKRCWb2maBujGoC66rd53LO1Xt3g5/vRqAUeO44z18FZJvZ8bZrDSHwHhxx5P04osI519DmMW3HDQW+ceS1eK8nv81yADCzMWb2ktf9dRD4ER9+H49X627nXGubenODhrv6HkkPUrhHKefcJgKtqUlBo3PNzIKGCwi05ncTaLlnO+f6ercM59yRfu8fEWjVneacywA+DQQvB2/6ybqBwEbGC4FMAq1v2nmOLnGBbQ+/BBqALwKY2VDgd8BdQH/nXF9gfRef636gEvjzke0S7XifwDq9+jjL2UMgnI848n4cfQntPKbtuN3AD4Pet77OuVTn3DPtPPYRYBMw2nsf/4POv+49QL6ZBWdGAYEvOQlDCvcoYWbjzOwbZpbnDecT6Fr5IGi2AcBXzCzBzK4l0I2zwDm3F3gV+IWZZZhZnLcR9QLvcelALVBjZrnAhzYgnqJ0AkFYRaAb5EfdvPwfA98ys2QC2xQcUAFgZrdy7BdgZzQT6LdPA55sE3gAOOdqgO8DD5vZ1WaW6q33uWb2U2+2Z4DvmlmOmWV78z/VxVp+B9xpZjMtIM3MLjOz9HbmTQcOArVmNo7A9oRgZcCIDp5nCYHW+Le81zEbuAKY18V6pZco3KPHIQIb1ZaYWR2BUF9PoFviiCXAaAKtzh8C1zjnqrxpnyGwoawQOECgy2GwN+0/gdMJbIB9GXi+m2t/ksBP/FLv+T84/uxd9jKB1/Q551wh8AsCLesy4DRgUVcX6JxrAv6NwPaKxzsI+F8AdwPfJfBlspvAL4YXvVkeBJYDa4F1BDY4P9jFOpYT2Pj7GwKvsYjAdoP2/DuBX0mHCHwp/KXN9PuBJ7zunU+2eZ4mAmE+l8Dn57fAZ7xfiBKG7NguWIlWZnYL8FnnXFgeQCMi3UstdxGRKKRwFxGJQuqWERGJQmq5i4hEoZCdOCw7O9sNGzYsVE8vIhKRVqxYUemcyznRfCEL92HDhrF8+fJQPb2ISEQys50nnkvdMiIiUUnhLiIShRTuIiJRKKyuxNTc3ExJSQkNDW1PhBf5kpOTycvLIyEh4cQzi4icorAK95KSEtLT0xk2bBjHnrwwsjnnqKqqoqSkhOHDh4e6HBGJAWHVLdPQ0ED//v2jKtgBzIz+/ftH5S8SEQlPYRXuQNQF+xHR+rpEJDyFVbeMSKg459hf10RCfBwpCT4SfP9q9/hbHQ3Nfg43+0lJ8JGW1P6/ze799SzcUkFtQwv9UhPol5ZIVloiA9KTyO+XSlycvuCl9yjcJeYtKqrkp//cxJqSmqPjfHFGUnwcLX5Hk/9fV5Yzg5E5fZicm8lpeZkUZKWydMd+3txUztby2g6fIy3Rx/jBGUwcksG4wRnUNbZQXFXHzqp6dlTWkZYYzxc/MpIrJg/p9JdAQ7OfxuZWmvytNPtbaWxpZW/1YXZU1bGjoo7iqjrAuGZ6HheOH0C8L+x+qEsP6lS4m9klwK8JXF399865H3cw3ycIXOThDO8iAjGjpaWF+Pj4Docl/KzZXc3PXtnMe0WVDMlM5luXjCUhLo6GZj8NLYHgjPcFWvIpiXEkJ/jYX9fE+tIa3i2q5PlVgSvMxccZM0dk8akz8pkzbgCDMpM5UN/MgbomDtQ3sbe6gcK9B9mwp4bnVpRQ1+QHICM5nuHZaUwf2o/N+w7x1Xmr+e1b2/j6RWP42MSB7Xbl7a9r4uW1e3hhVSkrd1V3+NqS4uMYnp1GdX0zr28sY1BGMjfMLOC6M/IZkJHcMytUwsoJ08e7RuTDwEVACbDMzOZ7V7QJni8d+CqBq/1EtCeffJKf//znmBmTJ0/mBz/4AbfddhuVlZXk5OTwxz/+kYKCAm655RaSk5NZtWoV5557LhkZGWzbto3t27dTUFDAM8+0dxlLCbWmllbueX4tz68sJSstke9dPoEbZxaQnNDR5VDbV3awgR2VdUwckkF68rG7uKYmxpPbN+VDj2ltdZRWH6ZPUjz90hKPGf/yur386vUt3PnUCsYPzmD8oHTSkuJJS4onNdHH2pJqFm6uoKXVMXZgOl/56Gj6piSQEB9Hki+OeJ8xKCOZYdlpDMpIJi7OaPG38uamcv7fBzv55Wtb+O83tnLV1Fy+MHskowaE7vrVzjn21DSQ6IsjJz0pZHVEs840Lc8Eipxz2wHMbB6BixkXtpnvB8BP6Kbra/7n/22gcM/B7ljUUROGZHDfFROPO8+GDRt48MEHWbx4MdnZ2ezfv5+bb7756O3xxx/nK1/5Ci++GLhSWklJCYsXL8bn83H//fdTWFjIe++9R0rKh/+xJfSaWlr54tMreX1jGV+cPZIvzB75oWDurIEZyQzsYis4Ls7Iz0ptd/wVU4Ywd9Ig/r56D09+sJOlxfupa2yhrslPU0srAzOSuP284Vw9LZfxgzM69XzxvjgunjiIiycOYkdlHU8sLmbesl08v6qEuZMG8cXZoxgzMJ11pTWs2LmfZcUH2FlVx0fGDeDa6fkn/QWwtqSaTfsOkZEcT0ZKAhnJCfjijFW7qlm6o4qlO/azpyaw91hBVirTh/bj9KH9mJrXlxE5aR1u15DO68wazCVw7ccjSghcq/MoMzsdyHfOvWxmHYa7md0B3AFQUFDQ9Wp7wZtvvsm1115LdnY2AFlZWbz//vs8/3zgsqE33XQT3/rWt47Of+211+Lz/avFd+WVVyrYw1RwsD9w1UQ+c/awUJf0IfG+OD4xPY9PTM87ZnxTSysJPjulva6GZ6dx/5UT+fKcUTy+aAdPLt7JgnX7SIyPo6ml9eg8gzOT+f27O3j07e1MH9qPT87IY8LgTMwC2xwMo09SPHn9Uo7ZPuCc4/1tVTy8sIhFRVUdlUF2nyRmjsji88OyaGzxs3JnNe9ureQFr5sLYFBGMsOz0xiWnUqcGYeb/NQ1tVDf5CctMZ4Zw/px1oj+jB+cgU8bqtt1yl+P3oWBf0nHF+U9yjn3GPAYwIwZM457lZATtbDDRVpa2nGHpXc1+1t57J3tlBw4zEfHDeC80dkkJ/giItiPJzG++zaG9u+TxDc/No47zh/JM0t3UVXbyPSh/Zg+NOtoF0n5oQZeWFnKX5fv5tt/W9fuctKT4pmYm8FpuZkM7Z/G31aWsGpXNTnpSfzHpeP42MRB1DX6OdjQzMHDzTS0tDJpSAbDs9M+9CXlnKPkwGHWldawo7KO7RV17Kis5dUNZQCkJvlITYgnNcnHzqp6/rlh39EazhiexY0zC5gzboB2OQ7SmXAvBfKDhvO8cUekA5OAhd6KHQTMN7MrI3Gj6pw5c/j4xz/O3XffTf/+/dm/fz/nnHMO8+bN46abbuLpp59m1qxZoS5T2rGjso6vzVvFmpIaUhN9PLN0FykJPs4fk019k593t1ZGZLD3lMyUBO68YGS70wakJ/P5C0Zyx/kjWFdaQ9nBRpxzOAJBfKC+mfWlNawvreGJxTtp8reS2zeFH1w9iWun53V5+4VZoLuqvS6r9uyraWCJ173z1qZybn9iOROHZPDlOaO4eMKgo78oWlsduw/UU3LgMM3+Vlqdo7UVHDA1v2+39vfXNrawtewQKYk+hmenkRTftXXQ3ToT7suA0WY2nECoXwfccGSic64GyD4ybGYLgX+PxGAHmDhxIt/5zne44IIL8Pl8TJs2jf/5n//h1ltv5Wc/+9nRDaoSPpxz/HX5bu6fX0hifBy/vfF0Lhw/kCU7qnhlwz5eKyyj7GCjgv0kmBmT8/oed56mllZ27a9naP/UY44P6EmDMpO5amouV03NpdnfyourSvntwm3c+dRKxg5MZ2JuBlvLaikqr+Vws7/dZST64rhiyhBuP284E4YcfxtG+cEGVu+upvxQI/5WR0urw9/aSm2jny37DrFx30F2VtUfnd8XZwztn8qYAelMK+jLjWcNpU8vb0fo1DVUzexS4CECu0I+7pz7oZk9ACx3zs1vM+9COhHuM2bMcG0v1rFx40bGjx/ftVcQQaL99fWmxhY/W8tqWV9aw6uFZby5qZxzRvbnF5+cwuDMY7d5tLY6quqatFdGlGvxt/Lyur08snAbB+qbGDMw3bv1YWj/NBLj44gzw2dGk9/P31fv4dnlJRxu9nP2iP5cPmUwhtHS2kqz33G4qYXCvQdZvav66MbftsxgeP80xg/OYNygdMYOSudwc+CzubX8EFvLatleWUd2nyTuvmgMn5yRd8rHG5jZCufcjBPOF6oLZCvcpTO2V9SypewQe6ob2FtzmD01DeyoqGNr+SGa/YHPbkZyPHfNGcVnzxuho0ClS2rqm5m3bBdPLC5uN8Dzs1KYkteXqfl9mVbQl/x+qcT74vDFGQk+I8EXd8JfK6t2HeCHL29k+c4DjBnYh3svHc/sMTknvX1A4R6Gov31dZemllb+uWEfT70f2B3wiKT4OIb0TSGvXwqTcjOZNCSTSbkZOrRfTlmLv5V9BxuIjwscL5AQF0difBwpid3Tb+6c45UN+/jxPzZRXFXPdy8bz2dnjTipZXU23MNuZ1LnXFRu8Q7Vl2gkqaxt5InFxTyzdDeVtY0UZKVy79xxnDsqmyF9U+iXmhCVnw0JvXhfHHn9Orcx92SYGZdMGsyccQN5eslOLjttcI891xFhFe7JyclUVVVF3Wl/j5zPPTlZh323p7K2kcfe2c6T7xfT2NLKnLEDuOnsoZw/OkctcokqifFx3Hpu71zTIazCPS8vj5KSEioqKkJdSrc7ciUm+Zeq2kYe9UK9qaWVq6fmctecUYzICd1h8SLRIqzCPSEhQVcqihEHG5r5xCOL2bW/XqEu0gPCKtwlNjjnuPdv69h94DB//txZnDWif6hLEok6OsGz9LqnPtjJy+v28s2PjVWwi/QQhbv0qvWlNfzgpY18ZGwOd5zkrmAicmIKd+k1Bxua+eLTK+nfJ5FffnKq9oQR6UHqc5de0dDs59vPraW0+jB//fxZx1yoQkS6n8JdetTWskM8s3Q3z68qobq+mXvnjmP60KxQlyUS9RTu0iNW7NzPjxZsYsXOAyT4jIsnDOL6Mws4d5Q2oIr0BoW7dLvd++u57U/LSUv08R+XjuPfTs8ju4/OyCjSmxTu0q0amv184ekVtDrHnz93FsOydWUqkVBQuEu3cc7xvRfXs770IL//zAwFu0gIaVdI6Tbzlu3m2RUlfHnOKC6cMDDU5YjENIW7dIs1u6u57+8bmDU6m69dOCbU5YjEPIW7nLL6pha++PRKctKT+O/rpuHTwUkiIac+dzllj769ndLqwzx759k6OEkkTKjlLqdkX00Dj76zjcsmD+aMYTo4SSRcKNzllPzslc20tsI9l4wLdSkiEkThLidtXUkNf1tZwq3nDSM/q+euPykiXadwl5PinOPBlwvJSkvkSx8ZFepyRKQNhbuclFcLy1iyYz9fv2gMGckJoS5HRNpQuEuXNbW08l8LNjJqQB+uPyM/1OWISDsU7tIlzjl+tGAjxVX1fOey8cT79BESCUf6z5ROc87x01c286fFxdxyzjBmj8kJdUki0gGFu3Tar9/YyiMLt3HDzALuu2ICZjoSVSRcKdylU367sIiHXt/KNdPzePCqSQp2kTCncJcTevL9Yn76z81cOWUIP/nEZF3YWiQCKNzluBpb/Pzsn5uZNTqbX35yik4KJhIhFO5yXO9uqeRQYwu3nzdce8aIRBD9t8pxLVi3l8yUBM4dlR3qUkSkCxTu0qGGZj+vFZbxsYkDSVCrXSSi6D9WOvTu1kCXzGWTh4S6FBHpok6Fu5ldYmabzazIzO5pZ/qdZrbOzFab2XtmNqH7S5Xe9vLaPfRNTeCckf1DXYqIdNEJw93MfMDDwFxgAnB9O+H9Z+fcac65qcBPgV92e6XSqxqa/by+sZyPTRikLhmRCNSZ/9ozgSLn3HbnXBMwD7gqeAbn3MGgwTTAdV+JEgrvbKmgtrGFyyYPDnUpInISOnMN1Vxgd9BwCTCz7Uxm9iXgbiARmNPegszsDuAOgIKCgq7WKr3o5XV76ZeawNnqkhGJSN32e9s597BzbiTwbeC7HczzmHNuhnNuRk6OTjoVrhqa/bxeWMYlk9QlIxKpOvOfWwoEn7Q7zxvXkXnA1adSlITWws0V1DX5ufQ0dcmIRKrOhPsyYLSZDTezROA6YH7wDGY2OmjwMmBr95Uove1ol8wIdcmIRKoT9rk751rM7C7gFcAHPO6c22BmDwDLnXPzgbvM7EKgGTgA3NyTRUvPaWj288bGMq6amqvTDYhEsM5sUMU5twBY0Gbc94Puf7Wb65IQeXNTOfVNfi7XXjIiEU1NMznGC6tKGZCexFnqkhGJaAp3Oaq6vomFm8u5csoQndpXJMIp3OWol9ftpdnvuHpabqhLEZFTpHCXo/6+ag+jBvRh4pCMUJciIqdI4S4AlByoZ2nxfj4+LVfXRxWJAgp3AeDvq/cAcOUUnd5XJBoo3AXnHC+uKuWMYf3Iz0oNdTki0g0U7kLh3oNsLa/lqqnakCoSLRTuwourSknwGZfpXDIiUUPhHuP8rY75a/ZwwZgB9EtLDHU5ItJNFO4x7oPtVZQdbOTj2rddJKoo3GPccytK6JMUz0fHDwh1KSLSjRTuMWxP9WH+b80erpmeR3KCL9TliEg3UrjHsMff24EDbj9veKhLEZFupnCPUTX1zTyzdBdXTB6sfdtFopDCPUY9tWQndU1+Pn/ByFCXIiI9QOEegxqa/fxx0Q4uGJPD+ME6SZhINFK4x6C/rSyhsraJO9VqF4laCvcY4291/O6d7UzJy+SsEVmhLkdEeojCPca8smEfxVX13HnBSJ3aVySKKdxjiHOOR9/exvDsNC6eOCjU5YhID1K4x5DnV5aypqSGOy8YoWukikQ5hXuMqKxt5AcvFzJ9aD+unZ4f6nJEpIcp3GPE/fM3UN/o5yefOI04tdpFop7CPQa8XljGS2v38uU5oxg1ID3U5YhIL1C4R7mDDc1898X1jBuUrqNRRWJIfKgLkJ71k39sovxQA4/eNJ3EeH2Xi8QK/bdHsVc37OPpJbu4/bzhTMnvG+pyRKQXqeUehZr9rfzi1S3879vbmDgkg7svGhvqkkSklynco0zJgXq+8swqVu6q5saZBXzv8gm6EIdIDFK4R5E3N5XxtXmrcQ5+c8M0Lp88JNQliUiIKNyjREOzn688s5r8rFT+99OnM7R/WqhLEpEQ0gbVKPHmpnJqG1v4zqXjFewionCPFvNX7yG7TxJnj+wf6lJEJAwo3KPAwYZm3txczuWTB+uEYCICdDLczewSM9tsZkVmdk870+82s0IzW2tmb5jZ0O4vVTry6oYymlpauWKKNqCKSMAJw93MfMDDwFxgAnC9mU1oM9sqYIZzbjLwHPDT7i5UOjZ/zR7y+qVweoEOVBKRgM603M8Eipxz251zTcA84KrgGZxzbznn6r3BD4C87i1TOlJV28iiokqumDJEV1YSkaM6E+65wO6g4RJvXEduB/7R3gQzu8PMlpvZ8oqKis5XKR1asG4v/lbHVVPVJSMi/9KtG1TN7NPADOBn7U13zj3mnJvhnJuRk5PTnU8ds+av2cOYgX0YNygj1KWISBjpTLiXAsGX7snzxh3DzC4EvgNc6Zxr7J7y5HhKqw+zrPgAV2pDqoi00ZlwXwaMNrPhZpYIXAfMD57BzKYBjxII9vLuL1Pa89KaPQDaS0ZEPuSE4e6cawHuAl4BNgJ/dc5tMLMHzOxKb7afAX2AZ81stZnN72Bx0o3mr9nDlPy+OiJVRD6kU+eWcc4tABa0Gff9oPsXdnNdcgJrdlezYc9Bvnd5271SRUR0hGpEWrO7mpv/uJSBGUlcrb1kRKQdCvcI88H2Km743QdkJCfw3J3n0L9PUqhLEpEwpFP+RpC3NpVz51MrKMhK5anPzmRgRnKoSxKRMKVwjxD/XL+PLz+zkrGD0nnytplkpSWGuiQRCWMK9wiwvaKWu/+6mkm5mTxx25lkJCeEuiQRCXPqcw9zjS1+vjJvFYnxcTxy43QFu4h0ilruYe7nr2xmfelBHrtpOoMy1ccuIp2jlnsYW7i5nN+9u4ObzhrKxRMHhbocEYkgCvcwVX6ogX9/dg1jB6bzncvGh7ocEYkw6pYJA845tpTVsr+uiZrDTVTXN/P8ylIONbTw58+dRXKCL9QlikiEUbiHgR8t2Mjv3t1xzDhfnPGjj09izMD0EFUlIpFM4R5ixZV1/HFRMZdNHsyNZxaQmZpAv9RE+qUmkpKoFruInByFe4j9/NXNJPjiuO/yCQzQEaci0k20QTWE1pZU89LavXx21nAFu4h0K4V7iDjn+PE/NpGVlsgd548IdTkiEmXULdPN7vv7et7eUsGA9GRyMpIYmJ5MQVYK18zIp0/Sv1b3O1srWbytivuumEC6jjoVkW6mcO9GG/bU8MT7O5mS3xcMCvccZOHBcuqa/PzmrSK+cfFYPjkjHwN+/I9N5GelcMPMglCXLSJRSOHejR56fSvpyfE8eduZZKb8qzW+Znc1D75cyL3Pr+NPi4qZNTqbjXsP8uvrppIUrz1iRKT7qc+9m6wrqeG1wjI+N2vEMcEOMCW/L3/9/Nk8cuPpHG728/v3djBxSAZXTNZVlESkZ6jl3k1+9foWMlMSuPXcYe1ONzPmnjaYOeMH8PfVe5g+tB9xcda7RYpIzFDLvQsef28HVz+8iOLKumPGr9p1gDc3lXPH+SNOuHE0Kd7HJ2fkMzKnT0+WKiIxTuHeSYcamvnV61tYvbuaq3+7iCXbq45O+9XrW8lKS+SWc4aFrkARkSAK907685JdHGpo4X+un0ZWaiKf/sMS/raihOXF+3lnSwV3XjCCtCT1colIeFAadUJji58/vLeDc0f154opQzh/dA5feHoF33h2DTnpSWT3SeKms4aFukwRkaPUcu+EF1aWUn6okS9cMAqAzNQEnrjtTK47I5+KQ418cfZIneRLRMKKWu4n4G91PPrOdk7LzeTcUf2Pjk/wxfFf/3Yat5w7jLE6La+IhBm13E/g1Q372FFZxxdmj8Ts2F0XzYxxgzI+NF5EJNQU7sfhnOORt7cxPDuNj+kapiISQRTux7F4WxVrS2q44/wR+HTAkYhEEIX7cfzv29sYkJ7Ev52eG+pSRES6ROHegdLqw7y7tZLPnD1UJ/cSkYijcO/A64VlAFx62uAQVyIi0nUK9w68WriPkTlpjNA5YEQkAinc21FT38yS7fu5aIL2kBGRyKRwb8dbm8tpaXVcPHFgqEsRETkpnQp3M7vEzDabWZGZ3dPO9PPNbKWZtZjZNd1fZu96rbCMnPQkpub1DXUpIiIn5YThbmY+4GFgLjABuN7MJrSZbRdwC/Dn7i6wtzW2+Fm4uZwLxw/UxTREJGJ15twyZwJFzrntAGY2D7gKKDw/szzjAAAMQElEQVQyg3Ou2JvW2gM19qrF26qoa/Jz8QR1yYhI5OpMt0wusDtouMQb12VmdoeZLTez5RUVFSeziB736oYy0hJ9nD2y/4lnFhEJU726QdU595hzboZzbkZOTk5vPnWntLY6Xt9YxgVjc0hO0IFLIhK5OhPupUB+0HCeNy7qrC6ppuJQIxepS0ZEIlxnwn0ZMNrMhptZInAdML9nywqN1wrL8MUZc8Yq3EUksp0w3J1zLcBdwCvARuCvzrkNZvaAmV0JYGZnmFkJcC3wqJlt6Mmie8qrG/Yxc3gWmakJoS5FROSUdOpKTM65BcCCNuO+H3R/GYHumoi1raKWbRV13HTW0FCXIiJyynSEKlBd38S3nluLL864SBflEJEoEPPXUN1bc5jP/GEpO6vq+c3108jtmxLqkkRETllMh3tR+SE+84elHGpo4YnbztS+7SISNWI23FftOsCtf1pGgi+Ov3z+bCYMyQh1SSIi3SYmw905x9f/spqM5ASeun0mBf1TQ12SiEi3iskNqtsr6yiuqudz549QsItIVIrJcF+4OXBem9ljwu8UCCIi3SFGw72cUQP6kJ+lVruIRKeYC/f6phaWbN/PR8aq1S4i0Svmwn1xURVN/lZmjx0Q6lJERHpMzIX7wi3lpCX6mDGsX6hLERHpMTEV7s453tpUwTmjskmK1/naRSR6xVS4b6uopbT6MB9Rl4yIRLmYCve3Nnm7QGpjqohEudgK983ljB2YzhCdHExEolzMhHttYwvLiver1S4iMSFmwn1RUSXNfqddIEUkJsRMuC/cXEGfpHjtAikiMSEmwt05x8LN5Zw3KpsEX0y8ZBGJcTGRdFvKatlb06D+dhGJGTER7u9uDewCeb7OAikiMSImwn1RUSUjctK0C6SIxIyoD/emllaW7NjPeaOyQ12KiEivifpwX7XrAPVNfs5VuItIDIn6cF9UVEmcwdkj+4e6FBGRXhP14f5uUSVT8vuSkZwQ6lJERHpNVIf7wYZm1uyuVn+7iMScqA7397dV0epQf7uIxJyoDvdFRZWkJPg4vUCnHBCR2BLV4f5eUSUzR2SRGB/VL1NE5EOiNvX2VB9me0Wd+ttFJCZFbbi/V1QJwHmjFe4iEnuiNtwXFVWS3SeRsQPTQ12KiEivi8pwb211LCqq5NxR2ZhZqMsREel1URnum8sOUVnbpP52EYlZnQp3M7vEzDabWZGZ3dPO9CQz+4s3fYmZDevuQrviva2B/nbt3y4iseqE4W5mPuBhYC4wAbjezCa0me124IBzbhTwK+An3V1oV7y1uZyROsWviMSwzrTczwSKnHPbnXNNwDzgqjbzXAU84d1/Dviohaizu7K2kQ+2VzF30uBQPL2ISFjoTLjnAruDhku8ce3O45xrAWqAD52G0czuMLPlZra8oqLi5Co+gVc27KPVwaWnKdxFJHb16gZV59xjzrkZzrkZOTk9c8m7f6zbx7D+qYwfrF0gRSR2dSbcS4H8oOE8b1y785hZPJAJVHVHgV2xv66J97dXcelpg7ULpIjEtM6E+zJgtJkNN7NE4Dpgfpt55gM3e/evAd50zrnuK7NzXt2wD3+rU5eMiMS8+BPN4JxrMbO7gFcAH/C4c26DmT0ALHfOzQf+APw/MysC9hP4Auh1L6/bS0FWKhOHZITi6UVEwsYJwx3AObcAWNBm3PeD7jcA13ZvaV1zoK6Jxduq+NysEeqSEZGYFzVHqL5WWOZ1yQwKdSkiIiEXNeG+YP1e8vqlcFpuZqhLEREJuagI95r6ZhYVVWovGRERT1SE+6uF+2j2ay8ZEZEjoiLc/7F+H7l9U5iSpy4ZERGIgnDfXlHLu1srmDtpkLpkREQ8ER3uzf5Wvv6X1aQmxvPZWSNCXY6ISNjo1H7u4eq/39jKmpIafnvj6QzKTA51OSIiYSNiW+7Livfz8FtFXDM9TxtSRUTaiMhwP9TQzNf/spq8fqncf+XEUJcjIhJ2IrJb5r75G9hTfZhn7zyHPkkR+RJERHpUxLXcX1q7h+dXlnLXnNFMH9ov1OWIiISliAv3zJQELpowkC/PGRXqUkREwlbE9WnMGp3DrNE9cxUnEZFoEXEtdxEROTGFu4hIFFK4i4hEIYW7iEgUUriLiEQhhbuISBRSuIuIRCGFu4hIFDLnXGie2KwC2NnJ2bOByh4sp7up3p6lentWpNULkVfzqdQ71Dl3wiM5QxbuXWFmy51zM0JdR2ep3p6lentWpNULkVdzb9SrbhkRkSikcBcRiUKREu6PhbqALlK9PUv19qxIqxcir+Yerzci+txFRKRrIqXlLiIiXaBwFxGJQmEd7mZ2iZltNrMiM7sn1PW0x8weN7NyM1sfNC7LzF4zs63e37C5HqCZ5ZvZW2ZWaGYbzOyr3viwrNnMks1sqZmt8er9T2/8cDNb4n02/mJmiaGuNZiZ+cxslZm95A2Hbb1mVmxm68xstZkt98aF5ecBwMz6mtlzZrbJzDaa2dnhWq+ZjfXW65HbQTP7Wm/UG7bhbmY+4GFgLjABuN7MJoS2qnb9Cbikzbh7gDecc6OBN7zhcNECfMM5NwE4C/iSt17DteZGYI5zbgowFbjEzM4CfgL8yjk3CjgA3B7CGtvzVWBj0HC41/sR59zUoH2vw/XzAPBr4J/OuXHAFALrOSzrdc5t9tbrVGA6UA+8QG/U65wLyxtwNvBK0PC9wL2hrquDWocB64OGNwODvfuDgc2hrvE4tf8duCgSagZSgZXATAJH98W391kJ9Q3I8/5h5wAvARbm9RYD2W3GheXnAcgEduDtDBLu9bap8WJgUW/VG7YtdyAX2B00XOKNiwQDnXN7vfv7gIGhLKYjZjYMmAYsIYxr9ro4VgPlwGvANqDaOdfizRJun42HgG8Brd5wf8K7Xge8amYrzOwOb1y4fh6GAxXAH71ur9+bWRrhW2+w64BnvPs9Xm84h3tUcIGv5rDb39TM+gB/A77mnDsYPC3canbO+V3gZ20ecCYwLsQldcjMLgfKnXMrQl1LF5znnDudQBfol8zs/OCJYfZ5iAdOBx5xzk0D6mjTpRFm9QLgbWO5Eni27bSeqjecw70UyA8azvPGRYIyMxsM4P0tD3E9xzCzBALB/rRz7nlvdFjXDOCcqwbeItCt0dfM4r1J4fTZOBe40syKgXkEumZ+TfjWi3Ou1PtbTqA/+EzC9/NQApQ455Z4w88RCPtwrfeIucBK51yZN9zj9YZzuC8DRnt7GSQS+EkzP8Q1ddZ84Gbv/s0E+rXDgpkZ8Adgo3Pul0GTwrJmM8sxs77e/RQC2wc2Egj5a7zZwqZe59y9zrk859wwAp/ZN51zNxKm9ZpZmpmlH7lPoF94PWH6eXDO7QN2m9lYb9RHgULCtN4g1/OvLhnojXpDvZHhBBsgLgW2EOhj/U6o6+mgxmeAvUAzgVbF7QT6WN8AtgKvA1mhrjOo3vMI/ARcC6z2bpeGa83AZGCVV+964Pve+BHAUqCIwE/dpFDX2k7ts4GXwrler6413m3Dkf+zcP08eLVNBZZ7n4kXgX5hXm8aUAVkBo3r8Xp1+gERkSgUzt0yIiJykhTuIiJRSOEuIhKFFO4iIlFI4S4iEoUU7iInwcxmHznjo0g4UriLiEQhhbtENTP7tHc++NVm9qh3ErJaM/uVd374N8wsx5t3qpl9YGZrzeyFI+fYNrNRZva6d075lWY20lt8n6Dzij/tHf0rEhYU7hK1zGw88CngXBc48ZgfuJHAEYPLnXMTgbeB+7yHPAl82zk3GVgXNP5p4GEXOKf8OQSOSIbAGTW/RuB6AyMInFdGJCzEn3gWkYj1UQIXSFjmNapTCJygqRX4izfPU8DzZpYJ9HXOve2NfwJ41jvvSq5z7gUA51wDgLe8pc65Em94NYHz+r/X8y9L5MQU7hLNDHjCOXfvMSPNvtdmvpM9B0dj0H0/+n+SMKJuGYlmbwDXmNkAOHpd0KEEPvdHztB4A/Cec64GOGBms7zxNwFvO+cOASVmdrW3jCQzS+3VVyFyEtTSkKjlnCs0s+8SuMpQHIEzd36JwAUezvSmlRPol4fAqVf/1wvv7cCt3vibgEfN7AFvGdf24ssQOSk6K6TEHDOrdc71CXUdIj1J3TIiIlFILXcRkSiklruISBRSuIuIRCGFu4hIFFK4i4hEIYW7iEgU+v/xVNCxvTBx0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "corrs_pd.plot(x='epoch', y='corr', title='Spearman Rank Correlation')\n",
    "plt.savefig('spcorr.png', dpi=300)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}