Skip to content

Instantly share code, notes, and snippets.

@amqdn

amqdn/amqdn-crl.ipynb

Last active July 26, 2022 14:17
Show Gist options
  • Save amqdn/f3ba1ea30e4e21c24617f6d7aec75212 to your computer and use it in GitHub Desktop.
Save amqdn/f3ba1ea30e4e21c24617f6d7aec75212 to your computer and use it in GitHub Desktop.
Download ZIP
Implementing Class Rectification Loss in fast.ai
Display the source blob
Display the rendered blob
Raw
amqdn-crl.ipynb
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": "## Class Rectification Loss"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "https://arxiv.org/abs/1804.10851\n\nLet's try to implement Class Rectification Loss. First, we need to understand what a loss function is doing -- what goes in, what comes out? Let's make a study of different loss functions and see what we can learn. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "%reload_ext autoreload\n%autoreload 2\n%matplotlib inline",
"execution_count": 1,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "from fastai.vision import *",
"execution_count": 2,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "np.random.seed(4)",
"execution_count": 3,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### L1 Loss"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "loss = nn.L1Loss()\nloss_per_sample = nn.L1Loss(reduction='none')",
"execution_count": 4,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "In some hypothetical scenario, we can imagine that we have a batch of 15 images, and we have 15 different labels for those images. In practice, we don't use L1 Loss this way, but since we're thinking about batch-wise class rectification, we can think of L1 Loss in these terms to help us grasp the problem in a simple way. \n\nThe loss function expects the \"output\" of the model (i.e., its predictions), and then it will compare those predictions against the ground truth. As we've said, we can imagine this as 15 predictions from the model and 15 ground truth labels. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "input = torch.randn(3, 5, requires_grad=True)\ninput # 15 \"predictions\" from the model",
"execution_count": 5,
"outputs": [
{
"data": {
"text/plain": "tensor([[-0.0289, -1.4645, 0.4495, 0.6747, -0.8097],\n [-1.2833, -0.8002, -0.4095, 0.3529, 0.4514],\n [ 1.0435, -1.6748, 1.1579, 1.0776, 0.2149]], requires_grad=True)"
},
"output_type": "execute_result",
"execution_count": 5,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "target = torch.eye(3, 5)\ntarget # 15 \"ground truth\" labels",
"execution_count": 6,
"outputs": [
{
"data": {
"text/plain": "tensor([[1., 0., 0., 0., 0.],\n [0., 1., 0., 0., 0.],\n [0., 0., 1., 0., 0.]])"
},
"output_type": "execute_result",
"execution_count": 6,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "output = loss_per_sample(input, target)\noutput # L1 loss first calculates the absolute value of the distances between input x and target y...",
"execution_count": 7,
"outputs": [
{
"data": {
"text/plain": "tensor([[1.0289, 1.4645, 0.4495, 0.6747, 0.8097],\n [1.2833, 1.8002, 0.4095, 0.3529, 0.4514],\n [1.0435, 1.6748, 0.1579, 1.0776, 0.2149]], grad_fn=<L1LossBackward>)"
},
"output_type": "execute_result",
"execution_count": 7,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "output = loss(input, target)\noutput # ...then calculates the mean; in other words, the mean absolute error",
"execution_count": 8,
"outputs": [
{
"data": {
"text/plain": "tensor(0.8596, grad_fn=<L1LossBackward>)"
},
"output_type": "execute_result",
"execution_count": 8,
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Remember that the objective of SGD is to find the minimum for this loss. "
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Triplet Loss"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "PyTorch includes Triplet Loss in their available loss functions, and since this is the base for Class Rectification Loss after we've hard-mined our samples, let's also take a look at how it works. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "triplet_loss = nn.TripletMarginLoss(margin=1.0, p=2)",
"execution_count": 21,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The \"triplet\" in triplet loss is composed of three samples: an anchor sample (a random sample), a positive sample (a random other sample of the same class as the anchor), and a negative sample (a random sample that is not of the same class as the anchor). The similarities (or distances) between the anchor sample $a$ and the positive sample $p$, as well as between anchor $a$ and the negative sample $n$, are calculated using Euclidean Norm: $d_{+} = ||f(a) - f(p)||_{2}$ and $d_{-} = ||f(a) - f(n)||_{2}$. \n\nThe \"margin\" in `TripletMarginLoss` refers to \"margin ranking loss\", which aims to rank the values of these distances such that $d_{-}$ > $d_{+}$ + _µ_. Why? If we enforce a ranking such that $d_{-}$ is always greater than $d_{+}$ plus some margin _µ_, then we are making sure that all negative samples $n$ are farther away from $a$ than our positive samples $p$. And the way we tell the model we want it to optimize for this condition is by a loss function _λ_$(d_{+}, d_{-}) =$ _max_(0, _µ_ + $d_{+}$ - $d_{-}$), otherwise known as _ReLU_(_µ_ + $d_{+}$ - $d_{-}$). \n\nNote that `TripletMarginLoss` expects (a, p, n), and so the selection of these samples is left up to the user."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "a = torch.randn(3, 5, requires_grad=True)\na # Some pre-chosen sample of a class",
"execution_count": 28,
"outputs": [
{
"data": {
"text/plain": "tensor([[ 0.3072, -0.0931, 1.5000, 0.6430, -0.3071],\n [ 0.0776, -0.7049, -0.1978, -0.8530, -0.9692],\n [-1.4200, -0.8154, -1.2950, -0.3744, -0.6828]], requires_grad=True)"
},
"output_type": "execute_result",
"execution_count": 28,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "p = torch.randn(3, 5, requires_grad=True)\np # Some pre-chosen sample of the same class as a",
"execution_count": 29,
"outputs": [
{
"data": {
"text/plain": "tensor([[-1.7797, -0.3953, -0.2961, -0.6337, -0.4672],\n [-0.9017, 1.3158, 1.6002, -0.5278, 0.0329],\n [-0.9086, 0.7668, 0.8678, -1.7287, 0.6816]], requires_grad=True)"
},
"output_type": "execute_result",
"execution_count": 29,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "n = torch.randn(3, 5, requires_grad=True)\nn # Some pre-chosen sample of a different class from a",
"execution_count": 30,
"outputs": [
{
"data": {
"text/plain": "tensor([[-0.9549, 0.0301, -1.1895, 0.4843, 0.8937],\n [-1.5880, 0.3522, -0.1486, 1.2759, -0.7453],\n [-1.1284, 0.2808, -0.3532, -0.0723, 0.3850]], requires_grad=True)"
},
"output_type": "execute_result",
"execution_count": 30,
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "output = triplet_loss(a, p, n)\noutput",
"execution_count": 31,
"outputs": [
{
"data": {
"text/plain": "tensor(1.4959, grad_fn=<MeanBackward1>)"
},
"output_type": "execute_result",
"execution_count": 31,
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Class Rectification Loss (Class+Rel)"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Class Rectification Loss (Class+Rel) can be thought of as an expanded Triplet Margin Loss. \n\nWhat is Class + Rel? The authors explore multiple variants of Class Rectification Loss (CRL), but the one that produces the best results is a combination of _class_-level hard mining and _relative_ comparison. \n\nClass-level hard mining looks for those samples that the model is most wrong about. If the sample is of the same class as the anchor $a$ but the model provided a low probability score for that class, then that sample is considered a hard-positive. Similarly, if the sample is of a different class from $a$ but the model provided a high probability score, then that sample is considered a hard-negative. Class-level hard mining sorts the samples according to the worst offenders, for both hard-positive and hard-negative. \n\nThe next part of class rectification involves comparing these mined samples. Relative comparison compares the relative distances of these samples from each other. Specifically, at the class level, we compare directly the prediction probabilities outputed by the model, such that $d_{+} = |P(a) - P(p)|$ and $d_{-} = P(a) - P(n)$, where $P$ represents the probability score(s) for that sample. Note that this constitutes one of the main differences between CRL Class-level Triplet Loss and `TripletMarginLoss` above: the distances $d$ calculated here are the distances between the probabilities that the model is predicting and the ground truth, instead of the Euclidean distance between the samples themselves (which is used in the instance-level variants of CRL). \n\nThe other main difference between CRL and `TripletMarginLoss` is that it is a batch-wise operation. That is, for every mini-batch, CRL defines the majority and minority classes of that batch, mines for hard-positives and hard-negatives relative to those minority classes, and then calculates triplet loss across those minority triplets. The triplet loss is calculated for every single sample of the minority class against some number $k$ of the hard-positives and hard-negatives each, which means we form a triplet for every single anchor against $k$ hard-positives and hard-negatives. If we have 10 anchors and $k = 25$, then we will have $10 * k * k$ triplets. This can be thought of as an expanded form of Online Triplet Loss. \n\nFinally, it is important to note that CRL always focuses the triplet calculation on the minority class, whatever that minority happens to be batch-to-batch. This ensures that the model is incrementally optimizing triplet loss for the minority of every batch, which is the main innovation of CRL in addressing highly imbalanced datasets. Note that the same operation applies if there are multiple minority classes such that their sum representation in the batch is $<=$ 50%.\n\nIn order to construct our loss function, let's now play with a toy dataset to find out what's going in and what's coming out of the model."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "path = untar_data(URLs.MNIST_SAMPLE)\ndata = ImageDataBunch.from_folder(path)",
"execution_count": 9,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "data.show_batch(3, figsize=(5,5))",
"execution_count": 5,
"outputs": [
{
"data": {
"text/plain": "<Figure size 360x360 with 9 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAFgCAYAAADpZ/FJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAH6hJREFUeJzt3Xu0lUUZx/HfCKgIiiGXBAOXyC0lzfsVQVpCgmGChEboCkGRFC0vJd5vKVgaqIlIBKJ5IRUrzOXiopKtJbE0lUukIYqQAql4JcC3P2jmzHbvc/Zl3nfvs9/9/azF8mHO3u8ez3CeM/POvDMmiiIBAEq3U6UrAADVjkQKAIFIpAAQiEQKAIFIpAAQiEQKAIFIpAAQqCYTqTFmtjFmvTFmszFmlTHmnErXCaUxxnz8pT/bjTFTKl0vlKZa29PU4oJ8Y8wBkl6PomiLMaaHpEWSBkZRtLSyNUMIY0wLSe9KOjmKoucqXR+Eqab2rMkeaRRFy6Io2mL/+v8/XSpYJcRjqKT3JD1f6YogFlXTnjWZSCXJGHO3MeZTSSslrZc0r8JVQrizJM2KanGYlU5V0541ObS3jDFNJB0tqY+kW6Mo2lrZGqFUxphOklZL2j+KotWVrg/CVFt71myPVJKiKNoeRdFiSftIGlvp+iDISEmLq+GHDgWpqvas6UTqaSrukVa7kZJmVroSiE1VtWfNJVJjTDtjzHBjTEtjTBNjTH9JZ0haUOm6oTTGmGMkdZT0aKXrgnDV2J5NK12BCoi0Yxh/j3b8Ilkj6aIoiuZWtFYIcZakx6Io+qjSFUEsqq49a3qyCQDiUHNDewCIG4kUAAKRSAEgEIkUAAKVddbeGMPMVh5RFJlK16FQtGd+tGe61Nee9EgBIBCJFAACkUgBIBCJFAACkUgBIBCJFAACkUgBIBCJFAACkUgBIFAt7keqk046SZJ0zTXXuLJevXq5ePLkyZKkJ554wpUtX77cxZ9++mnSVQRQReiRAkAgEikABCrrDvmV3BShRYsWLl6/fn1WWb7vw7p161zct29fSdIbb7wRZxVtPdjkIkVoz3Rh0xIASAiJFAAC1cys/Q9/+EMX77bbbkW/v0OHDi6+4IILJEmXX365K9uyZUtA7YDq1KdPHxcvXLgwlmsuWrRIkvTss8+6smuvvTaWayeFHikABKqZyabhw4e7eNasWZKkJk2auLJSvg+HHXaYi19++eWA2tVhcqJhw4YNkyQ9/PDDruzSSy+VJN12220FX2fo0KEu7tmzpyTp1FNPdWWdOnWSJLVt27b0yiqd7ZlELzQf20uV6nqqleilMtkEAAkhkQJAoJoZ2vvsxNOECRNc2YoVK1x85513SpL2228/V2YfG/UxtC9/ey5btkyS1KNHD1dm/w1v27bNlb311lsuXr16tSSpffv2ruyAAw5w8U47ZfcnNm3aJElq165dUH3T2J71De1zTRKV4oQTTsj6nHzs2m6/HklgaA8ACamZ5U++Tz75RFLmb7y3337bxfaJpxEjRjR4naOPPtrFcfVIke2OO+5wcbdu3bK+bsyOTkKzZs1cWZcuXXLGCOf3+JLsCfo/n/4GQ7l6qn7P+LrrrpNU3skoeqQAEIhECgCBanKyyfKHgv4w/ic/+YmkzAmNXPz3PPTQQ7HUKY2TE/mcd955LrYTfCeffLIr6969u4vtxND111/vyuzkoK9fv34unj9/viRpr732cmVz5sxxsZ14srd8JOn000+XJD399NPF/K9kqcX2TJod2ucb7idx24HJJgBICIkUAAKlftbezvIee+yxrswO3XfddVdXtu+++xZ9bX+dKUrnt8PYsWOzyuw6UEkaMmSIJGnlypWubOvWrVnXfOSRR7LK/Nl/fx2pdeWVV7o4dEiP5Nhhuj9cz3WL0p/Jtys7kkKPFAACVf1kU8eOHSVJp512mivz13ceeeSRkqTOnTvnqo+Li/k+LFiwQJI0cOBAV5arV1SKWpyc2GOPPVw8ceJESZk9zgceeMDFGzZsKPr6I0eOlCTdfffdrqx58+YuXrt2raTMXurHH39c9OfkUovtWQn5fn7j6pEy2QQACSGRAkCgqp9suuuuuyRJgwYNKttn/vznP5cU33C+1m3evNnF/prSEIceeqiL7b8Rfzj/0UcfuXj8+PGS4hvOo3Gwj4qWAz1SAAhU9T3SnXfeWVJpN5P97dM+++wzF2/fvl1S5tlOL7zwgoufe+65oj8L5WWXuEm5z+iyE4aS9MQTT5SlTohXfZuS2GVRbFoCAFWERAoAgap+aG/XjxWzDvTdd9+VJN1yyy2uzN/V2w7z/V3z/bWpV111laTGf0Rsrbnvvvtc7G96Yj311FMuHjVqVFnqhPjl2rTEF7pDfynokQJAIBIpAASq+qH9vHnzJGU+3vf555+7+JVXXsl4nVT3yGF960C7du3a4Gf6h96hMvzz5u3jwf5Z9bvvvruL7ZrRm2++2ZW9//77SVcRCfE3I8klycPv6kOPFAACVVWPdJdddpEkXXHFFa7M7nru9xI3btxY9LW/8pWvuNjfbg2N0zHHHONifzMSy39yyT4t5a8FRnXJ1wst13HM9aFHCgCBSKQAEKiqhvadOnWSJE2YMMGV2UmDX//6166s0KF9kyZNXHz55Ze7uH///g2+7+233y7o+ohf69atJUljxoxp8HX+mtG4DiZEefkH2uU63M4fwldiOO+jRwoAgUikABCoqob2udjZ9tmzZ7uyYcOGufif//ynpMzD7eyOUaNHj3ZlF198cYOf4687zHWOOspj+vTpkqQBAwZkfe2ZZ55x8QUXXFC2OiFedhhf30y9Hcb7M/WVRo8UAAJVfY/U6tWrl4vnzp3rYrsBSZs2bVxZs2bNJEnt27d3Zfk2PZkzZ46Lly1bFlZZFKV3794uPvHEE7O+/sknn0iSfvazn7myUtYSo3GobzMSq5w73xeKHikABCKRAkCgqhrar1mzRlLd2eeSdNlll2W9rlu3bkGfs3jxYknSpEmTXFm+R9QQL3/TkUsvvdTFLVu2zHrtk08+KUl66aWXkq8YEpFvzag/nK/0mtFc6JECQCBTzM7ywR9mTCwfZieLpLrt8UpZCuEfmGd3zZekLl26SMo8EK9coigq/hS/ComrPXPxd7C/9957s77ub5VoTy+wWyY2JrRnYfzTJnJNNpVyuGUS6mtPeqQAEIhECgCBqmqyyfJ3tj///PMlSSNHjnRl/rCwXbt2We+3+1LecMMNruwvf/mLiysxpEemXGfR+/yz6BvjkB7Fybd2NJf6bgfY23zlnJSiRwoAgUikABCoKmft06zWZ3nthjJvvvmmK/Mf5d22bZsk6ZBDDnFljfmR3Vpvz0L5a0dLWbNdrqNGmLUHgIRU5WQT0uvss8+WlNkL9Y0dO1ZS4+6Fonh+L9I+xVTfBBTb6AFACpFIASAQk02NDJMTO8ycOdPFI0aMcHHPnj0lSatWrUrqo2NFe6YLk00AkBB6pI0MPZh0oT3ThR4pACSERAoAgco6tAeANKJHCgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBSKQAEIhECgCBajKRGmNmG2PWG2M2G2NWGWPOqXSdUDraMz2MMR9/6c92Y8yUStcrHxNFUaXrUHbGmAMkvR5F0RZjTA9JiyQNjKJoaWVrhlLQnulkjGkh6V1JJ0dR9Fyl69OQmuyRRlG0LIqiLfav///TpYJVQgDaM7WGSnpP0vOVrkg+NZlIJckYc7cx5lNJKyWtlzSvwlVCANozlc6SNCuqgmFzTQ7tLWNME0lHS+oj6dYoirZWtkYIQXumhzGmk6TVkvaPomh1peuTT832SCUpiqLtURQtlrSPpLGVrg/C0J6pMlLS4mpIolKNJ1JPU3FPLU1oz+o3UtLMSleiUDWXSI0x7Ywxw40xLY0xTYwx/SWdIWlBpeuG4tGe6WOMOUZSR0mPVrouhaq5e6TGmLaS5kg6SDt+kayRNDmKomkVrRhKQnumjzFmqqTdoij6QaXrUqiaS6QAELeaG9oDQNxIpAAQiEQKAIFIpAAQqGk5P8wYw8xWHlEUmUrXoVC0Z360Z7rU1570SAEgEIkUAAKRSAEgEIkUAAKRSAEgEIkUAAKVdflTJY0cOdLFXbrs2GHtyiuvdGU77VT3O+W1116TJN177705rzVlSqM/iwtAGdEjBYBAZd39qRILfvfdd19J0vPP152ftffee2e9zpi6dbb5viff+MY3JEnLly+PoYaZWMCdLrRnurAgHwASQiIFgECpn2zq3bu3JKlNmzaxXfOyyy6TJI0ePdqVbd3KgZVJadu2rYunTdux8f0pp5zS4Hv8ycO7775bkvTwww+7sueeey7OKqLG0SMFgEAkUgAIlPpZe+sHP6g7R6tVq1YFvefcc891cc+ePV1sZ/ivuOIKV3brrbeGVlESs7zW2LF1x9L379/fxQMHDizo/f7Q/osvvpAkffjhh67Mb9vf//73JdczH9ozXZi1B4CE1EyPtBRdu3Z18YoVK1xse6Tz5s1zZfkmPwpFD2YH/6myUaNGudj2LvPJ1SP1+b3TcePGScqcjIpLLbVnu3btJEkHHXSQKxs8eHDJ1+vVq5eLX3311ayvn3TSSS7+7LPPXOyvGbcmTpwoSXrrrbdKro9EjxQAEkMiBYBAqV9HmiR/faONN2zYUKnqpMqYMWNc/Mknn7i40FtR48ePb/Dr/oTj7NmzJUmff/65K5s7d25Bn4M6t912myTpzDPPzPpaMY9g53rPscceW3A9DjzwwKzPee+99yRJ119/fcHXKQY9UgAIxGRTDnajE/sEk5TZQ7K/KZcsWeLKBg0aJEnauHFj0GfX0uREkgYMGOBi23b5JgSffvppF9v2DJX29vzOd77jYruMzO9Jetd2cSk90mLylH2f/5533nlHktStWzdXtmXLloKv6dWDySYASAKJFAAC1fRk09ChQ118/PHHu3jEiBGS8j8BNXXqVBeHDukR7vDDD3exPyy0G50cccQRrizXnrT+1+1tgD/84Q+x1zNN/NsluYb0jYVd43rUUUe5smeffTa269MjBYBAJFIACJT6of2FF14oKXMGz+5NWt/hd/keQ7zzzjslSTNmzIitniiOv4Z3woQJkjKHmXblhZS7PXOVvfjiiy5mSF8Ye5BkY/fBBx9Iinc476NHCgCBUt8jvf322yXlXofml/k9FFvuP1Hj76h+ww03xF5P1O+EE05wsd3MpGnTun+6nTp1Crr+unXrJGWeeIDCvP/++y5uaLIp30TU2rVrXbx58+as9/g/q927d5ckNWnSpODPyrWRSZzokQJAIBIpAARK5dDe3109xJQpU1zsT0yhvHbbbTcX77fffrFc8+9//7uL7ekJ69evj+XateS8885z8de+9jVJ0iGHHJL1Ojtcl6S77rrLxc8884wkafny5a4s38Y///73vyXlP9DSvx3w2muvNfjaUPRIASBQKnuky5Yti+U6Q4YMcTE90sq57rrrYr/mnnvu6WK/x4vi+L3Hvn37SpJOPfVUV2aXRy1atMiVlXIUtj/Bu9deexX9fn+LxCTQIwWAQCRSAAiU+v1I7RrE7373u/V+TZIOPvhgF+d66sU/snfYsGFxVjFD2vevLIW/GYk9oK5z584NvqeYJ9WuueYaSdKNN95YahXrRXvGY8GCBS7u3bt3g6+160j/+te/urJ+/fpJKm0PUh/7kQJAQkikABAo9UP7hvjr0BYuXOjinj17Nvg+O4ucxKOiDAUbZod1/pnnuUyePNnF+Yb2VrNmzUqvWD1oz9L5s/+zZs1ycb5VFnZob2/ZSPHdtmFoDwAJaZQ90hYtWrj4oIMOcrHdLs1fh/bYY49JkrZu3erK3nzzzaLr5k9eLF68WFLuXdSlum30LrrooqI/Jx96MPHYvn27i+mRFqaxtWd9mwrlYyca+/Tp48ri2j6PHikAJIRECgCBGuUjov46sVw7lffv39/FN910k6S6HbAlafbs2S5uaPhd3+F3PDJYnQYPHlzS+5588smYa4IQl1xyiaTcewQX4p577pGUuTFN0uiRAkAgEikABGqUs/Zf//rXXfzHP/7RxfkeC8zFzuDlm/Ur5fC78ePHF12ffJjlLUzHjh1dPHXqVEnSoYce6srsOeZS/vYcNGiQJOnpp5+Os4qSaM9CHXvssS62j4P6x8nky1P+7k4tW7aMuXZ1mLUHgIQ0yskmf7ds/+mGxx9/XFJxPVPbG8n3G62YG9srVqwo+PORn79ueNKkSZKkc88915XZHqNv2rRpLs613tcfYVj1HWa4cePGImuMuA0cONDF9R1q15BKH41OjxQAApFIASBQo5xsqo/dTGTEiBGuzB5c1qFDh/o+U1L+4bp/Fvbq1aslZR6G5m96kMSkhFWLkxNdu3Z1sX9bxypmIrCh91x66aWu7I477ii6nqWoxfYsRvPmzSVlnjtv9wau71z7XPz32yNPksBkEwAkpKp6pLnYiSd/mZS/DZ7d1KRHjx6u7N57723wmvfff78k6cMPP4ytnoWqxR6Mv1mI3ZjG/lcqrUf6xhtvuNg+6eIfA+xvcpOkWmzPYlx77bWSpKuuuirra8W0uz95SI8UAKoQiRQAAjXKdaTFWLNmjaT8O6aj8fKH2Zs2bSr5Ov7mI0OGDAmqE8rD3lrMdYuxmLXdc+fOjbdiRaJHCgCBSKQAEKjqh/ZIFzs8//73v+/K/HPtrXXr1rl4zJgxkqSlS5cmXDs0JvPnz3ex3UioUuiRAkCgql9HmjasO0wX2rNhdutDfx1427ZtbX1cmZ+nVq5cKUkaMGCAK1u7dm2i9fTqwTpSAEgCiRQAAjG0b2QYCqYL7VmYI4880sV2mO8P7YcPH+5ie6jdhg0bylS7OgztASAh9EgbGXow6UJ7pgs9UgBICIkUAAKVdWgPAGlEjxQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAApFIASAQiRQAAtVkIjXGzDbGrDfGbDbGrDLGnFPpOqF0tGe6VGN7miiKKl2HsjPGHCDp9SiKthhjekhaJGlgFEVLK1szlIL2TJdqbM+a7JFGUbQsiqIt9q///9OlglVCANozXaqxPWsykUqSMeZuY8ynklZKWi9pXoWrhAC0Z7pUW3vW5NDeMsY0kXS0pD6Sbo2iaGtla4QQtGe6VFN71myPVJKiKNoeRdFiSftIGlvp+iAM7Zku1dSeNZ1IPU3VyO/BoCi0Z7o0+vasuURqjGlnjBlujGlpjGlijOkv6QxJCypdNxSP9kyXam3PmrtHaoxpK2mOpIO04xfJGkmToyiaVtGKoSS0Z7pUa3vWXCIFgLjV3NAeAOJGIgWAQCRSAAhEIgWAQE3L+WHGGGa28oiiyFS6DoWiPfOjPdOlvvakRwoAgUikABCIRAoAgUikABCIRAoAgUikABCIRAoAgUikABCIRAoAgcr6ZBNQqKZN6/5pdu/e3cXf+973JEmtW7d2ZYMHD84qmzFjRoPXnzt3rosXLlwoSdq2bVtAjVHL6JECQCASKQAEKusO+WyKkB+bXOxwyy23uPiSSy7J9dkuLvTfcH3veeqppyRJp5xyStH1zIf2LN7ee+8tSTr88MNd2be+9a2s1w0fPtzFbdu2dfHatWslSY8//rgru+666yRJmzZtCqobm5YAQEJIpAAQKPWz9s2aNZMkHXfcca6sX79+kqQJEya4si+++MLF77//viTpnnvucWXvvPOOi3/7299Kkj777LP4KwxJ0n777ZezfMmSJZKkP//5z1mvPemkkxq8Zrt27XKWH3HEEZKkXXbZxZVt2bKl8Moir/3339/FPXr0cPGQIUMk1Q3nJenAAw/MKsvH//nt0KGDJGncuHGurFevXpKkvn37FlPtgtEjBYBAqZls8nsT/nrCO+64Q1Ldb74v1cfFxXwf1qxZI0m68cYbXVm+dYuFYnJiB789p0yZ4uKHHnpIkrRgwYKCr9WxY0dJ0ltvveXKcrW3P3kxZ86cwivbgLS350471fXFRo8eLUm6/PLLs17XqlUrF++5555ZX9+6dauL3333XUnSI4884sr80d9vfvObrPcff/zxLr766qslSfvuu68rsz//Tz75ZD3/J4VhsgkAEkIiBYBAVT/Z1Lx5c0mZw7+zzz470c/s3LmzpMzbBQ888IAk6b///W+in10r/MmeMWPGFP1+O5yXMiemEC9/SG0ndz7++OOs173wwgsu9ieRHn74YUnS/fff78ryTeK2aNFCkvTTn/7UlQ0cONDF7du3lyQNHTrUlYUO6fOhRwoAgap+ssn+1rGTEEXWx8WlfB8++OADF3/zm9+UJL399ttFX8eX9smJJNiJjK5du7qyBx980MVdunSRVH97b9iwQZL01a9+Nfa60Z7x8CcfZ86cKUk6/fTTXdmyZctcfO2110qSHnvssdjrwWQTACSERAoAgap+ssneeC6GHX6//vrrrsy/QV7o5hX+2tHQIT2Kc9FFF7l41KhRkqSePXvmfG2u2zZ+e/3yl7+MuXYIYYfxdqMRSRo5cqSL7QYl06dPd2X+2lX7ZGI50SMFgEAkUgAIVPVD+yeeeEJS5vBt2LBhWa/z15muXLlSkrT77ru7MrsnZSHsjLA/9EB5+etE6xvSf9ns2bNdfMEFF7h48+bN8VUMJbG3Z6S69aH+xjUfffSRi+3RMvPmzStT7fKjRwoAgap+HWkpbA/G3/jC32E7F39d4vnnny8p9xMcoVh3WJg99tjDxXZzi1y7qPvsIXdS5hpEfz1w3GjPhnXq1EmS9Morr7gyO1K0mwNJ0llnneXi559/vky1y8Y6UgBICIkUAALVzNB+0qRJLrbDhL322suV5fo+LF++3MVHHXWUiz/99NMkqmjrwVCwSHaYf9NNN7mysWPHZr3Of0R0/vz5LraTF0mceEB7ZvMfxbW3W7p16+bK7H6ktl2kzDXf27Ztk5Q5AVUuDO0BICGp7JH6m1fYI1n933h2V+98m5b45zT17t3bxf5N8LjRgymd3VJRytw27cQTT5SUuZu7f8aPXRrnPy0VF9ozW/fu3V384osvSpJatmxZ8PvtJjP+kqmXXnrJxevWrQutYr3okQJAQkikABAoNUN7u+ekJD3zzDMutuvU6qmPi/N9HwYNGuTiJHdcZygYD/+AtaVLl0rKPAwt136kxRz/Wyjas2F2Tff48eNdmb1F8+1vf9uV+RPDuaxdu9bFN998syRp6tSpsdXTYmgPAAkhkQJAoKrftMTyZ+Xt4XT1sWdo+7cAWrdu7WJ/zSiqk//Y52GHHSYpc2MaW4bKWrFihSTpvPPOy/qav6lQkyZNXDxgwABJ0oQJE1yZf9vG7k26fv16V8bhdwDQyKWmR+o/heTfuM7FTj6sWrXKlfm903JOwCF5dsf0JNcXIn71PblkD7r0D7z0f+btiQe/+MUvXNmf/vQnSdL27dtjr6dEjxQAgpFIASBQataRFsNOLPn7kR544IENvod1pNkaS3vmM3fuXEmZbcg60mxJtKedBGrTpo0r+9vf/hb3x2ifffZxca5HuHfddVdJdRPNpWIdKQAkhEQKAIFSM2ufz9lnn+3iiy++WFLmcN4f6m3ZskWStGjRIlf28ssvJ1vBFGrVqpWkzP0n//GPfyT2eQcffLCLr776ahfbIX19uz/ZoT/iN27cOEnS4Ycf7sr69OkTy7Xbt2/v4lmzZmV9ffLkyS62e5gmhR4pAARKZY/UP8ju9ttvlySddtpprmznnXdu8P0vvPCCJGngwIEJ1K522EMCL7zwQlcW14TOIYcc4uL+/ftLyuyFNmvWzMV2tOH3Qu+77z4X+0czI162nQ499NCg6/g/s/bgQv+IdTv6keomg6+55hpXlvSkOj1SAAhEIgWAQFU/tLdrQs8880xX5h985h9r0JBf/epXLp44cWJMtattxx13nKTMNYQzZ8508b/+9S9JufeF9Scncm0w4m9oscsuuxRUn2nTprnYH86Hri1E/V577TVJ0jHHHOPKevTo4eKVK1cWdB1/UxN7u87nrwk/44wzJEmbN28urrIB6JECQKCqerLJ9kLsccqS9KMf/UiStP/++xd9vd/97ncu/vGPf+xi+6RLJaTpSRg7KeB/n+u5josL/feY7z3+JjZ2UsLvkZZLmtqzFHaS99FHH3VlM2bMcPE555yT9R47WXzLLbe4Mn/5ol2e6PdMr7/++qyvJ4EnmwAgISRSAAhUVUN7e1693VW7GP5Qz04sTZ8+PaQ6iUjTUHCPPfaQJF155ZWuzD973urdu7eLDzjgAEnSkiVLXFm+TS7shIb/ntdff93F5Zx0+LI0tWcp7FNtr776qitr2rRujnvx4sVZ77FPqHXo0MGV+e+365Pteu9yYmgPAAkhkQJAoKoa2tvZ+mKG5HYzA/+gLP9QrMam1oeCaUN77mDXdkrS7NmzC3rPDTfc4GJ/A5L//Oc/8VWsSAztASAhVdUjbWiy6Y033nCx/5vswQcflJS5YUVjRg8mXWjPdKFHCgAJIZECQKCqGtrXAoaC6UJ7pgtDewBICIkUAAKRSAEgEIkUAAKVdbIJANKIHikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABCKRAkAgEikABPofRHsKFSmTVR4AAAAASUVORK5CYII=\n"
},
"output_type": "display_data",
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "data.batch_size",
"execution_count": 34,
"outputs": [
{
"data": {
"text/plain": "64"
},
"output_type": "execute_result",
"execution_count": 34,
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "fast.ai provides a wonderful callback API that allows us to modify the training process in almost any way, including, among other things, intervening when the loss is about to be calculated. Before we write our callback, let's first summarize what we need from the model so that we know at which stages of the training we want to step in. \n\nIn order to implement CRL, we need to know, for every batch:\n1. Which class is the minority class?\n2. For that minority class, which samples of the same (right) class have the lowest probability prediction from the model (hard-positives)? \n3. For that minority class, which samples of a different (wrong) class had the highest probability prediction (hard-negative)? \n\nAfter reviewing the various callback methods available, there are two places we can step in: `on_batch_begin` and `on_loss_begin`. Let's try an example."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "model = simple_cnn((3, 16, 16, 2))",
"execution_count": 10,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# https://docs.fast.ai/callback.html#callback\nclass PrintLabelsAndOutput(LearnerCallback):\n def __init__(self, learn:Learner):\n super().__init__(learn)\n self.iters = 0 # Manage the number of printouts we get\n def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:\n if self.iters < 3: print('Targets --> ' + str(last_target[:3]))\n def on_loss_begin(self, last_output:Tensor, **kwargs:Any) -> Tensor:\n if self.iters < 3: print('Outputs --> |0| |1|\\n' + str(last_output[:3]))\n self.iters += 1",
"execution_count": 13,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn = Learner(data, model, callback_fns=[PrintLabelsAndOutput])",
"execution_count": 14,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn.fit(1)",
"execution_count": 15,
"outputs": [
{
"data": {
"text/html": "Total time: 00:01 <p><table style='width:300px; margin-bottom:10px'>\n <tr>\n <th>epoch</th>\n <th>train_loss</th>\n <th>valid_loss</th>\n <th>time</th>\n </tr>\n <tr>\n <th>0</th>\n <th>0.066576</th>\n <th>0.055267</th>\n <th>00:01</th>\n </tr>\n</table>\n",
"text/plain": "<IPython.core.display.HTML object>"
},
"output_type": "display_data",
"metadata": {}
},
{
"text": "Targets --> tensor([1, 0, 1], device='cuda:0')\nOutputs --> |0| |1|\ntensor([[ 0.0000, 7.8249],\n [10.2639, 2.9326],\n [ 1.5382, 4.5256]], device='cuda:0', grad_fn=<SliceBackward>)\nTargets --> tensor([0, 1, 1], device='cuda:0')\nOutputs --> |0| |1|\ntensor([[7.9085, 2.0232],\n [1.2495, 5.1679],\n [0.2476, 6.8007]], device='cuda:0', grad_fn=<SliceBackward>)\nTargets --> tensor([1, 1, 0], device='cuda:0')\nOutputs --> |0| |1|\ntensor([[1.7952, 8.2486],\n [2.0123, 4.3107],\n [9.9175, 1.7809]], device='cuda:0', grad_fn=<SliceBackward>)\n",
"name": "stdout",
"output_type": "stream"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "What are we seeing here? First, for our purposes, even though the batch size is 64, we are only taking a look at the first three samples of each batch, up to three batches. \n\nSo \"targets\" refers to the ground truth classes of the first three samples of our mini-batch. For our toy dataset, we only have two possible classes: `0` and `1`, referring to whether the number is a 3 or a 7. \"Outputs\" refers to the output of the model -- that is, the prediction the model has made as to whether the given sample is of class `0` or class `1`. If the first number in a pair is higher, then the model thinks the sample is more likely to be of class `0`; similarly, if the second number is higher, then the model thinks the sample is more likely to be of class `1`. If you run the above cell multiple times, you should see these predictions become more and more \"confident.\"\n\nAt this point, we should have everything we need in order to implement CRL. Let's first sort our batch according to majority and minority classes. Note that our toy dataset is balanced, so we won't necessarily see a benefit here. In fact, our code below may return the same class as minority and majority because of this. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "from collections import Counter",
"execution_count": 11,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "class SortMinorityClass(LearnerCallback):\n def __init__(self, learn:Learner):\n super().__init__(learn)\n self.iters = 0 # Manage the number of printouts we get\n def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:\n if self.iters < 2:\n targets = last_target.cpu()\n target_indices = set(enumerate(targets))\n class_tally = Counter(targets)\n minority_class = min(class_tally, key=class_tally.get) # Find the class with the least num of samples\n minority_indices = {i[0] for i in target_indices if i[1] == minority_class}\n majority_class = max(class_tally, key=class_tally.get)\n majority_indices = {i[0] for i in target_indices if i[1] == majority_class}\n print('Minority Indices: ' + str(minority_indices))\n print('Majority Indices: ' + str(majority_indices))\n self.iters += 1",
"execution_count": 34,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn = Learner(data, model, callback_fns=[SortMinorityClass])",
"execution_count": 35,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn.fit(1)",
"execution_count": 36,
"outputs": [
{
"data": {
"text/html": "Total time: 00:01 <p><table style='width:300px; margin-bottom:10px'>\n <tr>\n <th>epoch</th>\n <th>train_loss</th>\n <th>valid_loss</th>\n <th>time</th>\n </tr>\n <tr>\n <th>0</th>\n <th>0.051920</th>\n <th>0.036756</th>\n <th>00:01</th>\n </tr>\n</table>\n",
"text/plain": "<IPython.core.display.HTML object>"
},
"output_type": "display_data",
"metadata": {}
},
{
"text": "Minority Indices: {0, 1, 2, 3, 5, 7, 10, 11, 13, 15, 17, 19, 20, 21, 22, 23, 26, 28, 30, 34, 35, 38, 44, 49, 50, 51, 52, 53, 54, 56, 58, 59, 60, 62}\nMajority Indices: {0, 1, 2, 3, 5, 7, 10, 11, 13, 15, 17, 19, 20, 21, 22, 23, 26, 28, 30, 34, 35, 38, 44, 49, 50, 51, 52, 53, 54, 56, 58, 59, 60, 62}\nMinority Indices: {0, 3, 4, 5, 7, 11, 12, 14, 15, 18, 20, 22, 23, 26, 32, 35, 36, 39, 42, 44, 46, 48, 51, 52, 53, 54, 57, 58, 60, 63}\nMajority Indices: {0, 3, 4, 5, 7, 11, 12, 14, 15, 18, 20, 22, 23, 26, 32, 35, 36, 39, 42, 44, 46, 48, 51, 52, 53, 54, 57, 58, 60, 63}\n",
"name": "stdout",
"output_type": "stream"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now that we have the indices for every sample sorted by which class they belong to, we can use those indices to split up the predictions from the model and sort those predictions into the worst offenders. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "model = simple_cnn((3, 16, 16, 2))",
"execution_count": 4,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "from collections import Counter",
"execution_count": 5,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "class ClassLevelHardMining(LearnerCallback):\n \"\"\"An implementation of class-level hard sample mining that is intended only for the two-class case.\"\"\"\n def __init__(self, learn:Learner):\n super().__init__(learn)\n self.sample_counts = Counter(learn.data.y.items)\n self.omega = None # Class imbalance measure\n self.minority_class = None\n self.minority_indices = None\n self.majority_class = None\n self.majority_indices = None\n def _make_omega(self, b, a):\n if self.omega is not None:\n return self.omega\n return (self.sample_counts[b] - self.sample_counts[a]) / self.sample_counts[b]\n def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:\n targets = last_target.cpu()\n target_indices = set(enumerate(targets))\n class_tally = Counter(targets.tolist())\n self.minority_class = min(class_tally, key=class_tally.get) # Find the class with the least num of samples\n self.minority_indices = tensor([i[0] for i in target_indices if i[1] == self.minority_class])\n self.majority_class = max(class_tally, key=class_tally.get)\n self.majority_indices = tensor([i[0] for i in target_indices if i[1] == self.majority_class])\n self.omega = self._make_omega(self.majority_class, self.minority_class)\n def on_loss_begin(self, last_output:Tensor, **kwargs:Any) -> Tensor:\n predictions = last_output.cpu()\n # Every minority sample is also treated as an anchor\n minority_predictions = anchors = predictions[self.minority_indices][:, self.minority_class]\n majority_predictions = predictions[self.majority_indices][:, self.majority_class]\n k = len(minority_predictions) if len(minority_predictions) < 25 else 25\n bottom_k_hard_pos = torch.sort(minority_predictions)[0][:k]\n top_k_hard_neg = torch.sort(majority_predictions, descending=True)[0][:k]\n return {'last_output': (anchors, bottom_k_hard_pos, top_k_hard_neg, predictions, self.omega)}",
"execution_count": 37,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "This callback will provide us with a set of anchors, the bottom-$k$ hard-positives, and the top-$k$ hard-negatives. Our loss function will then need to form triplets for every single anchor against all pairs of hard-positives and hard-negatives and then calculate the average sum (class probability) distances across all triplets. "
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "class ClassRectificationLoss(nn.Module):\n def __init__(self, eta=0.01, margin=0.5):\n super(ClassRectificationLoss, self).__init__()\n self.eta = eta # Subparameter of α\n self.margin = margin # Triplet Loss\n self.ce = CrossEntropyFlat()\n def _relative_comparison(self, a, p, n):\n # Cartesian product of all combinations\n num_a, num_p, num_n = len(a), len(p), len(n)\n a = a.view(-1, 1).expand(num_a, num_p**2).reshape(-1)\n p = p.view(-1, 1).expand(num_p, num_n).reshape(-1).repeat(num_a)\n n = n.repeat(num_a * num_p)\n a, p, n = torch.stack([a, p, n]) # Simplify tensor math\n d_p = (a - p)\n d_n = (a - n)\n losses = F.relu(self.margin + d_p - d_n)\n # TODO: Remove those losses == 0, then avg over the remaining triplets\n return losses.sum() / len(a)\n def forward(self, last_output, targets, reduction='none'):\n targets = targets.cpu()\n a, p, n, predictions, omega = last_output\n alpha = (self.eta * omega)\n return (alpha * self._relative_comparison(a, p, n)) + ((1 - alpha) * self.ce(predictions, targets))",
"execution_count": 38,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn = Learner(data, model, loss_func=ClassRectificationLoss(), callback_fns=[ClassLevelHardMining])",
"execution_count": 39,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn.fit_one_cycle(5)",
"execution_count": 40,
"outputs": [
{
"data": {
"text/html": "Total time: 00:13 <p><table style='width:300px; margin-bottom:10px'>\n <tr>\n <th>epoch</th>\n <th>train_loss</th>\n <th>valid_loss</th>\n <th>time</th>\n </tr>\n <tr>\n <th>0</th>\n <th>0.040992</th>\n <th>0.033535</th>\n <th>00:02</th>\n </tr>\n <tr>\n <th>1</th>\n <th>0.037327</th>\n <th>0.028478</th>\n <th>00:02</th>\n </tr>\n <tr>\n <th>2</th>\n <th>0.034604</th>\n <th>0.021936</th>\n <th>00:02</th>\n </tr>\n <tr>\n <th>3</th>\n <th>0.024023</th>\n <th>0.021723</th>\n <th>00:02</th>\n </tr>\n <tr>\n <th>4</th>\n <th>0.019512</th>\n <th>0.020085</th>\n <th>00:02</th>\n </tr>\n</table>\n",
"text/plain": "<IPython.core.display.HTML object>"
},
"output_type": "display_data",
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "learn.show_results(rows=5)",
"execution_count": 41,
"outputs": [
{
"data": {
"text/plain": "<Figure size 1440x1440 with 25 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHkAAATuCAYAAABZD/8jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XuYn+OZOPD7kahT0qhIuusQoc5F6rSqqqWlqKJC0Npu0zZt+bV1PvS0Lm3ZYqUU1U2zpVJrlaC7Ti3qbFNKUEEoJaxTkDifeX9/zOSS1ff5mpl8Z+adZz6f68o1M/ed533vtLlnJrf3O3eqqioAAAAAGNgW6+8CAAAAAFh0hjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAKAYKaWJKaUqpXRbP9exVWcdD/ZnHQDA4GLIAwBkpZQ2Symdn1J6IqX0WkrpsZTS5Sml8f1dW0+llB7sHMBs1abrXd15vYntuB4AQE8Z8gAAtVJKEyLihojYNSLmRcSvI+K6iFgzIj7f4tzifVJgLyvlzwEADB6GPADA30gpLR0R/xYRQyLi7IhYv6qqr1RVtUdErBYRR3T+vrGdT7FUKaV9UkqPRsRlnbkNUkq/Syk9lVJ6MqV0YUpprYXuseDc2M6Pj+z8+FedHy946dX1KaUTUkrPpJQeSSntvdA1VkgpXZZSejGldF1ErPouf64HI2KVzg+vWvAEzjvu9fOU0vMR8b131vTOulNKV0fExztTp3fGj3zHPQ9KKT2eUpqbUjq0q/8fAAB0lyEPAFBni4hYrvP9H1RV9caCRFVVb1ZVdVfNmaMj4tKI+J+U0t9HxDURsV1E/DEibo2Iz0TE1Sml9/Wgli0i4qaIWCEipqSU3tuZOysito2IhyLigYg4/F2udVpEPN/5/nkR8dOIWPjPskVEfKLzun/tQm3TI+KRzvcv77zeHxfKj4mISRHxPxExKiKOTSmt0YXrAgB029D+LgAAaKTRC73/YERESumYWGiIUlVVeseZCVVVXdn5ew+LiGUj4uqqqj7TGbs1Ij4UERMi4hfdqGVeRHwsIt6MiJcjYpmIWDOl9Hi8/RTNp6qqejil9GREHJS7UFVVP0wpfTkihkfEKVVVXd1Z27qdv+X5iNisqqpnOuNHtiqsqqpTUkq7R8SKEXFWVVW/6jy3VedveSsiPlFV1eMppTnRMfQZFxF/6fKfHgCgizzJAwDUmbvQ+yt3vr0+Op5wyblhoffHdr69e6HY7M63q0S9IZn43VVVvVJV1esR8WJnbFh0DFYiIl6uqurhzvfvbVFfV9y5YMBTJ6WUqzHn8aqqHu98f8F1h/WoMgCAd2HIAwDUuSE6nqCJiPhOSilVVXVRRPxr7kBVVa8u9OGDnW/XXii24OfxzOl8+1Ln2wUvvVovc+k3Fnq/Wuj9BS+TWiqltGAQtWauvoW82fm27vugV9/x8YKhUqsaW10vVzsAQNsZ8gAAf6Oqqpci4hvR8XKjL0XErSmlKRHx4y5e4syIeDYitk4p/XdK6XcRsWFEPBEdP8cmouPn9EREnJJSmhoRu3Szxv+NiGs7P7wspTQtIr7ZhaMLnvr5YUrpxIUGRHUW1PjplNLk6Pgh1Lnr7d95vXFdqAEAoO0MeQCAWlVVnR0dP/Pmouh4ydaXImL9iPh9RHz1Xc4+GhFbR8emrS0iYpOIuDgitq6qasETQt+KiDui4+f0rBQRp/egzL0j4oroeAnYmhHxky6cOTIi7ouIzSNi/4h4f4s/xxURcVJ0/CygXSPilJrfNjki/hwR63Zezw9WBgD6RaoqTw4DAAAADHSe5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgCgkVJKv0opVSmlEzs/ntj58dWLcM0jO6/xq3bVCQDQFIY8AEC3pJQe7ByULPj1VErp9ymlTXr51ndFxE8jYnpXfvNC9Y1dKPzHzmtc1vbqAAD62dD+LgAAGLAuiogHIuLjEfGpiNg0pbR2VVVzF/5NKaXFq6p6fVFvVlXVTRFx0yJe43cR8btFrQUAoIk8yQMA9NQvq6raLyI+2fnx+yLicws9QbNPSunR6HxqJqW0Xkrp4pTS3JTSkyml81JKYxZcLKX00ZTSHSmlF1NK0yJiyYVvVvdyrZTSh1NKl3Ve74WU0h9TSkunlKqFjj7QeW6rupdrpZR2TSn9KaX0fEppTkrpZymlZTtzYxf683w5pfRQSml+SumEhc5vlFK6LqX0XGcNs1JK+7brf2QAgK7yJA8A0GMppcWi40meBW5e6P2jI+K3EfF4SunvIuLaiBgWHU8AvScixkfEuimlD0XEUhFxYUQsGxFXRsSo6Hg6qNW9PxgRV0fEEhFxXUT8pbOW90THS7L27/ytp0fEcxHxvzXX2CEizo+I1yLi7IjYOCL+X0R8ICK2f8dvP7LzPntFxAEppYuqqvpDRJwUEVtExHkRMT8i1u68DgBAnzLkAQB66oJ3fHxhRDy20McTqqq6MiIipXRodDzpc3dEPNSZfzI6BiJbR8Ty0THguS8itqmqqkop3RIRG7W4/77RMeD576qqdum8z5CIqKqqOiCltGDI88Oqqh7szL/zGt/qfPsvVVX9IKW0fOefYbuU0prRMfxZYLeqqv6UUlopIj4WERtGxB8iYvHO/CXR8XKyeyLirRZ1AwD0CkMeAKCnLoqOoczTEXFLdPysm1UWyt+w0PtjO9+u0/lrYatHxDKd7/+lqqoFL7W6N1oPeVbtfPvHBYGqqt7sYu3vrOvuzvNPpZSeioi/i44/y18W+r23dr59pvPtsM63B0XEqRHx7xGRIuKFiDgiIk4IAIA+5GfyAAA99cuqqg6squqoqqouXWg4ExERVVW9utCHD3a+Pb+qqrTgV0T8fUT8MiIe6cyvkd5+3GbNd7n/A51vN1sQSCktttD5BU/TtPp+Z0Fda3eeHxkdTxVFRMx5x5/njQXvvuMaN1dVNS46nlTaKjqe7DkmpeQ/pgEAfco3HwBAX/iPiPhuRIxPKf0+OoYrH4iOn6GzRkRcHBHPRsdTPVeklF6LjpdDtfJvETEpInbp/GHM90bElhGxeXQ8bfNwdDyNc0pK6d6I+F7NNX4WETtExHdTSqtFx8/SGRoRl1dVde871q/nXNj5MrH7I2JEdLyE7OmI6O5TRQAAi8STPABAr6uq6tHoGOhcFBEfioh/jIgVo2PI8lRVVfMjYueIuDM6hjTPRscPMm51zVnR8eTMFRGxXkTs3Xluwc/ROTw6ftjy9tHxQ5iXqrnGxRGxR+d9d4+OIc2UiNizG3+8qyNihc777xgRf4qIPd/5ZBMAQG9Lvv8AAAAAGPg8yQMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMhTsJTSmSmlx1JKz6WU7k0pTervmgC9CU2lN6F59CU0k95srlRVVX/XQC9JKX0wIu6rqurVlNLaEXF1ROxYVdUt/VsZDG56E5pJb0Lz6EtoJr3ZXJ7kKVhVVXdWVfXqgg87f32gH0sCQm9CU+lNaB59Cc2kN5vLkKdwKaVTU0ovRcTsiHgsIi7p55KA0JvQVHoTmkdfQjPpzWbycq1BIKU0JCI2j4itIuLYqqpe79+KgAi9CU2lN6F59CU0k95sHk/yDAJVVb1ZVdX1EbFSROzb3/UAHfQmNJPehObRl9BMerN5DHkGl6HhdZLQRHoTmklvQvPoS2gmvdkQhjyFSimNTintlVIallIaklLaLiI+FxFX9ndtMJjpTWgmvQnNoy+hmfRms/mZPIVKKY2KiOkRMS46hnlzIuKkqqqm9mthMMjpTWgmvQnNoy+hmfRmsxnyAAAAABTAy7UAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8hUopvfCOX2+mlE7u77qAiJTSmSmlx1JKz6WU7k0pTervmgC9CU2kL6GZ9GZzWaE+CKSUlomIJyLi01VVXdvf9cBgl1L6YETcV1XVqymltSPi6ojYsaqqW/q3Mhjc9CY0j76EZtKbzeVJnsFh94iYGxHX9XchQERVVXdWVfXqgg87f32gH0sCQm9CE+lLaCa92VyGPIPDFyNiWuWxLWiMlNKpKaWXImJ2RDwWEZf0c0lA6E1oIn0JzaQ3m8nLtQqXUhoTEQ9ExOpVVT3Q3/UAb0spDYmIzSNiq4g4tqqq1/u3IiBCb0IT6UtoJr3ZPJ7kKd8/RcT1BjzQPFVVvVlV1fURsVJE7Nvf9QAd9CY0j76EZtKbzWPIU75/iogz+rsIoKWh4TXM0ER6E5pHX0Iz6c2GMOQpWErpIxGxYkSc29+1AB1SSqNTSnullIallIaklLaLiM9FxJX9XRsMZnoTmkdfQjPpzWbzM3kKllKaEhFLV1X1hf6uBeiQUhoVEdMjYlx0DNrnRMRJVVVN7dfCYJDTm9A8+hKaSW82myEPAAAAQAG8XAsAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKMLQvb5ZSssqLQaOqqtTfNXSV3mQw0ZvQTHoTmklvQjPletOTPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFGNrfBQAAAN03ZMiQ2vjEiROzZ9ZYY41s7ktf+lJtfNSoUd2qa1GklLK5v/zlL9nc+eefXxufPHly9syTTz7Z9cKgH6211lq18cMOOyx7Zvjw4dncoYcems39/d//fW38gQceyJ554oknsjn6nid5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFSFVV9d3NUuq7m0E/q6oqvwO0YfTm21qtm7z44ouzuREjRtTG58yZkz1z7LHHZnNvvPFGbfzGG2/MnqFr9GbfWGqppbK5bbbZpja+ww47ZM/ss88+i1xTV+20007ZXKvPAywavdl9q6++em38nnvu6eNKmmvWrFnZ3Cc/+cls7qmnnuqNcgYkvdn/Lrjggtr4Lrvskj3z3HPPZXOvv/56Njdy5Mja+KOPPpo90+rr5q233prNsWhyvelJHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiA7VrQS2wiGJiGDBmSzR1//PHZ3Ny5c2vjr776avbM97///Wzur3/9a2184sSJ2TOtNojwNr3ZPq220U2bNi2b23nnnWvjKeX/r3nooYeyufnz52dzOaNGjcrmlltuuWxu+vTptfHTTz89e+aqq67qemGDmN7svmHDhtXGzzjjjOyZrbfeOpu79957a+MzZ87MnrniiiuyuU984hPZ3CabbJLN5Sy77LLZ3BprrNHt6/385z/P5r75zW92+3ql0pt9473vfW82d+edd9bGV1xxxd4qp1tazRRyn3Ouvfba3ipn0LBdCwAAAKBghjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQACvUe1GrtZFf/OIXs7mPfvSj2dyqq65aG7/ooouyZ1qtln3sscdq4//1X/+VPTNjxoxsjrdZN8m72WKLLbK53FrJxx9/PHtm7bXXzuaef/75rhdWOL3ZPr/4xS+yua985SvZ3Jw5c2rjX//617NnZs2alc3lvpa1Mnbs2Gxu0qRJ2dzEiRNr463q23777bta1qCmN9un1Srm3PeSERG33357b5TTNjvvvHM2d8EFF3T7eq+88ko2t84662RzDz30ULfvNZDpzb7R6mtP7uvtSy+9lD3TKtfKkksuWRsfNmxYj653wgkn1MYPPvjgHl2Pt1mhDgAAAFAwQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogBXqbbDffvvVxv/1X/81e2bo0KHZXEr5LYXt/v8rd6/XX389e+aRRx7J5j73uc9lczfeeGPXCyuAdZO8mxVXXDGb+9nPflYbX2ONNbJnWq1kf+aZZ7peWOH0ZvtsvfXW2dz06dOzuRkzZtTGP/OZzyxyTb1t2rRptfFx48Zlz2y11VbZ3Pz58xe1pGLoTd5Nu1eov/zyy9nc2LFjs7mnnnqq2/cayPRm3xgxYkQ2N3v27Nr4t7/97eyZM844o0d1jBkzpjZ+9dVXZ8+06pd58+bVxlt93Wz1703eZoU6AAAAQMEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFCC/4on/o9VPOz/ssMNq4//2b/+WPfPLX/5ykWta2JJLLpnN7bbbbtncCiusUBvfbrvtsmdWWWWVbO7kk0/O5v7hH/4hm4PBaOTIkdnckCFDauOTJ0/OnrFBi7521VVXZXNrrrlmNtdqw+RAtd5662VzK6+8cjZnuxb0nzfffDObG2wbtOh/zz77bDa37rrr1safe+65ttfx0EMP1cb32muv7Jnrrrsum1tuueVq41//+tezZ4444ohsjnfnSR4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAHK22HaS77zne9kc//4j/9YG7/66qt7qZruuemmm7p9Zty4cdnczJkzs7mNN9642/eCwWqNNdbI5pZccsnaeG6tJTTN008/3d8l9IqUUrfiQPeMHTs2m5s8eXJb73XCCSe09XrQW+bPn9/fJbSs4ZVXXsnmFl988dr4Uksttcg1Uc+TPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAlih3kXf/va3+7uExmi1JvaGG27ow0qg/RZbLD/73mOPPbK5MWPGdPtejzzySDa37bbbdvt6QHvk1r1GRIwYMaI2fscdd2TPPPzww4tcEzTVsGHDepRbfvnla+O//e1vs2dWXXXVrhfWBUsssUQ2t/TSS2dzL730UlvrgIFgwoQJ2dzw4cP7sBLejSd5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFsEKdWp/85CezuaqqsrnjjjuuN8qBPrPzzjtnc//xH//R1nvNnj07m7v11ltr43fddVdbawD+1uWXX57NbbnllrXxqVOnZs/Mnz9/kWuC/rTGGmtkc2eeeWY2t8kmm/RGOW1z2GGHZXNbbbVVNnfttdfWxn/0ox9lz7zwwgtdrgt627LLLpvN/fu//3ttfPz48W2t4d57723r9XibJ3kAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAKnVpqS23yylvrtZHxo+fHhtfJlllsmeabVp49VXX13kmrpqxIgRtfF77rkne6ZVfeutt1429/zzz3e9sAJUVZX6u4auKrU3cyZMmJDNtdqgNWTIkG7f65lnnsnm3ve+92VzL730Um382GOPzZ659NJLs7ndd989m/v2t7+dzZVIbxLRupcOPfTQbC73fdP222+fPdNqWxdv05v9b9SoUbXxP//5z9kzo0eP7q1yBpwbb7wxm/ve976XzV111VW9UU7b6M3mWmGFFbK5H/7wh9ncF7/4xWyuJ9/v9sTtt9+ezU2fPj2bO/3007O5Rx99dJFqGmhyvelJHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAYb2dwEluOaaa2rj48aNy56ZOXNmNjdv3rxsLrdO7pVXXsmeaeWYY46pjedWaEZEfOMb38jmBtuadAam1VZbLZt74YUXsrk//elP2dz5559fG//DH/6QPdOTtbOtPj9ce+212dzIkSOzudNOO602fu+993a9MOhHY8eOzeaOOOKI2vgXvvCFttZw9tlnZ3OvvfZaNrfBBhtkc08++eQi1QQD0U033ZTN3XDDDdncBRdcUBtv9X3re9/73mxuhx12yOZyNttss2zuRz/6UTa33Xbb1cZffPHFbtfA4LLPPvtkc1/+8pfbeq+q6tl2+pRqt3y3/Ldyq9yXvvSlbG7bbbetjT/wwAPZMyXyJA8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACpp6vQenSzlPruZn3o5JNPro23WtnYFLl1zK3W8eXWuPN/VVVVvy+wgUrtzcHmvvvuy+YeffTRbp9r9+rNptCbA9P++++fzZ1wwgltvdfcuXOzudz3TUsssUT2zLLLLtujOiZPnlwb/+///u/smeuuu65H92oCvdlcub+LERHf/OY3s7lbb72129f83e9+lz3z/PPPZ3M9sdhi+f/effjhh2dzRx11VFvrOPbYY2vj3/3ud9t6n57Sm8215ZZbZnNHH310j6550UUX1cavv/76Hl3vK1/5Sm18r732yp5ZaqmlenSvhx9+uDa+3nrrZc+0+/NKX8r1pid5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoAC2a7XB4osvXhtfbrnlsmdWXXXVbG6LLbbI5nIbOg488MDsmVY/nfwXv/hFbXzffffNnqFrbCKgr7XaCPT0009nc7nPA+3eWNQUenNgarXdZ9KkSdlcbkvIpZdemj1z5plndr2wTiussEI294lPfCKbO+KII7K51VdfvTZ+++23Z89suOGG2VzT6c2BafTo0dncCy+8kM299NJLvVFO24wZMyabe+CBB9p6r6lTp9bGW2277Ut6k96wyy67ZHO5jXMREWuuuWa379Xq89RTTz3V7es1he1aAAAAAAUz5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACWKFeiM022yybu+qqq7K5119/vTa+2mqrZc+0WsXM26ybZFEsscQStfFvfetb2TPf//73s7nhw4dnc9ttt11t/IorrsieGcj05sDU6u/wGmuskc3NnDmzN8ppm7Fjx2Zzxx13XG18o402yp5p9f1A079+602apC9XqE+cOLE2/utf/7qt9+kpvUlf22effbK5U089tdvXs0IdAAAAgAHHkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKMLS/C6A9brzxxmzuyCOPzOaOOeaY2nhupXJExFlnndXluoC8pZdeOpvbfffda+PHHnts9sxrr72WzbVaRVnqqnTK8vzzz2dzTV+T3sqDDz6Yzc2ZM6c2vttuu2XPtFqvfvnll3e5Lga2r371q9ncuHHjsrm77767Nj5lypTsmTfeeKPrhTXMYovl/3v3+PHj23qvl19+OZu79tpr23ovGOieeOKJtl5vrbXWyuYG8gr1HE/yAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYIX6IHDcccdlc7kV6oceemj2jBXq0B4bbrhhNnf66ad3+3q///3vs7mpU6d2+3oANNsGG2xQG//+97+fPbPSSit1+z7bbLNNNnfFFVdkcz/72c+6fa92Gz16dDa39957Z3PHH398W+v41a9+lc3NmTOnrfeCgeDTn/50Ntfuzx333HNPW6/XdJ7kAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgALYrkWtqqr6uwQowvvf//5s7vrrr8/mHn744dr4aaedlj2T25YHDExbbbVVbfyOO+7Inrn88st7qRqaKPc1picbtFrZeeeds7mddtopm1tmmWWyuZtvvrnbday++urZ3K677lobX2WVVbJn1lprrW7X0Mrtt9+ezZ188sltvReDy7Rp02rjrbbHnX/++dnclVde2e0aRo4cmc2NHz8+m/vyl79cG3/f+96XPbPYYj17FuWPf/xjbfzFF1/s0fUGKk/yAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYIU6QBtst912tfEjjzwye2bWrFnZXG4V7H333detuoBmO+WUU7K5cePG1ca/973v9VY5DDDXXXddbXzKlCnZM1//+tfbWkNKKZv78Y9/3NZ7NcVNN91UG993332zZ+65557eKodBYPjw4bXxT33qU9kzrXID2bx587K5nXbaqTb+8ssv91Y5jeRJHgAAAIACGPIAAAAAFMCQBwAAAKAAhjykYfZcAAAgAElEQVQAAAAABTDkAQAAACiAIQ8AAABAAVJVVX13s5T67mZ0yVtvvVUbv+2227JnNtpoo94qpyhVVeV3ijaM3uyaVVZZJZv705/+VBvfb7/9smfOPvvsRa6J7tObA9P666+fzb3nPe/J5m655ZbeKOdvbLzxxtncgQcemM3ttttu2dztt99eG//whz/c9cIGEL3ZPksuuWQ2d8IJJ2RzX/va13qjnMZ64403srmTTjopm/vBD35QG3/hhRcWuaYm0pv9b/XVV6+NX3755dkzrb5vbYLcv0MjIu68885sbpdddsnmHnzwwUUpacDJ9aYneQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAtmsNAocffng2d8wxx9TG99577+yZs846a5FrGgxsIuh/m266aW18qaWWyp454IADsrnf/OY32Vxuu0+rDVqvv/56Nkfv0ZsD07Rp07K5Vps2HnjggW7fK6X8X5Hc902rrrpq9sywYcOyuenTp2dzZ5xxRm38kksuyZ4ZyPRm31h88cWzudzf1X333Td7ZsSIEdncIYcc0vXCeskpp5ySzf3+97/P5krts57Qm8210korZXOTJk3K5tZdd91sbvfdd6+N33DDDdkzd999dzZ37bXX1sZnz56dPXPzzTdnc7zNdi0AAACAghnyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAGsUC/E6NGjs7k5c+Zkc7fddlttfJtttsmeefHFF7te2CBm3WT/O++882rj8+bNy56ZNWtWNvfrX/86m2t1TZpFbw5Mq6++ejZ32WWXZXOrrLJKt+/VkxXqrb7Wnnvuudncj370o2zuhRdeyOZKpDehmfQmNJMV6gAAAAAFM+QBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAlih3os++MEPZnOf/OQns7lzzjknm9too41q49/5zneyZzbccMNs7uMf/3ht/JZbbsmeoWusm4Rm0pvQTHoTmklvQjNZoQ4AAABQMEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAV6v1k9913z+Z22mmnbC63Qv3OO+/MnjnqqKOyuVmzZmVzLBrrJqGZ9CY0k96EZtKb0ExWqAMAAAAUzJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABbNeCXmITATST3oRm0pvQTHoTmsl2LQAAAICCGfIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAfp0hToAAAAAvcOTPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkKdgKaUzU0qPpZSeSyndm1Ka1N81AXoTmkpvQvPoS2gmvdlcqaqq/q6BXpJS+mBE3FdV1asppbUj4uqI2LGqqlv6tzIY3PQmNJPehObRl9BMerO5PMlTsKqq7qyq6tUFH3b++kA/lgSE3oSm0pvQPPoSmklvNpchT+FSSqemlF6KiNkR8VhEXNLPJQGhN6Gp9CY0j76EZtKbzeTlWoNASmlIRGweEVtFxLFVVb3evxUBEXoTmkpvQvPoS2gmvdk8nuQZBKqqerOqqusjYqWI2Le/6wE66E1oJr0JzaMvoZn0ZvMY8gwuQ8PrJKGJ9CY0k96E5tGX0Ex6syEMeQqVUhqdUtorpTQspTQkpbRdRHwuIq7s79pgMNOb0Ex6E5pHX0Iz6c1m8zN5CpVSGhUR0yNiXHQM8+ZExElVVU3t18JgkNOb0Ex6E5pHX0Iz6c1mM+QBAAAAKICXawEAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkKllI6M6X0WErpuZTSvSmlSf1dE6A3oan0JjSPvoRm0pvNZYV6wVJKH4yI+6qqejWltHZEXB0RO1ZVdUv/VgaDm96EZtKb0Dz6EppJbzaXJ3kKVlXVnVVVvbrgw85fH+jHkoDQm9BUehOaR19CM+nN5jLkKVxK6dSU0ksRMTsiHouIS/q5JCD0JjSV3oTm0ZfQTHqzmbxcaxBIKQ2JiM0jYquIOLaqqtf7tyIgQm9CU+lNaB59Cc2kN5vHkzyDQFVVb1ZVdX1ErBQR+/Z3PUAHvQnNpDehefQlNJPebB5DnsFlaHidJDSR3oRm0pvQPPoSmklvNoQhT6FSSqNTSnullIallIaklLaLiM9FxJX9XRsMZnoTmklvQvPoS2gmvdlsfiZPoVJKoyJiekSMi45h3pyIOKmqqqn9WhgMcnoTmklvQvPoS2gmvdlshjwAAAAABfByLQAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACjA0L68WUrJKi8GjaqqUn/X0FV6k8FEb0Iz6U1oJr0JzZTrTU/yAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABRgaH8XAAAAtM/IkSOzufHjx/dZHRtvvHE297Wvfa02nlLKnqmqKps75ZRTauN33HFH9kwrf/jDH7K5v/71rz26JkBf8CQPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAqdUqwrbfLKW+u9kAtvnmm2dzBx54YG18woQJba1hxowZ2dyJJ56YzZ1zzjltrWMgq6oqvwO0YQZbb44YMSKbmz9/fo+umVv52upz7I033pjNbbbZZrXx448/PnvmrbfeyubOPvvsbO62227L5kqkN+kNW2+9dTa33HLLtfVeN9xwQzb3+OOPt/VefUlvts+4ceOyuZkzZ/ZhJXn/+7//WxtfaaWV+riSevfff382l1vXftJJJ/VWOf1Kb0Iz5XrTkzwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQANu1etEee+yRzR1wwAHZXKvtWu328MMP18ZXXnnlHl1vzJgx3b5XqWwiaK5//ud/zuaOPPLIviukD/3mN7/J5g499NDa+COPPNJb5fQrvVmebbbZJpvbbbfdauMrrLBC9swOO+zQ7RqGDBmSzeW27/XUm2++mc1dc8012dwXvvCF2vhjjz22yDW1g95sn+WXXz6bO/fcc7O5j33sY22t47DDDsvmLr300tp4q011rXzta1+rja+33no9ul4rt956a23805/+dPbM3Llz215HX9Gb5VlllVWyudznj69+9as9uleuN88///zsmbPOOiuba3VusLFdCwAAAKBghjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQACvUu6jVSvEJEybUxidPntyje7VaNX7iiSfWxn/yk5/06F45rf68Dz30UDa35557ZnPnnHPOItU00Fg32VyTJk3K5qZMmdKHlTTDLbfcUhtvtcb2xRdf7K1yep3ebK4tt9wym/vpT3+azY0bNy6ba/f68oFs2rRptfGJEyf2bSEZerNvDB06NJsbMmRIbfzQQw/NnrntttuyuYsvvjiba/e/QRZffPHa+GKL5f+b9j333JPNtfpe+K677qqNb7vtttkzjz/+eDbXdHqzuXLrySMidt1112xuo402yuZGjhxZG2/19bRVP0+dOrU23qq+Vv8e3nTTTbO5wcYKdQAAAICCGfIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAfI7FAehnq4Nz5kxY0Y2N3369Gyu3evQgf/rtNNOy+buv//+bG6HHXbI5u69995uX6+Vb3zjG7XxpZdeOntm3XXXzeZafX7beOONa+M/+MEPsmcOOeSQbA566te//nU2N2bMmD6ro9XX75tvvrk2/uc//7m3yvkb++23Xza3/vrrZ3P/9E//VBtvygp1+sYbb7zR7dxRRx3VW+W0TW6F8ze/+c3smdGjR/foXpdcckltfCCvSaf7xo8fn83tv//+tfELLrgge2adddbJ5tZee+3a+JZbbpk902qt+cyZM7O5n/70p7Xx2bNnZ8+cf/752VzOJpts0u0zdI0neQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAtmstZPPNN+/RuXPPPbc2fvDBB2fPPPzwwz26V1+ZPHlyf5cAveatt97K5q666qoe5dqtJ/dqtZUhtwkkIr+1aIsttsieGT58eDb3/PPPZ3PQyuKLL57NnXfeedncFVdckc3NmjWrNn7XXXdlz7T6O9xqM1G75TaPjBgxokfXmzt37qKUA422/PLL18aPO+64Hl3v5ZdfzuZafU2lLK2+tzrjjDOyudw21FbfW6WUsrncpqwpU6Zkz7Ta5HXZZZdlc+02atSo2niuZyMipk6d2lvlDAqe5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFCDl1rH1ys1S6rubtdnKK6+czTV9HXor55xzTm18woQJ2TMzZszI5j7ykY8sck2lqKoqvwexYQZyb/K2FVdcMZubPXt2Npdb89nKmmuumc3df//93b5eX9KbzbXnnntmc62+9jz00EO9UU6f2HXXXbO5E088sTbe6nuSVt/Xbb/99rXxyy+/PHumL+lNIiLWX3/9bO7iiy/O5t773vfWxocPH54988QTT2Rzn//857O5q6++Opsrkd6sN378+Gyu1XrwnOuuuy6bu/vuu7t9vb6UW5Meke+zJ598Mnvm/e9//yLXNBjketOTPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAlihXohW61QPOOCAbO6ggw7q9r3GjBmTzQ3kdfLtZt0kvWGHHXbI5s4999xsbqmllur2ve68885sbquttsrm5s2b1+179SW9yaJYYoklauMbbbRR9kxudXlExGGHHdbte82ePTt75vDDD8/mLrzwwmyuCfRmeVp9z3jggQfWxnfaaafsmVVXXTWbmzt3bm18+vTp2TOnn356Njdz5sxsbrDRm0S0XpN+ySWXZHO5r4+tvqe97LLLul7YIGaFOgAAAEDBDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAEP7uwC6J7cq/YYbbuj2mYj8yvM999yz22eA7hk2bFg29/GPf7w2/tvf/jZ7ZrHFeja3v++++2rj2267bfZM09ekw6LYbLPNsrmpU6fWxtdbb73eKudvHHzwwdncpZde2md1MHi0Wp283377ZXOt/q4uscQSi1TTO1100UW18W9961ttvQ+U7mMf+1htfP/998+eya1Jj4g44ogjauPWpPceT/IAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAWzXaqCDDjoom5s8eXK3rzdjxoxsLrdFywYtaI9WG6pabR3Zcssta+M93aB16623ZnPf/e53a+NPPPFEj+4FTbLWWmvVxv/zP/8ze2adddbJ5tq9EagncptPIiL+53/+J5t79tlne6McCjJhwoTa+Kmnnpo9s9xyy/VWOd0yceLE2vhnPvOZHl3v5ZdfzuYOOOCA2viVV16ZPfPCCy/0qA7oDePHj8/mcv/eHDNmTPbM0Ucf3e1cqxpafR3+7Gc/m82llGrjVVVlz/z4xz/O5s4///xsrsk8yQMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAV6r2o1brzVmvSe6LVmvTp06dncyuttFJt3Ap16J511123Nt5qTfO8efOyuSWXXLI2/sorr2TP7LnnntlcqxXqjzzySDYHA0Grr6nf+c53auMjR47s0b3uv//+2vjtt9+ePdPq6/CIESOyua9+9au18cMPPzx7ptUK9QsvvDCbg4j8157eWJM+f/782vhrr73Wo+sttlj9f7sePXp0j67XygUXXFAbv+qqq7JnTj755Gwud+65557rXmGwkFGjRmVzRx11VDaXW5X+9NNPZ8+06rO33nqrNt5qrXluFXpExJNPPpnN5f4Nu/baa2fP5L5PiLBCHQAAAIB+ZMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABbBCvQ1WXnnl2ni716RH5NfCbb755tkzrXLdvU9ExLnnnpvNHXzwwd2+F5Qgt172fe97X/ZMq9xf//rX2vjPf/7z7JmLLroom4OSzZ07N5sbNmxYbbzVWtQTTjghm5s5c2Zt/OWXX86e6am11lqrNr7RRhu1/V4QEXHHHXfUxlutW+6padOm1cbvv//+Hl0v1+uHHnpo9sxqq62WzX3+85/vdg1bb711j3I33nhjbfyzn/1s9kyrz3sQEbHrrrtmc7mvLxH51eYjR47Mnpk0aVI2N2XKlNr47Nmzs2euu+66bO6pp57K5nJfi2+66absmRJ5kgcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKkHI/PbtXbpZS392sAXJbt95Nq81W7ZbbvNVqI9fkyZOzuRkzZmRzH/nIR7peWAGqqkr9XUNXDbbe7A0f/ehHa+PXXHNNj6534okn1sZtsFt0erM8Sy21VDY3ZsyY2vg999zTW+V0yzbbbJPNnXfeebXx4cOHZ8/ssssu2dyFF17Y9cL6gd6kr/Xkc0dExG9/+9va+JprrrnINS3sQx/6UDaX24TWG/TmwLTOOutkc1tuuWU2d/fdd9fGW228aortt9++Nn7xxRdnz1x22WXZ3A477LDINfWmXG96kgcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUAAr1Gmrc845J5v78Ic/XBtvtaJyILNucnB5z3veUxu/9tprs2c23XTTbC53rtW65TfffDOb4216s97yyy/f7TPz5s3L5t56661FKacoPVmTHpFflT5jxozsmR133DGbe+aZZ7K5JtCbDBSHHHJIbfzYY49t632sUO8+vVm+8ePHZ3M///nPa+MjR47Mnvm7v/u7bO6pp57qemH9wAp1AAAAgIIZ8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABhvZ3AZRljz32yOYeeuih2vhBBx2UPfOTn/xkkWuCvvDaa6/VxidOnJg9c80112RzH/vYx2rjhx9+ePbMv/zLv2RzEBGxwQYbZHOXX355Nvfcc8/Vxlut933xxRe7XtgAMnbs2Nr4ySefnD2T6+eI/Jr0iIhnn322Nj5p0qTsmaavSYeBYskll8zmWvV0T8ycObM23vT1zdBblllmmWxu77337va5CRMmZM+U2Gee5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMAKdfrMIYccUhs//vjjs2esUGege8973pPNDR3a/U/Biy1mNk/PXXHFFdnc7Nmzs7nc6tGBvCZ95MiR2VyrVavHHXdcbXzYsGE9quPmm2/O5g466KDa+N13392jezF4tPr7OGLEiGzukUce6Y1y+tUqq6ySzX3605/O5g488MBs7gMf+EC365g1a1Y2t+OOO9bG586d2+37QAmmTZuWze2yyy7Z3FlnnVUbv+CCCxa5poHEvxYAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgALZr0e9WXnnl/i4BumTMmDHZ3AYbbFAbz22Vi4hYdtllu12DTRssilGjRmVzrTYdPvHEE71Rzt9o1RM93Yb1qU99qja+6aabZs8ss8wy2VzOK6+8ks2dd9552dx+++2Xzc2fP7/bdUBExNFHH53NbbbZZtlcbqPNaaedtsg1tcO4ceOyuW984xu18fXWW69H12vljTfe6FY8IuKYY47J5nxtp2S5r6k33XRT9sw666yTzU2ZMiWb23fffbteWME8yQMAAABQAEMeAAAAgAIY8vD/2bvTKLuqan/Ya5kQukjfgwQEJLSiMQLCJSA9KF0AQWyQIL0oAaJRBASkE6S5CkKGICqI0ooSJASCKNIIaoQg3ZWeKITGQAIBkv3/gL7e+7rnIVWpOmfXqucZIx8yf2OdvcSaVZU5TtUEAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABQgV1XVvofl3L6HtVG0Avypp55q802a7Wc/+1mXz+y55569cJP2qKoqd/oOc6udvbnCCiuE2UknnVRbP+ecc8IzkydP7vId3vWueL694447htl5550XZq3+d3XH2WefXVs/+uijwzNz5szp0TuUqj/3ZquPkeeffz7M/vSnP3X5WXfccUeYbbzxxrX1zTbbLDwz//zzd/kO3TVjxowwu/rqq2vrJ598cnjmoYcemuc79Qf9uTd7WquPxy9/+cttvEnfNWvWrDDbe++9a+s///nPe+s6HaU3mRdXXXVVbX3nnXcOz1x77bVhNnr06DB78skn5/5iBYh60zt5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFsEK9B0Rr3L70pS+FZ6644oowO/LII+f5Tr0pWhmfUkpnnnlmmO2xxx619ZVXXjk805fX0Fs3WW/11VcPs2jN8PTp08MzrdaVrrTSSrX1RRZZJDwzbNiwMOtp3/72t8Psq1/9am39zTff7K3r9Bv9uTcfeeSRMFtttdV68lGN8cILL9TWTz/99PDMBRdcEGatPh8xb/pzb/a0NdZYI8x++MMfhtnw4cNr6zk34/+al156Kczuvffe2vpRRx3VrWfNmTMnzKZMmdKt1+yr9CYppbTwwguH2a677hpmP/rRj2rr0Wr1lFLafffd5/5i/ZgV6gAAAAAFM+QBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAA27V60Z577hlmrX5jeLSFqpVW27q6ozt3eCfR1rBWG4b6MpsI6i2zzDJhdscdd9TWV1lllV66Tc+56KKLauunnnpqeKbV9rg33nhjnu9Evf7cm8svv3yYRdsvUkppxIgRtfUBAwZ06x7R9x5vvfVWeOaggw4Ks2nTpoVZtLXv4YcfDs/QGf25N5vic5/7XG19hRVWCM+ccMIJ3XpWtPlu7Nix4Zknn3wyzG666aZu3YN3pjdJqfU2rJ133jnMTjnllC7VU0pp5syZc3+xfsx2LQAAAICCGfIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAaxQ72POPPPM2vp73vOe8MxGG20UZtG5VivZ77zzzjBrda7VuugSWTfZdWussUZt/cQTTwzP7LHHHl1+zpw5c8LsjDPOCLPx48eHWdQXb7755txfjLbQm1233HLL1dY/8pGPdOv1opXnt912W7dejzLoTWgmvVmeI444Isyi74Vfe+218MzJJ5/crYx5Y4U6AAAAQMEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAVqhDL7FuEppJb0Iz6U1oJr1Znu9973thtummm9bWv/71r4dnrrnmmnm+E11nhToAAABAwQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoABWqEMvsW4SmklvQjPpTWgmvQnNZIU6AAAAQMEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgALkqqo6fQcAAAAA5pF38gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQ56C5Zx/nHOemnOennN+OOe8f6fvBOhNaCq9Cc2jL6GZ9GZz5aqqOn0HeknOeZ2U0qNVVc3KOQ9NKd2aUtqxqqp7O3sz6N/0JjST3oTm0ZfQTHqzubyTp2BVVU2pqmrWv/76zz+rdfBKQNKb0FR6E5pHX0Iz6c3mMuQpXM75vJzzzJTSgymlqSml8R2+EpD0JjSV3oTm0ZfQTHqzmfy4Vj+Qcx6QUto4pbR5Sum0qqre7OyNgJT0JjSV3oTm0ZfQTHqzebyTpx+oqmp2VVW/TSmtlFI6uNP3Ad6mN6GZ9CY0j76EZtKbzWPI078MTH5OEppIb0Iz6U1oHn0JzaQ3G8KQp1A552VyznvlnAfnnAfknLdNKe2dUrql03eD/kxvQjPpTWgefQnNpDebze/kKVTOeemU0pUppfent4d5T6SUzq2qalxHLwb9nN6EZtKb0Dz6EppJbzabIQ8AAABAAfy4FgAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkKdgOecf55yn5pyn55wfzjnv3+k7AXoTmkpvQvPoS2gmvdlcVqgXLOe8Tkrp0aqqZuWch6aUbk0p7VhV1b2dvRn0b3oTmklvQvPoS2gmvdlc3slTsKqqplRVNetff/3nn9U6eCUg6U1oKr0JzaMvoZn0ZnMZ8hQu53xeznlmSunBlNLUlNL4Dl8JSHoTmkpvQvPoS2gmvdlMflyrH8g5D0gpbZxS2jyldFpVVW929kZASnoTmkpvQvPoS2gmvdk83snTD1RVNbuqqt+mlFZKKR3c6fsAb9Ob0Ex6E5pHX0Iz6c3mMeTpXwYmPycJTaQ3oZn0JjSPvoRm0psNYchTqJzzMjnnvXLOg3POA3LO26aU9k4p3dLpu0F/pjehmfQmNI++hGbSm83md/IUKue8dDfrjiAAACAASURBVErpypTS+9Pbw7wnUkrnVlU1rqMXg35Ob0Iz6U1oHn0JzaQ3m82QBwAAAKAAflwLAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABBrbzYTlnv+WZfqOqqtzpO8wtvUl/ojehmfQmNJPehGaKetM7eQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAgzs9AVopmWXXTbMxo4dG2aHH354l5/1wgsvhNnSSy/d5dcDAACA/sg7eQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABbBCvR/Ya6+9wmyjjTaqrR988MHhmYED4w+bqqrm/mLzcAY6Yd999w2zE088sba+0korhWe687H/yiuvhNkJJ5wQZhdffHGYTZ8+vbb+1ltvzf3FAGiMpZZaKszOOeecMNt777174zq1cs619QkTJoRnxowZE2aTJ0+e5ztBScaOHRtmX/jCF8Lsmmuuqa2PGzcuPPPwww+H2cyZM8OM3uGdPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAuR2rq/OOduV3UtGjBgRZjfddFOYDRgwoDeu0yUzZswIs1b/u/74xz/2xnV6TFVV9btBG0hv/tugQYPCbPz48WG2xRZb1NZbrTx/7rnnwmyJJZaorS+++OLhme46/fTTa+vHHntseObNN9/s8Xu0i96EZtKbPefcc88Ns0MPPbSNN+lZU6dODbOhQ4fW1l999dXeuk6/oTf7pve+971hNnHixDBbZZVVuvys++67L8xuvPHG2vovf/nL8Mxtt93W5Tv0R1FveicPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMB2rUKsueaaYdbqt6evsMIKXX7W3XffHWYf/vCHu/x6rbTaPrT88sv36LN6mk0EfdMiiywSZq16Kfpc2mqLyT333BNm66yzTm19+PDh4ZlvfetbYRZt62plzJgxYXbmmWd2+fWaQm+2x/zzzx9mH/zgB9t4k3ovvPBCmD388MNtvAn/ojd7zte+9rUwW2yxxcLsscceq6232i7ZyhprrBFmO+20U239kEMO6dazRo8eXVs/55xzuvV6/JveLE+rr9HR9uWll146PHPZZZeF2dprr11bHzx4cHim1eecgw8+OMyeffbZMCuR7VoAAAAABTPkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAJYod4PLLnkkmE2aNCgLr/eq6++GmbRiryUUvrd737X5We98sorYdZqBWgTWDdZnuOPPz7M7rzzztr6r371q166zX9q1X8XX3xxmH3oQx+qrY8bNy48c9BBB839xRpGb9Y77LDDwmzzzTevrS+//PLhmVbrWTfYYIO5vtfcyDn+vzT6PufFF18MzzzyyCNhduONN9bWTzjhhPAMc0dv9i/R54irrroqPLP99tuHWXRuzz337NrF+A96k3mx5ppr1ta32GKL8My5554bZi+//HKYrbXWWrX1F154ITzTl1mhDgAAAFAwQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAowMBOX4De19Mr4xZddNEwO+aYY3r0WZMnT+7R14N50WqFehM88MADYdZqXXRk5MiRYTZmzJgwmz59epefRXsce+yxYXb00UeH2YILLtjlZ5199tlhNmnSpC6/XivdWaG+9957h2c23HDDMBs+fHht/d577w3PXH/99WEG/dWsWbNq6/fdd194ptUK9T//+c/zfCeg5z300ENdqqeU0qc//ekw23jjjcNs/vnnn/uLFcw7eQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAtmtRa7HFFguzfffdN8x22GGHLj/rjTfeCLPTTjuty68H/dVSSy0VZiuvvHKXX+/GG28Ms1dffbXLr0fnfeMb3wizOXPmhNntt99eW99xxx3DM6+88srcX6wDWm0TGz16dJidccYZtfXrrrsuPLPddtuF2U033RRm0NcNHBj/U2OrrbaqrX/hC1/o1rNabbgDmmezzTYLs2iTZUop3X333WHWnW2yJfJOHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAaxQ72OWXXbZ2vonP/nJ8MwWW2wRZt///vdr6/vtt1945mMf+1iYdUerNXjjx4/v0WfBvNhrr73CLFqRPHLkyPDMd77znTCbOXPm3F/sn3beeecwGzp0aJg9+OCDtfUjjzwyPNNq3TbNtccee3Tr3KRJk2rrTV+T3spuu+0WZp/61KfCrKqqLtVTSmnUqFFhZoU6fd2QIUPC7OSTTw6zVl9Tu2O11VarrX/gAx8Izzz00ENh1p2vw8B/WnDBBWvrrb5vbdV/hx56aJi9/vrrc3+xgnknDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgALnVys8ef1jO7XtYHzZixIgwu/DCC2vrq6++em9dp0tee+21MDviiCNq69dee2145vnnn5/nO3VKVVW503eYW/2tNwcMGBBm++67b5iNHTs2zFZdddV5uVJHXXXVVbX1Pffcs803aQ+9WZ5Wq+F33HHH2vouu+wSnhk8eHCY5Vz/4TNt2rTwTKv73XbbbWHW3+jN5ho4cGCYTZgwIcxafU/bBJMnTw6zk046Kcyuvvrq3rhOY+lN3smwYcPC7Ktf/Wptfddddw3PnHjiiWF23HHHzf3FChf1pnfyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYIV6A+28885h1vSVjY8//niYrbbaau27SANYN9lcyy23XJg988wz3XrNiRMn1tbvuOOO8MyRRx4ZZgsttFC37tEdVqg3V3/rze6aPXt2mPX09zkzZsyore+www7hmdtvv71H71AqvdlcgwYNCrNbbrklzFqtXr/xxhu7fI899tgjzIYMGVJbX2CBBbr8nJRSmjNnTpideuqptfXjjz8+PNPq81TT6c3+Zb755qutH3vsseGZVt/TRp8/Tj/99PDM17/+9TDry73U06xQBwAAACiYIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgALEv/KejnnxxRfD7NVXX62tDx48ODzzyiuvhNmf//zn2vpaa60VnlliiSXCrNXWou233762fsMNN4RnoDeMHDmyx18z2oZ1xBFHdPlMSik9+eSTtfWnnnoqPLPwwguH2QYbbBBmH/nIR2rrH/rQh8Iz99xzT5hBu+XcvsUv0dfhqGdTSmnJJZcMsxdeeGGe7wS97Y033gizTTfdtG33OO6448Ls/e9/f219m222Cc+02hbU6mv0V7/61dp6q34+++yzwwya5OCDD66tf+1rX+vW65100km19Vb9x7zxTh4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAGsUG+g3/zmN2G23Xbb1dYXWWSR8MzLL78cZnfddVdt/dprrw3PfPzjHw+zBRZYIMxarVeHdrrzzjt7/DWjNeStTJgwIcwOOuig2voTTzwRnmm1Qn3ixIlh9uEPf7i2vv/++4dnrFCnSUaNGtWtLPLBD34wzKKvZY899lh45umnnw6z6OtwSikdccQRtfVnn302PAP91eTJk7tUTymlxx9/PMwuvfTSMBswYEBt/X3ve194BvqKK664ora+3nrrhWfWXHPNMDvqqKNq6w8++GB45rLLLgsz3pl38gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACpCrqmrfw3Ju38PaaIkllqitr7TSSuGZww8/PMx++tOfhtnf//73ub/YXHjttddq69/61rfCM61WqLcSrWO++OKLu/V6TVdVVe70HeZWqb0ZGThwYJh96UtfCrNWqx7/53/+p8v3eOSRR8Lsrbfe6vLrtXLDDTeE2TbbbFNbHzduXHgmWvHeF+hN3kmrFepLLrlkbb3VqvaPfvSjYRZ9D5FSSk899VRt/bTTTgvPfO973wuzptObNMlzzz0XZtHngTfeeCM8s8gii4TZm2++OfcX6wC9yTt517vi945MnDixtr7iiiuGZzbYYIMwi/792h9FvemdPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFCAeMVMP9Rqw8W3v/3tMFt77bVr68OGDevWPT73uc9161x3vPTSS7X1119/vW13gHZrtbnqjDPOaONNetYyyywTZkOGDGnjTaBv+8Mf/tDlMzfddFOYbbLJJmG2yy67hNno0aNr662+J1l//fXD7JBDDgkz4P/6/ve/H2ZjxoyprQ8aNCg8s/zyy4fZk08+OfcXgwaaM2dOmF177bW19bPPPjs8s84664TZPffcM/cX66e8kwcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUAAr1P+Xww8/PMw+/elPt/Em7bP44ot3+gpAFyy22GJhNmHChDBbc801wyxaKd+X18lDk9x+++3dyiZPnlxbv+SSS8IzBx54YJdf74ILLgjPQH+19tpr9+jrjRw5MszOOuusHn0WNMnEiRO7fGarrbYKMyvU35l38gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACtAvV6hvv/32tfWxY8d26/VmzpxZWz/22GPDMy+//HK3nrX66qvX1kePHh2eGTRoULee1R2zZ88Os+i/E/CfolXp++67b3hmvfXW69azfvSjH9XWH3300W69HjD3hg0bFmbHHHNMbb2qqm4967vf/W5t3Qr1/mW++eYLs4033ri2vuyyy4Znxo8fH2YzZsyY+4t1wMILLxxmrVY4R+66664wO++887r8elCCHXfcsctnHn/88Z6/SD/inTwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAL0yxXqU6dOra23Wv89cGD8n+raa6+trUdrid/JWmutFWbDhw+vrb/00kvhmVZrL7vjscceC7NWa+N/+tOf9ug96KxWPbHooouG2QsvvNAb1+mT1l133TD7whe+UFvff//9u/WsSZMmhdmXv/zlbr0mlOpjH/tYmC200EK19VZrzT/xiU+E2Q477BBm888/f5ef1cqJJ57YrXOUZcsttwyz66+/vsuvN2HChDA76qijwmzKlCldflZ3LLbYYmF2+eWXh9kCCyzQ5We1Wic/a9asLr8elGCXXXaprbfqiVa9xDvzTh4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAoQO7uhoZuPSzn9j2sG374wx+G2Sc/+ckwyzn3xnXaItooNm7cuPDM2WefHWaPPPLIPN+pFFVV9ZkPjO705gc+8IEwO+ecc8Js9913D7Pnnnuuq9dojKWWWqq23qpfRo4cGWaDBg2qrbfaRBBt+ksppcMPPzzMpk2bFmYlKr03SzVs2LAwa7XN8tlnn62tr7HGGuGZ5ZZbLsyizYK98f1U9P1Fq2e12qB10kkn1dZbbRdtJ73ZHueff36YHXDAAT36rOnTp4fZNddcU1v/5S9/2a1nDRkypLZ+2GGHhWdWWWWVbj0r2ig2atSo8Ez0uagv0Ju8k89+9rNhdtFFF9XWW21ebvVvb/4t6k3v5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMAK9bn0zDPPhFmrVas9bc6cObX1hx9+ODzzgx/8IMz+9re/1dZbraNl7pS+bnK//fYLs3HjxoXZMcccE2annHJKV6/R0lprrVVbX2eddcIz22+/fZitu+66YbbYYovV1ldfffXwTCsPPPBAbf3MM88Mz7Tqdf6t9N4s1ejRo8PsjDPOCLOe/j6nO2vNW3n11VfD7Lrrrqutt1qT3ur7gabTm+0xdOjQMJs0aVJtfZlllumt6/yHVr0U9V9viL4Op5TSrrvuWlt/9NFHe+s6HaU3O+9d76p/b0ar1eWXXHJJmEX/pkwp7rMhQ4aEZ2677bYwm2+++WrrO+ywQ3jmj3/8Y5jxb1aoAwAAABTMkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYIV6D/jiF79YW99kk03CMyNHjgyz73znO2F255131tZ/8pOfhGfojNLXTS6xxBJh9vzzz4fZrFmzwuyZZ57p6jVaWnzxxbtU7w2PPPJImLVah37llVfW1l966aV5vlN/V3pvlup973tfmLX6envaaafV1lt9DmvlySefrK0/++yz4ZlWa83POeecMJs8efLcX6wAerPzlltuudr6+eefH57Zaaedeus6PeLll18Osx/84AdhNmbMmDCbPXv2vFypz9GbnbfQQgvV1n/zm9+EZx5//PEwe/XVV8MsWte+zz77hGda2WuvvWrrP/vZz7r1evybFeoAAAAABTPkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQANu1oJeUvokg+s37KaX0iU98IsyOPfbYMGu1PacJrrrqqjCbMGFCbb3V5rsZM2bM853outJ7k/8r+rzS3e1aTzzxRG196tSp3Xo9/k1vNlerr/kHHnhgmH384x8Ps2233bbL97jiiivCLNok9N3vfjc889RTT3X5Dv2R3myuESNGhNmpp54aZhtuuGGY5Vz/f/ekSZPCM9/85jfD7JZbbqmtt3MOUSrbtQAAAAAKZsgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABbBCHXqJdZPQTHoTmklvQjPpTWgmK9QBAAAACmbIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFCAXFVVp+8AAAAAwDzyTh4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMhTsJzzj3POU3PO03POD+ec9+/0nQC9CU2lN6F59CU0k95srlxVVafvQC/JOa+TUnq0qqpZOeehKaVbU0o7VlV1b2dvBv2b3oRm0pvQPPoSmklvNpd38hSsqqopVVXN+tdf//lntQ5eCUh6E5pKb0Lz6EtoJr3ZXIY8hcs5n5dznplSejClNDWlNL7DVwKS3oSm0pvQPPoSmklvNpMf1+oHcs4DUkobp5Q2TymdVlXVm529EZCS3oSm0pvQPPoSmklvNo938vQDVVXNrqrqtymllVJKB3f6PsDb9CY0k96E5tGX0Ex6s3kMefqXgcnPSUIT6U1oJr0JzaMvoZn0ZkMY8hQq57xMznmvnPPgnPOAnPO2KaW9U0q3dPpu0J/pTWgmvQnNoy+hmfRms/mdPIXKOS+dUroypfT+9PYw74mU0rlVVY3r6MWgn9Ob0Ex6E5pHX0Iz6c1mM+QBAAAAKIAf1wIAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIULOf845zz1Jzz9Jzzwznn/Tt9J0BvQlPpTWgefQnNpDebywr1guWc10kpPVpV1ayc89CU0q0ppR2rqrq3szeD/k1vQjPpTWgefQnNpDebyzt5ClZV1ZSqqmb966///LNaB68EJL0JTaU3oXn0JTST3mwuQ57C5ZzPyznPTCk9mFKamlIa3+ErAUlvQlPpTWgefQnNpDebyY9r9QM55wEppY1TSpunlE6rqurNzt4ISElvQlPpTWgefQnNpDebxzt5+oGqqmZXVfXblNJKKaWDO30f4G16E5pJb0Lz6EtoJr3ZPIY8/cvA5OckoYn0JjST3oTm0ZfQTHqzIQx5CpVzXibnvFfOeXDOeUDOeduU0t4ppVs6fTfoz/QmNJPehObRl9BMerPZ/E6eQuWcl04pXZlSen96e5j3RErp3KqqxnX0YtDP6U1oJr0JzaMvoZn0ZrMZ8gAAAAAUwI9rAQAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAowMB2Pizn7Lc8029UVZU7fYe5pTfpT/QmNJPehGbSm9BMUW96Jw8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUICBnb5AyY444ogw+/rXvx5miy++eJef9fDDD4fZVlttFWZPPfVUl58FAAAANI938gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACpCrqmrfw3Ju38Ma4Omnnw6z5Zdfvm33aLVe/aSTTqqtX3rppb11nX6jqqrc6TvMrf7Wm/RverM95ptvvjDbbrvtauurrLJKeGattdbq1j2WWWaZ2vrIkSPDM62+N7ruuutq69dee2145te//nWYPfbYY2HW3+hNUkpp+PDhYXbAAQeE2bLLLltb//jHPx6eadXrf/zjH8Psoosuqq1/97vfDc/0ZXqzPIMHDw6zo446qra+zjrrhGcWXXTRMNt6661r6y+++GJ45pZbbgmzSy65JMyuv/762no7Zx7tFPWmd/IAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApghXovasoK9VZmz55dW3/00UfDMx/72MfC7K9//es836kU1k12XrSOee+9927vRQIjRowIs2jdZDudc845YTZ69Og23qRn6c32WHHFFcPsySefbONNOq/VKubNNtsszGbOnNkb12ksvdm/rL322rX1W2+9NTyz5JJLhlnO9R8+3f23TvR6KaV0zTXX1NZHjhzZrWc1nd4sz3vf+94wa/XvwKb74he/WFv/7//+7zbfpD2sUAcAAAAomCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACDOz0BeisAQMG1NbXXHPN8My6664bZrZr0W4nnXRSmEW/YX/BBRfsret0SavNHe3cfBjZZpttOn0F+rC33norzGbMmFFbX3jhhXvrOh31gQ98IMxOOOGEMDvqqKN64zrQCEcffXRtvdUGrWeffTbMHnrooS7fYYkllgizDTbYoMuvByW75557wuz+++/v0WftscceYdbqe4Wddtqptl7qdq2Id/IAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApghXov+vznPx9mu+22W7deM1oPeemll4Znjj322DA74IADunyHaOVlSildd911XX49mBe77rprmDVlVXpf9Yc//KHTV6AP+/vf/x5m++yzT239Jz/5SXjmvvvuC7MnnngizMaNGxdmPWmLLbYIs7Fjx4bZqFGjwuyss86qrT/zzDNzfzFoqNVXX722ftFFF4VnWvXStGnTunyHMWPGhNn6668fZq1WSUNf8Morr4RZ9LXn+OOP79brRdZbb70w22uvvbr8eimlNGnSpG6dK4138gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACpCrqmrfw3Ju38P4/7RaI33mmWfW1g888MDwzFNPPRVmI0aMCLNWK25LVFVV7vQd5lZf7s0pU6aE2ZprrtnGm3RdzvGHSDs/N7/00ku19R133DE8c/fdd/fWdXqd3myuJZZYIsxmzJgRZrNmzeqN63TJHnvsEWaXX355t17z9NNPr623WiPdl+nN8gwePDjMrrzyytp6q689s2fP7vIdhg4dGmY33XRTmN18881htu+++3b5Hn2Z3uSdDBw4MMwOPvjg2vro0aPDM0OGDAmzVt+DbrbZZrX1N954IzzTl0W96Z08AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUID412BTjNdeey3MLr300tp6q+1aM2fODLMXX3xx7i8GPeDOO+8Ms6effrpHnzV9+vQwi7bgRBsFUkrps5/97DzfqSf86le/qq335Q1a9E194WvICiusUFs/7rjjevxZw4YN6/HXhHZabrnlwuyqq66qrbfaoNVqK+Xw4cNr61dccUV4JurnlPrfVlh4JyuuuGKYnXLKKWH2qU99qsvP+v3vfx9mI0eODLNSt2h1lXfyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYIV6IZZddtkw22ijjcLs0EMP7fKzhgwZEmYbb7xxmE2YMKHLz4J3MmrUqLY9a9CgQWG24YYb1tZHjBjRW9fpksMOOyzMfvrTn7bxJtB8q666aphde+21tfW11lqrx+8xa9asHn9NaKdHH320y9k222wTnjnqqKPCbKuttqqtV1UVnmlln332CbO//OUvtfXLL7+8W8+Cdhs4MB4DHHLIIbX10aNHh2dWXnnlMIvWmrf6/nPs2LFh9uyzz4YZb/NOaxTyxgAAIABJREFUHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAaxQL8See+4ZZmeffXaPPmvSpElh9vvf/75HnwXtNv/884fZRhttFGY333xzb1znP/zjH/8Is4kTJ4ZZqzWVL7300jzdCdphyy23DLNRo0b16LNafU3NOffos1555ZUwi9bYQgmGDx9eW7/22mvDM62+Rnd3VXrkve99b5hdcskltfXp06eHZ8aPHz/Pd4L/v1ar0Ft93fzyl78cZptvvvm8XOk/PP/887X1KVOmhGfWW2+9MHvuuefC7K233pr7ixXMO3kAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAXIPb1usOXDcm7fw/qZu+++O8yGDRvWo8/addddw+y6667r0Wf1ZVVV9eye3V7U33pz0KBBYXbccceFWat1k93RahVz9Ll55513Ds9cf/3183yn/kBvNlernvjJT34SZnvssUdvXKctbr755jDbZptt2niTztOb/csVV1xRW99tt93CMw899FCYfetb36qtP/LII+GZFVdcMczGjBkTZhtssEFt/Xe/+1145r/+67/CrOn0ZuctvvjitfVbb701PNNqDXlfdsstt4TZ5ZdfXlu/+OKLwzOzZ8+e5zt1StSb3skDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABRjY6QvQ9xx66KFh1mqrwLRp03rjOhAaOLD+U9zxxx8fnmm1TaMJWm0d2XbbbcPs/PPPD7No88hbb7019xeDHtBq890qq6zSvov0sD/96U9htssuu7TxJtAcl156aW39Zz/7WXjm6quvDrOe3pAzderUMPv5z39eW19uueV69A7wL5tsskltvdUGrTfffDPMWn1duu+++2rrTz/9dHhm+vTpYfbKK6/U1lttvhs7dmyYbb311mH20Y9+tLa+wgorhGdOOOGEMOurvJMHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFCAXFVV+x6Wc/se1s/cfffdYTZs2LC23WPy5MlhdtZZZ9XWf/SjH/XWdTqqqqrc6TvMrVJ7c8stt6yt33jjjW2+Sb2c4w+Rdn5uPuOMM2rrX/nKV9p2h3bSm33TZZddFmaf+MQn2niTrrvuuuvCbNddd23jTZpNb9JX/OIXv6itDx06NDyzxhpr9NZ1ep3e7LyBAwfW1j/1qU+FZ26//fYwa7W+vAkWWmihMDvuuOPC7Igjjqitv/766+GZ3XbbLcwmTpwYZk0Q9aZ38gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmCFei9aYoklwuzVV18NszfeeKPLzxo1alSYHXbYYWG28MIL19ZXW221Lt/hnTzzzDO19Q022CA88+KLL/b4PdrFusnO22qrrWrrTVmh/q53xXP2OXPmtPEm9c4555wwGz16dBtv0rP0Zt/0nve8J8xarVdffPHFa+t/+ctfwjO33XZbmO2000619c033zw8k3P8IXfIIYeE2YUXXhhmJdKb9BW//OUva+trrrlmeMYK9fbQm/3bueeeW1tv9e/h3/zmN2G2xRZb1Nab8H16SlaoAwAAABTNkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAFs1+pFrTb4TJo0KcxOPfXU3rhOraWWWqq2vs8++4RnTjjhhDAbPHhwl++w//77h9nFF1/c5ddrCpsIOm/LLbesrTdlu1arjTvt/NwcabU54DOf+UyYXX755b1xnR6jN9tj/fXXD7N11lmntv7AAw+EZyZPnjzPd+pNP/jBD8Ls05/+dJhNmzYtzJZddtl5uVKfozdpklYbMMePH19bb7UFMPq81xfoTfqKIUOG1NZbfX+x4IILhtlCCy1UW3/99de7drFeYrsWAAAAQMEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMCQBwAAAKAAAzt9gZK1WpW4+uqrh9nNN98cZk8++WRt/e9///vcX+x/iVa3nnPOOeGZVqudzzrrrG7dA3rDb3/729r6Rz7ykfDMQQcd1FvX+Q/dWaG+++67h2eiNY/d1Wp97Hzzzdejz6JvWm+99cLs1ltvDbNFF120tv7GG2+EZy688MIw+8Y3vhFmL774Ypj1pKOPPjrM1l577W5lG2+8cW39jjvumPuLAd3yyU9+Msy23nrr2vpGG23UW9ehEAMGDAizVv/++t73vldbv//+++f5TiX5xz/+UVt/66232nyTzvJOHgAAAIACGPIAAAAAFMCQBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAaxQ75BVVlklzO68884wi1aot1q73tO22mqrtj0L5sWsWbNq63fffXd4plXWBN/85jfD7Fe/+lWYrbrqqr1xHfqJwYMH19ZbfTxGa9JbGTRoUJgddthhYbbjjjuG2ahRo2rrv/71r+f+YnPh+eefD7NW/52uvvrqMDvvvPNq65tuuml4ZsaMGWEGKaW05ZZbhtlFF13U5de74YYbwuyBBx4IsyuvvLK2Hq1ATqnnP77XW2+9MDvllFPCbMKECbX1P/zhD/N8J8ow33zz1dbvuuuu8MxLL70UZlalz53tttuutv7ud7+7zTfpLO/kAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwAr1XjRz5swef82VV165tv65z32ux5/V0+bMmVNbj9ZcA/+p1aro5Zdfvo03oT8ZM2ZMbb3Vx2M7rbrqqmF244031tYvuOCC8MzJJ58cZtGK2zfeeCM8013rr79+bb1Vrz/66KM9fg/6nqWXXjrMWn18r7TSSl1+1uc///kun0kppbPOOqu2PmXKlPDMueeeG2YvvPBCbT1aZd3qDimlNHXq1DA75phjauuzZ88Oz9C/DBxY/8/sDTbYIDyz884799Z1irLpppuG2fnnn9/l15swYUKY9dV/p3onDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAdq1edNJJJ4XZRRddFGY55964Tsc98cQTtfXLLruszTehvxs6dGiYLbLIIm27x+abbx5mW221VW19k002Cc/MP//883ql/+O1114Ls2nTpvXos2i2hRZaqMtnWm2buvTSS2vrN9xwQ3hm9913D7M999wzzKLNOocddlh4plV222231dZb9cQKK6wQZtAb1ltvvTAbPnx4G2/Sdeuuu26YXXjhhT36rL/97W9hFm3QSimle++9t0fvASml9JnPfCbMfvGLX7TxJu2x2WabhdkBBxwQZiNHjgyz6Hvh+++/Pzxz5JFHhllVVWHWZN7JAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogBXqveiHP/xhmG244YZhduCBB4ZZ09ert1pFefrpp7fxJvQX7373u8NsxIgRtfVx48aFZ5Zeeul5vtPcatXP7VzZ+Mgjj9TWW62Rvvnmm3vrOjTQ+eefX1v/61//Gp4ZP358mD3++ONdvkOr12tlp512qq0vsMAC3Xq9Vitfe9qcOXO6VId/6emvIa1er7tfy6Jz3b179Hp/+ctfwjP77bdfmN11113dugd012677RZm5557bm19woQJvXWd/9Dqe+6tt946zLbbbrva+jLLLBOeede7eva9KBdccEGYTZkypUef1QTeyQMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKEBu55renHP7HtaHHXzwwWH29a9/vba+7LLL9tZ1/sOll14aZqecckqYtVphWaKqqpq97/5/aXpvLrTQQmF20UUXhdnuu+/eG9fpMe1cof7888+H2Te/+c3a+ne+850evUNT6M3+Zckll6ytjx49Ojzzla98pbeu0yU333xzbX2bbbZp803aQ2/2nAEDBoTZWWedFWaHHXZYb1ynxzz00ENhdtxxx9XWr7nmmvDMm2++Oc936g/0ZtdF3+Mdf/zx4Zno33n8X9OmTQuz/fbbr7Z+/fXXh2faOQ/paVFveicPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFMB2LeglNhH0nKFDh4ZZtH0mpfZuneuO7mzXarUJ5N577w2z3XbbLcyee+65MCuR3uSdrL/++mG2884719Z32WWX8MwGG2wQZq+//nqXnzVx4sTwTF+mN9tjvvnmC7PoY27XXXcNz+y1115hdtddd4XZfffdV1v/3e9+F5658sorw2zGjBlhxrzRmz2nVf+NHTs2zFpt5Wq6+++/v7be3V6fNGlSmM2ePXvuL1YA27UAAAAACmbIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAWwQh16iXWT7fHtb387zA4//PAefdb06dPD7LTTTqutb7PNNuGZm266Kcyiz83RGsqUUrr++uvDjH/Tm9BMehOaSW9CM1mhDgAAAFAwQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogBXq0Eusm4Rm0pvQTHoTmklvQjNZoQ4AAABQMEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAAAAAAFMOQBAAAAKIAhDwAAAEABDHkAAAAACmDIAwAAAFAAQx4AAACAAhjyAAAAABTAkAcAAACgAIY8AAAAAAUw5AEAAAAogCEPAAAAQAEMeQAAAAAKYMgDAAAAUABDHgAAAIACGPIAAAAAFCBXVdXpOwAAAAAwj7yTBwAAAKAAhjwAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAApgyAMAAABQAEMeAAAAgAIY8hQs5/zjnPPUnPP0nPPDOef9O30nQG9CU+lNaB59Cc2kN5srV1XV6TvQS3LO66SUHq2qalbOeWhK6daU0o5VVd3b2ZtB/6Y3oZn0JjSPvoRm0pvN5Z08BauqakpVVbP+9dd//lmtg1cCkt6EptKb0Dz6EppJbzaXIU/hcs7n5ZxnppQeTClNTSmN7/CVgKQ3oan0JjSPvoRm0pvN5Me1+oGc84CU0sYppc1TSqdVVfVmZ28EpKQ3oan0JjSPvoRm0pvN4508/UBVVbOrqvptSmmllNLBnb4P8Da9Cc2kN6F59CU0k95sHkOe/mVg8nOS0ER6E5pJb0Lz6EtoJr3ZEIY8hco5L5Nz3ivnPDjnPCDnvG1Kae+U0i2dvhv0Z3oTmklvQvPoS2gmvdlsfidPoXLOS6eUrkwpvT+9Pcx7IqV0blVV4zp6Mejn9CY0k96E5tGX0Ex6s9kMeQAAAAAK4Me1AAAAAApgyAMAAABQAEMeAAAAgAIY8gAAAAAUwJAHAAAAoACGPAXLOf845zw15zw95/xwznn/Tt8J0JvQVHoTmkdfQjPpzeayQr1gOed1UkqPVlU1K+c8NKV0a0ppx6qq7u3szaB/05vQTHoTmkdfQjPpzebyTp6CVVU1paqqWf/66z//rNbBKwFJb0JT6U1oHn0JzaQ3m8uQp3A55/NyzjNTSg+mlKamlMZ3+EpA0pvQVHoTmkdfQjPpzWby41r9QM55QEpp45TS5iml06qqerOzNwJS0pvQVHoTmkdfQjPpzebxTp5+oKqq2VVV/TaltFJK6eBO3wd4m96EZtKb0Dz6EppJbzaPIU//MjD5OUloIr0JzaQ3oXn0JTST3mwIQ55C5ZyXyTnvlXMenHMekHPeNqW0d0rplk7fDfozvQnNpDehefQlNJPebDa/k6dQOeelU0pXppTen94e5j2RUjq3qqpxHb0Y9HN6E5pJb0Lz6EtoJr3ZbIY8AAAAAAXw41oAAAAABTDkAQAAACiAIQ8AAABAAQx5AAAAAAowsJ0Pyzn7Lc/0G1VV5U7fYW7pTfoTvQnNpDehmfQmNFPUm97JA/y/9u4/Vuu6/B/4+8ARIkRRZEk4AcGWLjxUIATILJCB4QTkx5ChGwG6ANeKJhMhrJX9GAobiY6A1LTNQBeEIIpENnRhiIlop1jQELMjMFFCEjnff76bn32+7+v+nnOfc+77Pq/zePx5PXfd79cGr/Pj4j0uAAAAEmDIAwAAAJAAQx4AAACABBjyAAAAACTAkAcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMCQBwAAACABhjwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQAEMeAAAAgAQY8gAAAAAkwJAHAAAAIAGGPAAAAAAJMOQBAAAASIAhDwAAAEACDHkAAAAAEmDIAwAAAJCA6nIfAKCSXHzxxWE2cODAMFu0aFFufciQIUWd48Ybb8ytb9y4sajPg0rSr1+/3Prf/va3sOfpp58Os8WLF4fZnj17Gn4wACiT1atXh9nMmTNz6+3axe9snD17NszWr18fZlu3bs2tb9mypahn/fvf/w4zWoY3eQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMCQBwAAACABhjwAAAAACaiqr68v3cOqqkr3MCiz+vr6qnKfoaHa2t2srq4Os0JrmkeOHNkSx8l14MCB3Prs2bPDnp07d7bUcZLibpZftEK9tra2qM87depUmPXp0ye3bqVr5XE3oTK5m6Wxbdu2MPva176WW6+qiv9oSvl7/unTp8Ns1apVufVHH320qGfV1dWF2ZEjR4r6zNYqupve5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAE2K5VJjNmzAizwYMHh1n//v1z64X+J/GpU6eG2e7du3Prd911V9izffv2MOMTNhGUX7t2+XPsF198MewZOHBgs55h8+bNYfbXv/41zObNm5db37FjR9gzbty4MDt79myYtTXuZvlF27UKbYibPn16mD3//PNhtn///tz6F77whbCH8nA3aQmjRo0Ksy9/+cthdtNNN4VZ9LNCoZ/H77777jD75S9/GWaVwN0sjR49eoTZypUrc+vjx48Pe0r8e37JznHo0KEw+8Y3vpFb//3vf9+sZ6gUtmsBAAAAJMyQBwAAACABhjwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJghXqZ7Nq1K8wuuuiiMPv73/+eWx8zZkzYU8yf8XvvvRdmkydPDjPr1T9h3WTz6datW5h17NgxzNauXZtbv+6665p8pv9tzZo1ufU5c+YU9XkbNmzIrRdalTl69Ogwczc/4W6WX9euXXPrU6ZMCXt+8YtfhNmiRYvC7J577smtF1qd/Morr4QZLcfdbFtqamoaVc+yLFuwYEGYffazn82tR19vsqzw2udiFPq8l156KcyGDh3arOdobu5m5Zo/f36YFfod8JZbbgmz/fv359YnTZoU9nTq1KmoczS3jz/+OLc+ceLEsGfz5s0tdZwWZ4U6AAAAQMIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMCQBwAAACABVqiXSefOncOs0PrFDz/8MLd+wQUXhD19+/YNs/vvvz+3fvXVV4c9DzzwQJgVWuPX1lg32Xg9evTIrW/cuDHsiVYlZlmWDRo0qMln+p927NgRZtdff31u/b///W9Rz/r+97+fWy+0KnrdunVhNmvWrKLOkSJ3Mz2FVreePHkyt75p06awZ8KECWF29uzZhh+MRnE3W6fLL788zG677bYwmzdvXm79nHPOKeoc//nPf3LrK1asCHtuvvnmMOvVq1ejz1DoZ/g77rgjzFauXNnoZ5WSu5me6OfWLMuygwcP5tbr6urCns985jNhtnfv3gafq6miO3jrrbeGPb/61a9a6jgtzgp1AAAAgIQZ8gAAAAAkwJAHAAAAIAGGPAAAAAAJMOQBAAAASEB1uQ/QVkXbPooVbRTIsiybOnVqmA0YMKDRz3r77bcb3QMN0bVr19x6v379wp7zzjuvWc+wZs2aMFu4cGGYFbtFK1Jd7cszNFShTaHR99sbbrgh7OnQoUOYRVsuIQWf+tSncuuPPfZY2FNoG10xW3wLbbB78sknw2zJkiW59cOHD4c91113XZgVs11r165dYfbggw82+vOgKQptquvSpUuYffe7382tF9qude2114ZZoa1zzS36mtPWvnd7kwcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkAA7ehPx6quvhlmfPn0a/Xnr1q0Ls2XLljX686Ah3njjjdx6bW1t2DNw4MAwe++993Lrd911V9hTKStOb7/99nIfAVqNQqtRn3vuudz6+PHjw55Pf/rTRT0LWrvOnTvn1mtqapr9WdFa9ocffjjs2b59e5gNGzYst/7b3/427Ln88svDrJAdO3bk1hcuXBj2nDlzpqhnQbGuvPLKMIvuX7EKrUmP1poXq9Aq9/nz5+fW169f36xnqHTe5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMAK9RZUaHV5ofWshw8fDrNoJfT999/f8IP9D2vXrs2tnzp1qqjPg5YwadKkMLvzzjvDbOXKlbn1N998s8ln+t+idcwdO3YMeyZPnhxm5557bqPPsGHDhkb3QOq2bt2aW7/xxhvDnl27doXZO++8k1s/duxY2DNv3rwwe+utt8IMSu3o0aO59X79+oU9d999d5j97Gc/C7PTp083/GD/16BBg8Ls0Ucfza337t270c/Jsiyrra0Ns1tuuSW3fuTIkaKeBTTczp07w6ytrUqPeJMHAAAAIAGGPAAAAAAJMOQBAAAASIAhDwAAAEACDHkAAAAAEmDIAwAAAJCAqvr6+tI9rKqqdA8roVGjRuXWn3jiibCn0MrU2bNnh9lLL73U8INRVvX19VXlPkNDpXo3i9G9e/cw+8lPfhJm0TrVqqrm/Wuwbt26MJs7d26YFbOqNlXuJlmWZWvWrCmqb+LEibn1zp07hz0ffvhhmH3xi18MswMHDjT8YAlwN9uWSy65JLde6HvtTTfdFGYdOnTIrRf6XSdau55lWbZ27dow+8Mf/hBmKXI3W6dOnTqF2cKFC8OsV69ezfqsQve2udXU1OTWX3/99ZKdoZSiu+lNHgAAAIAEGPIAAAAAJMCQBwAAACABhjwAAAAACTDkAQAAAEiA7VrN4He/+11ufeDAgWFPoQ05zz77bJjt3r07t/7++++HPZSHTQTpeeGFF8Js6NChJTxJvmXLloXZ8uXLw+zIkSMtcZyK5W7SEkaMGBFm0c8JWZZltbW1YTZ69Ojc+rFjxxp+sFbE3WydvvSlL4VZ//79w+x73/tebr2YzT5ZFm+znDx5ctizcePGMPvoo4+KOkeK3E3+f9q3bx9mn//853Prhe5fsV8H6urqcusjR44Me/bv31/UsyqB7VoAAAAACTPkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQAEMeAAAAgARYoV4mr7zySphdddVVYXbgwIHc+t69e8Oe1atXh1mhde00jXWT6Sl0l2bOnFnCkzTem2++GWZjx47Nrf/zn/9sqeOUlbtJqe3cuTPMrrnmmjAbPHhwbn337t1NPlMlcjdbp8WLF4fZ0qVLS3aOH/3oR7n1e+65J+w5c+ZMSx0nKe4mLaF3795htm3btjDr27dvmLVrl/8Oy+233x72PPTQQ2FW6axQBwAAAEiYIQ8AAABAAgx5AAAAABJgyAMAAACQAEMeAAAAgAQY8gAAAAAkwAr1MunevXuYDRgwIMwWLVqUWx8xYkTYc+LEiTB7/PHHc+vf/OY3wx4axrrJ9EybNi3Mrrjiitz6nDlzwp4nnngizKJ7O2vWrLDnggsuCLPq6uowq62tza3fcccdYc+zzz4bZpXO3aTUfvCDH4RZ9H09y7Ls6quvzq2//PLLTT5TJXI3W6dCP9PefPPNYXbfffc16znefffd3Prs2bPDno0bNzbrGVLlbjafLl26hNlll10WZnV1dbn1I0eONPlMlWjcuHFhtn79+jA755xzcuvHjx8Pe6699tow27dvX5hVAivUAQAAABJmyAMAAACQAEMeAAAAgAQY8gAAAAAkwJAHAAAAIAG2ayVi8eLFYXbnnXeGWefOnXPrf/rTn8Ke8ePHh9nbb78dZm2NTQSU2ujRo8Ns1apVYda7d+/c+rJly8KeFStWhNlbb70VZpXA3aTUbNdqGHezbTnvvPNy6zfccEPYs2DBgjCLttNGW4myLMv69+8fZu+8806YtTXuZuNFW7TWrVsX9hT6HevQoUO59b59+zbuYAl48cUXwyz6vllo5jF27Ngwq/RtsrZrAQAAACTMkAcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASUF3uA9A8Cq1n3bRpU5itXr06tz5w4MCwZ+3atWFWaAUdVIqf/vSnYXb8+PEwu/fee1viOM1m27ZtYVbobj7zzDO59e985zthT9euXcNszpw5YQYAWZZlJ06cyK0/9thjYc9XvvKVMKupqcmtd+vWLeyxQp2Wctlll+XWC61JL6RXr1659VtvvTXsefjhh4t6VqXbsGFDmEUr1AuZMWNGmFX6CvWIN3kAAAAAEmDIAwAAAJAAQx4AAACABBjyAAAAACTAkAcAAAAgAYY8AAAAAAmwQr0N2Lt3b5gtWbIkt15oNd3o0aObfCYohb59++bWZ8+eHfZ06dIlzF5//fUw27x5c279448/DntKqba2Nsz27NmTW7/00kvDnosuuqjJZ4K2omvXruU+AiShT58+je45fPhwmEXf/6Cp6urqcuuHDh0Ke6I16YWMHTs2zFJdod7cDh48WO4jNDtv8gAAAAAkwJAHAAAAIAFVtXyhAAAHMklEQVSGPAAAAAAJMOQBAAAASIAhDwAAAEACDHkAAAAAEmCFehu3ZcuW3PqmTZvCnkmTJrXUcaBZHThwILe+ffv2sGfChAlh9tRTT4XZkiVLcus//OEPw55S6tixY5h179690Z9XU1PTlONAmzJmzJhyHwFajfPPPz/Milkx/Y9//CPMjh071ujPg4Y4cuRIbn3kyJFhz7Zt28Ksb9++ufWpU6eGPSdOnAizOXPmhFkl6NSpU5h99atfDbN27fLfYTl79mzY88ILLzT8YK2EN3kAAAAAEmDIAwAAAJAAQx4AAACABBjyAAAAACTAkAcAAAAgAbZrtXHR/1zer1+/Ep8ESmfKlClh9pvf/CbMxo8fH2ZLly7NrRfaALB+/fowW7duXW59xIgRYU+0eSHLsmzu3LlhduWVV4ZZZN++fY3ugZRNnz49zC655JIwe+6558LsjTfeaNKZoJINGzYst/7QQw+FPVdccUWjn1Poey2U2sGDB8Os0CbGl19+ObdeaBvdxIkTw6zQ1tVHHnkkt/7uu++GPWfOnAmz999/P7c+YMCAsGfBggVhNnTo0DCLtmidOnUq7InO15p5kwcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkICq+vr60j2sqqp0D6NBVqxYkVufP39+2HP8+PEw69atW5PPlIr6+vqqcp+hodzNT1x88cVhtmXLljC76qqrmvUc0SrK9u3bhz1VVaX7KzdkyJAw2717d8nOUQx3s/x69uyZW7/wwguL+rxrrrmm2c6QZVk2fPjwRn/e4MGDw+zo0aNhdv3114fZq6++2uhztGbuZttSW1ubW+/Xr19Rn7d169bc+oQJE8Ke06dPF/WstsbdLL9f//rXufUpU6aEPaX8Pf+jjz4Ks+h7YKGfuZv7HHPnzg171q5d2+znKJXobnqTBwAAACABhjwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQgOpyH4Dmcf7554fZAw88EGbjxo3LrW/atCnsWbBgQcMPBq3Mv/71rzD7+te/HmZz5szJrc+aNSvs6dGjR5hVV5fuy/PevXtz68uXLw97/vznP7fUcWhF2rWL/61o6dKlYfatb30rt37uueeGPYVWHT///PNhFt3poUOHhj2XXnppmEWefPLJMPvxj38cZn/5y18a/SxoLRYtWhRmvXv3zq0XWvv84IMPhtm3v/3t3Lo16aRg2rRpufU9e/aEPffee29LHef/0aFDhzBriVXpkdtuuy23/sgjj5TsDJXAmzwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQgKpC/4N9sz+sqqp0D0vUhRdemFt/6qmnwp7hw4eHWV1dXW591KhRYc++ffvCjE/U19dXlfsMDeVutpyampowmzlzZpgNGTIkt96zZ8+wZ8OGDWH2wQcfhFm0faFQT2vmbjafQYMGhdkzzzwTZvfdd19u/bXXXgt7/vjHP4bZ0aNHw4zWw90sv+jnv+nTp4c9w4YNC7Nog1aWZVn79u1z65MnTw57Cn1dOXnyZJjRNO5m5YruUZZl2YwZM8JszJgxYTZp0qRGn6OqKv4rEs0b1q9fH/YcOnQozFatWhVmBw8eDLMURXfTmzwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQAEMeAAAAgARYoV4mHTp0CLNCqyN//vOf59arq6vDnscffzzMli9fnlvfv39/2EPDWDcJlcndhMrkbpbfzp07c+vDhw8Pe4pZnZxlWfb000/n1qdNmxb2fPDBB2FGy3E3oTJZoQ4AAACQMEMeAAAAgAQY8gAAAAAkwJAHAAAAIAGGPAAAAAAJMOQBAAAASIAV6mXSpUuXMNu0aVOYvfbaa7n1aBV6lmXZgQMHGn4wmo11k1CZ3E2oTO5m+X3uc5/LrW/fvj3s6dmzZ5ht2bIlzKZMmZJbP3nyZNhDebibUJmsUAcAAABImCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMAKdWgh1k1CZXI3oTK5m1CZ3E2oTFaoAwAAACTMkAcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMCQBwAAACABhjwAAAAACTDkAQAAAEhAVX19fbnPAAAAAEATeZMHAAAAIAGGPAAAAAAJMOQBAAAASIAhDwAAAEACDHkAAAAAEmDIAwAAAJAAQx4AAACABBjyAAAAACTAkAcAAAAgAYY8AAAAAAkw5AEAAABIgCEPAAAAQAIMeQAAAAASYMgDAAAAkABDHgAAAIAEGPIAAAAAJMCQBwAAACABhjwAAAAACTDkAQAAAEiAIQ8AAABAAgx5AAAAABJgyAMAAACQAEMeAAAAgAT8HyxO2+/bMn6tAAAAAElFTkSuQmCC\n"
},
"output_type": "display_data",
"metadata": {
"needs_background": "light"
}
}
]
}
],
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"mimetype": "text/x-python",
"pygments_lexer": "ipython3",
"version": "3.7.1",
"nbconvert_exporter": "python",
"name": "python",
"file_extension": ".py",
"codemirror_mode": {
"name": "ipython",
"version": 3
}
},
"gist": {
"id": "0453f149b41c46d060cbc7ecf0b5632d",
"data": {
"description": "Implementing Class Rectification Loss in fast.ai",
"public": false
}
},
"_draft": {
"nbviewer_url": "https://gist.github.com/0453f149b41c46d060cbc7ecf0b5632d"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
@micimize
Copy link

micimize commented Jul 4, 2020

Great explanation, but I don't think the counter usage works, as the targets are tensors and not plain numbers. That's why your Majority and Minority Indices are the same.

Here is some of the preliminary work I did that fixes this using torch.unique and and generalizes up to SortMinorityClass:

from dataclasses import dataclass

def indices_of(occurences, given_class):
    "extract a tensor of indices of the given instance class"
    return tensor([
        i for i, encoding in enumerate(occurences)
        if torch.all(torch.eq(encoding, given_class))
    ])

@dataclass
class BatchClassification:
    __slots__ = ['encoding', 'indices', 'frequency']
    encoding: Tensor # class encoding as a tensor
    indices: Tensor # indices within the batch
    frequency: int # number of occurrences in the class

def classifications_of(targets: Tensor, descending_frequency=False) -> t.Iterable[BatchClassification]:
    class_encodings, class_indices, class_counts = torch.unique(targets, dim=0, return_counts=True, return_inverse=True)
    return sorted([
        BatchClassification(encoding, indices_of(class_indices, encoding), frequency)
        for encoding, frequency in zip(class_encodings, class_counts)
    ], key=lambda bc: bc.frequency, reverse=descending_frequency)

class SortMinorityClass(LearnerCallback):
    def __init__(self, learn:Learner):
        super().__init__(learn)
        self.iters = 0  # Manage the number of printouts we get
    def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:
        if self.iters < 2:
            for index, bc in enumerate(classifications_of(last_target)):
                print(f'frequency group {index}, count {str(bc.frequency)}: {bc}')
            self.iters += 1

Also, since the mining results are a function of outputs and targets anyways, couldn't we implement the entire mining operation within the loss function? I'm kinda new to ML so maybe I'm missing something

@amqdn
Copy link
Author

amqdn commented Jul 17, 2020

@micimize

Thanks for taking the time to do that! I like the idea of using torch.unique to accomplish the sample counting. As you can see, I published this over a year ago, and I notice my relative inexperience with PyTorch and ML back then shows.

Re: mining results being a function of outputs and targets... If I understand your question: You're asking why it is that I included the mining operation in a callback_fn instead of inside the loss.forward. In looking at my code and trying to remember why, I think I couldn't come up with a way to retain the indices of the majority/minority classes using just the loss module. Since the majority/minority class designation changes from batch-to-batch dynamically in this paper, it's necessary to keep track of the indices (I think) in order to calculate the loss properly. That doesn't mean it's not possible, but I hadn't found a way.

@2foil
Copy link

2foil commented Sep 21, 2020

Great explanation, but I don't think the counter usage works, as the targets are tensors and not plain numbers. That's why your Majority and Minority Indices are the same.

Here is some of the preliminary work I did that fixes this using torch.unique and and generalizes up to SortMinorityClass:

from dataclasses import dataclass

def indices_of(occurences, given_class):
    "extract a tensor of indices of the given instance class"
    return tensor([
        i for i, encoding in enumerate(occurences)
        if torch.all(torch.eq(encoding, given_class))
    ])

@dataclass
class BatchClassification:
    __slots__ = ['encoding', 'indices', 'frequency']
    encoding: Tensor # class encoding as a tensor
    indices: Tensor # indices within the batch
    frequency: int # number of occurrences in the class

def classifications_of(targets: Tensor, descending_frequency=False) -> t.Iterable[BatchClassification]:
    class_encodings, class_indices, class_counts = torch.unique(targets, dim=0, return_counts=True, return_inverse=True)
    return sorted([
        BatchClassification(encoding, indices_of(class_indices, encoding), frequency)
        for encoding, frequency in zip(class_encodings, class_counts)
    ], key=lambda bc: bc.frequency, reverse=descending_frequency)

class SortMinorityClass(LearnerCallback):
    def __init__(self, learn:Learner):
        super().__init__(learn)
        self.iters = 0  # Manage the number of printouts we get
    def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:
        if self.iters < 2:
            for index, bc in enumerate(classifications_of(last_target)):
                print(f'frequency group {index}, count {str(bc.frequency)}: {bc}')
            self.iters += 1

Also, since the mining results are a function of outputs and targets anyways, couldn't we implement the entire mining operation within the loss function? I'm kinda new to ML so maybe I'm missing something

Actually, the Counter works well here.
Because when passing the tensors to Counter, the author calls the tolist() method of torch.tensor object.
Check this bellow demo👇:

屏幕快照 2020-09-21 23 08 18

@micimize
Copy link

@2foil ah you're right – but it seems we're both right, because I was looking at the output of SortMinorityClass, which uses Counter(targets), which is later fixed in the final ClassLevelHardMining callback

@2foil
Copy link

2foil commented Sep 22, 2020 •
edited
Loading

@micimize Okay, I got it. Thanks for your explanation 😊.

@2foil
Copy link

2foil commented Sep 22, 2020 •
edited
Loading

@amqdn Thanks for your tutorial ❤️, it helps me a lot when implementing CRLloss.

Here I have one question.
Now I'm dealing with one training dataset, which has multiple majority and minority classes.
So how to compute the omega in CRLloss?

@amqdn
Copy link
Author

amqdn commented Sep 23, 2020 via email •
edited
Loading

@2foil
Copy link

2foil commented Sep 24, 2020

@amqdn Got it 😊, thanks for your explanation. ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment