Skip to content

Instantly share code, notes, and snippets.

@morganmcg1

morganmcg1/fasthugs_demo_buggy.ipynb

Created April 5, 2020 20:33
Show Gist options
  • Save morganmcg1/e5874baa24797f1046d6bc60753cdbb3 to your computer and use it in GitHub Desktop.
Save morganmcg1/e5874baa24797f1046d6bc60753cdbb3 to your computer and use it in GitHub Desktop.
Download ZIP
fasthugs : get_preds in ClassificationInterpretation.from_learner(learn) not working
Display the source blob
Display the rendered blob
Raw
fasthugs_demo_buggy.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# FastHugs\n",
"This notebook gives a full run through to fine-tune a text classification model with **HuggingFace 🤗 transformers** and the new **fastai-v2** library.\n",
"\n",
"## Things You Might Like (❤️ ?)\n",
"**FastHugsTokenizer:** A tokenizer wrapper than can be used with fastai-v2's tokenizer.\n",
"\n",
"**FastHugsModel:** A model wrapper over the HF models, more or less the same to the wrapper's from HF fastai-v1 articles mentioned below\n",
"\n",
"**Vocab:** A function to extract the vocab depending on the pre-trained transformer (HF hasn't standardised this processes 😢).\n",
"\n",
"**Padding:** Padding settings for the padding token index and on whether the transformer prefers left or right padding\n",
"\n",
"**Vocab for Albert-base-v2**: .json for Albert-base-v2's vocab, otherwise this has to be extracted from a SentencePiece model file, which isn't fun\n",
"\n",
"**Model Splitters:** Functions to split the classification head from the model backbone in line with fastai-v2's new definition of `Learner`\n",
"\n",
"## Housekeeping\n",
"### Pretrained Transformers only for now 😐\n",
"Initially, this notebook will only deal with finetuning HuggingFace's pretrained models. It covers BERT, DistilBERT, RoBERTa and ALBERT pretrained classification models only. These are the core transformer model architectures where HuggingFace have added a classification head. HuggingFace also has other versions of these model architectures such as the core model architecture and language model model architectures.\n",
"\n",
"If you'd like to try train a model from scratch HuggingFace just recently published an article on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train). Its well worth reading to see how their `tokenizers` library can be used, independent of their pretrained transformer models.\n",
"\n",
"### Read these first 👇\n",
"This notebooks heavily borrows from [this notebook](https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers) , which in turn is based off of this [tutorial](https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta) and accompanying [article](https://towardsdatascience.com/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2). Huge thanks to Melissa Rajaram and Maximilien Roberti for these great resources, if you're not familiar with the HuggingFace library please given them a read first as they are quite comprehensive.\n",
"\n",
"### fastai-v2 ✌️2️⃣\n",
"[This paper](https://www.fast.ai/2020/02/13/fastai-A-Layered-API-for-Deep-Learning/) introduces the v2 version of the fastai library and you can follow and contribute to v2's progress [on the forums](https://forums.fast.ai/). This notebook uses the small IMDB dataset and is based off the [fastai-v2 ULMFiT tutorial](http://dev.fast.ai/tutorial.ulmfit). Huge thanks to Jeremy, Sylvain, Rachel and the fastai community for making this library what it is. I'm super excited about the additinal flexibility v2 brings. 🎉\n",
"\n",
"### Dependencies 📥\n",
"If you haven't already, install HuggingFace's `transformers` library with: `pip install transformers`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"# CUDA ERROR DEBUGGING\n",
"# https://lernapparat.de/debug-device-assert/\n",
"import os\n",
"os.environ['CUDA_LAUNCH_BLOCKING'] = \"1\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"%reload_ext autoreload\n",
"%autoreload 2\n",
"\n",
"from fastai2.basics import *\n",
"from fastai2.text.all import *\n",
"from fastai2.callback.all import *\n",
"\n",
"from transformers import AlbertForSequenceClassification, AlbertTokenizer, AlbertConfig\n",
"\n",
"# from transformers import BertForSequenceClassification, BertTokenizer, BertConfig\n",
"# from transformers import DistilBertForSequenceClassification, DistilBertTokenizer, DistilBertConfig\n",
"# from transformers import RobertaForSequenceClassification, RobertaTokenizer, RobertaConfig\n",
"import json"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"path = untar_data(URLs.IMDB_SAMPLE)\n",
"model_path = Path('models')\n",
"df = pd.read_csv(path/'texts.csv')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## FastHugs Tokenizer\n",
"This tokenizer wrapper is initialised with the pretrained HF tokenizer, you can also specify the max_seq_len if you want longer/shorter sequences. Given text it returns tokens and adds separator tokens depending on the model type being used."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"class FastHugsTokenizer():\n",
" \"\"\" \n",
" transformer_tokenizer : takes the tokenizer that has been loaded from the tokenizer class\n",
" model_type : model type set by the user\n",
" max_len : override default sequence length, typically 512 for bert-like models\n",
" \"\"\"\n",
" def __init__(self, transformer_tokenizer=None, model_name = 'roberta', max_seq_len=None, **kwargs): \n",
" self.tok = transformer_tokenizer\n",
" self.max_seq_len = ifnone(max_seq_len, self.tok.max_len)\n",
" self.model_name = model_name\n",
" self.pad_token_id = self.tok.pad_token_id\n",
" \n",
" def do_tokenize(self, t:str):\n",
" \"\"\"Limits the maximum sequence length and add the special tokens\"\"\"\n",
" CLS = self.tok.cls_token\n",
" SEP = self.tok.sep_token\n",
"# import pdb\n",
"# pdb.set_trace()\n",
" #print(t)\n",
" if 'roberta' in model_name:\n",
" tokens = self.tok.tokenize(t, add_prefix_space=True)[:self.max_seq_len - 2]\n",
" else:\n",
" tokens = self.tok.tokenize(t)[:self.max_seq_len - 2]\n",
" #print(tokens)\n",
" return [CLS] + tokens + [SEP]\n",
"\n",
" def __call__(self, items): \n",
" for t in items: yield self.do_tokenize(t)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## FastHugs Model\n",
"This `nn.module` wraps the pretrained transformer model, initialises it with is config file. If you'd like to make configuration changes to the model, you can do so in this class. The `forward` of this module is taken straight from Melissa's notebook above and its purpose is to create the attention mask and grab only the logits from the output of the model (as the HF transformer models also output the loss)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# More or less copy-paste from https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers/data\n",
"class FastHugsModel(nn.Module):\n",
" def __init__(self, pretrained_model_name, model_class, config_dict, n_class, max_seq_len=None):\n",
" super(FastHugsModel, self).__init__()\n",
" self.config = config_dict #config_class.from_pretrained(pretrained_model_name)\n",
" self.config.num_labels = n_class\n",
" if max_seq_len is not None: self.config.max_position_embeddings = max_len\n",
" \n",
" self.transformer = model_class.from_pretrained(pretrained_model_name, config = self.config, \n",
" cache_dir=model_path/f'{pretrained_model_name}')\n",
" \n",
" def forward(self, input_ids, attention_mask=None):\n",
" attention_mask = (input_ids!=1).type(input_ids.type()) \n",
" logits = self.transformer(input_ids, attention_mask = attention_mask)[0] \n",
" return logits"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Padding\n",
"Pass the initialised transformer tokenizer to set the index for the padding token and the side padding should be applied; e.g. BERT, Roberta prefers padding to the right, so we set `pad_first=False`"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def transformer_padding(transformer_tokenizer): \n",
" if transformer_tokenizer.padding_side == 'right': \n",
" pad_first=False\n",
" return partial(pad_input_chunk, pad_first=pad_first, pad_idx=transformer_tokenizer.pad_token_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Lets get training\n",
"### Select our HuggingFace model, tokenzier and config"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Grab the model, tokenizer and config that we'd like to use"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoConfig\n",
"\n",
"model_name = 'roberta-base' \n",
"\n",
"model_class = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
"tokenizer_class = AutoTokenizer.from_pretrained(model_name)\n",
"config_dict = AutoConfig.from_pretrained(model_name)\n",
"tfmr_splitter = model_name.split('-')[0] + '_clas_splitter'"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"#model_class, tokenizer_class, config_class, pretrained_model_name, tfmr_splitter = models_dict[model_type]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also change the max sequence length for the tokenizer and transformer here. If its not set it will default to the pretrained model's default, typically `512`. 1024 or even 2048 can also be used depending on your GPU memory"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"max_seq_len = None "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Geting HuggingFace Tokenizer into fastai-v2\n",
"Intialise the tokenizer needed for the pretrained model, this will download the `vocab.json` and `merges.txt` files needed. Specifying `cache_dir` will allow us easily access them, otherwise they will be saved to a Torch cache folder here `~/.cache/torch/transformers`. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"transformer_tokenizer = tokenizer_class.from_pretrained(model_name, cache_dir=model_path/f'{model_name}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Create fasthugstok function:** Lets incorporate the `transformer_tokenizer` into fastai-v2's framework by specifying a fucntion to pass to `Tokenizer.from_df`. (Note `from_df` is the only method I have tested)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"fasthugstok = partial(FastHugsTokenizer, transformer_tokenizer = transformer_tokenizer, \n",
" model_name=model_name, max_seq_len=None)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Set up fastai-v2's Tokenizer.from_df:** We pass `rules=[]` to override fastai's default text processing rules"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"tok_fn = Tokenizer.from_df(text_cols='text', res_col_name='text', tok_func=fasthugstok, rules=[])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ## Vocab\n",
" Model and vocab files will be saved with files names as a long string of digits and letters (e.g. `d9fc1956a0....f4cfdb5feda.json` generated from the etag from the AWS S3 bucket as described [here in the HuggingFace repo](https://github.com/huggingface/transformers/issues/2157). For readability I prefer to save the files in a specified directory and model name so that it can be easily found and accessed in future.\n",
" \n",
"(Note: To avoid saving these files twice you could look at the `from_pretrained` and `cached_path` functions in HuggingFace's `PreTrainedTokenizer` class definition to find the code that downloads the files and maybe modify them to download directly to your specified directory withe desired name. I haven't had time to go that deep.)\n",
"\n",
"Load vocab file into a `list` as expected by fastai-v2. The HF pretrained tokenizer vocabs come in different file formats depending on the tokenizer you're using; BERT's vocab is saved as a .txt file, RoBERTa's is saved as a .json and Albert's has to be extracted from a SentencePiece model"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"def get_vocab(transformer_tokenizer, pretrained_model_name):\n",
" if pretrained_model_name in ['bert-base-uncased', 'distilbert-base-uncased']:\n",
" transformer_vocab = list(transformer_tokenizer.vocab.keys())\n",
" else:\n",
" transformer_tokenizer.save_vocabulary(model_path/f'{pretrained_model_name}')\n",
" suff = 'json'\n",
" if pretrained_model_name in ['albert-base-v2']:\n",
" with open(model_path/f'{pretrained_model_name}/alberta_v2_vocab.{suff}', 'r') as f: \n",
" transformer_vocab = json.load(f) \n",
" else:\n",
" with open(model_path/f'{pretrained_model_name}/vocab.{suff}', 'r') as f: \n",
" transformer_vocab = list(json.load(f).keys()) \n",
" return transformer_vocab"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"transformer_vocab = get_vocab(transformer_tokenizer, model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup Data\n",
"### Create Dataset\n",
"Lets add our custom tokenizer function (`tok_fn`) and `transformer_vocab` here"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"splits = ColSplitter()(df)\n",
"x_tfms = [attrgetter(\"text\"), tok_fn, Numericalize(vocab=transformer_vocab)]\n",
"dsets = Datasets(df, splits=splits, tfms=[x_tfms, [attrgetter(\"label\"), Categorize()]], dl_type=SortedDL)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataloaders\n",
"Here we use our `transformer_padding()` wrapper when loading the dataloader"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"bs = 4\n",
"dls = dsets.dataloaders(bs=bs, before_batch=transformer_padding(transformer_tokenizer))"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>text</th>\n",
" <th>category</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;s&gt; ĠI Ġwas Ġfortunate Ġenough Ġto Ġmeet ĠGeorge ĠPal Ġ( and Ġstill Ġhave Ġmy ĠDS : TM OB Ġposter Ġaut ographed Ġby Ġhim ) Ġat Ġa Ġconvention Ġshortly Ġafter Ġthe Ġrelease , Ġand Ġasked Ġhim Ġwhy Ġhe Ġchose Ġto Ġdo Ġthe Ġfilm Ġ\" camp \". ĠBefore Ġhe Ġcould Ġanswer , Ġtwo Ġstudio Ġfl acks Ġintercepted Ġand Ġlect ured Ġme Ġon</td>\n",
" <td>negative</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>&lt;s&gt; ĠThis Ġfilm Ġsat Ġon Ġmy ĠT ivo Ġfor Ġweeks Ġbefore ĠI Ġwatched Ġit . ĠI Ġdreaded Ġa Ġself - ind ul gent Ġy upp ie Ġflick Ġabout Ġrelationships Ġgone Ġbad . ĠI Ġwas Ġwrong ; Ġthis Ġwas Ġan Ġeng ross ing Ġexc ursion Ġinto Ġthe Ġscrewed - up Ġlib id os Ġof ĠNew ĠYorkers .&lt; br Ġ/ &gt;&lt; br</td>\n",
" <td>positive</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>&lt;s&gt; ĠThis Ġis Ġan Ġamazing Ġfilm Ġto Ġwatch Ġor Ġshow Ġyoung Ġpeople . ĠAside Ġfrom Ġa Ġvery Ġbrief Ġnude Ġscene , Ġit Ġgives Ġan Ġinteresting Ġglimpse Ġinto Ġcolonial Ġrule Ġin ĠAfrica Ġthat Ġyou 'll Ġrarely Ġfind Ġin Ġother Ġfilms . ĠIt Ġdoes Ġbear Ġa Ġsuperficial Ġsimilarity Ġto ĠOUT ĠOF ĠAfrica , Ġbut Ġwithout Ġall Ġthe Ġromantic Ġfl uff . ĠThe</td>\n",
" <td>positive</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"dls.show_batch(max_n=3, trunc_at=60)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (Alternatively) Factory dataloader\n",
"Here we set:\n",
"- `tok_tfm=tok_fn` to use our HF tokenizer\n",
"- `text_vocab=transformer_vocab` to load our pretrained vocab\n",
"- `before_batch=transformer_padding(transformer_tokenizer)` to use our custom padding function "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Factory\n",
"fct_dls = TextDataLoaders.from_df(df, text_col=\"text\", tok_tfm=tok_fn, text_vocab=transformer_vocab,\n",
" before_batch=transformer_padding(transformer_tokenizer),\n",
" label_col='label', valid_col='is_valid', bs=bs)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>text</th>\n",
" <th>category</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;s&gt; ĠI Ġwas Ġfortunate Ġenough Ġto Ġmeet ĠGeorge ĠPal Ġ( and Ġstill Ġhave Ġmy ĠDS : TM OB Ġposter Ġaut ographed Ġby Ġhim ) Ġat Ġa Ġconvention Ġshortly Ġafter Ġthe Ġrelease , Ġand Ġasked Ġhim Ġwhy Ġhe Ġchose Ġto Ġdo Ġthe Ġfilm Ġ\" camp \". ĠBefore Ġhe Ġcould Ġanswer , Ġtwo Ġstudio Ġfl acks Ġintercepted Ġand Ġlect ured Ġme Ġon</td>\n",
" <td>negative</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>&lt;s&gt; ĠDirect ed Ġby Ġthe Ġduo ĠY ud ai ĠYam ag uchi Ġ( Battle field ĠBaseball ) Ġand ĠJun ' ichi ĠYam amoto Ġ\" Meat ball ĠMachine \" Ġis Ġapparently Ġa Ġremake Ġof ĠYam amoto 's Ġ1999 Ġmovie Ġwith Ġthe Ġsame Ġname . ĠI Ġdoubt ĠI 'll Ġever Ġget Ġa Ġchance Ġto Ġsee Ġthe Ġoriginal Ġso ĠI 'll Ġjust Ġstick</td>\n",
" <td>positive</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>&lt;s&gt; ĠD ressed Ġto ĠKill Ġstarts Ġoff Ġwith ĠKate ĠMiller Ġ( Ang ie ĠDickinson ) Ġhaving Ġa Ġsexually Ġexplicit Ġnightmare , Ġlater Ġon Ġthat Ġday Ġshe Ġvisits Ġher Ġpsychiatrist ĠDr . ĠRobert ĠElliott Ġ( Michael ĠC aine ) Ġfor Ġa Ġsession Ġin Ġwhich Ġshe Ġadmits Ġto Ġbe Ġsexually Ġfrustrated Ġ&amp; Ġun ful filled Ġin Ġher Ġcurrent Ġmarriage . ĠKate Ġthen</td>\n",
" <td>positive</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fct_dls.show_batch(max_n=3, trunc_at=60)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Splitters\n",
"HuggingFace's models with names such as: `RobertaForSequenceClassification` are core transformer models with a classification head. Lets split the classification head from the core transformer backbone to enable us use progressive unfreezing and differential learning rates.\n",
"\n",
"You can split the model into 3 groups by modifying the splitter function like so:\n",
"\n",
"`\n",
"def roberta_clas_splitter(m):\n",
" \"Split the classifier head from the backbone\"\n",
" groups = [nn.Sequential(m.transformer.roberta.embeddings,\n",
" m.transformer.roberta.encoder.layer[0],\n",
" m.transformer.roberta.encoder.layer[1],\n",
" m.transformer.roberta.encoder.layer[2],\n",
" m.transformer.roberta.encoder.layer[3],\n",
" m.transformer.roberta.encoder.layer[4],\n",
" m.transformer.roberta.encoder.layer[5],\n",
" m.transformer.roberta.encoder.layer[6],\n",
" m.transformer.roberta.encoder.layer[7],\n",
" m.transformer.roberta.encoder.layer[8])]\n",
" groups+= [nn.Sequential(m.transformer.roberta.encoder.layer[9],\n",
" m.transformer.roberta.encoder.layer[10],\n",
" m.transformer.roberta.encoder.layer[11],\n",
" m.transformer.roberta.pooler)]\n",
" groups = L(groups + [m.transformer.classifier])\n",
" return groups.map(params)\n",
"`\n",
"\n",
"**Classification Head Differences**\n",
"\n",
"Interestingly, BERT's classification head is different to RoBERTa's\n",
"\n",
"BERT + ALBERT:\n",
"\n",
"`\n",
"(dropout): Dropout(p=0.1, inplace=False)\n",
"(classifier): Linear(in_features=768, out_features=2, bias=True)\n",
"`\n",
"\n",
"DistilBERT's has a \"pre-classifier\" layer:\n",
"\n",
"`\n",
"(pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
"(classifier): Linear(in_features=768, out_features=2, bias=True)\n",
"(dropout): Dropout(p=0.2, inplace=False)`\n",
"\n",
"RoBERTa's:\n",
"\n",
"`(classifier): RobertaClassificationHead(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" (out_proj): Linear(in_features=768, out_features=2, bias=True))`"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def bert_clas_splitter(m):\n",
" \"Split the classifier head from the backbone\"\n",
" groups = [nn.Sequential(m.transformer.bert.embeddings,\n",
" m.transformer.bert.encoder.layer[0],\n",
" m.transformer.bert.encoder.layer[1],\n",
" m.transformer.bert.encoder.layer[2],\n",
" m.transformer.bert.encoder.layer[3],\n",
" m.transformer.bert.encoder.layer[4],\n",
" m.transformer.bert.encoder.layer[5],\n",
" m.transformer.bert.encoder.layer[6],\n",
" m.transformer.bert.encoder.layer[7],\n",
" m.transformer.bert.encoder.layer[8],\n",
" m.transformer.bert.encoder.layer[9],\n",
" m.transformer.bert.encoder.layer[10],\n",
" m.transformer.bert.encoder.layer[11],\n",
" m.transformer.bert.pooler)]\n",
" groups = L(groups + [m.transformer.classifier]) \n",
" return groups.map(params)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"def albert_clas_splitter(m):\n",
" groups = [nn.Sequential(m.transformer.albert.embeddings,\n",
" m.transformer.albert.encoder.embedding_hidden_mapping_in, \n",
" m.transformer.albert.encoder.albert_layer_groups,\n",
" m.transformer.albert.pooler)]\n",
" groups = L(groups + [m.transformer.classifier]) \n",
" return groups.map(params)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"def distilbert_clas_splitter(m):\n",
" groups = [nn.Sequential(m.transformer.distilbert.embeddings,\n",
" m.transformer.distilbert.transformer.layer[0], \n",
" m.transformer.distilbert.transformer.layer[1],\n",
" m.transformer.distilbert.transformer.layer[2],\n",
" m.transformer.distilbert.transformer.layer[3],\n",
" m.transformer.distilbert.transformer.layer[4],\n",
" m.transformer.distilbert.transformer.layer[5],\n",
" m.transformer.pre_classifier)]\n",
" groups = L(groups + [m.transformer.classifier]) \n",
" return groups.map(params)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"def roberta_clas_splitter(m):\n",
" \"Split the classifier head from the backbone\"\n",
" groups = [nn.Sequential(m.transformer.roberta.embeddings,\n",
" m.transformer.roberta.encoder.layer[0],\n",
" m.transformer.roberta.encoder.layer[1],\n",
" m.transformer.roberta.encoder.layer[2],\n",
" m.transformer.roberta.encoder.layer[3],\n",
" m.transformer.roberta.encoder.layer[4],\n",
" m.transformer.roberta.encoder.layer[5],\n",
" m.transformer.roberta.encoder.layer[6],\n",
" m.transformer.roberta.encoder.layer[7],\n",
" m.transformer.roberta.encoder.layer[8],\n",
" m.transformer.roberta.encoder.layer[9],\n",
" m.transformer.roberta.encoder.layer[10],\n",
" m.transformer.roberta.encoder.layer[11],\n",
" m.transformer.roberta.pooler)]\n",
" groups = L(groups + [m.transformer.classifier])\n",
" return groups.map(params)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"splitters = {'bert_clas_splitter':bert_clas_splitter,\n",
" 'albert_clas_splitter':albert_clas_splitter,\n",
" 'distilbert_clas_splitter':distilbert_clas_splitter,\n",
" 'roberta_clas_splitter':roberta_clas_splitter}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load Model with configs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we can tweak the HuggingFace model's config file before loading the model"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"class FastHugsModel(nn.Module):\n",
" def __init__(self, transformer, config_dict, n_class, max_seq_len=None):\n",
" super(FastHugsModel, self).__init__()\n",
" self.config = config_dict #config_class.from_pretrained(pretrained_model_name)\n",
" self.config.num_labels = n_class\n",
" if max_seq_len is not None: self.config.max_position_embeddings = max_len\n",
" self.transformer = transformer\n",
" \n",
"# self.transformer = model_class.from_pretrained(pretrained_model_name, config = self.config, \n",
"# cache_dir=model_path/f'{pretrained_model_name}')\n",
" \n",
" def forward(self, input_ids, attention_mask=None):\n",
" attention_mask = (input_ids!=1).type(input_ids.type()) \n",
" logits = self.transformer(input_ids, attention_mask = attention_mask)[0] \n",
" return logits"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"fasthugs_model = FastHugsModel(transformer=model_class, config_dict=config_dict, \n",
" n_class=dls.c, max_seq_len=max_seq_len)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# fasthugs_model = FastHugsModel(model_class=model_class, config_class=config_dict,\n",
"# pretrained_model_name = model_name, \n",
"# n_class=dsets.c, max_seq_len=max_seq_len)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Initialise everything our Learner"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"opt_func = partial(Adam, decouple_wd=True)\n",
"\n",
"#cbs = [MixedPrecision(clip=0.1), SaveModelCallback()]\n",
"\n",
"loss = CrossEntropyLossFlat() #LabelSmoothingCrossEntropy\n",
"\n",
"splitter = splitters[tfmr_splitter]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Time to train\n",
"### Create our learner"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"learn = Learner(dls, fasthugs_model, opt_func=opt_func, splitter=splitter, \n",
" loss_func=loss, #cbs=cbs, \n",
" metrics=[accuracy]).to_fp16()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stage 1 training\n",
"Lets freeze the model backbone and only train the classifier head. `freeze_to(1)` means that only the classifier head is trainable"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"learn.freeze_to(1) "
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"FastHugsModel (Input shape: ['4 x 512'])\n",
"================================================================\n",
"Layer (type) Output Shape Param # Trainable \n",
"================================================================\n",
"Embedding 4 x 512 x 768 38,603,520 False \n",
"________________________________________________________________\n",
"Embedding 4 x 512 x 768 394,752 False \n",
"________________________________________________________________\n",
"Embedding 4 x 512 x 768 768 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"Dropout 4 x 12 x 512 x 512 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 590,592 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 3072 2,362,368 False \n",
"________________________________________________________________\n",
"Linear 4 x 512 x 768 2,360,064 False \n",
"________________________________________________________________\n",
"LayerNorm 4 x 512 x 768 1,536 False \n",
"________________________________________________________________\n",
"Dropout 4 x 512 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 768 590,592 False \n",
"________________________________________________________________\n",
"Tanh 4 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 768 590,592 True \n",
"________________________________________________________________\n",
"Dropout 4 x 768 0 False \n",
"________________________________________________________________\n",
"Linear 4 x 2 1,538 True \n",
"________________________________________________________________\n",
"\n",
"Total params: 125,237,762\n",
"Total trainable params: 592,130\n",
"Total non-trainable params: 124,645,632\n",
"\n",
"Optimizer used: functools.partial(<function Adam at 0x7f94cac51320>, decouple_wd=True)\n",
"Loss function: FlattenedLoss of CrossEntropyLoss()\n",
"\n",
"Model frozen up to parameter group number 1\n",
"\n",
"Callbacks:\n",
" - ModelToHalf\n",
" - TrainEvalCallback\n",
" - Recorder\n",
" - ProgressCallback\n",
" - MixedPrecision"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets find a learning rate to train our classifier head"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"device(type='cuda', index=0)"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dls.device"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"SuggestedLRs(lr_min=0.00010000000474974513, lr_steep=6.309573450380412e-07)"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEKCAYAAADw2zkCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZxcZZ3v8c+vqve9091ZO3sIEPakERnEAQcEwQuIK4yDKMo4MnpnRh2541ydq1edq9c7KOowDCKOIl42R0AEEYcBBISEEAghkKU7SaeT9Fad7q7q7tqe+aOqQxN67z59avm+X696UXXqVJ3fk0Pqm3Oe5zzHnHOIiEh+C/hdgIiI+E9hICIiCgMREVEYiIgICgMREUFhICIiQIHfBUxVfX29W7Fihd9liIhklU2bNnU65xrGej/rwmDFihVs3LjR7zJERLKKme0Z732dJhIREYWBiIgoDEREBIWBiIigMBARERQGIiKCwkBEJCs8tPUgHX1Dnn2/wkBEJMO19w7ymTs2873f7fBsGwoDEZEM9+OnW4glk3z0rJWebcOzMDCzW82s3cy2jrPOOWb2gpm9bGb/6VUtIiLZKhKN89Nn9vLOdQtYUV/u2Xa8PDK4DbhwrDfNrAb4AXCJc+4E4P0e1iIikpXu3tTK4YEYnzh7lafb8SwMnHOPA93jrHIlcK9zbm96/XavahERyUaJpOOHTzZz2rIaNiyv9XRbfvYZrAVqzewxM9tkZleNtaKZXWtmG81sY0dHxxyWKCLin0e2HWJPV4RPnL0KM/N0W36GQQGwAbgYuAD4n2a2drQVnXM3O+eanHNNDQ1jzsAqIpJTbnliN0vnlXLBCQs935afYdAKPOScCzvnOoHHgVN8rEdEJGM8vzfExj0hPnbWSoIBb48KwN8w+CVwtpkVmFkZcAbwio/1iIhkjFue2E1VSQEfaFo6J9vz7OY2ZnYHcA5Qb2atwJeBQgDn3E3OuVfM7CHgRSAJ3OKcG3MYqohIvuiJRHlo60E+cfYqyovn5h5knm3FOXfFJNb5FvAtr2oQEclGuzr6STo4Y9W8OdumrkAWEckwzZ0RAFbUeXeR2dEUBiIiGaalM0wwYCydVzZn21QYiIhkmOauMEtrSykMzt1PtMJARCTDtHSGPZ2HaDQKAxGRDOKco7kzPKf9BaAwEBHJKB19Q0SiCVbqyEBEJH81d4YBdJpIRCSftXSlwmCVwkBEJH/t7gxTFAywuKZ0TrerMBARySAtnWGWziudk8npRlIYiIhkkJbOyJx3HoPCQEQkYySTjpauuR9WCgoDEZGMcbB3kKF4cs5HEoHCQEQkYwwPK53rkUSgMBARyRh+XWMACgMRkYzR0hmmuCDAwqqSOd+2wkBEJEMMdx4H5nhYKSgMREQyRnNnmBX1c3cPg5EUBiIiGSCeSLK3O+JLfwEoDEREMkJbzyCxhPNlJBEoDEREMkJzeoI6Py44A4WBiEhGaEkPK/VjKgpQGIiIZITmzjDlRUEaKot92b7CQEQkAzR3hlleV47Z3A8rBYWBiEhGaOkKs7LBn1NEoDAQEfFdLJGkNTTASp86j0FhICLiu33dERJJ59s1BqAwEBHx3b7QAADL6/y5+hgUBiIivmsNRQBorJ3b+x6PpDAQEfFZa2iAgoAxv3LuZysdpjAQEfFZa2iAxTWlBH2YrXSYwkBExGetoYivp4jAwzAws1vNrN3Mtk6w3ulmljCz93lVi4hIJtsfGsjdMABuAy4cbwUzCwL/B3jYwzpERDLWYCxBe98QjbX+jSQCD8PAOfc40D3Bap8G7gHavapDRCSTtfWkhpUuqcndI4NxmdkS4D3ATX7VICLit9b0NQa5fJpoIjcAX3DOJSZa0cyuNbONZraxo6NjDkoTEZkbR8Jgnr+niQp83HYT8PP0DH31wEVmFnfO/fvRKzrnbgZuBmhqanJzWqWIiIf290QoCBgLfJq6ephvYeCcWzn83MxuAx4YLQhERHJZa2iARTUlFAT9HenvWRiY2R3AOUC9mbUCXwYKAZxz6icQESEVBo01/p4iAg/DwDl3xRTWvdqrOkREMllrKMLZxzT4XYauQBYR8ctQPMGh3iHfRxKBwkBExDdtPYMAvl9wBgoDERHf7M+QawxAYSAi4ptMuI/BMIWBiIhPWkMDBAPGwir/7mMwTGEgIuKT1lCEhVX+X2MACgMREd+0ZsDU1cMUBiIiPkmFgf8jiUBhICLii2g8yaG+QR0ZiIjkswOHB3AuM0YSgcJARMQXr9/HQKeJRETyViZdYwAKAxERX7SGBggYLKz2/xoDUBiIiPiiNTTAoupSCjPgGgNQGIiI+GJ/aIAlGXKKCBQGIiK+aA1FMqa/ABQGIiKe29sV4R/ue5kt+3qA1DUGB3sHaazJnDDw7R7IIiL54tdbD3DbUy3c9lQLb101j0tOWULSZc6wUtCRgYiI57ojUYqCAf7+4uPZ0xXh737xEpA5w0pBRwYiIp7rCceoKSvk42ev4qozV3D/ljae2d3Fqctq/C7tCIWBiIjHuiNR5pUXAVBUEOC9Gxp574ZGn6t6I50mEhHxWE8kSk1Zod9ljEthICLise7w60cGmUphICLisZ5IjJoyhYGISN5KJh2hSJR5CgMRkfzVNxgn6VCfgYhIPuuORAHUZyAiks9C6TCo1WkiEZH8FQqnw0BHBiIi+SsUiQFQqz4DEZH8pSMDEREhFIlSEDAqizN79h+FgYiIh0KRKDVlRZiZ36WMy7MwMLNbzazdzLaO8f6fmtmL6cdTZnaKV7WIiPglFI5lfH8BeHtkcBtw4TjvNwN/7Jw7GfgqcLOHtYiI+KI7Es34/gLwMAycc48D3eO8/5RzLpR++QyQWfO5iojMgp5INO+PDKbiGuDXY71pZtea2UYz29jR0TGHZYmIzEx3OJbxVx9DBoSBmZ1LKgy+MNY6zrmbnXNNzrmmhoaGuStORGQGnHPpexlkfhj4OtbJzE4GbgHe5Zzr8rMWEZHZ1jcUJ550GT9jKUzyyMDMVptZcfr5OWb2GTOb0c07zWwZcC/wZ86512byXSIimagnnLr6ONNnLIXJnya6B0iY2Rrgh8BK4GfjfcDM7gCeBo41s1Yzu8bMPmlmn0yv8iWgDviBmb1gZhun1wQRkcyULTOWwuRPEyWdc3Ezew9wg3PuRjPbPN4HnHNXTPD+x4GPT3L7IiJZZ3jG0mzoM5jskUHMzK4APgI8kF6W+cc9IiI+Gp6XKBuODCYbBh8FzgS+5pxrNrOVwE+9K0tEJPsNz1iaDR3IkzpN5JzbBnwGwMxqgUrn3D96WZiISLYLhaMEDCpLMnuSOpj8aKLHzKzKzOYBW4Afmdn/87Y0EZHsFopEqS0rIhDI7EnqYPKniaqdc73A5cCPnHMbgPO8K0tEJPulZizNju7VyYZBgZktAj7A6x3IIiIyjlCWTEUBkw+DrwAPA7ucc8+Z2Spgh3dliYhkv1CWTEUBk+9Avgu4a8Tr3cB7vSpKRCQXhCJRTmmc0WQNc2ayHciNZvaL9M1qDpnZPWamKadFRMbgnCMUjlFTnlt9Bj8C7gMWA0uA+9PLRERkFJFogmgimRXXGMDkw6DBOfcj51w8/bgN0FzSIiJj6E5ffVybY2HQaWYfNrNg+vFhQFNOi4iMoSd99XE23PISJh8GHyM1rPQgcAB4H6kpKkREZBTDM5Zmwy0vYZJh4Jzb65y7xDnX4Jyb75y7jNQFaCIiMoqe4TDIsSOD0fzNrFUhIpJjQjnaZzCazJ9sQ0TEJ92RGGZQXZpDp4nG4GatChGRHNMTiVJdWkgwCyapgwmuQDazPkb/0Teg1JOKRERyQHc4mjXXGMAEYeCcq5yrQkREcklPJJY1M5bCzE4TiYjIGLrD0ayZsRQUBiIinujJohlLQWEgIuKJ7oiODERE8tpANMFgLKk+AxGRfBZKX32cTaOJFAYiIrNsOAzUZyAiksdC4dSMpeozEBHJY6Esm7EUFAYiIrMulGUzloLCQERkVoWH4mzaEwKgJksmqYMJpqMQEZHJ6egb4sdPtfCTZ/ZweCDGJacspiCYPf/eVhiIiMzQLza38oV7XiKWSHLBuoVc+8erWL+s1u+ypkRhICIyQ3c+18ri6hJuvfp0VjVU+F3OtGTPMYyISIba3dnP+uW1WRsE4GEYmNmtZtZuZlvHeN/M7LtmttPMXjSz9V7VIiLilf6hOId6h1idxUEA3h4Z3AZcOM777wKOST+uBf7Zw1pERDzR3BEGYFV9uc+VzIxnYeCcexzoHmeVS4F/cynPADVmtsirekREvLC7sx8gq08Rgb99BkuAfSNet6aXvYmZXWtmG81sY0dHx5wUJyIyGbs6wpjB8royv0uZET/DYLS7RI92v2Wcczc755qcc00NDQ0elyUiMnm7O/pprC2lpDDodykz4mcYtAJLR7xuBNp8qkVEZFp2d4RZVZ/dp4jA3zC4D7gqParorcBh59wBH+sREZmSZNLR3BlmVUN2dx6DhxedmdkdwDlAvZm1Al8GCgGcczcBDwIXATuBCPBRr2oREfHCwd5BBmKJrB9WCh6GgXPuigned8B1Xm1fRMRru4eHlebAkYGuQBYRmabhYaW5cGSgMBARmabdHWHKi4LMryz2u5QZUxiIiEzTro5+VjVUYDbaSPnsojAQEZmm3R25MZIIFAYiItMyGEvQdnggJ64xAIWBiMi0NHeGcS43RhKBwkBEZFpyaVgpKAxERKZld0dqWOnKLJ+6epjCQERkGnZ3hllcXUJZUW7cPVhhICIyDbvTw0pzhcJARGSKnHM5NawUFAYiIlPW0T9E31A86291OZLCQERkil4fSaTTRCIieSvXhpWCwkBEZMp2d/RTUhhgcXWp36XMGoWBiMgU7eroZ0VdOYFA9k9QN0xhICIyBb2DMZ7a1cXpK+b5XcqsUhiIiEzBA1sOMBRP8r4NjX6XMqsUBiIiU3DXpn2sXVDByY3VfpcyqxQGIiKTtLO9j817e3j/hqU5cUObkRQGIiKTdNemVoIB47LTlvhdyqxTGIiITEI8keTe5/dz7rHzaciBex4fTWEgInlvKJ7AOTfuOo/v6KCjb4j3N+VWx/EwhYGI5LWWzjCnfeURrrr1WXa294253p3PtVJXXsQ7jps/h9XNHYWBiOS1bz/yGknn2LKvhwtveIKvPrCN3sHYG9bpDkd5dPshLjttCYXB3PzZzI27MoiITMPLbYe5f0sb1527mo+dtZJvP/Iat/6+mV++sJ+LT1rE8YuqOG5RFU/v6iKWcDl7iggUBiKSx7718KtUlxZy7dtXU11ayNffcxJXvmUZ33z4Ve7e1Eo4mjiy7klLqjluYZWP1XpLYSAieenZ5m4ee7WD6991HNWlhUeWn7ikmn/72FtIJh2toQG2HejltUN9vH1tg4/Vek9hICJ5xznHNx/azvzKYj5y5opR1wkEjGV1ZSyrK+PCExfObYE+yM2eEBGRcfxuezsb94T47+cdQ2lR0O9yMoLCQETySjLp+NbDr7KirowPNC31u5yMoTAQkbwRTyT53N1b2H6wj8++89icHSY6HZ7+SZjZhWb2qpntNLPrR3l/mZn9h5ltNrMXzewiL+sRkfw1FE9w3c+e597n9/PZ89fy7pMX+V1SRvGsA9nMgsD3gfOBVuA5M7vPObdtxGp/D9zpnPtnM1sHPAis8KomEclPkWicP//JJp7Y0cmX/9s6PnrWSr9LyjhejiZ6C7DTObcbwMx+DlwKjAwDBwwP3K0G2jysR0TyQO9gjIe3HmQoniSRdMSTjl+92MYL+3r45vtOVj/BGLwMgyXAvhGvW4EzjlrnH4DfmNmngXLgPA/rEZE8cNNju/jBY7vesKykMMCNV6znYp0aGpOXYTDanR+OnhbwCuA259y3zexM4CdmdqJzLvmGLzK7FrgWYNmyZZ4UKyLZzznHr146wB+truOGD51KQSBAMGCUFAYoLtAQ0vF42YHcCow8HmvkzaeBrgHuBHDOPQ2UAPVHf5Fz7mbnXJNzrqmhIbevAhSR6Xu5rZc9XREuOWUx8ytLmFdeRHVpoYJgErwMg+eAY8xspZkVAR8C7jtqnb3AnwCY2fGkwqDDw5pEJIc9+NIBggHjnSfk/hXDs82zMHDOxYG/BB4GXiE1auhlM/uKmV2SXu2zwCfMbAtwB3C1m+gOEyIio3DO8WD6FNG88iK/y8k6ns5N5Jx7kNRw0ZHLvjTi+TbgLC9rEJH88HJbLy1dEf78j1f7XUpW0uV3IpIThk8RXaBTRNOiMBCRrKdTRDOnMBCRrLftQOoU0UUn6TqC6VIYiEjW0ymimVMYiEhWS50iOsiZq3SKaCYUBiKS1bYd6KW5M6xTRDOkMBCRrBVLJLnx0Z3pU0QL/C4nqykMMkhzZ5ifPN3Cod5Bv0sRyXiDsQR/8dNNPPTyQT5/wbHUVRT7XVJW8/SiM5lYLJHk0VcO8dNn9vLkzk4Avvnwq3zxouP54OlLMRttvj+R/NY/FOfjP36OPzR389XLTuTP3rrc75KynsJgDjV3hvn11gO09w7R2Z967GwP09k/xOLqEj57/lr+aE0933xoO9ff+xK/fKGNb1x+EgurS9jZ3s+rB/vY0d5PwKCmrJCa0iJqygqZX1XC4poSGiqKFR6S80LhKFf/6Fm2tvVywwdP5dJTl/hdUk6wbJsKqKmpyW3cuHHOtzsYS3DVD5/lUN8gxy6o5LiFlRy7sIrTV9Qyv6pk3M8ePDzIdx7dwZ0b95FIOiqLC6ivLKauvIhFNaVccspi3nHcfIKB1A95Mun4+XP7+MaDrzAYT5B0kEim9lNh0HAO4sk377eiggCLq0s4qbGGy05dzNvXNmTFPV4PR2I0d4UpCBhFBQEKAkYwYAzFkwzFkgzGEwxEE/QOxjg8EKN3IE54KE5DZTFL55WybF4ZjbVllBRqZspcNhhLcPsf9vKD/9hJ31CcH1y5nvPWqZ9gssxsk3Ouaaz3dWQwSV/65VaebenmvOPns7Ojn9++coikg4DB2cc08L4NjZy/bsGRH6T+oTgtnWHu39LGbU+1kHSOD5+xjOvOXTNheAQCxpVnLOMdx83n5sd3U14c5NiFqQBaUVdOMGCEowlC4Sg9kRiHegfZ3zNAW88AraEBntzRwf1b2phXXsS7T17Ee9c3csrSmkm1M55I8vzeHg4PxIhE40SiCWKJJGsaKjh5aQ0VxTP7X2YwlqCzf4jXDvXx9K4untrVxbYDvUz13yRmvOkzx8yvoGnFPJqW19K0opbG2rIjAQupIYhd4Si72vvZ1RGmuCDAWWvqWVg9/v4Qf8UTSe59fj83/PY12g4P8rY19Vz/ruM4cUm136XlFB0ZTMJdG/fx+btf5C/PXcPnLjgWSP2ovXaoj0e2HeKeTa20HR6kqqSAtQsq2dMdoaNvCEj9aL3ntCX89XlrWTqvbE7qjSWSPP5aB7/YvJ9Hth1iKJ5kw/JarnnbSt65bgEFoxwtJJKO+7bs5zu/3UFLV2TU7zWDtfMrOWVpNWVFBcQSSeIJRyyZfMNtixwQjSePhEkkmuDwQIyu/iHC0cSR9YoKAqxfVsOZq+o5YXEVSeeIJVzqe5OO4oIAJYVBigsClBYFqSoppLo09SgpDNDZH2Vvd4R93RGaO8Nsae1h054QfYPxI9soLQxSUVJARXEBoUgqPI+2dkEFb1vTwOnpAFlck5oHX6fc/Le3K8KnfraJrft7OWVpDX97wbGcteZNtzyRSZjoyEBhMIHtB3u57Pu/57Sltfz042e84V+aw5JJx1O7urh70z7aegZZXlfGyoZyVtaVc+KS6jkLgdH0Dca4Z1Mrt/6+hb3dERprS7l8fSOLqkuoLSuirqKItp4BvvvoDnZ1hDluYSWfOncNK+vKKS0KUl4cJGjGtgO9vLCvh817e9i6/zDRRJKiYICCoFEQCBA4Kl8KgwHKiwooLQpSlv4hr68opq6iiIaKYhrnlbJ+We2sn9pJJh2vtfexaU+Ijr4hwkNx+ofi9A3GqSwpZM38CtbMr2B1Qzl9g3Ge2NHBEzs6eba5m6H46zfYKy4IsKSmlIXVJSysLmFRdapPpqyogJKiIKWFqT+blfXlLKwqUXB44NFXDvHX//8FAL5++UlcfNIi/TnPgMJgHLFEctxz6v1DcS753pP0Dcb51WfexvzK7D2dkEg6Htl2iB8+uZvnWkJven/tggr+6ry1XHjCQgKjBF6uG4wl2Nnez/6eAQ70DNB2OHXq7eDhQQ4eHuRQ7+Co/TQAFcUFrE4HTFVJIcXpWywWFwSoryhiQVUJi6pLWVhVgsPRE0n1ffQMxOgbjBFOh1V4KEHPQJTO/iidfUN0hYeIRBNHjoxKCoIEA8ZALEF46PVTeA2VqbBaWF3CgsoSastTR09VpYVUlaSOooqCAQqDAQoLAgQMAmYYYGap0C8KTuqH1jlHa2iA7Qf7eOVAL9sP9rI/NEBhMEBxejtlxQUsn1fGqoYKVjWUs7q+guqywjd9z472fn7z8kH+87UO6iuKeeuqOt66qo7VDeV859Ed3Pi7naxbVMVNH97Asjr//kGVKxQGY/jd9kNcd/tm/u7i40cdlpZMOj798838+qUD3P7xt3Lm6roZbzNTDEQThCJRusNRQpEohnHm6rpRj3okJZF09ESiDMQSDMYSDMaS9A7E2N0ZZmd7Pzva+9jdESY8FE91fMeTE3/pKCqKC6ivKKK+opj6imLKioIMxZMMxhIMxBLEk47yoiBlxQWUF6XCoaNviAPp0OoKR6e13YCltl1ZUsixCyt598mLOH/dAipLUj/inf1D/OL5/fz8ub3s6ggf+dzyujKWzSsjkXRE0+3uG4zRGhp4Q3hWlhTQWFtGY20pdeVF/KG5m+bO1Pec3FhNdzhKa2gASN28fjCW5P0bGvnqZSdqYMAsURiMYn/PABd95wki0TjxpOOfPnAql532+vC0WCLJ5+7awi9faOP6dx3HJ3WzDJki5xxD8SSd/UMcPDzIgfTRRcCMmrLX+z6qSgspL071aZQXBUftz5mKaDx5ZNTV8CMaTxJLpB7ReJKkS3W+J53DOcdALEHfYOro5PBAjGebu9nfM0BRQYBz1jYQDBi/feUQsYRjw/JaLjllMScuqea4hZWUjzGgIJZIsrc7QnNHmObOMK2hCK2hAfb3DHCod5CTGmu44IQFnH/8giMDKvZ1R/hDczeb9nRz+op5XL6+cUZ/FvJGCoOjRONJPnjz0+w41M89f/FH/MN9L/NsSzf/8uENnLduAYOxBNfd/jyPbm/nby88lk+ds2YWqxfJfMmkY/O+Hh54sY1fvXiAeNJx+WlL+ODpSzlmQaXf5ck0KQyO8r8f2MYtTzbz/SvXc/HJi+gfivOn//oMrxzs43tXnMYPn2zm2ZZuvnrpiXxYVzVKnhv+fVDHbfabKAwy/4qkWfSblw9yy5PNXHXmci4+OTXDYUVxAbd99C2sqCvj2p9sYtOeEDd88FQFgQipEFAQ5Ie8CYN93RE+d9cWTlpSzRcvPv4N79WWF/HTa87gghMW8K9XNenydhHJO3lzBfKO9j5Ki4J8/8r1FBe8eXTC/KoS/uXPxjyCEhHJaXkTBu84bgH/+fl6DVMTERlF3pwmAhQEIiJjyKswEBGR0SkMREREYSAiIgoDERFBYSAiIigMREQEhYGIiJCFE9WZWQewZ8SiauDwJJ/XA50z2PzI75zqOqMtP3rZeK+Hn49cNpP2zKQtY703mfrHeq59M3Gdk11H++bN+yPb2zLW86m0Z7lzrmHMd116TvNsfQA3T/Y5sHG2tjXVdUZbfvSy8V6PaMPIZdNuz0zaMp32aN9o38z1vsmltnjVnpGPXDhNdP8Un8/Wtqa6zmjLj1423uv7x1hnumbSlrHem0z94z2fCe2b8d/Lx32TS20Z7/msyLrTRDNhZhvdOPN5Z5tcak8utQVyqz1qS+aazfbkwpHBVNzsdwGzLJfak0ttgdxqj9qSuWatPXl1ZCAiIqPLtyMDEREZhcJAREQUBiIiojA4wszONrObzOwWM3vK73pmwswCZvY1M7vRzD7idz0zZWbnmNkT6f1zjt/1zJSZlZvZJjN7t9+1zJSZHZ/eL3eb2V/4Xc9MmNllZvavZvZLM3un3/XMlJmtMrMfmtndk1k/J8LAzG41s3Yz23rU8gvN7FUz22lm14/3Hc65J5xznwQeAH7sZb3jmY22AJcCS4AY0OpVrZMxS+1xQD9Qgo/tmaW2AHwBuNObKidvlv7evJL+e/MBwLchm7PUln93zn0CuBr4oIflTmiW2rPbOXfNpDc6W1ev+fkA3g6sB7aOWBYEdgGrgCJgC7AOOInUD/7Ix/wRn7sTqMrmtgDXA3+e/uzd2b5vgED6cwuA27O8LecBHyL1g/PubN836c9cAjwFXJntbUl/7tvA+lzYN+nPTeo3oIAc4Jx73MxWHLX4LcBO59xuADP7OXCpc+4bwKiH52a2DDjsnOv1sNxxzUZbzKwViKZfJryrdmKztW/SQkCxF3VOxiztm3OBclJ/iQfM7EHnXNLTwscwW/vGOXcfcJ+Z/Qr4mXcVj22W9o0B/wj82jn3vLcVj2+W/95MSk6EwRiWAPtGvG4FzpjgM9cAP/KsoumbalvuBW40s7OBx70sbJqm1B4zuxy4AKgBvudtaVM2pbY4574IYGZXA51+BcE4prpvzgEuJxXSD3pa2dRN9e/Np0kduVWb2Rrn3E1eFjcNU903dcDXgNPM7H+kQ2NMuRwGNsqyca+wc8592aNaZmpKbXHORUgFW6aaanvuJRVwmWjK/58BOOdum/1SZsVU981jwGNeFTNDU23Ld4HvelfOjE21PV3AJyf75TnRgTyGVmDpiNeNQJtPtcxULrUFcqs9udQWyK325FJbwOP25HIYPAccY2YrzayIVKfdfT7XNF251BbIrfbkUlsgt9qTS20Br9vjZ4/5LPa83wEc4PWhlNekl18EvEaqB/6LfteZb23JtfbkUltyrT251Ba/2qOJ6kREJKdPE4mIyCQpDERERGEgIiIKAxERQWEgIiIoDEREBIWB5Agz65/j7d1iZutm6SG8rmUAAAL/SURBVLsSZvaCmW01s/vNrGaC9WvM7FOzsW2RYbrOQHKCmfU75ypm8fsKnHPx2fq+CbZ1pHYz+zHwmnPua+OsvwJ4wDl34lzUJ/lBRwaSs8yswczuMbPn0o+z0svfYmZPmdnm9H+PTS+/2szuMrP7gd9Y6g5rj1nqLl7bzez29DTHpJc3pZ/3W+rOclvM7BkzW5Bevjr9+jkz+8okj16eJjU7JWZWYWaPmtnzZvaSmV2aXucfgdXpo4lvpdf9fHo7L5rZ/5rFP0bJEwoDyWXfAf7JOXc68F7glvTy7cDbnXOnAV8Cvj7iM2cCH3HOvSP9+jTgr0jdf2AVcNYo2ykHnnHOnUJqyvBPjNj+d9Lbn3BCMTMLAn/C6/PNDALvcc6tB84Fvp0Oo+uBXc65U51zn7fULRqPITXf/anABjN7+0TbExkpl6ewFjkPWJf+xzxAlZlVAtXAj83sGFJTABeO+MwjzrnuEa+fdc61ApjZC8AK4MmjthMldXcpgE3A+ennZwKXpZ//DPi/Y9RZOuK7NwGPpJcb8PX0D3uS1BHDglE+/870Y3P6dQWpcMjEe1lIhlIYSC4LAGc65wZGLjSzG4H/cM69J33+/bERb4eP+o6hEc8TjP53JuZe73wba53xDDjnTjWzalKhch2pefX/FGgANjjnYmbWQuo+0Ecz4BvOuX+Z4nZFjtBpIsllvwH+cviFmZ2afloN7E8/v9rD7T9D6vQUpKYbHpdz7jDwGeBzZlZIqs72dBCcCyxPr9oHVI746MPAx8xsuBN6iZnNn6U2SJ5QGEiuKDOz1hGPvyH1w9qU7lTdxut3ffom8A0z+z2pm4x75a+AvzGzZ4FFwOGJPuCc20zqRucfAm4nVf9GUkcJ29PrdAG/Tw9F/ZZz7jekTkM9bWYvAXfzxrAQmZCGlop4xMzKSJ0Ccmb2IeAK59ylE31OxA/qMxDxzgbge+kRQD3Ax3yuR2RMOjIQERH1GYiIiMJARERQGIiICAoDERFBYSAiIigMREQE+C/GoIbBsaJwdQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.lr_find(suggestions=True)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.LineCollection at 0x7f94b6d2cc90>"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEKCAYAAADw2zkCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhkZZn38e+dfesknaX3fWFfGogNqDiNICDj0G7jgCujwjhur+PIq77O6Iwjo6PjzKij4/QgoLJ4KTgICOJy2SJLI91AQ9tN0xudTm9JJ+nuJJVUparu94+qQDpkT05OVer3ua66UstTde6HdOrHc55znmPujoiI5La8sAsQEZHwKQxERERhICIiCgMREUFhICIiKAxERAQoCLuAsaqrq/MlS5aEXYaISFbZtGnTEXevH+r1rAuDJUuWsHHjxrDLEBHJKma2d7jXtZtIREQUBiIiojAQEREUBiIigsJARERQGIiICAoDEZGs8IsthzjSGQ3s8xUGIiIZ7uCxbj5251N86zc7AttGYGFgZjebWbOZbRmmzRoze8bM/mhmvwuqFhGRbLbu4d0kHT540bLAthHkyOBW4IqhXjSzauA7wFXufjrw5wHWIiKSlY50RrnzD428edV8FtaUBbadwMLA3R8G2oZp8k7gp+7emG7fHFQtIiLZ6uZH9hCNJ/nwxcsD3U6YcwYnATPNbL2ZbTKz94ZYi4hIxjnW3csPH9/LlWfMZXl9RaDbCnOhugLgPOASoBR43Mw2uPsLAxua2fXA9QCLFi2a0iJFRMLyg8depCMaD3xUAOGODJqAX7h7l7sfAR4Gzh6sobuvc/cGd2+orx9yBVYRkWmjKxrn5kf38PpTZnH6vKrAtxdmGPwMuMjMCsysDDgf2BZiPSIiGeOOJxppj/TykYtXTMn2AttNZGZ3AmuAOjNrAr4AFAK4+3fdfZuZ/QJ4FkgCN7n7kIehiojkingiyf/8fjcXLqvlvMUzp2SbgYWBu18zijZfA74WVA0iItmosS1Cc0eUGy4/ecq2qTOQRUQyzK6WLgBWzAr2CKL+FAYiIhlmV0snAMsCPpy0P4WBiEiG2dXcSf2MYqpKC6dsmwoDEZEMs/tIF8vry6d0mwoDEZEM4u7sbO6c0l1EoDAQEckobV0xjnX3Br78xEAKAxGRDNJ3JJF2E4mI5LC+I4k0MhARyWG7WzopLshjfnXplG5XYSAikkF2tXSxtK6cvDyb0u0qDEREMsiulk6WT+GZx30UBiIiGSIaT7CvLTLl8wWgMBARyRh7WyMkfeqPJAKFgYhIxtjVHM6RRKAwEBHJGH2HlS6t08hARCRn7WrpYl5VCeXFU395eoWBiEiGCOtIIlAYiIhkBHdnd0tXKPMFoDAQEckIzR1ROqNxloVwJBEoDEREMkKYRxKBwkBEJCOEtUBdH4WBiEgG2NXSRXlRPrMri0PZvsJARCQD7GpJXd3MbGoXqOujMBARyQCpI4nCmTwGhYGISOgisTj7j3aHNl8ACgMRkdDtbY0AsFQjAxGR3NXYlgqDxTUKAxGRnLUvHQaLaspCq0FhICISssa2CJUlBVSVFYZWg8JARCRkjW0RFtWGNyqAAMPAzG42s2Yz2zJCu1eZWcLM3h5ULSIimayxLRLqLiIIdmRwK3DFcA3MLB/4F+ChAOsQEclYyaTT1NbNwukaBu7+MNA2QrOPAXcDzUHVISKSyQ539BBLJKf1yGBYZjYfeAvw3VG0vd7MNprZxpaWluCLExGZIo2t4R9JBOFOIP8H8Gl3T4zU0N3XuXuDuzfU19dPQWkiIlNjbwYcVgow9RfafFkD8KP0okx1wJVmFnf3e0KsSURkSu1ri5BnMK+6NNQ6QgsDd1/ad9/MbgXuVxCISK5pbIswr7qUwvxwj/QPLAzM7E5gDVBnZk3AF4BCAHcfcZ5ARCQXZMJhpRBgGLj7NWNoe21QdYiIZLJ9bREuPXV22GXoDGQRkbB0ReMc6YyFfo4BKAxEREKzrz0zjiQChYGISGgy5RwDUBiIiISmMUPOMQCFgYhIaPa1RZhRXEB1iEtX91EYiIiEpLEtwsKaMtIn34ZKYSAiEpLGtgiLQ76OQR+FgYhICJJJZ197d0bMF4DCQEQkFM0dUWLxZEacYwAKAxGRUGTSkUSgMBARCYXCQEREaMyQpav7KAxEREKwry3C3KpSigoy42s4M6oQEckxmbJ0dR+FgYhIwLpjCW76/W52Nne89FymhUGYl70UEckJv93ezJd+vo0v/XwbF62s413nL6alI8qiDDnhDDQyEBEJ3JHOKAB/9bplbD/UwYdu2wSQMecYgEYGIiKBa+2MAXDD5Sfzt5edzINbDvK77S28dkVdyJW9TGEgIhKwtq4Y1WWFFKQver921XzWrpofclUn0m4iEZGAtXXFqCkvCruMYSkMREQC1tYVo6ZMYSAiktM0MhAREVq7YtRWKAxERHJWMum0RzQyEBHJacd7ekkknZry4rBLGZbCQEQkQK1dqXMMajUyEBHJXW3pMJipMBARyV1tGhmIiEhfGGgCWUQkh+V8GJjZzWbWbGZbhnj9XWb2bPr2mJmdHVQtIiJhae2MUV6UT0lhftilDCvIkcGtwBXDvL4H+BN3Pwv4J2BdgLWIiISirStKTYafcAYBrlrq7g+b2ZJhXn+s38MNwIKgahERCUtrVyzjzzGAzJkz+ADw4FAvmtn1ZrbRzDa2tLRMYVkiIhPTHolRU1YYdhkjCj0MzOxiUmHw6aHauPs6d29w94b6+vqpK05EZILaOrNjZBDqxW3M7CzgJuCN7t4aZi0iIpPN3bNikToIcWRgZouAnwLvcfcXwqpDRCQokViCaDyZ8YeVQoAjAzO7E1gD1JlZE/AFoBDA3b8LfB6oBb5jZgBxd28Iqh4RkamWLecYQLBHE10zwusfBD4Y1PZFRMKWLYvUQQZMIIuITFftWbJIHSgMREQCo5GBiIjQ1hUFsmPOYFRhYGbLzaw4fX+NmX3czKqDLU1EJLu1dsUoys+jojjUo/hHZbQjg7uBhJmtAL4HLAXuCKwqEZFpIHXCWRHpIyYz2mjDIOnuceAtwH+4+98Ac4MrS0Qk+7V1xbJiFxGMPgx6zewa4H3A/ennMn+xDRGREGXL2ccw+jD4S+BC4EZ332NmS4HbgitLRCT7tUdizCzLjjAY1ayGu28FPg5gZjOBGe7+lSALExHJdn1zBtlgtEcTrTezSjOrATYDt5jZvwVbmohI9orGE3RE41lxjgGMfjdRlbsfB94K3OLu5wGXBleWiEh2a+/qBciKq5zB6MOgwMzmAu/g5QlkEREZQmv6hLPpNjL4IvAQsMvdnzSzZcCO4MoSEcluL69YmvkXtoHRTyD/BPhJv8e7gbcFVVQQ1qxZA8D69etDrUNEcsNEl6+e6u+s0U4gLzCz/zWzZjM7bGZ3m5kuYC8iMoRsupYBjH430S3AvcA8YD5wX/o5EREZRFtXjDyD6tLsOD93tGFQ7+63uHs8fbsV0JXpRUSG0NqVOuEsLy/z1yWC0YfBETN7t5nlp2/vBnQBexGRIWTTCWcw+jB4P6nDSg8BB4G3k1qiQkREBpFNi9TBKMPA3Rvd/Sp3r3f3We7+ZlInoImIyCDaItmzSB1M7Epnn5y0KkREppm2ruxZpA4mFgbZMSsiIjLFEkmnPRLLmrOPYWJh4JNWhYjINHI0EsM9e84xgBHOQDazDgb/0jegNJCKRESy3EsnnFVkx1IUMEIYuPuMqSpERGS6aE2HQa7sJhIRkUEcPt4DTKPdRCIiMjruzlON7dzy6Iv8YsshigrymFtVEnZZo6YwEBGZoBePdPGxO5/muf3HmFFSwLWvXsJ7L1xCdRYdWqowEBGZoNs27GX7oQ6+9OYzeMs58ykvzr6v1sDmDMzs5vSS11uGeN3M7JtmttPMnjWzc4OqRUQkSDuaO1kxq4J3X7A4K4MAgp1AvhW4YpjX3wisTN+uB/4rwFpERAKzs7mTlbMrwi5jQgILA3d/GGgbpsla4AeesgGoTl9nWUQka3RF4+w/2s2KeoXBeM0H9vV73JR+TkQka+xq6QTQyGACBlvbaNAlLszsejPbaGYbW1paAi5LRGT0dhxOhcGKWdl9jm6YYdAELOz3eAFwYLCG7r7O3RvcvaG+XhdYE5HMsaO5k4I8Y3FtWdilTEiYYXAv8N70UUUXAMfc/WCI9YiIjNnO5k6W1pVTmJ/dCzoEdgyUmd0JrAHqzKwJ+AJQCODu3wUeAK4EdgIRdOU0EclCO5s7OG1eZdhlTFhgYeDu14zwugMfCWr7IiJB6+lN0NgW4aqz54VdyoRl97hGRCREe450kXRYMTu7J49BYSAiMm47mtOHlc7K7sNKQWEgIjJuOw93kGewtK487FImTGEgIjJOO1s6WVRTRklhftilTJjCQERknHYc7sz6k836KAxERMahN5Fkz5GurF+Goo/CQERkHPa2RognPesXqOujMBARGYedzR1A9i9Q10dhICIyDn0L1C3XyEBEJHftbOlkfnVp1l7ZbCCFgYjIOKSOJJoeowJQGIiIjFki6exq6ZwWZx73URiIiIzR/vZuovGkRgYiIrlsxzQ7kggUBiIiY9a3QN2K+ulx9jEoDERExux321tYXFtGVVlh2KVMGoWBiMgY7Grp5PHdrbyjYeHIjbOIwkBEZAzufKKRgjxTGIiI5Kqe3gR3PdXE5WfMoX5GcdjlTCqFgYjIKD245SBHI728a/WisEuZdAoDEZFRun1DI8vqyrlweW3YpUw6hYGI5LytB46z7eDxYds8f+g4G/e2887zF2FmU1TZ1JkeKyyJiIxTa2eUv1j3OB09cdaumsenLjuZhTVlr2h3xxONFBXk8bZzF4RQZfA0MhCRnPaN3+wgEkvw3gsX84sth7jk67/jxp9v5fDxnpfaRGJx/vep/bzpzLnMLC8KsdrgaGQgIjlrx+EObn+ikXeuXsQX157BX69Zzr//6gVuemQP//P7PdSUF3HS7ApKCvPpiMZ55/nTb+K4j8JARHLWjQ9so6won795w0kAzK0q5atvP5vrLlrG73cc4YXDHWw/3MFzTcc4d1E15y2eGXLFwVEYiEhO+t0LLazf3sLnrjyVmgG7flbOnsHK2S+vO+TuANNy4riPwkBEck48keRL929lcW0Z73314hHbT+cQ6KMJZBHJOT96ch87mjv57BtPobggP+xyMoLCQERyyqM7j/CVB59n9dIaLj99TtjlZIxAw8DMrjCz7Wa208w+M8jri8zst2b2tJk9a2ZXBlmPiOS2nz7VxLW3/IH51aV84+pVObH7Z7QCCwMzywe+DbwROA24xsxOG9Ds74Afu/s5wNXAd4KqR0Ryl7vz7d/u5JM/3kzD4hp+/KELmVtVGnZZGSXICeTVwE533w1gZj8C1gJb+7VxoDJ9vwo4EGA9IpID3J3bnmjk4NFuEkknnnQa2yL8auth1q6ax1fffpbmCQYRZBjMB/b1e9wEnD+gzT8AvzSzjwHlwKWDfZCZXQ9cD7Bo0fQ96UNEJm5z0zH+/p4t5OcZhflGvhmFBXl89OIVfPINJ5GXp11DgwkyDAb7L+4DHl8D3OruXzezC4EfmtkZ7p484U3u64B1AA0NDQM/Q0TkJfc8vZ+igjw2/t2lVJZMn8tSBi3ICeQmoP+lgBbwyt1AHwB+DODujwMlQF2ANYnINBZPJLn/2QNccsosBcEYBRkGTwIrzWypmRWRmiC+d0CbRuASADM7lVQYtARYk4hMY4/uauVIZ4y1q+aHXUrWCSwM3D0OfBR4CNhG6qihP5rZF83sqnSzvwWuM7PNwJ3Atd533reIyBj97On9VJYUcPEp9WGXknUCXY7C3R8AHhjw3Of73d8KvCbIGkQkN0RicR764yH+7Ox5OlpoHHQGsohMC7/aepiuWEK7iMZJYSAi08LPnjnA3KoSzl9aE3YpWUlhICJZr60rxsMvtHDV2fN0HsE4KQxEJOv9/NkDxJOuXUQToDAQkax3zzMHOGl2BafOnTFyYxmUwkBEstqDzx1k09521q6ar1VIJ0BhkGF6ehPct/kAX//ldg4d6wm7HJGM9r1H9vDhO57inEXVvOfCka9YJkPTZS8zQDLpbD14nJ9s3Mc9zxzgWHcvADc/sodPXHoS175mCYX5ym2RPomk80/3b+XWx17kitPn8B9Xr6KkUOcWTITCYBxaO6Mc6+5lUU0ZBWP8ku6Kxrltw15+90ILrZ0xWrtitEdiJJJOUUEeV5w+h3c0LGTBzFK+eP9WbnxgGz/ZtI8bLj+FaDzBjsOd7GzppLE1QmlhPpWlhVSXFVJdWsjsyhLmVJUwr7qEuVWlzK0q0bBZpp3Wziif/elz/HLrYT7w2qX8vytPJV9HEE2YwmCMth/q4K3feZSuWIKi/DyW1ZezYlYFq5fWcOWZc6mrKB70fZ3RON9/7EVu+v1u2iO9nDm/isW1ZZy7uJra8mIW1pRyxelzqSp7eXGt772vgV9tPcw/3reV636wEQAzWFxTxuLacqLxBE3tEbYe6KUtEqOn94TFXllcW8afnjmXK8+cy+nzKjM2GI50Rtn4Yhubm44RTyQpzM+jID+PwjwjnnSi8STReIJoPElXNM7x7l6O96R+lhUXsKC6lPkzS1kws5QV9RWctbCaimL9055uWjqirHt4F7dtaCQaT/D5N53G+1+7NOyypg39xYzB0UiM636wkbLiAv7+Taexp7WLnYc7eWbfUe5/9iD/eN9WXruijrWr5rGsvoK9rV00tkbY2xbh19sOczTSy8Un1/PxS1ZyzqKZI27PzLjs9DlctLKex3YdYU5VCcvrKwYdDrs7x3viHDrWw8Fj3TS2Rfj1tmb+++HdfGf9LpbUlvG2cxfwF6sXMmtGyYj9/OHje3l2/zG6YwkisTiRWIKyonzOnF/FmQuqOWtBFUtqy0m605tIEk84iQHLSiWTTndvgkgskfoZTdDaFaWtK0ZbV4xDx3rY1NjO7pYuAArzjYK8vNTnJV/+rKKCPIrTt4riAipLC6ksKWTWjGI6o3G2HTzOr7YdJhZPpv+7wUmzZnDOomqW11dQXlxAeXE+5UUFRONJXmztYndLF3uOdBKNJ3nVkhouXF7LBUtrTwhjyQzHunv55m92cPsTe4nFk6xdNZ+Pvn4Fy+srwi5tWrFsWxeuoaHBN27cOOb3rVmzBoD169ePa7vxRJJrb3mSP+xp487rL+C8xSd+mW8/1MG9m/fzs2cO0NTefcJrsyuLWbWwmg+vWcHZC6vHtf3xauuK8cs/HuLezQd4bFcrhfnGFWfM5T0XLOZVS2aeMFpoPt7DTY/s4fYNe+mKJThpdgUVxQWUFRVQUpjP8e5ethw4RiSWmHBdZlBbXsRZC6p51ZIaVi+dyRnzq15aU8bd6U04BXk2qpOIkknnSGeUbYc6eLqxnacbj/LMvqMvzb8MNLuymKV15eSZ8VRjOz29Scxg5awKFswsY05VCXPTu93mVJUwu7KE2TNKqCwtyNgR1nS0aW8bH7/zGQ4d7+HNq+bzkYuXsyxHQmCi31kDmdkmd28Y6nWNDEbpyw8+zyM7j/DVt5/1iiAAOHnODG6Ycwqfuuxknt53lNbOGItry1g4s4zSovAmtmrKi7h69SKuXr2I3S2d3LahkZ9s2sd9mw9QkGep+YayIipLCtiy/zjxZJI/O3sef71mOafMqXzF5yWSzp4jnTzbdIz97d3k5xuFeXkU5Bv5eXbCFY3MjNLCfEqLUreywnxqyouoKS+iuqxo2P28ZkZRwei/dPPyjFmVJcyqLOFPTkqtWOnudETjRKIJOqNxIrE4+XnGktpyyvvtRorGE2zed4zHd7XybNNRDh7r4Zl9R2nrir1iO0UFeZQV5VNSkE9xYR6lhfksqiljWX0Fy+rLWVZXTmVpIUX5eS+NaKpKC0ecW0omnUhvgkg0ztHu3vR8UmoUFe1NUlyY99I2zYzu9GitL5jrZxSnAquymLqKYqpKCwM/6CCeSLK3LcKOwx10RhMUF6T6XFSQR01ZEUtqy0ccaSWSztON7exrj3D2gmqW1pVjZiSSqWsWf+M3O5hXXcJdH7pwVKNpGb+cD4O2rhj/8uDzvOfCxZwxv2rQNndtauJ7j+zh2lcv4R0NCwdt08fMODdD/9Euq6/g8392Gp+6/CQeeO4Qu1o6ORrp5WgkNYn9jlct4LqLlrG4tnzIz8jPM1bMmsGKWZl/co+ZUVlSOOJFTooL8lm9tIbVA9a06elNcPh4D4ePR9M/e2jpjNIdS9DTm6CnNzWHsftIF7/d3kxvYvBRdp5BXUXqy3rWjGKS6V16x7t7Odbdmw6qiY+2BkodYFDAjJJUMBTlW3o+xsgzw4z0T6O8KJ+K4gIqSgqoLCmkYclMLlhWe0KgJJPOE3vauHfzfp7Zd4xdzZ3EEslhKoCZZYUsri1ncW0Zc6tKmZ8+uKErFue3zzfzuxdaaI+8PHqrLS/ivMUzaeuKsXFvO1edPY8vveUMXahmCuR0GMQTST56x1M8tquVB7cc5PYPXsCZC04MhLs3NfGZu5/l1ctr+dyfnhpSpZOrrKiAt5+3IOwyMl5JYX76i2zocOwTTyTZ197Ni61dRKIJovEEsXiSnt4EbV0xDh+P0tzRw8FjPeTnGZWlBcyurGBGcSEzSgpentcoTn0Z11YUUVteTG1FEcUFeUTTn9XTm8TdKS1KzYH0jTpbOqIvBVdLRw8dPXGO9/RyvDtOZzROLJGkt+8WdxIkSXpq9JRIOgePpkZPnT1xOmNx3KGqtJBLTp3F60+ZxbaDx7nn6QPsP9pNeVE+DUtqeN3KOlbOnsFJsyuoKi0kFk8SjSeJJZIc6YiytzXCntYuXjzSxdONR3ng2METArOmvIiLT57F60+dxbK6CjY3HWXji+1s3NtGe1eMf/3zs3nbuTqRbKrkdBj88wPP89iuVm64/GTu/EMj77ppA7d98HzOWlCNu/PN3+zk33/9Aq9ZUct/vfs8HesvQyrIz2NpXTlL60YOjvEYaRy2sKaMhTVlk7Ktnt4ED7/QwkN/PMyvtx3mp0/tJ8/gopX1/N8rTuay0+aMa9dnMukc6Ypy4GgPBpwxv+qEXYWnzavkmtWLgFRIKQSmVs6Gwd2bmrj50T385WuW8JGLV7B21TyuXreBd9/0BLf85Wru/EMjd21q4m3nLuDLbz2TogIFgeSGksJ8Ljt9DpedPofeRJJnm46ycGYZsyqHPwptJHl5xqwZJSMezQYoCEKQk2Gwed9RPvu/z6V2/VyZ2vWzYGYZP7r+Aq5et4G3/ddjAPyfS1byiUtX6h+m5KzC/DzOW6zrA+SCnAuD5o4e/uqHm6ivKOY/33nuCUd59AXCp+9+ljevms+fjzBZLCIyXeRcGDS1d5NnsO6951FTXvSK1xfMLOP2D14QQmUiIuHJuTA4d9FM1t9wseYARET6yclvRAWBiMiJ9K0oIiIKAxERURiIiAgKAxERQWEgIiIoDEREBIWBiIigMBAREQIOAzO7wsy2m9lOM/vMEG3eYWZbzeyPZnZHkPWIiMjgAluOwszygW8DbwCagCfN7F5339qvzUrgs8Br3L3dzGYFVY+IiAwtyJHBamCnu+929xjwI2DtgDbXAd9293YAd28OsB4RERlCkAvVzQf29XvcBJw/oM1JAGb2KJAP/IO7/2LgB5nZ9cD1AIsWLRpXMevXrx/X+0REwjDV31lBjgwGuyLMwCuGFwArgTXANcBNZlb9ije5r3P3BndvqK+vn/RCRURyXZBh0AT0vzrMAuDAIG1+5u697r4H2E4qHEREZAoFGQZPAivNbKmZFQFXA/cOaHMPcDGAmdWR2m20O8CaRERkEIGFgbvHgY8CDwHbgB+7+x/N7ItmdlW62UNAq5ltBX4L3ODurUHVJCIigzP3gbvxM1tDQ4Nv3Lgx7DJERLKKmW1y94ahXtcZyCIiojAQERGFgYiIoDAQERGycALZzFqAvUAVcKzfS8M97rvf/7k64Mg4yxi4rbG0Gex59UV9GW2do2mjvmRmXwZ7baT+jXR/LH1Z7O5Dn7Xr7ll5A9aN9nHf/QHPbZysbY+lzWDPqy/qi/oy/fsymtqHqn+Yfo27LwNv2byb6L4xPL5viDaTte2xtBnsefVlcqgvQz+vvkyOifRlsNdG6t9o7k+KrNtNNFnMbKMPc8xtNlFfMpP6kpnUl8Fl88hgotaFXcAkUl8yk/qSmdSXQeTsyEBERF6WyyMDERFJUxiIiIjCQEREFAavYGYXmdl3zewmM3ss7HomwszyzOxGM/uWmb0v7HomwszWmNnv07+bNWHXM1FmVm5mm8zsTWHXMhFmdmr6d3KXmf112PVMlJm92cz+x8x+ZmaXhV3PRJjZMjP7npndNZr20yoMzOxmM2s2sy0Dnr/CzLab2U4z+8xwn+Huv3f3DwH3A98Pst7hTEZfgLWkrkXdS+qqcqGYpL440AmUkP19Afg08ONgqhydSfp72Zb+e3kHEOrhmpPUn3vc/TrgWuAvAix3WJPUl93u/oFRb3Syzl7LhBvwOuBcYEu/5/KBXcAyoAjYDJwGnEnqC7//bVa/9/0YqMzmvgCfAf4q/d67srwveen3zQZuz/K+XErqyn/XAm/K5r6k33MV8BjwzrD6Mpn9Sb/v68C506Qvo/rbL2AacfeHzWzJgKdXAzvdfTeAmf0IWOvuXwYGHaKb2SLgmLsfD7DcYU1GX8ysCYilHyaCq3Z4k/V7SWsHioOoczQm6fdyMVBO6g+528wecPdkoIUPYrJ+L+5+L3Cvmf0cuCO4ioc3Sb8bA74CPOjuTwVb8dAm+W9mVKZVGAxhPrCv3+Mm4PwR3vMB4JbAKhq/sfblp8C3zOwi4OEgCxuHMfXFzN4KXA5UA/8ZbGljNqa+uPvnAMzsWuBIGEEwjLH+XtYAbyUV0A8EWtn4jPVv5mOkRm5VZrbC3b8bZHFjNNbfTS1wI3COmX02HRpDyoUwsEGeG/ZMO3f/QkC1TNSY+uLuEVLBlonG2pefkgq3TDTmf2MA7n7r5JcyYWP9vawH1gdVzCQYa3++CXwzuHImZHutPGEAAAOaSURBVKx9aQU+NNoPn1YTyENoAhb2e7wAOBBSLROlvmQm9SVzTaf+BNqXXAiDJ4GVZrbUzIpITdzdG3JN46W+ZCb1JXNNp/4E25cwZ/8DmIG/EzjIy4dSfiD9/JXAC6Rm4j8Xdp3qi/qSCbfp1Jfp1p8w+qKF6kREJCd2E4mIyAgUBiIiojAQERGFgYiIoDAQEREUBiIigsJApgkz65zi7d1kZqdN0mclzOwZM9tiZveZWfUI7avN7MOTsW2RPjrPQKYFM+t094pJ/LwCd49P1ueNsK2Xajez7wMvuPuNw7RfAtzv7mdMRX2SGzQykGnLzOrN7G4zezJ9e036+dVm9piZPZ3+eXL6+WvN7Cdmdh/wS0tdXW29pa7i9byZ3Z5e4pj08w3p+52WuqLcZjPbYGaz088vTz9+0sy+OMrRy+OkVqfEzCrM7Ddm9pSZPWdma9NtvgIsT48mvpZue0N6O8+a2T9O4n9GyREKA5nOvgH8u7u/CngbcFP6+eeB17n7OcDngX/u954Lgfe5++vTj88BPkHq2gPLgNcMsp1yYIO7n01qqfDr+m3/G+ntj7igmJnlA5fw8nozPcBb3P1c4GLg6+kw+gywy91XufsNlro840pS692vAs4zs9eNtD2R/nJhCWvJXZcCp6X/Zx6g0sxmAFXA981sJaklgAv7vedX7t7W7/Ef3L0JwMyeAZYAjwzYTozU1aUANgFvSN+/EHhz+v4dwL8OUWdpv8/eBPwq/bwB/5z+Yk+SGjHMHuT9l6VvT6cfV5AKh0y7hoVkMIWBTGd5wIXu3t3/STP7FvBbd39Lev/7+n4vdw34jGi/+wkG/5vp9Zcn34ZqM5xud19lZlWkQuUjpNbUfxdQD5zn7r1m9iKpa0APZMCX3f2/x7hdkZdoN5FMZ78EPtr3wMxWpe9WAfvT968NcPsbSO2egtRyw8Ny92PAx4FPmVkhqTqb00FwMbA43bQDmNHvrQ8B7zezvkno+WY2a5L6IDlCYSDTRZmZNfW7fZLUF2tDelJ1Ky9f9emrwJfN7FFSFxkPyieAT5rZH4C5wLGR3uDuT5O60PnVwO2k6t9IapTwfLpNK/Bo+lDUr7n7L0nthnrczJ4D7uLEsBAZkQ4tFQmImZWR2gXkZnY1cI27rx3pfSJh0JyBSHDOA/4zfQTQUeD9IdcjMiSNDERERHMGIiKiMBARERQGIiKCwkBERFAYiIgICgMREQH+Px2TVCvHedOgAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.recorder.plot_lr_find()\n",
"plt.vlines(9.1e-8, 0.6, 1.1)\n",
"plt.vlines(0.069, 0.6, 1.1)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.705750</td>\n",
" <td>0.633187</td>\n",
" <td>0.625000</td>\n",
" <td>00:06</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.561620</td>\n",
" <td>0.536927</td>\n",
" <td>0.820000</td>\n",
" <td>00:06</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.518291</td>\n",
" <td>0.503816</td>\n",
" <td>0.830000</td>\n",
" <td>00:06</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(3, lr_max=1e-3)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"learn.save('roberta-fasthugs-stg1-1e-3')"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3xV9f348dc7N5vsPRgJEMKeYSiyBAG3ba2C2mrraJ3VtlZt3db+/Nrptu5WwYXbIggKKpuw9x4JAbLIIjv5/P44Nzc3ISEXuEluwvv5eNxHzv2czznncyB538/9nM8QYwxKKaU6L6/2LoBSSqnWpYFeKaU6OQ30SinVyWmgV0qpTk4DvVJKdXLe7V2AxqKiokxSUlJ7F0MppTqUNWvW5Bpjopva53GBPikpifT09PYuhlJKdSgicqC5fdp0o5RSnZwGeqWU6uQ00CulVCfncW30Sil1qqqqqsjMzKS8vLy9i9Lq/P396dq1Kz4+Pi4fo4FeKdXhZWZmEhwcTFJSEiLS3sVpNcYY8vLyyMzMJDk52eXjtOlGKdXhlZeXExkZ2amDPICIEBkZecrfXDTQK6U6hc4e5Ouczn1qoO/gNh8qZMXevPYuhlLKg2mg7+Auf2EpM15ZwfGK6vYuilJnrYKCAl588cVTPu6iiy6ioKCgFUrUkAb6Dq5u4Zjvd+a0c0mUOns1F+hrampOetzcuXMJCwtrrWI5aKDv4HrHBAGwL+94O5dEqbPX/fffz549exg6dCgjR45k0qRJXHPNNQwaNAiAK664ghEjRjBgwABeeeUVx3FJSUnk5uayf/9++vXrx80338yAAQOYOnUqZWVlbiufdq/s4LzsD2YO5Ja2c0mU8gyPfbGFrVlFbj1n/4QQHrl0QLP7n3rqKTZv3sz69etZvHgxF198MZs3b3Z0gXzjjTeIiIigrKyMkSNH8pOf/ITIyMgG59i1axfvvvsur776KldddRUfffQR1113nVvKr4G+g6uorgW0Rq+UJxk1alSDfu7PPvssn3zyCQAZGRns2rXrhECfnJzM0KFDARgxYgT79+93W3k00Hdw5VVWG+DOo8UYY86aLmZKNedkNe+20qVLF8f24sWLWbhwIcuXLycwMJCJEyc22Q/ez8/PsW2z2dzadONSG72ITBeRHSKyW0Tub2J/dxFZJCLrRGSjiFzktO8B+3E7RGSa20quACirqsHmJRSUVpF5zH2/GEop1wUHB1NcXNzkvsLCQsLDwwkMDGT79u2sWLGijUvnQo1eRGzAC8AFQCawWkQ+N8Zsdcr2IPCBMeYlEekPzAWS7NszgAFAArBQRPoYY07+KFq5rLyqhpFJ4azYm8/mQ4V0iwhs7yIpddaJjIxk7NixDBw4kICAAGJjYx37pk+fzssvv8zgwYNJTU1lzJgxbV4+V5puRgG7jTF7AUTkPeBywDnQGyDEvh0KZNm3LwfeM8ZUAPtEZLf9fMvdUPaznjGG8qpahnYLJ33/MTYdKuTCQfHtXSylzkqzZ89uMt3Pz4+vvvqqyX117fBRUVFs3rzZkf773//erWVzpekmEchwep9pT3P2KHCdiGRi1ebvPIVjEZFbRCRdRNJzcrQ/uKvqHsSGBHjTJzaYTYcK27lESilP5Eqgb+rpnmn0fibwljGmK3AR8LaIeLl4LMaYV4wxacaYtOjoJpc8VE2oqLICvb+3jWHdw1h3sIBKe/BXSqk6rgT6TKCb0/uu1DfN1LkR+ADAGLMc8AeiXDxWnaYye48bfx8b4/tEU1JRzf0fb2znUimlPI0rgX41kCIiySLii/Vw9fNGeQ4CkwFEpB9WoM+x55shIn4ikgykAKvcVfizXbkj0HsxPiUaP28v/rfxMDW1J3xpUkqdxVoM9MaYauAOYD6wDat3zRYReVxELrNn+x1ws4hsAN4FbjCWLVg1/a3APOB27XHjPuXV1j9lgI+NAF8bT1wxkIrqWg7o4CmllBOXBkwZY+ZiPWR1TnvYaXsrMLaZY58EnjyDMqpmlNvb6P18rM/rfnFWx6dV+/LpGR3UbuVSSnkWndSsA6uqsQd6bxsA/eKDGZQYytPzdziadZRSnicoyKqIZWVlceWVVzaZZ+LEiaSnp7vlehroO7C6HjY+Nuu/0dvmxQMX9SX/eCVfbNBn3kp5uoSEBObMmdPq19FA34FV1tQF+vperOf0jCQlJoh3Vhxor2Ipdda57777GsxH/+ijj/LYY48xefJkhg8fzqBBg/jss89OOG7//v0MHDgQgLKyMmbMmMHgwYO5+uqrdZpiZamr0ft6139eiwg/GdGVp77azpHCcuJC/dureEq1j6/uhyOb3HvOuEFw4VPN7p4xYwZ33303t912GwAffPAB8+bN45577iEkJITc3FzGjBnDZZdd1uzEgy+99BKBgYFs3LiRjRs3Mnz4cLcVX2v0HVhdG72vreF/44Q+1qAzXXVKqbYxbNgwsrOzycrKYsOGDYSHhxMfH88f//hHBg8ezJQpUzh06BBHjx5t9hzff/+9Y/75wYMHM3jwYLeVT2v0HZgj0Hs3DPR944KJCfbju505XDWyW1OHKtV5naTm3ZquvPJK5syZw5EjR5gxYwazZs0iJyeHNWvW4OPjQ1JSUpPTEztrrWnGtUbfgTV+GFtHRJjQJ5olu3OprtEpEZRqCzNmzOC9995jzpw5XHnllRQWFhITE4OPjw+LFi3iwIGTPzcbP348s2bNAmDz5s1s3Oi+Ue4a6DuwyhprBGzjGj3A+D7RFJZVsSFTJzpTqi0MGDCA4uJiEhMTiY+P59prryU9PZ20tDRmzZpF3759T3r8rbfeSklJCYMHD+bpp59m1KhRbiubNt10YM3V6AHO6x0FwIq9eYzoEd6m5VLqbLVpU/1D4KioKJYvb3pG9pKSEsBaHLxueuKAgADee++9VimX1ug7sOYexgKEd/Gla3gA2w67d5FkpVTHo4G+A2uqe6WzvnHB7DjS9PJmSqmzhwb6DqyqphYvAZtX00/q+8aFsDf3OBXVOh2C6vyMOTtmbT2d+9RA34FV1tQ2W5sHSI0LpqbWsP5gARsyCtqwZEq1LX9/f/Ly8jp9sDfGkJeXh7//qQ2E1IexHVhldW2TD2Lr9IsPBuDqV6xV5zc8PJXQQJ82KZtSbalr165kZmZyNixF6u/vT9euXU/pGA30HVhVTS1+J6nRJ0V2ISrIl9ySSgDWZRxjYmpMWxVPqTbj4+NDcnJyexfDY2nTTQfWUo3e2+bFneenON6v2JvPst25nP+3xRyvqG6LIiqlPIAG+g6sqsacNNADXD40wbH9xYYs/m/+DvbmHmflvrzWLp5SykNooO/AWnoYCxAW6MuqP03mX1cP5VBBGZsPWSNlZ6882OkfXCmlLBroO7CWmm7qxAT7c+mQBLpFBDgWDl+4LZvkB+ZSVF7V2sVUSrUzDfQdWFVNLb4212a7s3kJlw9JBKBbRIAjfdlubcJRqrPTQN+BVbvQRu/shrFJAEztH8eXd54HwK/fWcPslQdbo3hKKQ/hUpQQkekiskNEdovI/U3s/6eIrLe/dopIgdO+Gqd9n7uz8Ge76traZkfFNiUqyI8ND0/l/gv7MjAx1DHZ2R8/2eRo0lFKdT4t9qMXERvwAnABkAmsFpHPjTFb6/IYY+5xyn8nMMzpFGXGmKHuK7KqU11j8PM5tS9lzgOmekQGsubAMcCa5XKsfcZLpVTn4kqUGAXsNsbsNcZUAu8Bl58k/0zgXXcUTp1cda3B5nX6rW8PXtyfOyb1xt/Hi8/WH3JjyZRSnsSVKJEIZDi9z7SnnUBEegDJwLdOyf4iki4iK0TkimaOu8WeJ/1sGMLsLjW1Bp9TaLppLKKLL7+flsq0AXEs3JatzTdKdVKuBPqmIklzEWEGMMcY4zxdYndjTBpwDfAvEel1wsmMecUYk2aMSYuOjnahSArqavRnvsbklH6x5B+vZEOmTnymVGfkSqDPBJxXmO4KZDWTdwaNmm2MMVn2n3uBxTRsv1dnoLqmFm8Xu1eezOjkCADW2tvrlVKdiyuBfjWQIiLJIuKLFcxP6D0jIqlAOLDcKS1cRPzs21HAWGBr42PV6ak5wzb6OjEh/nQND2D1/nw3lEop5WlajBLGmGrgDmA+sA34wBizRUQeF5HLnLLOBN4zDcfV9wPSRWQDsAh4yrm3jjoz1bUGbzc03QCMS4lm6e48XaREqU7IpWmKjTFzgbmN0h5u9P7RJo5bBgw6g/Kpk6hxUxs9wJR+Mby76iAr9+Yzvo8+J1GqM9GRsR1YdW0tPm5oowcY2zsKfx8vFmw9ClgzXb6z4kCDPKWV1WQXlbvlekqptqMLj3Rg1TXuq9H7+9iY0i+WT9cfwtsmvLl0P2A9qE2JtVaq+s1761mw9Sjbn5iOv4/NLddVSrU+rdG3ofKqGrcu+GG10bvvv/AXY5MoLq92BHmAma+uoLbW8MKi3Y7a/qLt2W67plKq9Wmgb0PXvraSAY/Mdynv2oPHuOz5JSzbk9tsHne20QMM7x5OiH/9l7xxKVHkllTy8bpD/HX+Dkf6Rvuc9kqpjkEDfRuqm1cmq6Cs5bz7j7Exs5C/f72z2TzVtbVu63UDICLceF5PAN78xUj+cdVQROA/y/Y3yLcnu8Rt11RKtT5to28Df/5yK7tz6oPjZ+uzuHXiCQOEGyittLo5bs0qarbmXl1j3DJgytldk3szITWaIV1DERFGdA8nvdFAqj05GuiV6ki0Rt8GXluyj8U76ufweXr+dsb85Rt+9vpKFu3IprTyxHb70iorrayqhgN5x0/Yb4w540nNmiIiDO0Whoj1AXLn5PrFxV++bgS/mZzCnpzjfL3liFuvq5RqPRroW9mx45WO7aggP96/ZQzXn5OEzUv4YVcuv3hzNU/+b9sJx5VW1A9cyjh2YlNP3fxj7my6acqEPtH8YXoq14zuzvSBcdx5fm8SwwJ4RxcrUarD0KabVlbXZPPmL0YyKTUGgNE9I3nk0v68s/Ig/1ywk9mrDnLhwHjOS4li86FCukUEUlpZg81LqKk1ZOSXnnDe6tpaALc+jG3ObRN7O7a9bV5cOaIrz3yzi0U7sh33pJTyXFqjb2V1NfroIL8G6SLCz8b0YMl9k/CxefHN9qMYY7jkuSVc/e/llFVVkxQZiK/Ni4xjJwb6uimFW7tG35RbJ/aib1wwv3hzNT97fSW1Or2xUh5NA30r2JpVxL5cq129sKwKgNAAnybzBvp60zcumF1HSyix97HffqSY4xU1BPl50z8hhHmbj1BVU9vguKoaK7i2RY2+MX8fG89fMxyAH3blcrCJbxxKKc+hgb4VXPTsD0z622KKy6vqA31g04EeoHdMENuPFDvyApRV1hDo682d5/fmQF4pH6ZnNjimrkZ/KouDu1PvmCA++NU5gParV8rTaaB3s8rq+pr3Xe+uI6ugHC+BIN/mH4eM6RlJbkkFC+0jTwH25R0n0NfG+X1jSOsRzh8/2cTPXl/pqPW3ZRt9c4Z0CyXIz5v/+2q7znqplAfTQO9m++1dIYd1D2PRjhzeWLqPWgNeJwnIlw5OwOYlfLnxsCMtp7iCQD9vRIQnrhgIWM0kLy/ew+ZDhe3aRl/Hz9vGkz8ayKGCMp0WQSkPpoHezdZnWMvxPXH5QB6+pL9LxwT42ogL8Wfb4aIG6YH2icP6xYfwr6uHAvD8ot1c8twSqtuxjd7ZxYPiiejiy7zN2q9eKU/VqQP9az/sZceR4ja95idrD5Ec1YUBCSH88rxkl49LCPPnuH007PXn9LCnBTj2XzEsEV+n9vjquhq9m0fGnipvmxfn9Y5iye48XVxcKQ/VaQN9RXUNf/7fNi59fkmrnP/Y8UrmbzmC84Jaxhg2ZhYwoU+0Y2Tp69en8fr1aS2eL9EpqN9zQR8W/nY8d57fu0Geuy+oH6VaN1+Ou0fGno6LB8eTW1LBT19exiEX5vFRSrWt9o8SrSTf3n/d+eGoO93/8UZ+9fYaHvpss2OKgiNF5RyvrKFXTJAj3+R+sUzuF9vi+epq7742L0IDfOgdE3xCu/5tE3vz8nVWt8Z1B635Z3zauekGYGr/WH57QR+2Hi7iwU82tXdxlFKNdPpA727GGH72+krmbzlKn9gg3llxkMl//44tWYXsts/q2Cu6yymft1d0/YdD3beBpkxMjcHP24sfdlnTF7d3Gz1Y5b1rcgq3TujNoh05fLPtaMsHKaXaTKcL9CUV1RzIO94g0FfXuK9Wn3mszBFkP7v9PD657Vz8vL24+T/pzFlj9XXv7VSjd1VKrHVMZQtl9fexkZYUzsp9+UD7t9E7u2V8T/rHh3DfR5uanKhNKdU+Ol2gv2P2Wib8dTHZRRWOtEc+39LiMP2M/FJSH/yK5XvyWJ9R0Gy/8Lpg/tntYwnwtTGsezgvXjeCrMJyPlufBZw43YEr6j4cRiVHtJj33F5Rjm1PaKOvE+Br48FL+pFbUsETX25jd3bbPghXSjWt001qtjHTGqX5uw83ONJmrTzIiB7h/Hh412aPW743j4rqWma+ugKAruEBvHnDSMd6qQAFpZU8880uAFLj6tPHp0QRG+LHUfuHy8maXpoT6OvNR7ee06AJpznn9Ip0bMcEn/qHSmsanRxJQqg/7646yAfpGfzwh0kNeg8ppdqeS9VBEZkuIjtEZLeI3N/E/n+KyHr7a6eIFDjtu15Edtlf17uz8E3pHx/SZPrS3XknPe5oYTkAgb5W3/XSyhoufm4Jby7dB8ChgjLHnPLPzBjaYHFsEWHFA5NJjQ3mplPoUtnYiB4RhAX6tphvcGKoY7uP0weRJ7B5Ce/eMoYbzk2iptbw5cas9i6SUme9Fmv0ImIDXgAuADKB1SLyuTFma10eY8w9TvnvBIbZtyOAR4A0wABr7Mc2XLLIjSK61AfKO8/vzXPf7gZoMANkVkEZz36zi0cvG+AI2PvzSokN8WPFA5MREfbmlHD77HU8+b9tjEyK4JLnrG6a/j5eXDwo/oTrigjz7xnfWrfVgLfNix8PTwTjGQ9jG+sR2YVHLxvAxswCXl+yj2tG9yDIr9N9eVSqw3ClRj8K2G2M2WuMqQTeAy4/Sf6ZwLv27WnAAmNMvj24LwCmn0mBW+LcnXJgYihbHpvGZUMSWLUvn9tmraGyupYHPt7Ee6szSN9vfd4Ul1fx/a4cUuNCHM0uPaODmDGyG9W1xhHkAeJC/PFup4nEnP3jqqH8wz5a1lPdN70vR4sqmK+jZpVqV65ErEQgw+l9pj3tBCLSA0gGvj2VY0XkFhFJF5H0nJycxrtPifND1LAAH7r4eZMUGQjA3E1H+H5njqN2X1JhzRa5bE8eOcUV/Hp8zwbn6tlEN8mYYP8zKt/ZZFRyBIlhAcy3Lzu4L/e4owuqUqrtuBLom2obaK4LywxgjjGmLtq6dKwx5hVjTJoxJi06OtqFIjWvvKq+Rl83NXBvp3bsm/6bzt4c+wCnwnLySipYtS8fERjaPazBuVLtx907LZUBCVbbf0yIZz389GQiwuieEaw9WIAxhkl/W8yUf3zX3sVS6qzjSsNpJtDN6X1XoLknbDOA2xsdO7HRsYtdL96pc67R1y32MaaZLotHiys496lvqaiupUdkIIGNphKOCfFn/cMXEBrgw56cErZkFRF1Gl0nz2bDuoXx8dpDZDqte1taWX3Cv7VSqvW4UqNfDaSISLKI+GIF888bZxKRVCAcWO6UPB+YKiLhIhIOTLWntZoKpzb6ukAfE+LPVWkndq18afEeR/4RPcKbPF9YoC8iwoQ+1jcNT+vl4ulG2j9kl+3JdaRtySpqLrtSqhW0WK0yxlSLyB1YAdoGvGGM2SIijwPpxpi6oD8TeM84zfJljMkXkSewPiwAHjfG5Lv3FhpyDvQBTl0gn75yCLdP6s22w8W8vWI/k/vG8viXVsehm8clc/eUPic97+VDEzm3VxThJ1kpSp0oNTaY+FB/Fm2vf/ayMbOQkUktDwxTSrmHS9+fjTFzgbmN0h5u9P7RZo59A3jjNMt3yiqqa7hoUByPXjrghIFLPSK70COyC9MHxgFw4aA4BCEu1LUHrNEeNjipIxARxvSMZO6m+kVVNmYWnOQIpZS7tX8/QTerqKolxN+HmJCWg3d8aIDLQV6dvmHdwxp809p+WKdGUKotdapAP3/LEbKLK/Dz7lS31eENSKgfydsjMpCswjJKK6u56T+r2ZOj3S2Vam2dKiL+6u01wMnXZ1Vtr4d9HAPAmORIisurWbgtm4Xbsnn4s83tWDKlzg6dKtDXySmuaDmTajORTtNSjOllPYRde8AalVxUptMZK9XaOk2gL6+q7z9/tKi8HUuiGnN+KN49whpt/Nay/QAcLizTtWaVamWdJtAXlVc5VnbSaQo8z3Mzh/HczGEM7RbGvdNSHbX83JJKVu9v1R63Sp31xHlxa0+QlpZm0tPTT/v473bmMLRbmGOwlPJMxhiOV9Yw+NH53DGpN7+dmtreRVKqQxORNcaYtKb2dbpx6HUjWJVnExGC/LzpFx9C+oFWm7VaKUUnarpRHVNaj3DWZxS4dV3fpmzKLKSssunlIZXq7DTQq3Y1IimC0soaNh0qpLCsqlWuUVxexaXPL2Hqv3TmTHV20kCv2lWafTK5H724jCGPfd2g95S7HMiz1h/IyC/jQN5xt59fKU+ngV61q4SwAPo5rfP79vIDbr9GXaAH+HSdrmGrzj4a6FW7+9X4nvSPDyHIz5v/m7edg06B2R0O5lvnO7dXJP9cuJNlu3NbOEKpzkUDvWp3VwxLZO5vxvH1PePx8hJ++Z/V1LpxENXB/ONEdvHljvN7A3DNayvJLakfPV1ba3TOHdWpaaBXHiMhLID7pvdld3YJu5pZW7a21pzyurMH8krpFhHIqKQIfO0T3i11qtW//P0eJv/9O3Yd1Vk1VeekgV55lKn9YwGYt9laULy8qobNhwod+2evOsiUf3zH8j15Lp/zQF4pPSID8bZ5se3x6YQH+vD11qOO/T/stIL+Dg30qpPSQK88StfwAKb0i+WZb3ayaEc2z36zi0ueW8Kz3+zCGMP3O62Vqj5em9nsOUorq/nTJ5s4mFdKZXUthwvL6BFhzaBp8xIuHZLAgq1HKa+qobK6lhX7rA+NxTtymj2nUh2ZBnrlUUSEZ2cOpU9sMH/8eBP5xysB+MeCnaw9WOCodf+wK5empu/47/L9DHr0a2atPMjsVQc5mF9KrYGkqC6OPMO7h1NZXUvmsTJeW7KXutPMWZPJK9/vafV7VKqtaaBXHifQ15ubx/XkcGE5hwrKHOkvLtrN0aJyAn1tHCkq57P1WWzJqm/Wycgv5eHPtjhmw3z5uz1M+Yc1SKpndJAjX7eIAEf+T9YeIsDHxsLfjmdgYgjPfbubvBKd5lp1LhrolUcamGitSrV8Tx6JYQHcf2FfvtmeTXlVLZcPTQTg7vfXc/GzSxyBfdvhIgDev2UM41KiGpyvZ3R9jb6bvRkn41gphwrKuHZ0d3rHBDOhTzTF5dWMfHJhqwzcUqq9aKBXHqlXdBcCfGxU1xqC/b25dnR3x77RyRGE+NfPx7f2oDUp2n77qNfUuGDHh0GdEP/62Uyjg/wI9LWxPqOA0soax0ynFw9KAKDWwLfbs1vnxpRqBy4FehGZLiI7RGS3iNzfTJ6rRGSriGwRkdlO6TUist7++txdBVedm7fNy1ErD/H3IdjfB38f69c1JtiP128YyZ8u6gfA1iyrJr8vt5SwQB/CAn25ckRXvrt3IgAXDYprcG4RYfrAOD5eewiAsEAr0PdPCGH3kxcS4u+tgV51Ki0GehGxAS8AFwL9gZki0r9RnhTgAWCsMWYAcLfT7jJjzFD76zL3FV11dhNTYwDIL7UeyF4+xKqlRwf7MTIpgpvGJRPi7+0I9EeLykkIDXAc3yOyCx/fdi7/vHroCee+eFC8Yzs0sH6pQ2+bF2N6RrKmhamTv91+lKe+2n6ad6ZU23KlRj8K2G2M2WuMqQTeAy5vlOdm4AVjzDEAY4xWh9QZG9zVaqevm4jsiSsG8t9fjiIlNhiwaubDe4Tz8bpM9uaUUFZZQxc/W4NzDO8ejp93wzSAEfbJ1IATFqkZ0i2MfbnHKbB/wDjbcaSYP8zZwC/fSufl7/ZoW77qEFwJ9IlAhtP7THuasz5AHxFZKiIrRGS60z5/EUm3p1/R1AVE5BZ7nvScHO3LrCx97AG9d4z109fbi/GNFpZ55NIBVNUYvt56lLKqGvx9TgzqTQlzqsWHNQr0I5OsBcwX7WhYX5m3+QjT/vU9H6TX9+E/4OZ5eZRqDa6sMCVNpDXuwOwNpAATga7ADyIy0BhTAHQ3xmSJSE/gWxHZZIxp0FnZGPMK8ApYSwme4j2oTsrX24tPbx9LYlhAs3mSo7qQGhvM0t25lFXWEBvi5/L5/X28KK+qJdi/4Z9BWo9wkqO68NGaQ0wbEMfts9ayyGkw1biUKNYeOMbxyhr25JSQGhd86jenVBtypUafCXRzet8VaDzXaybwmTGmyhizD9iBFfgxxmTZf+4FFgPDzrDM6iwytFsY0cEnD95je0exal8+BWWVBLhYowd46doR9IruQkKjDxIvL+GcXpFszCzgy42HGwR5H5vw3MxhrHnoAgJ8bCzeoa2UyvO5EuhXAykikiwivsAMoHHvmU+BSQAiEoXVlLNXRMJFxM8pfSyw1V2FVwpgbO9IKqprOVpUQYCv64F+Ut8YvvndxCabe/rHh1BUXs2OIw3nv/nrlUMIC/TF38fGj4Yn8kF6Jl9vOXLG96BUa2ox0BtjqoE7gPnANuADY8wWEXlcROp60cwH8kRkK7AIuNcYkwf0A9JFZIM9/SljjAZ65VZ1g6sAl9voW1LXTv/6kn0N0useEAP86aJ+JEUG8sKi3W65plKtxZU2eowxc4G5jdIedto2wG/tL+c8y4BBZ15MpZoXE+xHkJ83JRXVBJ5Cjf5kUuOCGZkUzur9x/C1eVFpX7w8KbJ+hG0XP2+uHd2DJ+duIyO/1DHiVilPoyNjVYcnIo4geypt9C2pO+f4PlG8cUMaf/nRILy8GvZNOL+f1dd/3NOLtKul8lga6FWnUKSDEToAACAASURBVNfbxl1NN2BNlQDWAK3z+8ZyjdM0DHV6RnXhR8Os3sZ1c+gr5Wk00KtOIcoelP283fcrHd7F137O5j88RIS//3QIYYE+rNyX77ZrK+VOGuhV66ithextbXa5yCArKBeUVrntnHWNNNLUSBInXl5CamwwK/fmUWVvy1fKk2igV61j0wfw4jnw2R1Q0vp9zeNC/AE4Xum+dvK6ZqAuvi33WegeEcje3OM8/632wFGeRwO9ah19psM5t8OGd+HZ4bD0GahuvQU9Zo7qzg3nJvHrCT3dds6rR3bjVxN68uuJvVrMe8PYJADma5965YGkqeXY2lNaWppJT09v72Iod8ndDV//CXbOg/BkmPpn6Htxy+0hHdBf5m7jraX7Wf/IBQS68C1AKXcSkTXGmLSm9mmNXrWuqN5wzftw3cfg7QfvXwv/vRyObmnvkrnd5L4xVNbU0v/h+ZRUVLMho6DBUohKtRcN9Kpt9J4Mv14KF/4VDm+Al8+DL38Lx/Pau2RuMzIpgi72AVur9uVx+QtLufjZH9q5VEppoFdtyeYNo2+Bu9bByJthzVvw3DBY/iLUuK+3THvx8hLSH7wAX5sXv3zLan4sKK3C05pH1dlHA71qe4ERcNHTcOsySBwB8x+weujs/Lq9S3bGAnxt3DsttUHaH+ZsbKfSKGXRQK/aT0xfq+1+5vtgamH2T+Gdn0DOjvYu2Rn5xdgkfG1eJEVaUyh8uCaTg7pAiWpHGuhV+xKB1Olw2wqY+iRkrLZq91/dB2UnX7fVU3nbvFjz0BS+vmcCX/1mHABrD3bMe1GdgwZ65Rm8feHcO+CutTD857DqFXh2GKx6FWqq27t0pyzY3wdfby9SYoLw9/Hi2+3ZFJZ1/OcQqmPSQK88S5couPRf8KsfIHYgzP291UNnz6L2Ltlp8bZ5MSAhlM83ZDH9X9+3d3HUWUoDvfJMcQPh+i/g6negugzevgLenQl5e1o+1sPUrXl7uLBce+CodqGBXnkuEeh3Kdy2EqY8Cvu+hxdGw9cPQnlhe5fOZUO7hTm2tzdamlCptqCBXnk+H3847x64cy0MuRqWPW/Nn7PmLaj1/MU+bjg3iTdvGEkXXxuv/rDXkX68oppb31nDMwt3tWPp1NlAA73qOIJj4fIX4JZFENkbvvgNvDIB9i9p75KdlJeXMKlvDJP7xbJiT/1I4FX78vlq8xH+uXAn2w4XkZFfyur9Oqe9cj8N9KrjSRgGv5wHV74BZQXw1sXw/s/g2P72LtlJDekWRlZhOdlF5QBsPVwEgI9NuOKFpYx7ehE/fXk5ReVVPP7FVl5zqv0rdSY00KuOSQQG/gTuWA2T/gS7F8Lzo2DhY1Dhme3g/eNDANh5tASAHUeKSQwL4PZJvamorl+w5HcfbOCNpfv48/+26Tq0yi000KuOzScAJvwB7kiHAVfAkn/Ac2mwfra1ypUHqe99U+b42S0igBE9wh154kP9WbD1qOP9AR1Rq9zApUAvItNFZIeI7BaR+5vJc5WIbBWRLSIy2yn9ehHZZX9d766CK9VAaCL8+BW4cSGEdoVPb4XXJsPBle1dMofYUGtd2yOFVtNNTnEF0cH+nNMzkt9d0IfVf5rCX340qMExe3NK2rycqvNpMdCLiA14AbgQ6A/MFJH+jfKkAA8AY40xA4C77ekRwCPAaGAU8IiIhKNUa+k2Em5cAD96BYoPwxtTYc6NUJjZ3iXDz9tGVJAvWc6BPsgPb5sXd05OITrYj+HdG/551DXzKHUmXKnRjwJ2G2P2GmMqgfeAyxvluRl4wRhzDMAYU7dI6DRggTEm375vATDdPUVXqhleXlY3zDvXwPg/wPYvreacRf8PKtu3KSQu1J+sgjJKK6s5XllDVLBvg/2hgT7855ejuHdaKgMSQli2J7edSqo6E1cCfSKQ4fQ+057mrA/QR0SWisgKEZl+CsciIreISLqIpOfk5LheeqVOxrcLnP8n64Ft6nT47il4Pg02fgjtNEI1JSaYnUeLOVpkrZ8bHeR3Qp4JfaK5fVJvJvSJZs2BY7pKlTpjrgT6phb3bPxX4g2kABOBmcBrIhLm4rEYY14xxqQZY9Kio6NdKJJSpyCsO/z0LfjFV9ZcOh/fBK9PhUNr2rwoAxJCOFxYzt+/tqZi7hUT1Gze68b0oNYY5qS3f7OT6thcCfSZQDen912BrCbyfGaMqTLG7AN2YAV+V45Vqm30OBduXgyXPW/1uX/1fPjk11B0uM2KMCAhFIAvNx5mVHLECW3yzhLCAogPDWB/3vG2Kp7qpFwJ9KuBFBFJFhFfYAbweaM8nwKTAEQkCqspZy8wH5gqIuH2h7BT7WlKtQ8vLxj+M6v9fuzdsPkjeG4EfP9XqGr9JpL+CSGO7ZST1ObrdIsI4GC+drFUZ6bFQG+MqQbuwArQ24APjDFbRORxEbnMnm0+kCciW4FFwL3GmDxjTD7wBNaHxWrgcXuaUu3LPwQueAxuXwm9JsG3f7YGXG35tFXb70MDfIgOttrlE+z96k+mW3ggGRro1RlyqR+9MWauMaaPMaaXMeZJe9rDxpjP7dvGGPNbY0x/Y8wgY8x7Tse+YYzpbX+92Tq3odRpiugJM2bBzz+3gv+H11tTKhze0GqXTI7sAkBIgE/LeaO7kF1cQXZxeauVR3V+OjJWKYCeE+BX38Ml/4Sc7fDvCfD5nVCS3fKxp+jRywYQG+LHeb2jWsw7uW8sAF9tOuL2cqizh3jaQghpaWkmPT29vYuhzmZlBfDd07Dq3+AdABPuhdG/Bu8Tu0K2NmMMlzy3hIrqWhbcMx6RpjqyKQUissYYk9bUPq3RK9VYQBhM/4u1YHmPc2HBw9aCJ9v/1+b970WEGaO6szu7hIx862Hxn7/cyucbtPOacp0GeqWaE5UC134A130ENl947xr47+VwdGubFmN4d2uFqvWZBazYm8drS/Zx17vrqK31rG/jynNpoFeqJb2nwK1L4cKnrYe0L4+FL38Lx/NaPtYN+sQGE+BjY/W+/AYzW+7M9szpmJXn0UCvlCtsPjD6V3DXOhh5k7WM4XPDYMVLUFPVqpf2sXkxvk8U87ccYcXePGJDrGcFGzM7zrq5qn1poFfqVARGwEV/tWr4iSNg3v3w0rmwa0GrXnZKv1iyiyvYklXEZUMSCPbzZmNmQateU3UeGuiVOh0x/eC6j2Hm+9YC5bOuhHeuhJydrXK5tKQIx3ZSVBcGJoayKbOQg3mlXPPqCvJKKlrluqpz0ECv1OkSsWbFvG0FTP0zZKyEl86Br+6HsmNuvVRSZKBju3tEIAMSQtiQWcj4vy5i2Z483llx0K3XU52LBnqlzpS3L5x7J9y5FoZdZ/W/f3Y4rHoVaqrdcgkRoXuEFey7hQdyft+YBvv1waw6GQ30SrlLUDRc+ow1wjZ2AMz9Pfx7HOxZ5JbTz7ppNL+9oA89IgM5t3cUq/402bFv2e5cR3fLp+dtZ/zTi9h1tJgDecdZvqdtegcpz6WBXil3ixsE138BV70Nlcfh7Svg3ZmQt+eMTtstIpC7Jqc4RsfGBPtz3Zju9I8P4VhpFY9/uZUfduXw3uoMDuaX8r9Nh7nq38uZ+eoKrvr3cnfcmeqgdAoEpVpTVTmseBF++DtUV8CYX8P4e8E/1G2XKK+q4cJnfmBfbsN568MDfThWWt/1c1RSBHdNTuG8lJbn2FEdj06BoFR78fGHcb+15r8ffDUse96a/37Nf6zeOm7g72Pj/gv7Nkg7r3eUI8jfM6UPAKv253Pd6yu5ffZaVu7V5pyziQZ6pdpCcBxc8QLcsggiesEXd8ErE2D/UrecftqAOJbefz4jk8KZlBrNjeOSHfsuHhzXIO//Nh7mzaX73XJd1TF4t3cBlDqrJAyDX86DLR/D1w/DWxdB/8vhgicgvMcZnToxLIAPfnUOxkBlTa0jvXdMMF/ccR5JUYEMevRrAL7flUN5VQ3+PrYzuqbqGLRGr1RbE4GBP4E7VsPEP1qjap8fCd88DhUlZ3hqwctL8PexMfeuccy+aTQAg7qGEuxfv9BJaWUNgx/7mmqnD4S2ll1UTk6xDvRqCxrolWovvoEw8T64I92q1f/wd6v9fv1sqD3zANw/IYRzGy1u8sux9U06ldW1LN6Rc8bXcZUxhpv+s5qn522nptYw4a+LGfnkQn784lIO5ulyia1JA71S7S00EX7yKty40Nr+9FZ4bTIcXOn2Sz10ST/2/b+L2PXkhUQF+fF+egZg1a7LKt3zcLg5K/fls3BbNi8u3sPKvXmUVVnXW3uwgC826vz6rUkDvVKeottIK9j/6N9QfBjemApzboTCTLddQkQQEXxsXvxkRCLfbs8mu7icUX/5hn4Pz+O7nTm8vXw/Y5/6lqyCstO6xrqDx/hkXSZHChuuc7vjSP3o3c/WW4F96f3n0zcumGV7ck/7nlTLNNAr5Um8vGDIDKs5Z/y9sP1LeC4NFj8Fle5t3rgqrRs1tYa/zd/hSLv+jVU89NkWDhWU8fWWU1+nNru4nB+9uIx73t/AI59vbrAvq9D64PC1efF+egZeAnEh/pzXO4rV+49RXtW63yjOZi4FehGZLiI7RGS3iNzfxP4bRCRHRNbbXzc57atxSv/cnYVXqtPyC4LzH7Qe2KZOh8X/z3pgu2mO25Yz7BUdxI+HJ/JBetPfGL7ZfuoLo8/bXP/hsCWrqMG+I4XldIsI4NaJvQCoNWDzEsamRFFZXUvfh+axJUvn2G8NLQZ6EbEBLwAXAv2BmSLSv4ms7xtjhtpfrzmllzmlX+aeYit1lgjrDj99C37xlTUX/kc3whvT4NAat5z+5+ckObYfuqT+zzos0IeVe/M5XFjG8QrXJ2ZbuS+fxLAA7r+wL5nHyhr0qtl2uIj4kADunpICQL/4EADO7RXpyOO8gpZyH1dq9KOA3caYvcaYSuA94PLWLZZSqoEe58Iti+Gy5yB/L7x6PnxyKxQdPqPT9rcHW4Ap/epnxPzV+F5U1tRyzv/7lgGPzOe5b3Zx57vryG1h3vv1BwsY1j2Mkfb589P35wOweEc2O4+W0C8+GBFh46NT+fDX5wDg523jxWuHA1BR3X7dPTszVwJ9IpDh9D7TntbYT0Rko4jMEZFuTun+IpIuIitE5IqmLiAit9jzpOfktF13L6U6FC8bDP+5NR3y2N/A5jlWd8zv/2bNqXMafL29+OjWc5h39zh6RHZxpF+V1pVxTnPi/H3BTr7YkMX3O5v/+yyvquFQQRl944IZlBhKgI+N/yzfT2lltWPZw99PSwUgxN+HIL/68ZoXDYonMSyAw6f5ALg1GWM4kHe85YwezJVAL02kNW4k/AJIMsYMBhYC/3Ha190+0c41wL9EpNcJJzPmFWNMmjEmLTo62sWiK3WW8g+BCx6H21dCr0nw7RPwwkjY8ulptd+P6BFB3zirZv/hr8/hq9+MIzLIj17RQQAkhPo78u7JaX5AV1G5NbdOaKAvvt5e/Gh4Iiv25vO3+TvJyC8lJtivwaCtxhLC/MkqPL0PrNa0aEc2E/66mA/SM1rO7KFcCfSZgHMNvSvQoNOrMSbPGFP3ne5VYITTviz7z73AYmDYGZRXKVUnoifMmAU//xx8g+HD6+GtS+DwxtM+5cikCEfb+W8mp3DDuUl8/dsJzLppNMlRXdib03zNtqjMassP8bdq6vdNsyZaW59xjIP5pY6FU5rTOyaIzYcKyT9eedrlbw3rM6xvI28vP9DOJTl9rgT61UCKiCSLiC8wA2jQe0ZE4p3eXgZss6eHi4iffTsKGAtsdUfBlVJ2PSdYi51c/A/I3gr/Hg+f3wUlZ9YMGt7Fl0cvG0CQnzdje0eRGhvMxsxCmpvavNheow8JsGrtoYE+XDemO7uyS8hwIdD/bEwSpZU1DXruuFNZZc1pra2766jV//9A3vFm793TtRjojTHVwB3AfKwA/oExZouIPC4idb1o7hKRLSKyAbgLuMGe3g9It6cvAp4yxmigV8rdbN4w8ka4ax2MuQ3Wz4LnhsPSZ6HaPTXkCanRHCooY8fRppctLCqvq9HXN8/0iw+huLyarMJyurYQ6PvFBxMa4MOmQwVuKW9jM19dwYg/Lzzl43ZlW81VReXVPPjp5g4Z7F3qR2+MmWuM6WOM6WWMedKe9rAx5nP79gPGmAHGmCHGmEnGmO329GXGmEH29EHGmNdb71aUUgSEwfS/WAuWdx8DCx6CF0fD9rln3P9+sn2d2oXNdIEsKrPX6P2dHrIOrP+y31KNXkQY3DWULzcebvYaZ2J9hvUBkl3s+nMAYwyZx0qZlGo9O5y18iDLOuDSjDoyVqnOKCoFrv0Qrv0IvLzhvZnWkoZHT/8LdUyIP0O6hrJwW9MDqYoaNd2A1fxT17umW3hAi9f45XnJFJdXc9N/03ljyT5qas+89lxSUc0v31rteL/J3gOotLL6pA+XAb7bmUN5VS0TU2PY/Ng0vASufW0lD326+aTHeRoN9Ep1ZilT4NZlMP3/IGsdvDwW/vc7a7v61Nurp/SLZX1GAWsO5J+wr/5hbMOeNffZV7/qae/FczKTUmMc/esf/3Irs1ceYE9OSYsB+WT+tzGLb7dn0yfWuv4Pu3KZu+kw/R+ez+S/f0dpZfMDwm540/qASAgLIMjPm+Qoqwvq2ysOsD+343S51ECvVGdn87HWqr1rPaTdCOlvwisT4S+J8PI4+PxOWP0aZK5psT/+tIHWalV3zF53Qlt1QWklvjYv/H0ahpWfjenB9iemEx3s51JxRyZF8N9fjgLgoc+2MPnv33Hxsz9QcgojdOs8PW879320iZ5RXZh/93gmpkbz1rL93DZrrSPPst0Nm2Iqqmt45LPNvLFknyOtb1wwAPdOq1+y8buTjCnwNLrClFJni8AIuPhvcN49kLkaDq+HrPWw7QtY+18rj9ggph/ED4X4IZAwFGIHWnPnA31ig7l1Yi9eWryH/XmljhouwJGicmJD/RA5cejNqa5kNb5PNOP7RDsGaJVX1fL9zhwuGhTfwpENfbLuEAAPXtIPEeEP0/o65uCPC/Env7SSlfvymNI/ltySCu56d90JbfDv3jyGbvbnC9MHxrHnLxcx+i/f8MjnW5g+MI7YEH88nQZ6pc42oYnWa4B9oLoxUJgBhzdYgf/wetg5D9a/Y+0XL4hKtYJ+/BB+lpDKf6Wchz/bzL3TUlm1L5/rxvTgSGE5cW4Mes9cPZQjReWkxAQx7IkF/LDr1AN9dLAfKbHBnN83FrAWY1n5x8mM/ss3XDgojvT9x9h22OpF9P7qjCYftI7oEd7gvc1LmDYgllkrDzJv8xGuPzfptO7PGNPkh2Jr0ECv1NlOxJo8Law79LvUSjMGirKs4F9X89+zCDa8SwKw2U/YcyCeTf9O5nBtErcs6M2Wym5MHHzCwPfTFt7Fl/AuvgCMTo5gxd4Tnwu0pKS82lEbrxMb4s+i308kMSyAhz/bzPwtRzDGOPrLA7x83XB+/Y7VvOPrfWIL95+vGMj8LUfZfOjUZttcd/AYUUF+3DtnAwmhAfzj6qGnfE+nQwO9UupEIvU1/74X1acXH4HDG6jKWMuRZd8yunobP7Ittfb5Q9aeRJgzxmr2iR8K8YMhILzpa5yCET0iWLgtm8LSKkIDm59GobGi8uoG3T3r1DU5De8eznurM9hxtJjdOSWMS4ni6SsHEx8awIMX96NndJcTjgWrK+jAxBA2uRjo//jJJsb1juJWp2cDYI0zuGlccqvX7DXQK6VcFxwHwXH49pnGoHN+T+7xCsq8S/jbW+/jn7uZ6xOPQcZq2PxR/THhSQ3b/OOHWs8LTkFKjNVjZndOyQlNKSdTUlF10vl1Jtj7x0//1w8A3HheMvGhVjfQm8b1POm5ByaE8sOuXMqrak76DCK7uJzZKw8ye+VBR1pCqDWvz5Nzt9EnLpgJfVp3ji8N9Eqp0xIa6GOvXQfx0D2/abjzeB4cqWvztzf/bP3U6eDukDDEXvMfZv0Maj7Y9bYH+j3Zrgf6qppayqtqG8yS2VhsiD8+NqGqxupBNKx7mEvnBhiYGEJNreGfC3bywEX9msxzqKCMez/c0CDtkUv784uxyVz2/BI2ZhZy/RurWPfQBY5mqtaggV4p5X5dIqHX+darTtkxa8K1w+vrH/xu+6J+f0hifZOP/cEvwVZ3zm4Rgfj7eLHtSBGuKrFPyRDcRNONsyuGJvLhGmuVrbQern/TGGHP+8bSffx+Wio+toZt+TnFFYx96tsTjosMsrqZfnrbWK5+ZTmr9x9j6Z5cLhmc4PK1T5UGeqVU2wgItyZg6zmhPq28EI5saljz3/EVjpnQg+Igfgi2hKH8ItKfXfsrwfS3niG0oNgR6E/epv/EFQO5bkwP8ksriQt1vddQdLCf46HthowC0pIafkg4r5k7fUAc8+xr8EYFWTV3Ly/hnZtGk/rgPO7/aBNT+sWecjdUV2mgV0q1H/9QSDrPetWpKLGCv3PNf/cC7jPW6lO1f43GK6FRzT+02wnB/6h9TpuWavT+PjaGdHO9ycbZOT2jEIElu3NPCPR1UmKCeO6aYUz++3cczC9tMHLYz9uGiDVNw/urM067q2ZLNNArpTyLXxD0OMd61ak8zv4tq3hjzidcG1pAavEe2PNPMDXW/oCIhg9744fw4epCuvjaGJMc2fR13CA00IdBiaEs3Z3L3VP6NNgX4GOF17dvHI2PzYtnZgzlb1/vcDxvqPPJbWO54oWlvLvqID8/p0er9MDRQK+U8ny+XUgaNonvFkJecCgvXDMcqsqsSdoOr6uv+S97HmqtydUeliB+7t+b0CVL6z8AwpPBy70zv4ztHcVLi/fwzbajjO8T7WirLyyrol98iKM5aFj3cGbdNOaE44d2C+PPVwzkwU83syGzkKGn+e3iZDTQK6U6jJSYIHYftU9w5hMAXUdYrzrVFdbiK1nr+e7reQyQfbDyZaixz8nvFwJxg+ubfOKHQmTvMwr+59kD/Y3/SeeJKwbyszE9AGs2z6b68Dfl8qEJbD1cRGiA62METoUGeqVUh5ESG8ziHTm8vXw/141popnD2w8ShkHCMP68oDvn9Y7irz/uD9nbGrb5r34Nqu0TuPkGQdyghn39o/pYi7G7YGRSBDePS+bVH/ax9sCx+kBfVnXCqNzmBPv78JcfDXL1n+GUaaBXSnUY14zqzrLduTz02Rb6xocwsokHoIWlVTw5dyuHC8utGrLNxxqhGz+4PlNNNeTuaDi/z9r/QFWptd87wAr+zjX/6L7WSl6N+Hp78aeL+7Mvt5Qlu3PZk1NCr+ggisurT5iyub1ooFdKdRjdIgKZffMYzn3qW+6YvZaFv51wQvfJ9ZkFfJBu9YsPaa4pxOYNsQOs19BrrLTaGsjd1XB+n/WzYdUr1n5vfyu/c80/uh94W90lz+0VycJtR5n89++4aFAcRWVVhAR4Roj1jFIopZSLuvh589SPB3HrrLX8sCv3hBkt6xYpB1xuIwesppqYvtZryNVWWm0t5O+x1/ztD303fQjp9lVRbb4Q0x8ShnJl+AA+kQp2mG7M3VTXZ961OfhbmwZ6pVSHc0H/WEIDfPh2e3YTgb5+gZLqM12K0MvLWpYxKgUGXWml1dbCsX0Na/5bPiWk/C2+8INa8WZbTSKbapOZYi6AzDHWNwGflpdSbC0a6JVSHY63zYuJqdHM3XSYiupazu0VycxR3YGGNfpWmRXSywsie1mvgT+20oyBggNweAMmcx1lq77jUtbT5fvF8D3Wgi7RfRu2+ccNBN+mZ8d0Nw30SqkO6cfDu/LZ+iy+2GC96gO9VaN/7LIBzBjVrW0KI2LN0hmehK3/5aRNxb6gS2Z9zf/wBti1ANbPsh/jZfXucW7zjxsEfsFuL55LgV5EpgPPADbgNWPMU4323wD8FThkT3reGPOafd/1wIP29D8bY/7jhnIrpc5yE/pEM/um0Vzz2kqgfsWmYvsc9K01nYDLRCCsm/Xqd4mVZgwUH3bq7bMB9n0HG9+z9scMgNuWub0oLQZ6EbEBLwAXAJnAahH53BiztVHW940xdzQ6NgJ4BEjDmqVojf3YY24pvVLqrHZu7yieuGIgD326mcOF5SSEBVBUfvI56NuVCIQkWK/UC+vTi49aQb+2qvljz4Arw8FGAbuNMXuNMZXAe8DlLp5/GrDAGJNvD+4LgOmnV1SllDpRaqzV1LHjiLUUYHF5dYsTmXmc4FjoMxX6Xtwqp3cl0CcCGU7vM+1pjf1ERDaKyBwRqWsYc+lYEblFRNJFJD0nJ8fFoiullFOgt6/5WlRW5TEDlTyFK4G+qcfWjfssfQEkGWMGAwuBunZ4V47FGPOKMSbNGJMWHd26S2oppTqX0EAf4kL82ZRprd+aW1JBVHDrrdbUEbkS6DMB50fXXYEs5wzGmDxjTIX97avACFePVUqpMzV1QCxfbz1CbkkF2UUVxAS7voDI2cCVQL8aSBGRZBHxBWYAnztnEBHnEQuXAdvs2/OBqSISLiLhwFR7mlJKuc3lQxOoqjEs35NHcUU10cGeMSLVU7T4xMIYUy0id2AFaBvwhjFmi4g8DqQbYz4H7hKRy4BqIB+4wX5svog8gfVhAfC4MSa/Fe5DKXUW62Nvp7/z3XWAtei3qufSo2ljzFxgbqO0h522HwAeaObYN4A3zqCMSil1Uo27U3a4XjetTP81lFKdwlu/GEleSSVbsooYlxLV3sXxKBrolVKdwsTUGAB+MqKFjGch9y6eqJRSyuNooFdKqU5OA71SSnVyGuiVUqqT00CvlFKdnAZ6pZTq5DTQK6VUJ6eBXimlOjkx5gxXSXczEckBDpziYVFAbisUpz3ovXiuznQ/ei+e6UzupYcxpsl53j0u0J8OEUk3xqS1dzncQe/Fc3Wm+9F78UytdS/adKOUUp2cBnqllOrkOkugf6W9C+BGei+eqzPdaWVBDQAABGBJREFUj96LZ2qVe+kUbfRKKaWa11lq9EoppZqhgV4ppTq5Dh/oRWS6iOwQkd0icn97l6clIvKGiGSLyGantAgRWSAiu+w/w+3pIiLP2u9to4gMb7+Sn0hEuonIIhHZJiJbROQ39vQOdz8i4i8iq0Rkg/1eHrOnJ4vISvu9vC8ivvZ0P/v73fb9Se1Z/qaIiE1E1onIl/b3HfJeRGS/iGwSkfUikm5P63C/YwAiEiYic0Rku/3v5py2uJcOHehFxAa8AFwI9Admikj/9i1Vi94CpjdKux/4xhiTAnxjfw/WfaXYX7cAL7VRGV1VDfzOGNMPGAPcbv/374j3UwGcb4wZAgwFpovIGOD/gH/a7+UYcKM9/43AMWNMb+Cf9nye5jfANqf3HfleJhljhjr1Me+Iv2MAzwDzjDF9gSFY/z+tfy/GmA77As4B5ju9fwB4oL3L5UK5k4DNTu93APH27Xhgh33738DMpvJ54gv4DLigo98PEAisBUZjjVL0bvz7BswHzrFve9vzSXuX3ekeutqDxvnAl4B04HvZD0Q1Sutwv2NACLCv8b9tW9xLh67RA4lAhtP7THtaRxNrjDkMYP8ZY0/vMPdn/7o/DFhJB70fe1PHeiAbWADsAQqMMdX2LM7lddyLfX8hENm2JT6pfwF/AGrt7yPpuPdigK9FZI2I3GJP64i/Yz2BHOBNe5PaayLShTa4l44e6KWJtM7UX7RD3J+IBAEfAXcbY4pOlrWJNI+5H2NMjTFmKFZteBTQr6ls9p8eey8icgmQbYxZ45zcRFaPvxe7scaY4VhNGbeLyPiT5PXke/EGhgMvGWOGAcepb6ZpitvupaMH+kygm9P7rkBWO5XlTBwVkXgA+89se7rH35+I+GAF+VnGmI/tyR32fgCMMQXAYv5/O/evEkcUxXH8eyAkBglZBTsLWQjpgoVFQAvBYGFhtYUguEWeIgTyCCEvkDpCAhGxda3VgP9WBaMQUBJJEbC2OCnuGZGFRZFkxzv8PjDMzL1TnAN3zjD3DpPWHWpm9iC6rsd7lUv0PwX+9DbSrsaBWTP7ASySpm8+kGcuuPvP2P8GvpIewjmOsTPgzN3X4/wLqfD/91xyL/SbwLP4muAhMAcslxzTXSwDzThukua6i/aFWH1/CVwUr3j3gZkZ8BE4dPf317qyy8fMhsysFsePgVekhbI1oBGXdeZS5NgAWh4TqWVz9zfuPuzuI6R7ouXu82SYi5n1m9mT4hiYBtpkOMbc/Rw4NbPn0TQFHNCLXMpeoPgHCxwzwBFpPvVt2fHcIt5PwC/gkvTEfk2aD10Fvsd+MK410ldFJ8AeMFZ2/B25TJBeJXeB7dhmcswHeAFsRS5t4F2014EN4Bj4DDyK9r44P47+etk5dMlrEljJNZeIeSe2/eIez3GMRXyjwLcYZ0vAQC9y0S8QREQqLvepGxERuYEKvYhIxanQi4hUnAq9iEjFqdCLiFScCr2ISMWp0IuIVNxfGA23RDw1NJsAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.recorder.plot_loss()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stage 2 training\n",
"And now lets train the full model with differential learning rates"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"learn.unfreeze()"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"#learn.summary()"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"SuggestedLRs(lr_min=1.0964782268274575e-05, lr_steep=0.00363078061491251)"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEKCAYAAAD5MJl4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhV1d3+//fnZCQzIWEMSJgFFJCIIipgEYdatK1a6STO/qq11tZWfz5Pbe1gWzuoba36KNrW2ToUhzrVIqNCEAEZZSZhykQImZOzvn/kgIEMJJCdfZLcr+vK5Tn7rHP2ZxGTO2uvvdc25xwiIiL1BfwuQEREwo/CQUREGlA4iIhIAwoHERFpQOEgIiINKBxERKSBSL8LaK20tDQ3cOBAv8sQEelQli1blu+cS29p+w4XDgMHDiQ7O9vvMkREOhQz29aa9jqsJCIiDSgcRESkAYWDiIg0oHAQEZEGFA4iItKAwkFERBrocKeyioiEg6qaIAWlleSXVLGvvIpxA7qTENN5fqV2np6IiHhse0EZT320jVeX57K3pPKw14b3SuQf10ygZ1KsT9W1LYWDiMhR/Hf9Xv6+aCtzN+QRMGPaiT0Z1TeZtIQY0hKiKa+u5c6XV3HZI4t56prT6J8a53fJx03hICLSjEUb87nqiaX0TIzhlnOGMnPCAHonNxwdDEiNY9YTS7ns4cU8de0EhvRM9KHatqMJaRGRZqzZtR+At289m++fO6zRYAAYN6A7z99wOrXOcfkjH/LGyl3k7iuno96KWSMHEZFm5BSVkxATSUpc1FHbjuidxIs3TOSbj3/ETc98DEBibCQn9k5i+qheXHNmJmbmdcltQuEgItKM3H3l9Evp1uJf6gPT4nnvtsms3rmftbv2s273flbsKOYXb6xlz/4K/v8LT2x1QDjn+OO7G5g+qjej+yUfSzdaTeEgItKM3KJy+nXv1qr3xEZFMP6E7ow/oTtQ98v97jmr+b/5WwiYcccFIw4LiNx95Sz8LJ+Lx/UlJjKiwef9bdFWHnx/I0GHwkFEJBzk7is/9Ev+WJkZP5sxiqBzPDJvMxjccf4ItheW8de5m3jp4xyqax2vrdzJI98aT1z057+a567fyz2vr+Hckb247dxhx9udFlM4iIg04UBlDcXl1a0eOTTGzLhnxmicg0c+2Ez21iI+2bGPiIDx9QkDGNAjnl++sYZvP76Ex2edSnK3KD7bU8J3n1nO8N5J3P+1sQQC7TdfoXAQEWlCblE5AP1Sjj8cAAIB4+cXjyZgxj+X5XD1pIFcd9agQxfO9UmO5XvPLWfmox/y4MyxXPO3bGKiInjsyizi2/nqa4WDiEgTcveVAbTJyOGgQMD4+SWj+emMUUQcMRK48KQ+xEVHcONTy5j+x3lERgR4/vrT2yycWlVnu+9RRKSDODhyyPDgl/ORwXDQlOE9+cc1pzGwRzx/uHwM4wYc33zHsdLIQUSkCTn7yomOCJCWENOu+z11YCrv/3BKu+7zSBo5iIg0IbeonL4pse06ERwuFA4iIk3I3df6axw6C4WDiEgTcovKfZkMDgcKBxGRRlTW1LK3pJJ+KR1/+e1j4Vk4mNlsM9trZp8202aKmX1iZqvN7AOvahERaa1d+yqAtj2NtSPxcuTwJHB+Uy+aWQrwEDDDOTcKuMzDWkREWiV3X9teANfReBYOzrl5QGEzTb4OvOyc2x5qv9erWkREWuvQNQ4aObS7YUB3M5trZsvM7NtNNTSz680s28yy8/Ly2rFEEemqcvaVEzCavLlPZ+dnOEQC44EvAucB/2tmjS456Jx71DmX5ZzLSk9Pb88aRaSLyi0qp1dSLFERXfO8HT+vkM4B8p1zpUCpmc0DxgAbfKxJRASAnKKyLjvfAP6OHP4FnGVmkWYWB5wGrPWxHhGRQ7ryBXDg4cjBzJ4FpgBpZpYD3A1EATjnHnbOrTWzt4CVQBB4zDnX5GmvIiLtpTbo2F1c0aVHDp6Fg3NuZgva3Afc51UNIiLHYs/+CmqCrkuPHLrmTIuISDO6+jUOoHAQEWmgq1/jAAoHEZEGDo4c+mrkICIiB+UUlZMaH01cdNe9H5rCQUTkCLn7uu5S3QcpHEREjpDbxS+AA4WDiMhhnHNd/gI4UDiIiBymsLSKiuqgRg5+FyAiEk4OXeOgkYOIiBx08BoHjRxEROSQgyOHrnwBHCgcREQOs3rnftITY0juFuV3Kb5SOIiI1LNsWxFZJ3THzPwuxVcKBxGRkL0lFWwvLGP8Cd39LsV3CgcRkZCPtxUBcIrCQeEgInLQsm1FREcGGN032e9SfKdwEBEJyd5WxJiMZKIj9atR/wIiIkBFdS2f5hbrkFKIwkFEBFiVW0x1rSPrhFS/SwkLCgcREermGwBOGZDicyXhQeEgIkJdOAxKi6dHQozfpYQFz8LBzGab2V4z+/Qo7U41s1ozu9SrWkREmuOc4+NtRZpvqMfLkcOTwPnNNTCzCOA3wNse1iEi0qytBWUUlFbp4rd6PAsH59w8oPAozb4LvATs9aoOEZGjOTjfkKVwOMS3OQcz6wd8GXjYrxpERACWbSskKTaSwekJfpcSNvyckL4f+LFzrvZoDc3sejPLNrPsvLy8dihNRLqSZaH5hkCgay+2V5+f4ZAFPGdmW4FLgYfM7JLGGjrnHnXOZTnnstLT09uzRhHp5IrLq9mw54AOKR0h0q8dO+cyDz42syeB151zr/pVj4h0TR9v12J7jfEsHMzsWWAKkGZmOcDdQBSAc07zDCJy3NbvLuF///UpU4anc+kpGfRMim31Z3y8rYiIgDG2vy5+q8+zcHDOzWxF21le1SEindcjH2xi2bYilmwp5PfvbGDq8J5ccWp/zhnRs0XzB/srqnlu6Q7Gn9CduGjfDqSEJf1riEiHtK+sitdX7WLmhP5cNSmTF7J38NKyXN5bu4eJg3rw20tPpn9qXLOfcd9b6yk4UMnsK09tp6o7Di2fISId0ivLc6mqCTJzwgAGpydw5wUnsvjOc7j3KyexKreY8++fx3NLtuOca/T9y7YV8dRH25h1RiYnZej+DUdSOIhI2MorqWRHYVmD7c45nl2ynTH9UxhV78Y8UREBZk4YwFu3nsXJGSnc8fIqrnpyKbuKyw97f3VtkLteWUXvpFhumz7M8350RAoHEQlbP3hxBRf9aQG7iysO275sWxEb9hzg6xP6N/q+jO5xPH3tafxsxig+3FzAtN9/wGPzN1NdGwTg8QVbWLe7hJ/NGEVCjI6uN0bhICJhqaSimsWb8ikur+aHL64gGPz88NAzS7aTEBPJRSf3bfL9gYBx5RkDeefWyUzITOUXb6zlS39awJwVO7n/vQ1MH9mL6aN6t0dXOiSFg4iEpXkb8qmudXz1lAwWbMxn9sItABSXVfPGyl1cMq4v8S34q39AjzhmzzqVR741nv3l1dzy7HIizPjpjFFed6FD03hKRMLSf9btISUuit989SSKy6v57VvrOXNoGh9uKqAyNBHdUmbGeaN6c9bQNB6fv4XhvRPpm9LNw+o7PoWDiISd2qBj7vo8pg7vSWREgN989STOu38+33v2E4LOMSYj+bCJ6JaKi47ku18Y6kHFnY8OK4lI2Fm+vYjC0irOGdETgB4JMdx32cms31PCZ3sPtGrUIMdG4SAiYee9tXuJDBiTh3++0ObU4T257qxM+ibH8qUxTU9ES9vQYSURCTvvr9vDhMxUkmKjDtt+1xdH8qPzRxAVob9rvaZ/YREJKzsKy9iw5wBfOLFXo68rGNqH/pVFJKy8t3YPAF8IzTeIPxQOIhJW3l+3l8Hp8QxMi/e7lC5N4SAiYaOkorpuuYsmDilJ+1E4iEjYmP9Z3VXR5+iQku8UDiISNt5bu4fkblGM1y07fadwEJGwsG73ft5YuYtzR/YiUmck+U7fARHxXVlVDTc9/TFJ3aL48fkj/C5HUDiISDuprg3yjw+3sXNfeYPX7v7Xajbnl3L/18aSnhjjQ3VyJIWDiLSLV5bn8r+vfsp5f5zH80s/v33nq8tzeXFZDjdNGcKkIWk+VykHafkMEfGcc47ZC7YwpGcCPeKj+fFLq3hj1W5uPHsQd72yilMHdufWaVotNZx4NnIws9lmttfMPm3i9W+Y2crQ1yIzG+NVLSLir4UbC1i3u4Qbzh7Es9edzj0Xj2LplkK+/thHREUGeOCKcZqEDjNejhyeBP4M/L2J17cAk51zRWZ2AfAocJqH9YiITx5bsJm0hBhmjO1LIGB8e+JApgzryR/eXc9lWf11450w5Fk4OOfmmdnAZl5fVO/ph0CGV7WIiH827i1h7vo8bjt3GDGREYe2D+gRx/1XjPOxMmlOuIzjrgH+3dSLZna9mWWbWXZeXl47liUix+vxBVuJjgzwjdN0g56OxPdwMLOp1IXDj5tq45x71DmX5ZzLSk9Pb6qZiISZwtIqXv44h6+e0o8eCTpFtSPx9WwlMzsZeAy4wDlX4GctItL2nv5wG5U1Qa6elOl3KdJKvo0czGwA8DLwLefcBr/qEBFvVNbU8rfF25g8LJ2hvRL9LkdaybORg5k9C0wB0swsB7gbiAJwzj0M/AToATxkZgA1zrksr+oRkfb12Pwt5B+o5JozNWroiLw8W2nmUV6/FrjWq/2LiH8+2JDH795Zz0Un9+GsobrquSPyfUJaRDqX7QVl3PLscob3SuS3l55M6MiAdDAKBxFpM2VVNVz/j2ycczzyrfHERWuFno5K3zkRaRPOOX780irW7ynhiVmnckIP3QO6I9PIQUTaxIvZOby2Yic/nD6cKcN1m8+OrkXhYGaDzSwm9HiKmd1iZineliYiHclrK3cyKD2e70wZ7Hcp0gZaOnJ4Cag1syHA40Am8IxnVYlIh1JZU8vSrYWcPTRdE9CdREvDIeicqwG+DNzvnPs+0Me7skSkI1m+fR8V1UHOGNzD71KkjbQ0HKrNbCZwJfB6aFuUNyWJSEezaGM+AYPTFQ6dRkvD4SpgIvBL59wWM8sEnvKuLBHpSBZszOfkjBSSYvU3Y2fRolNZnXNrgFsAzKw7kOic+7WXhYlIx1BSUc2KnGJunDzI71KkDbX0bKW5ZpZkZqnACuAJM/uDt6WJSEewZEshtUHHpCFaJqMzaelhpWTn3H7gK8ATzrnxwDTvyhKRjmLhxgJiIgOcMqC736VIG2ppOESaWR/gcj6fkBYRYeHGfE4dmEpsVMTRG0uH0dJwuAd4G9jknFtqZoOAz7wrS0Q6grySStbvKeGMITpLqbNp6YT0i8CL9Z5vBr7qVVEi0jEs2pQPwJmab+h0WjohnWFmr5jZXjPbY2YvmVmG18WJSHhbtLGApNhIRvVN9rsUaWMtPaz0BDAH6Av0A14LbRORTiqnqIyK6tomX3fOsWBjPhMH9yAioCUzOpuWhkO6c+4J51xN6OtJIN3DukTERwUHKvnC7z9gyn1zeTF7B7VB16DN9sIycveV6xTWTqql4ZBvZt80s4jQ1zeBAi8LExH//GfdXiprgiTGRnL7P1dy0Z8WMG9D3qHXD44aAIVDJ9XSm/1cDfwZ+CPggEXULakhIp3Qu2v20Cc5lrdvPZs3Vu3it2+v49uzlxATGaA26KgJjSR6J8UyKE039emMWnq20nZgRv1tZnYrcL8XRYmIfyqqa5n/WR6Xje9PIGB8aUxfpo/qxQvZOeQUlREZMCIDASIDxqmZqVqiu5M6ntuE3obCQaTTWfBZPhXVQc4d2evQtpjICL51+gk+ViXt7XhuE9rsnwtmNjt06uunTbxuZvagmW00s5Vmdspx1CIibeS9tXtIiInk9EG6sK0rO55waHj6wuGeBM5v5vULgKGhr+uBvx5HLSLSBoJBx3tr9zJ5eDrRkbrFfFfW7GElMyuh8RAwoFtz73XOzTOzgc00uRj4u3POAR+aWYqZ9XHO7Wq+ZBHxyvId+8g/UMn0eoeUpGtqNhycc4ke7rsfsKPe85zQtgbhYGbXUze6YMCAAR6WJNK1vbd2D5EBY8qwnn6XIj7zc9zY2JxFo4eqnHOPOueynHNZ6em69k7EK++u2cOEzFSS43RHt67Oz3DIAfrXe54B7PSpFpEub0t+KRv3HjjsLCXpuvwMhznAt0NnLZ0OFGu+QaRt/H3xVu54aSUb95a0+D3vrdkDwLQTFQ5yfNc5NMvMngWmAGlmlgPcDUQBOOceBt4ELgQ2AmXoimvpot76dBcvf5zLfZeNIbnb8R/O2VFYxi9eX0tVbZDns3dw4eg+3DR1CCP7JjX7vnfX7GFE70T6p8Yddw3S8XkWDs65mUd53QE3ebV/kY7i/XV7eWfNHvbMXsI/rplAUuzxBcQf3t2AGbxxy5m8uWoXf1u0jTdW7WLaib34/rlDG11ee9GmfLK3FXLz1CHHtW/pPHQis4jPCkurSYqNZM3OYr79+BJKKqqP+bNW7yzm1U9yuWpSJqP6JnP7eSNY+ONz+P60YSzZUsAXH1zAd55exmd7SnDOsWhjPpc/spiv/99HpCXEcOn4/kffiXQJVvcHfMeRlZXlsrOz/S5DpM185aGFdIuO4MqJA/nO0x9zUkYyf796AgkxkeTuK2fFjmK2FpRy3qheDOnZ/Nnl3569hBU79jHvR1MbHKIqLq/m8fmbeXzBFsqqaxmSnsBnew/QKymGGycPZuaEAboPdCdmZsucc1ktbe/ZYSURaZnC0ipO6p7C9FG9+fPXT+HmZz7mwgfnU15VS/6BqkPtfvfOes4b2ZvvTB3MyRkpDT5n4cZ85m3I464LT2x07iK5WxS3TR/OrEmZPDJvE4s3FfCzGaP42qn9FQrSgMJBxGcFpVX0iI8G4PzRvXnoG6fw8AebyExLYGz/ZMb0T6FXUixPf7iNJxdt5a3VuzlzSBrfmngCU4anExMZQTDo+PW/19EvpRvfmtj8Anmp8dHcecGJ7dE16cAUDiI+qqoJUlJRQ/e46EPbpo/qzfRRvRu0vW36cK47exDPfLSdxxds4YZ/LCO5WxQXntSHXkkxrMot5veXjdEoQNqEwkHER/vK6g4bpSZEH6VlncTYKG6YPJhrzsxkwcZ8Xl2ey6vLcymvrmVE70QuGdfPy3KlC1E4iPiooLQuHA4eVmqpyIgAU4b3ZMrwnpRW1jB3fR6j+iYREdCNd6RtKBxEfFQYCofUVoZDffExkXzx5D5tVZIIoOscRHx1rCMHEa8pHER8VBQKh+4KBwkzCgcRHxWUVmHGYWcriYQDhYOIjwpLK0npFqWJZAk7CgcRHxWWVh3XZLSIVxQOIj4qOKBwkPCkcBDxUVGZwkHCk8JBxEd1h5Vi/C5DpAGFg4hPgkFHUVm1rnGQsKRwEPFJcXk1tUGnw0oSlhQOIj4paIOlM0S8onAQ8UlRmcJBwpfCQcQnBQcUDhK+FA4iPjm4ImuPFt7LQaQ9eRoOZna+ma03s41mdkcjrw8ws/+a2XIzW2lmF3pZj0g4KSytBLSukoQnz8LBzCKAvwAXACOBmWY28ohm/wO84JwbB1wBPORVPSLhpqC0ivjoCN3WU8KSlyOHCcBG59xm51wV8Bxw8RFtHJAUepwM7PSwHpGwUlRa1eLbg4q0Ny/DoR+wo97znNC2+n4KfNPMcoA3ge829kFmdr2ZZZtZdl5enhe1irS7Al0dLWHMy3BobA1id8TzmcCTzrkM4ELgH2bWoCbn3KPOuSznXFZ6eroHpYq0v8LSKl0dLWHLy3DIAfrXe55Bw8NG1wAvADjnFgOxQJqHNYmEjcLSKk1GS9jyMhyWAkPNLNPMoqmbcJ5zRJvtwBcAzOxE6sJBx42k03POUVBapdNYJWx5Fg7OuRrgZuBtYC11ZyWtNrN7zGxGqNkPgOvMbAXwLDDLOXfkoSeRTqesqpaqmqAugJOwFenlhzvn3qRuorn+tp/Ue7wGmORlDSLhqFDrKkmY0xXSIj44uOieJqQlXCkcRHxw6OpohYOEKYWDiA8KS6sBjRwkfCkcRHxwcOSgOQcJVwoHER8UlFYRHREgIcbTc0JEjpnCQcQHhQeqSI2PxqyxhQRE/KdwEPFBYWmVJqMlrCkcRHxQWKZ1lSS8KRxEfFBYWqXJaAlrCgcRHxyccxAJVwoHkXZWWVNLSWWNDitJWFM4iLSzotAFcJqQlnCmcBBpZ4VaV0k6AIWDSDvTiqzSESgcRNpZQWjpDN3oR8KZwkGknR0cOegWoRLOFA4i7aywtAozSFE4SBhTOIi0s8LSKrrHRRMR0LpKEr4UDiLtTFdHS0egcBBpZ7v3V5CeEON3GSLNUjiItLMt+aVkpsf7XYZIszwNBzM738zWm9lGM7ujiTaXm9kaM1ttZs94WY+I3wpLq9hXVs2gNIWDhDfPbkNlZhHAX4BzgRxgqZnNcc6tqddmKHAnMMk5V2RmPb2qRyQcbMk/AMAgjRwkzHk5cpgAbHTObXbOVQHPARcf0eY64C/OuSIA59xeD+sR8d2mvFIABqUl+FyJSPO8DId+wI56z3NC2+obBgwzs4Vm9qGZne9hPSK+25JfSlSEkdG9m9+liDTLy7ubN3YSt2tk/0OBKUAGMN/MRjvn9h32QWbXA9cDDBgwoO0rFWknm/MOMCA1jsgInQsi4c3L/0NzgP71nmcAOxtp8y/nXLVzbguwnrqwOIxz7lHnXJZzLis9Pd2zgkW8tiW/lEwdUpIOwMtwWAoMNbNMM4sGrgDmHNHmVWAqgJmlUXeYabOHNYn4pjbo2FpQxmBNRksH4Fk4OOdqgJuBt4G1wAvOudVmdo+ZzQg1exsoMLM1wH+B251zBV7VJOKnnfvKqaoJkqnTWKUD8HLOAefcm8CbR2z7Sb3HDrgt9CXSqW3KO3gaqw4rSfjTrJhIO9mSX3caq0YO0hEoHETayea8UhJjI0nTTX6kA1A4iLSTLfmlDEqLx0xLdUv4UziItJPNeQc03yAdhsKhg8krqeRXb65l7a79fpcirVBeVcvO4grNN0iHoXDoYH7xxhoenbeZCx+cz63PLWd7QZnfJUkLHJyM1oJ70lF4eiprOPlwcwF/eGcDj83KIik26pg+Y9m2QuZ/lk90ZICYyAhiIgPEx0TQKzGWnkmx9E6OJSHGu3/S7K2F/OuTnVw1aSCxURE8sXALr6/cxddPG8B3zxlKemLb3ECmNuhYvr2I2KgI0hNjSI2PJkrLPRwXnakkHU2XCYfoyABLthby7uo9fHV8Rqve65zj8QVb+NWbawkeuTrUEdISovnReSO4LCujwcSjc46VOcVkpse3OqBqg46fvraaPsmx3H7ecOKiI5l1xkAe/M9nPP3Rdl5alsONkwdz7VmD6BYd0arPru9AZQ3fe3Y5/1n3+QK5ZtAjPpqTM1I4Y3APTh/Ug5F9kgjoHsgttjl0jYPCQTqKLhMO4/qnkNG9G3NW7GxVOFRU13Lny6t4ZXku543qxX2XjSE6IkBlTZDKmlpKKmrYu7+SPfsr2L2/gvfX7uVHL63k35/u4t6vnEzv5Fig7q/++95ez0dbChnRO5Gnrj2NtFbcKvLF7B18mrufB64YS1x03betV1Isv/zySVxzZia/eWsdv393A099tI0fnDucsQNSKC6vprismuLyajLT4xnXP6XZM2V27ivnmr9ls2FPCXdeMILMtHjyDlSSV1LJzn3lZG8t4v1QaCR3i+KqSQO5eeoQLSLXAlvyS+mTHHvoeycS7qzuIuWOIysry2VnZx/Te3/z1joenbeZpXdNa9EN3nP3lXPDP7JZvXM/t00bxk1Thxz1r+Vg0PH3xVv59VvriI4I8L1pw1jwWR7/XZ9HWkIMl2Vl8MTCLWR0j+OZa0+jZ1LsUesoLq/mnN/NZVB6PC/cMLHJX/BLtxbyqzfXsnz7vkZfH5ORzNVnZnLB6D5ERx7+C/2THfu47u/ZVFTV8udvnMLkYY0vcLi7uILFm/N569PdvL16DxMGpnL/FWPpm+L/EtTF5dV8sCGPycPSSe52bIcOvXLxXxaSEBPB09ee7ncp0kWZ2TLnXFaL23elcFizcz8XPjifX1wymm+efkKT7YJBx/PZO7j3zbU4B3/82limjezVqn1tzS/lhy+uIHtbEcndorhx8mCuPOME4qIj+XBzAVc/uZTeSbE8c93ph0YXTfn562uYvXALr918JqP7JTfb1jnH/M/yKS6vJrlbFMndokiMjWThxnyeWLiVzfml9EqKYerwnlTVBCmtqqGsqpYlWwpJT4xh9qxTGdYrsUV9fGV5Dv/zyqdERQa479IxTB6WztKthfxn7V7eX7eHvSWVDO2ZwPDeiQzvncTJGcmMH9Ddk8NRzjlufGoZb6/eQ0xkgPNH9+ZrWf05fVCPQ/sLBh2VNcEWHXarrKnl3TV7eCE7h8LSSq6cOJBLxvU7prkX5xxjfvYOM8b25ReXnNTq94u0BYVDM5xzTPvDB6QlxPD8DRMbbbNxbwl3vryKpVuLOC0zlV9/9eRjPk5cG3Qs21bEiD6JDeYYsrcWMuuJpfRIiObXXzmZxNhIIiOMqIgANbWOgtJKCkur2F1cwa//vY7LsjK49ysnH1MdBwWDjg825DF74RbW7NxPXEwE8dGRxEVH0D81jv+9aGSrDnVB3eGS7z77MZ/m7ic+OoLSqlqiIwOcMbgHA3vEs2FPCet3l1BQWgVARvduXDo+g0vHZ5DRPe64+lPfm6t28Z2nP+aqSQOpDTpeXZ7L/ooa0hNjiAoYJRU1HKiqwTmYMjyduy48kaGNhOD63SU8v3QHryzPoaismr7JsSR1i2Ld7hL6pXTjhsmDuDyrP7FRLZ/XyT9QSdYv3uMnF43k6jMz26zPIq2hcDiKB977jPv/s4HFd3zhsL/YnXM8NHcT97+3gbjoSO668MRGJ5Xb0vLtRXx79hJKKmqabdcvpRtzbp5Ej1b+4m4vlTW1PPTfTeQdqGTq8J5MGtKjwbH1vJJKFm3K58XsHBZuygdg4qAenDOiJxMH9+DE3sc+wb2vrIppf/iA3smxvPqdSURGBKioruXt1bt5f91eoiICJMZGkhgbRVVNkKc/2kZZVS0zJ/Tn+9OGERkRYM6KnbyYvYOVOcVERRjTR/bm8lP7c+aQNAIGc9fn8ef/bnXNrT0AAAs7SURBVGTZtiJ6JsZwz8WjOX907xbVt3RrIZc9vJgnrjqVqcN1m3Txh8LhKDbnHeCc33/A/3zxRK49a9Ch7a8sz+H7z6/giyf14WcXj2r1X9DHandxBet276e61lFTG6SqNkhEwEiNj6ZHfAzd46NIjYvuVJO+OwrLeOnjHOZ8spPNoVM8U+KimDAwlaRuUdQGHbVBR9A5+qfGMX5Ad045oXuT80Q/eGEF//oklzk3n8nIvklH3X9haRUPvLeBpz7aTreoCKprg1TWBBnRO5HLs/pzybh+je7LOcdHWwq557U1rNm1nxlj+vLTGaMOa7tzXzlLtxYyITOVPsl18zDPL93Oj19axbzbpzKgR9uNlkRaQ+HQAhf9aT4RgQD/umkSUPfL6sIH5jOiTyLPXT+RCJ2i2W52FZezeFMBizYVkL21kKqaIIGAEREwDMgpKqcmdP7woLR4Jg9P5xunDWBIz7pDQh9syOPK2Uu4eeoQfnje8Fbte+PeAzw0dyOJMZFcltWfUX2TWjRSrK4N8te5m/jT+5+R3C2KH503gp3F5by7Zg+rd9ZduZ4QE8nt5w3nm6efwG/fXscTC7ay9ufn6/8t8Y3CoQUe+WAT9/57HR/cPoWM7nFc8ehi1u0q4c3vnUX/VP1lF07Kq2pZlVvMsm1FZG+tuwixqjbI6YNSmTlhAL99az2xUQHeuOWsVs0DtIW1u/bzwxdXsHrnfsxg/IDuTBvZi3H9U/jzfzcy/7N8xg1IIeigrLKGd2+b3K71idTX2nDokiddXzSmL/f+ex2vr9yFc46lW4v449fGKBjCULfoCCZkpjIhMxUYTP6BSl7I3sEzH23ne899ghn888aJ7R4MACf2SeLVmyaxdEshw3onHnYockJmKq9+ksvPX19LYWkV01t5tpuI37rkyAHg0r8uYkdRGQUHqrjwpD48cMVYLaXcgdQGHfM25FFVG+S8US2bGPZDYWkVf527kcnDenLm0DS/y5EuTCOHFvrSmL7cPWc1/VK68fNLRisYOpiIgDF1RPif+ZMaH81dXxzpdxkirdZ5ToFppS+N6cupA7vzwBVjw+5qWhERv3XZkUNqfDQv3niG32WIiISlLjtyEBGRpnkaDmZ2vpmtN7ONZnZHM+0uNTNnZi2eLBEREe94Fg5mFgH8BbgAGAnMNLMGM3NmlgjcAnzkVS0iItI6Xo4cJgAbnXObnXNVwHPAxY20+znwW6DCw1pERKQVvAyHfsCOes9zQtsOMbNxQH/n3OvNfZCZXW9m2WaWnZeX1/aViojIYbwMh8YuHDh0xZ2ZBYA/Aj842gc55x51zmU557LS0xu/CY2IiLQdL8MhB+hf73kGsLPe80RgNDDXzLYCpwNzNCktIuI/L8NhKTDUzDLNLBq4Aphz8EXnXLFzLs05N9A5NxD4EJjhnDv+tTFEROS4eHYRnHOuxsxuBt4GIoDZzrnVZnYPkO2cm9P8JzRu2bJl+Wa2rd6mZKC4hY/TgPxj2W8jn3ksbRp77chtrekPHF+f2rs/Rz4/+Lg9+9NcO/UnvH+GumJ/jtx2PP1p+t7IjXHOdegv4NGWPqYulNpkX8fSprHXjtzWmv4cb5/auz/NfF/arT/NtVN/wvtnqCv2pyV9aMv+1P/qDFdIv9bKx221r2Np09hrR27rzP058vlrTbQ5Vi39nKbaqT/h/f9cV+zPkdu87s8hHW7J7uNhZtmuFUvWdgSdrU/qT3hTf8JbW/anM4wcWuNRvwvwQGfrk/oT3tSf8NZm/elSIwcREWmZrjZyEBGRFlA4iIhIAwoHERFpQOEQYmZnmdnDZvaYmS3yu57jZWYBM/ulmf3JzK70u57jZWZTzGx+6Hs0xe962oKZxZvZMjO7yO9a2oKZnRj6/vzTzP4/v+s5XmZ2iZn9n5n9y8ym+13P8TKzQWb2uJn9syXtO0U4mNlsM9trZp8esb1FNxsCcM7Nd87dCLwO/M3Leo+mLfpD3fLo/YBq6ta58k0b9ccBB4BYOkd/AH4MvOBNla3TRj9Da0M/Q5cDvp4e2kb9edU5dx0wC/iah+UeVRv1Z7Nz7poW77Strqbz8ws4GzgF+LTetghgEzAIiAZWUHfToZOoC4D6Xz3rve8FIKmj9we4A7gh9N5/doL+BELv6wU83Qn6M4269cZmARf52Z+26lPoPTOARcDXO0N/Qu/7PXBKJ+pPi34feLa2Untyzs0zs4FHbD50syEAM3sOuNg5dy/Q6DDezAYAxc65/R6We1Rt0R8zywGqQk9rvav26Nrq+xNSBMR4UWdLtdH3ZyoQT90Pc7mZvemcC3paeDPa6nvk6tZMm2NmbwDPeFdx89roe2TAr4F/O+c+9rbi5rXxz1CLdIpwaEJjNxs67SjvuQZ4wrOKjk9r+/My8CczOwuY52Vhx6hV/TGzrwDnASnAn70t7Zi0qj/OubsAzGwWkO9nMDSjtd+jKcBXqAvvNz2t7Ni09mfou9SN8JLNbIhz7mEvizsGrf3+9AB+CYwzsztDIdKkzhwOzd5sqDHOubs9qqUttKo/zrky6sIuXLW2Py9TF3jhqtX/vwE4555s+1LaTGu/R3OBuV4V0wZa258HgQe9K+e4tbY/BcCNLf3wTjEh3YSj3Wyoo1F/wltn6w90vj6pP63QmcOh2ZsNdUDqT3jrbP2Bztcn9ac1/JyBb8OZ/GeBXXx+2uY1oe0XAhuom9G/y+861R/1J1y/Oluf1J/j/9LCeyIi0kBnPqwkIiLHSOEgIiINKBxERKQBhYOIiDSgcBARkQYUDiIi0oDCQToFMzvQzvt7zMxGttFn1ZrZJ2b2qZm9ZmYpR2mfYmbfaYt9izRF1zlIp2BmB5xzCW34eZHOuZq2+ryj7OtQ7Wb2N2CDc+6XzbQfCLzunBvdHvVJ16SRg3RaZpZuZi+Z2dLQ16TQ9glmtsjMlof+Ozy0fZaZvWhmrwHvWN3d5+Za3Z3N1pnZ06FlnAltzwo9PmB1d91bYWYfmlmv0PbBoedLzeyeFo5uFlO32iZmlmBm/zGzj81slZldHGrza2BwaLRxX6jt7aH9rDSzn7XhP6N0UQoH6cweAP7onDsV+CrwWGj7OuBs59w44CfAr+q9ZyJwpXPunNDzccCt1N13YRAwqZH9xAMfOufGULc8+nX19v9AaP9HXRDNzCKAL/D5+jgVwJedc6cAU4Hfh8LpDmCTc26sc+52q7uF5VDq1vcfC4w3s7OPtj+R5nTmJbtFpgEjQ3/sAySZWSKQDPzNzIZSt8RxVL33vOucK6z3fIlzLgfAzD4BBgILjthPFXV32wJYBpwbejwRuCT0+Bngd03U2a3eZy8D3g1tN+BXoV/0QepGFL0aef/00Nfy0PME6sIiHO/jIR2EwkE6swAw0TlXXn+jmf0J+K9z7suh4/dz671cesRnVNZ7XEvjPzPV7vPJu6baNKfcOTfWzJKpC5mbqLuPwDeAdGC8c67azLZSdw/tIxlwr3PukVbuV6RJOqwkndk7wM0Hn5jZ2NDDZCA39HiWh/v/kLrDWVC3nHKznHPFwC3AD80siro694aCYSpwQqhpCZBY761vA1eb2cFJ7X5m1rON+iBdlMJBOos4M8up93Ubdb9os0KTtGv4/C5YvwXuNbOF1N2k3Su3AreZ2RKgD1B8tDc455ZTd6P4K4Cnqas/m7pRxLpQmwJgYejU1/ucc+9Qd9hqsZmtAv7J4eEh0mo6lVXEI2YWR90hI2dmVwAznXMXH+19IuFAcw4i3hkP/Dl0htE+4Gqf6xFpMY0cRESkAc05iIhIAwoHERFpQOEgIiINKBxERKQBhYOIiDSgcBARkQb+H5J75vd+PryKAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.lr_find(suggestions=True)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.LineCollection at 0x7f94b690ba50>"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEKCAYAAAD5MJl4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5dn/8c+VPSFkgSxA2PdVtggioqho1Sq4V7SLVkut1tbazf7aR5/a2s0uj3UpbnVpFWtxt1jEVkEEhKDIEhZJ2AKEhAQCCSHb3L8/ZsCQfZvMJPm+X695OefMPedct0PmmvvcyzHnHCIiItWFBDoAEREJPkoOIiJSi5KDiIjUouQgIiK1KDmIiEgtSg4iIlJLWKADaK6kpCQ3cODAQIchItKhrF279qBzLrmp5Ttcchg4cCAZGRmBDkNEpEMxs13NKa/LSiIiUouSg4iI1KLkICIitSg5iIhILUoOIiJSi5KDiIjU0uGGsoqIBIPjFVUUlJRTUFxGUWkFE/snEhvZeb5SO09NRET87LMDR3n+o928+ek+CkrKT3ltWEosz908hd7x0QGKrm0pOYiINMA5x5vr9/P3lbtYvbOQ8FDjwjG9GN07jp7dIugZG0lpRRU/fWUDVz26gudunsrQlNhAh91qSg4iIg1Yui2f7yz4hAE9Y7j74pFcPbkvSbGRtcoNSe7G1/66hmvmr+CvN57OxP6JAYi27ahDWkSkAVtzjwLw5h1nces5Q+pMDABj+sTz8remERcdzvVPfMQTy7JZlV3A4WPldZYPdmo5iIg0IOdQKfHR4cRFhTdadkDPbiy89UxueS6D+xdtPrk/NS6SWaNSufeyMUSEtew3uXMOM2vRe1tCyUFEpAE5h47RN7HpnczJ3SN57bYzyTtaxub9R9iae5T1e4t4/qPdHDhSxiM3TCQyLLRZMVRUebj9+Y+5fGIal4zr3dwqtIguK4mINCDnUGmzkgOAmZEaF8XMESl885whPHL9JO6bM4Z3Nx/gm39by/GKqpNlPR7HkswD/PTVDRw4crzWsZxz3PP6Rt7JPMDR4xWtrk9TqeUgIlIP5xw5h0o5e3iTb4NQr69OG0h4aAj/79UN3PJsBo9+eRLvZh5g/tIsth0oBryd3899fQqDkz8f7TR/aTYLVu/htplD+NLp/VsdR1Op5SAiUo/CknJKK6qa3XKoz9wp/Xng6vGsyDpI+i/f5a6XPiXEjAevm8Crt51JaXkV18xfyYacIgDeWr+P3/57C5ee1psfXDiiTWJoKrUcRETqkXOoFIC+iTFtdsyrJ/clOjyUVz7O4YYz+nPuiJSTHc3/vHUaX3lqNdc9vpI7Zw3ngXe2kj4gkd9fM56QkPbrjAYlBxGRen2eHNp21vMXT+vNF0+r3bE8ODmWV247k68+tZr7F21mYM8YHv9qOlHhzevAbgtKDiIi9cg5dAyAtDZODg1JjYvipW9OY/6yLK47vR89ukW027mrU3IQEalHc+Y4tKX4mHB+fNHIdj1nTeqQFhGpR3PnOHQmSg4iIvVoyRyHzkLJQUSkDifmOLTlSKWOxG/Jwcz+amZ5ZraxgTIzzWydmW0ys6X+ikVEpLnaeo5DR+PPlsMzwEX1vWhmCcCjwGzn3BjgGj/GIiLSLP6Y49CR+C05OOeWAYUNFLkeeMU5t9tXPs9fsYiINJe/5jh0FIHscxgOJJrZ+2a21sy+GsBYREROEYg5DsEkkPMcwoDJwPlANLDSzFY557bVLGhm84B5AP37t9/CUyLSdQVqjkOwCGTLIQf4t3OuxDl3EFgGjK+roHPucedcunMuPTm59asjiog0pivPcYDAJofXgRlmFmZmMcBUYHMj7xERaRddeY4D+PGykpktAGYCSWaWA9wLhAM45+Y75zab2b+B9YAHeNI5V++wVxGR9tKW93HoqPyWHJxzc5tQ5gHgAX/FICLSEl19jgNohrSISC17D3ftOQ6g5CAiUktXn+MASg4iIrV09TkOoOQgIlJLV5/jAEoOIiK1dPVhrKDkICJSS1efAAdKDiIip+jq93E4QclBRKSaQ8cqOFbetec4gJKDiMgpToxUUstBRERO0hwHLyUHEZFqdhdqjgMoOYiInGJDThFpCdFdeo4DKDmIiJzknGP1zkJOH5gY6FACTslBRMRnd+Ex8o+WkT6wR6BDCTglBxERnzU7DwFwupKDkoOIyAkZOwuJjw5nWEpsoEMJOCUHERGfNTsLSR+QSEiIBTqUgFNyEBEBCorLyMovUX+Dj5KDiAiQsetEf4NGKoGSg4gI4O1viAgLYVzf+ECHEhSUHERE8I5UmtA3gciw0ECHEhT8lhzM7K9mlmdmGxspd7qZVZnZ1f6KRUSkIaXlVWzcW0S6Limd5M+WwzPARQ0VMLNQ4LfAYj/GISLSoE/2HKLS4zS/oRq/JQfn3DKgsJFidwAvA3n+ikNEpDEZOw9hBpMGqOVwQsD6HMwsDbgCmN+EsvPMLMPMMvLz8/0fnIh0KWt2FjIitTvx0V17sb3qAtkh/X/Aj51zVY0VdM497pxLd86lJycnt0NoItJVVFZ5+HjXIV1SqiEsgOdOB140M4Ak4BIzq3TOvRbAmESki9mSe5SS8ip1RtcQsJaDc26Qc26gc24gsBC4TYlBRJrjWHklC1bvZo/vBj0tsWant2tULYdT+a3lYGYLgJlAkpnlAPcC4QDOuUb7GUREGvPEsh386d1tAEwb3JNr0vty0dhexEQ07avNOceiDftJS4imT0LXvvNbTX5LDs65uc0oe6O/4hCRzqmyysOC1buZMqgHZw1NYuHaHO566VPueX0Td84axk3TBxHayAJ6L67Zw5qdh/j1lePaKeqOQzOkRaRDendzHrlHjvONGYP5zvnDWPrDmfxj3hlMGdSDX/5rM1fPX8H2vKP1vv/AkeP8atFmzhjcg+tO79eOkXcMSg4iErScc1RWeep87e+rdtEnPorzRqYAYGZMHdyTp76WzoPXTWDHwRIueXA5j7y3nYoax3DO8T+vbaS80sNvrjwN38AYqUbJQUSC1o8WrufSh5ZzrLzylP3Z+cUs336Q66f2r3XpyMyYMyGNJd87h1mjU3hg8VYu/fNyVmYVnCzz9sZc3sk8wPcuGM7ApG7tUpeORslBRIJSWWUVizbsZ0vuUe57M/OU157/aDdhIca1DVwOSu4eyaM3TOaxr0ymuKySuU+s4o4Fn7Al9wj3vL6JsWlx3HLWIH9Xo8MK5DwHEZF6rcgqoKS8itMHJvLimj2cNSyJS0/rQ2l5FQvX5nDR2F6kdI9q9DhfGNOLc4Yn85f3s5i/NIs3P91HaIjx7NdPJyxUv4/ro+QgIkFpSeYBYiJCeeamKdzw5Ef85JUNjO+bwMrsAopKK/jyGQOafKyo8FC+d8Fwrp7clz8u2cZpfeMZ00f3bWiIkoOIBB2Px/Fu5gHOGZ5Mt8gwHpo7kUse/IDvvvgJFVWOYSmxTB3U/Elr/XrE8KcvTfBDxJ2P2lQiEnTW7y0i72gZF4xOBbxf6r+8Yiwf7z7Mhr1FfPmMARph5GdKDiISdJZk5hIaYieHqQLMmZDG3Cn9SIqN4IpJaQGMrmvQZSURCTpLMg9w+sBEEmIiTtn/qyvG8T+Xjm7y8hjScmo5iEhQ2VVQwrYDxVwwulet18xMiaGdKDmISFBZknkAgAt9/Q0SGEoOIhJU3sk8wMhe3enXIybQoXRpSg4iEjQKS8rJ2Fl4cpSSBI6Sg4gEjf9uycPjUHIIAkoOIhI0Fm/KpVdcFOPSNHs50JQcRCQoLMk8wJLMA1wxKU0T3IKAkoOIBNyewmN8/6V1jEuL585ZwwIdjqDkICLtpKC4jFueXcPr6/binDu5v6yyitue/xgHPHL9JCLDQgMXpJyk5CAi7eLpD3fy7uY8vvviOm55NoP9RaUA/PKtzWzYW8QfrhlP/54avhosNNVQRPyupKySv63axQWjUzljcE9+v3grF/xxGZeN78OC1buZd/ZgLhxTe0a0BI7fWg5m9lczyzOzjfW8foOZrfc9VpjZeH/FIiKB9Y81eygqreDWc4Zw81mDWHzn2YzvF8+C1btJH5DID78wItAhSg3+bDk8AzwMPFfP6zuAc5xzh8zsYuBxYKof4xGRAKis8vDU8h2kD0hk8oBEAPr3jOHvN09l2WcHGd83nnDdkS3o+O0Tcc4tAwobeH2Fc+6Qb3MV0NdfsYhI4Pxrw372Hi5l3tmDT9lvZpwzPLnWyqsSHIIlXd8MvF3fi2Y2z8wyzCwjPz+/HcMSkdZwzvH4smwGJ3dj1ijNeu5IAp4czOxcvMnhx/WVcc497pxLd86lJycnt19wItIqK7IK2LTvCN+YMZiQEE1s60gCOlrJzE4DngQuds4VBDIWEWl7jy3LJik2kism6s5tHU3AWg5m1h94BfiKc25boOIQEf/YkFPEsm353HjmAKLCNbGto/Fby8HMFgAzgSQzywHuBcIBnHPzgXuAnsCjvnVUKp1z6f6KR0Taz6GScm5/4WOSu0fy5TMGBDocaQG/JQfn3NxGXr8FuMVf5xeRwKio8nDb8x+Te+Q4/5h3hkYjdVAB75AWkc7lF29lsjK7gF9fMY6J/RMDHY60kJKDiLSZ5z/axXMrdzHv7MFcNVlTlzoyJQcRaRNrdxVy7+ubmDkimR9fNDLQ4UgrKTmISJt4ZsUu4qPD+fPciYRqTkOHp+QgIq3m8Tg+3H6Qc4YnExcVHuhwpA0oOYhIq2XuP0JhSTlnDUsKdCjSRpqUHMxsiJlF+p7PNLPvmFmCf0MTkY7ig88OAnDWUCWHzqKpLYeXgSozGwo8BQwCXvBbVCLSoXzwWT4je3UnJS4q0KFIG2lqcvA45yqBK4D/c859D+jtv7BEpKMoLa8iY+chtRo6maYmhwozmwt8DXjLt0+9TiLC6p2FlFd5mDFcKyZ3Jk1NDjcB04D7nXM7zGwQ8Hf/hSUiHcUH2/KJCA1hysAegQ5F2lCT1lZyzmUC3wEws0Sgu3PuN/4MTEQ6huXbD5I+MJHoCK282pk0dbTS+2YWZ2Y9gE+Bp83sj/4NTUSCXd6R42zJPcqMYbqk1Nk09bJSvHPuCHAl8LRzbjIwy39hiUhHsHy7dwjrDM1v6HSamhzCzKw3cC2fd0iLSBe3/LOD9OgWwejecYEORdpYU5PDfcBiIMs5t8bMBgOf+S8sEQm0tbsKOXDkeL2vO+dYvv0g04cm6f7QnVBTO6T/Cfyz2nY2cJW/ghKRwNp7uJSr568kIjSEm6YP4lvnDCE+5tTR69sOFJN3tIwZmt/QKTUpOZhZX+AhYDrggOXAd51zOX6MTUQC5J1NuTgH5wxP5rFlWbzw0S6+NXMo549KocrjqKxyvLV+H4DWU+qkmnqb0KfxLpdxjW/7y759F/gjKBEJrMWbchmeGsvjX01n8/4j/O7fW/it71Hd8NRY+iREByhK8aemJodk59zT1bafMbM7/RGQiARWYUk5q3cUcvu5QwEY1TuOp2+awro9h8k5dIywECMsJITQUGNEavcARyv+0tTkcNDMvgws8G3PBQoaeoOZ/RW4FMhzzo2t43UDHgQuAY4BNzrnPm5q4CLiH+9mHsDj4Atjep2yf0K/BCb002LMXUVTRyt9He8w1lxgP3A13iU1GvIMcFEDr18MDPM95gF/aWIsIuJHizflkpYQzZg+Gp7alTUpOTjndjvnZjvnkp1zKc65y/FOiGvoPcuAwgaKzAGec16rgATfXAoRCZDisko+2H6QC8ek4m3cS1fVmjvB3dXKc6cBe6pt5/j2iUiALN2aT3mlp9YlJel6WpMcWvuzoq73uzoLms0zswwzy8jPz2/laUWkPos35dKzWwSna4XVLq81yaHOL/JmyAH6VdvuC+yr80TOPe6cS3fOpScna4EvEX8or/Tw3pY8Zo1KJVQznru8BkcrmdlR6k4CBrR2cPMbwLfN7EVgKlDknNvfymOKCHD4WDlllR5Sm3HbzhVZBzlaVskXxqb6MTLpKBpMDs65Fg9iNrMFwEwgycxygHvx3T3OOTcfWIR3GOt2vENZGxv9JNIpHSopJ+dQKeP6xrfJ8ao8juuf+IitB45y2Wm9uXXmEEb2anzk0eJNuXSLCOXMIZrxLE2f59Bszrm5jbzugNv9dX6RjuLR97fz5PId/Paq07g2vV/jb2jEyx/nkLn/CBeMTuWdzAO8tm4f541M4baZQ0ivpy+hyuNYknmAmSNTiArXTXvEj8lBRJpmf9FxnIMfLVxPRZWHG6YOaPGxjpVX8vvFW5nQL4HHvzKZotIKnlu5i6c/3MHV81cyfWhPvnv+cKYM8iYJj8fxTmYuD7+3nYPF5XxxnEaTi5eSg0iAFZaUMy4tnpTukfz01Y1UVHq4cfqgk697PI5Dx8rpGRvZ6LGeWLaDvKNlPHrDJMyMhJgIvnP+MG6ZMYgXPtrN/KXZXPvYSs4c0pMLRqfywke7+SyvmIE9Y/jdVadx8VgNYRUvJQeRACsoLqd/zxgeuX4Sdyz4mP99M5N9Rccxgw05RWzYW8TR45WkD0jkm+cM4fyRKXXePyHvyHEeW5bFxWN71bp8FBMRxi0zBnPD1AE8/9Eu5i/NZkVWAcNTY3nwugl8cVxvwkJbM3hROhslB5EAKygpZ9KABCLCQnj4+knc+eI6Hl+WTXioMap3HLPH9yGlexT/XLuHbzyXwZDkbsw7ezCzx6cRHfF5/8Afl2yjosrD3RePrPdc0RGhJ5NEVn4xo3vH6UY9UiclB5EAOnHJqEe3CADCQ0N4aO5EfviFEfROiCIy7PMv/9vPHcKijbk8tjSLH7+8gfvezOSC0anMnuBNHi9l7OGm6YMY0LNbo+eNjghlbFrbjI6SzknJQSSAikorqPI4enb7vD8hJMQYmFT7Cz4sNITZ4/tw2Wm9WZVdyOvr9vL2xlxeW+edOxofHc4d5w1tt9ilc1NyEAmggpJyAHrGRjT5PWbGtCE9mTakJ/fNGcuybfm8vTGX80elkBDT9OOINETJQSSACorLAE5pOTRHRFgIs0anMmu0ZjVL29LwBJEAKvS1HE70OYgECyUHkQA62ILLSiLtQclBJIAKi73JIVF9BRJklBxEAqigpIy4qDAiwvSnKMFF/yJFAqigpJykJiyLIdLelBxEAqiwuFyd0RKUlBxEAqigpEyd0RKUlBxEAqiwpJweLZzjIOJPSg4iAeLxOApLyumpy0oShJQcRALkcGkFHqc5DhKclBxEAuTE0hnqkJZgpOQgEiAnFt3TUFYJRkoOIgGidZUkmPk1OZjZRWa21cy2m9nddbze38zeM7NPzGy9mV3iz3hEgsnJFVnV5yBByG/JwcxCgUeAi4HRwFwzG12j2M+Al5xzE4HrgEf9FY9IsDlxWUnrKkkw8mfLYQqw3TmX7ZwrB14E5tQo44A43/N4YJ8f4xEJKgXF5STEhBMeqqu7Enz8ebOfNGBPte0cYGqNMv8LvGNmdwDdgFl+jEckqHgnwKnVIMHJnz9ZrI59rsb2XOAZ51xf4BLgb2ZWKyYzm2dmGWaWkZ+f74dQRdrfweIyTYCToOXP5JAD9Ku23Zfal41uBl4CcM6tBKKApJoHcs497pxLd86lJycn+ylckfblnR2tYawSnPyZHNYAw8xskJlF4O1wfqNGmd3A+QBmNgpvclDTQLqEgpJyemikkgQpvyUH51wl8G1gMbAZ76ikTWZ2n5nN9hX7PvANM/sUWADc6JyreelJpNOp8jgOHSsnSZeVJEj5s0Ma59wiYFGNffdUe54JTPdnDCLB6PCxcpzTBDgJXhpDJxIAJ+Y49NTSGRKklBxEAqCg2Jcc1HKQIKXkIBIABSW+FVnVIS1BSslBJABOLLqnoawSrJQcRALgYPGJdZXCAxyJSN2UHEQCoLCkjMSYcMK0rpIEKf3LFAkAraskwU7JQSQADhaXaxirBDUlB5EA8K6rpJaDBC8lhw5m5syZzJw5M9BhNFlHi7e9FBSX6bKSBDUlB5F2Vlnl4XBphS4rSVBTchBpZ4eOVeCcZkdLcFNyEGlnJyfAaXa0BDElB5F2VlDsWzpDLQcJYkoOIu3sxIqsSepzkCCm5CDSzk5cVlLLQYKZkoNIOysoLsMMEmOUHCR4KTmItLOCknISYyIIDbFAhyJSLyUHkXZWUKx1lST4KTmItLP9R46TGqfOaAluSg4i7cg5R3ZeMUOSYwMdikiD/JoczOwiM9tqZtvN7O56ylxrZplmtsnMXvBnPCKBln+0jKNllQxO6hboUEQaFOavA5tZKPAIcAGQA6wxszecc5nVygwDfgJMd84dMrMUf8UjEgyy8ksAGJKiloMEN3+2HKYA251z2c65cuBFYE6NMt8AHnHOHQJwzuX5MR6RgMvKLwbQZSUJev5MDmnAnmrbOb591Q0HhpvZh2a2yswuqutAZjbPzDLMLCM/P99P4Yr4X1Z+MTERofSKiwp0KCIN8mdyqGsQt6uxHQYMA2YCc4EnzSyh1puce9w5l+6cS09OTm7zQEXaS3Z+CYOSuhGiOQ4S5PyZHHKAftW2+wL76ijzunOuwjm3A9iKN1mIdEpZ+RqpJB2DP5PDGmCYmQ0yswjgOuCNGmVeA84FMLMkvJeZsv0Yk0jAHK+oYu/hUiUH6RD8lhycc5XAt4HFwGbgJefcJjO7z8xm+4otBgrMLBN4D/ihc67AXzGJBNKOgyU4B4OTNYxVgp/fhrICOOcWAYtq7Lun2nMH3OV7iHRqGqkkHYlmSIu0k6y8EsxgkCbASQeg5CDSTrLyi0lLiCY6IjTQoYg0SslBpJ1kHyxmsC4pSQeh5NABVYVpAlVH4/E4svJKGKLOaOkglBw6mOKk0exJv4Mbn17NhpyiQIcjTZR75DilFVXqjJYOo8skh6PHK3h93V68A6RaxuNx5B8to6i0guMVVXg8LT9WSxSVVlA4YCZhpYWs23OYyx5ezjf/lsGW3CPtGoc0X7ZvwT0NY5WOwq9DWYPJkswD3PXSp6QlRJM+sEez37+74Bjfen4tm/ad+kUcHR5KSlwkqd2jSImLZEDPGG6aPoik2La/mcsf39mKJyya1M0Leee3z/PX5Tt58oNs3sn8gDnj+/D9C0fQr0dMq8/z9ob9/HHJNqLCQ0mKjSApNpKUuEjGpcUzZVBP3cWsBU4MYx2qloN0EF0mOVw4phdR4Rt4bd3eZieH97bkcec/1uGc4+6LRxIWYpRXeSir8FBcVkne0TIOHDnOxr1FvL0xlxc+2s29l41hzoQ+mHnX0Kmo8vDy2hyeXL6Di8b04vsXDj/5WlNs2lfE31btovuBdUQey6N7VDjfnTWMr505gPlLs3lmxQ7+tWE/N0wdwLfPG3pKciqv9BAeao2ez+NxPPifz3jwP58xsld3kmIjyC8uY/P+oxwsLqPS11IakdqdMwb3YO7U/ozsFdes/5ddVVZ+Md0jw0jurjvAScfQZZJDbGQYF4zuxb/W7+eeS8cQEdb4FbUTX5Z//u9njOoVx/wvT6Z/z4Z/mX924Cg/XLieO/+xjjc+3cfPZ49hZVYBD733GXsKS+kTH8XD722nqLSCn88e06QF2Dwexz2vbyIxJoKYPctPeS0hJoK7Lx7JTdMH8uB/PuNvq3bxUsYe0hKiOXK8wncJzENaQjRfmTaAL6X3I7GOX/7Hyiv5/kuf8vbGXK6e3Jf7rxhLZNjnQy7LKz1s2HuYVdmFrMou4KWMHBas3sOPLhrB16cPCpqF5I6VVxITEXz/rLPzSxicEtusHwQigRR8f0V+dPmEPrz56T6Wbctn1ujUBsvuPFjCz17byPLtB7lqkvfLMiq88fHpw1K78/K3zuTpD3fw+3e2MuN37wEwLi2en984hnNHpPCbt7fw2LJsSsor+d1VpxEW2nCievnjHNbuOsQDV5/GI/8pq7NMalwUv7piHLecNYjHlmZTVFpBfHQ4cdFhdI8KZ0XWQX7z9hb+tGQbcyb04byRqZRXeSgtr6SkrIp/rs1ha+4RfvbFUdx81qBaX2IRYSFMHtCDyQN6cPu5QyksKedHC9fzy39tZum2fP5wzXhS4qIoq6xi9Y5C3t+aT97RMg6nTSP82EGy84sZ0LMboX5MIvOXZvG7f29h+tAkrp7clwtH92rVnILCknLeWLeXwmMVXHd6P/okRLf4WFn5xUwb3LPF7xdpb9aaDtpASE9PdxkZGS16b0WVhyn3v8v0oUk8fP2kOsscr6hi/tIsHn0/i8jQEP7fF0dx3en9WvSLb3fBMZ5buZOpg3sya1TKyWM453j4v9v5w5JtXDy2Fw9eN7FWS6bK4zh0rJwDR47z1adWM6BnDAtvPZPzzjsXgPfff7/Z8WzNPcqzK3fy6sd7Ka2oOuW1hJhw/u9LE5g5ouk343PO8cLq3fzirUyiw0OZ1D+RldkFHCuvIiIshJTukeQUHgNfvbtHhnHJuN5cNbkvpw9MbNNf0Vtyj3DZQ8sZ1TuOguJy9h4upXtkGOePSiE8NITiskqKyyopLa/iorG9+Mq0Aae0jE6oqPKwdGs+C9fm8J8tB6iocphBqBmzJ/Th1nOGMDy1e7NiKy6rZOy9i/nhF0Zw+7lD26rKIs1iZmudc+lNLt+VkgPAz17bwMK1OWT87AJiI09tOH24/SA/fXUDOwuOMXt8H372xVGk+PGmLE8t38Ev3sokPNSICA0hPCyEsJAQKj0eikorOPHRhIYYr98+nbFp8cycORNoWXI4oai0gl0FJcREhBETEUq3iDC6RYY22oKpz/a8Yn788nryj5ZxzvBkzh2ZzLTBSURHhHL2eRdQEd2Du37+e1ZlF/L2xv0cK6+if48YrpiYxjkjkhmXFk94C88N3i/0yx/5kNyi47zzvbNJjIlg1Y4CXl67l6Xb8ggPDSE2MozYqDAqqjxs3HuEvonR/ODCEcwe34eQEGPTviJeXruXNz7dy8HicpJiI7h8QhpXp/clNjKMp5bv4MXVeyitqGLWqFTuuXR0o5cYT9iQU8RlDy9n/pcncdHY3i2up0hrKDk0Yu2uQq76y0r+cM14rprc95T91z62igE9YrhvzljOGpbUFuE26t3MA2TsOkbnbnUAAAuoSURBVERllYeKKg8VHkeoGYndIugRE05itwhG945jmO/Xalskh/ZUM96SskoWb8rl5Y9zWJFVgHMQExFK+sAeTB3Ug+5RYVRWOTzO+0hLiGHSgAR6x9d/SefP//mMPy7Zxl9umMTF4xr/8v3gs3x+vWgLmfuPMLp3HB7n2JJ7lPBQ4/yRqVw1uS8zRyTXSliHSsp5buUunvggm0qPhx9cOIKbpg865VJZlcex42Ax/XrEnGyZvL5uL999cR1Lvnf2yc9RpL01Nzl0qT4HgEn9E+nXI5rX1u09mRyKSiv4zoJ19EmI4rVvTycuKrzd4pk1OrXR/o/OpFtkGFdO6suVk/pSUFzG6h3eDu5V2YU8sHhrve/rFRfFpAEJzBiWzGXj+5xs9WXuO8JD//2My8b3aVJiAJgxLJnpdyTx+qd7efi/24mNCucXc8Zw6Wl96uysPyGxWwTfnTWMa0/vy89e3cgv/7WZNz/dx72zx5BbdJz/bM7j/a15FJSUMzi5G7+8fCxnDkkiK6+YEKPJLQ2RYNDlkoOZMWd8Go++v528o8dJjo3kJ6+s58CR4yz81pntmhi6up6xkVw8rvfJL/UjxyuoqPQQGmKEhBiGd5TPx7sP8cnuw6zddYhFG3L5xVuZzJnQh2vS+/GzVzcSHx3BfbPHNOvcISHGFRP7csXEvo0XrqF3fDRPfi2dN9fv5+dvbOLKR1cAEB8dzswRyUzol8DTH+7k+ic+4sqJaeQdLaN/tZaESEfQ5ZIDwOUT+/Dwe9t569P9RIWHsmhDLndfPJIJ/WrdvlraUV2JeXy/BMb3S+Cm6d4O8E/2HGbBR7t59ZO9LFi9B4DHvjK5wV/8/mBmzB7fhxlDk3hz/T5G9opjUv+Ek/02c6f059H3tvOXpVlUVDnOH9n0jn6RYNAlk8PQlO6M6RPHsyt3klt0nBnDkpg3Y3Cgw5JGmBmT+icyqX8i/3PZaF7/ZC9VHscXxvQKWEyJ3SL46rSBtfZHhYdy14UjmD0hjT8t2cbF4wIXo0hLdMnkAHD5hDTuX7SZpNgI/nDt+KCZxCVNExcVzlfq+FIONkNTYnnkhrqHTYsEsy6z8F5Nl09MY1TvOP70pQmkdNcS2CIi1XXZlkNy90je/u6MQIchIhKUumzLQURE6ufX5GBmF5nZVjPbbmZ3N1DuajNzZtbkCRoiIuI/fksOZhYKPAJcDIwG5prZ6DrKdQe+A3zkr1hERKR5/NlymAJsd85lO+fKgReBOXWU+wXwO+C4H2MREZFm8GeHdBqwp9p2DjC1egEzmwj0c869ZWY/qO9AZjYPmAfQv39/P4TacXSUNZVO6GjxioiXP1sOdU0cOLnKn5mFAH8Cvt/YgZxzjzvn0p1z6cnJyW0YooiI1MWfySEH6Fdtuy+wr9p2d2As8L6Z7QTOAN5Qp7SISOD5MzmsAYaZ2SAziwCuA9448aJzrsg5l+ScG+icGwisAmY751q+HreIiLQJvyUH51wl8G1gMbAZeMk5t8nM7jOz2f46r4iItJ5fZ0g75xYBi2rsu6eesjP9GYuIiDSdZkiLiEgtSg4iIlKLkoOIiNSi5CAiIrWYc67xUkHEzPKBXdV2xQNFTXyeBBxsxemrH7MlZep6rea+5tQHWlen9q5Pze0Tz9uzPg2VU32C+2+oK9an5r7W1GeAc67ps4idcx36ATze1OdARludqyVl6nqt5r7m1Ke1dWrv+jTwubRbfRoqp/oE999QV6xPU+rQlvWp/ugMl5XebObztjpXS8rU9VrNfZ25PjW336ynTEs19Tj1lVN9gvvfXFesT819/q7PSR3uslJrmFmGc65TLc/R2eqk+gQ31Se4tWV9OkPLoTkeD3QAftDZ6qT6BDfVJ7i1WX26VMtBRESapqu1HEREpAmUHEREpBYlBxERqUXJwcfMZpjZfDN70sxWBDqe1jKzEDO738weMrOvBTqe1jKzmWb2ge8zmhnoeNqCmXUzs7VmdmmgY2kLZjbK9/ksNLNvBTqe1jKzy83sCTN73cwuDHQ8rWVmg83sKTNb2JTynSI5mNlfzSzPzDbW2H+RmW01s+1mdndDx3DOfeCcuxV4C3jWn/E2pi3qA8zBex/vCrx35QuYNqqPA4qBKDpHfQB+DLzknyibp43+hjb7/oauBQI6PLSN6vOac+4bwI3Al/wYbqPaqD7Zzrmbm3zStppNF8gHcDYwCdhYbV8okAUMBiKAT4HRwDi8CaD6I6Xa+14C4jp6fYC7gW/63ruwE9QnxPe+VOD5TlCfWXjvjngjcGkg69NWdfK9ZzawAri+M9TH974/AJM6UX2a9H3g15v9tBfn3DIzG1hj9xRgu3MuG8DMXgTmOOd+DdTZjDez/kCRc+6IH8NtVFvUx8xygHLfZpX/om1cW30+PoeASH/E2VRt9PmcC3TD+8dcamaLnHMevwbegLb6jJxzb+C9F/y/gBf8F3HD2ugzMuA3wNvOuY/9G3HD2vhvqEk6RXKoRxqwp9p2DjC1kffcDDztt4hap7n1eQV4yMxmAMv8GVgLNas+ZnYl8AUgAXjYv6G1SLPq45z7KYCZ3QgcDGRiaEBzP6OZwJV4k/ei+soFUHP/hu7A28KLN7Ohzrn5/gyuBZr7+fQE7gcmmtlPfEmkXp05OVgd+xqc8eecu9dPsbSFZtXHOXcMb7ILVs2tzyt4E16wava/NwDn3DNtH0qbae5n9D7wvr+CaQPNrc+fgT/7L5xWa259CoBbm3rwTtEhXY8coF+17b7AvgDF0hZUn+DW2eoDna9Oqk8zdObksAYYZmaDzCwCb+ffGwGOqTVUn+DW2eoDna9Oqk9zBLIHvg178hcA+/l82ObNvv2XANvw9uj/NNBxqj6qT7A+OludVJ/WP7TwnoiI1NKZLyuJiEgLKTmIiEgtSg4iIlKLkoOIiNSi5CAiIrUoOYiISC1KDtIpmFlxO5/vSTMb3UbHqjKzdWa20czeNLOERsonmNltbXFukfponoN0CmZW7JyLbcPjhTnnKtvqeI2c62TsZvYssM05d38D5QcCbznnxrZHfNI1qeUgnZaZJZvZy2a2xveY7ts/xcxWmNknvv+O8O2/0cz+aWZvAu+Y9+5z75v3zmZbzOx53zLO+Pan+54Xm/eue5+a2SozS/XtH+LbXmNm9zWxdbMS72qbmFmsmf3HzD42sw1mNsdX5jfAEF9r4wFf2R/6zrPezH7ehv8bpYtScpDO7EHgT86504GrgCd9+7cAZzvnJgL3AL+q9p5pwNecc+f5ticCd+K978JgYHod5+kGrHLOjce7PPo3qp3/Qd/5G10QzcxCgfP5fH2c48AVzrlJwLnAH3zJ6W4gyzk3wTn3Q/PewnIY3vX9JwCTzezsxs4n0pDOvGS3yCxgtO/HPkCcmXUH4oFnzWwY3iWOw6u9Z4lzrrDa9mrnXA6Ama0DBgLLa5ynHO/dtgDWAhf4nk8DLvc9fwH4fT1xRlc79lpgiW+/Ab/yfdF78LYoUut4/4W+xye+7Vi8ySIY7+MhHYSSg3RmIcA051xp9Z1m9hDwnnPuCt/1+/ervVxS4xhl1Z5XUfffTIX7vPOuvjINKXXOTTCzeLxJ5na89xG4AUgGJjvnKsxsJ957aNdkwK+dc48187wi9dJlJenM3gG+fWLDzCb4nsYDe33Pb/Tj+VfhvZwF3uWUG+ScKwK+A/zAzMLxxpnnSwznAgN8RY8C3au9dTHwdTM70amdZmYpbVQH6aKUHKSziDGznGqPu/B+0ab7Omkz+fwuWL8Dfm1mH+K9Sbu/3AncZWargd5AUWNvcM59gvdG8dcBz+ONPwNvK2KLr0wB8KFv6OsDzrl38F62WmlmG4CFnJo8RJpNQ1lF/MTMYvBeMnJmdh0w1zk3p7H3iQQD9TmI+M9k4GHfCKPDwNcDHI9Ik6nlICIitajPQUREalFyEBGRWpQcRESkFiUHERGpRclBRERqUXIQEZFa/j9sUORrZZqZkgAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.recorder.plot_lr_find()\n",
"plt.vlines(6.3e-6, 0.4, 0.7)\n",
"plt.vlines(9.12e-7, 0.4, 0.7)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.421779</td>\n",
" <td>0.414670</td>\n",
" <td>0.850000</td>\n",
" <td>00:30</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.300052</td>\n",
" <td>0.301441</td>\n",
" <td>0.890000</td>\n",
" <td>00:32</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.259059</td>\n",
" <td>0.295043</td>\n",
" <td>0.890000</td>\n",
" <td>00:31</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(3, lr_max=slice(1e-6, 1e-5))"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"learn.save('roberta-fasthugs-stg2-3e-5')"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3hUVfrA8e+ZySST3hNKAgkYepUAVgRUBAtYcAXL2pB1V1dXd9eylrWu/nTti+5aV10UFcuiUhaQIkoL0kJNgAAhIZ2QXs/vj7mZTJJJGCDJJJP38zx5uPfcc2fOgfDOmXNPUVprhBBCeC6TuwsghBCibUmgF0IIDyeBXgghPJwEeiGE8HAS6IUQwsN5ubsAjUVEROi4uDh3F0MIITqVTZs25WqtI51d63CBPi4ujqSkJHcXQwghOhWl1MHmrknXjRBCeDgJ9EII4eEk0AshhIfrcH30QghxsqqqqkhPT6e8vNzdRWlzVquVmJgYLBaLy/dIoBdCdHrp6ekEBgYSFxeHUsrdxWkzWmvy8vJIT08nPj7e5ftc6rpRSk1WSu1RSqUqpR5ycv0WpVSOUmqL8TPL4drNSqkU4+dml0smhBAuKi8vJzw83KODPIBSivDw8JP+5nLCFr1SygzMAS4G0oGNSqkFWuudjbJ+prW+u9G9YcBfgURAA5uMewtOqpRCCHECnh7k65xKPV1p0Y8BUrXW+7XWlcA8YJqLr38JsFRrnW8E96XA5JMuZQezZMdRjhZ6fl+gEMIzuBLoewKHHc7TjbTGrlFKbVNKzVdKxZ7MvUqp2UqpJKVUUk5OjotFd4/aWs1vPt7EZa//CMCsDzcy9m/LKCyratdyfPhzGh+tTWvX9xRCOHfs2DHefPPNk77v0ksv5dixY21QooZcCfTOvic03q3kWyBOaz0MWAZ8eBL3orV+W2udqLVOjIx0OoO3wyivrgEgr6SSkopqlu3KJut4Ba8u28uZTy/lWGllm5ehrLKGvy7YweP/3dHm7yWEOLHmAn1NTU2L9y1cuJCQkJC2KpadK4E+HYh1OI8BMhwzaK3ztNYVxuk7wChX7+1syqtq7cf/WJFqP/7gpzTySyrZfNj26ZxfUskD87dyvLx1W/paaz5cm2Y/L6ts+RdJCNH2HnroIfbt28eIESMYPXo0EyZM4Prrr2fo0KEAXHnllYwaNYrBgwfz9ttv2++Li4sjNzeXtLQ0Bg4cyB133MHgwYOZNGkSZWVlrVY+V4ZXbgQSlFLxwBFgBnC9YwalVHetdaZxOhXYZRwvAf6mlAo1zicBD592qd2orKo+sL61ch8AY+PDWH8gH4CSimoA3v1xP58npdM3MoDfXND3tN/3eHkVc9cdIj7Cj+cX7banp2YXk1lYxpe/pLN2Xx6j48J475bRp/1+QnRWT367g50Zx1v1NQf1COKvVwxu9vrzzz9PcnIyW7ZsYeXKlVx22WUkJyfbh0C+//77hIWFUVZWxujRo7nmmmsIDw9v8BopKSl8+umnvPPOO/zqV7/iyy+/5MYbb2yV8p8w0Gutq5VSd2ML2mbgfa31DqXUU0CS1noBcI9SaipQDeQDtxj35iulnsb2YQHwlNY6v1VK7iblVQ1b0L4WM7edF28P9HUPaeta8kePN31oW1xRzdVv/kR1jaZXuB/3XpjAyF6hTfI5+r9Fu5m7/hDnnRHRIP2bLUd4b80B+/ny3dnU1mpMpq4xAkGIjmjMmDENxrm//vrrfP311wAcPnyYlJSUJoE+Pj6eESNGADBq1CjS0tJarTwuTZjSWi8EFjZKe9zh+GGaaalrrd8H3j+NMrrNmpRcUrOLuOXc+n+wxl0lqx4YT3F5tf0845gtsO/NKgZgw4Gmn2sH80rs1/fnlrByTw4pz07BYnbek5ZTVMHuo0UA7MgoBOC1GSP48xfbWLEnG4AXrhlGZU0tj36TTFZROd2DfU+pzkJ0di21vNuLv7+//XjlypUsW7aMtWvX4ufnx/jx452Og/fx8bEfm83mVu26kbVuWnDje+t54tuG0wUqjIexr1w3nPduTiQq0EpsmJ/9+qH8UrTW7DK+Ou7IOE5abkmD18gpqqCxPUeLuPn9DSzfldXk2uhnl7HpoG3qQUFpFd2CrEwb0ZO+UQHsz7G99rh+kfQOt5XjUF7pqVZZCHEKAgMDKSoqcnqtsLCQ0NBQ/Pz82L17N+vWrWvn0kmgd0lNbf1AobJK28PYniF+XDgwGgCL2cS4frbRQvtyinlxyR6KKqq558IELGbFu2v2N3i9ukD/9e/OYel94wB48MttrNqbw5PfNp6HVs/L6I4JsNq+iMUZgV0pCA/wpneYv1GGEucvIIRoE+Hh4Zx77rkMGTKEP//5zw2uTZ48merqaoYNG8Zjjz3GWWed1e7lk7VuGiksraKyppbIwPqvUQWllUQE2M7r+uh9LeYG93102xheWbqX15an8KbxkPbSod3Yn1PM4uQsnp42hHX788kuKufRb5IBGNAtCB8vE2Piw+xdPF4mRXlVDVbj9atrbB8sM8f04k+T+vHEtzsZHWfrz48OsgIQ5ueNxWwiNsyXiABv/v3zAfx9zEwb4Wy6gxCiLXzyySdO0318fFi0aJHTa3X98BERESQnJ9vT//SnP7Vq2Ty6RX84v/Skhx/e9/kW7vio4Q5X+SWV1NRq7v7kF9ak5gJgtTT9q0uIDrAfv3DNMAZ0C+L8hAhyiyvYk1XEzHfWce+8LVRU24K3r7cZk0nx4vRh9vv255Yw4LHFfPCT7QHrcaP/v390AOEBPrwxcyS/PjsOgKgg24dPkK9tFTulFOf0jWBvVjH3zttCZXX9UFAhRNflcYH+cH4pVTW1aK05/4UVXDnnJz5edxCtm8zTaqKovIofU3LYc7SowQdEXnElmYVlfLctk3//nAZgb3E7SogKtB9PHtoNgAkDovC1mHnjh9QGeR1Hz/QO9+ez2WcxtGewPa2uC6fAmIAV4ufd5P2iAm0t+oiA+mu3nBtnP07NLm65wkKILsGjAn12UTnnv7CCvy/ZY5/YtCeriMe+SebgCR5Qaq35+5I9VNVoyqpq2JlZaL/2yDfb2XK44TRlZ4E+PqL+SXuQ1dbKjgq0csXw7ny/LdN+7d4LE/jPrLEN7h3bJ5w3Zo5skLZsZxYpWbYHPCF+Tdeeruuz7x1e/75n9grlrRvOBGBnZuuOJRZCdE4e1Ue/wxjpsv5APqWV1Q2upWQXE+cQiBvbll7Ih2vr99ZNSqtfYHN/Tgl3f7K5QX5f76aB3tvL9rkZ3+h9RvYK5fOkdAA2PnJRg/5/R3ER/oT6WSgotY3Bn+XQhRTqpEV/8aBorh0Vw4NTBjRInzS4G74WMzsyCpk+Ksbpewkhug6PatFvT7e1wrsFWSlt1De/6wSt29xi20iYS40ul7q++PduTnSa3+rl/K9uy+MX893vz2uQdk7fcHy8TIzvH9lskK9TN8Ln12f3bpDu7D5/Hy9evHa4/UFxHbNJEebvzQc/pbE4+WiL7yeE8HweFejrZqGWVdU0CfTpBQ27bkY9vZRfv7/Bfl43k/VPk/oT5u/Njym2QD/Eod88wKf+C5BXM5ObQvy88fdp+EWpd7g/O568hA9cWJrg1RkjGB4TzOOXD2L7E5O4cEAUf7gogR4hJzcBalRv28icb7d26qWFhBCtwKMCfbkR3HOLK5p03WQ3mqSUV1LJ6r31SyIXGaNbgn0tDIuxBXelIMzf2/6Q9H/GmPdT4WU2ubRhwMQB0fz37vPwMpsItFp475bR/OGifif9fs9cNQRvL5N9bZ6fUnOdTtQSQrS/gADbCL2MjAymT5/uNM/48eNJSkpyeu1keVQffV1QyzpewVVv/tzgmmOQq5vd6ui4sZ58oNXC8JgQVu7JwdtswmI28ckdYzlWWkV0kJURsSHcdp7rezW6S5DVwoUDotiTVcTerCJueHc9Fw6IkgXPhOhAevTowfz589v8fTwq0NdNZqrrb3d02FiaQCnltGV7vLwaq8WEt5fJ3qKve7gaaLUQaIyi+eauc9uq+K0uITqQRclHueYt24feoXxZGkGItvDggw/Su3dvfve73wHwxBNPoJRi9erVFBQUUFVVxTPPPMO0aQ0350tLS+Pyyy8nOTmZsrIybr31Vnbu3MnAgQPbfZniTqOsqvnJUcfLq3npf3v50yX9yTpeH+jLKmvw9TZzvKzKPiRybJ9wLhwQxR8n9W/zMrelX5/dm4/WpnGstIoAHy8yC8uprdWkZBezP6eYyUO6dZl9NkUXsughOLq9dV+z21CY8nyzl2fMmMEf/vAHe6D//PPPWbx4Mffddx9BQUHk5uZy1llnMXXq1Gb/z7311lv4+fmxbds2tm3bxplnntlqxfewQN/yTNBFyZn86ZL+HMqvXwsmt7iC2DA/jpdX2WeYBvh4eUQXR0SAD0vvu4BD+aWkZBXx0FfbOVxQyiWvrgZs305GxLb97jZCeLqRI0eSnZ1NRkYGOTk5hIaG0r17d+677z5Wr16NyWTiyJEjZGVl0a1bN6evsXr1au655x4Ahg0bxrBhw5zmOxUeFejLK2sI9rU02b914oAoFHDkmO2r0Deb60eiZBfZAn1ucSVBVo/66wBswzIjA32wmG2tiJ9S8+zXFm3PlEAvPE8LLe+2NH36dObPn8/Ro0eZMWMGc+fOJScnh02bNmGxWIiLi3O6PLGjtvqG7VGjbsqqaujlsGRwHavFREyoL0cKyqit1WxMy7cvDJZTVMGGA/lsOJBvH5LoifpFB+JlUnyxqX6v9v25ssqlEK1lxowZzJs3j/nz5zN9+nQKCwuJiorCYrGwYsUKDh482OL948aNY+7cuQAkJyezbdu2ViubRwX68mYCvY+XmZhQP4oqqtmZeZzSyhrOT7AtK5xTXMHT39nWlbl8WI92LW97slrMTBnanc2HbEs5xIT6ku1k9yshxKkZPHgwRUVF9OzZk+7du3PDDTeQlJREYmIic+fOZcCAAS3e/9vf/pbi4mKGDRvGCy+8wJgxY1qtbB7VV1FWVUNUkA+PXT6IqppaAny8ePSbZHKLK4gOti0A9tryFADO7huOaRlkHy9nX04xv0qMYbiHd2NcNrS7fQLV6Lgwft6X6+YSCeFZtm+vfwgcERHB2rVrneYrLrYtOBgXF2dfntjX15d58+a1Sbk8KtDXreN+uzHOfY+x/d629ELumnAGAEt32nZwGtwjiDB/H7alF1JaWcOwGM8O8mCrc52YUF9yiiqoqdWYZX9ZITyax3TdVNXUUlWjG2wIkhAVgK/FzP0X92uyVoyftxcxob6sMmbH9u8WiKeLCbUto3DZsO5EB1mp1UirXoguwGNa9M52fjKZFLuengzY1pqv89zVQwF4cupgHpi/jSE9gxnVy3MfxNZRSpH85CX4eJk4XlbFnBWpvL48hR4hvvSJ8Jcx9aJTq5sQ6elc2VujMY9p0detP291snwwNFyQbOaYXgAMjw1hyX3jeOlXwzF1ke6LAB8vLGYT4QE+zBzTi41pBVz40io+2XAIgLdX7+N14zmGEJ2F1WolLy/vlIJgZ6K1Ji8vD6vVelL3eUyLPiLAm+QnL7FvxtGYUorZ4/rIuHEHM0bH8vLSvQB8/csRbhjbm78t3A3A1OE9Wly/X4iOJCYmhvT0dHJyck6cuZOzWq3ExJzcPhMeE+iVUg1a7c785dKB7VSaziEqyMr6v1zIcwt38c2WDN5fc8B+LTmjUAK96DQsFgvx8R1/sUF38ZiuG3FqooOsJETbHkQ/ZcwnAMg8JmPshfAUEugFU4c3nSj27MJdHDM2JhdCdG4S6AWxYX7cck4cAH+aVL/JyUdrD1JYVsUfP9/KwTxZLkGIzkoCvQDgooHRAJyfEMlZfcIA2JFRyN+X7OHLX9L5dMPhlm4XQnRgLgV6pdRkpdQepVSqUuqhFvJNV0pppVSicR6nlCpTSm0xfv7ZWgUXreu8hAh2Pz2Z4bEh/Of2sVw1sidr9+WxP9c2VXtHRqGbSyiEOFUnDPRKKTMwB5gCDAJmKqUGOckXCNwDrG90aZ/WeoTxc2crlFm0Easx2czLbGJ8/0iOl1fblzVefyCfwtKqlm4XQnRQrrToxwCpWuv9WutKYB4wzUm+p4EXABmu4QHOdJgpPKh7EJXVtSxKznRjiYQQp8qVQN8TcOygTTfS7JRSI4FYrfV3Tu6PV0ptVkqtUkqd7+wNlFKzlVJJSqmkrjDhoTPoGeKL1WL79bigfyTdgqysTpF/GyE6I1cCvbOppvZ5xkopE/AK8Ecn+TKBXlrrkcD9wCdKqaDGmbTWb2utE7XWiZGRka6VXLQpk0nZF4KLCPDhnDPC2XCgwM2lEkKcClcCfToQ63AeA2Q4nAcCQ4CVSqk04CxggVIqUWtdobXOA9BabwL2Af0QncI5fSIA2/IS8eH+5BZX2BePE0J0Hq4E+o1AglIqXinlDcwAFtRd1FoXaq0jtNZxWus4YB0wVWudpJSKNB7mopTqAyQA+1u9FqJNPHvVEN6YOZJJg7oRE2Zb4ji9oMzNpRJCnKwTBnqtdTVwN7AE2AV8rrXeoZR6Sik19QS3jwO2KaW2AvOBO7XW+adbaNE+vMwmrhjeA19vMz1DbFs0XvPWz3yedJjK6lo3l04I4SqXFjXTWi8EFjZKe7yZvOMdjr8EvjyN8okOIi7CFugLy6p4YP42jpdVMev8Pm4ulRDCFTIzVrgkKtDK4j+cz4a/XIjZpFi3X76YCdFZSKAXLhvQLYioICtXjujJz/tyOZxf6u4iCSFcIIFenLTZ4/qggOcW7XJ3UYQQLpBAL05a/26BzBzTi8XJR2W4pRCdgAR6cUpG9AqhVsO+nGJ3F0UIcQIS6MUp6WfsSpWSJYFeiI5OAr04JXHh/niZFHuzigCoqdUnuEMI4S4S6MUp8fYyER/hz96sIrTWXPLqaqbN+YnqGplIJURHI4FenLJQf2+W7crm5aV7Sc0uZuvhY6zdn+fuYgkhGpFAL07ZBf1sK42+8UOqPW35rmx3FUcI0QwJ9OKU/WZcHxKiAgAIsnoRH+Evm4gL0QFJoBenzMtsotp4CPvUtCEkRAVw5FgZWsuDWSE6Egn04rT89oK+AEwaHE2PEF/2ZhUT//BCrvvXWqrkwawQHYIEenFafjU6lrTnL8PP24sga/1iqOsP5LPpoOxIJURHIIFetJrbGy1bnJSWz5qUXDeVRghRRwK9aDXBvhaevnIIAIFWL/7+v73c+N56Vu2VTcWFcCcJ9KJV3Ti2F9uemMT/XTPMnnZEth8Uwq0k0ItWpZQiyGphSI9ge1pBaaUbSySEkEAv2kTPUF/7cXqBbFAihDtJoBdtwmxSxCjbLNkDuTKJSgh3kkAv2sb2+fzo9wBPxm5m3f58Zn24UWbNCuEmEuhF2+g7EdX7HG7OeZFHvT5mxa5M/vj5VneXSoguSQK9aBt+YXDDlxzp92tmeS3iY+tLHM7MpFbWrRei3UmgF23H7EXP69+g9JKXGauS+YRHyNy/zd2lEqLLkUAv2pzf2bdz4NJPCVHFhH96KTplmbuLJESXIoFetIuewy9kWuUzHKgKQ8+9FtbOAVnlUoh2IYFetAtfbzPpOpJrKp9gm/85sOQv8N+7obrC3UUTwuNJoBft5tHLBlKKlbur74NxD8CW/8CHV0Cx7EolRFtyKdArpSYrpfYopVKVUg+1kG+6UkorpRId0h427tujlLqkNQotOqdZ5/fhgcn9SS+soPDsB2D6B5C5Dd6eAJky9FKItnLCQK+UMgNzgCnAIGCmUmqQk3yBwD3Aeoe0QcAMYDAwGXjTeD3RRQ3sHgTAjoxCGHI13LYY0PD+ZNjxjXsLJ4SHcqVFPwZI1Vrv11pXAvOAaU7yPQ28AJQ7pE0D5mmtK7TWB4BU4/VEF5XYOxQ/bzNfbjpiS+gxAu5YAdFD4IubYcVzUCs7UwnRmlwJ9D2Bww7n6UaanVJqJBCrtf7uZO8VXUug1cKkQdGsSXVYoz4wGm75DkbcAKuetwX8SlkuQYjW4kqgV07S7OPilFIm4BXgjyd7r8NrzFZKJSmlknJyZJMKTzegexBZxysoLKuqT/TygWlz4JK/we7v4L1L4Ngh9xVSCA/iSqBPB2IdzmOADIfzQGAIsFIplQacBSwwHsie6F4AtNZva60TtdaJkZGRJ1cD0ekkRAUA8MrSvQ0vKAVn3wXXf2EL8m9PgINr3VBCITyLK4F+I5CglIpXSnlje7i6oO6i1rpQax2htY7TWscB64CpWuskI98MpZSPUioeSAA2tHotRKcyslcoAN9sOeI8Q8JFcMdysAbbhl/+8lE7lk4Iz3PCQK+1rgbuBpYAu4DPtdY7lFJPKaWmnuDeHcDnwE5gMXCX1rrm9IstOrMwf28eu3wQx0qryC1uZsJURIIt2MedBwt+D4segprq9i2oEB7Cy5VMWuuFwMJGaY83k3d8o/NngWdPsXzCQ/WLtnXf7M0qIiLAx3km31C4YT4sfQzWvQk5u+HaD2zpQgiXycxY4Rb9owMB2Hu0iPKqGgpKmtlX1uwFk5+Dqf+AtDXwzoWQs9d5XiGEUxLohVtEBvoQ4mdhT1Yxf5i3hZFPLyX7eDmbDuY7v+HMm+Dmb6G8EN69CGQFTCFcJoFeuIVSin5RgezMPM7iHUcBGPO35Vzz1lpqmtucpPfZMHslhPSCT66Fn/8hK2AK4QIJ9MJtLugfydbDx5qkZxwra/6mkFi4fQkMuBz+9wi13/yOJVsPNv/hIISQQC/c584L+jKhf9N5E4fyS1u+0dsfrv0Qxj+MaesnRHx5Dat+SW6jUgrR+UmgF25jNimuGx3bJD01u/jEN5tMMP4hHrH8mYHqEGOXXgMZW9qglEJ0fhLohVuNjQ/HbFK8et0Ilt1/AQlRAby7Zj/ahb736ppavigbxfTKv1Jdi20FzOSv2r7QQnQyEuiFW4X6e7Pvb5dy5cienBEVwHWjYzmcX8ax0qoT3nsgt4TK6lp26jiuV89R220ozL8VfnhWVsAUwoEEetGhxIb5AZBeUP9AtqqmadAuKq/i959uBmB8/0h2HLcyIPUuigbOgNUvwOc3QYULXUBCdAES6EWHEhPqC8DjC5J5/L/JXDnnJxIeWcScFamArbtm/f48/vFDKruPFgHw2oyRAFRi4a7i20gZ+RfYsxDemwQFB91TESE6EAn0okOpa9FvPnSMj9YeZIsx/HLlnmyKK6p5c+U+rnt7HQu21i+CGuxr4ZFLBwKwOiWXi9cOoWbmF1CYDu9MgLSf2r8iQnQgEuhFhxJktXB2n3D7+RszR3LLOXFsTCtgyF+XsCjZNrkqs7C8wX13jOvDJYOj7ecv7OsJd/xgWxfno6mw6d/tUn4hOiIJ9KLD+fPk/vbjXmF+jIkPs5/vyjxuP06ICuDF6cPs52Pj6z8gPll3CB3eF2Yth/gL4Nt7qVzwR1kBU3RJEuhFh1PXTw+2rpyJA6Ka5LloYDRf/e4crk2sH4c/c0wvbjs3nt+O70tRRTX5JZXgGwLXf86mHtfj/cu7VH54JZTWr6dTXVNL8pHCJq9fUlHNk9/uoKj8xKN/hOjoJNCLDifC34fuwVYuGRxNmL83VouZpfeNsy9t/MilA3n35kQCrZYG9/l6m3n8ikGMibN9A5j40irbBbMXz1TfyJ+rZmM+vA7evRBy9gDw5Lc7ufyNNSzbmUVmYf1In41p+XzwUxqr9srWlqLzc2k9eiHak8mk+PGBCXiZ69shCdGBhPp5A9Ar3K/F+wd2DwKgsKyK6ppavvwlnbLKGr6oGU/P2GHcm/ck+u2JVF31Hh+vs+2DM+ujJAB+emgiPUN87fvZ7j1aBMOcv48QnYW06EWH5Bjk6zwwuT8xob6Mdeizd6ZbsJU/X2Lr5992pJAHv9xuH4r56t4wHg5/jV0V4Vg+m8Fs87c47ld/xRtrAOwTtvZkFbVGdYRwKwn0otMY1TuMNQ9OJMRo2bek7sNg86Gmq2PO2wvTK//Kwtox/MXyKS9Z3sIH28Yn+cYGKPYWfZZMuhKdnwR64ZESogLxNpv4bOMhe9rlw7rbj8uwcnfVPeSN/hPXmNewtvvLRFJAoI+tN7OuRZ+WV0J5lWxzLDo3CfTCIwX7Wbh+bC97i/zhKQP429VDuXZUDAA3jO3FmgcnEn7ZY/CrjwkrTmVF0JPEVe6lvKqGZbuyANu+Ji6tpilEByaBXnis303oaz+eNLgbQVYLV47syblnhHPnBX2JCTUe6g6aCrf/D2X24gvvJ1n+xZsN1sS//I01EuxFpyaBXnisqECr/TjE1zYU89wzIpg76yz7Ugt23YZScvP/2KHO4LK9j/Inr8+IDfGxX/7Xqn3U1GqqnSywJkRHJ4FedAlBvpYT5onqFsuCYW/yafUE7vb6Lyt7vYc/trH1248UcvuHG5n0yuq2LqoQrU4CvfBoUYG2VrnZpFzKf27/HjxcPYs51tmYU5bwc+TzDPbNZ/fRIlbuyWF/bglHWtrTVogOSAK98GiL7j2f7+85z+X8Fw+K5tXrRnLuDY/AjV8SXJXN99bHef2sEvvM3PX789qquEK0CQn0wqOFB/gwuEewy/mVUlw5sicjYkOg7wS4YwX4RzB1229ZeE4qSsHBvBNsXi5EByOBXoiWhPeFWcugzwS8Ft3P3/0+5kje8RPfJ0QHIoFeiBOxBsP1n8E5v+eamkX8ev/9DVbAFKKjk0AvhCtMZpj0DF/E/IUBlTvQ70yE7N3uLpUQLnEp0CulJiul9iilUpVSDzm5fqdSartSaotSao1SapCRHqeUKjPStyil/tnaFRCiPVUMmcF1FY9RW1EM714Eexa7u0hCnNAJA71SygzMAaYAg4CZdYHcwSda66Fa6xHAC8DLDtf2aa1HGD93tlbBhXCHYTHBbNYJrLrgCwjvA5/OgDWv2NZKEKKDcqVFPwZI1Vrv11pXAvOAaY4ZtNaOT6f8cVz3VQgP0i86EIDk4gC4dTEMvgqWPQFf/waqylu+WQg3cSXQ9wQOO5ynG2kNKKXuUkrtw9aiv8fhUrxSarNSapVS6nxnb6CUmq2USlJKJeXkyI4+ouOyWsx0DyMo80IAAB6nSURBVLaSllsC3n4w/X2Y+Chs+wz+fSkcz3R3EYVowpVA72xKYZMWu9Z6jta6L/Ag8KiRnAn00lqPBO4HPlFKBTm5922tdaLWOjEyMtL10gvhBvER/hzIK7GdKAXj/gzXzbU9nH1nAhzZ5N4CCtGIK4E+HYh1OI8BMlrIPw+4EkBrXaG1zjOONwH7gH6nVlQhOoa+kQGkZBU3XOBs4OVw+//AZIEPLoVtX7ivgEI04kqg3wgkKKXilVLewAxggWMGpVSCw+llQIqRHmk8zEUp1QdIAPa3RsGFcJcx8WEUV1STnNFo4lS3ITB7BfQcBV/NgmVPQq2sdinc74SBXmtdDdwNLAF2AZ9rrXcopZ5SSk01st2tlNqhlNqCrYvmZiN9HLBNKbUVmA/cqbWWmSaiUzu7bzgAP+/LbXrRPwJu+gZG3QJrXoZ510OF7Dsr3EvpDjYsLDExUSclJbm7GEK0aPKrq4kI8OE/s8Y6z6A1bHgHFj8EEf1g5qcQFt++hRRdilJqk9Y60dk1mRkrxCk4p28EG9PyqahuZj9ZpWDsbLjpKyjKhHcmwoEf27eQQhgk0AtxCs7pG05FdS2/HDzWcsY+4+GOH8A/Ej6+Eja+2x7FE6IBCfRCnIKxfcLw9jKxYGtLA9AM4X1h1lLoOxG+/yN8dz/UVLV9IYUwSKAX4hQEWi1cPbInX/2STklFtT09+Ugh/9txtOkN1mCYOQ/OvReS3oOPr2LvgYN0tGdkwjNJoBfiFE0b0ZOK6lpW7bXN5tZac/kba5j98SaOFjZdDuG2j37h3ryr4Kq3qTm0HusHF/LT2jXtXWzRBUmgF+IUjY4LxdvLxNbDtn7642X1LfvdR5tuTvLD7mz+uyWDCUujuabsUXxUFWOW/wr2LGq3MouuSQK9EKfIy2yiT4Q/qdnFAGQV1bfi7/goyXkXDnAgt4Qt+gymVjxDlqUXfDoTfnxZVsAUbUYCvRCnoW9kAPtybIHesbumqkYz++P6NW8c+/HrZBHG/QHPwZBrYPmT8NUdUFXW9oUWXY4EeiFOw5CewaTllZKSVUTW8eaXKV62K6vB+cwxscwYHcvG9DIuOfRrskY/ANvnwwdT4LgLI3mEOAkS6IU4DTNGx2I2Kb7dmmEP9NFBPvbrldW1zF1/kHvnbbGnXTa0O89dPYxLh3YHYE92MU8VToEZn0BuCrw9AdJlBUzReiTQC3EaQv29iQn1ZfGOo+zPKSHEz8KL04fbr6fllfDI18n28/V/uZA5N5wJwHlnRPDi9GGM7x/J99syWVQ1Em5fCl4+tpb91s/avT7CM0mgF+I0BVq92JtVzFebj9AtyMq4fpF8e/d5AKw/UL+G379uGkV0kNV+bjIprk2M5Z83jiIiwJsvNqVD9CC4YwXEjoGvZ8PSx6G2mWUWhHCRBHohTpNJ1e/NE2UE8j6R/gA89o2tNT8sJpjx/Z1vqmO1mJkypDs/7M7myjk/gX843PQ1JN4GP71mG5VT3nS4phCukkAvxGl6+Vf1XTXdjP55fx8vegTXt97nzhqLj5e52dfo3822F+2Ww8coKKmkGjNc/gpc9hKkLoN3L4K8fW1UA+HpJNALcZrOiArk9ZkjARjaM9ieHuRrAeDpK4cQaLW0+Bp9IvztxxvT8kl8dhlxD33PQutl8OtvoCTbtgLm/lVtUAPh6STQC9EKpg7vwb6/XcpNZ8fZ03wsthZ8TKjvCe8f0SuEwT1s2yl/uy2TY6W2Rc9W7M6G+HG2FTADu8HHV9nWuTcmV1XX1PLC4t1Mm/MT2UXND+8UXZsEeiFaidmkGpxfNrQbAPHh/s6yN+Dn7cX395zPBf0i+dZhRcykgwXszynmmZ/L2Tfta0iYBAv/BN/dB9WVLN2ZxZsr97H18DE2HJDN24RzXu4ugBCe6o7z+3D5sB70CDlxi77O+P6R9kXSEqICSMkuZuJLtu6agtIYXpoxF354Gta8ArkpZPZ8wn7v3qziVi2/8BzSoheijSilTirIQ/1+tAB/nNSvwbUdGYVgMsNFT8DV70D6Rq7ceBNj/DKJC/cjJUv2phXOSaAXogNJiAq0Hw/uEcxzVw+1n6dkF1NeZYypH/YrDl35JVWVZXxQ+whX+W3lYF5pexdXdBIS6IXoQBz7+YN8LfSLDrCf19Rq9hwtYulO27o5/zvWkysqnqXAL557c/7KlIL/NFgBs7qmljkrUsktrmi/CogOSQK9EB1UoI8XZzi08AF+8/Em7vgoieQjhaTllVBujaTnH34gJXoKv2celZ/fCpWlbEzL58eUXF5csoe/LtjhphqIjkIexgrRwfxmXB/+s+4gJpMi2NfCo5cNpH+3QG56bwNHjYXTUrKLOJBbQnxkAMrbj9RzX+bLz4J5cNdnlGSn8vsjszmKrb9/V4bMqu3qpEUvRAfz8KUD2fHUZPv5rPP7cN4ZEQT61LfL7vtsK1sPF9IrzA+APlGB/LNmKuvH/gOfwv0s8HmMkSoFgEP5pVRUy3o5XZkEeiE6AaUU15/Vq0FacUU14f7egG1tHW8vE8trz2T1uE8o097M836Gq0w/Ul2r7btgia5JAr0QncQ1Z8Y0SQszAr3FbGJAt0B2Zh7nsFcc0yqfZlNtAq94v8VDXp9wxeur+TElp72LLDoICfRCdBIJUbYRODNGxxJgdOPUBXqAwT2C2JFxnILSSo4RyK+rHmJDxNXc6fUd71r+zsKNe9xSbuF+EuiF6CSUUqQ8O4Xnrh5KiJ9tkTTHQD+oRzDHSqvYaTx8rcaLb3reT9745znftJ27D/xWVsDsolwK9EqpyUqpPUqpVKXUQ06u36mU2q6U2qKUWqOUGuRw7WHjvj1KqUtas/BCdDUWswmllD3QB/vWr4o5qLttUbSfUnPtaSUV1YSP/y3/7P0S/lUFxgqYK9u1zML9Tji8UillBuYAFwPpwEal1AKt9U6HbJ9orf9p5J8KvAxMNgL+DGAw0ANYppTqp7WWIQBCnIYRsSEkHzmOn3f9GvcDuweiFJRU1tA/OhCTSXHXhDMAMMWfz+UpT7Mq8l+YP77atpOVdwB4+xs/jsfGucXPybVG+cwtL78sOgZXxtGPAVK11vsBlFLzgGmAPdBrrR0H6voDddPzpgHztNYVwAGlVKrxemtboexCdFmPXjaIc/tGMLJXqD3Nz9uLHsG+HDlWxrSRPfjd+DPs1/pHB5Kuo3izz5uM1G9yXlAxVBZDaT4cOwyVJbbzymKorXa9IGafRh8Qzj4QHI4tfi18cBjHFl9Q6sTvLVzmSqDvCRx2OE8HxjbOpJS6C7gf8AYmOty7rtG9PZ3cOxuYDdCrV6/Gl4UQjVgtZqYM7d4k/c4L+rBiTw5Xj2w4QqdftG2G7UurMoGr+PH6CcQaY/CbqK6EqhIj+Nd9ADR3XgyVpU3TSw83PK8qOYnaKSPo+534g0O+fbjElUDv7KNVN0nQeg4wRyl1PfAocPNJ3Ps28DZAYmJik+tCCNfcdHZcg81P6sSG+eLnbaa00tZrunZfXvOB3svb9uMb6vz6qaitheqyZj4onH2IlDbN1+TbRwnUVrleBrOP8eHh5IPCw799uBLo04FYh/MYIKOZvADzgLdO8V4hRBtQStE3MoDtRwoBWLs/j1+Njj3BXa3IZKoPkkS13uue1LePZtJP+9uHi91W3v5gOUE+n0CwBrXe34/BlUC/EUhQSsUDR7A9XL2+QVWVStBapxinlwF1xwuAT5RSL2N7GJsAbGiNggshTs7rM0eyK/M432/P5Od9uWitUZ2gNdoiT/v20eNMmL2i9epiOGGg11pXK6XuBpYAZuB9rfUOpdRTQJLWegFwt1LqIqAKKMDWbYOR73NsD26rgbtkxI0Q7hEf4U98hD9F5VV8vy2TXZlFDOrR+q3HTs+d3z6swSd+nVOgtO5YXeKJiYk6KSnJ3cUQwmPlFFUw9m/L8PYy8a+bErmgX6S7iyRagVJqk9Y60dk1mRkrRBcTGejDazNGUl5Vy83vb6Cw7CQeaIpOSQK9EF3QFcN78LvxfQFIyz2Zh49QVVPLw19tY96GQ21RNNEGJNAL0UVdfaZtSsv+3JNbwvhgXgmfbjjMQ19tb4tiiTYggV6ILio2zA+Tgr1ZJxfoK6pr7ceFpdLt0xlIoBeii/LxMjM2PpzvtmVQU+v6oIxKh0B/MP/kun2Ee0igF6ILu/Gs3hzOL+O7ba7PY3Rs0afllbZFsUQrk0AvRBc2ZUg3egRbWbozy+V7HFv0h/KkRd8ZSKAXogszmRRDegazM/P4iTMbKqVF3+lIoBeiixvYPYgDuSWUVrq2PHFljS3Qh/hZOCgt+k5BAr0QXdyI2BC0hi2HjrmUv65FnxAVIC36TkICvRBd3Jm9Q1EKNqYVuJS/LtD3jQwgp6iC8ipZvqqjk0AvRBcX7Guhf3QgG9PyXcpfUVMf6AHSC8rarGyidUigF0IwJj6MXw4VUFFdQ2Zhy4Hb3qKP8gfgcIF033R0EuiFEFw8KJrSyhr6P7qYs5/7gb1ZRc3mrQv0/aJtm5GvSclt8bW11uSXVLZqecXJkUAvhOD8hEhuGFu/X3OysROVMxXVtj757sG+XDGsB59vPExLy50v2XGUM59eelKTskTrkkAvhADg9xMT7Md7jrbcojebFGaTYlTvUIoqqskpqmg2/74c2xDMD39Oa7WyipMjgV4IAUB0kI/9+LOkw/YFywpKKht0vVRW1+JttoWOugey/92SwftrDjh93TJjQ/Ij8tDWbSTQCyEA2wbiS/4wjjdmjuRYaRVr9+fy+cbDjHx6KRNfWsm3WzOordVU1tTi7WUEeuOB7LMLd/HUdzspqWg66epYme1DIqOwnOPlnrva5cfrDpL4zNIWn2+4iwR6IYRd/26BXDK4Gz5eJjYcKODNlakAHCut4vefbmbxjqO2Fr0R6LsFWQnz97bfP/ivS6httBJmYVl98Hd1UlZn9MvBAnKLK/nvliPuLkoTEuiFEA14e5kY1COIHRmFHD1ezqzz4rl8WHcAFifbAr2PEeiVUgRZvRrc33jdnGOllfSLDsDLpPh5X177VMINlPFnykmu798eJNALIZqIj/Dnl0MFlFfVEhPqyz+uPxM/bzMLtmbwzZYjWC1me97/u2YY916YwLWjYgBYt79hMC8sq6J7sC/nJUTwyfqDLT64BSivquGGd9fxyyHXZup2FHXLN6fmSKAXQnQC8eH+VNXYumBiQv0AeOW6EQDUaugT4W/PO7ZPOPdd3I8Xrx1ORIBPkxZtYVkVIX4WHrt8EOVVtby2fG+L730gt4SfUvO4e+4vrVmlNlc37DQtt8Tpswp3kkAvhGiijzGaxs/bzLDYYAAuGdyNK4b3AOqDf5P7Ivw54LDZuNaao4XlRAb40DcygPH9I/kp1dbiLyyr4p5PNzeZTFVsBMmMwvLWrVQzXlm6l+cX7W5xLoAr6lr0tRq2tzAPwR0k0Ashmrh4UDRv3zSKbX+dRFSg1Z5+ozGpKjEu1Ol98RH+bEjLJ6eogsLSKt5cuY+K6lp6h9s+GEb2CuVAbgn5JZUs2HKEBVszeHVZwxZ+XnH7zaItr6rhteUp/HPVPjYdLLAPBT3V1xrQLRCArYdP7qHzsdJKFm7PPGG31qmSQC+EaMLby8Skwd3wMjcMEWP7hLPhkQuZMqSb0/vOOSMcgLdX7+M/6w/y4pI9gG0jcoCz+9qu/2/HUYJ8LQBkHGvYci8orQ/0bb35+DGH15/+z7W8tjzFfr4jo5B0Yx0fVz4AKqpr6RZsJTrIhz0nOcRyb1Yxv5v7S4sT1U6HBHohxEmJCrSilHJ6bdqIngzoFsg7Px6wB3moD/TDY4LpG+nPd9sy7V0dy3ZlcesHG+xdJ45dOWltvLFJ426jXw4W8M3mI2xPL+Sy19cw5dUfWbE7m4GPL2bOilRW7c1he3ohSU5W+qyoso1G6hcd6HQsvda62b77unKE+ltaoVZNeZ04ixBCnLoHJve3P7xVSnFWn3D+uyWD8f0j7XlW7MlhzopU7p6YQEGjQD88NqTNylb37SGxdyj5JZVsSMtng0MQL6qoZtXeHIAGH1wAac9f1uC8oroGHy8zvcP9+fdPaWQXlTfo9vrP+kM89k0yax+eSPdgX6flcJyT0JqkRS+EaFXnnhFhP+4T6c/vxp/R4BvAyF6hFFdUs9mYPJXy7BQmDojin6v2U1ldS35pJZGBPnibTSe1l+2pqGtJ/+3qocw6v4/TPD4W52Eyt7iCFxbvZugTS0jNLqaiuharxcT1Y3pRXVvL8wt38591B+35lyQfBWDHkeNNXufhr7YDEOrXNoHepRa9Umoy8BpgBt7VWj/f6Pr9wCygGsgBbtNaHzSu1QDbjayHtNZTW6nsQogO6MHJA7j13DhSsoo5IyqgyfVzjH7677dn4msxYzGbmDmmFz/szmZjWj75JZV0C7LSI8S3zWfS1rWkQ/28uWZUT5bvymL57uwGeTYfdF6GbenH+GF3NkXl1SzYmkFFdS0+XmbiIvwZ3COYrzYf4avNR5gwIIrPNh5ml/GhdSi/lP05xfj7eBEdZOXrX+pn0jrOT2hNJ2zRK6XMwBxgCjAImKmUGtQo22YgUWs9DJgPvOBwrUxrPcL4kSAvhIfz9jIRE+rHhAFR9r55Rz1CfBna0zZks66hf3bfcEwK1h/Ip6CkklB/b0bEBLP9SCE1tac37LEldS36ED8LPl5m3rtlNM9dPdR+PdDHq0FXzv9dM5Rx/WxdTnfN3cxu4+Hp5kMFVFTV2GcMj40Ps9/zw+5sXl+eQp7xXnuOFjHxpVWc9dxy+3u3NVe6bsYAqVrr/VrrSmAeMM0xg9Z6hda6bpuZdUBM6xZTCOFJHpjcH4BSYzRLgI8XA7oF8cvBAvJLKwnzszCiVwillTWkZDd8sFleVUPykULyik9/v9pjpVUEWb2wOIwuqvsQArhihG3eQL/oAGaP68MVw3vw0W1jiA7yoczhvTcdLKCkssbezXPxoGj7tb836ttfa8wcrhu23x577roS6HsChx3O04205twOLHI4tyqlkpRS65RSVzq7QSk128iTlJOT40KRhBCd2Vl9wpukjeodyuZDBeQWVRLm78OIWNtY/XkbDjfI996aA1z+xhpGPbOMJ7/dcVrlyC+pbPIAtL8xFh5gxuhYeof78eTUIfzl0oH4edt6u/tF1+cZ1y/S/oHl42XrekmMC2PKkG4M6BZIYVnDIaKH8uu3XnxjeQpzVuwD4MXpw06rLi1xJdA7G0fl9LuUUupGIBF40SG5l9Y6EbgeeFUp1bfJi2n9ttY6UWudGBkZ2fiyEMLDWMwmeob4MiymvvWcGBdKSWUNZVU1hPlbiAv344J+kXy87qB9+0KgwRDFTzccZuJLK5sE0xPJOFbG6r05FJTauokal+3/rhnK5785m2ExIaz68wT7+P86dc8eRseF8vZNo+gbaRtVVGVsnG42Kd66cRTv3pwI2GYYQ31XVZ2Xlu7l6PFyfLxMXJsYe1J1OBmuPIxNBxxLEAM02RNMKXUR8AhwgdbaPr1La51h/LlfKbUSGAnsO40yCyE8wOoHJmByCHxnO7Tyg/28UUpx5cgerNqbw8G8EhKMVnTjVub+nBI2Hcxn4oBoXHXbvzey+2gRvcP9OCOy6QPj60b3cnJXve7BtmGTg7oHYbWYeXjKQGZ9lETW8YaTv2JC/fj3raNJiA6koKQSi9nEJa+ubvJ6FQ4fZG3BlRb9RiBBKRWvlPIGZgALHDMopUYC/wKmaq2zHdJDlVI+xnEEcC6ws7UKL4TovMwm1WDYZVSQlaRHL+L3E89gktHH3SfCFoQvfmU1n6w/BDifpXogt7RJWh3H9fEXbc8k+UihfSLWwbzSJi16V/QOt7Xg6+6dOCCKh6YM4L6L+zXJO75/FD1DfBnSM7hBt9CE/u3Xe3HCFr3WulopdTewBNvwyve11juUUk8BSVrrBdi6agKAL4x/uLphlAOBfymlarF9qDyvtZZAL4RwKiLAhz9O6m8/7xNZv0rmsl1ZXD+2l73rxt/bzH0X9+OZ73fZhy6CbatDjcakFLd8sAGL2cS/bx1D8pFCfutkRcxTmaQ0aVA0r80YwWRjKQiTSXHnBU16pZ36962jiQz04b9bMlixp32eSbo0jl5rvRBY2CjtcYfji5q572dgqLNrQghxIoFWC9/fcx7v/niA5buyqK3VlFbV0CfSnx/+OB6wDcn8MSWHxcmZXDK4G5NfXc3+3IZLJxw5Vmaf4dpYTKiv0/SWKKWYNqKlMSnNG98/CrB9W+kebOX7bZl0Dzn5MpwMmRkrhOjQBvcIZlTvUI6XV3P0eDlllTX2h5sA4xIiyDpewZ3/+YX5m9KbBHmAN1eksuXwMfpE+nN+QkSDaxMHRLV5HZzx9TZz67nxzP/tObwxc2SbvpesdSOE6PD6Gg9M9+UUU1JRjZ+lPnSdl1Df1/3n+dsa3HfZsO74WczMNfr3LxkczaVDu1NaWcOvz+7N8bKqZtfW9yQS6IUQHV7fKFtf/Yc/H2T9gXwu6Fcf3OPCmw/U/5g5ksqaWr7YlG57ncgApo3oecrdLp2VdN0IITq8yAAfzuwVwrJdWUDDNeuVUqx5cAJJj17UoEun7lrdJCaAAd2D2qfAHYwEeiFEh6eUYu6ss5gx2jalJ61RP3xMqB8RAT4kGBOZLhoYzSvXDbdff3DyAM5PiODSZjZM8XQS6IUQnYKvt5nHr7Ctp1i3cXljY41JV1NH9OCqkfVLbv12fF8+vn1skx2zugrpoxdCdBp+3l48fvkgRvV2vmft/Rf3IyrQp9mtDrsqCfRCiE7ltvPim71mtZib3UCkK+ua32OEEKILkUAvhBAeTgK9EEJ4OAn0Qgjh4STQCyGEh5NAL4QQHk4CvRBCeDgJ9EII4eGU1s6nEruLUioHOHiSt0UAuW1QHHeQunRcnlQfqUvHdDp16a21dro/YYcL9KdCKZWktU50dzlag9Sl4/Kk+khdOqa2qot03QghhIeTQC+EEB7OUwL92+4uQCuSunRcnlQfqUvH1CZ18Yg+eiGEEM3zlBa9EEKIZkigF0IID9fpA71SarJSao9SKlUp9ZC7y3MiSqn3lVLZSqlkh7QwpdRSpVSK8Weoka6UUq8bddumlDrTfSVvSikVq5RaoZTapZTaoZS610jvdPVRSlmVUhuUUluNujxppMcrpdYbdflMKeVtpPsY56nG9Th3lt8ZpZRZKbVZKfWdcd4p66KUSlNKbVdKbVFKJRlpne53DEApFaKUmq+U2m38vzm7PerSqQO9UsoMzAGmAIOAmUqpQe4t1Qn9G5jcKO0hYLnWOgFYbpyDrV4Jxs9s4K12KqOrqoE/aq0HAmcBdxl//52xPhXARK31cGAEMFkpdRbwf8ArRl0KgNuN/LcDBVrrM4BXjHwdzb3ALofzzlyXCVrrEQ5jzDvj7xjAa8BirfUAYDi2f5+2r4vWutP+AGcDSxzOHwYedne5XCh3HJDscL4H6G4cdwf2GMf/AmY6y9cRf4D/Ahd39voAfsAvwFhssxS9Gv++AUuAs41jLyOfcnfZHeoQYwSNicB3gOrEdUkDIhqldbrfMSAIOND477Y96tKpW/RAT+Cww3m6kdbZRGutMwGMP6OM9E5TP+Pr/khgPZ20PkZXxxYgG1gK7AOOaa2rjSyO5bXXxbheCIS3b4lb9CrwAFBrnIfTeeuigf8ppTYppWYbaZ3xd6wPkAN8YHSpvauU8qcd6tLZA71ykuZJ40U7Rf2UUgHAl8AftNbHW8rqJK3D1EdrXaO1HoGtNTwGGOgsm/Fnh62LUupyIFtrvckx2UnWDl8Xw7la6zOxdWXcpZQa10LejlwXL+BM4C2t9UighPpuGmdarS6dPdCnA7EO5zFAhpvKcjqylFLdAYw/s430Dl8/pZQFW5Cfq7X+ykjutPUB0FofA1Zie+4QopTyMi45ltdeF+N6MJDfviVt1rnAVKVUGjAPW/fNq3TOuqC1zjD+zAa+xvYh3Bl/x9KBdK31euN8PrbA3+Z16eyBfiOQYIwm8AZmAAvcXKZTsQC42Ti+GVtfd136r42n72cBhXVf8ToCpZQC3gN2aa1fdrjU6eqjlIpUSoUYx77ARdgelK0AphvZGtelro7TgR+00ZHqblrrh7XWMVrrOGz/J37QWt9AJ6yLUspfKRVYdwxMApLphL9jWuujwGGlVH8j6UJgJ+1RF3c/oGiFBxyXAnux9ac+4u7yuFDeT4FMoArbJ/bt2PpDlwMpxp9hRl6FbVTRPmA7kOju8jeqy3nYvkpuA7YYP5d2xvoAw4DNRl2SgceN9D7ABiAV+ALwMdKtxnmqcb2Pu+vQTL3GA9911roYZd5q/Oyo+z/eGX/HjPKNAJKM37NvgND2qIssgSCEEB6us3fdCCGEOAEJ9EII4eEk0AshhIeTQC+EEB5OAr0QQng4CfRCCOHhJNALIYSH+394qk/6xRaekQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"learn.recorder.plot_loss()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Lets Look at the model's predictions"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"('positive', tensor(1), tensor([0.1808, 0.8192]))"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.predict(\"This was a really good movie, i loved it\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if dl is None: dl = learn.dls[ds_idx]\n",
"cls(dl, *learn.get_preds(dl=dl, with_input=True, with_loss=True, with_decoded=True, act=None))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"self.preds, self.input"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Automatic pdb calling has been turned ON\n"
]
}
],
"source": [
"%pdb"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"ename": "IndexError",
"evalue": "index 197 is out of bounds for dimension 0 with size 197",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-47-3c9a1c919336>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mfastai2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minterpret\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m#interp = Interpretation.from_learner(learn)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0minterp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mClassificationInterpretation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_learner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m~/fastai2/fastai2/interpret.py\u001b[0m in \u001b[0;36mfrom_learner\u001b[0;34m(cls, learn, ds_idx, dl, act)\u001b[0m\n\u001b[1;32m 24\u001b[0m \u001b[0;34m\"Construct interpretatio object from a learner\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 25\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_preds\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 28\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtop_losses\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlargest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/learner.py\u001b[0m in \u001b[0;36mget_preds\u001b[0;34m(self, ds_idx, dl, with_input, with_decoded, with_loss, act, inner, reorder, **kwargs)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'decodes'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnoop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mreorder\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'get_idxs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/torch_core.py\u001b[0m in \u001b[0;36mnested_reorder\u001b[0;34m(t, idxs)\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 614\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 615\u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 616\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 617\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/torch_core.py\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 614\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 615\u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 616\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 617\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/torch_core.py\u001b[0m in \u001b[0;36mnested_reorder\u001b[0;34m(t, idxs)\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 614\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 615\u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 616\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 617\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastcore/fastcore/foundation.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(cls, x, *args, **kwargs)\u001b[0m\n\u001b[1;32m 39\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 41\u001b[0;31m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 42\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_newchk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 43\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastcore/fastcore/foundation.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, items, use_list, match, *rest)\u001b[0m\n\u001b[1;32m 312\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mitems\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 313\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0muse_list\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0m_is_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 314\u001b[0;31m \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0muse_list\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0m_listify\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 315\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmatch\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 316\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mis_coll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mmatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastcore/fastcore/foundation.py\u001b[0m in \u001b[0;36m_listify\u001b[0;34m(o)\u001b[0m\n\u001b[1;32m 248\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 249\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_is_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 250\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mis_iter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 251\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 252\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/torch_core.py\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 614\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 615\u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 616\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 617\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/fastai2/fastai2/torch_core.py\u001b[0m in \u001b[0;36mnested_reorder\u001b[0;34m(t, idxs)\u001b[0m\n\u001b[1;32m 612\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 614\u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 615\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 616\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mIndexError\u001b[0m: index 197 is out of bounds for dimension 0 with size 197"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"> \u001b[0;32m/home/morgan/fastai2/fastai2/torch_core.py\u001b[0m(614)\u001b[0;36mnested_reorder\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 612 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 613 \u001b[0;31m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 614 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 615 \u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 616 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> self.preds\n",
"*** NameError: name 'self' is not defined\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/torch_core.py\u001b[0m(615)\u001b[0;36m<genexpr>\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 613 \u001b[0;31m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 614 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 615 \u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 616 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 617 \u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastcore/fastcore/foundation.py\u001b[0m(250)\u001b[0;36m_listify\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 248 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 249 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_is_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 250 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mis_iter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 251 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 252 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastcore/fastcore/foundation.py\u001b[0m(314)\u001b[0;36m__init__\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 312 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mitems\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 313 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0muse_list\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0m_is_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 314 \u001b[0;31m \u001b[0mitems\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0muse_list\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0m_listify\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 315 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mmatch\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 316 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mis_coll\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mmatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastcore/fastcore/foundation.py\u001b[0m(41)\u001b[0;36m__call__\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 39 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 40 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m---> 41 \u001b[0;31m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 42 \u001b[0;31m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_newchk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 43 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/torch_core.py\u001b[0m(615)\u001b[0;36mnested_reorder\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 613 \u001b[0;31m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 614 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 615 \u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 616 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 617 \u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/torch_core.py\u001b[0m(615)\u001b[0;36m<genexpr>\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 613 \u001b[0;31m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 614 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 615 \u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 616 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 617 \u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/torch_core.py\u001b[0m(615)\u001b[0;36mnested_reorder\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 613 \u001b[0;31m \u001b[0;34m\"Reorder all tensors in `t` using `idxs`\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 614 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 615 \u001b[0;31m \u001b[0;32melif\u001b[0m \u001b[0mis_listy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 616 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 617 \u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Expected tensor, tuple, list or L but got {type(t)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/learner.py\u001b[0m(232)\u001b[0;36mget_preds\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 230 \u001b[0;31m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 231 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'decodes'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnoop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 232 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mreorder\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'get_idxs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 233 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 234 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> u\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/interpret.py\u001b[0m(26)\u001b[0;36mfrom_learner\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 24 \u001b[0;31m \u001b[0;34m\"Construct interpretatio object from a learner\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 25 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m---> 26 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_preds\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 27 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 28 \u001b[0;31m \u001b[0;32mdef\u001b[0m \u001b[0mtop_losses\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlargest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> self.preds\n",
"*** NameError: name 'self' is not defined\n",
"ipdb> d\n",
"> \u001b[0;32m/home/morgan/fastai2/fastai2/learner.py\u001b[0m(232)\u001b[0;36mget_preds\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m 230 \u001b[0;31m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 231 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'decodes'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnoop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m--> 232 \u001b[0;31m \u001b[0;32mif\u001b[0m \u001b[0mreorder\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'get_idxs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 233 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[0;32m 234 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\n",
"ipdb> self.preds\n",
"*** AttributeError: 'Learner' object has no attribute 'preds'\n",
"ipdb> l 200.234\n",
"\u001b[1;32m 195 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_train\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 196 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 197 \u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mCancelEpochException\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_cancel_epoch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 198 \u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_epoch'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 199 \u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 200 \u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mCancelFitException\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_cancel_fit'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 201 \u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_fit'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 202 \u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 203 \u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mds_idx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcbs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 204 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 205 \u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madded_cbs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcbs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_logging\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\n",
"ipdb> l 200,234\n",
"\u001b[1;32m 200 \u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mCancelFitException\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_cancel_fit'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 201 \u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'after_fit'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 202 \u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 203 \u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mds_idx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcbs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 204 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 205 \u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madded_cbs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcbs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_logging\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 206 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_before_epoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 207 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 208 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_after_epoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 209 \u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'final_record'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 210 \u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 211 \u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mdelegates\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGatherPredsCallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 212 \u001b[0m def get_preds(self, ds_idx=1, dl=None, with_input=False, with_decoded=False, with_loss=False, act=None,\n",
"\u001b[1;32m 213 \u001b[0m inner=False, reorder=True, **kwargs):\n",
"\u001b[1;32m 214 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mds_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshuffled\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdrop_last\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 215 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mreorder\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'get_idxs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 216 \u001b[0m \u001b[0midxs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_idxs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 217 \u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mget_idxs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 218 \u001b[0m \u001b[0mcb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGatherPredsCallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwith_input\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwith_input\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwith_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwith_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 219 \u001b[0m \u001b[0mctx_mgrs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_logging\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madded_cbs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 220 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwith_loss\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mctx_mgrs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_not_reduced\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 221 \u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mExitStack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mstack\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 222 \u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mmgr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mctx_mgrs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menter_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmgr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 223 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin_epoch\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0minner\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0m_before_epoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 224 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_epoch_validate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 225 \u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mafter_epoch\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0minner\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0m_after_epoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 226 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mact\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mact\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'activation'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnoop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 227 \u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mall_tensors\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 228 \u001b[0m \u001b[0mpred_i\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwith_input\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 229 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 230 \u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 231 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwith_decoded\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss_func\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'decodes'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnoop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpred_i\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[1;32m 232 \u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mreorder\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'get_idxs'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnested_reorder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0midxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m--> 233 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0m\u001b[1;32m 234 \u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\n",
"ipdb> event.after_epoch\n",
"'after_epoch'\n",
"ipdb> c\n"
]
}
],
"source": [
"from fastai2.interpret import *\n",
"#interp = Interpretation.from_learner(learn)\n",
"interp = ClassificationInterpretation.from_learner(learn)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#interp.plot_top_losses(3)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5-final"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment