Skip to content

Instantly share code, notes, and snippets.

@awni

awni/MLX LM LoRA Fine Tune.ipynb

Last active April 16, 2025 00:22
Show Gist options
  • Save awni/773e2a12079da40a1cbc566686c84c8f to your computer and use it in GitHub Desktop.
Save awni/773e2a12079da40a1cbc566686c84c8f to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
MLX LM LoRA Fine Tune.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "1055c3f3",
"metadata": {},
"source": [
"### LoRA Fine-Tuning with MLX LM\n",
"\n",
"In this notebook, we'll walk through how to [LoRA fine-tune](https://arxiv.org/abs/2106.09685) an LLM with MLX LM. We'll use the [HellaSwag](https://rowanzellers.com/hellaswag/) dataset for common sense reasoning as an example. An outline:\n",
"\n",
"1. Download the dataset and prepare it in the right format for MLX LM.\n",
"2. Setup and run LoRA training. We'll show how to capture the training logs and plot some statistics to visualize the performance.\n",
"3. Evaluate on the test set. We'll compute the final question-answer accuracy of the fine-tuned model.\n",
"4. Fuse the resulting adapters into the base model and upload to Hugging Face.\n",
"5. Discuss tips for debugging accuracy and efficiency."
]
},
{
"cell_type": "markdown",
"id": "21397627",
"metadata": {},
"source": [
"### Install dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "664272fb",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!pip install mlx-lm\n",
"!pip install matplotlib"
]
},
{
"cell_type": "markdown",
"id": "dd27c693",
"metadata": {},
"source": [
"### Preprocess Data\n",
"We'll start by downloading an already pre-processed version of the HellaSwag dataset from [LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "61698208",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HellaSwag stats: 39905 training examples and 10042 test examples.\n",
"An example:\n",
"\n",
"{\n",
" \"instruction\": \"Please choose the correct ending to complete the given sentence: Removing ice from car: Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then\\n\\nEnding1: , the man adds wax to the windshield and cuts it. Ending2: , a person board a ski lift, while two men supporting the head of the person wearing winter clothes snow as the we girls sled. Ending3: , the man puts on a christmas coat, knitted with netting. Ending4: , the man continues removing the snow on his car.\\n\\nAnswer format: ending1/ending2/ending3/ending4\",\n",
" \"input\": \"\",\n",
" \"output\": \"the correct answer is ending4\",\n",
" \"answer\": \"ending4\"\n",
"}\n"
]
}
],
"source": [
"import json\n",
"import numpy as np\n",
"from pathlib import Path\n",
"from urllib import request\n",
"\n",
"save_dir = \"/tmp/hellaswag\"\n",
"\n",
"def download_and_save(save_dir):\n",
" base_url = \"https://raw.githubusercontent.com/AGI-Edgerunners/LLM-Adapters/main/dataset/hellaswag/\"\n",
" save_dir = Path(save_dir)\n",
" save_dir.mkdir(parents=True, exist_ok=True)\n",
" for name in [\"train.json\", \"test.json\"]:\n",
" out_file = save_dir / name\n",
" if not out_file.exists():\n",
" request.urlretrieve(base_url + name, out_file)\n",
"\n",
"def load_json(dataset):\n",
" download_and_save(save_dir)\n",
" with open(f\"{save_dir}/{dataset}.json\", \"r\") as fid:\n",
" return json.load(fid)\n",
"\n",
"train_set, test_set = load_json(\"train\"), load_json(\"test\")\n",
"print(f\"HellaSwag stats: {len(train_set)} training examples and {len(test_set)} test examples.\")\n",
"print(\"An example:\\n\")\n",
"print(json.dumps(train_set[0], indent=4))"
]
},
{
"cell_type": "markdown",
"id": "9a514d79",
"metadata": {},
"source": [
"Next, let's split the training set into a training and a validation set. We'll pull out a randomly chosen 10% for validation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9b607237",
"metadata": {},
"outputs": [],
"source": [
"# Seed for reproducibility\n",
"np.random.seed(43)\n",
"perm = np.random.permutation(len(train_set))\n",
"valid_size = int(0.1 * len(train_set))\n",
"valid_set = [train_set[i] for i in perm[:valid_size]]\n",
"train_set = [train_set[i] for i in perm[valid_size:]]"
]
},
{
"cell_type": "markdown",
"id": "b259eb69",
"metadata": {},
"source": [
"### Fine-Tune\n",
"\n",
"For fine-tuning, we'll use Microsoft's [Phi-3 mini](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct). At 3.8 billion parameters, Phi-3 mini is a high-quality model that is also fast to fine-tune on most Apple silicon machines. Also, it has a [permissive MIT License](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/LICENSE).\n",
"\n",
"First, import all the packages and functions we need."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c3ff309a",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import mlx.core as mx\n",
"import mlx.optimizers as optim\n",
"from mlx.utils import tree_flatten\n",
"from mlx_lm import load, generate\n",
"from mlx_lm.tuner import train, evaluate, TrainingArgs\n",
"from mlx_lm.tuner.datasets import CompletionsDataset\n",
"from mlx_lm.tuner import linear_to_lora_layers\n",
"import tqdm\n",
"import os\n",
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\""
]
},
{
"cell_type": "markdown",
"id": "87628d24",
"metadata": {},
"source": [
"Next, setup the LoRA parameters and make the training arguments. See the [training argument class](https://github.com/ml-explore/mlx-examples/blob/81318ad4a8b2ca5fd1431a42db2b0244d16be851/llms/mlx_lm/tuner/trainer.py#L31-L63) for a more detailed list of training parameters. \n",
"\n",
"Recall the LoRA update is $W^\\top \\mathbf{x} + c \\cdot \\mathbf{a} \\mathbf{b}^\\top \\mathbf{x}$ where $\\mathbf{a}$ has shape `(D, rank)`.\n",
"\n",
"With that in mind, the LoRA parameters to attend to are:\n",
"- `lora_layers`: The number of Transformer blocks from the top of the model to adapt.\n",
"- `rank`: The rank of the low-rank adapters. A larger rank implies more adapter parameters per linear layer.\n",
"- `scale`: This is the constant $c$ that scales the low-rank update."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "f0851dc8",
"metadata": {},
"outputs": [],
"source": [
"# Make a directory to save the adapter config and weights\n",
"adapter_path = Path(\"adapters\")\n",
"adapter_path.mkdir(parents=True, exist_ok=True)\n",
"\n",
"lora_config = {\n",
" \"num_layers\": 8,\n",
" \"lora_parameters\": {\n",
" \"rank\": 8,\n",
" \"scale\": 20.0,\n",
" \"dropout\": 0.0,\n",
"}}\n",
"\n",
"# Save the LoRA config to the adapter path\n",
"with open(adapter_path / \"adapter_config.json\", \"w\") as fid:\n",
" json.dump(lora_config, fid, indent=4) \n",
"\n",
"training_args = TrainingArgs(\n",
" adapter_file=adapter_path / \"adapters.safetensors\",\n",
" iters=200,\n",
" steps_per_eval=50\n",
")"
]
},
{
"cell_type": "markdown",
"id": "56fefd19",
"metadata": {},
"source": [
"Next, load the Phi-3 mini model. Note this may take a few minutes to download from Hugging Face if you haven't downloaded it before."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "fb0b16f2",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5987d49c0f3d4011ba368e4004c0cf54",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Fetching 13 files: 0%| | 0/13 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model_path = \"microsoft/Phi-3-mini-4k-instruct\"\n",
"model, tokenizer = load(model_path)"
]
},
{
"cell_type": "markdown",
"id": "6609c92a",
"metadata": {},
"source": [
"After loading the model, freeze it's parameters so we don't train them. Then convert linear layers to LoRA layers using the MLX LM utility `linear_to_lora_layers`. The adapters in the `LoRA` layers are not frozen, so they will be included in the model's `trainable_parameters`. Check-out the [LoRA layer implementation](https://github.com/ml-explore/mlx-examples/blob/81318ad4a8b2ca5fd1431a42db2b0244d16be851/llms/mlx_lm/tuner/lora.py#L72-L104) to see how it all works.\n",
"\n",
"By default, MLX LM only adapts the query, key, and value projection matrices for Phi-3. You can specify the layers to adapt by setting `lora_parameters[\"keys\"]` to a list of layer names. In this case it defaults to `[\"attn.qkv_proj\"]`. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "50e1ab3a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of trainable parameters: 3145728\n"
]
}
],
"source": [
"# Freeze the base model\n",
"model.freeze()\n",
"\n",
"# Convert linear layers to lora layers\n",
"linear_to_lora_layers(model, lora_config[\"num_layers\"], lora_config[\"lora_parameters\"])\n",
"\n",
"num_train_params = (\n",
" sum(v.size for _, v in tree_flatten(model.trainable_parameters()))\n",
")\n",
"print(f\"Number of trainable parameters: {num_train_params}\")"
]
},
{
"cell_type": "markdown",
"id": "d97656ab",
"metadata": {},
"source": [
"The last step before training is to pre-process the data sets. The pre-processing applies a chat-template and special tokens if necessary. It also tokenizes the data and converts the tokens to integer ids. We'll use a `CompletionsDataset` for this example which does the all of that. For more details, see the [documentation on supported formats](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/LORA.md#Data)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1c5add0b",
"metadata": {},
"outputs": [],
"source": [
"# Configure the tokenizer to add the eos token id at the end of each example\n",
"def make_dataset(ds):\n",
" return CompletionsDataset(\n",
" ds,\n",
" tokenizer,\n",
" prompt_key=\"instruction\",\n",
" completion_key=\"output\",\n",
" mask_prompt=False,\n",
" )\n",
"train_set, valid_set = make_dataset(train_set), make_dataset(valid_set)"
]
},
{
"cell_type": "markdown",
"id": "827d1590",
"metadata": {},
"source": [
"Now we're ready to put it all together and actually train the model. We'll use `Adam` for the optimizer, but you can specify any [optimizer](https://ml-explore.github.io/mlx/build/html/python/optimizers/common_optimizers.html) with any [scheduler](https://ml-explore.github.io/mlx/build/html/python/optimizers/schedulers.html). We also added a custom class to capture the training and validation loss to plot it later."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "984516d3",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Starting training..., iters: 200\n",
"Iter 1: Val loss 2.355, Val took 28.889s\n",
"Iter 10: Train loss 2.122, Learning Rate 1.000e-05, It/sec 0.277, Tokens/sec 319.293, Trained Tokens 11539, Peak mem 16.357 GB\n",
"Iter 20: Train loss 1.726, Learning Rate 1.000e-05, It/sec 0.332, Tokens/sec 287.068, Trained Tokens 20195, Peak mem 16.357 GB\n",
"Iter 30: Train loss 1.745, Learning Rate 1.000e-05, It/sec 0.438, Tokens/sec 441.289, Trained Tokens 30259, Peak mem 16.357 GB\n",
"Iter 40: Train loss 1.507, Learning Rate 1.000e-05, It/sec 0.409, Tokens/sec 440.940, Trained Tokens 41051, Peak mem 16.357 GB\n",
"Iter 50: Val loss 1.554, Val took 27.485s\n",
"Iter 50: Train loss 1.580, Learning Rate 1.000e-05, It/sec 0.317, Tokens/sec 343.995, Trained Tokens 51911, Peak mem 16.357 GB\n",
"Iter 60: Train loss 1.556, Learning Rate 1.000e-05, It/sec 0.369, Tokens/sec 404.053, Trained Tokens 62875, Peak mem 16.357 GB\n",
"Iter 70: Train loss 1.502, Learning Rate 1.000e-05, It/sec 0.407, Tokens/sec 441.838, Trained Tokens 73719, Peak mem 16.357 GB\n",
"Iter 80: Train loss 1.517, Learning Rate 1.000e-05, It/sec 0.409, Tokens/sec 405.316, Trained Tokens 83636, Peak mem 16.357 GB\n",
"Iter 90: Train loss 1.556, Learning Rate 1.000e-05, It/sec 0.389, Tokens/sec 365.788, Trained Tokens 93040, Peak mem 16.357 GB\n",
"Iter 100: Val loss 1.537, Val took 25.555s\n",
"Iter 100: Train loss 1.457, Learning Rate 1.000e-05, It/sec 0.286, Tokens/sec 291.589, Trained Tokens 103248, Peak mem 16.357 GB\n",
"Iter 100: Saved adapter weights to adapters/adapters.safetensors and adapters/0000100_adapters.safetensors.\n",
"Iter 110: Train loss 1.561, Learning Rate 1.000e-05, It/sec 0.451, Tokens/sec 413.650, Trained Tokens 112412, Peak mem 16.357 GB\n",
"Iter 120: Train loss 1.452, Learning Rate 1.000e-05, It/sec 0.481, Tokens/sec 426.075, Trained Tokens 121272, Peak mem 16.357 GB\n",
"Iter 130: Train loss 1.487, Learning Rate 1.000e-05, It/sec 0.492, Tokens/sec 433.570, Trained Tokens 130088, Peak mem 16.357 GB\n",
"Iter 140: Train loss 1.504, Learning Rate 1.000e-05, It/sec 0.383, Tokens/sec 380.423, Trained Tokens 140028, Peak mem 16.357 GB\n",
"Iter 150: Val loss 1.521, Val took 27.191s\n",
"Iter 150: Train loss 1.516, Learning Rate 1.000e-05, It/sec 0.375, Tokens/sec 408.976, Trained Tokens 150932, Peak mem 16.357 GB\n",
"Iter 160: Train loss 1.526, Learning Rate 1.000e-05, It/sec 0.528, Tokens/sec 407.316, Trained Tokens 158640, Peak mem 16.357 GB\n",
"Iter 170: Train loss 1.427, Learning Rate 1.000e-05, It/sec 0.561, Tokens/sec 433.830, Trained Tokens 166368, Peak mem 16.357 GB\n",
"Iter 180: Train loss 1.508, Learning Rate 1.000e-05, It/sec 0.354, Tokens/sec 444.070, Trained Tokens 178916, Peak mem 16.583 GB\n",
"Iter 190: Train loss 1.497, Learning Rate 1.000e-05, It/sec 0.459, Tokens/sec 436.021, Trained Tokens 188421, Peak mem 16.583 GB\n",
"Iter 200: Val loss 1.522, Val took 27.453s\n",
"Iter 200: Train loss 1.491, Learning Rate 1.000e-05, It/sec 0.499, Tokens/sec 416.295, Trained Tokens 196769, Peak mem 16.583 GB\n",
"Iter 200: Saved adapter weights to adapters/adapters.safetensors and adapters/0000200_adapters.safetensors.\n",
"Saved final weights to adapters/adapters.safetensors.\n"
]
}
],
"source": [
"# Put the model in training mode:\n",
"model.train()\n",
"\n",
"# Make the optimizer:\n",
"opt = optim.Adam(learning_rate=1e-5)\n",
"\n",
"# Make a class to record the training stats:\n",
"class Metrics:\n",
" train_losses = []\n",
" val_losses = []\n",
" def on_train_loss_report(self, info):\n",
" self.train_losses.append((info[\"iteration\"], info[\"train_loss\"]))\n",
" def on_val_loss_report(self, info):\n",
" self.val_losses.append((info[\"iteration\"], info[\"val_loss\"]))\n",
"\n",
"metrics = Metrics()\n",
"\n",
"# Train model:\n",
"train(\n",
" model=model,\n",
" tokenizer=tokenizer,\n",
" args=training_args,\n",
" optimizer=opt,\n",
" train_dataset=train_set,\n",
" val_dataset=valid_set,\n",
" training_callback=metrics,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "b8d043b8",
"metadata": {},
"source": [
"The adapters are saved every 100 iterations along with the final adapters in `adapters.safetensors`."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "ac329358",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0000100_adapters.safetensors adapter_config.json\r\n",
"0000200_adapters.safetensors adapters.safetensors\r\n"
]
}
],
"source": [
"!ls adapters/"
]
},
{
"cell_type": "markdown",
"id": "2b7e23ee",
"metadata": {},
"source": [
"Next, let's plot the training and validation losses to see how well the adapters fit the data."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "f1ffd638",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAG1CAYAAAAFuNXgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnPklEQVR4nO3dd3hUZfr/8fdMekgjkJAEAqEKkS5FQBQVBVQUG4oFWV1dFV1ZV78u61r3p6C761oXt9jBriCiooCANKUGCJ0QSICEkpBO2sz5/THMQCQ9M5mSz+u65srJmXOec09OkrnnqSbDMAxEREREfITZ3QGIiIiIOJOSGxEREfEpSm5ERETEpyi5EREREZ+i5EZERER8ipIbERER8SlKbkRERMSnKLkRERERn6LkRkRERHyKkhsRERHxKW5NbmbMmMHgwYMJDw8nNjaWCRMmsGvXrnqf//HHH2MymZgwYYLrghQRERGv4tbkZvny5UydOpWff/6ZRYsWUVFRweWXX05xcXGd5+7fv59HHnmEkSNHNkOkIiIi4i1MnrRw5rFjx4iNjWX58uVceOGFNR5nsVi48MILufPOO1mxYgV5eXnMmzevXtewWq0cPnyY8PBwTCaTkyIXERERVzIMg8LCQhISEjCba6+b8W+mmOolPz8fgOjo6FqPe/bZZ4mNjeWuu+5ixYoVtR5bVlZGWVmZ4/tDhw6RnJzc9GBFRESk2WVmZtKhQ4daj/GY5MZqtTJt2jRGjBhB7969azxu5cqVvPXWW6SkpNSr3BkzZvDMM8+ctT8zM5OIiIjGhisiIiLNqKCggMTERMLDw+s81mOSm6lTp5KamsrKlStrPKawsJDbb7+d//73v7Rt27Ze5U6fPp2HH37Y8b39hxMREaHkRkRExMvUp0uJRyQ3DzzwAAsWLOCnn36qtaopLS2N/fv3M378eMc+q9UKgL+/P7t27aJr165VzgkKCiIoKMg1gYuIiIjHcWtyYxgGDz74IHPnzmXZsmV07ty51uN79uzJ1q1bq+z7y1/+QmFhIa+88gqJiYmuDFdERES8gFuTm6lTp/Lhhx/y1VdfER4eTnZ2NgCRkZGEhIQAMHnyZNq3b8+MGTMIDg4+qz9OVFQUQK39dERERKTlcGtyM2vWLABGjRpVZf8777zDlClTAMjIyKhzyJeIiIgnsFgsVFRUuDsMrxUYGOiU93yPmuemORQUFBAZGUl+fr46FIuIiFMYhkF2djZ5eXnuDsWrmc1mOnfuTGBg4FnPNeT92yM6FIuIiHgze2ITGxtLaGioJoltBPsku1lZWXTs2LFJP0MlNyIiIk1gsVgciU2bNm3cHY5Xi4mJ4fDhw1RWVhIQENDoctSZRUREpAnsfWxCQ0PdHIn3szdHWSyWJpWj5EZERMQJ1BTVdM76GapZylmsFjiwGoqOQFg76DQczH7ujkpERKTFUXLjDNvnw8LHoODw6X0RCTD2BUi+2n1xiYiINKOkpCSmTZvGtGnT3BqHmqWaavt8+HRy1cQGoCDLtn/7fPfEJSIiXsViNViTlsNXKYdYk5aDxeq6mVpMJlOtj6effrpR5a5bt4577rnHucE2gmpumsJqsdXYUN0voAGYYOGfoOeVaqISEZEaLUzN4pmvt5OVX+rYFx8ZzFPjkxnbO97p18vKynJsf/LJJzz55JPs2rXLsS8sLMyxbRgGFosFf/+6U4aYmBjnBtpIqrlpigOrz66xqcKAgkO240RERKqxMDWL+2ZvrJLYAGTnl3Lf7I0sTM2q4czGi4uLczwiIyMxmUyO73fu3El4eDjfffcd5513HkFBQaxcuZK0tDSuueYa2rVrR1hYGIMHD2bx4sVVyk1KSuLll192fG8ymfjf//7HtddeS2hoKN27d2f+fNe3aCi5aYqiI849TkREvJ5hGJSUV9brUVhawVPzt9VY/w/w9PztFJZW1Ks8Zy468Kc//YmZM2eyY8cO+vbtS1FREVdccQVLlixh06ZNjB07lvHjx5ORkVFrOc888wwTJ05ky5YtXHHFFdx6663k5uY6Lc7qqFmqKcLaOfc4ERHxeicrLCQ/+b1TyjKA7IJS+jz9Q72O3/7sGEIDnfPW/uyzz3LZZZc5vo+OjqZfv36O7//6178yd+5c5s+fzwMPPFBjOVOmTGHSpEkAPP/887z66qusXbuWsWPHOiXO6qjmpik6DbeNiqKmcfkmiGhvO05ERMSLDBo0qMr3RUVFPPLII/Tq1YuoqCjCwsLYsWNHnTU3ffv2dWy3atWKiIgIjh496pKY7VRz0xRmP9tw708nY0twqqkOHDtTnYlFRFqQkAA/tj87pl7Hrk3PZco76+o87t3fDGZI5+h6XdtZWrVqVeX7Rx55hEWLFvH3v/+dbt26ERISwg033EB5eXmt5fx6GQWTyYTVanVanNVRctNUyVfDxPfPnufGHAA3vKV5bkREWhiTyVTvpqGR3WOIjwwmO7+02n43JiAuMpiR3WPwM7t3BuRVq1YxZcoUrr32WsBWk7N//363xlQTNUs5Q/LVMC0V7lgAV71iS2ysFRDVyd2RiYiIB/Mzm3hqfDJwdgcH+/dPjU92e2ID0L17d7788ktSUlLYvHkzt9xyi8trYBpLyY2zmP2g80gYNOV0bc2m2W4NSUREPN/Y3vHMum0gcZHBVfbHRQYz67aBLpnnpjFeeuklWrduzfDhwxk/fjxjxoxh4MCB7g6rWibDmePGvEBBQQGRkZHk5+cTERHhmouk/QgfXAvBkfDH3RAQXPc5IiLilUpLS0lPT6dz584EBzf+/73FarA2PZejhaXEhgczpHO0R9TYNKfafpYNef9WnxtX6HwRRCZCfibsXAB9bnB3RCIi4uH8zCaGdW3j7jB8gpqlXMHsB/1vsW2raUpERKRZKblxFXtys28Z5NU+B4CIiIg4j5IbV2mdBJ0vBAxI+cjd0YiIiLQYSm5cacDttq8ps8FDh8uJiIj4GiU3rtRrPARF2pql9q9wdzQiIiItgpIbVwoIgT7X27bVsVhERKRZKLlxtQG32b7umA8n89waioiISEug5MbVEgZCbDJUlkLqF+6ORkRExOcpuXE1k+l07Y2apkRExEeMGjWKadOmOb5PSkri5ZdfrvUck8nEvHnzXBoXKLlpHn1vArM/HN4IR7a5OxoREfFEVgukr4Ctn9u+Wi0uu9T48eMZO3Zstc+tWLECk8nEli1bGlTmunXruOeee5wRXpMpuWkOrdrCOeNs25vmuDcWERHxPNvnw8u94b2r4Iu7bF9f7m3b7wJ33XUXixYt4uDBg2c998477zBo0CD69u3boDJjYmIIDQ11VohNouSmudjnvNnyMVSWuzcWERHxHNvnw6eToeBw1f0FWbb9LkhwrrrqKmJiYnj33Xer7C8qKuKzzz5jwoQJTJo0ifbt2xMaGkqfPn346KPaJ6T9dbPUnj17uPDCCwkODiY5OZlFixY5/XXURMlNc+l6KYTFQUkO7F7o7mhERMRVDAPKi+v3KC2A7/4PMKoryPZl4WO24+pTnlFdOWfz9/dn8uTJvPvuuxhnnPPZZ59hsVi47bbbOO+88/jmm29ITU3lnnvu4fbbb2ft2rX1Kt9qtXLdddcRGBjIL7/8wptvvsljjz1Wr3OdQauCNxc/f+g/CVb+09axOPlqd0ckIiKuUFECzyc4qTDDVqMzM7F+h//5MAS2qtehd955J3/7299Yvnw5o0aNAmxNUtdffz2dOnXikUcecRz74IMP8v333/Ppp58yZMiQOstevHgxO3fu5PvvvychwfazeP755xk3blz9XkcTqeamOfU/NWpq7yJbdaOIiIib9OzZk+HDh/P2228DsHfvXlasWMFdd92FxWLhr3/9K3369CE6OpqwsDC+//57MjLqtxD0jh07SExMdCQ2AMOGDXPJ66iOam6aU9tu0HEYZKyBzR/ByIfdHZGIiDhbQKitBqU+DqyGOTfUfdytn0On4fW7dgPcddddPPjgg7zxxhu88847dO3alYsuuogXXniBV155hZdffpk+ffrQqlUrpk2bRnm5d/QZVc1Ncztzzpt6to2KiIgXMZlsTUP1eXS9BCISAFNNhUFEe9tx9SnPVFM51Zs4cSJms5kPP/yQ999/nzvvvBOTycSqVau45ppruO222+jXrx9dunRh9+7d9S63V69eZGZmkpV1upXi559/blBsTaHkprklT4CAVpCbBhnNd6NFRMQDmf1g7Aunvvl1YnLq+7Ezbce5QFhYGDfddBPTp08nKyuLKVOmANC9e3cWLVrE6tWr2bFjB7/73e84cuRIvcsdPXo0PXr04I477mDz5s2sWLGCxx9/3CWvoTpKbppbUBj0vta2rRmLRUQk+WqY+D5ExFfdH5Fg2+/iASh33XUXJ06cYMyYMY4+Mn/5y18YOHAgY8aMYdSoUcTFxTFhwoR6l2k2m5k7dy4nT55kyJAh/Pa3v+W5555z0Ss4m8kwWlbbSEFBAZGRkeTn5xMREeGeIDJ+hrfH2GpwHtkFQeHuiUNERJqstLSU9PR0OnfuTHBwcOMLslpsfXCKjkBYO1sfGxfV2Hiq2n6WDXn/Vodid0gcCm26Qc5e2DYPBt7u7ohERMTdzH7QeaS7o/AJapZyBy2mKSIi4jJKbtyl3yQw+UHmz3B8j7ujERER8RlKbtwlPA66X2bbVu2NiIiI0yi5cSd709Tmj8BS6d5YRESkSVrY+ByXcNbPUMmNO3UfA6FtbT3j9y52dzQiItIIAQEBAJSUlLg5Eu9nnwHZz69po8Q0Wsqd/AOh382w5nXY9AGcM9bdEYmISAP5+fkRFRXF0aNHAQgNDcXUwJmCxbaS+LFjxwgNDcXfv2npiZIbd+t/qy252b0Qio5BWIy7IxIRkQaKi4sDcCQ40jhms5mOHTs2OTlUcuNu7ZKh/XlwaANs+QSGP+DuiEREpIFMJhPx8fHExsZSUVHh7nC8VmBgIGZz03vMKLnxBANusyU3mz6AYVMbvPCZiIh4Bj8/vyb3F5GmU4diT9D7evAPhmM74dBGd0cjIiLi1ZTceILgSEi+xra96QP3xiIiIuLllNx4CvucN6lfQLmGE4qIiDSWkhtP0ekCiOoEZQWw42t3RyMiIuK1lNx4CrP5jMU01TQlIiLSWEpuPEm/SYAJ9q+A3HR3RyMiIuKVlNx4kqhE6HqxbTvlQ/fGIiIi4qWU3Hgae9NUyodgtbg3FhERES+k5MbTnHMlBEdBwUHYt8zd0YiIiHgdJTeeJiAY+k60bW+a7d5YREREvJCSG09kb5rauQBKct0bi4iIiJdRcuOJ4vtBXB+wlMPWz90djYiIiFdRcuOpBtxu+6o5b0RERBrErcnNjBkzGDx4MOHh4cTGxjJhwgR27dpV6zn//e9/GTlyJK1bt6Z169aMHj2atWvXNlPEzajPjeAXCNlbIGuzu6MRERHxGm5NbpYvX87UqVP5+eefWbRoERUVFVx++eUUFxfXeM6yZcuYNGkSS5cuZc2aNSQmJnL55Zdz6NChZoy8GYRGQ88rbdub5rg3FhERES9iMgzDcHcQdseOHSM2Npbly5dz4YUX1usci8VC69atef3115k8eXKdxxcUFBAZGUl+fj4RERFNDdm19i6G2dfbhob/cZdtJJWIiEgL1JD3b4/qc5Ofnw9AdHR0vc8pKSmhoqKixnPKysooKCio8vAaXS6GiPZQmge7vnV3NCIiIl7BY5Ibq9XKtGnTGDFiBL179673eY899hgJCQmMHj262udnzJhBZGSk45GYmOiskF3P7Af9b7Fta84bERGRevGY5Gbq1Kmkpqby8ccf1/ucmTNn8vHHHzN37lyCg6tvspk+fTr5+fmOR2ZmprNCbh725CbtR8g/6N5YREREvIBHJDcPPPAACxYsYOnSpXTo0KFe5/z9739n5syZ/PDDD/Tt27fG44KCgoiIiKjy8CrRXSBpJGBAykfujkZERMTjuTW5MQyDBx54gLlz5/Ljjz/SuXPnep334osv8te//pWFCxcyaNAgF0fpARyLac4Gq9W9sYiIiHg4tyY3U6dOZfbs2Xz44YeEh4eTnZ1NdnY2J0+edBwzefJkpk+f7vj+hRde4IknnuDtt98mKSnJcU5RUZE7XkLz6HU1BIbDif1wYJW7oxEREfFobk1uZs2aRX5+PqNGjSI+Pt7x+OSTTxzHZGRkkJWVVeWc8vJybrjhhirn/P3vf3fHS2gegaHQ53rbtjoWi4iI1Mqj5rlpDl41z82ZDq6H/10K/iHwyC4IjnR3RCIiIs3Ga+e5kVq0Pw9iekLlSUj90t3RiIiIeCwlN97CZDrdsVhNUyIiIjVScuNN+t4EZn84tB6O7nB3NCIiIh5JyY03CYuFHmNt26q9ERERqZaSG29jb5ra/DFYKtwbi4iIiAdScuNtul0GYe2g5Djs/t7d0YiIiHgcJTfexs8f+t1s21bTlIiIyFmU3Hij/qeapvb8AIXZ7o1FRETEwyi58UYxPSBxKBgWW98bERERcVBy463OnPOmZU0yLSIiUislN97q3GshIBRy9kDmWndHIyIi4jGU3HiroHBbggOw6QP3xiIiIuJBlNx4M3vT1La5UFbk3lhEREQ8hJIbb9ZxGER3gfIi2P6Vu6MRERHxCEpuvJkW0xQRETmLkhtv128SmMyQsRqO73V3NCIiIm6n5MZDWawGa9Jy+CrlEGvScrBYaxjuHZEA3UbbtlPmNF+AIiIiHsrf3QHI2RamZvHM19vJyi917IuPDOap8cmM7R1/9gkDbrPNVrz5I7j4cdsSDSIiIi2Uam48zMLULO6bvbFKYgOQnV/KfbM3sjA16+yTeoyD0DZQmAVpPzZTpCIiIp5JyY0HsVgNnvl6O9U1QNn3PfP19rObqPwDoa99MU3NeSMiIi2bkhsPsjY996wamzMZQFZ+KWvTc89+0j5qatd3UHzcNQGKiIh4ASU3HuRoYc2JTZ3HtUuGhIFgrYAtnzo5MhEREe+h5MaDxIYHN+04LaYpIiKi5MaTDOkcTXxkMKYanjdhGzU1pHN09Qf0vh78g+HoNji8yVVhioiIeDQlNx7Ez2ziqfHJAGclOPbvnxqfjJ+5hvQnJAp6XW3b1ozFIiLSQim58TBje8cz67aBxEVWbXqKjQhi1m0Dq5/n5kz2pqmtn0PFSRdFKSIi4rmU3Higsb3jWfnYJXx09/lEhgQA8Pot9UhsAJJGQlRHKMuHHQtcHKmIiIjnUXLjofzMJoZ1bUPv9hEApB8vrt+JZjP0t3cs1pw3IiLS8ii58XDdYsIASDtWVP+T+k8CTJC+HE4ccE1gIiIiHkrJjYfrGnsquTnagOQmqiN0GWXbTvnQ+UGJiIh4MCU3Hu50zU09m6Xs7B2LU+aA1erkqERERDyXkhsPZ6+5OZBTTFmlpf4n9rwKgiMhP9PWPCUiItJCKLnxcLHhQYQH+WM14EBOSf1PDAiGPhNt25rzRkREWhAlNx7OZDLR5VTtzd6G9LuB001TO76GkyecHJmIiIhnUnLjBRz9bhqa3MT3g3Z9wFJmm9RPRESkBVBy4wW6xrYCYG9DhoMDmExVF9MUERFpAZTceIFGzXVj13ci+AVCVgpkb3VuYCIiIh5IyY0XOD3XTTFWq9Gwk0Oj4ZwrbNub5jg5MhEREc+j5MYLdIwOJcDPxMkKC1kFpQ0vYMDttq9bPoHKMucGJyIi4mGU3HiBAD8zndqc6nfT0E7FAF0vhvAEOJkLu75zcnQiIiKeRcmNl2j0iCkAsx/0v8W2rY7FIiLi45TceIlGj5iysyc3aUsg/5CTohIREfE8Sm68RLfGLKB5pjZdodMFYFhh80dOjExERMSzKLnxEl2bMhzc7sw5b4wGjroSERHxEkpuvIQ9uTleVE5eSXnjCkm+GgLD4UQ6HFjtxOhEREQ8h5IbL9EqyJ/4yGCgCbU3ga2g93W2bXUsFhERH6Xkxot0O2Myv0azz3mzfR6UFjQ9KBEREQ+j5MaL2JumGj1iCqDDIGh7DlSUwLa5TopMRETEcyi58SJdmzpiCrSYpoiI+DwlN16ka4xtrpsmjZgC6HczmPzg4Fo4tssJkYmIiHgOJTdexN7nJiO3hNIKS+MLCouFHmNt26q9ERERH6PkxovEhAURHuyP1YADOSVNK8zeNLX5Y7BUND04ERERD6HkxouYTCZH7U2jFtA8U/fLoFUsFB+FPYucEJ2IiIhnUHLjZZwyUzGAX4Ct7w2oaUpERHyKkhsv47SaGzjdNLV7IRQeaXp5IiIiHkDJjZdxWs0NQMw50GEIGBbY8knTyxMREfEASm68jGOW4mNFWK1OWPxSi2mKiIiPUXLjZRJbhxDoZ6a0wsrh/JNNL/DcayEgFI7vgoPrm16eiIiImym58TL+fmaS2oYCTup3ExwByRNs25s+aHp5IiIibqbkxgud7nfThAU0z2Rvmkr9EsqdVKaIiIibKLnxQk4dMQXQaThEd4HyQtg+3zllioiIuImSGy/k1BFTYFtMs/+ttm3NeSMiIl7OrcnNjBkzGDx4MOHh4cTGxjJhwgR27ap7IcfPPvuMnj17EhwcTJ8+ffj222+bIVrP0c0Zq4P/Wr9JYDLDgZWQk+a8ckVERJqZW5Ob5cuXM3XqVH7++WcWLVpERUUFl19+OcXFNff7WL16NZMmTeKuu+5i06ZNTJgwgQkTJpCamtqMkbtXl1Org+cUl3OiuNw5hUa2h66X2rZTPnROmSIiIm5gMgzPmdzk2LFjxMbGsnz5ci688MJqj7npppsoLi5mwYIFjn3nn38+/fv3580336zzGgUFBURGRpKfn09ERITTYm9uI2b+yKG8k3x+7zAGJUU7p9Bt8+CzOyA8Af6QCmY/55QrIiLSRA15//aoPjf5+fkAREfX/Ga9Zs0aRo8eXWXfmDFjWLNmTbXHl5WVUVBQUOXhC+y1N07rdwNwzjgIiYbCw5C21HnlioiINCOPSW6sVivTpk1jxIgR9O7du8bjsrOzadeuXZV97dq1Izs7u9rjZ8yYQWRkpOORmJjo1LjdxekjpgD8g6DvTbZtzXkjIiJeymOSm6lTp5KamsrHH3/s1HKnT59Ofn6+45GZmenU8t3F6XPd2A04NWpq5zdQnOPcskVERJqBRyQ3DzzwAAsWLGDp0qV06NCh1mPj4uI4cqTqCtZHjhwhLi6u2uODgoKIiIio8vAFLqm5AYjrA/H9wVoBWz9zbtkiIiLNwK3JjWEYPPDAA8ydO5cff/yRzp0713nOsGHDWLJkSZV9ixYtYtiwYa4K0yPZa24yT5RQWmFxbuGOxTQ/0GKaIiLiddya3EydOpXZs2fz4YcfEh4eTnZ2NtnZ2Zw8eXpByMmTJzN9+nTH9w899BALFy7kH//4Bzt37uTpp59m/fr1PPDAA+54CW7TNiyQyJAADAPSjzu5aarPDeAXBEdSIWuzc8sWERFxMbcmN7NmzSI/P59Ro0YRHx/veHzyySeOYzIyMsjKynJ8P3z4cD788EP+85//0K9fPz7//HPmzZtXaydkX2QymejqihFTACGtodd427Y6FouIiJfxd+fF6zPFzrJly87ad+ONN3LjjTe6ICLv0i02jI0Zec7vdwO2pqnUz239bi7/fxAQ4vxriIiIuIBHdCiWxnHZiCmAzhdBZCKU5ttGTomIiHgJJTdezGUjpgDM5jMW01TTlIiIeA8lN17MXnOz71gRVqsLRjX1v8X2dd9yOHHA+eWLiIi4gJIbL5YYHUqgn5mySiuH8k7WfUJDte5ka57CgM0fOb98ERERF1By48X8zCY6t7WNmNrr7BFTdgNut33dNAesVtdcQ0RExImU3Hg5e7+bNFf0uwHodRUERUJ+Buz/yTXXEBERcSIlN17OZXPd2AWE2Cb1A9g02zXXEBERcSIlN16uqytHTNnZl2PYPh9OnnDddURERJxAyY2Xc+lcN3YJAyD2XLCUQeoXrruOiIiIEyi58XL25Ca3uJzc4nLXXMRkOmMxTTVNiYiIZ1Ny4+VCAv1oH2VbGsFl/W4A+k4EcwAc3gTZqa67joiISBMpufEBLp2p2K5VWzhnnG07ZY7rriMiItJESm58gKPfjSuTGzg9583mj6HSRU1gIiIiTaTkxgc4am5c2SwF0PUSCI+Hk7mw+zvXXktERKSRlNz4AJfPdWPn5w/9Jtm21bFYREQ8lJIbH2CvuTl44iSlFRbXXsw+amrvYig47NpriYiINIKSGx8Q3SqQqNAADAP2uXK+G4A2XaHjcDCsWkxTREQ8kpIbH2AymegW00z9bqDqnDeG4frriYiINICSGx/RbCOmAJKvgcAwyN0HGWtcfz0REZEGUHLjI5ptxBRAUBice61tWx2LRUTEwyi58RFdY0+NmGqOmhs4PefNtrlQVtg81xQREamHRiU3mZmZHDx40PH92rVrmTZtGv/5z3+cFpg0TLeYcAD2HS/GYm2GfjCJQ6BNd6gosSU4IiIiHqJRyc0tt9zC0qVLAcjOzuayyy5j7dq1PP744zz77LNODVDqp33rEAL9zZRXWjl04qTrL6jFNEVExEM1KrlJTU1lyJAhAHz66af07t2b1atXM2fOHN59911nxif15Gc20aWtrWlq77FmaibqdzOY/CDzFzi2u3muKSIiUodGJTcVFRUEBQUBsHjxYq6++moAevbsSVZWlvOikwbpGmsfMeXiuW7swuOg++W27RTV3oiIiGdoVHJz7rnn8uabb7JixQoWLVrE2LFjATh8+DBt2rRxaoBSf465bpqrUzGcbppK+QgsFc13XRERkRo0Krl54YUX+Pe//82oUaOYNGkS/fr1A2D+/PmO5ippfo6am+YYDm7XYwy0ioHio7YlGURERNzMvzEnjRo1iuPHj1NQUEDr1q0d+++55x5CQ0OdFpw0zJmzFBuGgclkcv1F/QKg702w5nVbx+Jzxrn+miIiIrVoVM3NyZMnKSsrcyQ2Bw4c4OWXX2bXrl3ExsY6NUCpvy4xrTCZIK+kgtzi8ua7sL1pavdCKDrafNcVERGpRqOSm2uuuYb3338fgLy8PIYOHco//vEPJkyYwKxZs5waoNRfcIAfHVqHAM3c7ya2F7QfBNZK2PJJ811XRESkGo1KbjZu3MjIkSMB+Pzzz2nXrh0HDhzg/fff59VXX3VqgNIwjjWmXL06+K9pMU0REfEQjUpuSkpKCA+3zYj7ww8/cN1112E2mzn//PM5cOCAUwOUhnHLiCmA3teBfwgc2wmHNjTvtUVERM7QqOSmW7duzJs3j8zMTL7//nsuv9w218nRo0eJiIhwaoDSMG4ZMQUQHGlbLRxg0wfNe20REZEzNCq5efLJJ3nkkUdISkpiyJAhDBs2DLDV4gwYMMCpAUrDOFYHb+6aGzjdNLX1Cygvaf7ri4iI0Mjk5oYbbiAjI4P169fz/fffO/Zfeuml/POf/3RacNJw9j43h/JOcrLc0rwX7zQCWidBeSHsmN+81xYRETmlUckNQFxcHAMGDODw4cOOFcKHDBlCz549nRacNFx0q0CiWwUCbmiaMpuhvxbTFBER92pUcmO1Wnn22WeJjIykU6dOdOrUiaioKP76179itVqdHaM0UNcY2wKazZ7cAPSfBJhg/wrI3df81xcRkRavUcnN448/zuuvv87MmTPZtGkTmzZt4vnnn+e1117jiSeecHaM0kDdHAtouiG5iewAXS+xbad82PzXFxGRFq9Ryy+89957/O9//3OsBg7Qt29f2rdvz/33389zzz3ntACl4dw2143dgNsgbYktuRk1Hcx+7olDRERapEbV3OTm5lbbt6Znz57k5uY2OShpmq7uHDEF0PNKCGkNBYdg31L3xCAiIi1Wo5Kbfv368frrr5+1//XXX6dv375NDkqaxj6RX/rxYixWN8wW7B8EfSbattWxWEREmlmjmqVefPFFrrzyShYvXuyY42bNmjVkZmby7bffOjVAabj2USEE+Zspq7SSmVtCUttWzR/EgNtg7b9h5zdQkguh0c0fg4iItEiNqrm56KKL2L17N9deey15eXnk5eVx3XXXsW3bNj74QLPTupvZbKJLjJtmKraL7wtxfcFSDls/c08MIiLSIpkMw3mrHG7evJmBAwdisTTz5HENUFBQQGRkJPn5+T69VMSDH23i682HmT6uJ7+7qKt7gvjlP/DdoxDXB+5d6Z4YRETEJzTk/bvRk/iJZ3PrXDd2fW4Av0DI3gpZm90Xh4iItChKbnyUW9eYsguNhp5X2bbVsVhERJqJkhsfdeZcN05seWw4+2KaWz6FilL3xSEiIi1Gg0ZLXXfddbU+n5eX15RYxIk6t22FyQT5Jys4XlROTHiQewLpMgoiOkDBQdj1DfS+3j1xiIhIi9GgmpvIyMhaH506dWLy5MmuilUaIDjAj8TWoYCb+92Y/aD/LbZtNU2JiEgzaFDNzTvvvOOqOMQFusWGkZFbQtqxIs7v0sZ9gfS/BX56EdKWQl4mRCW6LxYREfF56nPjw+wjptzaqRggujMkjQQM2PyRe2MRERGfp+TGhzlWB3fXAppnGnC77eum2WC1ujcWERHxaUpufJhjxJS7a24Aeo2HoAjIOwAHNKGfiIi4jpIbH2ZPbg7lnaSkvNK9wQSGnh4ppY7FIiLiQkpufFjrVoG0aRUIwD5Papra/hWU5rs3FhER8VlKbnxc11g3L6B5pvYDIaYXVJZC6hfujkZERHyUkhsfZ2+acvuIKQCT6fSMxWqaEhERF1Fy4+O6eVLNDUDfm8DsD4c2wJHt7o5GRER8kJIbH+cxc93YhcVAj7G27ZQ57o1FRER8kpIbH2evudl/vIRKi4fML2PvWLz5Y6gsd28sIiLic5Tc+LiEyBBCAvwot1jJPHHS3eHYdBsNYe2g5Djs+d7d0YiIiI9xa3Lz008/MX78eBISEjCZTMybN6/Oc+bMmUO/fv0IDQ0lPj6eO++8k5ycHNcH66XMZhNdTjVNecRkfgB+/tBvkm1bHYtFRMTJ3JrcFBcX069fP9544416Hb9q1SomT57MXXfdxbZt2/jss89Yu3Ytd999t4sj9W6OEVOe0qkYTo+a2vMDFGS5NxYREfEpDVoV3NnGjRvHuHHj6n38mjVrSEpK4ve//z0AnTt35ne/+x0vvPCCq0L0CY4RU55ScwPQtjskng+ZP8OWj+GCP7g7IhER8RFe1edm2LBhZGZm8u2332IYBkeOHOHzzz/niiuuqPGcsrIyCgoKqjxaGo+suYGqc94YhntjERERn+FVyc2IESOYM2cON910E4GBgcTFxREZGVlrs9aMGTOIjIx0PBITE5sxYs9wZs2N4UlJxLkTIKAV5OyFzF/cHY2IiPgIr0putm/fzkMPPcSTTz7Jhg0bWLhwIfv37+fee++t8Zzp06eTn5/veGRmZjZjxJ4hqW0oZhMUlFZyrKjM3eGcFhQO515r2970gXtjERERn+FVyc2MGTMYMWIEjz76KH379mXMmDH861//4u233yYrq/pOqUFBQURERFR5tDRB/n50jA4FIO2oByygeSZ701TqXCjzsGYzERHxSl6V3JSUlGA2Vw3Zz88PwLOaWzyQx/a76Xg+RHeFimLYPs/d0YiIiA9wa3JTVFRESkoKKSkpAKSnp5OSkkJGRgZga1KaPHmy4/jx48fz5ZdfMmvWLPbt28eqVav4/e9/z5AhQ0hISHDHS/AaHjliCrSYpoiIOJ1bk5v169czYMAABgwYAMDDDz/MgAEDePLJJwHIyspyJDoAU6ZM4aWXXuL111+nd+/e3HjjjZxzzjl8+eWXbonfm9hrbjxmAc0z9ZsEJjNkrIHje90djYiIeDmT0cLacwoKCoiMjCQ/P79F9b/ZcOAE189aTUJkMKunX+rucM42Z6JtKYYL/gCjn3Z3NCIi4mEa8v7tVX1upPG6naq5OZxfSnFZpZujqYa9aSrlI7B4YHwiIuI1lNy0EJGhAbQNCwJg3zEPGzEF0GMshLaBomxIW+LuaERExIspuWlBup5aQHPvsUI3R1IN/0Doe7NtW3PeiIhIEyi5aUFOj5jywJobON00tes7KD7u3lhERMRrKblpQRxz3XjacHC7dsmQMBCslbDlE3dHIyIiXkrJTQviqLnxxOHgdvbam40faDFNERFpFCU3LUjXU8nN/pxiKi1WN0dTg97Xg38wHNsBhze6OxoREfFCSm5akPiIYEID/aiwGGTklrg7nOqFREGvq23bmrFYREQaQclNC2I2m+hiHzHlon43FqvBmrQcvko5xJq0HCzWRjQt2Zumtn4O5R6ahImIiMfyd3cA0ry6xYSReqiANBfMdbMwNYtnvt5OVn6pY198ZDBPjU9mbO/4+heUNBKiOkJeBuxcAH0nOj1WERHxXaq5aWFcNWJqYWoW983eWCWxAcjOL+W+2RtZmJpV/8LMZuhvX0xTc96IiEjDKLlpYVwxYspiNXjm6+1U1wBl3/fM19sb1kTVfxJggvSf4MT+pgcpIiIthpKbFqarYyK/IpyxZqphGHyyNuOsGpsqxwBZ+aWsTc+tf8FRHaHLKNt2yodNilFERFoW9blpYTq1CcXPbKKwrJJjhWXERgQ3uIzSCgtr9uWwePsRluw4SnZBzYnNmY4W1u84hwG3wb6lsGkOXPQYmP0aHKuIiLQ8Sm5amCB/PzpGh5J+vJi9R4vqndzkFJXx486jLNlxlJ/2HKOk3HJGmWbKKuueNyc2vIGJVM+rIDgSCg5C+nLoeknDzhcRkRZJyU0L1KVtK9KPFzN30yFMJhNDOkfjZzZVOcYwDNKOFbN4xxEWbz/ChowTVSYMjosI5tJesYxObseQpGhGv7Sc7PzSavvdmIC4yGCGdI5uWKABwdBnIqz7r23OGyU3IiJSD0puWpiFqVn8vC8HgM82HOSzDQcdw7VH92rHhgMnbAnNjqOkH686XPzchAhG92rHZcntODchApPpdEL01Phk7pu9ERNUSXBMZzz/6wSqXgbcZktudiyAklwIbWCCJCIiLY7JcEavUi9SUFBAZGQk+fn5REREuDucZmUfrl3TDQ8N9KvS3BTgZ2JY17Zc1iuWS3q1o31USJ3l/3qem7iIYJ6+uoHz3JzJMODNkXBkK1zxdxhyd+PKERERr9aQ92/V3LQQtQ3XtisptxAZ4s+lPdsxOrkdI7u3JTw4oN7XGNs7nsuS41ibnsN9szeSd7KCGdf14eKesY0P3GSy1d4sfMw2542SGxERqYOGgrcQa9Nzax2ubffGLQN56ab+XNEnvkGJjZ2f+VRtT3I7AEcTWJP0nQh+gZC1GbK2NL08ERHxaUpuWoj6DsPOKS53yvVGdGsLwMq9x5teWGg0nHOFbTtlTtPLExERn6bkpoWo7zDsBg/XrsHwrm0A2J5VwAlnJEwDbrd93fIJVJY1vTwREfFZSm5aiCGdo4mPDKam8UombItcNni4dg1iI4LpHhuGYcAaZzRNdb0YwhPg5AnY9W3TyxMREZ+l5KaF8DObeGp8MsBZCU6Th2vXwN40tcoZTVNmP+h/i2170+ymlyciIj5LyU0LMrZ3PLNuG0hcZNWmp7jIYGbdNrDxw7VrYG+aWp3mhJobOJ3c7F0C+QedU6aIiPgcDQVvYU4P187laGEpseHB1c5Q7AxDu7TBbIL048UczjtJQh3z5NSpTVfodAEcWAmbP4ILH3VOoCIi4lNUc9MC2YZrt+Ga/u0Z1rWNSxIbgMiQAPp2iAKc1DQFtjlvwNY0Za17PSsREWl5lNyIS43oZmuaclpyk3w1BIbDif2Qsdo5ZYqIiE9RciMuNaLrqU7FaTk4ZaWPwFbQ+zrbtjoWi4hINZTciEsN7NSaIH8zxwrL2Hu0yDmF2ue82TYPSgucU6aIiPgMJTfiUsEBfgxKag04sWmqwyBoew5UnoRtXzqnTBER8RlKbsTlhp/RNOUU9sU0QU1TIiJyFiU34nIXnJrM7+d9OVRanDTCqd/NYPKDg+vg6E7nlCkiIj5ByY24XO/2kUQE+1NYWknqYSf1kQmLhR5jbdspqr0REZHTlNyIy/mZTZzfxclDwuF009Tmj8FS4bxyRUTEqym5kWbh1HWm7LpfBq1iofgY7PnBeeWKiIhXU3IjzcI+md/6AycorbA4p1C/AFvfG1DHYhERcVByI82ia0wYseFBlFda2XDghPMKtjdN7f4eCo84r1wREfFaSm6kWZhMJtc0TcWcAx2GgGGBLR87r1wREfFaSm6k2TiSG2fNd2N35pw3zljiQUREvJqSG2k29n43Ww/mkX/SiaObzr0WAkLh+G7bvDciItKiKbmRZhMfGUKXtq2wGrYJ/ZwmOAKSJ9i2N33gvHJFRMQrKbmRZjX8VO3Namf2u4HTTVOpX0J5sXPLFhERr6LkRprVCGevM2XXaThEd4HyItj+lXPLFhERr6LkRprVsK5tMJlg79EijhSUOq9gkwn632rb1pw3IiItmpIbaVZRoYGcmxABwOo0JzdN9ZsEJjMcWAU5ac4tW0REvIaSG2l2p+e7cXLTVGR76HqpbTtljnPLFhERr6HkRpqdvd/N6r3HMZw9L429Y3HKh2B10jIPIiLiVZTcSLMbnBRNoJ+Zw/mlpB938simc8ZBSDQUZkHaj84tW0REvIKSG2l2IYF+DOgYBbhg1JR/EPS9ybatOW9ERFokJTfiFvZ+N06f7wZgwKlRUzu/hWInJ08iIuLxlNyIW9iXYlizLwer1cn9buL6QHx/sFbA1k+dW7aIiHg8JTfiFn07RBEW5E9eSQXbswqcfwF7x+KNH2gxTRGRFkbJjbhFgJ+ZoZ2jAVjliqapPjeAXxAc3QZZKc4vX0REPJaSG3Gb4af63ax0RXIT0hp6jbdta8ZiEZEWRcmNuI293826/bmUVbpgThp709TWz6DipPPLFxERj6TkRtzmnHbhtA0LpLTCyqaMPOdfoPNFEJkIpfmw8xvnly8iIh5JyY24jclkYlhXFw4JN5vPWExTc96IiLQUSm7ErUZ0tTVNOX0yP7v+t9i+7lsOJw645hoiIuJRlNyIW9kn89ucmUdRWaXzL9C6k615CgM2f+T88kVExOMouRG3SowOpWN0KJVWg7XpLqq9GXC77eumOWC1uuYaIiLiMdya3Pz000+MHz+ehIQETCYT8+bNq/OcsrIyHn/8cTp16kRQUBBJSUm8/fbbrg9WXMY+amrlHhclN72ugqBIyM+A/T+55hoiIuIx3JrcFBcX069fP9544416nzNx4kSWLFnCW2+9xa5du/joo48455xzXBiluNpwe6fiNBd0KgYICLFN6gea80ZEpAXwd+fFx40bx7hx4+p9/MKFC1m+fDn79u0jOto2u21SUpKLopPmMvxUp+Kd2YUcLyqjbViQ8y8y4DZY/xZsnw9XnLBN8iciIj7Jq/rczJ8/n0GDBvHiiy/Svn17evTowSOPPMLJkzVP0FZWVkZBQUGVh3iWNmFB9IwLB2C1q0ZNJQyA2HPBUgapX7jmGiIi4hG8KrnZt28fK1euJDU1lblz5/Lyyy/z+eefc//999d4zowZM4iMjHQ8EhMTmzFiqS/7qCmXzHcDYDKdnrFYTVMiIj7Nq5Ibq9WKyWRizpw5DBkyhCuuuIKXXnqJ9957r8bam+nTp5Ofn+94ZGZmNnPUUh8XnEpuVrmq3w1A34lgDoDDmyA71XXXERERt/Kq5CY+Pp727dsTGRnp2NerVy8Mw+DgwYPVnhMUFERERESVh3ieIZ2j8TebyMw9SUZOiWsu0qotnHOqj1fKHNdcQ0RE3M6rkpsRI0Zw+PBhioqKHPt2796N2WymQ4cOboxMmqpVkD/9E6MAF9fe2Oe82fwxVJYDYLEarEnL4auUQ6xJy8FiNVx3fRERcTm3JjdFRUWkpKSQkpICQHp6OikpKWRkZAC2JqXJkyc7jr/lllto06YNv/nNb9i+fTs//fQTjz76KHfeeSchISHueAniRMPtTVOu6ncD0PUSCI+Hk7mw+zsWpmZxwQs/Mum/P/PQxylM+u/PXPDCjyxMzXJdDCIi4lJuTW7Wr1/PgAEDGDBgAAAPP/wwAwYM4MknnwQgKyvLkegAhIWFsWjRIvLy8hg0aBC33nor48eP59VXX3VL/OJc9nWm1qTlYHVV7YmfP/SbBMDRn97ivtkbycovrXJIdn4p983eqARHRMRLmQzDaFF18AUFBURGRpKfn6/+Nx6mvNJKv2d+4GSFhe8eGkmveBfdn5w0eG0gFswML32VI0SfdYgJiIsMZuVjl+BnNrkmDhERqbeGvH97VZ8b8W2B/maGdLYlGi5tmmrTlYLYwfhh5Xq/FdUeYgBZ+aWsTc91XRwiIuISSm7Eo9jXmXLZZH6n7G0/AYAb/ZZhS2Wq996a/Rw84aLRWyIi4hJKbsSj2NeZ+mVfDhUW163gXXHO1RQZwXQ2H2GwaVeNxy1MzWbki0u57X+/8FXKIUorLC6LSTyTRtOJeB+3ri0l8mvJ8RG0Dg3gREkFmzPzGJR0dn8YZ+gUH8s31vO5yW8ZE/2Wsa6yZ5XnTUBkSAC94sNZsy+XlXuPs3LvcSKC/bmmf3smDkqkd/sITCb1x/FlC1OzeObr7VU6ncdHBvPU+GTG9o53Y2QiUhvV3IhHMZtNDDs1amrVXtc0TZVWWLh39gY+qRwFwJV+v9CK0zNc29OVmdf34aN7hrHi/y7m95d2JyEymILSSj74+QDjX1/JuFdW8PbKdHKLy2u8lj71V+VNP4+FqVkaTSfipTRaSjzO7J8P8Jd5qQzpHM2nvxvm1LKtVoMHP9rEN1uziAz256fQ/yOyZD9vVl7JdmtnjhJFZlg/nri6z1mfzC1Wg9Vpx/l0/UG+35ZNeaWt2SzAz8ToXu2YOCiRkd3b4u9n+8ygT/1VedPPw2I1uOCFH89KbOw0mk6k+TXk/VvNUuJx7Itobso4QUl5JaGBzvs1/fsPu/hmaxYBfib+PXkQkSlDYct+7vX/xnGMEZyAyfwCcHWVc/3MJkZ2j2Fk9xjySyqYv/kQn64/yNZD+XyXms13qdm0iwji+oEdaBcRzNPzt53VVdn+qX/WbQM97g3dley1IN7y81ibnltjYgNVR9PZaxpFxHOoWUo8TlKbUNpHhVBhMVi3/4TTyv10fSb/WpYGwMzr+nJ+6SrY8ulZx5kKsuDTybB9fo1lRYYGcPuwJL5+8AK+/f1IfjMiidahARwpKONfy9J4qprEBk6Py3rm6+0e3STjTBarwTNfb/eqn8fRwpoTm8YcJyLNS8mNeByTycTwU5+GVztpvpvVacf585dbAXjwkm5cPyAeFj5G9cPAT+1b+Cew1j06KjkhgqfGn8vPf76Uf906kH6JkbUe39Lm0GlILYiniA0PdupxItK81CwlHmlEt7Z8tuGgUxbR3Hu0iHs/2ECl1eCqvvH8YXQPOLASCg7XcpYBBYfgtUEQkQBB4RAUdurrqUdgeJXvg4LCuSIuHL+BoTycmU0JQRi1fH5oKZ/6vbEWZEjnaOIjg+vsc2OfdFJEPIuSG/FI9pqbbYcLOFFcTutWgY0qJ6eojDvfXUdBaSUDO0bx9xv7YTaboOhI/Qo4sc/2aIAxwLZTH+gLjRCKCKHYCKaIkNPfE8KQHZ0hJ+aMxCnijMTJnkid2ucfBF447PxYYRkfr8us17GeVAviZzZx30VdeXL+thqPeWp8sjoTi3goJTfikWIjgukeG8aeo0Ws2ZfDFX0a3tm0tMLCPR9sICO3hMToEP47eRDBAX62J8Pa1a+Q0c9AVEcoK7Q9yotObRec+lp0xnO2r0ZZISZrJQDhppOEc/L0+PIz7foJap4/sCqzf401RnUmRmfWOAWG2xYPdTGr1eDT9Zk8/+0OCkoraz3WKbUgVgscWG1LWsPaQafhYPZrdHGGYbBk51HAtiyIfWSc3fUD23tUB2gRqUrJjXisEd3asudoEav2Hm9wcmMYBv/3+RY2HDhBeLA/70wZTJuwoNMHdBpua24qyKL6fjcm2/PDH2zwm6TJMPhhy37+/NEawkwnacVJwk99DTu13SagjFv7tyE2sPx0slR+RqJkT5zKC22FWivh5Anbo6n8Q+qRGNmTo1qa4gJbVVubtPdoIX/+MpW1+219aM5NiODqfgnM/G4ncPZP26CJtSDb59v6T53ZzBiRAGNfgOSraz6vFt9vO8Ly3ccI8DOx4MELyCkq52hhKdsO5/Ofn9L5fvsR/lRURtszf6dExGMouRGPNbxrG95dvb9R60z9c9Fu5m8+jL/ZxL9vO49useFVDzD72d78Pp2Mre7gzLfcU2+yY2c27tO/ycTl/Tpj9Qvmma+3sz+/1FF8bHgQ/mYTh/NL+c8GP16dNIDLkmupRbJaTyc9tdUaVZscFVatcao81X+k8qTtUXy04a+t6gutkvhYA8PILPZjb67BDdYQrgoMpU+X9vTtmohfyB7Ov7CSDzaeIKPYjyJszXNFRghdOsQ3vhZk+/xT9/BXKZN9xNvE9xuc4JSUV/Ls17bmqHsu7EKPduFw6hZd1TeB1Wk5pB4q4B8/7GbGdX0aF7eIuJQm8ROPVVBaQf9nfsBqwOo/XUJCVEi9zvtiw0H++NlmAF68vi8TByfWfHC1n/rb2xKbRn7qP5PFarA2PZejhaXEhtuaXorLK5k6ZyMr9hzHZIInrkzmzgs6N/ladaosP5UAFTQsMaousTKcu+6X1RyAudYao1/VMAWFQUAofHU/FNfU6fxU7du0rQ1KUl9YuJNZy9JoHxXC4ocvIiSw6rnr9udy45trMJlgwYMXcG5C7aPjRMQ5GvL+reRGPNqEN1aRkpnH327oy42DaklSTvl5Xw63v/ULFRaD+0Z15bGxPes8x9n9NeqjwmLlya+28dHaDADuGNaJJ65Kdsxu7NEMAypOQlkhBfm5vLt0C2t2HCCMk8QHV3Bjn0h6t/XDdFZyVE2NU0Wx6+Nt1wciO5xOihyJUthZTXEZRWZum72dPEsQf791BJf36VBtkQ9+tImvNx9mSOdoPrnnfK0x5uGq+5ChzuDeR8lNLZTceJe/fb+TN5amMaF/Ai/fPKDWY/cdK+Laf60m/2QFV/aJ57VJA2wjozyUYRj856d9zDjVF+WSnrG8NmkArYI8v7XYMAzmbz7MXxds53iRbW2tW4Z25LGxPYkMCah3OSdLy7n2le/JP3GC2we25v5h7WqoNSo4Y9+p5CgvEwrqNxKr0Rz9k6omRSWmUObvKCDPGswlfbvQIzG+auftwPBfnRcOfvX/uYjzeNOyHz7BhR8WldzUQsmNd1m99zi3/O8XYsKDWPvnS2v8hHyiuJxr/7WK/Tkl9E+M4uN7zj89MsrDfbs1iz98kkJZpZXk+AjenjKYuEjPGRb9axk5Jfzlq1R+2n0MgO6xYTx/XR8GN3IF9592H2Py22sxm2De1BH07RBVvxPTV8B7V9V93IWP2mpuzkyMys/ctn0tLsyjrDiPMEoJNNU+wqtR/IPP7rTt2K4pKaq+hslnEyUnvzHWtOyH/b+Ipy374fVc0Ln/TEpuaqHkxruUVljo98wPlFVaWfSHC+neLvysY8oqLdz+v7Ws3Z9Lh9YhzL1/BDHh3jWKZWPGCe5+bz05xeXERQTz9pTBJCe45/ezpir8CouVt1am8/Li3ZRWWAn0N/Pgxd343UVdCfRvWnPaQx9v4quUwyTHRzD/gRH1a56zWuDl3nWPeKtHn5uiskou/ccyjhSU8dCl3fnDxZ1Oj1arISmqOFnAZ6u3Q1kRg+L86dHa9KvjTh1rKWvUz6RWfkF11xRVNwKuSjIVYfvev3FzSDnd9vkYCx/DdMYboxGRgKmRb4xa/LSZ1dS5355KNqJz/69p4UzxGcEBfgxKas2qvTms2nv8rOTGMAz+9MVW1u7PJTzIn7enDPa6xAZgYMfWzJs6ginvrCXtWDE3vrma128ZyMU9Y5s1jpqq8CcPS+KrlEPszLYNTR/WpQ3PXdubLjFhTrnuE1cls2zXMbZnFfDOqv3cfWGXuk9y4oi3Vxbv5khBGR2jQ7lvVFfw97NNnNiq5kUxA4BWbQ7x0McphBzxY+nto6qvcXN05P515+wzvy/6VcfuGmqY7CPeLGVQUgYlTliexC+o7pqi+tYw+Tfyb2/7fIxPJ2NgVJkSyig4DJ9OxlTPN0ar1eBQ3kn2HitiyY4jWvy0uVgtdSxnY7ItZ9PzSpf3Z7RTzY14vH8t28uLC3dxWXI7/jt5UJXnXlm8h38u3o2f2cS7vxnMyO4xborSOfJLKrhvzgZWp+VgNsEzV5/L7cOSmuXaNVXhnykqNIDHr+jFDed1cHon2k/WZfDYF1sJCfDjhz9cSGJ0aP1ObOKIt13ZhVzx6gosVoO3pwzikp71nOARW3J945trWH/gBNcOaM8/b+pf73MbxVJRTVJUWGsNU/XJVOHpRMmZ/AIbmBSFQ0Aryr68n8Cy3Gon4bYaUB4SS/C9P4LJDIaVskoLB3NLyMgt5sDxIjJyisjIKebgiWLKK62YME49cGybz9jHGd//8fLuXNQ9xjYC0DAA44yvv9pnWH/1PNXsq085v36+urKruzYNO8exj1rOqa0cai77zHMKD8Pu7+v+/bhjAXQeWd/fprOoWaoWSm68z+bMPK55YxXhwf5seuIyR5PFVym2T80Az1/bh1uGdnRjlM5TXmnlz3O38vmGgwD89oLOTL+il0urzuuqwgcICfBj+aOjiI1wTX8gwzC46T8/szY9l4vPieHtKYPrn0A1sq/GmdesLnmuj60H87n6jZUYBnxx33DO69S6wWW4haWihpoi+wSSNSRF1TXXVZ5096sRb3D9W9DnhkafrmYp8Sm920cSHuRHYWkl/1qWxuCkaEwmePSzLYBtojVfSWzANt3/327oS+e2rfjb97v438p0DuSW8MrN/QkNdM2fbF0rdwOcrLCQdqzYZcmNyWTi+Wv7cMUrK1i66xjfbM3iqr4J9TvZ7NeoT4TzUg6xNj2X4AAzT41PbvD5AH06RHLjeR34dP1Bnv16G3PvH+HRo/Qc/AIgpLXt0VSWytO1RQ1JisqLKD2WTnDxwbovYZiwnKpvsdXHACYzJrMZk8mE2WTCbD71PSYMk4n8k5VUGqeOddThgBUzBmA2mWkXGYzJZAZMp2bcPvX1rH3mqs87jqvtnJr21X3O0cIydh4porTC6og3KMCP5PhI4iJDzo6nruvU+BqqK4eG/SxOZMDmOXX/ntR32RsnUHIjHm/R9mzKLbZ/Ty8t2g3Y/p4MA8ac244/1WcuGy9jMpmYenE3OrQO4dHPtrBo+xFu/s/P/O+OQU5bYLK4rJKNGSdYl57Lt6lZ9TrH1St3d4sN4/6Lu/Ly4j088/V2RnaPadDQ8oYoKK3guW9sw/AfvKQ7HVrXsxmsGo+MOYdvt2az+WA+czcd4vrzqp8fxxdU2+Hcz7/eiVJhaQXbDxew7dSjtHwZb/BknefNjH2RxIGX0y0mjG6xYcSEB9Vas2cCfj7V1ArV9waZdetAxjVi3TpXq3GUVwWQ5oGjvKwWSF9ad+f+TsObLSQlN+LRavojtzemXtEn3js+JTfSNf3bkxAVwj3vr2fLwXyufWM1b08ZTLfYsAZPSpZTVMa6/SdYtz+Xdftz2Xa4AIu1Ya3SzbFy932juvL15sOkHSvmhYU7ef5a1yxx8NIPuzleVEaXtq347cimzRAdGx7MA5d0Y+Z3O3lh4U7G9I4jzAvmK2qohs4ZY1uPq+BUMpPPtsMFHMgpqXKMmS4cDoomjlyq+xW2GpBNGy65fALDujesg/3Y3vHMum3gWTHblZRbGlRec7BYDZ75enttXXN55uvtXJYc5zmjvFy5nE0jqc+NeKz69AOJbyFDOfcfL+Y3764j/Xgxwf5mQgP9yS0pdzz/6zcYwzA4eOKkI5FZm55L2rGzZwNuHxXCkM7RnNepNS8v3k1OUXlNn7uaddjsL/tyuOk/PwPw2b3DGj2HTk22Hy7gqtdWYDXgg7uGOKUjelmlhcv/+RMHckq4f1RX/s/HahTrmjPm2Wt60zYs8FSNTD6phws4Vlj9MPj2USEkJ0RwbkIEveIiWDL3f8ys/BtAlQTHnnv/OeD/eO7Pf270796va5s2ZOTy9+930zo0gCV/HEV0Kw8ZDg+sScth0n9/rvO4j+4+v9GjvFw2Y7OLl7NRnxvxCfXpB9JShnImtW3Fl/cN58Y317D3WBGlleVVns/OL+Xe2Ru5eXAiJeUW1u2v/mfXo10Yg5OiGdI5msFJ0VXW62obFsh9szfW9LmraSt3N9DQLm24aVAin6zP5M9fbuWb349s8lw6dlarwRNfpWI14Mo+8U4bYRfk78dfrkzm7vfX878V6dw8uCMd2zS+qcuT1FWbAPDEV6lnPWcyQZe2rTg3IZJzEyIcX1v/KpkwuIv7P6zgyYD3SSDXsT+bNjxbcTsTbryzSb97fmZTlf8Rg5Jas2BzFjuzC3numx38Y2K/RpftbPVt+n3txz3sPVZEr7hwesSFExFcv+Zbl87YnHy1bbh3My9nUx0lN+Kx6vtH7up+IJ4iIiSAorKKap+zv8F8vO70cgT+ZhO920c6EplBnVqf9aZyppqq8OPcNFX99Ct6smTnEfYcLeLfy9N48NLuTin3840H2XDgBKGBfvzlql5OKdNudK9YLujWlpV7j/P8tzt48/bznFq+u9Q1Z4xdUttQhia1oXf7CJITIukVH16vTvBje8fDLfdy4/wRJBZtJpY8jhJFZlg/nrixj9N/9wL8zDx/XR+un7WaLzYe5Prz2jO8a1unXqOx6tv0uzoth9VpOY7vO7QOoWdcBL3iw+kZF0HP+HCS2rSqkhTWVPuWnV/KfbM3OqcvTyM79zubkhvxWPX9I2+OfiCeYG16LtkFdc92e/3A9lw/sAP9O0Y1eHTV2N7xXJYc5xGLDEaFBvLEVck89HEKry3dy5V945s8aWB+SQUzT63l9dCl3YmPrN9K8/VlMpl44qpkrnh1BQu3ZbN673GGd/OMN82GKK+0sjHjBCv2HGPFnuNsOZhfr/P+MLoH1/Rv36hrnv7dO69ZfvcGdmzNbUM78cHPB3h8birfPTTSI5ZsGdI5msiQAPJPVv9BBmzzTd00OJHd2YXszC4kK7+UgydOcvDESRbvOOI4LsjfzDlx4fSMC6dHu3D+tSzN5X15PGWRUiU34rGGdI4mPjKY7PzSWvuBDOns3P4Ynqq+NVQX9ohp0hvqr6vw3enqfgl8vuEgK/Yc5/G5qXx499AmTR74tx92kltcTvfYMO68oGmdiGtyTlw4tw3tyHtrDvDsgu0sePCCZl/tvaFvMIZhsO94MSt225KZNftyGtXZtqkfNJr7d+/Rsefw/bZs0o8X86+le3n48nOa7do12ZFVQHFZ9Wub2e/gzOuq1mbllZSzM7uQnVkF7MwuZEd2IbuzCzlZYWHLwfx6Jaf2GZtX7DnGqHMaNzO6Jy1SquRGPJaf2cRT45M9ph+Iu7XEmiyTycRzE/pw+cvLWbMvhy82HuKGRg6z3nownzm/ZAC2zq8BLkw4/nBZD77afJid2YV8tC6T28/v5LJr/Vp932DySspZtTfHUTtzKK/qRHxtwwK5oFtbRnaPYVjXNlw/a7XPfdCICA7g6avP5f45G5m1PI3x/RKqXb+uueSfrOD+ORuptBr0aR/BscJysgvqbiKOCg3k/C5tOL/L6cTQYjXIyC1hZ1YBO7ILWbrzCFsPFdQZw5R31hEe5E9sRBDtIoJpFxFs2w4PPvW9bX9MeFCVmq5mafJqAI2WEo/nSZ8G3Mk+eqyuNxhfHD02a1kaLyzc2ejRLVarwbWzVttmu+6fwCs3D3BRpKe9t3o/T83fRuvQAJY9cjGRoa5fybuuEU0PX9aDcouVn/YcZ8vBPM787x/oZ2Zw59aM7B7DyO5t6RUXUWWahYU1zBnj7StsG4bBb99bz5KdRxmc1JpP7hnmluklDMPg7vc3sHjHETq0DuGbB0cSFuzvtCae+o7Caoio0ADahQcTEx7IhgN5nKyovrbPWf+btPxCLZTceCdPacd1N199g6lLhcXK+NdWsjO7kOsGtuelif0bdP6Hv2Tw57lbCQvy58c/XuSyWZbPVGmxcsWrK9h9pIjfjEjiqfHnuvR69Zk64dd6tAtzJDNDO7chJLD2Pie++kHjUN5JLntpOSXlFmZc14dJQ5p/xvM3l6cx87udBPqZ+eK+4fTpEOnU8uv74ei7h0ZyvKicowWlHC0s40hBKUcKyjhSWMpR+3ZBKWWV1gbH0JTh66DkplZKbsTb+eobTF02ZZzgulmrMQyY89uhjKhnv6Lc4nIu+ccy8koqeOKqZO5yUV+b6qzcc5zb3voFP7OJhQ+NdGmTR30/mQ/v2oZrB7RnZPeY6lcxr4OvftD434p9/L9vdhAR7M+SP44iJryRK5w3ws/7crj1f79gsRo8d21vbh3qmmZMZ304MgyDgpOVHCks5UhBKQtTsx1NvrV55eb+je5wDg17/27eXm4i0mRje8ez8rFL+Oju83nl5v58dPf5rHzsEp9ObAAGdGzt6Lvy+NytlNZQBf5rLy7cSV5JBT3jwrljWPP1fQG4oHtbLktuh8Vq8NdvduDKz5L17XB+0+BEbhyU2KjEBk53+r2mf3uGdW3jE4kNwJThSfRuH0FBaSV/XbC92a57tKCUBz/ahMVqcN2A9tziwloj+3QPv773cZHBDar1NZlMRIYG0KNdOCO7x9R7Dbjm7A+oDsUiXsiTRjQ1p0fH2Ea37M8p4fUf9/LImNpHt2zMOOGY++evE3o3+6glgMev6MWyXUf5afcxlu46yiU9nb94YGFpBT/uOFqvY32pw7kz+fuZmXldX65+fSXzNx/muoHtGz1qqL4qLVYe/GgTxwrL6NEujP93be8mjQasD1dM9+CJI1tVcyMiXiM8OIBnrrb1XXlzeRq7jxTWeKzFavDEPNusudcP7OD0JRzqK6ltK8ew878u2EF5I/oq1KTSYuXDXzK4+O/L+Grz4VqPNWFrvvS2EU3NqXf7SH4zwnavnvgqlZMuXnvq7z/s5pf0XFoF+jHrtvMaPC9VYzm79s0+shVON3HZuWtkq5IbEfEqY86NY3SvdlRaDaZ/uRVrDYt/zvnlANsOFxAe7M/0K9y7ztMDF3ejbVgQ6ceLeW/1fqeU+dPuY1z56kr+PHcrx4vK6dy2Ffdd1BUTnvMG440evqwHCZHBZOae5OUlu112nUXbj/Dm8jQAXryhH12bOEGluzmryctZ1KFYRLzO4VOjW4rLLdV2wDxeVMbFf19GYWklz15zLpOHJbkn0DN8uj6T//t8C+FB/ix9dBRtwxrXYXXPkUKe+3YHy3YdAyAyJIBpo7tz69BOBPqbW2yHc2davP0Iv31/PX5mE18/cAHJCc59r8jIKeGq11ZQUFrZLCPpmpMrO5xrtFQtlNyI+IZ3VqXzzNfbCQ/2Z8nDVYd3//HTzXyx8SC920fw1dQLPKK2wmo1uOaNVWw9lM+kIYnMuK5vg84/XlTGPxft5uN1mVisBgF+JiYPS+LBS7oRFVp13h9fHdHUnO6bvYHvUrPpnxjFF/cNd9rPr7TCwg1vrib1UAEDO0bx8T3DnLYorK/TquAi4vMmD0ti3qZDbD6Yz9Pzt3H7sCSOFpZyoricLzYeBOCv1/T2mDd186l+CTe8uYaP12Vy69BO9G5f91wmpRUW3lm1nzeW7qXo1LT8Y85tx5/G9aJz21bVntNSO5w709NXn8vKPcdJycxjzi8HnFb798zX20k9VEB0q0Bev2WgEhsXUc2NiHitbYfzGf/aSqrrdjOiaxvm3H1+8wdVh99/tIn5mw8zJCmaT353fo2jYwzDYMGWLGZ+t9OxNELv9hH85crkKtPsi+t8sGY/T3y1jbAgfxY/fFGjh8/bfbHhIH/8bDMmE7x/5xBGdo9xUqQtg+a5EZEWITO3pNrEBmB1Wg4LU7OaN6B6+NO4ngQHmFm7P5evNx9mTVoOX6UcYk1aDpZTL2Zjxgmun7WaBz/axKG8k8RFBPPSxH7Mn3qBEptmdOvQTgzoGEVRWSVPz9/WpLJ2Zhfw+LytAEy7tIcSGxdTzY2IeKW6lhvw5LW2Xl68m5cX78FsokpyFhMeRKfoUNYfOAFASIAf917Ulbsv7Nxsw4Slqp3ZBVz16koqrQb/nTyIy5IbPk9RYWkFV7++ivTjxVzYI4Z3pwx2y/pV3k41NyLi89am59a6jpIBZOWXsjY9t/mCqid7X5lf1zodKyxzJDY3nteBZY+O4qHR3ZXYuFHPuAh+O7ILAE9+lero91RfhmHw2BdbSD9eTEJkMC/f1F+JTTNQciMiXqm+yw3U97jmYrEazPxuZ63HtA0LZOb1fWnXDAt8St0eurQ7idEhZOWX8tIPDZv75u1V+/l2azYBfibeuHVgg1e0l8ZRciMiXqm+ywh42nIDddU4ARwvKvfIGqeWKiTQj+cm9AHg3dXpbD2YX6/zNhzIZca3OwD4y5XJDOjY2mUxSlVKbkTEK9nXs6mpgt9Tlxvw1hqnlu7CHjFc0z8BqwF/+nILlZbal9E4XlTG1DmbqLQajO+XwORmXrS1pVNyIyJeyRPXs6kPb61xEnjiqmQiQwLYdriAd2tZRsNiNZj2cQrZBaV0jWnFjOv6uHxBTKlKyY2IeC1PW8+mPry1xkmgbVgQfz61Ttk/ftjNwRMl1R73yuLdrNx7nJAA24KYYUHqEN7c9BMXEa82tnc8lyXHec1yA/Yap/tmb8SEbVSXnSfXOInNxEGJfLHxEGvTc3nyq228dcegKrUyS3cd5dUf9wIw8/o+9GgX7q5QWzTNcyMi4gZa4NJ77T1axLhXfqLCYvD6pAG0CQviaGEpJhM8MS+V/JOV3HZ+R/7fqU7I4hxaOLMWSm5ExFNogUvv9dKi3by65OyJGAE6tQnlhz9cSJC/n3uC81FaOFNExAtogUvv1S2m+okYAQ7klLB051HVwLmROhSLiIg0gMVqMKOWiRhN2Fb/ttS08Jm4nJIbERGRBvDmpT9aCiU3IiIiDaCJGD2fkhsREZEG0ESMnk/JjYiISANoIkbPp+RGRESkAbx16Y+WRMmNiIhIA3nj0h8tiea5ERERaQRvW/qjJXFrzc1PP/3E+PHjSUhIwGQyMW/evHqfu2rVKvz9/enfv7/L4hMREamNfSLGa/q3Z1jXNkpsPIRbk5vi4mL69evHG2+80aDz8vLymDx5MpdeeqmLIhMRERFv5dZmqXHjxjFu3LgGn3fvvfdyyy234Ofn16DaHhEREfF9Xteh+J133mHfvn089dRT9Tq+rKyMgoKCKg8RERHxXV6V3OzZs4c//elPzJ49G3//+lU6zZgxg8jISMcjMTHRxVGKiIiIO3lNcmOxWLjlllt45pln6NGjR73Pmz59Ovn5+Y5HZmamC6MUERERd/OaoeCFhYWsX7+eTZs28cADDwBgtVoxDAN/f39++OEHLrnkkrPOCwoKIigoqLnDFRERETfxmuQmIiKCrVu3Vtn3r3/9ix9//JHPP/+czp07uykyERER8SRuTW6KiorYu3ev4/v09HRSUlKIjo6mY8eOTJ8+nUOHDvH+++9jNpvp3bt3lfNjY2MJDg4+a7+IiIi0XG5NbtavX8/FF1/s+P7hhx8G4I477uDdd98lKyuLjIwMd4UnIiIiXshkGIbh7iCaU35+PlFRUWRmZhIREeHucERERKQeCgoKSExMJC8vj8jIyFqP9Zo+N85SWFgIoCHhIiIiXqiwsLDO5KbF1dxYrVYOHz5MeHg4JpNz1gCxZ5O+Whvk668P9Bp9ga+/PtBr9AW+/vrAda/RMAwKCwtJSEjAbK59JpsWV3NjNpvp0KGDS8qOiIjw2V9W8P3XB3qNvsDXXx/oNfoCX3994JrXWFeNjZ3XTOInIiIiUh9KbkRERMSnKLlxgqCgIJ566imfnQnZ118f6DX6Al9/faDX6At8/fWBZ7zGFtehWERERHybam5ERETEpyi5EREREZ+i5EZERER8ipIbERER8SlKbprojTfeICkpieDgYIYOHcratWvdHVKjzZgxg8GDBxMeHk5sbCwTJkxg165dVY4ZNWoUJpOpyuPee+91U8QN8/TTT58Ve8+ePR3Pl5aWMnXqVNq0aUNYWBjXX389R44ccWPEDZeUlHTWazSZTEydOhXwzvv3008/MX78eBISEjCZTMybN6/K84Zh8OSTTxIfH09ISAijR49mz549VY7Jzc3l1ltvJSIigqioKO666y6Kioqa8VXUrLbXV1FRwWOPPUafPn1o1aoVCQkJTJ48mcOHD1cpo7r7PnPmzGZ+JTWr6x5OmTLlrPjHjh1b5RhPvodQ92us7u/SZDLxt7/9zXGMJ9/H+rw/1Od/aEZGBldeeSWhoaHExsby6KOPUllZ6fR4ldw0wSeffMLDDz/MU089xcaNG+nXrx9jxozh6NGj7g6tUZYvX87UqVP5+eefWbRoERUVFVx++eUUFxdXOe7uu+8mKyvL8XjxxRfdFHHDnXvuuVViX7lypeO5P/zhD3z99dd89tlnLF++nMOHD3Pddde5MdqGW7duXZXXt2jRIgBuvPFGxzHedv+Ki4vp168fb7zxRrXPv/jii7z66qu8+eab/PLLL7Rq1YoxY8ZQWlrqOObWW29l27ZtLFq0iAULFvDTTz9xzz33NNdLqFVtr6+kpISNGzfyxBNPsHHjRr788kt27drF1Vdffdaxzz77bJX7+uCDDzZH+PVS1z0EGDt2bJX4P/rooyrPe/I9hLpf45mvLSsri7fffhuTycT1119f5ThPvY/1eX+o63+oxWLhyiuvpLy8nNWrV/Pee+/x7rvv8uSTTzo/YEMabciQIcbUqVMd31ssFiMhIcGYMWOGG6NynqNHjxqAsXz5cse+iy66yHjooYfcF1QTPPXUU0a/fv2qfS4vL88ICAgwPvvsM8e+HTt2GICxZs2aZorQ+R566CGja9euhtVqNQzDu++fYRgGYMydO9fxvdVqNeLi4oy//e1vjn15eXlGUFCQ8dFHHxmGYRjbt283AGPdunWOY7777jvDZDIZhw4darbY6+PXr686a9euNQDjwIEDjn2dOnUy/vnPf7o2OCep7jXecccdxjXXXFPjOd50Dw2jfvfxmmuuMS655JIq+7zpPv76/aE+/0O//fZbw2w2G9nZ2Y5jZs2aZURERBhlZWVOjU81N41UXl7Ohg0bGD16tGOf2Wxm9OjRrFmzxo2ROU9+fj4A0dHRVfbPmTOHtm3b0rt3b6ZPn05JSYk7wmuUPXv2kJCQQJcuXbj11lvJyMgAYMOGDVRUVFS5nz179qRjx45eez/Ly8uZPXs2d955Z5VFYr35/v1aeno62dnZVe5bZGQkQ4cOddy3NWvWEBUVxaBBgxzHjB49GrPZzC+//NLsMTdVfn4+JpOJqKioKvtnzpxJmzZtGDBgAH/7299cUtXvSsuWLSM2NpZzzjmH++67j5ycHMdzvnYPjxw5wjfffMNdd9111nPech9//f5Qn/+ha9asoU+fPrRr185xzJgxYygoKGDbtm1Oja/FLZzpLMePH8disVS5SQDt2rVj586dborKeaxWK9OmTWPEiBH07t3bsf+WW26hU6dOJCQksGXLFh577DF27drFl19+6cZo62fo0KG8++67nHPOOWRlZfHMM88wcuRIUlNTyc7OJjAw8Kw3jHbt2pGdne2egJto3rx55OXlMWXKFMc+b75/1bHfm+r+Du3PZWdnExsbW+V5f39/oqOjve7elpaW8thjjzFp0qQqCxL+/ve/Z+DAgURHR7N69WqmT59OVlYWL730khujrb+xY8dy3XXX0blzZ9LS0vjzn//MuHHjWLNmDX5+fj51DwHee+89wsPDz2r29pb7WN37Q33+h2ZnZ1f7t2p/zpmU3Ei1pk6dSmpqapU+KUCVNu4+ffoQHx/PpZdeSlpaGl27dm3uMBtk3Lhxju2+ffsydOhQOnXqxKeffkpISIgbI3ONt956i3HjxpGQkODY5833r6WrqKhg4sSJGIbBrFmzqjz38MMPO7b79u1LYGAgv/vd75gxY4ZXTPN/8803O7b79OlD37596dq1K8uWLePSSy91Y2Su8fbbb3PrrbcSHBxcZb+33Mea3h88iZqlGqlt27b4+fmd1RP8yJEjxMXFuSkq53jggQdYsGABS5cupUOHDrUeO3ToUAD27t3bHKE5VVRUFD169GDv3r3ExcVRXl5OXl5elWO89X4eOHCAxYsX89vf/rbW47z5/gGOe1Pb32FcXNxZnfwrKyvJzc31mntrT2wOHDjAokWLqtTaVGfo0KFUVlayf//+5gnQybp06ULbtm0dv5e+cA/tVqxYwa5du+r82wTPvI81vT/U539oXFxctX+r9uecSclNIwUGBnLeeeexZMkSxz6r1cqSJUsYNmyYGyNrPMMweOCBB5g7dy4//vgjnTt3rvOclJQUAOLj410cnfMVFRWRlpZGfHw85513HgEBAVXu565du8jIyPDK+/nOO+8QGxvLlVdeWetx3nz/ADp37kxcXFyV+1ZQUMAvv/ziuG/Dhg0jLy+PDRs2OI758ccfsVqtjuTOk9kTmz179rB48WLatGlT5zkpKSmYzeazmnK8xcGDB8nJyXH8Xnr7PTzTW2+9xXnnnUe/fv3qPNaT7mNd7w/1+R86bNgwtm7dWiVRtSfrycnJTg9YGunjjz82goKCjHfffdfYvn27cc899xhRUVFVeoJ7k/vuu8+IjIw0li1bZmRlZTkeJSUlhmEYxt69e41nn33WWL9+vZGenm589dVXRpcuXYwLL7zQzZHXzx//+Edj2bJlRnp6urFq1Spj9OjRRtu2bY2jR48ahmEY9957r9GxY0fjxx9/NNavX28MGzbMGDZsmJujbjiLxWJ07NjReOyxx6rs99b7V1hYaGzatMnYtGmTARgvvfSSsWnTJsdooZkzZxpRUVHGV199ZWzZssW45pprjM6dOxsnT550lDF27FhjwIABxi+//GKsXLnS6N69uzFp0iR3vaQqant95eXlxtVXX2106NDBSElJqfJ3aR9dsnr1auOf//ynkZKSYqSlpRmzZ882YmJijMmTJ7v5lZ1W22ssLCw0HnnkEWPNmjVGenq6sXjxYmPgwIFG9+7djdLSUkcZnnwPDaPu31PDMIz8/HwjNDTUmDVr1lnne/p9rOv9wTDq/h9aWVlp9O7d27j88suNlJQUY+HChUZMTIwxffp0p8er5KaJXnvtNaNjx45GYGCgMWTIEOPnn392d0iNBlT7eOeddwzDMIyMjAzjwgsvNKKjo42goCCjW7duxqOPPmrk5+e7N/B6uummm4z4+HgjMDDQaN++vXHTTTcZe/fudTx/8uRJ4/777zdat25thIaGGtdee62RlZXlxogb5/vvvzcAY9euXVX2e+v9W7p0abW/l3fccYdhGLbh4E888YTRrl07IygoyLj00kvPeu05OTnGpEmTjLCwMCMiIsL4zW9+YxQWFrrh1ZyttteXnp5e49/l0qVLDcMwjA0bNhhDhw41IiMjjeDgYKNXr17G888/XyUxcLfaXmNJSYlx+eWXGzExMUZAQIDRqVMn4+677z7rQ6In30PDqPv31DAM49///rcREhJi5OXlnXW+p9/Hut4fDKN+/0P3799vjBs3zggJCTHatm1r/PGPfzQqKiqcHq/pVNAiIiIiPkF9bkRERMSnKLkRERERn6LkRkRERHyKkhsRERHxKUpuRERExKcouRERERGfouRGREREfIqSGxEREfEpSm5EpMVJSkri5ZdfdncYIuIiSm5ExKWmTJnChAkTABg1ahTTpk1rtmu/++67REVFnbV/3bp13HPPPc0Wh4g0L393ByAi0lDl5eUEBgY2+vyYmBgnRiMinkY1NyLSLKZMmcLy5ct55ZVXMJlMmEwm9u/fD0Bqairjxo0jLCyMdu3acfvtt3P8+HHHuaNGjeKBBx5g2rRptG3bljFjxgDw0ksv0adPH1q1akViYiL3338/RUVFACxbtozf/OY35OfnO6739NNPA2c3S2VkZHDNNdcQFhZGREQEEydO5MiRI47nn376afr3788HH3xAUlISkZGR3HzzzRQWFrr2hyYijaLkRkSaxSuvvMKwYcO4++67ycrKIisri8TERPLy8rjkkksYMGAA69evZ+HChRw5coSJEydWOf+9994jMDCQVatW8eabbwJgNpt59dVX2bZtG++99x4//vgj//d//wfA8OHDefnll4mIiHBc75FHHjkrLqvVyjXXXENubi7Lly9n0aJF7Nu3j5tuuqnKcWlpacybN48FCxawYMECli9fzsyZM1300xKRplCzlIg0i8jISAIDAwkNDSUuLs6x//XXX2fAgAE8//zzjn1vv/02iYmJ7N69mx49egDQvXt3XnzxxSplntl/Jykpif/3//4f9957L//6178IDAwkMjISk8lU5Xq/tmTJErZu3Up6ejqJiYkAvP/++5x77rmsW7eOwYMHA7Yk6N133yU8PByA22+/nSVLlvDcc8817QcjIk6nmhsRcavNmzezdOlSwsLCHI+ePXsCttoSu/POO++scxcvXsyll15K+/btCQ8P5/bbbycnJ4eSkpJ6X3/Hjh0kJiY6EhuA5ORkoqKi2LFjh2NfUlKSI7EBiI+P5+jRow16rSLSPFRzIyJuVVRUxPjx43nhhRfOei4+Pt6x3apVqyrP7d+/n6uuuor77ruP5557jujoaFauXMldd91FeXk5oaGhTo0zICCgyvcmkwmr1erUa4iIcyi5EZFmExgYiMViqbJv4MCBfPHFFyQlJeHvX/9/SRs2bMBqtfKPf/wDs9lWCf3pp5/Web1f69WrF5mZmWRmZjpqb7Zv305eXh7Jycn1jkdEPIeapUSk2SQlJfHLL7+wf/9+jh8/jtVqZerUqeTm5jJp0iTWrVtHWloa33//Pb/5zW9qTUy6detGRUUFr732Gvv27eODDz5wdDQ+83pFRUUsWbKE48ePV9tcNXr0aPr06cOtt97Kxo0bWbt2LZMnT+aiiy5i0KBBTv8ZiIjrKbkRkWbzyCOP4OfnR3JyMjExMWRkZJCQkMCqVauwWCxcfvnl9OnTh2nTphEVFeWokalOv379eOmll3jhhRfo3bs3c+bMYcaMGVWOGT58OPfeey833XQTMTExZ3VIBlvz0ldffUXr1q258MILGT16NF26dOGTTz5x+usXkeZhMgzDcHcQIiIiIs6imhsRERHxKUpuRERExKcouRERERGfouRGREREfIqSGxEREfEpSm5ERETEpyi5EREREZ+i5EZERER8ipIbERER8SlKbkRERMSnKLkRERERn/L/AcM1TmZA4QPSAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"train_its, train_losses = zip(*metrics.train_losses)\n",
"val_its, val_losses = zip(*metrics.val_losses)\n",
"plt.plot(train_its, train_losses, '-o')\n",
"plt.plot(val_its, val_losses, '-o')\n",
"plt.xlabel(\"Iteration\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.legend(['Train', \"Valid\"]);"
]
},
{
"cell_type": "markdown",
"id": "b28f216c",
"metadata": {},
"source": [
"### Evaluate\n",
"\n",
"The training and validation loss are only part of the story. For HellaSwag, we ultimately care about how good the model is at answering questions. To asses this, let's generate the actual `ending1`, `ending2`, `ending3`, or `ending4` responses with the fine-tuned model and measure the accuracy.\n",
"\n",
"First, let's split the last word off of each output in the test set to create a prompt without the answer."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "d96e4dcf",
"metadata": {},
"outputs": [],
"source": [
"test_set = [(t[\"instruction\"], *t[\"output\"].rsplit(\" \", maxsplit=1)) for t in test_set]"
]
},
{
"cell_type": "markdown",
"id": "8becd26a",
"metadata": {},
"source": [
"Next, we'll generate the response for each example in the test set and compare it to the ground-truth answer to measure the accuracy."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "b396980a",
"metadata": {},
"outputs": [],
"source": [
"def evaluate(model, tokenizer, num_test):\n",
" num_correct = 0\n",
" for prompt, completion, answer in tqdm.tqdm(test_set[:num_test]): \n",
" messages = [\n",
" {\"role\": \"user\", \"content\": prompt},\n",
" {\"role\": \"assistant\", \"content\": completion}\n",
" ]\n",
" tokens = tokenizer.apply_chat_template(messages, continue_final_message=True)\n",
" response = generate(model, tokenizer, tokens, max_tokens=2)\n",
" num_correct += (response==answer)\n",
" return num_correct / num_test"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "4cbc00b3",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████| 100/100 [00:28<00:00, 3.53it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Approximate test accuracy 0.730\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"# Increase this number to use more test examples\n",
"num_test = 100\n",
"test_acc = evaluate(model, tokenizer, num_test)\n",
"print(f\"Approximate test accuracy {test_acc:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "67fbba7f",
"metadata": {},
"source": [
"### Fuse Adapters\n",
"\n",
"Sometimes its convenient to fuse the adapters into the base model to create a single adapted model. MLX LM has a fuse script just for that.\n",
"\n",
"The adapted weights are: $\\tilde{W} = W + c \\cdot \\mathbf{b}^\\top \\mathbf{a}$. Note, this process can be destructive if the inputs are in low precision and they have very different magnitudes. Tuning the `scale` parameter, $c$, prior to fine-tuning can improve the model performance after fusion.\n",
"\n",
"To see more options for fusing the model, including how to upload to HuggingFace [check the documentation](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/LORA.md#fuse)."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "37854c9b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading pretrained model\n",
"Fetching 13 files: 100%|█████████████████████| 13/13 [00:00<00:00, 33762.20it/s]\n"
]
}
],
"source": [
"!mlx_lm.fuse --model {model_path}"
]
},
{
"cell_type": "markdown",
"id": "c349707e",
"metadata": {},
"source": [
"Once the adapters are fused, we can rerun the evaluation using the fused model to make sure it worked. By default the fused model will be saved to `fused_model`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1c45e3a",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 0%| | 0/100 [00:00<?, ?it/s]"
]
}
],
"source": [
"model, tokenizer = load(\"fused_model\")\n",
"test_acc = evaluate(model, tokenizer, num_test)\n",
"print(f\"Approximate test accuracy {test_acc:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "d0dc7f4c",
"metadata": {},
"source": [
"### Troubleshooting\n",
"\n",
"#### Results\n",
"\n",
"To figure out why your LoRA adapters are not working well it's critical to plot both the trianing loss and validation loss over the duration of fine-tuning. There are really only two cases to consider: underfitting or overfitting. And you can figure out which regime you are in based on the above plot.\n",
"\n",
"**Underfitting**: The trianing loss is not low enough and the validation loss closely matches the training loss. You could also measure the accuracy on the training set itself for question-answering style tasks like HellaSwag. If you are in this regime you have a few options to improve the results:\n",
"\n",
"- Use more adapters. Increase `lora_layers` or adapt more of the linear layers within a given block by setting `lora_parameters[\"keys\"]`.\n",
"- Use a higher rank. A higher rank means more parameters per adapter.\n",
"- If you are using dropout, decrease the droupout rate or turn it off entirely.\n",
"- Sometimes, underfitting issues are really optimization issues. In these cases it can be helpful to tune the learning rate or learning rate schedule.\n",
"- If none of the above works, try a bigger model. For example, try Phi-3 medium instead of Phi-3 tiny.\n",
"\n",
"**Overfitting**: The trianing loss keeps going down but the validation loss stops going down and even starts to go up. If you are in this regime you also have a few options:\n",
"\n",
"- The best thing to do is to use more trianing data if you have it.\n",
"- Contrary to the underfitting regime decreasing the capacity of the model can help. For example, use fewer adapters, a lower LoRA rank, or a smaller model size.\n",
"- If you are not using dropout, use it.\n",
"\n",
"If you find your adapters work well pre-fusion but stop working post-fusion, try tuning the `scale` parameter, $c$, prior to fine-tuning. Typically the adapters have a smaller magnitude than the weights, so using a larger scale helps.\n",
"\n",
"#### Memory Use\n",
"\n",
"Fine-tuning a large LM with LoRA requires a machine with a decent amount of memory. Here are some tips to reduce memory use should you need to do so. \n",
"\n",
"- Try quantization (QLoRA). You can use QLoRA by generating a quantized model with `mlx_lm.convert` and the `-q` flag or by using an already quantized model from HuggingFace.\n",
"\n",
"- Try using a smaller batch size. You can set the `batch_size` parameter in the `TrainingArgs` or pass `--batch-size` if you are using the CLI. The default is 4 so setting this to 2 or 1 will reduce memory consumption. Note, this may slow things down a little..\n",
"\n",
"- Reduce the number of layers to fine-tune with by setting `lora_layers` to a smaller value or passing `--lora-layers` if you are using the CLI. The default is `16`, so you can try `8` or `4`. This reduces the amount of memory needed for back propagation. It may also reduce the quality of the fine-tuned model and you may need to compensate with a larger `rank`.\n",
"\n",
"- Longer examples require more memory. If it makes sense for your data, one thing you can do is break your examples into smaller sequences when making the `train`, `valid`, and `test` data sets.\n",
"\n",
"- Gradient checkpointing lets you trade-off memory use (less) for computation (more) by recomputing instead of storing intermediate values needed by the backward pass. You can use gradient checkpointing by passing `grad_checkpoint=True` to the `TrainingArgs` or the `--grad-checkpoint` flag if using the CLI. Gradient checkpointing will be more helpful for larger batch sizes or sequence lengths with smaller or quantized models.\n",
"\n",
"### Next Steps\n",
"\n",
"- To learn more about MLX check-out the [GitHub repo](http://github.com/ml-explore/mlx) and [documentation](https://ml-explore.github.io/mlx/)\n",
"- For more on MLX LM check-out the [MLX LM documentation](https://github.com/ml-explore/mlx-examples/tree/main/llms#readme).\n",
"- Check out the other [MLX Examples](https://github.com/ml-explore/mlx-examples/tree/main). These are great as a learning resource or to use as a starting point for a new project.\n",
"- We also have an example of [LoRA fine-tuning in MLX Swift](https://github.com/ml-explore/mlx-swift-examples/tree/main/Applications/LoRATrainingExample)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment