Skip to content

Instantly share code, notes, and snippets.

@davideuler

davideuler/MLX LM LoRA Fine Tune.ipynb

Forked from awni/MLX LM LoRA Fine Tune.ipynb
Created June 4, 2024 00:14
Show Gist options
  • Save davideuler/8e5647e5aaadded867d4ed93d75a43c4 to your computer and use it in GitHub Desktop.
Save davideuler/8e5647e5aaadded867d4ed93d75a43c4 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
MLX LM LoRA Fine Tune.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "1055c3f3",
"metadata": {},
"source": [
"### LoRA Fine-Tuning with MLX LM\n",
"\n",
"In this notebook, we'll walk through how to [LoRA fine-tune](https://arxiv.org/abs/2106.09685) an LLM with MLX LM. We'll use the [HellaSwag](https://rowanzellers.com/hellaswag/) dataset for common sense reasoning as an example. An outline:\n",
"\n",
"1. Download the dataset and prepare it in the right format for MLX LM.\n",
"2. Setup and run LoRA training. We'll show how to capture the training logs and plot some statistics to visualize the performance.\n",
"3. Evaluate on the test set. We'll compute the final question-answer accuracy of the fine-tuned model.\n",
"4. Fuse the resulting adapters into the base model and upload to Hugging Face.\n",
"5. Discuss tips for debugging accuracy and efficiency."
]
},
{
"cell_type": "markdown",
"id": "21397627",
"metadata": {},
"source": [
"### Install dependencies"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "664272fb",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!pip install mlx-lm\n",
"!pip install matplotlib"
]
},
{
"cell_type": "markdown",
"id": "dd27c693",
"metadata": {},
"source": [
"### Preprocess Data\n",
"We'll start by downloading an already pre-processed version of the HellaSwag dataset from [LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters)."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "61698208",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"HellaSwag stats: 39905 training examples and 10042 test examples.\n",
"An example:\n",
"\n",
"{\n",
" \"instruction\": \"Please choose the correct ending to complete the given sentence: Removing ice from car: Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then\\n\\nEnding1: , the man adds wax to the windshield and cuts it. Ending2: , a person board a ski lift, while two men supporting the head of the person wearing winter clothes snow as the we girls sled. Ending3: , the man puts on a christmas coat, knitted with netting. Ending4: , the man continues removing the snow on his car.\\n\\nAnswer format: ending1/ending2/ending3/ending4\",\n",
" \"input\": \"\",\n",
" \"output\": \"the correct answer is ending4\",\n",
" \"answer\": \"ending4\"\n",
"}\n"
]
}
],
"source": [
"import json\n",
"import numpy as np\n",
"from pathlib import Path\n",
"from urllib import request\n",
"\n",
"save_dir = \"/tmp/hellaswag\"\n",
"\n",
"def download_and_save(save_dir):\n",
" base_url = \"https://raw.githubusercontent.com/AGI-Edgerunners/LLM-Adapters/main/dataset/hellaswag/\"\n",
" save_dir = Path(save_dir)\n",
" save_dir.mkdir(parents=True, exist_ok=True)\n",
" for name in [\"train.json\", \"test.json\"]:\n",
" out_file = save_dir / name\n",
" if not out_file.exists():\n",
" request.urlretrieve(base_url + name, out_file)\n",
"\n",
"def load_json(dataset):\n",
" download_and_save(save_dir)\n",
" with open(f\"{save_dir}/{dataset}.json\", \"r\") as fid:\n",
" return json.load(fid)\n",
"\n",
"train_set, test_set = load_json(\"train\"), load_json(\"test\")\n",
"print(f\"HellaSwag stats: {len(train_set)} training examples and {len(test_set)} test examples.\")\n",
"print(\"An example:\\n\")\n",
"print(json.dumps(train_set[0], indent=4))"
]
},
{
"cell_type": "markdown",
"id": "9a514d79",
"metadata": {},
"source": [
"Next, let's split the training set into a training and a validation set. We'll pull out a randomly chosen 10% for validation."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "9b607237",
"metadata": {},
"outputs": [],
"source": [
"# Seed for reproducibility\n",
"np.random.seed(43)\n",
"perm = np.random.permutation(len(train_set))\n",
"valid_size = int(0.1 * len(train_set))\n",
"valid_set = [train_set[i] for i in perm[:valid_size]]\n",
"train_set = [train_set[i] for i in perm[valid_size:]]"
]
},
{
"cell_type": "markdown",
"id": "35c38c4e",
"metadata": {},
"source": [
"Finally, put the data splits in the MLX LM training format. The format simply expects the data to be in a container which supports random access to the individual examples (e.g. a Python `list`):\n",
"```\n",
"[\"An example for the model.\", \"Another example for the model.\", ...]\n",
"```\n",
"For more details, see the [documentation on supported formats](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/LORA.md#Data)."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "ea738f2b",
"metadata": {},
"outputs": [],
"source": [
"def preprocess(dataset):\n",
" return [t[\"instruction\"] + \"\\n\" + t[\"output\"] for t in dataset]\n",
"\n",
"train_set, valid_set, test_set = map(preprocess, (train_set, valid_set, test_set))"
]
},
{
"cell_type": "markdown",
"id": "b259eb69",
"metadata": {},
"source": [
"### Fine-Tune\n",
"\n",
"For fine-tuning, we'll use Microsoft's [Phi-3 mini](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct). At 3.8 billion parameters, Phi-3 mini is a high-quality model that is also fast to fine-tune on most Apple silicon machines. Also, it has a [permissive MIT License](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/LICENSE).\n",
"\n",
"First, import all the packages and functions we need."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "c3ff309a",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import mlx.core as mx\n",
"import mlx.optimizers as optim\n",
"from mlx.utils import tree_flatten\n",
"from mlx_lm import load, generate\n",
"from mlx_lm.tuner import train, evaluate, TrainingArgs\n",
"from mlx_lm.tuner import linear_to_lora_layers\n",
"import tqdm"
]
},
{
"cell_type": "markdown",
"id": "87628d24",
"metadata": {},
"source": [
"Next, setup the LoRA parameters and make the training arguments. See the [training argument class](https://github.com/ml-explore/mlx-examples/blob/81318ad4a8b2ca5fd1431a42db2b0244d16be851/llms/mlx_lm/tuner/trainer.py#L31-L63) for a more detailed list of training parameters. \n",
"\n",
"Recall the LoRA update is $W^\\top \\mathbf{x} + c \\cdot \\mathbf{a} \\mathbf{b}^\\top \\mathbf{x}$ where $\\mathbf{a}$ has shape `(D, rank)`.\n",
"\n",
"With that in mind, the LoRA parameters to attend to are:\n",
"- `lora_layers`: The number of Transformer blocks from the top of the model to adapt.\n",
"- `rank`: The rank of the low-rank adapters. A larger rank implies more adapter parameters per linear layer.\n",
"- `scale`: This is the constant $c$ that scales the low-rank update."
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "f0851dc8",
"metadata": {},
"outputs": [],
"source": [
"# Make a directory to save the adapter config and weights\n",
"adapter_path = Path(\"adapters\")\n",
"adapter_path.mkdir(parents=True, exist_ok=True)\n",
"\n",
"lora_config = {\n",
" \"lora_layers\": 8,\n",
" \"lora_parameters\": {\n",
" \"rank\": 8,\n",
" \"scale\": 20.0,\n",
" \"dropout\": 0.0,\n",
"}}\n",
"\n",
"# Save the LoRA config to the adapter path\n",
"with open(adapter_path / \"adapter_config.json\", \"w\") as fid:\n",
" json.dump(lora_config, fid, indent=4) \n",
"\n",
"training_args = TrainingArgs(\n",
" adapter_file=adapter_path / \"adapters.safetensors\",\n",
" iters=200,\n",
" steps_per_eval=50\n",
")"
]
},
{
"cell_type": "markdown",
"id": "56fefd19",
"metadata": {},
"source": [
"Next, load the Phi-3 mini model. Note this may take a few minutes to download from HuggingFace if you haven't downloaded it before."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb0b16f2",
"metadata": {},
"outputs": [],
"source": [
"model_path = \"microsoft/Phi-3-mini-4k-instruct\"\n",
"model, tokenizer = load(model_path)"
]
},
{
"cell_type": "markdown",
"id": "6609c92a",
"metadata": {},
"source": [
"After loading the model, freeze it's parameters so we don't train them. Then convert linear layers to LoRA layers using the MLX LM utility `linear_to_lora_layers`. The adapters in the `LoRA` layers are not frozen, so they will be included in the model's `trainable_parameters`. Check-out the [LoRA layer implementation](https://github.com/ml-explore/mlx-examples/blob/81318ad4a8b2ca5fd1431a42db2b0244d16be851/llms/mlx_lm/tuner/lora.py#L72-L104) to see how it all works.\n",
"\n",
"By default, MLX LM only adapts the query, key, and value projection matrices for Phi-3. You can specify the layers to adapt by setting `lora_parameters[\"keys\"]` to a list of layer names. In this case it defaults to `[\"attn.qkv_proj\"]`. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "50e1ab3a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of trainable parameters: 786432\n"
]
}
],
"source": [
"# Freeze the base model\n",
"model.freeze()\n",
"\n",
"# Convert linear layers to lora layers\n",
"linear_to_lora_layers(model, lora_config[\"lora_layers\"], lora_config[\"lora_parameters\"])\n",
"\n",
"num_train_params = (\n",
" sum(v.size for _, v in tree_flatten(model.trainable_parameters()))\n",
")\n",
"print(f\"Number of trainable parameters: {num_train_params}\")"
]
},
{
"cell_type": "markdown",
"id": "827d1590",
"metadata": {},
"source": [
"Now we're ready to put it all together and actually train the model. We'll use `Adam` for the optimizer, but you can specify any [optimizer](https://ml-explore.github.io/mlx/build/html/python/optimizers/common_optimizers.html) with any [scheduler](https://ml-explore.github.io/mlx/build/html/python/optimizers/schedulers.html). We also added a custom class to capture the training and validation loss to plot it later."
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "984516d3",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Starting training..., iters: 200\n",
"Iter 1: Val loss 2.687, Val took 32.210s\n",
"Iter 10: Train loss 2.456, Learning Rate 1.000e-05, It/sec 0.511, Tokens/sec 583.542, Trained Tokens 11412, Peak mem 14.235 GB\n",
"Iter 20: Train loss 2.328, Learning Rate 1.000e-05, It/sec 0.708, Tokens/sec 605.538, Trained Tokens 19963, Peak mem 14.235 GB\n",
"Iter 30: Train loss 2.062, Learning Rate 1.000e-05, It/sec 0.621, Tokens/sec 625.616, Trained Tokens 30034, Peak mem 14.235 GB\n",
"Iter 40: Train loss 1.856, Learning Rate 1.000e-05, It/sec 0.577, Tokens/sec 613.944, Trained Tokens 40670, Peak mem 14.235 GB\n",
"Iter 50: Val loss 1.808, Val took 30.444s\n",
"Iter 50: Train loss 1.872, Learning Rate 1.000e-05, It/sec 4.770, Tokens/sec 5100.991, Trained Tokens 51365, Peak mem 14.235 GB\n",
"Iter 60: Train loss 1.904, Learning Rate 1.000e-05, It/sec 0.588, Tokens/sec 640.415, Trained Tokens 62257, Peak mem 14.235 GB\n",
"Iter 70: Train loss 1.737, Learning Rate 1.000e-05, It/sec 0.603, Tokens/sec 648.153, Trained Tokens 73010, Peak mem 14.235 GB\n",
"Iter 80: Train loss 1.810, Learning Rate 1.000e-05, It/sec 0.657, Tokens/sec 638.259, Trained Tokens 82730, Peak mem 14.235 GB\n",
"Iter 90: Train loss 1.652, Learning Rate 1.000e-05, It/sec 0.667, Tokens/sec 631.938, Trained Tokens 92203, Peak mem 14.235 GB\n",
"Iter 100: Val loss 1.756, Val took 28.710s\n",
"Iter 100: Train loss 1.811, Learning Rate 1.000e-05, It/sec 3.804, Tokens/sec 3831.672, Trained Tokens 102276, Peak mem 14.235 GB\n",
"Iter 100: Saved adapter weights to adapters/adapters.safetensors and adapters/0000100_adapters.safetensors.\n",
"Iter 110: Train loss 1.770, Learning Rate 1.000e-05, It/sec 0.678, Tokens/sec 610.968, Trained Tokens 111292, Peak mem 14.235 GB\n",
"Iter 120: Train loss 1.696, Learning Rate 1.000e-05, It/sec 0.734, Tokens/sec 631.704, Trained Tokens 119904, Peak mem 14.235 GB\n",
"Iter 130: Train loss 1.719, Learning Rate 1.000e-05, It/sec 0.729, Tokens/sec 636.505, Trained Tokens 128640, Peak mem 14.235 GB\n",
"Iter 140: Train loss 1.673, Learning Rate 1.000e-05, It/sec 0.660, Tokens/sec 653.183, Trained Tokens 138543, Peak mem 14.235 GB\n",
"Iter 150: Val loss 1.645, Val took 28.133s\n",
"Iter 150: Train loss 1.694, Learning Rate 1.000e-05, It/sec 5.227, Tokens/sec 5618.089, Trained Tokens 149291, Peak mem 14.235 GB\n",
"Iter 160: Train loss 1.691, Learning Rate 1.000e-05, It/sec 0.800, Tokens/sec 608.860, Trained Tokens 156906, Peak mem 14.235 GB\n",
"Iter 170: Train loss 1.653, Learning Rate 1.000e-05, It/sec 0.848, Tokens/sec 640.269, Trained Tokens 164458, Peak mem 14.235 GB\n",
"Iter 180: Train loss 1.638, Learning Rate 1.000e-05, It/sec 0.516, Tokens/sec 644.267, Trained Tokens 176950, Peak mem 14.235 GB\n",
"Iter 190: Train loss 1.579, Learning Rate 1.000e-05, It/sec 0.666, Tokens/sec 627.337, Trained Tokens 186373, Peak mem 14.235 GB\n",
"Iter 200: Val loss 1.758, Val took 31.519s\n",
"Iter 200: Train loss 1.658, Learning Rate 1.000e-05, It/sec 6.545, Tokens/sec 5317.109, Trained Tokens 194497, Peak mem 14.235 GB\n",
"Iter 200: Saved adapter weights to adapters/adapters.safetensors and adapters/0000200_adapters.safetensors.\n",
"Saved final adapter weights to adapters/adapters.safetensors.\n"
]
}
],
"source": [
"# Put the model in training mode:\n",
"model.train()\n",
"\n",
"# Make the optimizer:\n",
"opt = optim.Adam(learning_rate=1e-5)\n",
"\n",
"# Make a class to record the training stats:\n",
"class Metrics:\n",
" train_losses = []\n",
" val_losses = []\n",
" def on_train_loss_report(self, info):\n",
" self.train_losses.append((info[\"iteration\"], info[\"train_loss\"]))\n",
" def on_val_loss_report(self, info):\n",
" self.val_losses.append((info[\"iteration\"], info[\"val_loss\"]))\n",
"\n",
"metrics = Metrics()\n",
"\n",
"# Train model:\n",
"train(\n",
" model=model,\n",
" tokenizer=tokenizer,\n",
" args=training_args,\n",
" optimizer=opt,\n",
" train_dataset=train_set,\n",
" val_dataset=valid_set,\n",
" training_callback=metrics,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "b8d043b8",
"metadata": {},
"source": [
"The adapters are saved every 100 iterations along with the final adapters in `adapters.safetensors`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac329358",
"metadata": {},
"outputs": [],
"source": [
"!ls adapters/"
]
},
{
"cell_type": "markdown",
"id": "2b7e23ee",
"metadata": {},
"source": [
"Next, let's plot the training and validation losses to see how well the adapters fit the data."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "f1ffd638",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABtB0lEQVR4nO3deVyVdfr/8dc5rCKbKAgoKu4i7ltqLpW5VKYt38oss2wzncmZqWms37TOZNbU1FRjM21WtjdpZaW574p7KuaKggpuyKps59y/Pw4gyHbAczgHeD8fj/PwcG/nujnCufhsl8kwDAMRERGResLs6gBEREREHEnJjYiIiNQrSm5ERESkXlFyIyIiIvWKkhsRERGpV5TciIiISL2i5EZERETqFU9XB1DbrFYrJ06cICAgAJPJ5OpwRERExA6GYZCZmUlkZCRmc+VtMw0uuTlx4gRRUVGuDkNERERqICkpiZYtW1Z6TINLbgICAgDbNycwMNDF0YiIiIg9MjIyiIqKKv4cr0yDS26KuqICAwOV3IiIiNQx9gwp0YBiERERqVeU3IiIiEi9ouRGRERE6pUGN+ZGRETEWSwWC/n5+a4Oo87y9vaucpq3PZTciIiIXCbDMEhJSSEtLc3VodRpZrOZ6OhovL29L+s6Sm5EREQuU1FiExYWhp+fnxaJrYGiRXaTk5Np1arVZX0PldyIiIhcBovFUpzYNG3a1NXh1GmhoaGcOHGCgoICvLy8anwdDSgWERG5DEVjbPz8/FwcSd1X1B1lsVgu6zpKbkRERBxAXVGXz1HfQ3VLOYrVAkfXQ9ZJ8G8OrQeB2cPVUYmIiDQ4Sm4cIf57WPQEZJy4uC0wEkbPhpgbXReXiIhIA6RuqcsV/z18Nal0YgOQkWzbHv+9a+ISEZE6xWI12HDoLN/tOM6GQ2exWA1Xh1Rtbdq04fXXX3d1GGq5uSxWi63FhvL+AxqACRb9BTpfry4qERGp0KLdyTz3QzzJ6TnF2yKCfHlmbAyjYyMc/npVjW155plnePbZZ6t93c2bN9O4ceMaRuU4Sm4ux9H1ZVtsSjEg47jtuOghtRaWiIjUHYt2JzN13rYyfyanpOcwdd425tzV2+EJTnJycvHzL7/8kqeffpp9+/YVb/P39y9+bhgGFosFT8+qU4bQ0FCHxllT6pa6HFknHXuciIjUeYZhcD6vwK5HZk4+z3y/p8L2f4Bnv48nMyffrusZhn1dWeHh4cWPoKAgTCZT8de//fYbAQEB/Pzzz/Tp0wcfHx/Wrl3LoUOHGDduHM2bN8ff359+/fqxdOnSUte9tFvKZDLx3nvvcdNNN+Hn50eHDh34/nvnD9dQy83l8G/u2ONERKTOu5BvIebpxQ65lgGkZOTQ7dlf7Do+/vlR+Hk75qP9L3/5C//4xz9o27YtTZo0ISkpieuuu46///3v+Pj48PHHHzN27Fj27dtHq1atKrzOc889x8svv8wrr7zCm2++ycSJEzl69CghISEOibM8arm5HK0H2WZFUVHfpQkCW9iOExERqUOef/55rr32Wtq1a0dISAg9evTgoYceIjY2lg4dOvDCCy/Qrl27KltiJk+ezIQJE2jfvj0vvvgiWVlZxMXFOTV2tdxcDrOHbbr3V5OwJTjlNAeOfkmDiUVEGpBGXh7EPz/KrmPjElKZ/OHmKo+be28/+kdX3dLRyMtxnzd9+/Yt9XVWVhbPPvssP/74I8nJyRQUFHDhwgUSExMrvU737t2Lnzdu3JjAwEBOnTrlsDjLo+TmcsXcCLd9XHadG0xw87ta50ZEpIExmUx2dw0N6RBKRJAvKek55Y67MQHhQb4M6RCKh7l2V0C+dNbTY489xpIlS/jHP/5B+/btadSoEbfeeit5eXmVXufSGlEmkwmr1erweEtSt5QjxNwIM3bDPQttCU1ABGBAXparIxMRETfmYTbxzNgYoOwAh6KvnxkbU+uJTXnWrVvH5MmTuemmm+jWrRvh4eEcOXLE1WGVS8mNo5g9bNO9u98GA6fbtm1+D+wcuS4iIg3T6NgI5tzVm/Ag31Lbw4N8nTINvKY6dOjAt99+y44dO9i5cyd33nmn01tgakrdUs7QayIs/xuc3A1Jm6DVFa6OSERE3Njo2AiujQknLiGVU5k5hAX40j86xC1abIq89tpr3HfffQwaNIhmzZrxxBNPkJGR4eqwymUy7J0UX09kZGQQFBREeno6gYGBznuh76bB9nnQ7f/glvec9zoiIuJSOTk5JCQkEB0dja+vb9UnSIUq+15W5/Nb3VLO0u8B2797FkCWc0eFi4iIyEVKbpwlsie07AfWfNj2saujERERaTCU3DhTv/tt/275ECwFro1FRESkgVBy40wx48GvKWQcg/2LXB2NiIhIg6Dkxpm8fKHX3bbnmzWoWEREpDYouXG2vvcBJji8As4cdHU0IiIi9Z6SG2dr0ho6FtYY2fK+a2MRERFpAJTc1IaiaeHbP4W8bNfGIiIiUs8puakN7a6GJtGQmw67vnF1NCIiIpdt+PDhzJgxo/jrNm3a8Prrr1d6jslkYsGCBU6NC5Tc1A6zGfpNsT3f/K7qTYmISFlWCySssf0RnLDG9rWTjB07ltGjR5e7b82aNZhMJn799ddqXXPz5s08+OCDjgjvsqm2VG3pWVhvKmUXJMVBqwGujkhERNxF/Pew6AnIOHFxW2AkjJ4NMTc6/OWmTJnCLbfcwrFjx2jZsmWpfR9++CF9+/ale/fu1bpmaGioI0O8LGq5qS1+IRB7q+25poWLiEiR+O/hq0mlExuAjGTb9vjvHf6SN9xwA6GhocydO7fU9qysLL7++mvGjx/PhAkTaNGiBX5+fnTr1o3PP/+80mte2i114MABhg4diq+vLzExMSxZssTh91ERJTe1qX/hisXxCyDrtEtDERERJzEM2+QRex45GfDzn4HyhisUblv0hO04e65n57AHT09PJk2axNy5cylZP/vrr7/GYrFw11130adPH3788Ud2797Ngw8+yN13301cXJxd17dardx88814e3uzadMm3nnnHZ544gm7znUEdUvVpshe0KIPHN8K2z+GIX9ydUQiIuJo+efhxUgHXcywtei8FGXf4U+eAO/Gdh1633338corr7Bq1SqGDx8O2LqkbrnlFlq3bs1jjz1WfOzvfvc7Fi9ezFdffUX//v2rvPbSpUv57bffWLx4MZGRtu/Fiy++yJgxY+y7j8vk0pabWbNm0a9fPwICAggLC2P8+PHs27evyvPS0tKYNm0aERER+Pj40LFjR3766adaiNgBiqaFb/nQqYPFREREKtO5c2cGDRrEBx98AMDBgwdZs2YNU6ZMwWKx8MILL9CtWzdCQkLw9/dn8eLFJCYm2nXtvXv3EhUVVZzYAAwcONAp91Eel7bcrFq1imnTptGvXz8KCgp48sknGTlyJPHx8TRuXH7mmZeXx7XXXktYWBjffPMNLVq04OjRowQHB9du8DXV9SZY/CSkJ9nqTXW+3tURiYiII3n52VpQ7HF0PXx6a9XHTfwGWg+y77WrYcqUKfzud7/j7bff5sMPP6Rdu3YMGzaM2bNn88Ybb/D666/TrVs3GjduzIwZM8jLy6vW9V3FpcnNokWli0nOnTuXsLAwtm7dytChQ8s954MPPiA1NZX169fj5eUF2AYxVSQ3N5fc3NzirzMyMi4/8Mvh5Qu974Z1b9gGFiu5ERGpX0wmu7uGaHe1bVZURjLlj7sx2fa3uxrMHo6MEoDbbruNRx99lM8++4yPP/6YqVOnYjKZWLduHePGjeOuu+4CbGNo9u/fT0xMjF3X7dKlC0lJSSQnJxMREQHAxo0bHR5/RdxqQHF6ejoAISEhFR7z/fffM3DgQKZNm0bz5s2JjY3lxRdfxGIpv4tn1qxZBAUFFT+iouzst3SmonpTh5ar3pSISENm9rBN9wbAdMnOwq9Hv+SUxAbA39+f22+/nZkzZ5KcnMzkyZMB6NChA0uWLGH9+vXs3buXhx56iJMnT9p93REjRtCxY0fuuecedu7cyZo1a3jqqaeccg/lcZvkxmq1MmPGDAYPHkxsbGyFxx0+fJhvvvkGi8XCTz/9xF//+ldeffVV/va3v5V7/MyZM0lPTy9+JCUlOesW7NekDXQYaXu+5QOXhiIiIi4WcyPc9jEERpTeHhhp2+6EdW5KmjJlCufOnWPUqFHFY2T+3//7f/Tu3ZtRo0YxfPhwwsPDGT9+vN3XNJvNzJ8/nwsXLtC/f3/uv/9+/v73vzvpDsoyGYZ7LJc7depUfv75Z9auXVtmQaGSOnbsSE5ODgkJCXh42DLZ1157jVdeeYXk5OQqXycjI4OgoCDS09MJDAx0WPzVdmCJrZ/VNwj++Bt4V6+fVERE3EPRZ1J0dDS+vr41v5DVYhuDk3US/Jvbxtg4qcXGXVX2vazO57dbTAWfPn06CxcuZPXq1ZUmNgARERF4eXkVJzZg69tLSUkhLy8Pb29vZ4frGO2usbXgnDsCu7+B3pNcHZGIiLiS2QOih7g6inrBpd1ShmEwffp05s+fz/Lly4mOjq7ynMGDB3Pw4EGsVmvxtv379xMREVF3Ehuw1ZvqW1hvKk71pkRERBzFpcnNtGnTmDdvHp999hkBAQGkpKSQkpLChQsXio+ZNGkSM2fOLP566tSppKam8uijj7J//35+/PFHXnzxRaZNm+aKW7g8ve4CT19I+RWObXZ1NCIiIvWCS5ObOXPmkJ6ezvDhw4mIiCh+fPnll8XHJCYmlhpLExUVxeLFi9m8eTPdu3fn97//PY8++ih/+ctfXHELl8cvBGJvsT1XvSkRERGHcOmYG3vGMq9cubLMtoEDB9bqfHmn6nc/7PgU9syHkX8Hf/epqioiIvZzk/k5dZqjvoduMxW8wWrRGyJ7gyUPtn/i6mhERKSaihaUPX/+vIsjqfuKVkAuOWmoJtxitlSD1/8BWDDVVm9q8KMNbuqfiEhd5uHhQXBwMKdOnQLAz88Pk+nSBfmkKlarldOnT+Pn54en5+WlJ0pu3EHXmwvrTSXCgV+gU+1UTRUREccIDw8HKE5wpGbMZjOtWrW67ORQyY078PKFXnfD+n/ZpoUruRERqVNMJhMRERGEhYWRn5/v6nDqLG9vb8zmyx8xo+TGXfS9D9a/CYeWwdlD0LSdqyMSEZFq8vDwuOzxInL5NKDYXYREQ4drbc9Vb0pERKTGlNy4k34P2P7d/gnkadS9iIhITSi5cSftr4Hg1pCTDrv/5+poRERE6iQlN+7E7AH9CutNbVa9KRERkZpQcuNuet0NHj6QvBOOb3V1NCIiInWOkht3U7LeVNy7ro1FRESkDlJy447632/7d8+3kH3GtbGIiIjUMUpu3FGLPhDZS/WmREREakDJjbsqmha++QOwWlwbi4iISB2i5MZdxd4MjZoU1pta4upoRERE6gwlN+7KqxH0usv2fLMGFouIiNhLyY0763sfYIKDSyH1sKujERERqROU3LizkLbQfoTt+eb3XRuLiIhIHaHkxt31L6o3NU/1pkREROyg5MbdtR8Bwa0gJ8227o2IiIhUSsmNuzN7QN/CelNxqjclIiJSFSU3dUFxvakdcHybq6MRERFxa0pu6oLGTW3r3oCmhYuIiFRByU1dUbRi8e5vIfusa2MRERFxY0pu6ooWvSGiJ1hyVW9KRESkEkpu6gqT6eK08C3vq96UiIhIBZTc1CVdbwbfYEhLtK1aLCIiImUoualLvP0u1puK08BiERGR8ii5qWv6Fa55o3pTIiIi5VJyU9cU15syYMsHro5GRETE7Si5qYv6lag3lX/BtbGIiIi4GSU3bspiNdhw6Czf7TjOhkNnsVhLlF3ocC0EtYIL52zr3oiIiEgxT1cHIGUt2p3Mcz/Ek5yeU7wtIsiXZ8bGMDo2wlZvqt99sPRZ24rFvSa6LlgRERE3o5YbN7NodzJT520rldgApKTnMHXeNhbtTrZt6HU3eHjDie1wfKsLIhUREXFPSm7ciMVq8NwP8ZRX97to23M/xNu6qBo3s617AxD3Xm2FKCIi4vaU3LiRuITUMi02JRlAcnoOcQmptg1FKxbv/h+cT3V+gCIiInWAkhs3ciqz4sSm3ONa9IGIHqo3JSIiUoKSGzcSFuBbveNMpovTwjer3pSIiAgouXEr/aNDiAjyxVTJMRFBvvSPDrm4IfaWwnpTR+HgMmeHKCIi4vaU3LgRD7OJZ8bGAFSY4DwzNgYPc4m9JetNbVa9KRERESU3bmZ0bARz7upNeFD5XVTBft5lN/a9z/bvgSWQmuDE6ERERNyfyTCM8mYe11sZGRkEBQWRnp5OYGCgq8OpkMVqEJeQyqnMHMICfPnh1+N8timJzuEBLPzdlXh6XJKXfnIzHFoGg34PI19wTdAiIiJOUp3Pb7XcuCkPs4mB7ZoyrmcLBrZryuMjOxPUyIvfUjL5PC6x7AlF08K3f6J6UyIi0qApuakjmjT25rGRHQF4dcl+zmXnlT6gw8iL9ab2zHdBhCIiIu5ByU0dMqF/KzqHB5B2Pp/XluwvvdPsAX3vtT2P08BiERFpuJTc1CGeHmaeGdsVgE83HSX+REbpA3pPKqw3tU31pkREpMFSclPHDGzXlOu7RWA14Lkf9lBqPHjjZtD1Jtvzze+7JkAREREXU3JTB828rjO+XmY2JaTy467k0jv7qd6UiIg0bEpu6qCWTfyYOqw9AC/+uJcLeSXKLrTsC+HdoSAHts9zUYQiIiKu49LkZtasWfTr14+AgADCwsIYP348+/bts/v8L774ApPJxPjx450XpJt6aFhbWgQ34kR6DnNWHbq4w2S6OC18y/tgtbomQBERERdxaXKzatUqpk2bxsaNG1myZAn5+fmMHDmS7OzsKs89cuQIjz32GEOGDKmFSN2Pr5cH/+/6LgC8s+oQSannL+6MvRV8g+DcEdvCfiIiIg2IS5ObRYsWMXnyZLp27UqPHj2YO3cuiYmJbN1a+Uwfi8XCxIkTee6552jbtm0tRet+RseGM7BtU/IKrPz9x70Xd3j7Qc/CelOaFi4iIg2MW425SU9PByAkJKTS455//nnCwsKYMmVKldfMzc0lIyOj1KO+MJlMPHOjrZDmoj0prDt45uLOfoXfmwO/2FpwREREGgi3SW6sViszZsxg8ODBxMbGVnjc2rVref/993n3XftaJGbNmkVQUFDxIyoqylEhu4XO4YHcfUVrwDY1vMBSOMamaTtodzVgwJYPXBegiIhILXOb5GbatGns3r2bL774osJjMjMzufvuu3n33Xdp1qyZXdedOXMm6enpxY+kpCRHhew2/jCiI038vNh/Mot5G49e3FE0LXzbJ5Cf45rgREREapmnqwMAmD59OgsXLmT16tW0bNmywuMOHTrEkSNHGDt2bPE2a+FsIE9PT/bt20e7du1KnePj44OPj49zAncTQX5ePDaqE0/N381rS/YztkckTf19oOMoCIqC9CRbvameE1wdqoiIiNO5tOXGMAymT5/O/PnzWb58OdHR0ZUe37lzZ3bt2sWOHTuKHzfeeCNXXXUVO3bsqHddTtVxR79WxEQEkpFTwD9+Kaw7VbLe1GYNLBYRkYbBpcnNtGnTmDdvHp999hkBAQGkpKSQkpLChQsXio+ZNGkSM2fOBMDX15fY2NhSj+DgYAICAoiNjcXb29tVt+JyHmYTz95oqzv1xeZEdh+3Dc6mV2G9qeNb4fg2F0YoIiJSO1ya3MyZM4f09HSGDx9ORERE8ePLL78sPiYxMZHk5ORKriJF+keHcGOPSAwDnv2+sO6UfyjEjLcdoHpTIiLSAJiMUpUX67+MjAyCgoJIT08nMDDQ1eE4XHL6Ba7+xyou5Ft4446ejOvZAhI3wQcjwdMX/rgX/Cqfai8iIuJuqvP57TazpcQxIoIaMe0q26DqWT/9RnZuAUT1h/ButnpTOz51cYQiIiLOpeSmHrp/SFuiQhqRkpHDv1cetNWbKpoWvln1pkREpH5TclMP2epOxQDw7uoEjp7Nhm63gk8QnEuAQ8tdHKGIiIjzKLmpp0bGNGdIh2bkWaz87ce94N0Yek207dS0cBERqceU3NRTJpOJZ8bG4Gk2sST+JKv2n4a+hfWm9i+Gc0crv4CIiEgdpeSmHmsfFsCkgW0AeP6HPeQ3aQttr0L1pkREpD5TclPPPTqiA00be3PodDYfrT8C/e637diuelMiIlI/Kbmp54IaefH4qE4AvLH0AKcjr4LAlnD+LMQvcG1wIiIiTqDkpgH4v75RdGsRRGZuAa8sOVii3tR7rg1MRETECZTcNAC2ulO2qeFfbz3GnvBxYPaCY5vhxA7XBiciIuJgSm4aiD6tQ7ipVwsMA55acpLTrcYAcGrZW1isDaoCh4iI1HNKbhqQv4zpjLenmR1J6Ty8rxcAgQcXMOal71m0W8VJRUSkflBy04BsTzxHXoGt9MJWoyPx1tb4mvIZmr2YqfO2KcEREZF6QclNA2GxGjz3Q3yJLSY+tlwLwESPpZiw8twP8eqiEhGROk/JTQMRl5BKcnrpdW2+swwiw/Aj2nySK827SE7PIS4h1UURioiIOIaSmwbiVGbZBfsu4Ms3lqEA3O2xtMLjRERE6hIlNw1EWIBvudvnWUYAcLV5Gy04XeFxIiIidYWSmwaif3QIEUG+mC7ZftiIZI0lFg+TwUONV9E/OsQl8YmIiDiKkpsGwsNsqxIOlElw5hUOLL7DYwUe1rxajkxERMSxlNw0IKNjI5hzV2/Cg0p3PcV59+dCowi8887BngWuCU5ERMRBlNw0MKNjI1j7xNV8/sAVjO8ZCUDXliE0GjjFdoDqTYmISB2n5KYB8jCbGNiuKdOuag9A3JFzXIi9q7DeVBwk73RxhCIiIjWn5KYBax/mT2SQL3kFVjae9oCYcbYdar0REZE6TMlNA2YymRjWKRSAVftOQ7/7bTt+/RounHNhZCIiIjWn5KaBG9bRltys3n8aWl0BzWOh4ALs+MzFkYmIiNSMkpsGblD7ZniYTRw+k01i6gXoVzSw+H2wWl0bnIiISA0ouWngAn296NOqCQCrDpyGbreBTyCkHoLDK1wcnYiISPUpuZHS4258/KHnnbYdm993YVQiIiI1o+RGisfdbDh0hrwCK/Qt7Jra/zOkJbkwMhERkepTciPERATStLE32XkWth49B6EdIXoYGFbY8oGrwxMREakWJTeC2WxiaGHrzar9p20bi6aFb/sYCnJdFJmIiEj1KbkR4GLXVHFy0+k6CIiE82cg/jsXRiYiIlI9Sm4EgCEdmmEywd7kDE5m5ICHJ/S917ZTKxaLiEgdouRGAGjq70O3FkFA4YJ+AL3vsdWbStoEyb+6MDoRERH7KbmRYmW6pgKaQ8yNtudqvRERkTpCyY0UK0pu1hw4g8Vq2DYWDSze9TVcSHNNYCIiItWg5EaK9YwKJsDXk/QL+fx6LM22sdVACOsK+edVb0pEROoEJTdSzNPDzJXtmwEluqZMphL1pt5TvSkREXF7Sm6klDLjbgC6336x3lTCStcEJiIiYiclN1JK0WJ+O5PSOJedZ9vo4w89Jtieq96UiIi4OSU3UkpkcCM6NvfHasDag2cu7ijqmtr3k+pNiYiIW1NyI2WU2zUV2gmih9rqTW2d65rARERE7KDkRsoY1jEMsCU3hmFc3FFcb+oj1ZsSERG3peRGyujbpgmNvDw4nZnL3uTMizs6XQ8BEZB9GuK/d12AIiIilVByI2X4enkwsF1TAFYfKNE15eEJfVRvSkRE3JuSGynX0A6F693sO116R597wOwJSRshZZcLIhMREamckhsp17BOtnE3W46mkpVbcHFHQDh0Ub0pERFxX0pupFxtmvrRKsSPfIvBhkNnS+8sGlj861eqNyUiIm5HyY2Uy2QylZgSfqr0ztaDICzGVm9q5xcuiE5ERKRiSm6kQkXJzcp9l0wJv7TeVMl9IiIiLubS5GbWrFn069ePgIAAwsLCGD9+PPv27av0nHfffZchQ4bQpEkTmjRpwogRI4iLi6uliBuWge2a4uVh4ti5CyScyS69s/vt4B0AZw/A4ZUuiU9ERKQ8Lk1uVq1axbRp09i4cSNLliwhPz+fkSNHkp2dXeE5K1euZMKECaxYsYINGzYQFRXFyJEjOX78eC1G3jA09vGkX5sQ4JLVigF8AqDHHbbnGlgsIiJuxGQY7tOncPr0acLCwli1ahVDhw616xyLxUKTJk146623mDRpUpXHZ2RkEBQURHp6OoGBgZcbcr33n1WHmPXzb1zVKZQP7+1feuep3+DfA8Bkhhm7IKila4IUEZF6rzqf32415iY9PR2AkJAQu885f/48+fn5FZ6Tm5tLRkZGqYfYb1gn27ibDYfPkpNvKb0zrDO0GaJ6UyIi4lbcJrmxWq3MmDGDwYMHExsba/d5TzzxBJGRkYwYMaLc/bNmzSIoKKj4ERUV5aiQG4ROzQNoHuhDTr6VzUdSyx5QNC1860dQkFe7wYmIiJTDbZKbadOmsXv3br74wv6pxS+99BJffPEF8+fPx9fXt9xjZs6cSXp6evEjKSnJUSE3CCaTiaEdCqeEX7paMUDnonpTp2Cv6k2JiIjruUVyM336dBYuXMiKFSto2dK+cRv/+Mc/eOmll/jll1/o3r17hcf5+PgQGBhY6iHVU9Q1VWZQMYCHF/SZbHuugcUiIuIGXJrcGIbB9OnTmT9/PsuXLyc6Otqu815++WVeeOEFFi1aRN++fZ0cpVzZvhlmExw4lcXxtAtlD+hdWG8qcQOk7K79AEVEREpwaXIzbdo05s2bx2effUZAQAApKSmkpKRw4cLFD9BJkyYxc+bM4q9nz57NX//6Vz744APatGlTfE5WVpYrbqFBCPbzpmdUMACry2u9CYyAzjfYnqv1RkREXMylyc2cOXNIT09n+PDhREREFD++/PLL4mMSExNJTk4udU5eXh633nprqXP+8Y9/uOIWGoxhHW2FNMsddwPQ/wHbv79+BTnptRSViIhIWZ6ufHF7lthZuXJlqa+PHDninGCkUsM6hfLPpftZd/AM+RYrXh6X5MWtB0NoFzi911ZvasBDrglUREQaPLcYUCzur1uLIJr4eZGZW8COpLSyB6jelIiIuAklN2IXD7OJKyubEg62cgze/nBmPySsqsXoRERELqpRcpOUlMSxY8eKv46Li2PGjBn897//dVhg4n6KqoSXOyUcVG9KRETcQo2SmzvvvJMVK1YAkJKSwrXXXktcXBxPPfUUzz//vEMDFPcxtEMzAHYdT+dMVm75BxWtWPzbT5CuYqYiIlL7apTc7N69m/79bUUUv/rqK2JjY1m/fj2ffvopc+fOdWR84kbCAn2JibAtgrjmQAWtN2FdCutNWVRvSkREXKJGyU1+fj4+Pj4ALF26lBtvvBGAzp07l5q2LfVP8WrFFY27gYsDi7fOVb0pERGpdTVKbrp27co777zDmjVrWLJkCaNHjwbgxIkTNG3a1KEBinspGnez+sAZrNYKZkR1vgH8w231pn77oRajExERqWFyM3v2bP7zn/8wfPhwJkyYQI8ePQD4/vvvi7urpH7q3aoJ/j6epGbnsftEBYv1law3FaeBxSIiUrtqtIjf8OHDOXPmDBkZGTRp0qR4+4MPPoifn5/DghP34+1pZlC7pvwSf5LV+0/TvWVw+Qf2mQyrX4HE9XByDzTvWpthiohIA1ajlpsLFy6Qm5tbnNgcPXqU119/nX379hEWFubQAMX9DK1qSjjY6k11Ub0pERGpfTVKbsaNG8fHH38MQFpaGgMGDODVV19l/PjxzJkzx6EBivspGnezLTGN9Av5FR/Yr7De1M4vVW9KRERqTY2Sm23btjFkyBAAvvnmG5o3b87Ro0f5+OOP+de//uXQAMX9RIX40Ta0MRarwfqDZyo+sM2VENoZ8rNtCY6IiEgtqFFyc/78eQICAgD45ZdfuPnmmzGbzVxxxRUcPXrUoQGKe6pytWIorDdVuKif6k2JiEgtqVFy0759exYsWEBSUhKLFy9m5MiRAJw6dYrAwECHBijuqWRyU2l19+63F9ab2gdH1tRSdCIi0pDVKLl5+umneeyxx2jTpg39+/dn4MCBgK0Vp1evXg4NUNzTFW2b4uNpJjk9hwOnsio+0DfQluAAxL1bO8GJiEiDVqPk5tZbbyUxMZEtW7awePHi4u3XXHMN//znPx0WnLgvXy8PBrS1LdhY6WrFUKLe1I+qNyUiIk5Xo+QGIDw8nF69enHixIniCuH9+/enc+fODgtO3NvF1YqrSG6ax0DrwbZ6U9s+qoXIRESkIatRcmO1Wnn++ecJCgqidevWtG7dmuDgYF544QWsVqujYxQ3NayjrUr4psOpnM8rqPzgotYb1ZsSEREnq1Fy89RTT/HWW2/x0ksvsX37drZv386LL77Im2++yV//+ldHxyhuql2oPy2CG5FnsbLpcGrlB3cZC/7NIesk/LawdgIUEZEGqUbJzUcffcR7773H1KlT6d69O927d+eRRx7h3XffZe7cuQ4OUdyVyWSyb7ViKF1vSisWi4iIE9UouUlNTS13bE3nzp1JTa3iL3ipV+xa76ZIn8lg8oCj6+BkvHMDExGRBqtGyU2PHj146623ymx/66236N69+2UHJXXHoPZN8TSbSDiTzdGz2ZUfHBgJna+3PVfrjYiIOEmNqoK//PLLXH/99SxdurR4jZsNGzaQlJTETz/95NAAxb0F+nrRu3UT4hJSWb3/NHcPbFz5Cf0fgL3fw69fwohnbevgiIiIOFCNWm6GDRvG/v37uemmm0hLSyMtLY2bb76ZPXv28Mknnzg6RnFz1eqaajMEmnWCvCxbgiMiIuJgJqPStfOrZ+fOnfTu3RuLxeKoSzpcRkYGQUFBpKenq1SEg+w+ns4Nb67Fz9uDHU+PxNuzipx503/h58dtSc60TbYaVCIiIpWozud3jRfxEykSExFIM39vzudZ2HLUjgHlPe4Ar8aF9abWOj9AERFpUJTcyGUzm00M7VCNrinfQOhRWG9qs+pNiYiIYym5EYcY1qkwuamqzlSRohWL9y6EjBNOikpERBqias2Wuvnmmyvdn5aWdjmxSB12ZftmmEzwW0omJzNyaB7oW/kJzbtCq0GQuB62fgRXzaydQEVEpN6rVnITFBRU5f5JkyZdVkBSNzX196F7iyB2Hkvng7UJxEQGEhbgS//oEDzMFQwY7n9/YXIzF4Y+ZlvFWERE5DJVK7n58MMPnRWH1AMtghux81g6/1l9uHhbRJAvz4yNYXRsRNkTOhfVm0qx1ZvqelMtRisiIvWVxtyIQyzancxPu1PKbE9Jz2HqvG0s2p1c9iRPb+h9j+15nFYsFhERx1ByI5fNYjV47ofya0UVLaL03A/xWKzlLKlUXG9qLZza67QYRUSk4VByI5ctLiGV5PScCvcbQHJ6DnEJ5ayBE9QCOl9ne656UyIi4gBKbuSyncqsOLGx67h+D9j+3fkF5GQ4KCoREWmolNzIZQsLqGLad1XHRQ+FZh1Vb0pERBxCyY1ctv7RIUQE+VJRhSgTtllT/aNDKjjAdHFRv83vgePKnYmISAOk5EYum4fZxDNjYwDKJDhFXz8zNqbi9W7gYr2p07/B0XVOiVNERBoGJTfiEKNjI5hzV2/Cg0p3PTVp7M2cu3qXv85NSb5B0P022/M41ZsSEZGaU3IjDjM6NoK1T1zN5w9cwaB2TQEY1rFZ1YlNkaKuqd8WQkY56+KIiIjYQcmNOJSH2cTAdk3547UdAfhlz0nO5xXYd3J4LLQaCNYC2PaRE6MUEZH6TMmNOEWf1k2ICmlEdp6FJfEn7T+xqPVmy4dgyXdOcCIiUq8puRGnMJlM3NSzBQDztx+3/8QuN0LjsMJ6Uz86KToREanPlNyI04zvZUtu1hw4w+nMXPtO8vSGPoX1prRisYiI1ICSG3GatqH+9IgKxmI1+GHnCftP7HOvrd7UkTVw6jfnBSgiIvWSkhtxqpt6RgKwYEc1uqaCWkCnMbbnar0REZFqUnIjTjW2RyQeZhO/Hkvn4Kks+0/sX6LeVG6mc4ITEZF6ScmNOFVTfx+GdQwFYEF1BhZHD4OmHSAvU/WmRESkWpTciNPd1OvirCmr1c66USXrTcWp3pSIiNhPyY043bUxzfH38eR42gW2HD1n/4k97gAvPzi9F46ud16AIiJSr7g0uZk1axb9+vUjICCAsLAwxo8fz759+6o87+uvv6Zz5874+vrSrVs3fvrpp1qIVmrK18uDMbHhAMzffsz+ExsFX6w3tVn1pkRExD4uTW5WrVrFtGnT2LhxI0uWLCE/P5+RI0eSnZ1d4Tnr169nwoQJTJkyhe3btzN+/HjGjx/P7t27azFyqa6irqmFvyaTk2+x/8Sirqm9P0BmihMiExGR+sZkGO4zmOH06dOEhYWxatUqhg4dWu4xt99+O9nZ2SxcuLB42xVXXEHPnj155513yhyfm5tLbu7FBeQyMjKIiooiPT2dwMBAx9+ElMtqNRg8eznJ6TnMmdibMd3sLKYJ8P4oSNoIw5+E4U84L0gREXFbGRkZBAUF2fX57VZjbtLT0wEICQmp8JgNGzYwYsSIUttGjRrFhg0byj1+1qxZBAUFFT+ioqIcF7DYzWw2cWPhmjfVKscAF6eFb1W9KRERqZrbJDdWq5UZM2YwePBgYmNjKzwuJSWF5s2bl9rWvHlzUlLK77KYOXMm6enpxY+kpCSHxi32u7lXSwBW7DvFuew8+0/sMhYah0JmMuzT+CoREamc2yQ306ZNY/fu3XzxxRcOva6Pjw+BgYGlHuIancID6BIRSL7F4Mddyfaf6OkDvQvrTcVpYLGIiFTOLZKb6dOns3DhQlasWEHLli0rPTY8PJyTJ0+W2nby5EnCw8OdGaI4yM29alApHKDvvWAy2+pNna56Rp2IiDRcLk1uDMNg+vTpzJ8/n+XLlxMdHV3lOQMHDmTZsmWlti1ZsoSBAwc6K0xxoBt7RmI2wdaj50g8e97+E4NaQqfrbM9Vb0pERCrh0uRm2rRpzJs3j88++4yAgABSUlJISUnhwoULxcdMmjSJmTNnFn/96KOPsmjRIl599VV+++03nn32WbZs2cL06dNdcQtSTc0DfRncvhlQg9abomnhOz6H3GrUqRIRkQbFpcnNnDlzSE9PZ/jw4URERBQ/vvzyYi2hxMREkpMvjs8YNGgQn332Gf/973/p0aMH33zzDQsWLKh0ELK4l/E9bV1TC3Ycp1orEUQPg6btVW9KREQq5Vbr3NSG6syTF+fIzi2g79+WciHfwvxHBtGrVRP7T944Bxb9BcJiYOp6Ww0qERGp9+rsOjfSMDT28WRkV9t0/mpVCgfoMcFWb+pUPCSWv7aRiIg0bEpuxCWKyjH88Gsy+Rar/Sc2CoZu/2d7rmnhIiJSDiU34hJXtm9GM38fUrPzWL3/dPVOLq439T1knqz8WBERaXCU3IhLeHqYubGHrRzDt9XtmoroDlEDwFoA2z5yQnQiIlKXKbkRlynqmloaf5KMnGrWjOpXWG9qy4dgKXBwZCIiUpcpuRGXiW0RSPswf3ILrCzaVX5tsArF3Ah+zSDzhOpNiYhIKUpuxGVMJlNx6021F/Tz9IE+hfWmNmtgsYiIXKTkRlxqXE/buJuNCWc5kXahiqMv0aew3lTCatWbEhGRYkpuxKVaNvGjf3QIhgHf7ThRvZODo6DjGNvzze87PjgREamTlNyIy12sFH6seuUYAPoXTgvfqXpTIiJio+RGXG5Mtwi8PczsP5lFfHJG9U6OHg4h7SA3A3Z95YzwRESkjlFyIy4X1MiLa7qEATUox2A2X1zUb/P70LBKpYmISDmU3IhbKJo19d2OE1is1UxQek4Az0ZwcjckbnRCdCIiUpcouRG3MLxTGMF+XpzKzGX9oTPVO7lRE+heWG9K08JFRBo8JTfiFrw9zdzQPQKA+duq2TUFF7um4lVvSkSkoVNyI26jqGtq0Z4UzudVs6RCRA9o2R+s+bDtYydEJyIidYWSG3EbvVs1oVWIH+fzLPyypwatL/0L601tdZ96UxarwYZDZ/lux3E2HDpb/fFEIiJSbUpuxG2YTCbG17QcA0DMOFu9qYzjsP9nB0dXfYt2J3Pl7OVMeHcjj36xgwnvbuTK2ctZtDvZ1aGJiNRrSm7ErRR1Ta05cJpTmTnVO9nTB3pPsj3f/J6DI6ueRbuTmTpvG8nppe8hJT2HqfO2KcEREXEiJTfiVqKbNaZnVDBWA37YWYMEoG9hvanDK+HMAYfHZw+L1eC5H+IprwOqaNtzP8Sri0pExEmU3Ijbubn3xXIM1RbcCjqOtj13UetNXEJqmRabkgwgOT2HuITU2gtKRKQBUXIjbueG7pF4mk3sPp7BgZOZ1b9A0bTwHZ9BXrZjg7PDsXPn7Tqu2t1uIiJiFyU34nZCGnszvFMoUMOBxW2vgpC2tnpTv9ZevanEs+f528J4nvlut13HhwX4OjkiEZGGScmNuKXxJcoxWKs7NqVUvan3nFpvyjAMVu8/zZS5mxn2jxW8tzaB8/lWPMymCs8xARFBvvSPDnFaXCIiDZmnqwMQKc+ILs0J8PHkeNoF4o6kckXbptW7QM87YdkLtnpTSZug1RUOjS8rt4Bvtx1j7vojHD59setraMdQJg9qTU6elWmfbQMod2DxM2NjKk2ARESk5pTciFvy9fJgTLdwvtpyjAXbj1c/uWnUBLrdCts/sbXeOCi5OXw6i483HOWbrcfIyrUtFOjv48mtfVpy98DWtAv1Lz52jrk3z/0QX2pwcSMvD/55ew9Gx0Y4JB4RESnLZBhObLN3QxkZGQQFBZGenk5gYKCrw5FKbDh0lgnvbiTA15PNT43A18ujehc4sQP+OwzMXvDHePAPK/cwi9UgLiGVU5k5hAXYuotKtqpYrQar9p9m7vojrNp/unh729DG3DOwDbf0aYm/T/l/JxRde9Phs7y+7ABeHibW/+UaQgN8qncvIiINXHU+v9VyI25rQHQIkUG+nEjPYflvp7iuWzVbOyJ7Qst+cGwzbPsIhj5e5pBFu5PLtK5EBPnyzNgYBrVvxtdbjvHJhiMcOWubAWUywdWdwrhnUBuubN8McxVdSx5mEwPbNeWKtiGs3H+aHUlpzNt4lD9c27F69yIiInbTgGJxW2aziXGFA4u/rUmlcIB+hfWmtswtU2+qolWEk9NzeHjeNvr+bSkvLIznyNnzBPp6cv+V0ax8bDjvT+7H0I6hVSY2JZlMJu4fEg3AJxuPkpNvqdn9iIhIlZTciFsrKsewct8pUrPzqn+BmHHg1xQyjsH+RcWbK1tFuEhegZUOYY35+02xbHzyGv7fDTG0btq4+jEUGt01nBbBjUjNzqvZFHcREbGLkhtxax2bB9A1MpACq8GPv56o/gW8fMutN7V6/+lKVxEu8vy4WCYOaI2f9+X34Hp6mLl3cBsA3ltzuPpT3EVExC5KbsTtFbXefLzhCN/tOM6GQ2erVZfJ6DMZAxMcXsE/v/iR695Yw31zN9t17qnM3BrFXJHb+0UR4OPJodPZpQYni4iI42hAsbi9gMKZSAdOZfPoFzuAi4N+y5tSnZVbwM6kNLYnnmNbou3fly29uNZjG0G7Pya+YJLdr+3oVYQDfL24o38U765J4N01h7mqc/kzuEREpOaU3IhbW7Q7mb98u6vM9pT0HKbO28bbd/amY3hAqURm/8lMLm3Y+cJrJNeyjTt91tLylhfp2iaSW99ZT0p6TrnjbkxAuJNWEZ48OJoP1h1h/aGz7DmRTtfIIIe/hohIQ6bkRtxWZYN+i7ZN+2xbuftbBDeid+sm9IoKpnfrJsSEj4I5X+GbepiR1jXQZDLPjI1h6rxtmCi9inDRHChnrSLcIrgR13WL4IedJ3h/bQKv3dbT4a8hItKQKbkRtxWXkFrloF8D8DKb6NWqCb1aBdOrVRN6twomLLCc7qS+U+CXp2wDi3vfw+jYCObcVXYV4fBKurwc5f4ro/lh5wl+2HmCJ0Z3pnl58YqISI0ouRG3dSqz6tlMALNv7c7NvVtWfWDPO2H5C5CyC5LioNUARsdGcG1MeKUrFDtDj6hg+rcJIe5IKh+tP8KfR3d26uuJiDQkmi0lbsvewbwRQY3su6BfiK3eFJSaFl60ivC4ni0Y2K5prRW0nFK4qN+nmxI5n1dQxdEiImIvJTfitvpHhxAR5EtFqYYJ26ypag367Xe/7d/4BZDl2qnYI7o0p3VTP9Iv5PPN1mMujUVEpD5RciNuy8Ns4pmxMQBlEpwaD/qN7AUt+oIlz1ZvyoU8zCamXGlrvflgbUK11u4REZGKKbkRt1Y06Dc8qHQXVXiQL3Pu6l2zQb9FrTdbPgSra2s83dqnJUGNvDhy9jxL9550aSwiIvWFBhSL23P4oN+uN9lmTRXVm+p8vWMDrgY/b08mDmjFv1ce4v01CYzqGu6yWERE6gu13Eid4NBBv16+0Otu2/MSA4td5Z5BbfDyMBF3JJWdSWmuDkdEpM5TciMNU997ARMcWg5nDro0lOaBvoztEQnAe2sTXBqLiEh9oORGGqYmbaDjKNvzLR+4NBSgeGDxT7uSOZ52wcXRiIjUbUpupOEqGli8Yx7knXdpKF0jgxjUrikWq8Hcda5tvbFYDTYcOlujCuyuUhdjFhHn0YBiabjaXQNNouFcAqx4ESJ7gn9zaD0IzB61Hs4DQ9qy/tBZvohL4vfXdCDA16vWY1i0O7lMOYrKKrC7g7oYs4g4l1pupOEym6HVFbbnG96E/02Bj26A12Mh/vtaD2dYx1DahTYmM7eALzcn1frrL9qdzNR528rU8yqqwL5od3Ktx1SVuhiziDifkhtpuOK/h51flN2ekQxfTar1BMdsNnH/kLYAfLjuCAUWa629tj0V2J/7Id6tunvqYswiUjtcmtysXr2asWPHEhkZiclkYsGCBVWe8+mnn9KjRw/8/PyIiIjgvvvu4+zZs84PVuoXqwUWPQGVfTQu+kutL/J3U68WhDT25njaBRbvqb1F/aqqwG4Ayek5xCWk1lpMVamLMYtI7XBpcpOdnU2PHj14++237Tp+3bp1TJo0iSlTprBnzx6+/vpr4uLieOCBB5wcqdQ7R9dDxolKDjAg4zjMvR5++attNePDKyEt0akJj6+XB3dd0RqAd9ccxjBqp9XB3grs9h5XG+yOOcN9YhaR2uHSAcVjxoxhzJgxdh+/YcMG2rRpw+9//3sAoqOjeeihh5g9e7azQpT6KsvOVpHEDbZHSR7eENwKQtraBiSHtIWQaNvzJq3B0+eyQrv7ita8s+oQO5LS2JZ4jj6tq1EYtIbsrcBu73G1wd5Y/rvmMO3C/IltEeTkiETEXdSp2VIDBw7kySef5KeffmLMmDGcOnWKb775huuuu67Cc3Jzc8nNzS3+OiMjozZCFXfn39y+4/o9ACazbUZV6mE4d9RWdPPsQdujDBMEtbyY7JRMfEKiwSegypcMDfDhpp4t+HJLEu+uTqDP3c5Pbs5m5Va634Stnle1KrA7WVHV+Mq6pgD2nMjghjfXclOvFjw2qhMtghvVUoQi4ip1KrkZPHgwn376Kbfffjs5OTkUFBQwduzYSru1Zs2axXPPPVeLUUqd0HoQBEbaBg+XO+7GZNs/ZnbpaeFWi627KvUwpBYlPAmQesT2PD8b0pNsj4TVZS/bOLREi0906dYfvxAw2cpKTBkSzZdbklgcn8LRs9m0btrYGd8FAH7YeYIZX+4oeeflfkeqXYHdyYqqxj88b1uZfUVRvjA+ls1HUvluxwnmbz/Oj7uSuXdwGx4Z3p6gRrU/1V5EaofJqK1O/SqYTCbmz5/P+PHjKzwmPj6eESNG8Ic//IFRo0aRnJzM448/Tr9+/Xj//ffLPae8lpuoqCjS09MJDAx09G1IXRL/vW1WFFD647zwo/G2jyHmRvuvZxiQfbqcxKfw+YUqBrb6BJZq5Xk/3sSSFD/69+7LH28eZpu67mDf7TjOH77cgdWAW3q35JrOYbzwY3yZ1pAHhkTz1PUxDn/9y7X7eDo3vLm2zPZL17n59VgaL/60l42Hbe9BEz8vfnd1B+66ojXenpo0KlIXZGRkEBQUZNfnd51Kbu6++25ycnL4+uuvi7etXbuWIUOGcOLECSIiql6wqzrfHGkA4r+3zZoqObg4sAWMfql6iY09LqRdTHaKurmKWnwyKxvcDIaHD6YmbS52c5Vs/QluBR7Vb4X4dtsxHvt6J1YDbuvbklk3d8fDbMJiNYorsK/af5pvtx0ntkUgP0y/EpPJfVpuAKZ9to0ff01mXI8I7ujfutKq8YZhsPy3U8z6+TcOnsoCoHVTP54Y3ZkxseFud28iUlp1Pr/rVLfU+fPn8fQsHbKHh63LwE1yNKlrYm6EztfbZk9lnXTuCsWNgqFRL4jsVXZf/gXbeJ7i1p7DGKkJnEiIJ8xyCi9LLpzZZ3tcyuRROM6nbTljfdqAd9kura+3JPHn//2KYcCE/lH8fXw3zIXJQFEFdoAhHUL5eVcKu49nsObAGYZ2DHXgN+TyJJzJ5uddtkX6pl7Vns7hVfwlZzJxTZfmDOsYyldbjvHakv0cPXueRz7dRu9WwTx1fZdaGbwtIs7n0uQmKyuLgwcvDspMSEhgx44dhISE0KpVK2bOnMnx48f5+OOPARg7diwPPPAAc+bMKe6WmjFjBv379ycyMtJVtyF1ndkDooe4NgavRhDW2fYoZALWb0niL99sp0dAJl/e2hyvjCOF3VwJF1uBCi5A2lHb4/CKstf2Dy/V2rP+XCDz4nIJMJpz4xUxPH9jbHFic6mQxt7c0T+KD9cdYc7KQ26V3Px39WGsBlzdOazKxKYkTw8zdw5oxbiekfx39WH+u/ow2xLTuGXOBkZ3DeeJMZ2JbnYxISzZklVRq5CIuBeXdkutXLmSq666qsz2e+65h7lz5zJ58mSOHDnCypUri/e9+eabvPPOOyQkJBAcHMzVV1/N7NmzadGihV2vqW4pqUtyCyxcOXsFpzNzef32nozvdcn/c8OAzJQS3VwJpVp/yEmv9PqGbzCmSwc2F7X+BISDycSJtAsMfXkFBVaD+Y8MolerJk68Y/ucysjhytkryLNY+frhgfRrU/MWl5MZOfxzyX6+2pKE1QBPs4mJA1rx+2s6sPlIqupWibiJOjnmprYouZG65q3lB/jHL/vpGhnIwt9Vc9zL+dTiFp7tO7dx8LddtDafpIvPWQLyz1R+rpefrVsrpC0rTzdmaYofTaM684fbRkFQFHi4ruF31s97+c+qw/Rt3YRvpg5yyDX3n8xk1k97WbHvNAC+nmZyCsqWwCj67s+5q7cSHJFapOSmEkpupK45l53HwJeWkZNv5fMHrigeD1MdH65L4Lkf4gHbzKcnr+uCKf88nDtStrUnNcE2ld2opLaV2fOShQxLtP40aQNezlvsL/1CPoNfWk5WbgHv39OXa7rYuWaRndYfPMPffownPjmzwmOK1v1Z+8TV6qISqSX1dkCxSEPUpLE3t/ZpybyNiby/9nC1k5v31hzmbz/uBeDhYe14YnQnW+uPd2No3tX2uFRBni3BKZH47Px1O35ZibTxOI2XNa8wETpczisWrhFUnPRcMrvL9/JWCp638ShZuQV0ah7AVZ3CbButFocNCh/Uvhn/7/oY7nxvU4XHlKxbVZNkU6TecuDP4uVQciNSB9w3OJpPNyWydO8pDp3Ool2ov13n/WfVIWb9/BsA069qz59GdrSvW8vTG5q2sz0KGV3TuPbtdXiZDdZM7Ux4QXI5Y32OQG6GbaHDjONwtOwaNPg1LT2+p2Ti0zi0eCHD8uTkW/hwXQIADw9vaxsIXe50/kgYPbvG0/lPV7FicxF3qrUl4nJO+FmsKSU3InVA21B/runcnKV7T/LB2gT+flO3Ks/598qDvLzINnX80Ws6MGNEh8tay6VnVDCD2jVl/aGzvLM9l2dvHFJ2lplh2Mb5XNrNVfQ8+zScP2t7HN9S9kW8/Uu3+JRMggJb8PWWJM5k5dEiuBE3dI8ssRDjJb3rGcm27dVdiLFQXay1JeJSTvpZrCmNuRGpIzYePssd/92Ij6eZDTOvIaSxd4XHvrnsAK8u2Q/AH0Z05NERHRwSw9oDZ7jr/U34eplZ98TVNPWvZpHQ3MzS09hLruKcfozyCz/YGB7eJFpDOVQQSnh0DDFdusGaf9gSpXIVdo/N2FXtZnGL1eDK2ctJSc+pMKLmgT6s/8s1GnMjYrXA67GlW2xKqfnPYkkacyNSDw2IDqFbiyB2HU/n041H+d01ZRMWwzB4fekB3lh2AIDHR3Vi2lXtHRbD4PZNi2P4aP0R/jiyU/Uu4BMAEd1tj0sV5EJaYvnlK84dwWTJozXHae1xHBJ3QGJVL2YUdo2tr/Y6RkV1q6bO21ZhrS1vDzOZOfkE+1WcZIo0CAeWVpLYwOX8LNaUiqqI1BEmk4n7h0QD8NGGo+QWWErtNwyD15bsL05s/jKms0MTm6IYHhluG4czd/0RsnILHHdxTx9o1gE6joIrHobrXoaJX8PvtmA8lcKkwPe5M+9Jlrd/EgY/CpG97bvu4RW2AdLVNDo2gjl39SY8qHTXU6i/D/4+niSdu8Cd727iXHb1ry1S56UmwMY58NGN8MUE+87JOuncmEpQt5RIHZJvsTL05RUkp+fw0NC2xEQGEhbgS782TXhtyX7+vfIQAP/v+i7cP6StU2KwWA2ufW0Vh89k89R1XXhgqHNep6QVv53i3rmbaeztwfq/XEOQnxckrIGPbrDvAj6B0O4q6Dga2l8L/vavtFzeCsWHTmdx57sbOZOVR5eIQD69f0Cl3YQidZ7VAsc2w76fYf8iOP1b9a9xz8LLarnROjeVUHIjdd0fv9zBt9uPl9rW2NuD7DxbS87TN8Rw35XRTo3hq8222lTNA31Y/eer8PF07lTP297ZQNyR1NLVyYv7+ZOpcKyOl5/tcb7kgoUmaNHHluh0HAXh3SqdoVWRAyczmfDuJs5k5dI5PIDPHrhCCY7ULzkZcGi5LZk58Evp8W0mD9s0746jocNI+GRcJT+LtT/mRsmNSB2yaHcyU+dtq3CQ6x39onjplnLGszhYXoGtBSklI4eXbu7GHf1bOe21thxJ5dZ3NuDlYWLNn68u3U1UPEMDSv9SLUxWbvsYOt8AJ7YX/oJeDMk7S79AQCR0HGn7JR09DLz97I7t4KlM7vjvxQTn0/sHVH+QtYg7OXcE9i2y/bwcWQvW/Iv7fIOhw7WFLaDXQKMSpVjs+Vm8zNlSSm4qoeRG6qqiGTwl6xxdKqIWV80tWhwwulljlv5xmNNe8/6PNrN076mKE7dy19ZoAaNfKv+XacYJ21+h+3+xjcfJP39xn4cPRA+1teh0HGVbhbkKB09lMeHdjZzOzKVT8wA+fWAAzZTgSF1htcCxLbD/Z1tSc3pv6f1N29uSmU5jIOqKysuuVPdnsZqU3FRCyY3UVRsOnWXCuxurPK6mJRqqKzu3gMGzl5N2Pp+37+zN9d0dX2dpX0omo15fjckEy/44jLYVLV5Y01VR83Nsf50eWGz7SzXtkilYYTGFic5oaNmvwmseOp3FhP9u5FRmLh2b+/PZA1cowRH3lZtp627aV9iaeWl3U6uB0Gk0dBwDzao5KcGJKxRrKrhIPWTvari1tWpuYx9P7hnYhjeWHeDfKw9yXbfwy1oksDz/WWUbID0mNrzixAZsvzxrMlDRyxc6jLA9xrxsGyS5f7HtkbQRTsXbHmv/aWuCb3+tLdm5pEm+Xag/Xzx4BRPe3cj+k7ZE57MHriA0wDUJTnmDoLUeTwN37qgtgS/qbrKUmOXnG1T4f3u07WehZHdTddX0Z9HBlNyI1BHuuGru5EFt+O/qw+w5kcGaA2cY2tH+WUhVOXbuPN/ttDVvPzysXRVHO4DJBGFdbI8rZ9hWWj64zPaX7YElcOEc7PrK9jB5QKsrLrbqNOtI21B/vnhwIBP+u5EDhV1Vnz0woNZXMV60O5nnfogv1X0ZEeTLM2NjVMW8IbFa4PjWi7ObTsWX3h/SztbV1HG07f+yh5dr4nQSdUuJ1BFVrZrrqkrVz/8QzwfrEriibQhfPDjQYdd99vs9zF1/hCvbN2Pe/QMcdt0asRTAsbjCv3x/KTsuIbh18eyrIwG9uOP97aRk5NAutDGfP3AFYYG1k+BUNOC86H/DnLt6K8Gpz3Iz4dCKwv+ni0vPEixOyAvHzzRzzKrltUljbiqh5EbqsqIPLyh3PoJLPryS0y8w9OUV5FsMvn1kEL1bXUaTdqGzWbkMnr2cnHwrn94/gMHtmzkgUgc6d8SW5BxYDAmrSzfxezUmO2oIbyS2ZX5WVwJCW/JFLSQ4VQ04d1XyK06Wllg4u+nnst1NPkG2LtROY6D9CPALcV2cDqDkphJKbqSuc8duh8e/3snXW49xbUxz3p3U97Kv99ov+/jX8oN0bxnEd9MGO3wsj0PlZkHCqotjdbJSSu3+1RrNdp8B3PB/99K0fX8wO2dheHcbcC5OYrXaupuKZjed2lN6f0hb20DgTqNtA4PrUXeTBhSL1GOjYyO4NibcrQaMPjSsHd9sO8aS+JPsP5lJx+YBNb5WVm4BH204CsDUYe3cO7EB8PGHztfbHoZhW0fnwC+2roHjW+luTqB7fgJ89gUWv1A8OhWO02k73FZrywGsVoNNCRUVEC2ttgaciwPlZtmWLSia3ZR9+uI+k9k2Rbt4dlOHGi1KWd8ouRGpgzzMJrf667t9mD+jYsJZtCeFd1Yd4rXbetb4Wl/EJZJ+IZ+2zRozsmu444KsDSYTRPa0PYb9GTJPkrrzR3Yu/4p+lu34nz8N2+fZHmYvaHNl4Vidkba/uKsh32Jl4+GzLN6Twi97TnIqM9eu82p7gLPUUFrSxdlNl3Z9+gTaups6jrEtqlfHu5ucQd1SIuIQO5PSGPf2OjzNJlY+PpyWTexf6bdIboGFoS+v4GRGLrNv6cbt/Zy38nFtSko9z13/WUPLzO2M99vNTY134Zl+pPRBzTraZl91GFXh7JULeRZW7T/NL3tSWLr3JBk5FwuX+nt7UGAY5ORbK43lpl6RPHV9jNbhcTdWK5zYdnF208ndpfc3ib44u6n1oHrV3WQvjbmphJIbEeeZ+N5G1h08y+RBbXj2xq7VPr+2a1bVpmPnznPHfzdy7NwFWoc04qtbQwlNXknmrz8ScGozZqNEhfWigaAdR5HRcjjLEgtYtDuFVftPl0pemvl7c21MOKO6NmdQu2Ys/+1khQPOS34d1MiLJ0Z35o5+UZjdaHBxg1ufJy+7cHbTz7YB6tmnLu4zmSFqQInZTR0bfHeTkptKKLkRcZ51B88w8b1N+HqZWffE1dWqs2SxGlz7z1UcPl171cZr27Fz55nw7kaSUi/QzN8bk8nE6cxcAjjPEPOv3OD7KyM8d+Kdd674HKthYpvRgeWWXiy39iIrqCOjYiMYHRtO71ZNynz4VzbgvHmgL0/N3018cgYAvVsF87fx3YiJdP3vQmcOlHerpCn9WGHrTNFMuxLdid4BJWY3XQuN3afr2R0ouamEkhsR5zEMg3Fvr+PXY+n87ur2/GlkJ7vPXbQ7mYfnbSPQ15P1M6/B36d+Dgk8nnaBcW+t5UxWXrn7zVjpYTrE1R7buca8nRjz0VL7jcCWmIpqX0UPBa9GZa5R2Yd5gcXKRxuO8tov+8jOs+BhNnHvoDbMuLajy77nzlyfx+WzC63WwsKthbObTu4qvb9JG9vYmY6joPVg8FRl+YoouamEkhsR5yqZpKz7y9UE+FY9NsAwDMa/vY6dNUiK6hqL1WDgrGVVDgDu0TKIMd0iuK6VhVZn19r+0j+8EgpKzHbybARth0GHkbYPx6CWdseRkp7D8wv38NMu29T1og/8UV0dX0ajMs5cn8dlixrmZdveq30/22bOZZ0s8eJmaNn/4uym0E4NvrvJXkpuKqHkRsS5rFaDEYXdS09e15kHh1ZdOmH9wTPcWcPurLrmstajyb8ACWsurkCbcaz0/ubdbDOvOo6GFn3sKli4Yt8pnv5uN0mpFwC4unMYz93YlaiQ6g8Ir4kNh84w4d1NVR7XIawxoQG++Hp54OtlLvzXA1/Pkl+bi7d5e5p57oc9nDufX+71HL6oYfrxi60z5XY3XV04u2mkuptqSOvciIjLmM0mHh7Wjj9/8yvvrUngnkFtqhwYPKewQObtfaPqdWIDl1kA1atRYfIy0ramzqn4i4lOUpyty+PkLljzKvg1tX2QdhhpG8fhG1Tu61zVKYwlfxjG2ysO8s6qQyz/7RTrD53hd1d34IEhbfH2dPyig2eycll38Ayr9p9mSfzJqk8ADpzK5sCpbIfFYADJ6TnEJaTWbFkFqxWSt19cHTjlku6m4NYlZjepu6m2KbkREYcb37MF/1yyn+T0HL7ddpwJ/Sue0r3rWDprDpzBw2zi/iH1bxDxpRxWANVkguZdbY8hf4Lss3BwqS3ZObgMzp+FnZ/bHmZP22q1RYU+m7Yv1RXi6+XBn0Z2YlzPFvy/BbvYeDiVVxbvY/724/xtfCxXtL344V+Twbl5BVa2Hj3H6gOnWXPgNLuPZ9j1PSjpj9d2pHVTP3LzrVzIt5CTbyEn30pOQYnnxdstHDt3gQOnsqq8brUWNcw7b+tu2l84ILhkdxMmiOp/cXZTaGd1N7mQuqVExCneX5vACwvjadPUj2V/Gl7hB+C0T7fx465kburVgn/e3rN2g3SBWimAasmHpE0XW3XO7C+9P6StbT2dcgaxGobBgh3H+dvCvZzNtg16vrl3C568rgtbjqTaNTjXMAyOnD3P6v2nWb3/NBsOn+V8nqVUCF0iAhnasRlD2jXjsW92cjIj16HfD3u7/67uHMbfxscSGVx2YDZQ2N1U+H1MWFV6zJO3P7S72pbMdBgJjd2sBlo9ozE3lVByI1I7snMLGDx7OWnn83nrzl7c0D2yzDEJZ7K5+tWVGAYsmjGEzuEN42ey1gugph62raOyf5GtuKK1xDgU7wBod1XhAoIjwT8MgPTz+by8+Dc+i0vEMMDP26NMglIy5ldv64GftwerD5xh9f7THDt3odRxzfy9GdIhlCEdmnFlh2alWqac8f2oKoksydvDzJ0DWvHI8HaE+XtD8g7b92rfz5Dya+mDg1oVDgYebVth2rN+d6O6EyU3lVByI1J7Xl+6n9eXHiAmIpAff39lmVk4M7/9lc/jkrimcxjvT+7noihdw2VTlHMzC7tWFpVdOA5sA5GLWnUierA9KY0nv93F3pTMar2Ml4eJvq1DGNrRltDERARWumCgM74fVSVNfxrZkTUHzrAzIZnB5t2M9NzOdT6/EpB/pvTRLftdnN0U1kXdTS6i5KYSSm5Eas+57DwGz17O+TwLH93Xn2EdQ4v3nczIYcjsFeRZrHzz8ED6tml49XFcvric1VrYSrHYluwk7yi9PyACOlxLvP9AbvnFhwtUPg4oIsiXUV3DGdqxGQOim9K4muvmOOP7UVHS9OKIplxl2oaxfxHWQyvxKDG7Kdvw5XjTK2g54Gb8ul4H/qHlXVpqmZKbSii5EaldLyyM5/21CQyIDuHLhwYWb5/1017+s/ow/do04euHB7kwQimWmVJY0XyxrSxA/sXZSbmGFxutXVhmta2UfMwIK3P6G3f0ZFzPFrUZsV0sVoO4w2fJPbaNdqlraHl6FabknaWOMYKiOB42nP+mdOCL063Jw4sAH0+mDInmviujCbRjvSZxbsKu5KYSSm5Ealdy+gWGvryCfIvB/6YOok/rJqSfz2fQS8vIzrPwweS+XN25uavDlEsV5NrG5+xfTE78T/hmJZXavd/aguXW3iyz9GKb0QELHuWvzeNK+Rfg8KqLs5syk0vsNEHLvhdnN4XFgMmEYRgsiT/Ja0v281thV1xQIy8eGtaWewa2KdMa5fLWNzfi7K5WJTeVUHIjUvv+/M1OvtpyjGs6h3H/kLbM23iEH3el0Km5P4tmDK3VFXGl+iwWKxNf+oQe5zdwtcd2+pj242m6WMAzzWhMnEdvRoybhLnDCPBzYRdjRnLheKJFtsSmoMTAZq/GtsHTRbOb/Mu2PhWxWg1+2p3MP5fs59BpWwtW08beTB3ejruuaI2vl4frSzu4kdpYDVrJTSWU3IjUvkOns7jm1VVltgc38uKlW7o1uA+Cuqjk4NwAshhm/pWrPbYz3LyTJqYS68kUV7MuXFPH2eu9GAYk77w4u+nScUOBLS8OBm5zJXjZt85QEYvV4Lsdx3lj2QGOnj0PQFiAD1d1CuOrLUm1X9rBDTmzhEZJSm4qoeRGpPYV1Zu6VEP8IKjLymupaBHoxauD8rmiYLNt9tWpPaVPCmp1MdGpKrmwWuDoetvieP7NofWg8ktI5F+wlTgoqq6deaLETpNtxlfRdO3msQ5JrvItVr7ddox/LTvI8bQLlR7r8NIObu6ySopUg5KbSii5EaldtfVXndSOKseYpCXaEo4Dv9i6hUrWWPLyg7bDL66pE1hi7aP472HRE5BRIlEJjITRsyHmRttg56LF9A6tuKS7yc+2mF7H0bZrV9LddLlyCyzM/vk3Plh3pMpj3W4MkpN8t+M4j36xo8rjLnfAuWpLiYjbiEtIrTCxAQfU+JFa5WE2Vf4+BbeC/g/YHnnZthaWojV1Mk/Avp9sD4Dw7raExMsPlj0Hl3byZJyAr+6GJtFwLqH0vsAWFwcDtxlS7e6mmvLx9KBHVLBdx365ORGzCbq2CMK/mtPioe4MVnZYSREHUnIjIk51WYUipW7zbmxLPjqNsY2NSdlV2KqzGI5tsa3+e+kKwOUpSmxa9LGNnek4CsK7uWwxPXs/pBfsOMGCHScwmaBts8Z0bxlMtxZBdG8ZRExkIH7eFX8E16XByi2bNMJsAmsF/UBFrbP9o2tvoLmSGxFxKnf8q05cwGSCiO62x7DHIes0HFwC2+fB0XVVn3/bJ7buKTfQPzqEiCDfSks7BPh4ckXbEHafyCA5PYdDp7M5dDqb+duPA2A2Qfswf7q1CKZ7yyBiWwQRExFII2+PCmcepaTnMHXeNrcao3YmK5d7PoirNLEBeGZsTK22Oim5ERGnquqDwBV/1Ykb8A+FnneCh7d9yY0lz/kx2cnDbOKZsTFMnbcNE+WXdnjl/7oXJyCnM3PZfTydX4+ls+t4OruOp3EyI5f9J7PYfzKL/207Vnzd9qGNSUy9UO7PilF4/ed+iOfamHCXd1Gln8/n7vfjOHwmmxbBjXjkqna8tfxgqdamcBe1Nim5ERGnsueDoLb/qhM34m/nAo72HldLRsdGMOeu3mW6jsr7MA8N8OGqzmFc1fniQOeTGTnsKk52bInPmaxc9p3MojLuMkYtO7eAyXPj2JucQTN/H+bdP4DoZo25o18rtxgnpNlSIlIr6tIYAqlFVgu8HmtbfK+itr3ASJixq/xp4S7mqEG/hmFwMiOX99Yc5r21CVUe78pSFzn5Fu6bu5n1h84S1MiLLx+6gs7hzv881WwpEXE7o2MjuDYm3C3+qhM3YvawTff+ahJU1LY3+iW3TGzAjtljdjKZTIQH+XJNl+Z2JTeuGqOWb7Ey/bNtrD90lsbeHnx0X/9aSWyqS8mNiNQaR30QSD0TcyPc9nEF69y85DYDiWuDPYOVgxt5uWSMmsVq8KevdrJ07yl8PM28d08/eto5Lb62qVtKRETcg70rFNdzJUtdVPQBPXlQG566vgteHuZaickwDJ6cv4vP45LwNJt4d1LfUmOIakN1Pr9r57siIiJSFbMHRA+Bbrfa/m2AiQ1cHKwcHlS66ykiyJcxseEAzF1/hInvbqqV9aEMw+DvP+7l87gkzCZ4445etZ7YVJdabkRERNxQRYOVf9mTwp++2klmbgHNA33498Q+9GndxGlxvLH0AP9cuh+Al2/pzm39opz2WpVRbalKKLkREZG67vDpLB76ZCsHTmXh5WHi6bFduWtAK0wOXrX5vTWH+duPewF4+oYY7rsy2qHXrw51S4mIiNRjbUP9WTBtMNd3iyDfYvDXBbt57Otfycm3OOw1vohLLE5s/nRtR5cmNtXl0uRm9erVjB07lsjISEwmEwsWLKjynNzcXJ566ilat26Nj48Pbdq04YMPPnB+sCIiIm6ksY8nb93Ziyev64zZBP/bdoxb5qwnKfX8ZV/7h50nmDl/FwAPDW3L9KvbX/Y1a5NLk5vs7Gx69OjB22+/bfc5t912G8uWLeP9999n3759fP7553Tq1MmJUYqIiLgnk8nEg0PbMW/KAEIae7PnRAZj31rLmgOna3zNZXtP8ocvd2AYMHFAK/4yprPDu7uczW3G3JhMJubPn8/48eMrPGbRokXccccdHD58mJCQms3x15gbERGpj46nXeCReVvZeSwdswkeG9WJqcPaVSsxWX/oDJM/3ExegZXxPSN57baemN1koc16O+bm+++/p2/fvrz88su0aNGCjh078thjj3HhwoUKz8nNzSUjI6PUQ0REpL5pEdyILx8ayB39orAa8PKifTw8byuZOfl2nb8t8Rz3f7SFvAIr18Y055X/6+E2iU111ank5vDhw6xdu5bdu3czf/58Xn/9db755hseeeSRCs+ZNWsWQUFBxY+oKNdMYRMREXE2Xy8PXrqlO7Nu7oa3h5nFe04y7u11HDyVWel58ScymPxBHOfzLFzZvhlvTuhVawsEOkOd6pYaOXIka9asISUlhaCgIAC+/fZbbr31VrKzs2nUqFGZc3Jzc8nNzS3+OiMjg6ioKHVLiYhIvbYjKY2p87aSnJ5DY28PXr2tB6NjI8qsn9PU35s7393Imaw8+rRuwidT+uPn7X7Vmept4cyIiAhatGhRnNgAdOnSBcMwOHbsGB06dChzjo+PDz4+PrUZpoiIiMv1jArmh99dyfTPtrHxcCoPz9vGqK7N2ZmUTkrGxZWNzSawGhATEcgHk/u5ZWJTXXWqzWnw4MGcOHGCrKys4m379+/HbDbTsmVLF0YmIiLifpr5+zBvygAeGGJbo2bxnpOlEhuwJTYA9w5uQ1Ajr9oO0SlcmtxkZWWxY8cOduzYAUBCQgI7duwgMTERgJkzZzJp0qTi4++8806aNm3KvffeS3x8PKtXr+bxxx/nvvvuK7dLSkREpKHz9DDzlzFdCK4icXltyX4sVrcYqXLZXJrcbNmyhV69etGrVy8A/vjHP9KrVy+efvppAJKTk4sTHQB/f3+WLFlCWloaffv2ZeLEiYwdO5Z//etfLolfRESkLohLSCXtQuWzppLTc4hLSK2liJzLpR1rw4cPp7LxzHPnzi2zrXPnzixZssSJUYmIiNQv9lYPr40q47WhTo25ERERkeoLC/B16HHuTsmNiIhIPdc/OoSIIF8qWpLPBEQE+dI/umar/7sbJTciIiL1nIfZxDNjYwDKJDhFXz8zNgaPOroi8aWU3IiIiDQAo2MjmHNXb8KDSnc9hQf5Mueu3oyOjXBRZI5X91fqEREREbuMjo3g2pjwUisU948OqTctNkWU3IiIiDQgHmYTA9s1dXUYTqVuKREREalXlNyIiIhIvaLkRkREROoVJTciIiJSryi5ERERkXpFyY2IiIjUK0puREREpF5RciMiIiL1ipIbERERqVca3ArFhmEAkJGR4eJIRERExF5Fn9tFn+OVaXDJTWZmJgBRUVEujkRERESqKzMzk6CgoEqPMRn2pED1iNVq5cSJEwQEBGAyOaZQWEZGBlFRUSQlJREYGOiQa7qT+n5/oHusD+r7/YHusT6o7/cHzrtHwzDIzMwkMjISs7nyUTUNruXGbDbTsmVLp1w7MDCw3v5nhfp/f6B7rA/q+/2B7rE+qO/3B865x6pabIpoQLGIiIjUK0puREREpF5RcuMAPj4+PPPMM/j4+Lg6FKeo7/cHusf6oL7fH+ge64P6fn/gHvfY4AYUi4iISP2mlhsRERGpV5TciIiISL2i5EZERETqFSU3IiIiUq8oublMb7/9Nm3atMHX15cBAwYQFxfn6pBqbNasWfTr14+AgADCwsIYP348+/btK3XM8OHDMZlMpR4PP/ywiyKunmeffbZM7J07dy7en5OTw7Rp02jatCn+/v7ccsstnDx50oURV1+bNm3K3KPJZGLatGlA3Xz/Vq9ezdixY4mMjMRkMrFgwYJS+w3D4OmnnyYiIoJGjRoxYsQIDhw4UOqY1NRUJk6cSGBgIMHBwUyZMoWsrKxavIuKVXZ/+fn5PPHEE3Tr1o3GjRsTGRnJpEmTOHHiRKlrlPe+v/TSS7V8JxWr6j2cPHlymfhHjx5d6hh3fg+h6nss7+fSZDLxyiuvFB/jzu+jPZ8P9vwOTUxM5Prrr8fPz4+wsDAef/xxCgoKHB6vkpvL8OWXX/LHP/6RZ555hm3bttGjRw9GjRrFqVOnXB1ajaxatYpp06axceNGlixZQn5+PiNHjiQ7O7vUcQ888ADJycnFj5dfftlFEVdf165dS8W+du3a4n1/+MMf+OGHH/j6669ZtWoVJ06c4Oabb3ZhtNW3efPmUve3ZMkSAP7v//6v+Ji69v5lZ2fTo0cP3n777XL3v/zyy/zrX//inXfeYdOmTTRu3JhRo0aRk5NTfMzEiRPZs2cPS5YsYeHChaxevZoHH3ywtm6hUpXd3/nz59m2bRt//etf2bZtG99++y379u3jxhtvLHPs888/X+p9/d3vflcb4dulqvcQYPTo0aXi//zzz0vtd+f3EKq+x5L3lpyczAcffIDJZOKWW24pdZy7vo/2fD5U9TvUYrFw/fXXk5eXx/r16/noo4+YO3cuTz/9tOMDNqTG+vfvb0ybNq34a4vFYkRGRhqzZs1yYVSOc+rUKQMwVq1aVbxt2LBhxqOPPuq6oC7DM888Y/To0aPcfWlpaYaXl5fx9ddfF2/bu3evARgbNmyopQgd79FHHzXatWtnWK1WwzDq9vtnGIYBGPPnzy/+2mq1GuHh4cYrr7xSvC0tLc3w8fExPv/8c8MwDCM+Pt4AjM2bNxcf8/PPPxsmk8k4fvx4rcVuj0vvrzxxcXEGYBw9erR4W+vWrY1//vOfzg3OQcq7x3vuuccYN25chefUpffQMOx7H8eNG2dcffXVpbbVpffx0s8He36H/vTTT4bZbDZSUlKKj5kzZ44RGBho5ObmOjQ+tdzUUF5eHlu3bmXEiBHF28xmMyNGjGDDhg0ujMxx0tPTAQgJCSm1/dNPP6VZs2bExsYyc+ZMzp8/74rwauTAgQNERkbStm1bJk6cSGJiIgBbt24lPz+/1PvZuXNnWrVqVWffz7y8PObNm8d9991XqkhsXX7/LpWQkEBKSkqp9y0oKIgBAwYUv28bNmwgODiYvn37Fh8zYsQIzGYzmzZtqvWYL1d6ejomk4ng4OBS21966SWaNm1Kr169eOWVV5zS1O9MK1euJCwsjE6dOjF16lTOnj1bvK++vYcnT57kxx9/ZMqUKWX21ZX38dLPB3t+h27YsIFu3brRvHnz4mNGjRpFRkYGe/bscWh8Da5wpqOcOXMGi8VS6k0CaN68Ob/99puLonIcq9XKjBkzGDx4MLGxscXb77zzTlq3bk1kZCS//vorTzzxBPv27ePbb791YbT2GTBgAHPnzqVTp04kJyfz3HPPMWTIEHbv3k1KSgre3t5lPjCaN29OSkqKawK+TAsWLCAtLY3JkycXb6vL7195it6b8n4Oi/alpKQQFhZWar+npychISF17r3NycnhiSeeYMKECaUKEv7+97+nd+/ehISEsH79embOnElycjKvvfaaC6O13+jRo7n55puJjo7m0KFDPPnkk4wZM4YNGzbg4eFRr95DgI8++oiAgIAy3d515X0s7/PBnt+hKSkp5f6sFu1zJCU3Uq5p06axe/fuUmNSgFJ93N26dSMiIoJrrrmGQ4cO0a5du9oOs1rGjBlT/Lx79+4MGDCA1q1b89VXX9GoUSMXRuYc77//PmPGjCEyMrJ4W11+/xq6/Px8brvtNgzDYM6cOaX2/fGPfyx+3r17d7y9vXnooYeYNWtWnVjm/4477ih+3q1bN7p37067du1YuXIl11xzjQsjc44PPviAiRMn4uvrW2p7XXkfK/p8cCfqlqqhZs2a4eHhUWYk+MmTJwkPD3dRVI4xffp0Fi5cyIoVK2jZsmWlxw4YMACAgwcP1kZoDhUcHEzHjh05ePAg4eHh5OXlkZaWVuqYuvp+Hj16lKVLl3L//fdXelxdfv+A4vemsp/D8PDwMoP8CwoKSE1NrTPvbVFic/ToUZYsWVKq1aY8AwYMoKCggCNHjtROgA7Wtm1bmjVrVvz/sj68h0XWrFnDvn37qvzZBPd8Hyv6fLDnd2h4eHi5P6tF+xxJyU0NeXt706dPH5YtW1a8zWq1smzZMgYOHOjCyGrOMAymT5/O/PnzWb58OdHR0VWes2PHDgAiIiKcHJ3jZWVlcejQISIiIujTpw9eXl6l3s99+/aRmJhYJ9/PDz/8kLCwMK6//vpKj6vL7x9AdHQ04eHhpd63jIwMNm3aVPy+DRw4kLS0NLZu3Vp8zPLly7FarcXJnTsrSmwOHDjA0qVLadq0aZXn7NixA7PZXKYrp644duwYZ8+eLf5/Wdffw5Lef/99+vTpQ48ePao81p3ex6o+H+z5HTpw4EB27dpVKlEtStZjYmIcHrDU0BdffGH4+PgYc+fONeLj440HH3zQCA4OLjUSvC6ZOnWqERQUZKxcudJITk4ufpw/f94wDMM4ePCg8fzzzxtbtmwxEhISjO+++85o27atMXToUBdHbp8//elPxsqVK42EhARj3bp1xogRI4xmzZoZp06dMgzDMB5++GGjVatWxvLly40tW7YYAwcONAYOHOjiqKvPYrEYrVq1Mp544olS2+vq+5eZmWls377d2L59uwEYr732mrF9+/bi2UIvvfSSERwcbHz33XfGr7/+aowbN86Ijo42Lly4UHyN0aNHG7169TI2bdpkrF271ujQoYMxYcIEV91SKZXdX15ennHjjTcaLVu2NHbs2FHq57Jodsn69euNf/7zn8aOHTuMQ4cOGfPmzTNCQ0ONSZMmufjOLqrsHjMzM43HHnvM2LBhg5GQkGAsXbrU6N27t9GhQwcjJyen+Bru/B4aRtX/Tw3DMNLT0w0/Pz9jzpw5Zc539/exqs8Hw6j6d2hBQYERGxtrjBw50tixY4exaNEiIzQ01Jg5c6bD41Vyc5nefPNNo1WrVoa3t7fRv39/Y+PGja4OqcaAch8ffvihYRiGkZiYaAwdOtQICQkxfHx8jPbt2xuPP/64kZ6e7trA7XT77bcbERERhre3t9GiRQvj9ttvNw4ePFi8/8KFC8YjjzxiNGnSxPDz8zNuuukmIzk52YUR18zixYsNwNi3b1+p7XX1/VuxYkW5/y/vuecewzBs08H/+te/Gs2bNzd8fHyMa665psy9nz171pgwYYLh7+9vBAYGGvfee6+RmZnpgrspq7L7S0hIqPDncsWKFYZhGMbWrVuNAQMGGEFBQYavr6/RpUsX48UXXyyVGLhaZfd4/vx5Y+TIkUZoaKjh5eVltG7d2njggQfK/JHozu+hYVT9/9QwDOM///mP0ahRIyMtLa3M+e7+Plb1+WAY9v0OPXLkiDFmzBijUaNGRrNmzYw//elPRn5+vsPjNRUGLSIiIlIvaMyNiIiI1CtKbkRERKReUXIjIiIi9YqSGxEREalXlNyIiIhIvaLkRkREROoVJTciIiJSryi5ERERkXpFyY2INDht2rTh9ddfd3UYIuIkSm5ExKkmT57M+PHjARg+fDgzZsyotdeeO3cuwcHBZbZv3ryZBx98sNbiEJHa5enqAEREqisvLw9vb+8anx8aGurAaETE3ajlRkRqxeTJk1m1ahVvvPEGJpMJk8nEkSNHANi9ezdjxozB39+f5s2bc/fdd3PmzJnic4cPH8706dOZMWMGzZo1Y9SoUQC89tprdOvWjcaNGxMVFcUjjzxCVlYWACtXruTee+8lPT29+PWeffZZoGy3VGJiIuPGjcPf35/AwEBuu+02Tp48Wbz/2WefpWfPnnzyySe0adOGoKAg7rjjDjIzM537TRORGlFyIyK14o033mDgwIE88MADJCcnk5ycTFRUFGlpaVx99dX06tWLLVu2sGjRIk6ePMltt91W6vyPPvoIb29v1q1bxzvvvAOA2WzmX//6F3v27OGjjz5i+fLl/PnPfwZg0KBBvP766wQGBha/3mOPPVYmLqvVyrhx40hNTWXVqlUsWbKEw4cPc/vtt5c67tChQyxYsICFCxeycOFCVq1axUsvveSk75aIXA51S4lIrQgKCsLb2xs/Pz/Cw8OLt7/11lv06tWLF198sXjbBx98QFRUFPv376djx44AdOjQgZdffrnUNUuO32nTpg1/+9vfePjhh/n3v/+Nt7c3QUFBmEymUq93qWXLlrFr1y4SEhKIiooC4OOPP6Zr165s3ryZfv36AbYkaO7cuQQEBABw9913s2zZMv7+979f3jdGRBxOLTci4lI7d+5kxYoV+Pv7Fz86d+4M2FpLivTp06fMuUuXLuWaa66hRYsWBAQEcPfdd3P27FnOnz9v9+vv3buXqKio4sQGICYmhuDgYPbu3Vu8rU2bNsWJDUBERASnTp2q1r2KSO1Qy42IuFRWVhZjx45l9uzZZfZFREQUP2/cuHGpfUeOHOGGG25g6tSp/P3vfyckJIS1a9cyZcoU8vLy8PPzc2icXl5epb42mUxYrVaHvoaIOIaSGxGpNd7e3lgsllLbevfuzf/+9z/atGmDp6f9v5K2bt2K1Wrl1VdfxWy2NUJ/9dVXVb7epbp06UJSUhJJSUnFrTfx8fGkpaURExNjdzwi4j7ULSUitaZNmzZs2rSJI0eOcObMGaxWK9OmTSM1NZUJEyawefNmDh06xOLFi7n33nsrTUzat29Pfn4+b775JocPH+aTTz4pHmhc8vWysrJYtmwZZ86cKbe7asSIEXTr1o2JEyeybds24uLimDRpEsOGDaNv374O/x6IiPMpuRGRWvPYY4/h4eFBTEwMoaGhJCYmEhkZybp167BYLIwcOZJu3boxY8YMgoODi1tkytOjRw9ee+01Zs+eTWxsLJ9++imzZs0qdcygQYN4+OGHuf322wkNDS0zIBls3UvfffcdTZo0YejQoYwYMYK2bdvy5ZdfOvz+RaR2mAzDMFwdhIiIiIijqOVGRERE6hUlNyIiIlKvKLkRERGRekXJjYiIiNQrSm5ERESkXlFyIyIiIvWKkhsRERGpV5TciIiISL2i5EZERETqFSU3IiIiUq8ouREREZF65f8D6dKE6tbuP9UAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"train_its, train_losses = zip(*metrics.train_losses)\n",
"val_its, val_losses = zip(*metrics.val_losses)\n",
"plt.plot(train_its, train_losses, '-o')\n",
"plt.plot(val_its, val_losses, '-o')\n",
"plt.xlabel(\"Iteration\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.legend(['Train', \"Valid\"]);"
]
},
{
"cell_type": "markdown",
"id": "b28f216c",
"metadata": {},
"source": [
"### Evaluate\n",
"\n",
"The training and validation loss are only part of the story. For HellaSwag, we ultimately care about how good the model is at answering questions. To asses this, let's generate the actual `ending1`, `ending2`, `ending3`, or `ending4` responses with the fine-tuned model and measure the accuracy.\n",
"\n",
"First, let's split the last word off of each example in the test set to create a prompt without the answer."
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "d96e4dcf",
"metadata": {},
"outputs": [],
"source": [
"test_set = [t.rsplit(\" \", maxsplit=1) for t in test_set]"
]
},
{
"cell_type": "markdown",
"id": "8becd26a",
"metadata": {},
"source": [
"Next, we'll generate the response for each example in the test set and compare it to the ground-truth answer to measure the accuracy."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b396980a",
"metadata": {},
"outputs": [],
"source": [
"# Increase this number to use more test examples\n",
"num_test = 100\n",
"num_correct = 0\n",
"for prompt, answer in tqdm.tqdm(test_set[:num_test]):\n",
" response = generate(model, tokenizer, prompt, max_tokens=2)\n",
" num_correct += (response==answer)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "4cbc00b3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Approximate test accuracy 0.750\n"
]
}
],
"source": [
"test_acc = num_correct / num_test\n",
"print(f\"Approximate test accuracy {test_acc:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "67fbba7f",
"metadata": {},
"source": [
"### Fuse Adapters\n",
"\n",
"Sometimes its convenient to fuse the adapters into the base model to create a single adapted model. MLX LM has a fuse script just for that.\n",
"\n",
"The adapted weights are: $\\tilde{W} = W + c \\cdot \\mathbf{b}^\\top \\mathbf{a}$. Note, this process can be destructive if the inputs are in low precision and they have very different magnitudes. Tuning the `scale` parameter, $c$, prior to fine-tuning can improve the model performance after fusion.\n",
"\n",
"To see more options for fusing the model, including how to upload to HuggingFace [check the documentation](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/LORA.md#fuse)."
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "37854c9b",
"metadata": {},
"outputs": [],
"source": [
"!mlx_lm.fuse --model {model_path}"
]
},
{
"cell_type": "markdown",
"id": "c349707e",
"metadata": {},
"source": [
"Once the adapters are fused, we can rerun the evaluation using the fused model to make sure it worked. By default the fused model will be saved to `lora_fused_model`."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "c1c45e3a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Approximate test accuracy 0.750\n"
]
}
],
"source": [
"model, tokenizer = load(\"lora_fused_model\")\n",
"num_correct = 0\n",
"for prompt, answer in tqdm.tqdm(test_set[:num_test]):\n",
" response = generate(model, tokenizer, prompt, max_tokens=2)\n",
" num_correct += (response==answer)\n",
"test_acc = num_correct / num_test\n",
"print(f\"Approximate test accuracy {test_acc:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "d0dc7f4c",
"metadata": {},
"source": [
"### Troubleshooting\n",
"\n",
"#### Results\n",
"\n",
"To figure out why your LoRA adapters are not working well it's critical to plot both the trianing loss and validation loss over the duration of fine-tuning. There are really only two cases to consider: underfitting or overfitting. And you can figure out which regime you are in based on the above plot.\n",
"\n",
"**Underfitting**: The trianing loss is not low enough and the validation loss closely matches the training loss. You could also measure the accuracy on the training set itself for question-answering style tasks like HellaSwag. If you are in this regime you have a few options to improve the results:\n",
"\n",
"- Use more adapters. Increase `lora_layers` or adapt more of the linear layers within a given block by setting `lora_parameters[\"keys\"]`.\n",
"- Use a higher rank. A higher rank means more parameters per adapter.\n",
"- If you are using dropout, decrease the droupout rate or turn it off entirely.\n",
"- Sometimes, underfitting issues are really optimization issues. In these cases it can be helpful to tune the learning rate or learning rate schedule.\n",
"- If none of the above works, try a bigger model. For example, try Phi-3 medium instead of Phi-3 tiny.\n",
"\n",
"**Overfitting**: The trianing loss keeps going down but the validation loss stops going down and even starts to go up. If you are in this regime you also have a few options:\n",
"\n",
"- The best thing to do is to use more trianing data if you have it.\n",
"- Contrary to the underfitting regime decreasing the capacity of the model can help. For example, use fewer adapters, a lower LoRA rank, or a smaller model size.\n",
"- If you are not using dropout, use it.\n",
"\n",
"If you find your adapters work well pre-fusion but stop working post-fusion, try tuning the `scale` parameter, $c$, prior to fine-tuning. Typically the adapters have a smaller magnitude than the weights, so using a larger scale helps.\n",
"\n",
"#### Memory Use\n",
"\n",
"Fine-tuning a large LM with LoRA requires a machine with a decent amount of memory. Here are some tips to reduce memory use should you need to do so. \n",
"\n",
"- Try quantization (QLoRA). You can use QLoRA by generating a quantized model with `mlx_lm.convert` and the `-q` flag or by using an already quantized model from HuggingFace.\n",
"\n",
"- Try using a smaller batch size. You can set the `batch_size` parameter in the `TrainingArgs` or pass `--batch-size` if you are using the CLI. The default is 4 so setting this to 2 or 1 will reduce memory consumption. Note, this may slow things down a little..\n",
"\n",
"- Reduce the number of layers to fine-tune with by setting `lora_layers` to a smaller value or passing `--lora-layers` if you are using the CLI. The default is `16`, so you can try `8` or `4`. This reduces the amount of memory needed for back propagation. It may also reduce the quality of the fine-tuned model and you may need to compensate with a larger `rank`.\n",
"\n",
"- Longer examples require more memory. If it makes sense for your data, one thing you can do is break your examples into smaller sequences when making the `train`, `valid`, and `test` data sets.\n",
"\n",
"- Gradient checkpointing lets you trade-off memory use (less) for computation (more) by recomputing instead of storing intermediate values needed by the backward pass. You can use gradient checkpointing by passing `grad_checkpoint=True` to the `TrainingArgs` or the `--grad-checkpoint` flag if using the CLI. Gradient checkpointing will be more helpful for larger batch sizes or sequence lengths with smaller or quantized models.\n",
"\n",
"### Next Steps\n",
"\n",
"- To learn more about MLX check-out the [GitHub repo](http://github.com/ml-explore/mlx) and [documentation](https://ml-explore.github.io/mlx/)\n",
"- For more on MLX LM check-out the [MLX LM documentation](https://github.com/ml-explore/mlx-examples/tree/main/llms#readme).\n",
"- Check out the other [MLX Examples](https://github.com/ml-explore/mlx-examples/tree/main). These are great as a learning resource or to use as a starting point for a new project.\n",
"- We also have an example of [LoRA fine-tuning in MLX Swift](https://github.com/ml-explore/mlx-swift-examples/tree/main/Applications/LoRATrainingExample)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment