Skip to content

Instantly share code, notes, and snippets.

@firmai

firmai/automl_lightgbm.ipynb

Last active March 5, 2023 17:39
Show Gist options
  • Save firmai/2b099f9a55a3beadd611bd5bb5d2c25f to your computer and use it in GitHub Desktop.
Save firmai/2b099f9a55a3beadd611bd5bb5d2c25f to your computer and use it in GitHub Desktop.
Download ZIP
automl_lightgbm.ipynb
Display the source blob
Display the rendered blob
Raw
automl_lightgbm.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/firmai/2b099f9a55a3beadd611bd5bb5d2c25f/automl_lightgbm.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mr9UQuhpz5_1"
},
"source": [
"# Tune LightGBM with FLAML Library\n",
"\n",
"\n",
"## 1. Introduction\n",
"\n",
"FLAML is a Python library (https://github.com/microsoft/FLAML) designed to automatically produce accurate machine learning models \n",
"with low computational cost. It is fast and economical. The simple and lightweight design makes it easy \n",
"to use and extend, such as adding new learners. FLAML can \n",
"- serve as an economical AutoML engine,\n",
"- be used as a fast hyperparameter tuning tool, or \n",
"- be embedded in self-tuning software that requires low latency & resource in repetitive\n",
" tuning tasks.\n",
"\n",
"In this notebook, we demonstrate how to use FLAML library to tune hyperparameters of LightGBM with a regression example.\n",
"\n",
"FLAML requires `Python>=3.6`. To run this notebook example, please install flaml with the `notebook` option:\n",
"```bash\n",
"pip install flaml[notebook]\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "6GNRAyTnz5_4",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "4c8e1924-eec9-4f7f-8dbf-b88454a0add4"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
"Collecting flaml[notebook]\n",
" Downloading FLAML-1.1.3-py3-none-any.whl (224 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.2/224.2 KB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: pandas>=1.1.4 in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (1.3.5)\n",
"Requirement already satisfied: scikit-learn>=0.24 in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (1.2.1)\n",
"Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (1.10.1)\n",
"Collecting lightgbm>=2.3.1\n",
" Downloading lightgbm-3.3.5-py3-none-manylinux1_x86_64.whl (2.0 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m37.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: NumPy>=1.17.0rc1 in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (1.22.4)\n",
"Requirement already satisfied: xgboost>=0.90 in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (1.7.4)\n",
"Collecting openml==0.10.2\n",
" Downloading openml-0.10.2.tar.gz (158 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m159.0/159.0 KB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from flaml[notebook]) (3.5.3)\n",
"Collecting jupyter\n",
" Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)\n",
"Collecting liac-arff>=2.4.0\n",
" Downloading liac-arff-2.5.0.tar.gz (13 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Collecting xmltodict\n",
" Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from openml==0.10.2->flaml[notebook]) (2.25.1)\n",
"Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from openml==0.10.2->flaml[notebook]) (2.8.2)\n",
"Requirement already satisfied: wheel in /usr/local/lib/python3.8/dist-packages (from lightgbm>=2.3.1->flaml[notebook]) (0.38.4)\n",
"Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas>=1.1.4->flaml[notebook]) (2022.7.1)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn>=0.24->flaml[notebook]) (3.1.0)\n",
"Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from scikit-learn>=0.24->flaml[notebook]) (1.2.0)\n",
"Collecting qtconsole\n",
" Downloading qtconsole-5.4.0-py3-none-any.whl (121 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.0/121.0 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: nbconvert in /usr/local/lib/python3.8/dist-packages (from jupyter->flaml[notebook]) (6.5.4)\n",
"Requirement already satisfied: ipywidgets in /usr/local/lib/python3.8/dist-packages (from jupyter->flaml[notebook]) (7.7.1)\n",
"Requirement already satisfied: jupyter-console in /usr/local/lib/python3.8/dist-packages (from jupyter->flaml[notebook]) (6.1.0)\n",
"Requirement already satisfied: ipykernel in /usr/local/lib/python3.8/dist-packages (from jupyter->flaml[notebook]) (5.3.4)\n",
"Requirement already satisfied: notebook in /usr/local/lib/python3.8/dist-packages (from jupyter->flaml[notebook]) (6.3.0)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (3.0.9)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (4.38.0)\n",
"Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (8.4.0)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (0.11.0)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (1.4.4)\n",
"Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->flaml[notebook]) (23.0)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil->openml==0.10.2->flaml[notebook]) (1.15.0)\n",
"Requirement already satisfied: traitlets>=4.1.0 in /usr/local/lib/python3.8/dist-packages (from ipykernel->jupyter->flaml[notebook]) (5.7.1)\n",
"Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from ipykernel->jupyter->flaml[notebook]) (7.9.0)\n",
"Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.8/dist-packages (from ipykernel->jupyter->flaml[notebook]) (6.2)\n",
"Requirement already satisfied: jupyter-client in /usr/local/lib/python3.8/dist-packages (from ipykernel->jupyter->flaml[notebook]) (6.1.12)\n",
"Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->jupyter->flaml[notebook]) (0.2.0)\n",
"Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->jupyter->flaml[notebook]) (3.0.5)\n",
"Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->jupyter->flaml[notebook]) (3.6.2)\n",
"Requirement already satisfied: pygments in /usr/local/lib/python3.8/dist-packages (from jupyter-console->jupyter->flaml[notebook]) (2.6.1)\n",
"Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-console->jupyter->flaml[notebook]) (2.0.10)\n",
"Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (0.7.2)\n",
"Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (0.4)\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (4.9.2)\n",
"Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (0.2.2)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (2.1.2)\n",
"Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (0.8.4)\n",
"Requirement already satisfied: nbformat>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (5.7.3)\n",
"Requirement already satisfied: bleach in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (6.0.0)\n",
"Requirement already satisfied: jinja2>=3.0 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (3.1.2)\n",
"Requirement already satisfied: defusedxml in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (0.7.1)\n",
"Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (4.6.3)\n",
"Requirement already satisfied: jupyter-core>=4.7 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (5.2.0)\n",
"Requirement already satisfied: tinycss2 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (1.2.1)\n",
"Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from nbconvert->jupyter->flaml[notebook]) (1.5.0)\n",
"Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.8/dist-packages (from notebook->jupyter->flaml[notebook]) (0.13.3)\n",
"Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.8/dist-packages (from notebook->jupyter->flaml[notebook]) (21.3.0)\n",
"Requirement already satisfied: prometheus-client in /usr/local/lib/python3.8/dist-packages (from notebook->jupyter->flaml[notebook]) (0.16.0)\n",
"Requirement already satisfied: Send2Trash>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from notebook->jupyter->flaml[notebook]) (1.8.0)\n",
"Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.8/dist-packages (from notebook->jupyter->flaml[notebook]) (23.2.1)\n",
"Collecting qtpy>=2.0.1\n",
" Downloading QtPy-2.3.0-py3-none-any.whl (83 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 KB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->openml==0.10.2->flaml[notebook]) (1.26.14)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->openml==0.10.2->flaml[notebook]) (2022.12.7)\n",
"Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->openml==0.10.2->flaml[notebook]) (4.0.0)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->openml==0.10.2->flaml[notebook]) (2.10)\n",
"Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (57.4.0)\n",
"Requirement already satisfied: backcall in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (0.2.0)\n",
"Requirement already satisfied: decorator in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (4.4.2)\n",
"Requirement already satisfied: pexpect in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (4.8.0)\n",
"Requirement already satisfied: pickleshare in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (0.7.5)\n",
"Collecting jedi>=0.10\n",
" Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m32.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.8/dist-packages (from jupyter-core>=4.7->nbconvert->jupyter->flaml[notebook]) (3.0.0)\n",
"Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (4.3.3)\n",
"Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (2.16.3)\n",
"Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->flaml[notebook]) (0.2.6)\n",
"Requirement already satisfied: ptyprocess in /usr/local/lib/python3.8/dist-packages (from terminado>=0.8.3->notebook->jupyter->flaml[notebook]) (0.7.0)\n",
"Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.8/dist-packages (from argon2-cffi->notebook->jupyter->flaml[notebook]) (21.2.0)\n",
"Requirement already satisfied: webencodings in /usr/local/lib/python3.8/dist-packages (from bleach->nbconvert->jupyter->flaml[notebook]) (0.5.1)\n",
"Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->jupyter->flaml[notebook]) (0.8.3)\n",
"Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (22.2.0)\n",
"Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (0.19.3)\n",
"Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (5.12.0)\n",
"Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter->flaml[notebook]) (1.15.1)\n",
"Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter->flaml[notebook]) (2.21)\n",
"Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->flaml[notebook]) (3.15.0)\n",
"Building wheels for collected packages: openml, liac-arff\n",
" Building wheel for openml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for openml: filename=openml-0.10.2-py3-none-any.whl size=190317 sha256=395329abce31d07931a7023771e1db3defa5ef97bc7d02758a360a99b1ff75b3\n",
" Stored in directory: /root/.cache/pip/wheels/e9/c7/af/50e65ac06e82af181b9ce080c156cd4c76fa2ea8904f28c7cd\n",
" Building wheel for liac-arff (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for liac-arff: filename=liac_arff-2.5.0-py3-none-any.whl size=11732 sha256=5a2dde95dbc68d27a0909b4b82d3bfc54660afb777c3a6c985813df6798e6708\n",
" Stored in directory: /root/.cache/pip/wheels/a2/de/68/bf3972de3ecb31e32bef59a7f4c75f0687a3674c476b347c14\n",
"Successfully built openml liac-arff\n",
"Installing collected packages: xmltodict, qtpy, liac-arff, jedi, openml, lightgbm, qtconsole, flaml, jupyter\n",
" Attempting uninstall: lightgbm\n",
" Found existing installation: lightgbm 2.2.3\n",
" Uninstalling lightgbm-2.2.3:\n",
" Successfully uninstalled lightgbm-2.2.3\n",
"Successfully installed flaml-1.1.3 jedi-0.18.2 jupyter-1.0.0 liac-arff-2.5.0 lightgbm-3.3.5 openml-0.10.2 qtconsole-5.4.0 qtpy-2.3.0 xmltodict-0.13.0\n"
]
}
],
"source": [
"!pip install flaml[notebook];"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vRcAWvAQz5_5"
},
"source": [
"## 2. Regression Example\n",
"### Load data and preprocess\n",
"\n",
"Download [houses dataset](https://www.openml.org/d/537) from OpenML. The task is to predict median price of the house in the region based on demographic composition and a state of housing market in the region."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"tags": [],
"id": "vCcCHrjQz5_6",
"outputId": "ac6d5359-41dd-475d-ef3f-349dfdf4f319",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"download dataset from openml\n"
]
},
{
"output_type": "stream",
"name": "stderr",
"text": [
"DEBUG:openml.datasets.dataset:Saved dataset 537: houses to file /root/.openml/cache/org/openml/www/datasets/537/dataset.pkl.py3\n"
]
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Dataset name: houses\n",
"X_train.shape: (15480, 8), y_train.shape: (15480,);\n",
"X_test.shape: (5160, 8), y_test.shape: (5160,)\n"
]
}
],
"source": [
"from flaml.data import load_openml_dataset\n",
"X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=537, data_dir='./')"
]
},
{
"cell_type": "code",
"source": [
"X_train.head()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 206
},
"id": "ziCqdD1j0JJX",
"outputId": "5f105b39-4dcb-43c6-cc1b-2de87b2711ae"
},
"execution_count": 4,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
" median_income housing_median_age total_rooms total_bedrooms \\\n",
"19226 7.3003 19.0 4976.0 711.0 \n",
"14549 5.9547 18.0 1591.0 268.0 \n",
"9093 3.2125 19.0 552.0 129.0 \n",
"12213 6.9930 13.0 270.0 42.0 \n",
"12765 2.5162 21.0 3260.0 763.0 \n",
"\n",
" population households latitude longitude \n",
"19226 1926.0 625.0 38.46 -122.68 \n",
"14549 547.0 243.0 32.95 -117.24 \n",
"9093 314.0 106.0 34.68 -118.27 \n",
"12213 120.0 42.0 33.51 -117.18 \n",
"12765 1735.0 736.0 38.62 -121.41 "
],
"text/html": [
"\n",
" <div id=\"df-37cfb88c-d5c0-41cb-8768-f29d2932b66a\">\n",
" <div class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>median_income</th>\n",
" <th>housing_median_age</th>\n",
" <th>total_rooms</th>\n",
" <th>total_bedrooms</th>\n",
" <th>population</th>\n",
" <th>households</th>\n",
" <th>latitude</th>\n",
" <th>longitude</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>19226</th>\n",
" <td>7.3003</td>\n",
" <td>19.0</td>\n",
" <td>4976.0</td>\n",
" <td>711.0</td>\n",
" <td>1926.0</td>\n",
" <td>625.0</td>\n",
" <td>38.46</td>\n",
" <td>-122.68</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14549</th>\n",
" <td>5.9547</td>\n",
" <td>18.0</td>\n",
" <td>1591.0</td>\n",
" <td>268.0</td>\n",
" <td>547.0</td>\n",
" <td>243.0</td>\n",
" <td>32.95</td>\n",
" <td>-117.24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9093</th>\n",
" <td>3.2125</td>\n",
" <td>19.0</td>\n",
" <td>552.0</td>\n",
" <td>129.0</td>\n",
" <td>314.0</td>\n",
" <td>106.0</td>\n",
" <td>34.68</td>\n",
" <td>-118.27</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12213</th>\n",
" <td>6.9930</td>\n",
" <td>13.0</td>\n",
" <td>270.0</td>\n",
" <td>42.0</td>\n",
" <td>120.0</td>\n",
" <td>42.0</td>\n",
" <td>33.51</td>\n",
" <td>-117.18</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12765</th>\n",
" <td>2.5162</td>\n",
" <td>21.0</td>\n",
" <td>3260.0</td>\n",
" <td>763.0</td>\n",
" <td>1735.0</td>\n",
" <td>736.0</td>\n",
" <td>38.62</td>\n",
" <td>-121.41</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-37cfb88c-d5c0-41cb-8768-f29d2932b66a')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
" \n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
" <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
" </svg>\n",
" </button>\n",
" \n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" flex-wrap:wrap;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-37cfb88c-d5c0-41cb-8768-f29d2932b66a button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-37cfb88c-d5c0-41cb-8768-f29d2932b66a');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
" </div>\n",
" "
]
},
"metadata": {},
"execution_count": 4
}
]
},
{
"cell_type": "code",
"source": [
"y_train.head()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "E7a03nn-05of",
"outputId": "4f1d11f1-0e61-4fb1-c2bc-61966eff626a"
},
"execution_count": 5,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"19226 381300.0\n",
"14549 329300.0\n",
"9093 185400.0\n",
"12213 500001.0\n",
"12765 97500.0\n",
"Name: median_house_value, dtype: float64"
]
},
"metadata": {},
"execution_count": 5
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Wz2oV4Hhz5_7"
},
"source": [
"### Run FLAML\n",
"In the FLAML automl run configuration, users can specify the task type, time budget, error metric, learner list, whether to subsample, resampling strategy type, and so on. All these arguments have default values which will be used if users do not provide them. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"tags": [],
"id": "-umGlIdAz5_7"
},
"outputs": [],
"source": [
"''' import AutoML class from flaml package '''\n",
"from flaml import AutoML\n",
"automl = AutoML()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "QQFXJqdOz5_8"
},
"outputs": [],
"source": [
"settings = {\n",
" \"time_budget\": 60, # total running time in seconds\n",
" \"metric\": 'r2', # primary metrics for regression can be chosen from: ['mae','mse','r2','rmse','mape']\n",
" \"estimator_list\": ['lgbm'], # list of ML learners; we tune lightgbm in this example\n",
" \"task\": 'regression', # task type \n",
" \"log_file_name\": 'houses_experiment.log', # flaml log file\n",
" \"seed\": 7654321, # random seed\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"tags": [],
"id": "EeLWAngwz5_8",
"outputId": "863de1f8-4597-4082-a4e3-01386c9a5af2",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[flaml.automl.automl: 03-05 17:30:18] {2726} INFO - task = regression\n",
"[flaml.automl.automl: 03-05 17:30:18] {2728} INFO - Data split method: uniform\n",
"[flaml.automl.automl: 03-05 17:30:18] {2731} INFO - Evaluation method: cv\n",
"[flaml.automl.automl: 03-05 17:30:18] {2858} INFO - Minimizing error metric: 1-r2\n",
"[flaml.automl.automl: 03-05 17:30:18] {3004} INFO - List of ML learners in AutoML Run: ['lgbm']\n",
"[flaml.automl.automl: 03-05 17:30:18] {3334} INFO - iteration 0, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:18] {3472} INFO - Estimated sufficient time budget=1249s. Estimated necessary time budget=1s.\n",
"[flaml.automl.automl: 03-05 17:30:18] {3519} INFO - at 0.2s,\testimator lgbm's best error=0.7383,\tbest estimator lgbm's best error=0.7383\n",
"[flaml.automl.automl: 03-05 17:30:18] {3334} INFO - iteration 1, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:18] {3519} INFO - at 0.3s,\testimator lgbm's best error=0.7383,\tbest estimator lgbm's best error=0.7383\n",
"[flaml.automl.automl: 03-05 17:30:18] {3334} INFO - iteration 2, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:18] {3519} INFO - at 0.5s,\testimator lgbm's best error=0.3250,\tbest estimator lgbm's best error=0.3250\n",
"[flaml.automl.automl: 03-05 17:30:18] {3334} INFO - iteration 3, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:19] {3519} INFO - at 1.0s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:19] {3334} INFO - iteration 4, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:20] {3519} INFO - at 2.7s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:20] {3334} INFO - iteration 5, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:21] {3519} INFO - at 2.9s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:21] {3334} INFO - iteration 6, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:21] {3519} INFO - at 3.2s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:21] {3334} INFO - iteration 7, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:23] {3519} INFO - at 5.1s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:23] {3334} INFO - iteration 8, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:23] {3519} INFO - at 5.3s,\testimator lgbm's best error=0.1868,\tbest estimator lgbm's best error=0.1868\n",
"[flaml.automl.automl: 03-05 17:30:23] {3334} INFO - iteration 9, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:25] {3519} INFO - at 6.8s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:25] {3334} INFO - iteration 10, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:25] {3519} INFO - at 7.2s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:25] {3334} INFO - iteration 11, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:30] {3519} INFO - at 11.9s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:30] {3334} INFO - iteration 12, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:30] {3519} INFO - at 12.5s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:30] {3334} INFO - iteration 13, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:36] {3519} INFO - at 18.0s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:36] {3334} INFO - iteration 14, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:40] {3519} INFO - at 22.4s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:40] {3334} INFO - iteration 15, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:41] {3519} INFO - at 23.0s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:41] {3334} INFO - iteration 16, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:30:59] {3519} INFO - at 41.1s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:30:59] {3334} INFO - iteration 17, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:31:04] {3519} INFO - at 45.9s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:31:04] {3334} INFO - iteration 18, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:31:13] {3519} INFO - at 55.4s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:31:13] {3334} INFO - iteration 19, current learner lgbm\n",
"[flaml.automl.automl: 03-05 17:31:18] {3519} INFO - at 59.8s,\testimator lgbm's best error=0.1744,\tbest estimator lgbm's best error=0.1744\n",
"[flaml.automl.automl: 03-05 17:31:22] {3783} INFO - retrain lgbm for 4.2s\n",
"[flaml.automl.automl: 03-05 17:31:22] {3790} INFO - retrained model: LGBMRegressor(learning_rate=0.2647748120311755, max_bin=1023,\n",
" min_child_samples=29, n_estimators=44, num_leaves=81,\n",
" reg_alpha=0.0009765625, reg_lambda=0.028486834222229064,\n",
" verbose=-1)\n",
"[flaml.automl.automl: 03-05 17:31:22] {3034} INFO - fit succeeded\n",
"[flaml.automl.automl: 03-05 17:31:22] {3035} INFO - Time taken to find the best model: 6.765257835388184\n"
]
}
],
"source": [
"'''The main flaml automl API'''\n",
"automl.fit(X_train=X_train, y_train=y_train, **settings)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5SWVewM0z5_9"
},
"source": [
"### Best model and metric"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"tags": [],
"id": "0L7E3LCwz5_-",
"outputId": "c1a96772-ca72-47e8-ecba-548982924e9a",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Best hyperparmeter config: {'n_estimators': 44, 'num_leaves': 81, 'min_child_samples': 29, 'learning_rate': 0.2647748120311755, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.028486834222229064}\n",
"Best r2 on validation data: 0.8256\n",
"Training duration of best run: 4.246 s\n"
]
}
],
"source": [
"''' retrieve best config'''\n",
"print('Best hyperparmeter config:', automl.best_config)\n",
"print('Best r2 on validation data: {0:.4g}'.format(1-automl.best_loss))\n",
"print('Training duration of best run: {0:.4g} s'.format(automl.best_config_train_time))"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "PRt0r6Qpz5__",
"outputId": "18e7edd4-fb54-4270-cd86-6daeb475940b",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 126
}
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"LGBMRegressor(learning_rate=0.2647748120311755, max_bin=1023,\n",
" min_child_samples=29, n_estimators=44, num_leaves=81,\n",
" reg_alpha=0.0009765625, reg_lambda=0.028486834222229064,\n",
" verbose=-1)"
],
"text/html": [
"<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LGBMRegressor(learning_rate=0.2647748120311755, max_bin=1023,\n",
" min_child_samples=29, n_estimators=44, num_leaves=81,\n",
" reg_alpha=0.0009765625, reg_lambda=0.028486834222229064,\n",
" verbose=-1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LGBMRegressor</label><div class=\"sk-toggleable__content\"><pre>LGBMRegressor(learning_rate=0.2647748120311755, max_bin=1023,\n",
" min_child_samples=29, n_estimators=44, num_leaves=81,\n",
" reg_alpha=0.0009765625, reg_lambda=0.028486834222229064,\n",
" verbose=-1)</pre></div></div></div></div></div>"
]
},
"metadata": {},
"execution_count": 10
}
],
"source": [
"automl.model.estimator"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "d7xOUYC0z5__",
"outputId": "710ba25c-18d0-491b-c8fa-f5f696606863",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 283
}
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<BarContainer object of 8 artists>"
]
},
"metadata": {},
"execution_count": 11
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAD4CAYAAACOhb23AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAedElEQVR4nO3df3RdZZ3v8feHtLZQMAVaWZmCHMEMCLQNbUSBwiAqKnoRpVqFgcK47IAMKl5Gq7jGguMI1LkgikqdQUAQXeWHsIq2cIuF3kopSX8kLaWgtI5UBEUaflQqpN/7x35iD8f82Pl5Tk4+r7XOyt7Pfvbe330a+OR59s6JIgIzMzPr2W7lLsDMzGy4cGiamZnl5NA0MzPLyaFpZmaWk0PTzMwsp1HlLsAG14QJE6JQKJS7DDOzYaW5ufmPETGxtN2hWeUKhQJNTU3lLsPMbFiR9JvO2j09a2ZmlpND08zMLCeHppmZWU4OTTMzs5wcmmZmZjk5NM3MzHJyaJqZmeXk0DQzM8vJH25Q5Vq3tlGYe3e5yzCzEWjLZe8vdwkDziNNMzOznByaZmZmOTk0zczMcnJompmZ5eTQNDMzy8mhaWZmlpNDs4ikFwfhmKdImpuWT5V0WB+OsUxS40DXZmZmvePQHGQRcVdEXJZWTwV6HZpmZlYZHJqdUGa+pPWSWiXNSu0npFHfrZIelXSzJKVtJ6e2ZklXS1qU2s+W9G1JxwCnAPMlrZV0cPEIUtIESVvS8u6Sfixpo6Q7gN2LajtJ0oOSVktaKGnPoX13zMxGLn8iUOc+DDQAU4EJwMOSHkjbjgQOB34HrACOldQEXAscHxGbJd1SesCI+KWku4BFEXErQMrbzpwHbI+It0iaAqxO/ScAXwbeFREvSfoC8Dng0uKdJc0B5gDUvH5i394BMzP7Gx5pdm4GcEtEtEfE08D9wFvTtlUR8WRE7ATWAgXgUOCJiNic+vxNaPbS8cBNABHRArSk9reTTe+ukLQWmA0cWLpzRCyIiMaIaKzZo7afpZiZWQePNHtvR9FyO/17D19l1w8uY3P0F3BvRHy8H+c0M7M+8kizc8uBWZJqJE0kG/mt6qb/JuAgSYW0PquLfi8AexWtbwGmp+WZRe0PAKcDSDoCmJLaV5JNB785bRsn6e/zXJCZmfWfQ7Nzd5BNia4D7gM+HxG/76pzRPwZ+BSwWFIzWTi2ddL1x8C/Sloj6WDgG8B5ktaQ3Tvt8F1gT0kbye5XNqfz/AE4G7hFUgvwINnUsJmZDQFFRLlrqAqS9oyIF9PTtNcAj0fEleWua0xdfdTNvqrcZZjZCDSc/zSYpOaI+Jvfj/dIc+B8Mj2cswGoJXua1szMqogfBBogaVRZ9pGlmZkNHo80zczMcnJompmZ5eTQNDMzy8n3NKvc5Em1NA3jJ9jMzCqJR5pmZmY5OTTNzMxycmiamZnl5NA0MzPLyQ8CVbnWrW0U5t5d7jLMzIbUYH2En0eaZmZmOTk0zczMcnJompmZ5eTQNDMzy8mhaWZmlpND08zMLCeHZi9IerGH7eMlfapo/e8k3ZqWGySd3IdzzpN0Ue+rNTOzgebQHFjjgb+GZkT8LiJmptUGoNehaWZmlcOh2QeS9pS0VNJqSa2SPpg2XQYcLGmtpPmSCpLWS3odcCkwK22bVTqCTP0KafliSY9J+n/AIUV9Dpa0WFKzpOWSDh26qzYzM38iUN+8DHwoIp6XNAFYKekuYC5wREQ0AHSEYET8RdK/AY0R8S9p27zODixpOvAxspHpKGA10Jw2LwDOjYjHJb0N+A5wYifHmAPMAah5/cQBuFwzMwOHZl8J+A9JxwM7gUnAfgN07OOAOyJiO0AKYyTtCRwDLJTU0XdMZweIiAVkAcuYuvoYoLrMzEY8h2bfnAFMBKZHxCuStgBje3mMV3nt9HhP++8GbOsYxZqZ2dDzPc2+qQWeSYH5DuDA1P4CsFcX+5Ru2wJMA5A0DXhTan8AOFXS7pL2Av4XQEQ8D2yW9JG0jyRNHbhLMjOznjg0++ZmoFFSK3AW8ChARDwLrEgP9cwv2ecXwGEdDwIBtwH7SNoA/AvwWDrGauAnwDrg58DDRcc4A/iEpHXABuCDmJnZkFGEb3lVszF19VE3+6pyl2FmNqT6+6fBJDVHRGNpu0eaZmZmOTk0zczMcnJompmZ5eTQNDMzy8m/p1nlJk+qpamfN8TNzCzjkaaZmVlODk0zM7OcHJpmZmY5OTTNzMxy8oNAVa51axuFuXeXuwzrh/5+somZDRyPNM3MzHJyaJqZmeXk0DQzM8vJoWlmZpaTQ9PMzCwnh6aZmVlOIyI0JRUkrS/DeV/sZf95ki7qpL0s9ZuZ2WuNiNA0MzMbCCMpNGskfV/SBkn3SNpdUoOklZJaJN0haW8AScskNablCZK2pOXDJa2StDbtU5/a/7Go/VpJNR0nlfQ1SevSefZLbQVJ96VjLJX0xtJiJU1P+60Dzi9q77QGMzMbfCMpNOuBayLicGAbcBpwI/CFiJgCtAJf6eEY5wLfjIgGoBF4UtJbgFnAsam9HTgj9R8HrIyIqcADwCdT+7eAG9J5bwau7uRcPwAuSPt2W0PpjpLmSGqS1NS+va2HSzIzs7xGUmhujoi1abkZOBgYHxH3p7YbgON7OMaDwJckfQE4MCL+DLwTmA48LGltWj8o9f8LsKjonIW0fDTwo7T8Q2BG8UkkjU+1PVDUp7saXiMiFkREY0Q01uxR28MlmZlZXiMpNHcULbcD47vp+yq73puxHY0R8SPgFODPwM8knQiIbNTYkF6HRMS8tMsrERFF5+z3Z/12UYOZmQ2BkRSapdqA5yQdl9bPBDpGnVvIRo8AMzt2kHQQ8EREXA3cCUwBlgIzJb0h9dlH0oE9nPuXwMfS8hnA8uKNEbEN2CZpRlGf7mowM7MhMJJDE2A2MF9SC9AAXJravwGcJ2kNMKGo/0eB9Wka9gjgxoh4BPgycE86zr1AXQ/nvQA4J/U/E/hMJ33OAa5J51J3NeS6UjMz6zftmj20ajSmrj7qZl9V7jKsH/ynwcyGnqTmiGgsbR/pI00zM7PcHJpmZmY5OTTNzMxycmiamZnl1O/fG7TKNnlSLU1+kMTMbEB4pGlmZpaTQ9PMzCwnh6aZmVlODk0zM7Oc/CBQlWvd2kZh7t3lLsNsWPGnMFlXPNI0MzPLyaFpZmaWk0PTzMwsJ4emmZlZTg5NMzOznByaZmZmOTk0B4GkgqT1OfqcXrTeKOnqwa/OzMz6yqFZPgXgr6EZEU0R8enylWNmZj0ZkaGZRnmPSrpZ0kZJt0raQ9I7Ja2R1CrpOkljUv8tkq5I7askvTm1Xy9pZtFxX+ziXMslrU6vY9Kmy4DjJK2VdKGkEyQtSvvsI+mnklokrZQ0JbXPS3Utk/SEJIesmdkQGpGhmRwCfCci3gI8D3wOuB6YFRGTyT4t6byi/m2p/dvAVb04zzPAuyNiGjAL6JiCnQssj4iGiLiyZJ9LgDURMQX4EnBj0bZDgfcARwFfkTS69ISS5khqktTUvr2tF6WamVl3RnJo/jYiVqTlm4B3Apsj4rHUdgNwfFH/W4q+Ht2L84wGvi+pFVgIHJZjnxnADwEi4j5gX0mvT9vujogdEfFHskDer3TniFgQEY0R0VizR20vSjUzs+6M5M+ejZL1bcC+Oft3LL9K+sFD0m7A6zrZ70LgaWBq6vtyH2ottqNouZ2R/W9oZjakRvJI842SOkaMpwNNQKHjfiVwJnB/Uf9ZRV8fTMtbgOlp+RSyUWWpWuCpiNiZjlmT2l8A9uqituXAGQCSTgD+GBHP57koMzMbPCN5lLIJOF/SdcAjwKeBlcBCSaOAh4HvFfXfW1IL2Ujv46nt+8CdktYBi4GXOjnPd4DbJJ1V0qcFaE/7Xg+sKdpnHnBdOt92YHb/LtXMzAaCIkpnKaufpAKwKCKOyNl/C9CY7iMOK2Pq6qNu9lXlLsNsWPGfBjNJzRHRWNo+kqdnzczMemVETs9GxBYg1ygz9S8MWjFmZjZseKRpZmaWk0PTzMwsJ4emmZlZTiPynuZIMnlSLU1+EtDMbEB4pGlmZpaTQ9PMzCwnh6aZmVlODk0zM7Oc/CBQlWvd2kZh7t3lLsOGMX+knNkuHmmamZnl5NA0MzPLyaFpZmaWk0PTzMwsJ4emmZlZTg5NMzOznCouNCWNl/SpHvoUJJ2e41gFSeu72X62pG/3pc6B2N/MzIaXigtNYDzQbWgCBaDH0CwXSf79VzOzKlSJoXkZcLCktZLmp9d6Sa2SZhX1OS71uTCNKJdLWp1ex/TifAdIWibpcUlf6WiU9I+SVqVzXCupJrWfI+kxSauAY4v6Xy/pe5IeAq6Q1CBppaQWSXdI2jv166p9maQrJTVJ2ijprZJuT3X9e+ozTtLdktal92QWZmY2ZCoxNOcCv46IBmAl0ABMBd4FzJdUl/osj4iGiLgSeAZ4d0RMA2YBV/fifEcBpwFTgI9IapT0lnScY1Md7cAZ6dyXkIXlDOCwkmPtDxwTEZ8DbgS+EBFTgFagI5C7agf4S0Q0At8D7gTOB44Azpa0L/Be4HcRMTUijgAWd3ZBkuak8G1q397Wi7fCzMy6U+nTiDOAWyKiHXha0v3AW4HnS/qNBr4tqYEs4P6+F+e4NyKeBZB0ezrnq8B04GFJALuTBfPbgGUR8YfU/ycl51oYEe2SaoHxEXF/ar8BWNhVe9H+d6WvrcCGiHgqnecJ4IDU/p+SLgcWRcTyzi4oIhYACwDG1NVHL94LMzPrRqWHZl4XAk+TjUh3A17uxb6loRKAgBsi4ovFGySd2sOxXurFeTuzI33dWbTcsT4qIh6TNA04Gfh3SUsj4tJ+ntPMzHKqxOnZF4C90vJyYJakGkkTgeOBVSV9AGqBpyJiJ3AmUNOL871b0j6SdgdOBVYAS4GZkt4AkLYfCDwE/IOkfSWNBj7S2QEjog14TtJxqelM4P6u2vMWKunvgO0RcRMwH5jWi+s0M7N+qriRZkQ8K2lF+lWRnwMtwDqyEeDnI+L3kp4F2iWtA64HvgPcJukssvt8vRnxrQJuI7sfeVNENAFI+jJwj6TdgFeA8yNipaR5wIPANmBtN8edDXxP0h7AE8A5PbTnMZnsvu7OVNN5vdjXzMz6SRG+5VXNxtTVR93sq8pdhg1j/tNgNhJJak4PZr5GJU7PmpmZVaSKm54dDJLeA1xe0rw5Ij5UjnrMzGx4GhGhGRFLgCXlrsPMzIY3T8+amZnlNCJGmiPZ5Em1NPlBDjOzAeGRppmZWU4OTTMzs5wcmmZmZjk5NM3MzHLyg0BVrnVrG4W5d5e7DDPAny5kw59HmmZmZjk5NM3MzHJyaJqZmeXk0DQzM8vJoWlmZpaTQ9PMzCwnh6aZmVlOVR2aksZL+lQPfQqSTs9xrIKk9QNXnZmZDTdVHZrAeKDb0AQKQI+h2RuS/KERZmZVqNpD8zLgYElrJc1Pr/WSWiXNKupzXOpzYRpRLpe0Or2OyXMiSWdLukvSfcBSSftI+qmkFkkrJU1J/bpqnyfphnTu30j6sKQrUq2LJY1O/S6T9Eja/xtd1DJHUpOkpvbtbf19D83MLKn2EdFc4IiIaJB0GnAuMBWYADws6YHU56KI+ACApD2Ad0fEy5LqgVuAxpznmwZMiYg/SfoWsCYiTpV0InAj0ABc0kU7wMHAO4DDgAeB0yLi85LuAN4vaTnwIeDQiAhJ4zsrIiIWAAsAxtTVR87azcysB9U+0iw2A7glItoj4mngfuCtnfQbDXxfUiuwkCzA8ro3Iv5UdL4fAkTEfcC+kl7fTTvAzyPiFaAVqAEWp/ZWsmnkNuBl4L8lfRjY3ovazMysn0ZSaOZ1IfA02Yi0EXhdL/Z9qZ/n3gEQETuBVyKiY5S4ExgVEa8CRwG3Ah9gV6iamdkQqPbQfAHYKy0vB2ZJqpE0ETgeWFXSB6AWeCoF15lkI76+WA6cASDpBOCPEfF8N+09krQnUBsRPyML96l9rM3MzPqgqu9pRsSzklakXxX5OdACrAMC+HxE/F7Ss0C7pHXA9cB3gNsknUU2kuvr6HEecJ2kFrJp1Nk9tOexF3CnpLGAgM/1sTYzM+sD7ZoBtGo0pq4+6mZfVe4yzAD/PU0bPiQ1R8TfPARa7dOzZmZmA6aqp2cHg6T3AJeXNG+OiA+Vox4zMxs6Ds1eioglwJJy12FmZkPPoVnlJk+qpcn3kczMBoTvaZqZmeXk0DQzM8vJoWlmZpaTQ9PMzCwnPwhU5Vq3tlGYe3e5y7AK4Q8XMOsfjzTNzMxycmiamZnl5NA0MzPLyaFpZmaWk0PTzMwsJ4emmZlZTg5NMzOznHoMTUkFSesHqwBJvxysY/dX8bVLapR0dblrMjOz8in7hxtExDHlriGPiGgCmspdh5mZlU/e6dkaSd+XtEHSPZJ2l9QgaaWkFkl3SNobQNIySY1peYKkLWn5cEmrJK1N+9Sn9hfT1xPSvrdKelTSzZKUtp2c2polXS1pUVeFSpon6QZJyyX9RtKHJV0hqVXSYkmjU7/pku5Px1wiqa6ofZ2kdcD5Rcc9oeO8ko6S9KCkNZJ+KemQ1H62pNvTeR6XdEV3b6qk70pqSu/rJUXtnV6vpHGSrkvv4xpJH+ziuHPScZvat7d1V4KZmfVC3tCsB66JiMOBbcBpwI3AFyJiCtAKfKWHY5wLfDMiGoBG4MlO+hwJfBY4DDgIOFbSWOBa4H0RMR2YmKPeg4ETgVOAm4BfRMRk4M/A+1NwfguYmY55HfC1tO8PgAsiYmo3x38UOC4ijgT+DfiPom0NwCxgMjBL0gHdHOfiiGgEpgD/IGlKD9d7MXBfRBwFvAOYL2lc6UEjYkFENEZEY80etd2c3szMeiPv9OzmiFiblpvJQml8RNyf2m4AFvZwjAeBiyXtD9weEY930mdVRDwJIGktUABeBJ6IiM2pzy3AnB7O9fOIeEVSK1ADLE7tremYhwBHAPemwWwN8JSk8em6Hkj9fwi8r5Pj1wI3pNFyAKOLti2NiLZ0DY8ABwK/7aLOj0qaQ/bvUEf2w8Ju3VzvScApki5K62OBNwIbu303zMxsQOQNzR1Fy+3A+G76vsquEezYjsaI+JGkh4D3Az+T9M8RcV8P5+nrPdcd6Zw7Jb0SEZHad6ZjCtgQEUcX75RCM4+vko1ePySpACwrPXfS5TVIehNwEfDWiHhO0vUUvV9dEHBaRGzKWaeZmQ2gvv7KSRvwnKTj0vqZQMeocwswPS3P7NhB0kFkI6irgTvJpiTz2AQclMIJsqnP/toETJR0dKpttKTDI2IbsE3SjNTvjC72rwW2puWz+1jD64GXgDZJ+7FrRNvd9S4BLii613tkH89tZmZ90J/f05xNdk+thew+3qWp/RvAeZLWABOK+n8UWJ+mXY8guyfao4j4M/ApYLGkZuAFstDus4j4C1mgX54e+FkLdDzFew5wTapTXRziCuDr6Rr7NBqOiHXAGrL7oz8CVqT27q73q2RTwS2SNqR1MzMbIto1c1m5JO0ZES+mEdY1wOMRcWW56xosA3m9Y+rqo272VQNanw1f/nuaZvlIak4Par7GcPlEoE+mkd8GsqnRa8tbzqAbaddrZjYslP3DDfJIo6zXjLQknQN8pqTriog4nwqTHoAaU9J8ZkS0dta/s+s1M7PyGxah2ZmI+AHZ71RWvIh4W7lrMDOz/hsu07NmZmZlN2xHmpbP5Em1NPnhDzOzAeGRppmZWU4OTTMzs5wcmmZmZjk5NM3MzHLyg0BVrnVrG4W5d5e7DLNB5087sqHgkaaZmVlODk0zM7OcHJpmZmY5OTTNzMxycmiamZnl5NA0MzPLyaFpZmaWU9WGpqRlkhrT8s8kjR/AY58r6ayBOp6ZmQ0PI+LDDSLi5AE+3vcG8nhmZjY8VNRIU1JB0qOSrpf0mKSbJb1L0gpJj0s6StI4SddJWiVpjaQPpn13l/RjSRsl3QHsXnTcLZImpOWfSmqWtEHSnKI+L0r6mqR1klZK2q+bOudJuigtL5N0earnMUnHpfYaSd+QtF5Si6QLUvs7U92t6TrGFNX4dUlrJTVJmiZpiaRfSzq36Nz/KunhdMxLuqhvTjpGU/v2tn78i5iZWbGKCs3kzcB/Aoem1+nADOAi4EvAxcB9EXEU8A5gvqRxwHnA9oh4C/AVYHoXx/+niJgONAKflrRvah8HrIyIqcADwCd7UfOoVM9n07kB5gAFoCEipgA3SxoLXA/MiojJZCP984qO8z8R0QAsT/1mAm8HLgGQdBJQDxwFNADTJR1fWkxELIiIxohorNmjtheXYWZm3anE0NwcEa0RsRPYACyNiABayULoJGCupLXAMmAs8EbgeOAmgIhoAVq6OP6nJa0DVgIHkIUQwF+ARWm5OZ0rr9s72e9dwLUR8Wqq6U/AIen6Hkt9bkh1d7grfW0FHoqIFyLiD8COdE/2pPRaA6wm+6GiHjMzGxKVeE9zR9HyzqL1nWT1tgOnRcSm4p0k9XhgSSeQhdnREbFd0jKy0AV4JYUz6Ry9eW86auztfl0dp/i6O9ZHAQK+HhHX9uMcZmbWR5U40uzJEuACpZSUdGRqf4BsKhdJRwBTOtm3FnguBeahZFOfg+Ve4J8ljUo17QNsAgqS3pz6nAnc34tjLgH+SdKe6ZiTJL1hAGs2M7NuDMfQ/CowGmiRtCGtA3wX2FPSRuBSsqnSUouBUanPZWRTtIPlv4D/SXWuA06PiJeBc4CFklrJRpC5n8SNiHuAHwEPpv1vBfYa8MrNzKxT2jUjadVoTF191M2+qtxlmA06/z1NG0iSmiOisbR9OI40zczMyqISHwSqGJIuBj5S0rwwIr5WjnrMzKy8HJrdSOHogDQzM8ChWfUmT6qlyfd6zMwGhO9pmpmZ5eTQNDMzy8mhaWZmlpND08zMLCeHppmZWU4OTTMzs5wcmmZmZjk5NM3MzHJyaJqZmeXkv3JS5SS9QPZ3PIeLCcAfy11EL7jewTfcana9g2uo6j0wIiaWNvpj9Krfps7+vE2lktTkegfPcKsXhl/NrndwlbteT8+amZnl5NA0MzPLyaFZ/RaUu4Becr2Da7jVC8OvZtc7uMparx8EMjMzy8kjTTMzs5wcmmZmZjk5NKuUpPdK2iTpV5LmlrueDpKuk/SMpPVFbftIulfS4+nr3qldkq5O19AiaVoZ6j1A0i8kPSJpg6TPVHLNksZKWiVpXar3ktT+JkkPpbp+Iul1qX1MWv9V2l4YynqL6q6RtEbSokqvV9IWSa2S1kpqSm0V+f2Qahgv6VZJj0raKOnoSq1X0iHpfe14PS/ps5VUr0OzCkmqAa4B3gccBnxc0mHlreqvrgfeW9I2F1gaEfXA0rQOWf316TUH+O4Q1VjsVeB/R8RhwNuB89N7Wak17wBOjIipQAPwXklvBy4HroyINwPPAZ9I/T8BPJfar0z9yuEzwMai9Uqv9x0R0VD0+4KV+v0A8E1gcUQcCkwle58rst6I2JTe1wZgOrAduKOi6o0Iv6rsBRwNLCla/yLwxXLXVVRPAVhftL4JqEvLdWQfyABwLfDxzvqVsfY7gXcPh5qBPYDVwNvIPkFlVOn3B7AEODotj0r9NMR17k/2P8ITgUWAKrzeLcCEkraK/H4AaoHNpe9RpdZbUuNJwIpKq9cjzeo0Cfht0fqTqa1S7RcRT6Xl3wP7peWKuo40FXgk8BAVXHOa6lwLPAPcC/wa2BYRr3ZS01/rTdvbgH2Hsl7gKuDzwM60vi+VXW8A90hqljQntVXq98ObgD8AP0jT3/8laRyVW2+xjwG3pOWKqdehaRUlsh8XK+73oCTtCdwGfDYini/eVmk1R0R7ZNNb+wNHAYeWt6KuSfoA8ExENJe7ll6YERHTyKYGz5d0fPHGCvt+GAVMA74bEUcCL7FrahOouHoBSPewTwEWlm4rd70Ozeq0FTigaH3/1FapnpZUB5C+PpPaK+I6JI0mC8ybI+L21FzRNQNExDbgF2TTm+MldXzWdHFNf603ba8Fnh3CMo8FTpG0Bfgx2RTtNyu4XiJia/r6DNn9tqOo3O+HJ4EnI+KhtH4rWYhWar0d3gesjoin03rF1OvQrE4PA/XpCcTXkU1z3FXmmrpzFzA7Lc8mu2/Y0X5WekLu7UBb0RTNkJAk4L+BjRHxf4o2VWTNkiZKGp+Wdye7/7qRLDxndlFvx3XMBO5LP8kPiYj4YkTsHxEFsu/T+yLijEqtV9I4SXt1LJPdd1tPhX4/RMTvgd9KOiQ1vRN4pFLrLfJxdk3NdtRVGfWW4wavX4P/Ak4GHiO7n3VxuespqusW4CngFbKfgj9Bdk9qKfA48H+BfVJfkT0F/GugFWgsQ70zyKaCWoC16XVypdYMTAHWpHrXA/+W2g8CVgG/IpvyGpPax6b1X6XtB5Xxe+MEYFEl15vqWpdeGzr+26rU74dUQwPQlL4nfgrsXeH1jiObPagtaquYev0xemZmZjl5etbMzCwnh6aZmVlODk0zM7OcHJpmZmY5OTTNzMxycmiamZnl5NA0MzPL6f8D+Qk11tUizm0AAAAASUVORK5CYII=\n"
},
"metadata": {
"needs_background": "light"
}
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"plt.barh(automl.model.estimator.feature_name_, automl.model.estimator.feature_importances_)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "Qe25m3tdz5__"
},
"outputs": [],
"source": [
"''' pickle and save the automl object '''\n",
"import pickle\n",
"with open('automl.pkl', 'wb') as f:\n",
" pickle.dump(automl, f, pickle.HIGHEST_PROTOCOL)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"tags": [],
"id": "XdN8BFN5z6AA",
"outputId": "57ea068c-dd27-486a-84bd-546d5364002f",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Predicted labels [144777.73346527 222143.38670785 156547.88237569 ... 213541.68968855\n",
" 241946.12879879 260921.0767248 ]\n",
"True labels 14740 136900.0\n",
"10101 241300.0\n",
"20566 200700.0\n",
"2670 72500.0\n",
"15709 460000.0\n",
" ... \n",
"13132 121200.0\n",
"8228 137500.0\n",
"3948 160900.0\n",
"8522 227300.0\n",
"16798 265600.0\n",
"Name: median_house_value, Length: 5160, dtype: float64\n"
]
}
],
"source": [
"''' compute predictions of testing dataset ''' \n",
"y_pred = automl.predict(X_test)\n",
"print('Predicted labels', y_pred)\n",
"print('True labels', y_test)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"tags": [],
"id": "gUR9Ikbcz6AA",
"outputId": "2f64fb66-2656-40c9-b8ed-a281d58089f8",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"r2 = 0.8337155008815875\n",
"mse = 2198032739.332084\n",
"mae = 31463.024910135013\n"
]
}
],
"source": [
"''' compute different metric values on testing dataset'''\n",
"from flaml.ml import sklearn_metric_loss_score\n",
"print('r2', '=', 1 - sklearn_metric_loss_score('r2', y_pred, y_test))\n",
"print('mse', '=', sklearn_metric_loss_score('mse', y_pred, y_test))\n",
"print('mae', '=', sklearn_metric_loss_score('mae', y_pred, y_test))"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"tags": [],
"id": "PSEldr76z6AA",
"outputId": "d64fecb7-a776-4145-ecc1-749d72e5d34e",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"{'Current Learner': 'lgbm', 'Current Sample': 15480, 'Current Hyper-parameters': {'n_estimators': 4, 'num_leaves': 4, 'min_child_samples': 20, 'learning_rate': 0.09999999999999995, 'log_max_bin': 8, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 1.0}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 4, 'num_leaves': 4, 'min_child_samples': 20, 'learning_rate': 0.09999999999999995, 'log_max_bin': 8, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 1.0}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 15480, 'Current Hyper-parameters': {'n_estimators': 22, 'num_leaves': 4, 'min_child_samples': 18, 'learning_rate': 0.2293009676418639, 'log_max_bin': 9, 'colsample_bytree': 0.9086551727646448, 'reg_alpha': 0.0015561782752413472, 'reg_lambda': 0.33127416269768944}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 22, 'num_leaves': 4, 'min_child_samples': 18, 'learning_rate': 0.2293009676418639, 'log_max_bin': 9, 'colsample_bytree': 0.9086551727646448, 'reg_alpha': 0.0015561782752413472, 'reg_lambda': 0.33127416269768944}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 15480, 'Current Hyper-parameters': {'n_estimators': 28, 'num_leaves': 20, 'min_child_samples': 17, 'learning_rate': 0.32352862101602586, 'log_max_bin': 10, 'colsample_bytree': 0.8801327898366843, 'reg_alpha': 0.004475520554844502, 'reg_lambda': 0.033081571878574946}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 28, 'num_leaves': 20, 'min_child_samples': 17, 'learning_rate': 0.32352862101602586, 'log_max_bin': 10, 'colsample_bytree': 0.8801327898366843, 'reg_alpha': 0.004475520554844502, 'reg_lambda': 0.033081571878574946}}\n",
"{'Current Learner': 'lgbm', 'Current Sample': 15480, 'Current Hyper-parameters': {'n_estimators': 44, 'num_leaves': 81, 'min_child_samples': 29, 'learning_rate': 0.2647748120311755, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.028486834222229064}, 'Best Learner': 'lgbm', 'Best Hyper-parameters': {'n_estimators': 44, 'num_leaves': 81, 'min_child_samples': 29, 'learning_rate': 0.2647748120311755, 'log_max_bin': 10, 'colsample_bytree': 1.0, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.028486834222229064}}\n"
]
}
],
"source": [
"from flaml.data import get_output_from_log\n",
"time_history, best_valid_loss_history, valid_loss_history, config_history, metric_history = \\\n",
" get_output_from_log(filename=settings['log_file_name'], time_budget=60)\n",
"\n",
"for config in config_history:\n",
" print(config)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "sODWJTsyz6AA",
"outputId": "f99104b8-4328-4524-c328-16ce4572505c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 295
}
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbo0lEQVR4nO3df7xVdZ3v8dfbI+jRhJNxavAAQiNRqCVKes2m1HTAMqA0B7u3O9kdsZl0mvRi0KiZXm82zti1x1ANmuOPUREJEYskJ39U/ghQFATCIVLhYIk/8Fcn+fW5f6x1aLvdZ599Dmftvc9e7+fjcR5nr7W+e63PIdvv/f1+1w9FBGZmll971LoAMzOrLQeBmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPArAxJfyFpba3rMMuSg8DqlqSnJJ1Qyxoi4hcRMSar/UuaIOnnkl6VtFnS/ZImZXU8s1IcBJZrkppqeOxTgduAG4BhwLuAi4BP9mJfkuT/P1uv+D8c63ck7SFphqTfSHpB0lxJ+xdsv03S7yS9nH7bPrhg23WSvidpkaTXgePSnsf/lrQifc+tkvZO2x8raWPB+7tsm24/X9KzkjZJ+htJIemgEn+DgCuBSyPimoh4OSJ2RsT9EXFm2uZiSf9R8J6R6f72TJfvk3SZpAeAPwDTJS0rOs5XJC1MX+8l6Z8lPSPp95K+L6l5N//nsAbgILD+6BxgCvBR4ADgJWBWwfafAKOBdwKPAjcVvf+zwGXAfsAv03WnAROBUcD7gc+XOX7JtpImAucCJwAHAceW2ccYYDgwr0ybSnwOmEbyt3wfGCNpdMH2zwI3p68vB94DHJbW10bSA7GccxBYf/RF4B8jYmNEvAFcDJza+U05Iq6NiFcLtn1A0uCC998REQ+k38D/mK77TkRsiogXgTtJPiy70lXb04B/j4hVEfGH9NhdeUf6+9nK/uQuXZceb3tEvAzcAZwOkAbCe4GFaQ9kGvCViHgxIl4F/i8wdTePbw3AQWD90YHA7ZK2SNoCrAF2AO+S1CTp8nTY6BXgqfQ9Qwrev6HEPn9X8PoPwNvKHL+rtgcU7bvUcTq9kP4eWqZNJYqPcTNpEJD0BhakodQK7AM8UvDvdle63nLOQWD90QbgpIhoKfjZOyLaST78JpMMzwwGRqbvUcH7s7rl7rMkk76dhpdpu5bk7zilTJvXST68O/1ZiTbFf8vdQKukw0gCoXNY6HmgAzi44N9scESUCzzLCQeB1bsBkvYu+NmTZCz8MkkHAkhqlTQ5bb8f8AbJN+59SIY/qmUucIak90naB7iwq4aR3P/9XOBCSWdIGpROgn9Y0uy02WPARySNSIe2ZnZXQERsIzkT6Qpgf5JgICJ2AlcD35b0TgBJbZIm9PaPtcbhILB6t4jkm2znz8XAVcBC4KeSXgUeBo5K298APA20A6vTbVURET8BvgPcC6wrOPYbXbSfB/wV8AVgE/B74P+QjPMTEXcDtwIrgEeAH1VYys0kPaLbImJ7wfqvdtaVDpv9J8mkteWc/GAas2xIeh/wBLBX0QeyWV1xj8CsD0n6VHq+/tuBbwF3OgSs3jkIzPrWWcBzwG9IzmT629qWY9Y9Dw2ZmeWcewRmZjm3Z60L6KkhQ4bEyJEja12GmVm/8sgjjzwfESUvIOx3QTBy5EiWLVvWfUMzM9tF0tNdbfPQkJlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5Vy/O2vIzCxvFixv54rFa9m0pYMDWpqZPmEMU8a19dn+HQRmZnVswfJ2Zs5fSce2HQC0b+lg5vyVAH0WBh4aMjOrY1csXrsrBDp1bNvBFYvX9tkxHARmZnVs05aOHq3vDQeBmVkdO6CluUfre8NBUAMLlrdzzOX3MGrGjznm8ntYsLy91iWZWZ2aPmEMzQOa3rSueUAT0yf03cPlPFlcZdWY+DGzxtH5uXD+vBVs3bGTNp811P91NfFz/rwV3LLkmRpVZWb1bq8BezBuRAu3nnV0n+/bQ0NV1tUEz9YdO6tciZn1J2OHDmLyYdmMGmTaI5A0EbgKaAKuiYjLi7aPAK4HWtI2MyJiUZY11doBLc20lwiDtpbmTJLezKw7mfUIJDUBs4CTgLHA6ZLGFjW7AJgbEeOAqcB3s6qnXlRj4sfMrCeyHBo6ElgXEesjYiswB5hc1CaAQenrwcCmDOupC1PGtfHNTx/KwKbkn76tpZlvfvpQTxSbWc1kOTTUBmwoWN4IHFXU5mLgp5LOAfYFTii1I0nTgGkAI0aM6PNCq23KuLZdE8MeDjKzWqv1ZPHpwHURMQz4OHCjpLfUFBGzI2J8RIxvbS35yE0zM+ulLIOgHRhesDwsXVfofwFzASLiIWBvYEiGNZmZWZEsg2ApMFrSKEkDSSaDFxa1eQb4GICk95EEweYMazIzsyKZBUFEbAfOBhYDa0jODlol6RJJk9Jm5wFnSnocuAX4fEREVjWZmdlbZXodQXpNwKKidRcVvF4NHJNlDWZmVl6tJ4vNzKzGHARmZjnnIDAzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc45CMzMcs5BYGaWcw4CM7OccxCYmeWcg8DMLOccBGZmOecgMDPLOQeBmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzCznHARmZjmXaRBImihpraR1kmaU2P5tSY+lP09K2pJlPWZm9lZ7ZrVjSU3ALOBEYCOwVNLCiFjd2SYivlLQ/hxgXFb1mJlZaVn2CI4E1kXE+ojYCswBJpdpfzpwS4b1mJlZCVkGQRuwoWB5Y7ruLSQdCIwC7uli+zRJyyQt27x5c58XamaWZ/UyWTwVmBcRO0ptjIjZETE+Isa3trZWuTQzs8aWZRC0A8MLloel60qZioeFzMxqIssgWAqMljRK0kCSD/uFxY0kvRd4O/BQhrWYmVkXMguCiNgOnA0sBtYAcyNilaRLJE0qaDoVmBMRkVUtZmbWtcxOHwWIiEXAoqJ1FxUtX5xlDWZmVl6mQdAoFixv54rFa9m0pYMDWpqZPmEMU8aVPAHKzKzfcRB0Y8HydmbOX0nHtuSEpvYtHcycvxLAYWBmDcFB0I0rFq/dFQKdOrbt4Px5K7hlyTO93u/qZ19h7NBBu1uemdluq5frCOrWpi0dJddv3bFzt/Y7duggJh/mHoWZ1Z57BN04oKWZ9hJh0NbSzK1nHV2DiszM+pZ7BN2YPmEMzQOa3rSueUAT0yeMqVFFZmZ9yz2CbnROCJ8/bwVbd+ykzWcNmVmDcRBUYMq4tl0Twx4OMrNG46EhM7OccxCYmeWcg8DMLOccBGZmOecgMDPLOQeBmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzCznHARmZjlXNggkDZL05yXWvz+7kszMrJq6DAJJpwG/Bn4oaZWkDxZsvi7rwszMrDrK9Qi+BhwREYcBZwA3SvpUuk1ZF2ZmZtVR7nkETRHxLEBELJF0HPAjScOBqEp1ZmaWuXI9glcL5wfSUDgWmAwcnHFdZmZWJeV6BH9L0RBQRLwqaSJwWqZVmZlZ1XTZI4iIx4HfSrq3aP22iLipkp1LmihpraR1kmZ00eY0SavTCembe1S9mZnttrLPLI6IHZJ2ShocES/3ZMeSmoBZwInARmCppIURsbqgzWhgJnBMRLwk6Z09/xPMzGx3VPLw+teAlZLuBl7vXBkRf9/N+44E1kXEegBJc0jmF1YXtDkTmBURL6X7fK4HtZuZWR+oJAjmpz891QZsKFjeCBxV1OY9AJIeAJqAiyPiruIdSZoGTAMYMWJEL0oxM7OudBsEEXF9xscfTXI20jDg55IOjYgtRTXMBmYDjB8/3qeumpn1oSzvNdQODC9YHpauK7QRWJhOQP8WeJIkGMzMrEqyDIKlwGhJoyQNBKYCC4vaLCDpDSBpCMlQ0foMazIzsyKZBUFEbAfOBhYDa4C5EbFK0iWSJqXNFgMvSFoN3AtMj4gXsqrJzMzeqts5AknvAaYDBxa2j4jju3tvRCwCFhWtu6jgdQDnpj9mZlYDlZw1dBvwfeBqYEe25ZiZWbVVEgTbI+J7mVdiZmY1UckcwZ2S/k7SUEn7d/5kXpmZmVVFJT2Cv05/Ty9YF8C7+74cMzOrtkouKBtVjULMzKw2KjlraADJLak/kq66D/i3iNiWYV1mZlYllQwNfQ8YAHw3Xf5cuu5vsirKzMyqp5Ig+GBEfKBg+R5Jj2dVkJmZVVclZw3tKHxkpaR34+sJzMwaRiU9gunAvZLWkzy68kDgjEyrMjOzqqnkrKGfpU8SG5OuWhsRb2RblpmZVUuXQSDp+Ii4R9KnizYdJImI6M3DaszMrM6U6xF8FLgH+GSJbUHvnlpmZmZ1pssgiIivpy8vSR8as4skX2RmZtYgKjlr6Icl1s3r60LMzKw2ys0RvBc4GBhcNE8wCNg768LMzKw6ys0RjAFOBlp48zzBq8CZGdZkZmZVVG6O4A7gDklHR8RDVazJzMyqqJILypZL+hLJMNGuIaGI+EJmVZmZWdVUMll8I/BnwATgfmAYyfCQmZk1gEqC4KCIuBB4PSKuBz4BHJVtWWZmVi2VBEHncwe2SDoEGAy8M7uSzMysmiqZI5gt6e3AhcBC4G3ARZlWZWZmVVPJTeeuSV/ej59TbGbWcMpdUHZuuTdGxJV9X46ZmVVbuR7BfunvMcAHSYaFILm4bEmWRZmZWfV0OVkcEd+IiG+QnC56eEScFxHnAUcAIyrZuaSJktZKWidpRontn5e0WdJj6Y+fg2xmVmWVTBa/C9hasLw1XVeWpCZgFnAisBFYKmlhRKwuanprRJxdYb1mZtbHKgmCG4Alkm5Pl6cA11XwviOBdRGxHkDSHGAyUBwEZmZWQ91eRxARl5E8o/il9OeMiPhmBftuAzYULG9M1xU7RdIKSfMkDS+1I0nTJC2TtGzz5s0VHNrMzCrVZRBIGpT+3h94iuRWEzcCT6fr+sKdwMiIeD9wN3B9qUYRMTsixkfE+NbW1j46tJmZQfmhoZtJbkP9CMmjKTspXe7umoJ2oPAb/rB03S4R8ULB4jXAP3WzTzMz62PlbkN9cvq7t4+lXAqMTh9r2Q5MBT5b2EDS0Ih4Nl2cBKzp5bHMzKyXyl1Qdni5N0bEo91s3y7pbGAx0ARcGxGrJF0CLIuIhcDfS5oEbAdeBD7fw/rNzGw3lRsa+pcy2wI4vrudR8QiYFHRuosKXs8EZna3HzMzy065oaHjqlmImZnVRiXXEZDefnosb35C2Q1ZFWVmZtXTbRBI+jpwLEkQLAJOAn5JcqGZmZn1c5U8mOZU4GPA7yLiDOADJA+nMTOzBlBJEHRExE5ge3qR2XO8+foAMzPrxyqZI1gmqQW4muTisteAh7IsyszMqqfcdQSzgJsj4u/SVd+XdBcwKCJWVKU6MzPLXLkewZPAP0saCswFbomI5dUpy8zMqqXcg2muioijgY8CLwDXSvq1pK9Lek/VKjQzs0xVchvqpyPiWxExDjid5HkEvieQmVmD6DYIJO0p6ZOSbgJ+AqwFPp15ZWZmVhXlJotPJOkBfJzkYfVzgGkR8XqVajMzsyooN1k8k+SZBOdFxEtVqsfMzKqs3E3nur27qJmZ9X+VXFlsZmYNzEFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc45CMzMcs5BYGaWcw4CM7OccxCYmeVcpkEgaaKktZLWSZpRpt0pkkLS+CzrMTOzt8osCCQ1AbOAk4CxwOmSxpZotx/wZeBXWdViZmZdy7JHcCSwLiLWR8RWkgfbTC7R7lLgW8AfM6zFzMy6kGUQtAEbCpY3put2kXQ4MDwiflxuR5KmSVomadnmzZv7vlIzsxyr2WSxpD2AK4HzumsbEbMjYnxEjG9tbc2+ODOzHMkyCNqB4QXLw9J1nfYDDgHuk/QU8N+AhZ4wNjOrriyDYCkwWtIoSQOBqcDCzo0R8XJEDImIkRExEngYmBQRyzKsyczMimQWBBGxHTgbWAysAeZGxCpJl0ialNVxzcysZ7p8eH1fiIhFwKKidRd10fbYLGsxM7PSfGWxmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzCznHARmZjnnIDAzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc45CMzMcs5BYGaWcw4CM7OccxCYmeWcg8DMLOccBGZmOecgMDPLOQeBmVnOOQjMzHIu0yCQNFHSWknrJM0osf2LklZKekzSLyWNzbIeMzN7q8yCQFITMAs4CRgLnF7ig/7miDg0Ig4D/gm4Mqt6zMystCx7BEcC6yJifURsBeYAkwsbRMQrBYv7ApFhPWZmVsKeGe67DdhQsLwROKq4kaQvAecCA4HjM6zHzMxKqPlkcUTMiog/B74KXFCqjaRpkpZJWrZ58+bqFmhm1uCyDIJ2YHjB8rB0XVfmAFNKbYiI2RExPiLGt7a29l2FZmaWaRAsBUZLGiVpIDAVWFjYQNLogsVPAP+VYT1mZlZCZnMEEbFd0tnAYqAJuDYiVkm6BFgWEQuBsyWdAGwDXgL+Oqt6zMystCwni4mIRcCionUXFbz+cpbHNzOz7tV8stjMzGrLQWBmlnMOAjOznHMQmJnlnIPAzCznMj1rqF4sWN7OFYvXsmlLBwe0NDN9whimjGurdVlmZnWh4YNgwfJ2Zs5fSce2HQC0b+lg5vyVAA4DMzNyEARXLF67KwQ6dWzbwfnzVnDLkmcq3s/qZ19h7NBBfV2emVnNNfwcwaYtHSXXb92xs0f7GTt0EJMPcw/CzBpPw/cIDmhppr1EGLS1NHPrWUfXoCIzs/rS8D2C6RPG0Dyg6U3rmgc0MX3CmBpVZGZWXxq+R9A5IeyzhszMSmv4IIAkDPzBb2ZWWsMPDZmZWXkOAjOznHMQmJnlnIPAzCznHARmZjmniKh1DT0iaTPwdBebhwDPV7GcvuK6q6+/1u66q6u/1g1vrf3AiGgt1bDfBUE5kpZFxPha19FTrrv6+mvtrru6+mvd0LPaPTRkZpZzDgIzs5xrtCCYXesCesl1V19/rd11V1d/rRt6UHtDzRGYmVnPNVqPwMzMeshBYGaWcw0RBJImSloraZ2kGbWup1KSrpX0nKQnal1LT0gaLuleSaslrZL05VrXVAlJe0taIunxtO5v1LqmnpDUJGm5pB/VupaekPSUpJWSHpO0rNb1VEpSi6R5kn4taY2kun+SlaQx6b9z588rkv6h2/f19zkCSU3Ak8CJwEZgKXB6RKyuaWEVkPQR4DXghog4pNb1VErSUGBoRDwqaT/gEWBKvf+bSxKwb0S8JmkA8EvgyxHxcI1Lq4ikc4HxwKCIOLnW9VRK0lPA+IjoVxdmSboe+EVEXCNpILBPRGypcVkVSz8b24GjIqKri3CBxugRHAmsi4j1EbEVmANMrnFNFYmInwMv1rqOnoqIZyPi0fT1q8AaoO4f+BCJ19LFAelPv/gmJGkY8AngmlrXkgeSBgMfAX4AEBFb+1MIpD4G/Ka7EIDGCII2YEPB8kb6wYdSo5A0EhgH/KrGpVQkHV55DHgOuDsi+kXdwP8Dzgd21riO3gjgp5IekTSt1sVUaBSwGfj3dDjuGkn71rqoHpoK3FJJw0YIAqsRSW8Dfgj8Q0S8Uut6KhEROyLiMGAYcKSkuh+Sk3Qy8FxEPFLrWnrpwxFxOHAS8KV0SLTe7QkcDnwvIsYBrwP9af5xIDAJuK2S9o0QBO3A8ILlYek6y1A6xv5D4KaImF/renoq7ebfC0yscSmVOAaYlI61zwGOl/QftS2pchHRnv5+DridZDi33m0ENhb0GOeRBEN/cRLwaET8vpLGjRAES4HRkkalKTgVWFjjmhpaOun6A2BNRFxZ63oqJalVUkv6upnkBINf17SoCkTEzIgYFhEjSf77vici/keNy6qIpH3TEwpIh1b+Eqj7s+Qi4nfABklj0lUfA+r6ZIgip1PhsBA0wMPrI2K7pLOBxUATcG1ErKpxWRWRdAtwLDBE0kbg6xHxg9pWVZFjgM8BK9PxdoCvRcSi2pVUkaHA9enZFHsAcyOiX52K2Q+9C7g9+e7AnsDNEXFXbUuq2DnATekXzPXAGTWupyJp4J4InFXxe/r76aNmZrZ7GmFoyMzMdoODwMws5xwEZmY55yAwM8s5B4GZWc45CKyuSPp24d0SJS2WdE3B8r+kN1/r6v3XSTo1fX2fpLc8vFvSAEmXS/ovSY9KekjSSem2pyQN6UXdu47bxfZZ6d0gV0vqKLg75KmSFnVe39CXJA0td6dSSQMl/VxSvz+N3HaPg8DqzQPAhwAk7QEMAQ4u2P4h4MHdPMalJNcUHJLe+mAKsN9u7rOsiPhSemuLj5PcCOyw9GdeRHw8oxuanQtcXaamrcDPgL/K4NjWjzgIrN48CHTe9/1gkqtQX5X0dkl7Ae8DHpV0kaSlkp6QNDu92rlbkvYBzgTOiYg3ACLi9xExt0Tbc9P9P1HUS/mfklakzzW4scT7Lk17CE0V1vSUpCGSRqb3vr9O0pOSbpJ0gqQH0t7LkWn7fZU8y2JJekO0ru62ewpwV/qeg9P2j6W1j07bLAD+eyV1WuNyl9DqSkRskrRd0giSb/8PkdxN9mjgZWBlRGyV9K8RcQlA+mF8MnBnBYc4CHimu5vkSTqC5ErSowABv5J0P7AVuAD4UEQ8L2n/ovddQdK7OCN6d7XmQcBngC+Q3D7ls8CHSW4g9jWS3ss/ktxm4gvpkNISSf8ZEa8X1DEKeKkz7IAvAldFROeVsp0h9QTwwV7UaQ3EPQKrRw+ShEBnEDxUsPxA2uY4Sb+StBI4njcPH/WFDwO3R8Tr6TMM5gN/kR7rts6HrERE4fMkLgQGR8QXexkCAL+NiJURsRNYBfws3ddKYGTa5i+BGentPe4D9gZGFO1nKMltlDs9BHxN0leBAyOiI61/B7C1835Alk8OAqtHnfMEh5J8Y32YpEfwIeBBSXsD3wVOjYhDScbB965w3+uAEZIG9XnVyTf4I4p7CT30RsHrnQXLO/lTD17AKQXzDCMiYk3Rfjoo+DeJiJtJehUdwCJJxxe03Qv4427UbP2cg8Dq0YMkQz0vps8PeBFoIQmDB/nTB9zzSp6J0OXZOsUi4g8kd069Kh0i6bwr6WeKmv4CmCJpn/QmXp9K190DfEbSO9L3Fn7o3wVcDvw442/Yi4FzOudFJI0r0eZJ/tSDQNK7gfUR8R3gDuD96fp3AM9HxLYM67U65yCwerSS5Gyhh4vWvRwRz6dn2FxN0ltYTPJNvCcuIBk2WS3pCeBHwJvmDNJHcV4HLCF5+to1EbE8vbPtZcD9kh4Hrix6321pbQuV3Oo6C5eSPGZzhaRV6fKbpPMFv5F0ULrqNOCJdDjpEOCGdP1xwI8zqtP6Cd991KxBSfoUcEREXFCmzXxgRkQ8Wb3KrN74rCGzBhURt3cOYZWSDo0tcAiYewRmZjnnOQIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8u5/w8GB08qPuQG+QAAAABJRU5ErkJggg==\n"
},
"metadata": {
"needs_background": "light"
}
}
],
"source": [
"import numpy as np\n",
"\n",
"plt.title('Learning Curve')\n",
"plt.xlabel('Wall Clock Time (s)')\n",
"plt.ylabel('Validation r2')\n",
"plt.scatter(time_history, 1 - np.array(valid_loss_history))\n",
"plt.step(time_history, 1 - np.array(best_valid_loss_history), where='post')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-_cJdXc5z6AB"
},
"source": [
"## 3. Comparison with alternatives\n",
"\n",
"### FLAML's accuracy"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"tags": [],
"id": "_6WpkYtTz6AC",
"outputId": "ef70deab-f8bd-4726-b1cd-bb641c486d6f",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"flaml (4min) r2 = 0.8337155008815875\n"
]
}
],
"source": [
"print('flaml (4min) r2', '=', 1 - sklearn_metric_loss_score('r2', y_pred, y_test))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "acwScPcvz6AC"
},
"source": [
"### Default LightGBM"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "FuJ3qmWDz6AC"
},
"outputs": [],
"source": [
"from lightgbm import LGBMRegressor\n",
"lgbm = LGBMRegressor()"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "VfY9avJ4z6AC",
"outputId": "0dc9cb4e-92d2-444b-a9a3-5d80016888b4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 74
}
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"LGBMRegressor()"
],
"text/html": [
"<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LGBMRegressor()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LGBMRegressor</label><div class=\"sk-toggleable__content\"><pre>LGBMRegressor()</pre></div></div></div></div></div>"
]
},
"metadata": {},
"execution_count": 19
}
],
"source": [
"lgbm.fit(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"tags": [],
"id": "bxcTCAUIz6AD",
"outputId": "1ac414d8-598b-43f5-90ff-3e2e3479f29e",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"default lgbm r2 = 0.8296179648694404\n"
]
}
],
"source": [
"y_pred = lgbm.predict(X_test)\n",
"from flaml.ml import sklearn_metric_loss_score\n",
"print('default lgbm r2', '=', 1 - sklearn_metric_loss_score('r2', y_pred, y_test))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bppmMMT-z6AD"
},
"source": [
"### Optuna LightGBM Tuner"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "3HYXZpnvz6AD"
},
"outputs": [],
"source": [
"%%capture\n",
"!pip install optuna==2.8.0;"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"id": "cEp-_rsjz6AD"
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"train_x, val_x, train_y, val_y = train_test_split(X_train, y_train, test_size=0.1)\n",
"import optuna.integration.lightgbm as lgb\n",
"dtrain = lgb.Dataset(train_x, label=train_y)\n",
"dval = lgb.Dataset(val_x, label=val_y)\n",
"params = {\n",
" \"objective\": \"regression\",\n",
" \"metric\": \"regression\",\n",
" \"verbosity\": -1,\n",
" \n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"tags": [
"outputPrepend"
],
"id": "8Bg7Xl5hz6AD",
"outputId": "6126b938-93a6-4453-c1d1-2b57e71c452b",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"\u001b[32m[I 2023-03-05 17:31:30,787]\u001b[0m A new study created in memory with name: no-name-64b64790-2084-491d-840a-be6dcafa865a\u001b[0m\n",
"feature_fraction, val_score: inf: 0%| | 0/7 [00:00<?, ?it/s]/usr/local/lib/python3.8/dist-packages/lightgbm/engine.py:239: UserWarning: 'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. Pass 'log_evaluation()' callback via 'callbacks' argument instead.\n",
" _log_warning(\"'verbose_eval' argument is deprecated and will be removed in a future release of LightGBM. \"\n",
"feature_fraction, val_score: 2110496880.451937: 14%|#4 | 1/7 [00:01<00:09, 1.53s/it]\u001b[32m[I 2023-03-05 17:31:32,324]\u001b[0m Trial 0 finished with value: 2110496880.4519372 and parameters: {'feature_fraction': 0.7}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 29%|##8 | 2/7 [00:04<00:12, 2.42s/it]\u001b[32m[I 2023-03-05 17:31:35,377]\u001b[0m Trial 1 finished with value: 2153022743.5090523 and parameters: {'feature_fraction': 0.5}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 43%|####2 | 3/7 [00:05<00:07, 1.92s/it]\u001b[32m[I 2023-03-05 17:31:36,697]\u001b[0m Trial 2 finished with value: 2269709610.6730585 and parameters: {'feature_fraction': 0.4}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 57%|#####7 | 4/7 [00:07<00:05, 1.78s/it]\u001b[32m[I 2023-03-05 17:31:38,254]\u001b[0m Trial 3 finished with value: 2164192138.6021137 and parameters: {'feature_fraction': 0.8999999999999999}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 71%|#######1 | 5/7 [00:09<00:03, 1.72s/it]\u001b[32m[I 2023-03-05 17:31:39,879]\u001b[0m Trial 4 finished with value: 2176607939.415413 and parameters: {'feature_fraction': 1.0}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 86%|########5 | 6/7 [00:10<00:01, 1.65s/it]\u001b[32m[I 2023-03-05 17:31:41,374]\u001b[0m Trial 5 finished with value: 2110496880.4519372 and parameters: {'feature_fraction': 0.8}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 100%|##########| 7/7 [00:11<00:00, 1.57s/it]\u001b[32m[I 2023-03-05 17:31:42,786]\u001b[0m Trial 6 finished with value: 2139176233.3702793 and parameters: {'feature_fraction': 0.6}. Best is trial 0 with value: 2110496880.4519372.\u001b[0m\n",
"feature_fraction, val_score: 2110496880.451937: 100%|##########| 7/7 [00:11<00:00, 1.71s/it]\n",
"num_leaves, val_score: 2110496880.451937: 5%|5 | 1/20 [00:05<01:41, 5.33s/it]\u001b[32m[I 2023-03-05 17:31:48,137]\u001b[0m Trial 7 finished with value: 2130241010.388424 and parameters: {'num_leaves': 110}. Best is trial 7 with value: 2130241010.388424.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 10%|# | 2/20 [00:13<02:01, 6.73s/it]\u001b[32m[I 2023-03-05 17:31:55,834]\u001b[0m Trial 8 finished with value: 2091420708.6160116 and parameters: {'num_leaves': 234}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 15%|#5 | 3/20 [00:16<01:31, 5.35s/it]\u001b[32m[I 2023-03-05 17:31:59,556]\u001b[0m Trial 9 finished with value: 2102301030.3697517 and parameters: {'num_leaves': 54}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 20%|## | 4/20 [00:18<00:59, 3.74s/it]\u001b[32m[I 2023-03-05 17:32:00,814]\u001b[0m Trial 10 finished with value: 2172610704.883104 and parameters: {'num_leaves': 23}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 25%|##5 | 5/20 [00:19<00:42, 2.85s/it]\u001b[32m[I 2023-03-05 17:32:02,087]\u001b[0m Trial 11 finished with value: 2160379837.6086755 and parameters: {'num_leaves': 24}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 30%|### | 6/20 [00:20<00:31, 2.23s/it]\u001b[32m[I 2023-03-05 17:32:03,129]\u001b[0m Trial 12 finished with value: 2180424205.0269766 and parameters: {'num_leaves': 15}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 35%|###5 | 7/20 [00:29<00:59, 4.58s/it]\u001b[32m[I 2023-03-05 17:32:12,536]\u001b[0m Trial 13 finished with value: 2147379136.7329636 and parameters: {'num_leaves': 236}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2091420708.616012: 40%|#### | 8/20 [00:38<01:11, 5.97s/it]\u001b[32m[I 2023-03-05 17:32:21,474]\u001b[0m Trial 14 finished with value: 2117239365.5550628 and parameters: {'num_leaves': 249}. Best is trial 8 with value: 2091420708.6160116.\u001b[0m\n",
"num_leaves, val_score: 2077390161.428703: 45%|####5 | 9/20 [00:41<00:55, 5.07s/it]\u001b[32m[I 2023-03-05 17:32:24,566]\u001b[0m Trial 15 finished with value: 2077390161.4287028 and parameters: {'num_leaves': 64}. Best is trial 15 with value: 2077390161.4287028.\u001b[0m\n",
"num_leaves, val_score: 2077390161.428703: 50%|##### | 10/20 [00:48<00:57, 5.72s/it]\u001b[32m[I 2023-03-05 17:32:31,761]\u001b[0m Trial 16 finished with value: 2133122528.191451 and parameters: {'num_leaves': 178}. Best is trial 15 with value: 2077390161.4287028.\u001b[0m\n",
"num_leaves, val_score: 2077390161.428703: 50%|##### | 10/20 [00:48<00:48, 4.90s/it]\n",
" 0%| | 0/10 [00:00<?, ?it/s]\n",
" 0%| | 0/6 [00:00<?, ?it/s]\n",
" 0%| | 0/20 [00:00<?, ?it/s]\n",
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 1min 45s, sys: 3.71 s, total: 1min 49s\n",
"Wall time: 1min 1s\n"
]
},
{
"output_type": "stream",
"name": "stderr",
"text": [
"\n"
]
}
],
"source": [
"%%time\n",
"model = lgb.train(params, dtrain, valid_sets=[dtrain, dval], verbose_eval=10000,time_budget =60) "
]
},
{
"cell_type": "markdown",
"source": [
"Note below, the comparison is not fair, because the test set is used from a previous set and could now have been used in the training set above."
],
"metadata": {
"id": "wZJW35g12Qr0"
}
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"tags": [],
"id": "dORBLnwhz6AE",
"outputId": "1efe6e47-3e7c-4675-8040-6aa3b5b98fe0",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Optuna LightGBM Tuner r2 = 0.8462894079441026\n"
]
}
],
"source": [
"y_pred = model.predict(X_test)\n",
"from flaml.ml import sklearn_metric_loss_score\n",
"print('Optuna LightGBM Tuner r2', '=', 1 - sklearn_metric_loss_score('r2', y_pred, y_test))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D9khuebtz6AE"
},
"source": [
"## 4. Add a customized LightGBM learner in FLAML\n",
"The native API of LightGBM allows one to specify a custom objective function in the model constructor. You can easily enable it by adding a customized LightGBM learner in FLAML. In the following example, we show how to add such a customized LightGBM learner with a custom objective function."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "joWPFMllz6AE"
},
"source": [
"### Create a customized LightGBM learner with a custom objective function"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "Eehb_evaz6AE"
},
"outputs": [],
"source": [
"import numpy as np \n",
"\n",
"''' define your customized objective function '''\n",
"def some_crazy_random_loss(y_true, y_pred):\n",
" c = 0.5\n",
" residual = y_pred - y_true\n",
" grad = c * residual /(np.abs(residual) + c)\n",
" hess = c ** 2 / (np.abs(residual) + c) ** 2\n",
" # rmse grad and hess\n",
" grad_rmse = residual\n",
" hess_rmse = 1.0\n",
" \n",
" # mae grad and hess\n",
" grad_mae = np.array(residual)\n",
" grad_mae[grad_mae > 0] = 1.\n",
" grad_mae[grad_mae <= 0] = -1.\n",
" hess_mae = 1.0\n",
"\n",
" coef = [0.4, 0.3, 0.3]\n",
" return coef[0] * grad + coef[1] * grad_rmse + coef[2] * grad_mae, \\\n",
" coef[0] * hess + coef[1] * hess_rmse + coef[2] * hess_mae\n",
"\n",
"\n",
"from flaml.model import LGBMEstimator\n",
"\n",
"''' create a customized LightGBM learner class with your objective function '''\n",
"class MyLGBM(LGBMEstimator):\n",
" '''LGBMEstimator with some_crazy_random_loss as the objective function\n",
" '''\n",
"\n",
" def __init__(self, **config):\n",
" super().__init__(objective=some_crazy_random_loss, **config)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DQp_9bl7z6AE"
},
"source": [
"### Add the customized learner in FLAML"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"tags": [],
"id": "PEfXNsjCz6AF",
"outputId": "b8379141-d760-443c-a64a-b54c1fa02842",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[flaml.automl.automl: 03-05 17:32:32] {2726} INFO - task = regression\n",
"[flaml.automl.automl: 03-05 17:32:32] {2728} INFO - Data split method: uniform\n",
"[flaml.automl.automl: 03-05 17:32:32] {2731} INFO - Evaluation method: cv\n",
"[flaml.automl.automl: 03-05 17:32:32] {2858} INFO - Minimizing error metric: 1-r2\n",
"[flaml.automl.automl: 03-05 17:32:32] {3004} INFO - List of ML learners in AutoML Run: ['my_lgbm']\n",
"[flaml.automl.automl: 03-05 17:32:32] {3334} INFO - iteration 0, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:33] {3472} INFO - Estimated sufficient time budget=12007s. Estimated necessary time budget=12s.\n",
"[flaml.automl.automl: 03-05 17:32:33] {3519} INFO - at 1.2s,\testimator my_lgbm's best error=2.9883,\tbest estimator my_lgbm's best error=2.9883\n",
"[flaml.automl.automl: 03-05 17:32:33] {3334} INFO - iteration 1, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:34] {3519} INFO - at 1.9s,\testimator my_lgbm's best error=2.9883,\tbest estimator my_lgbm's best error=2.9883\n",
"[flaml.automl.automl: 03-05 17:32:34] {3334} INFO - iteration 2, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:34] {3519} INFO - at 2.7s,\testimator my_lgbm's best error=1.7086,\tbest estimator my_lgbm's best error=1.7086\n",
"[flaml.automl.automl: 03-05 17:32:34] {3334} INFO - iteration 3, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:35] {3519} INFO - at 3.6s,\testimator my_lgbm's best error=0.3474,\tbest estimator my_lgbm's best error=0.3474\n",
"[flaml.automl.automl: 03-05 17:32:35] {3334} INFO - iteration 4, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:37] {3519} INFO - at 4.8s,\testimator my_lgbm's best error=0.3474,\tbest estimator my_lgbm's best error=0.3474\n",
"[flaml.automl.automl: 03-05 17:32:37] {3334} INFO - iteration 5, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:39] {3519} INFO - at 7.5s,\testimator my_lgbm's best error=0.3015,\tbest estimator my_lgbm's best error=0.3015\n",
"[flaml.automl.automl: 03-05 17:32:39] {3334} INFO - iteration 6, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:41] {3519} INFO - at 9.7s,\testimator my_lgbm's best error=0.3015,\tbest estimator my_lgbm's best error=0.3015\n",
"[flaml.automl.automl: 03-05 17:32:41] {3334} INFO - iteration 7, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:42] {3519} INFO - at 10.5s,\testimator my_lgbm's best error=0.3015,\tbest estimator my_lgbm's best error=0.3015\n",
"[flaml.automl.automl: 03-05 17:32:42] {3334} INFO - iteration 8, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:47] {3519} INFO - at 15.8s,\testimator my_lgbm's best error=0.2721,\tbest estimator my_lgbm's best error=0.2721\n",
"[flaml.automl.automl: 03-05 17:32:48] {3334} INFO - iteration 9, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:50] {3519} INFO - at 17.9s,\testimator my_lgbm's best error=0.2721,\tbest estimator my_lgbm's best error=0.2721\n",
"[flaml.automl.automl: 03-05 17:32:50] {3334} INFO - iteration 10, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:56] {3519} INFO - at 24.5s,\testimator my_lgbm's best error=0.1833,\tbest estimator my_lgbm's best error=0.1833\n",
"[flaml.automl.automl: 03-05 17:32:56] {3334} INFO - iteration 11, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:32:59] {3519} INFO - at 27.7s,\testimator my_lgbm's best error=0.1833,\tbest estimator my_lgbm's best error=0.1833\n",
"[flaml.automl.automl: 03-05 17:32:59] {3334} INFO - iteration 12, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:00] {3519} INFO - at 28.6s,\testimator my_lgbm's best error=0.1833,\tbest estimator my_lgbm's best error=0.1833\n",
"[flaml.automl.automl: 03-05 17:33:00] {3334} INFO - iteration 13, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:02] {3519} INFO - at 29.8s,\testimator my_lgbm's best error=0.1833,\tbest estimator my_lgbm's best error=0.1833\n",
"[flaml.automl.automl: 03-05 17:33:02] {3334} INFO - iteration 14, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:03] {3519} INFO - at 31.0s,\testimator my_lgbm's best error=0.1833,\tbest estimator my_lgbm's best error=0.1833\n",
"[flaml.automl.automl: 03-05 17:33:03] {3334} INFO - iteration 15, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:04] {3519} INFO - at 32.6s,\testimator my_lgbm's best error=0.1762,\tbest estimator my_lgbm's best error=0.1762\n",
"[flaml.automl.automl: 03-05 17:33:04] {3334} INFO - iteration 16, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:05] {3519} INFO - at 33.7s,\testimator my_lgbm's best error=0.1762,\tbest estimator my_lgbm's best error=0.1762\n",
"[flaml.automl.automl: 03-05 17:33:05] {3334} INFO - iteration 17, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:13] {3519} INFO - at 40.8s,\testimator my_lgbm's best error=0.1760,\tbest estimator my_lgbm's best error=0.1760\n",
"[flaml.automl.automl: 03-05 17:33:13] {3334} INFO - iteration 18, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:14] {3519} INFO - at 42.4s,\testimator my_lgbm's best error=0.1760,\tbest estimator my_lgbm's best error=0.1760\n",
"[flaml.automl.automl: 03-05 17:33:14] {3334} INFO - iteration 19, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:16] {3519} INFO - at 44.2s,\testimator my_lgbm's best error=0.1760,\tbest estimator my_lgbm's best error=0.1760\n",
"[flaml.automl.automl: 03-05 17:33:16] {3334} INFO - iteration 20, current learner my_lgbm\n",
"[flaml.automl.automl: 03-05 17:33:28] {3519} INFO - at 56.0s,\testimator my_lgbm's best error=0.1760,\tbest estimator my_lgbm's best error=0.1760\n",
"[flaml.automl.automl: 03-05 17:33:29] {3783} INFO - retrain my_lgbm for 1.1s\n",
"[flaml.automl.automl: 03-05 17:33:29] {3790} INFO - retrained model: LGBMRegressor(colsample_bytree=0.8338647421161105,\n",
" learning_rate=0.6064061936041562, max_bin=1023,\n",
" min_child_samples=2, n_estimators=75, num_leaves=232,\n",
" objective=<function my_loss_obj at 0x7f0a698818b0>,\n",
" reg_alpha=0.0009765625, reg_lambda=0.006523200676964421,\n",
" verbose=-1)\n",
"[flaml.automl.automl: 03-05 17:33:29] {3034} INFO - fit succeeded\n",
"[flaml.automl.automl: 03-05 17:33:29] {3035} INFO - Time taken to find the best model: 40.83250284194946\n"
]
}
],
"source": [
"automl = AutoML()\n",
"automl.add_learner(learner_name='my_lgbm', learner_class=MyLGBM)\n",
"settings = {\n",
" \"time_budget\": 60, # total running time in seconds\n",
" \"metric\": 'r2', # primary metrics for regression can be chosen from: ['mae','mse','r2']\n",
" \"estimator_list\": ['my_lgbm',], # list of ML learners; we tune lightgbm in this example\n",
" \"task\": 'regression', # task type \n",
" \"log_file_name\": 'houses_experiment_my_lgbm.log', # flaml log file\n",
"}\n",
"automl.fit(X_train=X_train, y_train=y_train, **settings)"
]
},
{
"cell_type": "markdown",
"source": [
"Still a pretty high r-squared with some strange loss function, maybe the loss function choice doesn't matter that much for lightgbm models?"
],
"metadata": {
"id": "sX1gJUnM2neh"
}
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"tags": [],
"id": "hY5DtnuLz6AF",
"outputId": "9a1a5098-ca89-4118-b260-4189d6a5fdde",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Best hyperparmeter config: {'n_estimators': 75, 'num_leaves': 232, 'min_child_samples': 2, 'learning_rate': 0.6064061936041562, 'log_max_bin': 10, 'colsample_bytree': 0.8338647421161105, 'reg_alpha': 0.0009765625, 'reg_lambda': 0.006523200676964421}\n",
"Best r2 on validation data: 0.824\n",
"Training duration of best run: 1.115 s\n",
"Predicted labels [140277.26590347 276276.19120105 141045.10838511 ... 217332.04436863\n",
" 245233.80641789 297535.52242455]\n",
"True labels 14740 136900.0\n",
"10101 241300.0\n",
"20566 200700.0\n",
"2670 72500.0\n",
"15709 460000.0\n",
" ... \n",
"13132 121200.0\n",
"8228 137500.0\n",
"3948 160900.0\n",
"8522 227300.0\n",
"16798 265600.0\n",
"Name: median_house_value, Length: 5160, dtype: float64\n",
"r2 = 0.8354998590274428\n",
"mse = 2174446189.5088744\n",
"mae = 30640.6710859263\n"
]
}
],
"source": [
"print('Best hyperparmeter config:', automl.best_config)\n",
"print('Best r2 on validation data: {0:.4g}'.format(1-automl.best_loss))\n",
"print('Training duration of best run: {0:.4g} s'.format(automl.best_config_train_time))\n",
"\n",
"y_pred = automl.predict(X_test)\n",
"print('Predicted labels', y_pred)\n",
"print('True labels', y_test)\n",
"\n",
"from flaml.ml import sklearn_metric_loss_score\n",
"print('r2', '=', 1 - sklearn_metric_loss_score('r2', y_pred, y_test))\n",
"print('mse', '=', sklearn_metric_loss_score('mse', y_pred, y_test))\n",
"print('mae', '=', sklearn_metric_loss_score('mae', y_pred, y_test))"
]
}
],
"metadata": {
"interpreter": {
"hash": "0cfea3304185a9579d09e0953576b57c8581e46e6ebc6dfeb681bc5a511f7544"
},
"kernelspec": {
"display_name": "Python 3.8.0 64-bit ('blend': conda)",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"colab": {
"name": "automl_lightgbm.ipynb",
"provenance": [],
"include_colab_link": true
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment