Skip to content

Instantly share code, notes, and snippets.

@gschivley

gschivley/_NOTE.md

Last active March 26, 2020 11:54
Show Gist options
  • Save gschivley/09257d239a88fcbd8981ca5e0589321e to your computer and use it in GitHub Desktop.
Save gschivley/09257d239a88fcbd8981ca5e0589321e to your computer and use it in GitHub Desktop.
Download ZIP
FERC714_exploration.ipynb
Raw
_NOTE.md

Moved to github.com/gschivley/FERC_714

There has been more interest in this project than I first anticitpated, so I've moved this notebook to a full repository. The notebook in this gist will not be updated - all future changes will take place on the repo. You can fork the repo, submit pull requests, or open issues.

Raw
environment.yml
name: ferc-data
channels:
- conda-forge
dependencies:
- python=3.7
- numpy
- pandas=0.25.*
- pip
# - matplotlib=3.*
- joblib
- xlrd
# GIS dependencies from conda-forge in case ppl start using shapefiles
- conda-forge::fiona
- conda-forge::geopandas
- conda-forge::shapely
- conda-forge::descartes
Display the source blob
Display the rendered blob
Raw
ferc714_exploration.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "FERC714_exploration.ipynb",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/gschivley/09257d239a88fcbd8981ca5e0589321e/ferc714_exploration.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z_b2KbSpG2SW",
"colab_type": "text"
},
"source": [
"# FERC 714 hourly demand data\n",
"\n",
"This notebook extracts a couple years of hourly demand data from FERC 714 and starts exploring ways to match the FERC respondents to EIA utility/balancing authority entities in EIA-861. My goal is to match the FERC respondents with IPM regions. It's a working document that I'm sharing with the hope that other people will be able to check and improve on what I've done. If you have questions or want to suggest changes/additional data you can leave a comment in [the gist](https://gist.github.com/gschivley/09257d239a88fcbd8981ca5e0589321e), find on on twitter ([@gschivley](https://twitter.com/gschivley)), or email me at *greg at carbonimpact dot co*."
]
},
{
"cell_type": "code",
"metadata": {
"id": "bHLnKurAEF4i",
"colab_type": "code",
"outputId": "de75bd7c-9766-44c3-a6c8-e93dd06a4b57",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
}
},
"source": [
"!wget https://raw.githubusercontent.com/gschivley/EIA_Cleaned_Hourly_Electricity_Demand_Code/master/anomaly_screening.py"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"--2020-03-25 19:19:34-- https://raw.githubusercontent.com/gschivley/EIA_Cleaned_Hourly_Electricity_Demand_Code/master/anomaly_screening.py\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 19011 (19K) [text/plain]\n",
"Saving to: ‘anomaly_screening.py’\n",
"\n",
"\ranomaly_screening.p 0%[ ] 0 --.-KB/s \ranomaly_screening.p 100%[===================>] 18.57K --.-KB/s in 0.01s \n",
"\n",
"2020-03-25 19:19:34 (1.52 MB/s) - ‘anomaly_screening.py’ saved [19011/19011]\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "bdX3O0YiZuYx",
"colab_type": "code",
"colab": {}
},
"source": [
"from itertools import combinations, chain\n",
"import pandas as pd\n",
"import numpy as np\n",
"from pathlib import Path\n",
"import zipfile\n",
"import urllib\n",
"from joblib import Parallel, delayed\n",
"\n",
"from anomaly_screening import screen_timeseries, make_anomaly_summary\n",
"\n",
"cwd = Path.cwd()\n",
"pd.set_option(\"display.max_columns\", 100)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "S71Ky-ibeJ6g",
"colab_type": "code",
"colab": {}
},
"source": [
"# Download the FERC 714 data to a temp folder that google is nice enough to host.\n",
"url = 'https://www.ferc.gov/docs-filing/forms/form-714/data/form714-database.zip'\n",
"save_folder = cwd / \"FERC\"\n",
"save_folder.mkdir(parents=True, exist_ok=True)\n",
"urllib.request.urlretrieve(url, save_folder / 'form714-database.zip')\n",
"### Unzip it\n",
"data_path = save_folder / \"form714-database\"\n",
"with zipfile.ZipFile(save_folder / 'form714-database.zip', 'r') as zfile:\n",
" zfile.extractall(data_path)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "yEwru4tCfgfS",
"colab_type": "code",
"outputId": "9ea7894a-fe49-431e-d960-78b11d446680",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 224
}
},
"source": [
"df = pd.read_csv(\n",
" data_path / \"Part 3 Schedule 2 - Planning Area Hourly Demand.csv\",\n",
" parse_dates=[\"plan_date\"], infer_datetime_format=True\n",
")\n",
"df.head()"
],
"execution_count": 5,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_id</th>\n",
" <th>report_yr</th>\n",
" <th>report_prd</th>\n",
" <th>spplmnt_num</th>\n",
" <th>row_num</th>\n",
" <th>plan_date</th>\n",
" <th>timezone</th>\n",
" <th>hour01</th>\n",
" <th>hour02</th>\n",
" <th>hour03</th>\n",
" <th>hour04</th>\n",
" <th>hour05</th>\n",
" <th>hour06</th>\n",
" <th>hour07</th>\n",
" <th>hour08</th>\n",
" <th>hour09</th>\n",
" <th>hour10</th>\n",
" <th>hour11</th>\n",
" <th>hour12</th>\n",
" <th>hour13</th>\n",
" <th>hour14</th>\n",
" <th>hour15</th>\n",
" <th>hour16</th>\n",
" <th>hour17</th>\n",
" <th>hour18</th>\n",
" <th>hour19</th>\n",
" <th>hour20</th>\n",
" <th>hour21</th>\n",
" <th>hour22</th>\n",
" <th>hour23</th>\n",
" <th>hour24</th>\n",
" <th>hour25</th>\n",
" <th>timezone_f</th>\n",
" <th>hour01_f</th>\n",
" <th>hour02_f</th>\n",
" <th>hour03_f</th>\n",
" <th>hour04_f</th>\n",
" <th>hour05_f</th>\n",
" <th>hour06_f</th>\n",
" <th>hour07_f</th>\n",
" <th>hour08_f</th>\n",
" <th>hour09_f</th>\n",
" <th>hour10_f</th>\n",
" <th>hour11_f</th>\n",
" <th>hour12_f</th>\n",
" <th>hour13_f</th>\n",
" <th>hour14_f</th>\n",
" <th>hour15_f</th>\n",
" <th>hour16_f</th>\n",
" <th>hour17_f</th>\n",
" <th>hour18_f</th>\n",
" <th>hour19_f</th>\n",
" <th>hour20_f</th>\n",
" <th>hour21_f</th>\n",
" <th>hour22_f</th>\n",
" <th>hour23_f</th>\n",
" <th>hour24_f</th>\n",
" <th>hour25_f</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2</td>\n",
" <td>2006</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>100</td>\n",
" <td>2006-01-01</td>\n",
" <td></td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>2006</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>200</td>\n",
" <td>2006-01-02</td>\n",
" <td></td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>2006</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>300</td>\n",
" <td>2006-01-03</td>\n",
" <td></td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2</td>\n",
" <td>2006</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>400</td>\n",
" <td>2006-01-04</td>\n",
" <td></td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>2006</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>500</td>\n",
" <td>2006-01-05</td>\n",
" <td></td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_id report_yr report_prd spplmnt_num row_num plan_date \\\n",
"0 2 2006 12 0 100 2006-01-01 \n",
"1 2 2006 12 0 200 2006-01-02 \n",
"2 2 2006 12 0 300 2006-01-03 \n",
"3 2 2006 12 0 400 2006-01-04 \n",
"4 2 2006 12 0 500 2006-01-05 \n",
"\n",
" timezone hour01 hour02 hour03 hour04 hour05 hour06 hour07 hour08 \\\n",
"0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"\n",
" hour09 hour10 hour11 hour12 hour13 hour14 hour15 hour16 hour17 \\\n",
"0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n",
"\n",
" hour18 hour19 hour20 hour21 hour22 hour23 hour24 hour25 timezone_f \\\n",
"0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 \n",
"1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 \n",
"2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 \n",
"3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 \n",
"4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 \n",
"\n",
" hour01_f hour02_f hour03_f hour04_f hour05_f hour06_f hour07_f \\\n",
"0 0 0 0 0 0 0 0 \n",
"1 0 0 0 0 0 0 0 \n",
"2 0 0 0 0 0 0 0 \n",
"3 0 0 0 0 0 0 0 \n",
"4 0 0 0 0 0 0 0 \n",
"\n",
" hour08_f hour09_f hour10_f hour11_f hour12_f hour13_f hour14_f \\\n",
"0 0 0 0 0 0 0 0 \n",
"1 0 0 0 0 0 0 0 \n",
"2 0 0 0 0 0 0 0 \n",
"3 0 0 0 0 0 0 0 \n",
"4 0 0 0 0 0 0 0 \n",
"\n",
" hour15_f hour16_f hour17_f hour18_f hour19_f hour20_f hour21_f \\\n",
"0 0 0 0 0 0 0 0 \n",
"1 0 0 0 0 0 0 0 \n",
"2 0 0 0 0 0 0 0 \n",
"3 0 0 0 0 0 0 0 \n",
"4 0 0 0 0 0 0 0 \n",
"\n",
" hour22_f hour23_f hour24_f hour25_f \n",
"0 0 0 0 0 \n",
"1 0 0 0 0 \n",
"2 0 0 0 0 \n",
"3 0 0 0 0 \n",
"4 0 0 0 0 "
]
},
"metadata": {
"tags": []
},
"execution_count": 5
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "jU9KXMvnfgN4",
"colab_type": "code",
"outputId": "a0504a7b-17a8-42e2-a194-fa8de7126615",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 187
}
},
"source": [
"respondents = df.loc[df.report_yr >=2011,\"respondent_id\"].unique()\n",
"\n",
"respondents"
],
"execution_count": 6,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"array([101, 102, 110, 115, 116, 118, 119, 121, 122, 124, 125, 128, 133,\n",
" 135, 138, 139, 140, 141, 142, 143, 150, 151, 156, 157, 159, 160,\n",
" 161, 162, 163, 164, 165, 166, 169, 171, 172, 173, 174, 177, 178,\n",
" 180, 182, 183, 185, 186, 187, 190, 191, 193, 194, 195, 197, 199,\n",
" 200, 201, 206, 209, 210, 211, 212, 213, 214, 217, 218, 219, 220,\n",
" 221, 223, 225, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239,\n",
" 240, 241, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 257,\n",
" 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 271, 272, 273,\n",
" 274, 275, 277, 280, 282, 283, 284, 285, 287, 289, 292, 296, 297,\n",
" 298, 299, 307, 308, 311, 321, 328, 329])"
]
},
"metadata": {
"tags": []
},
"execution_count": 6
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "On5aCkQVfgD4",
"colab_type": "code",
"colab": {}
},
"source": [
"def hour_25_looks_real(df):\n",
" \"\"\"\n",
" The FERC demand data has a column hour25 for daylight savings. Determine if\n",
" the value there looks like it could be a continuation of the series.\n",
" Sometimes it seems to be a sum of all values for the day or something else\n",
" weird.\n",
" \"\"\"\n",
" df[\"hour_25_valid\"] = False\n",
" df[\"hour24_25_ratio\"] = df[\"hour24\"] / df[\"hour25\"]\n",
" df.loc[\n",
" (df[\"hour24_25_ratio\"] > 0.6)\n",
" & (df[\"hour24_25_ratio\"] < 1.5),\n",
" \"hour_25_valid\"\n",
" ] = True\n",
" \n",
" df.loc[df[\"hour_25_valid\"] == False, \"hour25\"] = np.nan\n",
" \n",
" return df"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "xxf4Kh12ftBO",
"colab_type": "code",
"colab": {}
},
"source": [
"# Only keeping 2011/2012 for now. Explore more of the data if you want!\n",
"corrected_df = hour_25_looks_real(\n",
" df.loc[(df.report_yr.isin([2011, 2012])), :].copy()\n",
")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "AVxgDpMyftOa",
"colab_type": "code",
"outputId": "bd85def2-1884-4a27-a8f9-d506f35daec3",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 306
}
},
"source": [
"# Check to see how many valid hour25 values there are for each respondent.\n",
"# Looks like 311 uses hour25 all the time...\n",
"hour_25_count = {}\n",
"for r in corrected_df.dropna()[\"respondent_id\"].unique():\n",
" hour_25_count[r] = len(corrected_df.query(\"respondent_id==@r\").dropna())\n",
"\n",
"hour_25_count"
],
"execution_count": 9,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{121: 2,\n",
" 139: 2,\n",
" 141: 1,\n",
" 163: 1,\n",
" 166: 1,\n",
" 169: 1,\n",
" 195: 2,\n",
" 212: 2,\n",
" 225: 2,\n",
" 235: 2,\n",
" 237: 1,\n",
" 241: 2,\n",
" 247: 2,\n",
" 259: 2,\n",
" 275: 1,\n",
" 308: 1,\n",
" 311: 365}"
]
},
"metadata": {
"tags": []
},
"execution_count": 9
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZDOBllPNftce",
"colab_type": "code",
"outputId": "d2536bbe-0c66-4552-8971-4e45371e864d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 844
}
},
"source": [
"# hour25 values do appear to be on dayslight savings change\n",
"corrected_df.query(\"hour25 > 0 & respondent_id != 311\").sort_values(\"report_yr\")"
],
"execution_count": 10,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_id</th>\n",
" <th>report_yr</th>\n",
" <th>report_prd</th>\n",
" <th>spplmnt_num</th>\n",
" <th>row_num</th>\n",
" <th>plan_date</th>\n",
" <th>timezone</th>\n",
" <th>hour01</th>\n",
" <th>hour02</th>\n",
" <th>hour03</th>\n",
" <th>hour04</th>\n",
" <th>hour05</th>\n",
" <th>hour06</th>\n",
" <th>hour07</th>\n",
" <th>hour08</th>\n",
" <th>hour09</th>\n",
" <th>hour10</th>\n",
" <th>hour11</th>\n",
" <th>hour12</th>\n",
" <th>hour13</th>\n",
" <th>hour14</th>\n",
" <th>hour15</th>\n",
" <th>hour16</th>\n",
" <th>hour17</th>\n",
" <th>hour18</th>\n",
" <th>hour19</th>\n",
" <th>hour20</th>\n",
" <th>hour21</th>\n",
" <th>hour22</th>\n",
" <th>hour23</th>\n",
" <th>hour24</th>\n",
" <th>hour25</th>\n",
" <th>timezone_f</th>\n",
" <th>hour01_f</th>\n",
" <th>hour02_f</th>\n",
" <th>hour03_f</th>\n",
" <th>hour04_f</th>\n",
" <th>hour05_f</th>\n",
" <th>hour06_f</th>\n",
" <th>hour07_f</th>\n",
" <th>hour08_f</th>\n",
" <th>hour09_f</th>\n",
" <th>hour10_f</th>\n",
" <th>hour11_f</th>\n",
" <th>hour12_f</th>\n",
" <th>hour13_f</th>\n",
" <th>hour14_f</th>\n",
" <th>hour15_f</th>\n",
" <th>hour16_f</th>\n",
" <th>hour17_f</th>\n",
" <th>hour18_f</th>\n",
" <th>hour19_f</th>\n",
" <th>hour20_f</th>\n",
" <th>hour21_f</th>\n",
" <th>hour22_f</th>\n",
" <th>hour23_f</th>\n",
" <th>hour24_f</th>\n",
" <th>hour25_f</th>\n",
" <th>hour_25_valid</th>\n",
" <th>hour24_25_ratio</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>44501</th>\n",
" <td>121</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>mst</td>\n",
" <td>338.00</td>\n",
" <td>340.00</td>\n",
" <td>332.00</td>\n",
" <td>328.00</td>\n",
" <td>336.00</td>\n",
" <td>334.00</td>\n",
" <td>345.00</td>\n",
" <td>358.00</td>\n",
" <td>368.00</td>\n",
" <td>386.00</td>\n",
" <td>382.00</td>\n",
" <td>388.00</td>\n",
" <td>375.00</td>\n",
" <td>375.00</td>\n",
" <td>388.00</td>\n",
" <td>367.00</td>\n",
" <td>380.00</td>\n",
" <td>395.00</td>\n",
" <td>432.00</td>\n",
" <td>424.00</td>\n",
" <td>427.00</td>\n",
" <td>418.00</td>\n",
" <td>392.00</td>\n",
" <td>374.00</td>\n",
" <td>364.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.027473</td>\n",
" </tr>\n",
" <tr>\n",
" <th>446989</th>\n",
" <td>259</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>CDT</td>\n",
" <td>2866.00</td>\n",
" <td>2822.00</td>\n",
" <td>2811.00</td>\n",
" <td>2795.00</td>\n",
" <td>2797.00</td>\n",
" <td>2844.00</td>\n",
" <td>2879.00</td>\n",
" <td>2962.00</td>\n",
" <td>3042.00</td>\n",
" <td>3104.00</td>\n",
" <td>3102.00</td>\n",
" <td>3085.00</td>\n",
" <td>3058.00</td>\n",
" <td>3056.00</td>\n",
" <td>3047.00</td>\n",
" <td>3043.00</td>\n",
" <td>3023.00</td>\n",
" <td>3048.00</td>\n",
" <td>3085.00</td>\n",
" <td>3220.00</td>\n",
" <td>3218.00</td>\n",
" <td>3178.00</td>\n",
" <td>3120.00</td>\n",
" <td>3024.00</td>\n",
" <td>2920.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.035616</td>\n",
" </tr>\n",
" <tr>\n",
" <th>410830</th>\n",
" <td>247</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>PST</td>\n",
" <td>1060.00</td>\n",
" <td>1028.00</td>\n",
" <td>945.00</td>\n",
" <td>965.00</td>\n",
" <td>1019.00</td>\n",
" <td>1076.00</td>\n",
" <td>1149.00</td>\n",
" <td>1221.00</td>\n",
" <td>1299.00</td>\n",
" <td>1323.00</td>\n",
" <td>1305.00</td>\n",
" <td>1283.00</td>\n",
" <td>1242.00</td>\n",
" <td>1217.00</td>\n",
" <td>1216.00</td>\n",
" <td>1225.00</td>\n",
" <td>1303.00</td>\n",
" <td>1410.00</td>\n",
" <td>1404.00</td>\n",
" <td>1383.00</td>\n",
" <td>1342.00</td>\n",
" <td>1267.00</td>\n",
" <td>1158.00</td>\n",
" <td>1072.00</td>\n",
" <td>981.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.092762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>387820</th>\n",
" <td>241</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>EST</td>\n",
" <td>685.00</td>\n",
" <td>658.00</td>\n",
" <td>633.00</td>\n",
" <td>630.00</td>\n",
" <td>640.00</td>\n",
" <td>662.00</td>\n",
" <td>693.00</td>\n",
" <td>720.00</td>\n",
" <td>749.00</td>\n",
" <td>768.00</td>\n",
" <td>772.00</td>\n",
" <td>770.00</td>\n",
" <td>772.00</td>\n",
" <td>760.00</td>\n",
" <td>762.00</td>\n",
" <td>762.00</td>\n",
" <td>797.00</td>\n",
" <td>890.00</td>\n",
" <td>893.00</td>\n",
" <td>872.00</td>\n",
" <td>838.00</td>\n",
" <td>791.00</td>\n",
" <td>730.00</td>\n",
" <td>683.00</td>\n",
" <td>645.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.058915</td>\n",
" </tr>\n",
" <tr>\n",
" <th>359332</th>\n",
" <td>235</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>MST</td>\n",
" <td>4045.00</td>\n",
" <td>3915.00</td>\n",
" <td>3856.00</td>\n",
" <td>3869.00</td>\n",
" <td>3881.00</td>\n",
" <td>3920.00</td>\n",
" <td>4061.00</td>\n",
" <td>4200.00</td>\n",
" <td>4300.00</td>\n",
" <td>4490.00</td>\n",
" <td>4544.00</td>\n",
" <td>4523.00</td>\n",
" <td>4491.00</td>\n",
" <td>4504.00</td>\n",
" <td>4471.00</td>\n",
" <td>4471.00</td>\n",
" <td>4512.00</td>\n",
" <td>4724.00</td>\n",
" <td>5269.00</td>\n",
" <td>5337.00</td>\n",
" <td>5249.00</td>\n",
" <td>5070.00</td>\n",
" <td>4781.00</td>\n",
" <td>4415.00</td>\n",
" <td>4110.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.074209</td>\n",
" </tr>\n",
" <tr>\n",
" <th>505792</th>\n",
" <td>275</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>CPT</td>\n",
" <td>91.00</td>\n",
" <td>86.00</td>\n",
" <td>84.00</td>\n",
" <td>84.00</td>\n",
" <td>83.00</td>\n",
" <td>86.00</td>\n",
" <td>84.00</td>\n",
" <td>91.00</td>\n",
" <td>98.00</td>\n",
" <td>99.00</td>\n",
" <td>97.00</td>\n",
" <td>97.00</td>\n",
" <td>89.00</td>\n",
" <td>94.00</td>\n",
" <td>96.00</td>\n",
" <td>95.00</td>\n",
" <td>96.00</td>\n",
" <td>94.00</td>\n",
" <td>98.00</td>\n",
" <td>110.00</td>\n",
" <td>108.00</td>\n",
" <td>105.00</td>\n",
" <td>105.00</td>\n",
" <td>102.00</td>\n",
" <td>99.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.030303</td>\n",
" </tr>\n",
" <tr>\n",
" <th>284094</th>\n",
" <td>212</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>EST</td>\n",
" <td>1528.00</td>\n",
" <td>1487.00</td>\n",
" <td>1421.00</td>\n",
" <td>1420.00</td>\n",
" <td>1448.00</td>\n",
" <td>1506.00</td>\n",
" <td>1599.00</td>\n",
" <td>1684.00</td>\n",
" <td>1750.00</td>\n",
" <td>1760.00</td>\n",
" <td>1735.00</td>\n",
" <td>1704.00</td>\n",
" <td>1669.00</td>\n",
" <td>1642.00</td>\n",
" <td>1613.00</td>\n",
" <td>1632.00</td>\n",
" <td>1735.00</td>\n",
" <td>1956.00</td>\n",
" <td>1965.00</td>\n",
" <td>1919.00</td>\n",
" <td>1850.00</td>\n",
" <td>1716.00</td>\n",
" <td>1586.00</td>\n",
" <td>1502.00</td>\n",
" <td>1446.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.038728</td>\n",
" </tr>\n",
" <tr>\n",
" <th>238805</th>\n",
" <td>195</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>CST</td>\n",
" <td>74.80</td>\n",
" <td>70.20</td>\n",
" <td>68.20</td>\n",
" <td>67.00</td>\n",
" <td>65.90</td>\n",
" <td>69.40</td>\n",
" <td>72.40</td>\n",
" <td>72.40</td>\n",
" <td>76.20</td>\n",
" <td>81.70</td>\n",
" <td>83.80</td>\n",
" <td>88.00</td>\n",
" <td>90.30</td>\n",
" <td>92.10</td>\n",
" <td>95.60</td>\n",
" <td>95.50</td>\n",
" <td>97.10</td>\n",
" <td>96.60</td>\n",
" <td>103.00</td>\n",
" <td>109.00</td>\n",
" <td>106.90</td>\n",
" <td>101.70</td>\n",
" <td>93.60</td>\n",
" <td>83.70</td>\n",
" <td>76.40</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.095550</td>\n",
" </tr>\n",
" <tr>\n",
" <th>327192</th>\n",
" <td>225</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>EST</td>\n",
" <td>446.00</td>\n",
" <td>405.00</td>\n",
" <td>399.00</td>\n",
" <td>395.00</td>\n",
" <td>389.00</td>\n",
" <td>391.00</td>\n",
" <td>408.00</td>\n",
" <td>429.00</td>\n",
" <td>441.00</td>\n",
" <td>479.00</td>\n",
" <td>517.00</td>\n",
" <td>540.00</td>\n",
" <td>544.00</td>\n",
" <td>552.00</td>\n",
" <td>546.00</td>\n",
" <td>552.00</td>\n",
" <td>553.00</td>\n",
" <td>554.00</td>\n",
" <td>566.00</td>\n",
" <td>599.00</td>\n",
" <td>592.00</td>\n",
" <td>571.00</td>\n",
" <td>535.00</td>\n",
" <td>503.00</td>\n",
" <td>468.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.074786</td>\n",
" </tr>\n",
" <tr>\n",
" <th>146765</th>\n",
" <td>163</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>CPT</td>\n",
" <td>417.00</td>\n",
" <td>400.00</td>\n",
" <td>384.00</td>\n",
" <td>388.00</td>\n",
" <td>385.00</td>\n",
" <td>387.00</td>\n",
" <td>399.00</td>\n",
" <td>419.00</td>\n",
" <td>443.00</td>\n",
" <td>484.00</td>\n",
" <td>489.00</td>\n",
" <td>479.00</td>\n",
" <td>471.00</td>\n",
" <td>469.00</td>\n",
" <td>474.00</td>\n",
" <td>460.00</td>\n",
" <td>463.00</td>\n",
" <td>486.00</td>\n",
" <td>513.00</td>\n",
" <td>527.00</td>\n",
" <td>520.00</td>\n",
" <td>504.00</td>\n",
" <td>474.00</td>\n",
" <td>438.00</td>\n",
" <td>408.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.073529</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91615</th>\n",
" <td>141</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>CST</td>\n",
" <td>270.00</td>\n",
" <td>259.00</td>\n",
" <td>253.00</td>\n",
" <td>250.00</td>\n",
" <td>247.00</td>\n",
" <td>248.00</td>\n",
" <td>260.00</td>\n",
" <td>267.00</td>\n",
" <td>275.00</td>\n",
" <td>283.00</td>\n",
" <td>295.00</td>\n",
" <td>296.00</td>\n",
" <td>303.00</td>\n",
" <td>307.00</td>\n",
" <td>306.00</td>\n",
" <td>308.00</td>\n",
" <td>313.00</td>\n",
" <td>322.00</td>\n",
" <td>352.00</td>\n",
" <td>345.00</td>\n",
" <td>338.00</td>\n",
" <td>329.00</td>\n",
" <td>309.00</td>\n",
" <td>293.00</td>\n",
" <td>257.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.140078</td>\n",
" </tr>\n",
" <tr>\n",
" <th>82850</th>\n",
" <td>139</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31000</td>\n",
" <td>2011-11-06</td>\n",
" <td>PST</td>\n",
" <td>562.00</td>\n",
" <td>564.00</td>\n",
" <td>541.00</td>\n",
" <td>552.00</td>\n",
" <td>560.00</td>\n",
" <td>583.00</td>\n",
" <td>613.00</td>\n",
" <td>661.00</td>\n",
" <td>695.00</td>\n",
" <td>718.00</td>\n",
" <td>714.00</td>\n",
" <td>682.00</td>\n",
" <td>647.00</td>\n",
" <td>626.00</td>\n",
" <td>603.00</td>\n",
" <td>596.00</td>\n",
" <td>599.00</td>\n",
" <td>637.00</td>\n",
" <td>703.00</td>\n",
" <td>700.00</td>\n",
" <td>696.00</td>\n",
" <td>677.00</td>\n",
" <td>629.00</td>\n",
" <td>585.00</td>\n",
" <td>547.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.069470</td>\n",
" </tr>\n",
" <tr>\n",
" <th>239171</th>\n",
" <td>195</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>31100</td>\n",
" <td>2012-11-06</td>\n",
" <td>CST</td>\n",
" <td>82.79</td>\n",
" <td>75.91</td>\n",
" <td>70.92</td>\n",
" <td>68.37</td>\n",
" <td>68.91</td>\n",
" <td>71.16</td>\n",
" <td>77.18</td>\n",
" <td>82.18</td>\n",
" <td>87.64</td>\n",
" <td>91.26</td>\n",
" <td>94.93</td>\n",
" <td>98.90</td>\n",
" <td>97.81</td>\n",
" <td>96.46</td>\n",
" <td>95.24</td>\n",
" <td>95.36</td>\n",
" <td>96.72</td>\n",
" <td>94.96</td>\n",
" <td>101.53</td>\n",
" <td>108.20</td>\n",
" <td>106.60</td>\n",
" <td>102.74</td>\n",
" <td>97.12</td>\n",
" <td>87.61</td>\n",
" <td>79.48</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.102290</td>\n",
" </tr>\n",
" <tr>\n",
" <th>157721</th>\n",
" <td>166</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>PST</td>\n",
" <td>213.20</td>\n",
" <td>206.12</td>\n",
" <td>192.77</td>\n",
" <td>192.43</td>\n",
" <td>193.20</td>\n",
" <td>200.71</td>\n",
" <td>212.40</td>\n",
" <td>229.44</td>\n",
" <td>247.77</td>\n",
" <td>256.13</td>\n",
" <td>259.82</td>\n",
" <td>260.43</td>\n",
" <td>260.01</td>\n",
" <td>252.14</td>\n",
" <td>250.24</td>\n",
" <td>253.34</td>\n",
" <td>260.63</td>\n",
" <td>288.69</td>\n",
" <td>290.01</td>\n",
" <td>281.82</td>\n",
" <td>270.71</td>\n",
" <td>251.55</td>\n",
" <td>231.91</td>\n",
" <td>213.71</td>\n",
" <td>193.13</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.106560</td>\n",
" </tr>\n",
" <tr>\n",
" <th>284458</th>\n",
" <td>212</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>EST</td>\n",
" <td>1501.00</td>\n",
" <td>1449.00</td>\n",
" <td>1372.00</td>\n",
" <td>1367.00</td>\n",
" <td>1386.00</td>\n",
" <td>1452.00</td>\n",
" <td>1523.00</td>\n",
" <td>1615.00</td>\n",
" <td>1710.00</td>\n",
" <td>1731.00</td>\n",
" <td>1785.00</td>\n",
" <td>1773.00</td>\n",
" <td>1782.00</td>\n",
" <td>1752.00</td>\n",
" <td>1775.00</td>\n",
" <td>1801.00</td>\n",
" <td>1917.00</td>\n",
" <td>2066.00</td>\n",
" <td>2052.00</td>\n",
" <td>1994.00</td>\n",
" <td>1910.00</td>\n",
" <td>1781.00</td>\n",
" <td>1646.00</td>\n",
" <td>1558.00</td>\n",
" <td>1372.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.135569</td>\n",
" </tr>\n",
" <tr>\n",
" <th>327556</th>\n",
" <td>225</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>EPT</td>\n",
" <td>532.00</td>\n",
" <td>444.00</td>\n",
" <td>423.00</td>\n",
" <td>415.00</td>\n",
" <td>406.00</td>\n",
" <td>406.00</td>\n",
" <td>423.00</td>\n",
" <td>430.00</td>\n",
" <td>451.00</td>\n",
" <td>493.00</td>\n",
" <td>532.00</td>\n",
" <td>568.00</td>\n",
" <td>597.00</td>\n",
" <td>612.00</td>\n",
" <td>634.00</td>\n",
" <td>642.00</td>\n",
" <td>654.00</td>\n",
" <td>647.00</td>\n",
" <td>655.00</td>\n",
" <td>676.00</td>\n",
" <td>652.00</td>\n",
" <td>621.00</td>\n",
" <td>580.00</td>\n",
" <td>539.00</td>\n",
" <td>492.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.095528</td>\n",
" </tr>\n",
" <tr>\n",
" <th>359696</th>\n",
" <td>235</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>MDT</td>\n",
" <td>3739.00</td>\n",
" <td>3676.00</td>\n",
" <td>3635.00</td>\n",
" <td>3606.00</td>\n",
" <td>3569.00</td>\n",
" <td>3619.00</td>\n",
" <td>3801.00</td>\n",
" <td>3921.00</td>\n",
" <td>4106.00</td>\n",
" <td>4295.00</td>\n",
" <td>4339.00</td>\n",
" <td>4338.00</td>\n",
" <td>4288.00</td>\n",
" <td>4212.00</td>\n",
" <td>4160.00</td>\n",
" <td>4163.00</td>\n",
" <td>4260.00</td>\n",
" <td>4470.00</td>\n",
" <td>4952.00</td>\n",
" <td>4961.00</td>\n",
" <td>4826.00</td>\n",
" <td>4637.00</td>\n",
" <td>4338.00</td>\n",
" <td>3954.00</td>\n",
" <td>3760.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.051596</td>\n",
" </tr>\n",
" <tr>\n",
" <th>369192</th>\n",
" <td>237</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>PST</td>\n",
" <td>399.00</td>\n",
" <td>394.00</td>\n",
" <td>394.00</td>\n",
" <td>394.00</td>\n",
" <td>400.00</td>\n",
" <td>410.00</td>\n",
" <td>420.00</td>\n",
" <td>433.00</td>\n",
" <td>431.00</td>\n",
" <td>430.00</td>\n",
" <td>425.00</td>\n",
" <td>419.00</td>\n",
" <td>413.00</td>\n",
" <td>414.00</td>\n",
" <td>403.00</td>\n",
" <td>406.00</td>\n",
" <td>416.00</td>\n",
" <td>430.00</td>\n",
" <td>425.00</td>\n",
" <td>424.00</td>\n",
" <td>415.00</td>\n",
" <td>411.00</td>\n",
" <td>401.00</td>\n",
" <td>391.00</td>\n",
" <td>397.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>0.984887</td>\n",
" </tr>\n",
" <tr>\n",
" <th>83214</th>\n",
" <td>139</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>PST</td>\n",
" <td>437.00</td>\n",
" <td>429.00</td>\n",
" <td>406.00</td>\n",
" <td>391.00</td>\n",
" <td>396.00</td>\n",
" <td>404.00</td>\n",
" <td>417.00</td>\n",
" <td>448.00</td>\n",
" <td>469.00</td>\n",
" <td>506.00</td>\n",
" <td>527.00</td>\n",
" <td>525.00</td>\n",
" <td>522.00</td>\n",
" <td>525.00</td>\n",
" <td>514.00</td>\n",
" <td>511.00</td>\n",
" <td>503.00</td>\n",
" <td>528.00</td>\n",
" <td>574.00</td>\n",
" <td>575.00</td>\n",
" <td>563.00</td>\n",
" <td>547.00</td>\n",
" <td>503.00</td>\n",
" <td>466.00</td>\n",
" <td>425.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.096471</td>\n",
" </tr>\n",
" <tr>\n",
" <th>388184</th>\n",
" <td>241</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>EST</td>\n",
" <td>684.00</td>\n",
" <td>654.00</td>\n",
" <td>626.00</td>\n",
" <td>624.00</td>\n",
" <td>632.00</td>\n",
" <td>645.00</td>\n",
" <td>684.00</td>\n",
" <td>708.00</td>\n",
" <td>744.00</td>\n",
" <td>775.00</td>\n",
" <td>792.00</td>\n",
" <td>802.00</td>\n",
" <td>816.00</td>\n",
" <td>818.00</td>\n",
" <td>819.00</td>\n",
" <td>830.00</td>\n",
" <td>880.00</td>\n",
" <td>944.00</td>\n",
" <td>943.00</td>\n",
" <td>914.00</td>\n",
" <td>880.00</td>\n",
" <td>827.00</td>\n",
" <td>766.00</td>\n",
" <td>713.00</td>\n",
" <td>641.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.112324</td>\n",
" </tr>\n",
" <tr>\n",
" <th>411194</th>\n",
" <td>247</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>PST</td>\n",
" <td>917.00</td>\n",
" <td>873.00</td>\n",
" <td>1671.00</td>\n",
" <td>832.00</td>\n",
" <td>839.00</td>\n",
" <td>872.00</td>\n",
" <td>914.00</td>\n",
" <td>981.00</td>\n",
" <td>1063.00</td>\n",
" <td>1125.00</td>\n",
" <td>1137.00</td>\n",
" <td>1143.00</td>\n",
" <td>1133.00</td>\n",
" <td>1130.00</td>\n",
" <td>1135.00</td>\n",
" <td>1141.00</td>\n",
" <td>1166.00</td>\n",
" <td>1268.00</td>\n",
" <td>1252.00</td>\n",
" <td>1230.00</td>\n",
" <td>1185.00</td>\n",
" <td>1127.00</td>\n",
" <td>1039.00</td>\n",
" <td>940.00</td>\n",
" <td>848.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.108491</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44865</th>\n",
" <td>121</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>mst</td>\n",
" <td>330.00</td>\n",
" <td>316.00</td>\n",
" <td>313.00</td>\n",
" <td>318.00</td>\n",
" <td>316.00</td>\n",
" <td>316.00</td>\n",
" <td>329.00</td>\n",
" <td>342.00</td>\n",
" <td>359.00</td>\n",
" <td>372.00</td>\n",
" <td>375.00</td>\n",
" <td>377.00</td>\n",
" <td>376.00</td>\n",
" <td>385.00</td>\n",
" <td>369.00</td>\n",
" <td>372.00</td>\n",
" <td>377.00</td>\n",
" <td>392.00</td>\n",
" <td>420.00</td>\n",
" <td>408.00</td>\n",
" <td>412.00</td>\n",
" <td>392.00</td>\n",
" <td>374.00</td>\n",
" <td>343.00</td>\n",
" <td>336.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.020833</td>\n",
" </tr>\n",
" <tr>\n",
" <th>447353</th>\n",
" <td>259</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>CST</td>\n",
" <td>3107.00</td>\n",
" <td>3076.00</td>\n",
" <td>3049.00</td>\n",
" <td>3046.00</td>\n",
" <td>3044.00</td>\n",
" <td>3052.00</td>\n",
" <td>3087.00</td>\n",
" <td>3154.00</td>\n",
" <td>3234.00</td>\n",
" <td>3294.00</td>\n",
" <td>3311.00</td>\n",
" <td>3294.00</td>\n",
" <td>3293.00</td>\n",
" <td>3296.00</td>\n",
" <td>3299.00</td>\n",
" <td>3321.00</td>\n",
" <td>3327.00</td>\n",
" <td>3350.00</td>\n",
" <td>3368.00</td>\n",
" <td>3465.00</td>\n",
" <td>3438.00</td>\n",
" <td>3382.00</td>\n",
" <td>3316.00</td>\n",
" <td>3201.00</td>\n",
" <td>3067.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.043691</td>\n",
" </tr>\n",
" <tr>\n",
" <th>163565</th>\n",
" <td>169</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>EDT</td>\n",
" <td>430.00</td>\n",
" <td>429.00</td>\n",
" <td>389.00</td>\n",
" <td>376.00</td>\n",
" <td>373.00</td>\n",
" <td>374.00</td>\n",
" <td>389.00</td>\n",
" <td>412.00</td>\n",
" <td>456.00</td>\n",
" <td>517.00</td>\n",
" <td>564.00</td>\n",
" <td>600.00</td>\n",
" <td>627.00</td>\n",
" <td>656.00</td>\n",
" <td>678.00</td>\n",
" <td>690.00</td>\n",
" <td>695.00</td>\n",
" <td>684.00</td>\n",
" <td>682.00</td>\n",
" <td>711.00</td>\n",
" <td>683.00</td>\n",
" <td>643.00</td>\n",
" <td>587.00</td>\n",
" <td>523.00</td>\n",
" <td>460.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.136957</td>\n",
" </tr>\n",
" <tr>\n",
" <th>576280</th>\n",
" <td>308</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>30900</td>\n",
" <td>2012-11-04</td>\n",
" <td>EPT</td>\n",
" <td>37.00</td>\n",
" <td>38.00</td>\n",
" <td>36.00</td>\n",
" <td>33.00</td>\n",
" <td>33.00</td>\n",
" <td>33.00</td>\n",
" <td>34.00</td>\n",
" <td>39.00</td>\n",
" <td>46.00</td>\n",
" <td>52.00</td>\n",
" <td>59.00</td>\n",
" <td>61.00</td>\n",
" <td>66.00</td>\n",
" <td>71.00</td>\n",
" <td>73.00</td>\n",
" <td>77.00</td>\n",
" <td>77.00</td>\n",
" <td>76.00</td>\n",
" <td>77.00</td>\n",
" <td>80.00</td>\n",
" <td>75.00</td>\n",
" <td>69.00</td>\n",
" <td>62.00</td>\n",
" <td>52.00</td>\n",
" <td>44.00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>True</td>\n",
" <td>1.181818</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_id report_yr report_prd spplmnt_num row_num plan_date \\\n",
"44501 121 2011 12 0 31000 2011-11-06 \n",
"446989 259 2011 12 0 31000 2011-11-06 \n",
"410830 247 2011 12 0 31000 2011-11-06 \n",
"387820 241 2011 12 0 31000 2011-11-06 \n",
"359332 235 2011 12 0 31000 2011-11-06 \n",
"505792 275 2011 12 0 31000 2011-11-06 \n",
"284094 212 2011 12 0 31000 2011-11-06 \n",
"238805 195 2011 12 0 31000 2011-11-06 \n",
"327192 225 2011 12 0 31000 2011-11-06 \n",
"146765 163 2011 12 0 31000 2011-11-06 \n",
"91615 141 2011 12 0 31000 2011-11-06 \n",
"82850 139 2011 12 0 31000 2011-11-06 \n",
"239171 195 2012 12 0 31100 2012-11-06 \n",
"157721 166 2012 12 0 30900 2012-11-04 \n",
"284458 212 2012 12 0 30900 2012-11-04 \n",
"327556 225 2012 12 0 30900 2012-11-04 \n",
"359696 235 2012 12 0 30900 2012-11-04 \n",
"369192 237 2012 12 0 30900 2012-11-04 \n",
"83214 139 2012 12 0 30900 2012-11-04 \n",
"388184 241 2012 12 0 30900 2012-11-04 \n",
"411194 247 2012 12 0 30900 2012-11-04 \n",
"44865 121 2012 12 0 30900 2012-11-04 \n",
"447353 259 2012 12 0 30900 2012-11-04 \n",
"163565 169 2012 12 0 30900 2012-11-04 \n",
"576280 308 2012 12 0 30900 2012-11-04 \n",
"\n",
" timezone hour01 hour02 hour03 hour04 hour05 hour06 \\\n",
"44501 mst 338.00 340.00 332.00 328.00 336.00 334.00 \n",
"446989 CDT 2866.00 2822.00 2811.00 2795.00 2797.00 2844.00 \n",
"410830 PST 1060.00 1028.00 945.00 965.00 1019.00 1076.00 \n",
"387820 EST 685.00 658.00 633.00 630.00 640.00 662.00 \n",
"359332 MST 4045.00 3915.00 3856.00 3869.00 3881.00 3920.00 \n",
"505792 CPT 91.00 86.00 84.00 84.00 83.00 86.00 \n",
"284094 EST 1528.00 1487.00 1421.00 1420.00 1448.00 1506.00 \n",
"238805 CST 74.80 70.20 68.20 67.00 65.90 69.40 \n",
"327192 EST 446.00 405.00 399.00 395.00 389.00 391.00 \n",
"146765 CPT 417.00 400.00 384.00 388.00 385.00 387.00 \n",
"91615 CST 270.00 259.00 253.00 250.00 247.00 248.00 \n",
"82850 PST 562.00 564.00 541.00 552.00 560.00 583.00 \n",
"239171 CST 82.79 75.91 70.92 68.37 68.91 71.16 \n",
"157721 PST 213.20 206.12 192.77 192.43 193.20 200.71 \n",
"284458 EST 1501.00 1449.00 1372.00 1367.00 1386.00 1452.00 \n",
"327556 EPT 532.00 444.00 423.00 415.00 406.00 406.00 \n",
"359696 MDT 3739.00 3676.00 3635.00 3606.00 3569.00 3619.00 \n",
"369192 PST 399.00 394.00 394.00 394.00 400.00 410.00 \n",
"83214 PST 437.00 429.00 406.00 391.00 396.00 404.00 \n",
"388184 EST 684.00 654.00 626.00 624.00 632.00 645.00 \n",
"411194 PST 917.00 873.00 1671.00 832.00 839.00 872.00 \n",
"44865 mst 330.00 316.00 313.00 318.00 316.00 316.00 \n",
"447353 CST 3107.00 3076.00 3049.00 3046.00 3044.00 3052.00 \n",
"163565 EDT 430.00 429.00 389.00 376.00 373.00 374.00 \n",
"576280 EPT 37.00 38.00 36.00 33.00 33.00 33.00 \n",
"\n",
" hour07 hour08 hour09 hour10 hour11 hour12 hour13 \\\n",
"44501 345.00 358.00 368.00 386.00 382.00 388.00 375.00 \n",
"446989 2879.00 2962.00 3042.00 3104.00 3102.00 3085.00 3058.00 \n",
"410830 1149.00 1221.00 1299.00 1323.00 1305.00 1283.00 1242.00 \n",
"387820 693.00 720.00 749.00 768.00 772.00 770.00 772.00 \n",
"359332 4061.00 4200.00 4300.00 4490.00 4544.00 4523.00 4491.00 \n",
"505792 84.00 91.00 98.00 99.00 97.00 97.00 89.00 \n",
"284094 1599.00 1684.00 1750.00 1760.00 1735.00 1704.00 1669.00 \n",
"238805 72.40 72.40 76.20 81.70 83.80 88.00 90.30 \n",
"327192 408.00 429.00 441.00 479.00 517.00 540.00 544.00 \n",
"146765 399.00 419.00 443.00 484.00 489.00 479.00 471.00 \n",
"91615 260.00 267.00 275.00 283.00 295.00 296.00 303.00 \n",
"82850 613.00 661.00 695.00 718.00 714.00 682.00 647.00 \n",
"239171 77.18 82.18 87.64 91.26 94.93 98.90 97.81 \n",
"157721 212.40 229.44 247.77 256.13 259.82 260.43 260.01 \n",
"284458 1523.00 1615.00 1710.00 1731.00 1785.00 1773.00 1782.00 \n",
"327556 423.00 430.00 451.00 493.00 532.00 568.00 597.00 \n",
"359696 3801.00 3921.00 4106.00 4295.00 4339.00 4338.00 4288.00 \n",
"369192 420.00 433.00 431.00 430.00 425.00 419.00 413.00 \n",
"83214 417.00 448.00 469.00 506.00 527.00 525.00 522.00 \n",
"388184 684.00 708.00 744.00 775.00 792.00 802.00 816.00 \n",
"411194 914.00 981.00 1063.00 1125.00 1137.00 1143.00 1133.00 \n",
"44865 329.00 342.00 359.00 372.00 375.00 377.00 376.00 \n",
"447353 3087.00 3154.00 3234.00 3294.00 3311.00 3294.00 3293.00 \n",
"163565 389.00 412.00 456.00 517.00 564.00 600.00 627.00 \n",
"576280 34.00 39.00 46.00 52.00 59.00 61.00 66.00 \n",
"\n",
" hour14 hour15 hour16 hour17 hour18 hour19 hour20 \\\n",
"44501 375.00 388.00 367.00 380.00 395.00 432.00 424.00 \n",
"446989 3056.00 3047.00 3043.00 3023.00 3048.00 3085.00 3220.00 \n",
"410830 1217.00 1216.00 1225.00 1303.00 1410.00 1404.00 1383.00 \n",
"387820 760.00 762.00 762.00 797.00 890.00 893.00 872.00 \n",
"359332 4504.00 4471.00 4471.00 4512.00 4724.00 5269.00 5337.00 \n",
"505792 94.00 96.00 95.00 96.00 94.00 98.00 110.00 \n",
"284094 1642.00 1613.00 1632.00 1735.00 1956.00 1965.00 1919.00 \n",
"238805 92.10 95.60 95.50 97.10 96.60 103.00 109.00 \n",
"327192 552.00 546.00 552.00 553.00 554.00 566.00 599.00 \n",
"146765 469.00 474.00 460.00 463.00 486.00 513.00 527.00 \n",
"91615 307.00 306.00 308.00 313.00 322.00 352.00 345.00 \n",
"82850 626.00 603.00 596.00 599.00 637.00 703.00 700.00 \n",
"239171 96.46 95.24 95.36 96.72 94.96 101.53 108.20 \n",
"157721 252.14 250.24 253.34 260.63 288.69 290.01 281.82 \n",
"284458 1752.00 1775.00 1801.00 1917.00 2066.00 2052.00 1994.00 \n",
"327556 612.00 634.00 642.00 654.00 647.00 655.00 676.00 \n",
"359696 4212.00 4160.00 4163.00 4260.00 4470.00 4952.00 4961.00 \n",
"369192 414.00 403.00 406.00 416.00 430.00 425.00 424.00 \n",
"83214 525.00 514.00 511.00 503.00 528.00 574.00 575.00 \n",
"388184 818.00 819.00 830.00 880.00 944.00 943.00 914.00 \n",
"411194 1130.00 1135.00 1141.00 1166.00 1268.00 1252.00 1230.00 \n",
"44865 385.00 369.00 372.00 377.00 392.00 420.00 408.00 \n",
"447353 3296.00 3299.00 3321.00 3327.00 3350.00 3368.00 3465.00 \n",
"163565 656.00 678.00 690.00 695.00 684.00 682.00 711.00 \n",
"576280 71.00 73.00 77.00 77.00 76.00 77.00 80.00 \n",
"\n",
" hour21 hour22 hour23 hour24 hour25 timezone_f hour01_f \\\n",
"44501 427.00 418.00 392.00 374.00 364.00 0 0 \n",
"446989 3218.00 3178.00 3120.00 3024.00 2920.00 0 0 \n",
"410830 1342.00 1267.00 1158.00 1072.00 981.00 0 0 \n",
"387820 838.00 791.00 730.00 683.00 645.00 0 0 \n",
"359332 5249.00 5070.00 4781.00 4415.00 4110.00 0 0 \n",
"505792 108.00 105.00 105.00 102.00 99.00 0 0 \n",
"284094 1850.00 1716.00 1586.00 1502.00 1446.00 0 0 \n",
"238805 106.90 101.70 93.60 83.70 76.40 0 0 \n",
"327192 592.00 571.00 535.00 503.00 468.00 0 0 \n",
"146765 520.00 504.00 474.00 438.00 408.00 0 0 \n",
"91615 338.00 329.00 309.00 293.00 257.00 0 0 \n",
"82850 696.00 677.00 629.00 585.00 547.00 0 0 \n",
"239171 106.60 102.74 97.12 87.61 79.48 0 0 \n",
"157721 270.71 251.55 231.91 213.71 193.13 0 0 \n",
"284458 1910.00 1781.00 1646.00 1558.00 1372.00 0 0 \n",
"327556 652.00 621.00 580.00 539.00 492.00 0 0 \n",
"359696 4826.00 4637.00 4338.00 3954.00 3760.00 0 0 \n",
"369192 415.00 411.00 401.00 391.00 397.00 0 0 \n",
"83214 563.00 547.00 503.00 466.00 425.00 0 0 \n",
"388184 880.00 827.00 766.00 713.00 641.00 0 0 \n",
"411194 1185.00 1127.00 1039.00 940.00 848.00 0 0 \n",
"44865 412.00 392.00 374.00 343.00 336.00 0 0 \n",
"447353 3438.00 3382.00 3316.00 3201.00 3067.00 0 0 \n",
"163565 683.00 643.00 587.00 523.00 460.00 0 0 \n",
"576280 75.00 69.00 62.00 52.00 44.00 0 0 \n",
"\n",
" hour02_f hour03_f hour04_f hour05_f hour06_f hour07_f hour08_f \\\n",
"44501 0 0 0 0 0 0 0 \n",
"446989 0 0 0 0 0 0 0 \n",
"410830 0 0 0 0 0 0 0 \n",
"387820 0 0 0 0 0 0 0 \n",
"359332 0 0 0 0 0 0 0 \n",
"505792 0 0 0 0 0 0 0 \n",
"284094 0 0 0 0 0 0 0 \n",
"238805 0 0 0 0 0 0 0 \n",
"327192 0 0 0 0 0 0 0 \n",
"146765 0 0 0 0 0 0 0 \n",
"91615 0 0 0 0 0 0 0 \n",
"82850 0 0 0 0 0 0 0 \n",
"239171 0 0 0 0 0 0 0 \n",
"157721 0 0 0 0 0 0 0 \n",
"284458 0 0 0 0 0 0 0 \n",
"327556 0 0 0 0 0 0 0 \n",
"359696 0 0 0 0 0 0 0 \n",
"369192 0 0 0 0 0 0 0 \n",
"83214 0 0 0 0 0 0 0 \n",
"388184 0 0 0 0 0 0 0 \n",
"411194 0 0 0 0 0 0 0 \n",
"44865 0 0 0 0 0 0 0 \n",
"447353 0 0 0 0 0 0 0 \n",
"163565 0 0 0 0 0 0 0 \n",
"576280 0 0 0 0 0 0 0 \n",
"\n",
" hour09_f hour10_f hour11_f hour12_f hour13_f hour14_f hour15_f \\\n",
"44501 0 0 0 0 0 0 0 \n",
"446989 0 0 0 0 0 0 0 \n",
"410830 0 0 0 0 0 0 0 \n",
"387820 0 0 0 0 0 0 0 \n",
"359332 0 0 0 0 0 0 0 \n",
"505792 0 0 0 0 0 0 0 \n",
"284094 0 0 0 0 0 0 0 \n",
"238805 0 0 0 0 0 0 0 \n",
"327192 0 0 0 0 0 0 0 \n",
"146765 0 0 0 0 0 0 0 \n",
"91615 0 0 0 0 0 0 0 \n",
"82850 0 0 0 0 0 0 0 \n",
"239171 0 0 0 0 0 0 0 \n",
"157721 0 0 0 0 0 0 0 \n",
"284458 0 0 0 0 0 0 0 \n",
"327556 0 0 0 0 0 0 0 \n",
"359696 0 0 0 0 0 0 0 \n",
"369192 0 0 0 0 0 0 0 \n",
"83214 0 0 0 0 0 0 0 \n",
"388184 0 0 0 0 0 0 0 \n",
"411194 0 0 0 0 0 0 0 \n",
"44865 0 0 0 0 0 0 0 \n",
"447353 0 0 0 0 0 0 0 \n",
"163565 0 0 0 0 0 0 0 \n",
"576280 0 0 0 0 0 0 0 \n",
"\n",
" hour16_f hour17_f hour18_f hour19_f hour20_f hour21_f hour22_f \\\n",
"44501 0 0 0 0 0 0 0 \n",
"446989 0 0 0 0 0 0 0 \n",
"410830 0 0 0 0 0 0 0 \n",
"387820 0 0 0 0 0 0 0 \n",
"359332 0 0 0 0 0 0 0 \n",
"505792 0 0 0 0 0 0 0 \n",
"284094 0 0 0 0 0 0 0 \n",
"238805 0 0 0 0 0 0 0 \n",
"327192 0 0 0 0 0 0 0 \n",
"146765 0 0 0 0 0 0 0 \n",
"91615 0 0 0 0 0 0 0 \n",
"82850 0 0 0 0 0 0 0 \n",
"239171 0 0 0 0 0 0 0 \n",
"157721 0 0 0 0 0 0 0 \n",
"284458 0 0 0 0 0 0 0 \n",
"327556 0 0 0 0 0 0 0 \n",
"359696 0 0 0 0 0 0 0 \n",
"369192 0 0 0 0 0 0 0 \n",
"83214 0 0 0 0 0 0 0 \n",
"388184 0 0 0 0 0 0 0 \n",
"411194 0 0 0 0 0 0 0 \n",
"44865 0 0 0 0 0 0 0 \n",
"447353 0 0 0 0 0 0 0 \n",
"163565 0 0 0 0 0 0 0 \n",
"576280 0 0 0 0 0 0 0 \n",
"\n",
" hour23_f hour24_f hour25_f hour_25_valid hour24_25_ratio \n",
"44501 0 0 0 True 1.027473 \n",
"446989 0 0 0 True 1.035616 \n",
"410830 0 0 0 True 1.092762 \n",
"387820 0 0 0 True 1.058915 \n",
"359332 0 0 0 True 1.074209 \n",
"505792 0 0 0 True 1.030303 \n",
"284094 0 0 0 True 1.038728 \n",
"238805 0 0 0 True 1.095550 \n",
"327192 0 0 0 True 1.074786 \n",
"146765 0 0 0 True 1.073529 \n",
"91615 0 2 0 True 1.140078 \n",
"82850 0 0 0 True 1.069470 \n",
"239171 0 0 0 True 1.102290 \n",
"157721 0 0 0 True 1.106560 \n",
"284458 0 0 0 True 1.135569 \n",
"327556 0 0 0 True 1.095528 \n",
"359696 0 0 0 True 1.051596 \n",
"369192 0 0 0 True 0.984887 \n",
"83214 0 0 0 True 1.096471 \n",
"388184 0 0 0 True 1.112324 \n",
"411194 0 0 0 True 1.108491 \n",
"44865 0 0 0 True 1.020833 \n",
"447353 0 0 0 True 1.043691 \n",
"163565 0 0 0 True 1.136957 \n",
"576280 0 0 0 True 1.181818 "
]
},
"metadata": {
"tags": []
},
"execution_count": 10
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "UO76JcZSg_EA",
"colab_type": "code",
"outputId": "78d6bc44-7c9b-4d7e-ba33-dfb83b8c80eb",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"# Same for a 0 value in hour02, which happens sometimes instead of skipping the hour\n",
"corrected_df.query(\"hour02 == 0 & hour01 != 0\").sort_values(\"report_yr\")"
],
"execution_count": 11,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_id</th>\n",
" <th>report_yr</th>\n",
" <th>report_prd</th>\n",
" <th>spplmnt_num</th>\n",
" <th>row_num</th>\n",
" <th>plan_date</th>\n",
" <th>timezone</th>\n",
" <th>hour01</th>\n",
" <th>hour02</th>\n",
" <th>hour03</th>\n",
" <th>hour04</th>\n",
" <th>hour05</th>\n",
" <th>hour06</th>\n",
" <th>hour07</th>\n",
" <th>hour08</th>\n",
" <th>hour09</th>\n",
" <th>hour10</th>\n",
" <th>hour11</th>\n",
" <th>hour12</th>\n",
" <th>hour13</th>\n",
" <th>hour14</th>\n",
" <th>hour15</th>\n",
" <th>hour16</th>\n",
" <th>hour17</th>\n",
" <th>hour18</th>\n",
" <th>hour19</th>\n",
" <th>hour20</th>\n",
" <th>hour21</th>\n",
" <th>hour22</th>\n",
" <th>hour23</th>\n",
" <th>hour24</th>\n",
" <th>hour25</th>\n",
" <th>timezone_f</th>\n",
" <th>hour01_f</th>\n",
" <th>hour02_f</th>\n",
" <th>hour03_f</th>\n",
" <th>hour04_f</th>\n",
" <th>hour05_f</th>\n",
" <th>hour06_f</th>\n",
" <th>hour07_f</th>\n",
" <th>hour08_f</th>\n",
" <th>hour09_f</th>\n",
" <th>hour10_f</th>\n",
" <th>hour11_f</th>\n",
" <th>hour12_f</th>\n",
" <th>hour13_f</th>\n",
" <th>hour14_f</th>\n",
" <th>hour15_f</th>\n",
" <th>hour16_f</th>\n",
" <th>hour17_f</th>\n",
" <th>hour18_f</th>\n",
" <th>hour19_f</th>\n",
" <th>hour20_f</th>\n",
" <th>hour21_f</th>\n",
" <th>hour22_f</th>\n",
" <th>hour23_f</th>\n",
" <th>hour24_f</th>\n",
" <th>hour25_f</th>\n",
" <th>hour_25_valid</th>\n",
" <th>hour24_25_ratio</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>63985</th>\n",
" <td>128</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>EDT</td>\n",
" <td>1300.00</td>\n",
" <td>0.0</td>\n",
" <td>1275.00</td>\n",
" <td>1283.00</td>\n",
" <td>1317.00</td>\n",
" <td>1383.00</td>\n",
" <td>1488.00</td>\n",
" <td>1641.00</td>\n",
" <td>1793.00</td>\n",
" <td>1750.00</td>\n",
" <td>1590.00</td>\n",
" <td>1472.00</td>\n",
" <td>1431.00</td>\n",
" <td>1403.00</td>\n",
" <td>1371.0</td>\n",
" <td>1363.00</td>\n",
" <td>1380.00</td>\n",
" <td>1416.00</td>\n",
" <td>1444.00</td>\n",
" <td>1551.00</td>\n",
" <td>1694.00</td>\n",
" <td>1612.00</td>\n",
" <td>1449.00</td>\n",
" <td>1272.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>545729</th>\n",
" <td>292</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td></td>\n",
" <td>31.00</td>\n",
" <td>0.0</td>\n",
" <td>27.00</td>\n",
" <td>27.00</td>\n",
" <td>28.00</td>\n",
" <td>29.00</td>\n",
" <td>28.00</td>\n",
" <td>28.00</td>\n",
" <td>28.00</td>\n",
" <td>33.00</td>\n",
" <td>31.00</td>\n",
" <td>31.00</td>\n",
" <td>30.00</td>\n",
" <td>31.00</td>\n",
" <td>32.0</td>\n",
" <td>31.00</td>\n",
" <td>31.00</td>\n",
" <td>32.00</td>\n",
" <td>31.00</td>\n",
" <td>37.00</td>\n",
" <td>39.00</td>\n",
" <td>35.00</td>\n",
" <td>34.00</td>\n",
" <td>32.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>543538</th>\n",
" <td>289</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td></td>\n",
" <td>82.00</td>\n",
" <td>0.0</td>\n",
" <td>82.00</td>\n",
" <td>75.00</td>\n",
" <td>80.00</td>\n",
" <td>76.00</td>\n",
" <td>81.00</td>\n",
" <td>80.00</td>\n",
" <td>84.00</td>\n",
" <td>87.00</td>\n",
" <td>91.00</td>\n",
" <td>91.00</td>\n",
" <td>93.00</td>\n",
" <td>97.00</td>\n",
" <td>96.0</td>\n",
" <td>93.00</td>\n",
" <td>95.00</td>\n",
" <td>98.00</td>\n",
" <td>99.00</td>\n",
" <td>108.00</td>\n",
" <td>105.00</td>\n",
" <td>105.00</td>\n",
" <td>98.00</td>\n",
" <td>89.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>525643</th>\n",
" <td>283</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CDT</td>\n",
" <td>132.00</td>\n",
" <td>0.0</td>\n",
" <td>127.00</td>\n",
" <td>125.00</td>\n",
" <td>124.00</td>\n",
" <td>126.00</td>\n",
" <td>129.00</td>\n",
" <td>131.00</td>\n",
" <td>136.00</td>\n",
" <td>142.00</td>\n",
" <td>147.00</td>\n",
" <td>146.00</td>\n",
" <td>149.00</td>\n",
" <td>148.00</td>\n",
" <td>145.0</td>\n",
" <td>144.00</td>\n",
" <td>146.00</td>\n",
" <td>148.00</td>\n",
" <td>150.00</td>\n",
" <td>162.00</td>\n",
" <td>169.00</td>\n",
" <td>162.00</td>\n",
" <td>152.00</td>\n",
" <td>141.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>505554</th>\n",
" <td>275</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CPT</td>\n",
" <td>99.00</td>\n",
" <td>0.0</td>\n",
" <td>97.00</td>\n",
" <td>94.00</td>\n",
" <td>94.00</td>\n",
" <td>93.00</td>\n",
" <td>106.00</td>\n",
" <td>109.00</td>\n",
" <td>105.00</td>\n",
" <td>104.00</td>\n",
" <td>111.00</td>\n",
" <td>106.00</td>\n",
" <td>107.00</td>\n",
" <td>99.00</td>\n",
" <td>94.0</td>\n",
" <td>91.00</td>\n",
" <td>93.00</td>\n",
" <td>92.00</td>\n",
" <td>101.00</td>\n",
" <td>101.00</td>\n",
" <td>105.00</td>\n",
" <td>112.00</td>\n",
" <td>106.00</td>\n",
" <td>95.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>485101</th>\n",
" <td>268</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>EST</td>\n",
" <td>526.00</td>\n",
" <td>0.0</td>\n",
" <td>496.00</td>\n",
" <td>478.00</td>\n",
" <td>471.00</td>\n",
" <td>475.00</td>\n",
" <td>491.00</td>\n",
" <td>499.00</td>\n",
" <td>530.00</td>\n",
" <td>569.00</td>\n",
" <td>594.00</td>\n",
" <td>604.00</td>\n",
" <td>607.00</td>\n",
" <td>604.00</td>\n",
" <td>596.0</td>\n",
" <td>593.00</td>\n",
" <td>596.00</td>\n",
" <td>608.00</td>\n",
" <td>639.00</td>\n",
" <td>690.00</td>\n",
" <td>688.00</td>\n",
" <td>660.00</td>\n",
" <td>612.00</td>\n",
" <td>572.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>449879</th>\n",
" <td>260</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>27800</td>\n",
" <td>2011-10-05</td>\n",
" <td>CST</td>\n",
" <td>51.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>449761</th>\n",
" <td>260</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>16000</td>\n",
" <td>2011-06-09</td>\n",
" <td>CST</td>\n",
" <td>50.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>446751</th>\n",
" <td>259</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CDT</td>\n",
" <td>3096.00</td>\n",
" <td>0.0</td>\n",
" <td>3079.00</td>\n",
" <td>3065.00</td>\n",
" <td>3063.00</td>\n",
" <td>3104.00</td>\n",
" <td>3152.00</td>\n",
" <td>3234.00</td>\n",
" <td>3258.00</td>\n",
" <td>3284.00</td>\n",
" <td>3290.00</td>\n",
" <td>3263.00</td>\n",
" <td>3250.00</td>\n",
" <td>3221.00</td>\n",
" <td>3227.0</td>\n",
" <td>3200.00</td>\n",
" <td>3193.00</td>\n",
" <td>3191.00</td>\n",
" <td>3192.00</td>\n",
" <td>3194.00</td>\n",
" <td>3313.00</td>\n",
" <td>3323.00</td>\n",
" <td>3286.00</td>\n",
" <td>3177.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>392330</th>\n",
" <td>243</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>PST</td>\n",
" <td>962.00</td>\n",
" <td>0.0</td>\n",
" <td>915.00</td>\n",
" <td>899.00</td>\n",
" <td>891.00</td>\n",
" <td>913.00</td>\n",
" <td>945.00</td>\n",
" <td>988.00</td>\n",
" <td>1027.00</td>\n",
" <td>1102.00</td>\n",
" <td>1151.00</td>\n",
" <td>1167.00</td>\n",
" <td>1180.00</td>\n",
" <td>1177.00</td>\n",
" <td>1154.0</td>\n",
" <td>1154.00</td>\n",
" <td>1169.00</td>\n",
" <td>1231.00</td>\n",
" <td>1280.00</td>\n",
" <td>1342.00</td>\n",
" <td>1346.00</td>\n",
" <td>1282.00</td>\n",
" <td>1176.00</td>\n",
" <td>1070.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>387582</th>\n",
" <td>241</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>EDT</td>\n",
" <td>700.00</td>\n",
" <td>0.0</td>\n",
" <td>687.00</td>\n",
" <td>658.00</td>\n",
" <td>658.00</td>\n",
" <td>660.00</td>\n",
" <td>686.00</td>\n",
" <td>714.00</td>\n",
" <td>743.00</td>\n",
" <td>784.00</td>\n",
" <td>806.00</td>\n",
" <td>831.00</td>\n",
" <td>846.00</td>\n",
" <td>848.00</td>\n",
" <td>846.0</td>\n",
" <td>842.00</td>\n",
" <td>853.00</td>\n",
" <td>871.00</td>\n",
" <td>888.00</td>\n",
" <td>926.00</td>\n",
" <td>921.00</td>\n",
" <td>887.00</td>\n",
" <td>828.00</td>\n",
" <td>765.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>322206</th>\n",
" <td>223</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CDT</td>\n",
" <td>1089.00</td>\n",
" <td>0.0</td>\n",
" <td>1074.00</td>\n",
" <td>1072.00</td>\n",
" <td>1079.00</td>\n",
" <td>1080.00</td>\n",
" <td>1119.00</td>\n",
" <td>1153.00</td>\n",
" <td>1171.00</td>\n",
" <td>1194.00</td>\n",
" <td>1231.00</td>\n",
" <td>1239.00</td>\n",
" <td>1261.00</td>\n",
" <td>1253.00</td>\n",
" <td>1238.0</td>\n",
" <td>1212.00</td>\n",
" <td>1208.00</td>\n",
" <td>1225.00</td>\n",
" <td>1245.00</td>\n",
" <td>1302.00</td>\n",
" <td>1329.00</td>\n",
" <td>1294.00</td>\n",
" <td>1217.00</td>\n",
" <td>1137.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>575678</th>\n",
" <td>308</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>EDS</td>\n",
" <td>43.00</td>\n",
" <td>0.0</td>\n",
" <td>39.00</td>\n",
" <td>39.00</td>\n",
" <td>41.00</td>\n",
" <td>42.00</td>\n",
" <td>46.00</td>\n",
" <td>51.00</td>\n",
" <td>55.00</td>\n",
" <td>57.00</td>\n",
" <td>57.00</td>\n",
" <td>58.00</td>\n",
" <td>56.00</td>\n",
" <td>58.00</td>\n",
" <td>55.0</td>\n",
" <td>56.00</td>\n",
" <td>58.00</td>\n",
" <td>61.00</td>\n",
" <td>61.00</td>\n",
" <td>64.00</td>\n",
" <td>67.00</td>\n",
" <td>64.00</td>\n",
" <td>54.00</td>\n",
" <td>45.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>283856</th>\n",
" <td>212</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>EDT</td>\n",
" <td>1510.00</td>\n",
" <td>0.0</td>\n",
" <td>1463.00</td>\n",
" <td>1444.00</td>\n",
" <td>1431.00</td>\n",
" <td>1450.00</td>\n",
" <td>1501.00</td>\n",
" <td>1574.00</td>\n",
" <td>1661.00</td>\n",
" <td>1735.00</td>\n",
" <td>1805.00</td>\n",
" <td>1837.00</td>\n",
" <td>1854.00</td>\n",
" <td>1848.00</td>\n",
" <td>1824.0</td>\n",
" <td>1824.00</td>\n",
" <td>1844.00</td>\n",
" <td>1862.00</td>\n",
" <td>1928.00</td>\n",
" <td>2036.00</td>\n",
" <td>2011.00</td>\n",
" <td>1899.00</td>\n",
" <td>1765.00</td>\n",
" <td>1639.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91377</th>\n",
" <td>141</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CST</td>\n",
" <td>257.00</td>\n",
" <td>0.0</td>\n",
" <td>252.00</td>\n",
" <td>249.00</td>\n",
" <td>250.00</td>\n",
" <td>255.00</td>\n",
" <td>262.00</td>\n",
" <td>274.00</td>\n",
" <td>277.00</td>\n",
" <td>287.00</td>\n",
" <td>289.00</td>\n",
" <td>299.00</td>\n",
" <td>300.00</td>\n",
" <td>303.00</td>\n",
" <td>301.0</td>\n",
" <td>311.00</td>\n",
" <td>325.00</td>\n",
" <td>327.00</td>\n",
" <td>340.00</td>\n",
" <td>351.00</td>\n",
" <td>353.00</td>\n",
" <td>343.00</td>\n",
" <td>324.00</td>\n",
" <td>302.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>121691</th>\n",
" <td>156</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td></td>\n",
" <td>71.00</td>\n",
" <td>0.0</td>\n",
" <td>66.00</td>\n",
" <td>66.00</td>\n",
" <td>68.00</td>\n",
" <td>64.00</td>\n",
" <td>70.00</td>\n",
" <td>69.00</td>\n",
" <td>74.00</td>\n",
" <td>79.00</td>\n",
" <td>81.00</td>\n",
" <td>83.00</td>\n",
" <td>86.00</td>\n",
" <td>83.00</td>\n",
" <td>86.0</td>\n",
" <td>86.00</td>\n",
" <td>83.00</td>\n",
" <td>85.00</td>\n",
" <td>85.00</td>\n",
" <td>98.00</td>\n",
" <td>93.00</td>\n",
" <td>90.00</td>\n",
" <td>89.00</td>\n",
" <td>80.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>137203</th>\n",
" <td>161</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>24400</td>\n",
" <td>2011-09-01</td>\n",
" <td>CPT</td>\n",
" <td>300.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.0</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>238567</th>\n",
" <td>195</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CDT</td>\n",
" <td>69.30</td>\n",
" <td>0.0</td>\n",
" <td>66.50</td>\n",
" <td>65.00</td>\n",
" <td>63.70</td>\n",
" <td>65.60</td>\n",
" <td>70.00</td>\n",
" <td>69.70</td>\n",
" <td>73.90</td>\n",
" <td>77.10</td>\n",
" <td>79.90</td>\n",
" <td>82.50</td>\n",
" <td>83.00</td>\n",
" <td>86.20</td>\n",
" <td>86.6</td>\n",
" <td>88.30</td>\n",
" <td>89.80</td>\n",
" <td>88.60</td>\n",
" <td>90.30</td>\n",
" <td>97.70</td>\n",
" <td>102.00</td>\n",
" <td>97.20</td>\n",
" <td>88.70</td>\n",
" <td>79.20</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>146527</th>\n",
" <td>163</td>\n",
" <td>2011</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7200</td>\n",
" <td>2011-03-13</td>\n",
" <td>CPT</td>\n",
" <td>395.00</td>\n",
" <td>0.0</td>\n",
" <td>390.00</td>\n",
" <td>393.00</td>\n",
" <td>397.00</td>\n",
" <td>411.00</td>\n",
" <td>431.00</td>\n",
" <td>475.00</td>\n",
" <td>503.00</td>\n",
" <td>520.00</td>\n",
" <td>514.00</td>\n",
" <td>511.00</td>\n",
" <td>510.00</td>\n",
" <td>502.00</td>\n",
" <td>469.0</td>\n",
" <td>520.00</td>\n",
" <td>521.00</td>\n",
" <td>556.00</td>\n",
" <td>564.00</td>\n",
" <td>628.00</td>\n",
" <td>593.00</td>\n",
" <td>563.00</td>\n",
" <td>544.00</td>\n",
" <td>507.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>219939</th>\n",
" <td>187</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>192.00</td>\n",
" <td>0.0</td>\n",
" <td>190.00</td>\n",
" <td>188.00</td>\n",
" <td>185.00</td>\n",
" <td>187.00</td>\n",
" <td>189.00</td>\n",
" <td>198.00</td>\n",
" <td>197.00</td>\n",
" <td>205.00</td>\n",
" <td>206.00</td>\n",
" <td>213.00</td>\n",
" <td>214.00</td>\n",
" <td>215.00</td>\n",
" <td>222.0</td>\n",
" <td>216.00</td>\n",
" <td>220.00</td>\n",
" <td>224.00</td>\n",
" <td>228.00</td>\n",
" <td>249.00</td>\n",
" <td>250.00</td>\n",
" <td>244.00</td>\n",
" <td>243.00</td>\n",
" <td>231.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>447115</th>\n",
" <td>259</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>3258.00</td>\n",
" <td>0.0</td>\n",
" <td>3210.00</td>\n",
" <td>3198.00</td>\n",
" <td>3201.00</td>\n",
" <td>3225.00</td>\n",
" <td>3283.00</td>\n",
" <td>3352.00</td>\n",
" <td>3379.00</td>\n",
" <td>3380.00</td>\n",
" <td>3357.00</td>\n",
" <td>3312.00</td>\n",
" <td>3257.00</td>\n",
" <td>3206.00</td>\n",
" <td>3134.0</td>\n",
" <td>3092.00</td>\n",
" <td>3053.00</td>\n",
" <td>3040.00</td>\n",
" <td>3051.00</td>\n",
" <td>3079.00</td>\n",
" <td>3193.00</td>\n",
" <td>3208.00</td>\n",
" <td>3148.00</td>\n",
" <td>3043.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>163327</th>\n",
" <td>169</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>EDT</td>\n",
" <td>469.00</td>\n",
" <td>0.0</td>\n",
" <td>435.00</td>\n",
" <td>415.00</td>\n",
" <td>406.00</td>\n",
" <td>410.00</td>\n",
" <td>428.00</td>\n",
" <td>451.00</td>\n",
" <td>483.00</td>\n",
" <td>538.00</td>\n",
" <td>585.00</td>\n",
" <td>614.00</td>\n",
" <td>630.00</td>\n",
" <td>632.00</td>\n",
" <td>631.0</td>\n",
" <td>626.00</td>\n",
" <td>622.00</td>\n",
" <td>624.00</td>\n",
" <td>627.00</td>\n",
" <td>646.00</td>\n",
" <td>659.00</td>\n",
" <td>625.00</td>\n",
" <td>574.00</td>\n",
" <td>512.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>284220</th>\n",
" <td>212</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>EST</td>\n",
" <td>1560.00</td>\n",
" <td>0.0</td>\n",
" <td>1506.00</td>\n",
" <td>1485.00</td>\n",
" <td>1464.00</td>\n",
" <td>1484.00</td>\n",
" <td>1533.00</td>\n",
" <td>1604.00</td>\n",
" <td>1649.00</td>\n",
" <td>1710.00</td>\n",
" <td>1727.00</td>\n",
" <td>1711.00</td>\n",
" <td>1680.00</td>\n",
" <td>1628.00</td>\n",
" <td>1591.0</td>\n",
" <td>1561.00</td>\n",
" <td>1531.00</td>\n",
" <td>1584.00</td>\n",
" <td>1643.00</td>\n",
" <td>1820.00</td>\n",
" <td>1846.00</td>\n",
" <td>1736.00</td>\n",
" <td>1602.00</td>\n",
" <td>1452.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>238931</th>\n",
" <td>195</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>72.69</td>\n",
" <td>0.0</td>\n",
" <td>68.98</td>\n",
" <td>67.44</td>\n",
" <td>67.21</td>\n",
" <td>67.54</td>\n",
" <td>71.01</td>\n",
" <td>72.34</td>\n",
" <td>74.31</td>\n",
" <td>80.62</td>\n",
" <td>85.86</td>\n",
" <td>89.01</td>\n",
" <td>91.63</td>\n",
" <td>94.28</td>\n",
" <td>94.7</td>\n",
" <td>95.77</td>\n",
" <td>96.37</td>\n",
" <td>97.43</td>\n",
" <td>99.79</td>\n",
" <td>107.57</td>\n",
" <td>109.76</td>\n",
" <td>106.17</td>\n",
" <td>97.21</td>\n",
" <td>88.15</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>122055</th>\n",
" <td>156</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>76.00</td>\n",
" <td>0.0</td>\n",
" <td>70.00</td>\n",
" <td>70.00</td>\n",
" <td>69.00</td>\n",
" <td>71.00</td>\n",
" <td>71.00</td>\n",
" <td>77.00</td>\n",
" <td>76.00</td>\n",
" <td>81.00</td>\n",
" <td>84.00</td>\n",
" <td>84.00</td>\n",
" <td>87.00</td>\n",
" <td>90.00</td>\n",
" <td>87.0</td>\n",
" <td>88.00</td>\n",
" <td>93.00</td>\n",
" <td>92.00</td>\n",
" <td>93.00</td>\n",
" <td>99.00</td>\n",
" <td>96.00</td>\n",
" <td>93.00</td>\n",
" <td>90.00</td>\n",
" <td>77.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>546093</th>\n",
" <td>292</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>30.00</td>\n",
" <td>0.0</td>\n",
" <td>29.00</td>\n",
" <td>29.00</td>\n",
" <td>31.00</td>\n",
" <td>28.00</td>\n",
" <td>29.00</td>\n",
" <td>32.00</td>\n",
" <td>30.00</td>\n",
" <td>34.00</td>\n",
" <td>33.00</td>\n",
" <td>32.00</td>\n",
" <td>32.00</td>\n",
" <td>33.00</td>\n",
" <td>32.0</td>\n",
" <td>30.00</td>\n",
" <td>33.00</td>\n",
" <td>33.00</td>\n",
" <td>33.00</td>\n",
" <td>38.00</td>\n",
" <td>38.00</td>\n",
" <td>37.00</td>\n",
" <td>35.00</td>\n",
" <td>29.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>526007</th>\n",
" <td>283</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CDT</td>\n",
" <td>122.00</td>\n",
" <td>0.0</td>\n",
" <td>117.00</td>\n",
" <td>114.00</td>\n",
" <td>114.00</td>\n",
" <td>116.00</td>\n",
" <td>120.00</td>\n",
" <td>121.00</td>\n",
" <td>125.00</td>\n",
" <td>131.00</td>\n",
" <td>136.00</td>\n",
" <td>139.00</td>\n",
" <td>142.00</td>\n",
" <td>139.00</td>\n",
" <td>137.0</td>\n",
" <td>136.00</td>\n",
" <td>137.00</td>\n",
" <td>141.00</td>\n",
" <td>146.00</td>\n",
" <td>156.00</td>\n",
" <td>158.00</td>\n",
" <td>151.00</td>\n",
" <td>142.00</td>\n",
" <td>131.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>322570</th>\n",
" <td>223</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CDT</td>\n",
" <td>889.00</td>\n",
" <td>0.0</td>\n",
" <td>911.00</td>\n",
" <td>911.00</td>\n",
" <td>810.00</td>\n",
" <td>829.00</td>\n",
" <td>892.00</td>\n",
" <td>924.00</td>\n",
" <td>952.00</td>\n",
" <td>983.00</td>\n",
" <td>1015.00</td>\n",
" <td>1038.00</td>\n",
" <td>1059.00</td>\n",
" <td>1069.00</td>\n",
" <td>1071.0</td>\n",
" <td>1066.00</td>\n",
" <td>1093.00</td>\n",
" <td>1130.00</td>\n",
" <td>1139.00</td>\n",
" <td>1167.00</td>\n",
" <td>1162.00</td>\n",
" <td>1121.00</td>\n",
" <td>1045.00</td>\n",
" <td>968.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>543902</th>\n",
" <td>289</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>CST</td>\n",
" <td>87.00</td>\n",
" <td>0.0</td>\n",
" <td>84.00</td>\n",
" <td>81.00</td>\n",
" <td>83.00</td>\n",
" <td>82.00</td>\n",
" <td>85.00</td>\n",
" <td>86.00</td>\n",
" <td>92.00</td>\n",
" <td>96.00</td>\n",
" <td>96.00</td>\n",
" <td>106.00</td>\n",
" <td>101.00</td>\n",
" <td>106.00</td>\n",
" <td>104.0</td>\n",
" <td>104.00</td>\n",
" <td>107.00</td>\n",
" <td>104.00</td>\n",
" <td>108.00</td>\n",
" <td>113.00</td>\n",
" <td>109.00</td>\n",
" <td>111.00</td>\n",
" <td>98.00</td>\n",
" <td>97.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64349</th>\n",
" <td>128</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>EDT</td>\n",
" <td>1534.00</td>\n",
" <td>0.0</td>\n",
" <td>1524.00</td>\n",
" <td>1545.00</td>\n",
" <td>1592.00</td>\n",
" <td>1669.00</td>\n",
" <td>1785.00</td>\n",
" <td>1944.00</td>\n",
" <td>2080.00</td>\n",
" <td>1996.00</td>\n",
" <td>1791.00</td>\n",
" <td>1625.00</td>\n",
" <td>1541.00</td>\n",
" <td>1474.00</td>\n",
" <td>1399.0</td>\n",
" <td>1353.00</td>\n",
" <td>1353.00</td>\n",
" <td>1381.00</td>\n",
" <td>1417.00</td>\n",
" <td>1557.00</td>\n",
" <td>1717.00</td>\n",
" <td>1647.00</td>\n",
" <td>1525.00</td>\n",
" <td>1367.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>387946</th>\n",
" <td>241</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>EST</td>\n",
" <td>697.00</td>\n",
" <td>0.0</td>\n",
" <td>676.00</td>\n",
" <td>663.00</td>\n",
" <td>649.00</td>\n",
" <td>662.00</td>\n",
" <td>679.00</td>\n",
" <td>691.00</td>\n",
" <td>715.00</td>\n",
" <td>734.00</td>\n",
" <td>747.00</td>\n",
" <td>755.00</td>\n",
" <td>752.00</td>\n",
" <td>750.00</td>\n",
" <td>738.0</td>\n",
" <td>732.00</td>\n",
" <td>733.00</td>\n",
" <td>741.00</td>\n",
" <td>751.00</td>\n",
" <td>822.00</td>\n",
" <td>836.00</td>\n",
" <td>804.00</td>\n",
" <td>746.00</td>\n",
" <td>688.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" <tr>\n",
" <th>354710</th>\n",
" <td>234</td>\n",
" <td>2012</td>\n",
" <td>12</td>\n",
" <td>0</td>\n",
" <td>7100</td>\n",
" <td>2012-03-11</td>\n",
" <td>EST</td>\n",
" <td>2940.00</td>\n",
" <td>0.0</td>\n",
" <td>2787.00</td>\n",
" <td>2628.00</td>\n",
" <td>2571.00</td>\n",
" <td>2589.00</td>\n",
" <td>2687.00</td>\n",
" <td>2822.00</td>\n",
" <td>3079.00</td>\n",
" <td>3430.00</td>\n",
" <td>3764.00</td>\n",
" <td>3983.00</td>\n",
" <td>4109.00</td>\n",
" <td>4196.00</td>\n",
" <td>4215.0</td>\n",
" <td>4243.00</td>\n",
" <td>4254.00</td>\n",
" <td>4242.00</td>\n",
" <td>4206.00</td>\n",
" <td>4297.00</td>\n",
" <td>4352.00</td>\n",
" <td>4113.00</td>\n",
" <td>3766.00</td>\n",
" <td>3315.00</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>inf</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_id report_yr report_prd spplmnt_num row_num plan_date \\\n",
"63985 128 2011 12 0 7200 2011-03-13 \n",
"545729 292 2011 12 0 7200 2011-03-13 \n",
"543538 289 2011 12 0 7200 2011-03-13 \n",
"525643 283 2011 12 0 7200 2011-03-13 \n",
"505554 275 2011 12 0 7200 2011-03-13 \n",
"485101 268 2011 12 0 7200 2011-03-13 \n",
"449879 260 2011 12 0 27800 2011-10-05 \n",
"449761 260 2011 12 0 16000 2011-06-09 \n",
"446751 259 2011 12 0 7200 2011-03-13 \n",
"392330 243 2011 12 0 7200 2011-03-13 \n",
"387582 241 2011 12 0 7200 2011-03-13 \n",
"322206 223 2011 12 0 7200 2011-03-13 \n",
"575678 308 2011 12 0 7200 2011-03-13 \n",
"283856 212 2011 12 0 7200 2011-03-13 \n",
"91377 141 2011 12 0 7200 2011-03-13 \n",
"121691 156 2011 12 0 7200 2011-03-13 \n",
"137203 161 2011 12 0 24400 2011-09-01 \n",
"238567 195 2011 12 0 7200 2011-03-13 \n",
"146527 163 2011 12 0 7200 2011-03-13 \n",
"219939 187 2012 12 0 7100 2012-03-11 \n",
"447115 259 2012 12 0 7100 2012-03-11 \n",
"163327 169 2012 12 0 7100 2012-03-11 \n",
"284220 212 2012 12 0 7100 2012-03-11 \n",
"238931 195 2012 12 0 7100 2012-03-11 \n",
"122055 156 2012 12 0 7100 2012-03-11 \n",
"546093 292 2012 12 0 7100 2012-03-11 \n",
"526007 283 2012 12 0 7100 2012-03-11 \n",
"322570 223 2012 12 0 7100 2012-03-11 \n",
"543902 289 2012 12 0 7100 2012-03-11 \n",
"64349 128 2012 12 0 7100 2012-03-11 \n",
"387946 241 2012 12 0 7100 2012-03-11 \n",
"354710 234 2012 12 0 7100 2012-03-11 \n",
"\n",
" timezone hour01 hour02 hour03 hour04 hour05 hour06 hour07 \\\n",
"63985 EDT 1300.00 0.0 1275.00 1283.00 1317.00 1383.00 1488.00 \n",
"545729 31.00 0.0 27.00 27.00 28.00 29.00 28.00 \n",
"543538 82.00 0.0 82.00 75.00 80.00 76.00 81.00 \n",
"525643 CDT 132.00 0.0 127.00 125.00 124.00 126.00 129.00 \n",
"505554 CPT 99.00 0.0 97.00 94.00 94.00 93.00 106.00 \n",
"485101 EST 526.00 0.0 496.00 478.00 471.00 475.00 491.00 \n",
"449879 CST 51.00 0.0 0.00 0.00 0.00 0.00 0.00 \n",
"449761 CST 50.00 0.0 0.00 0.00 0.00 0.00 0.00 \n",
"446751 CDT 3096.00 0.0 3079.00 3065.00 3063.00 3104.00 3152.00 \n",
"392330 PST 962.00 0.0 915.00 899.00 891.00 913.00 945.00 \n",
"387582 EDT 700.00 0.0 687.00 658.00 658.00 660.00 686.00 \n",
"322206 CDT 1089.00 0.0 1074.00 1072.00 1079.00 1080.00 1119.00 \n",
"575678 EDS 43.00 0.0 39.00 39.00 41.00 42.00 46.00 \n",
"283856 EDT 1510.00 0.0 1463.00 1444.00 1431.00 1450.00 1501.00 \n",
"91377 CST 257.00 0.0 252.00 249.00 250.00 255.00 262.00 \n",
"121691 71.00 0.0 66.00 66.00 68.00 64.00 70.00 \n",
"137203 CPT 300.00 0.0 0.00 0.00 0.00 0.00 0.00 \n",
"238567 CDT 69.30 0.0 66.50 65.00 63.70 65.60 70.00 \n",
"146527 CPT 395.00 0.0 390.00 393.00 397.00 411.00 431.00 \n",
"219939 CST 192.00 0.0 190.00 188.00 185.00 187.00 189.00 \n",
"447115 CST 3258.00 0.0 3210.00 3198.00 3201.00 3225.00 3283.00 \n",
"163327 EDT 469.00 0.0 435.00 415.00 406.00 410.00 428.00 \n",
"284220 EST 1560.00 0.0 1506.00 1485.00 1464.00 1484.00 1533.00 \n",
"238931 CST 72.69 0.0 68.98 67.44 67.21 67.54 71.01 \n",
"122055 CST 76.00 0.0 70.00 70.00 69.00 71.00 71.00 \n",
"546093 CST 30.00 0.0 29.00 29.00 31.00 28.00 29.00 \n",
"526007 CDT 122.00 0.0 117.00 114.00 114.00 116.00 120.00 \n",
"322570 CDT 889.00 0.0 911.00 911.00 810.00 829.00 892.00 \n",
"543902 CST 87.00 0.0 84.00 81.00 83.00 82.00 85.00 \n",
"64349 EDT 1534.00 0.0 1524.00 1545.00 1592.00 1669.00 1785.00 \n",
"387946 EST 697.00 0.0 676.00 663.00 649.00 662.00 679.00 \n",
"354710 EST 2940.00 0.0 2787.00 2628.00 2571.00 2589.00 2687.00 \n",
"\n",
" hour08 hour09 hour10 hour11 hour12 hour13 hour14 hour15 \\\n",
"63985 1641.00 1793.00 1750.00 1590.00 1472.00 1431.00 1403.00 1371.0 \n",
"545729 28.00 28.00 33.00 31.00 31.00 30.00 31.00 32.0 \n",
"543538 80.00 84.00 87.00 91.00 91.00 93.00 97.00 96.0 \n",
"525643 131.00 136.00 142.00 147.00 146.00 149.00 148.00 145.0 \n",
"505554 109.00 105.00 104.00 111.00 106.00 107.00 99.00 94.0 \n",
"485101 499.00 530.00 569.00 594.00 604.00 607.00 604.00 596.0 \n",
"449879 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 \n",
"449761 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 \n",
"446751 3234.00 3258.00 3284.00 3290.00 3263.00 3250.00 3221.00 3227.0 \n",
"392330 988.00 1027.00 1102.00 1151.00 1167.00 1180.00 1177.00 1154.0 \n",
"387582 714.00 743.00 784.00 806.00 831.00 846.00 848.00 846.0 \n",
"322206 1153.00 1171.00 1194.00 1231.00 1239.00 1261.00 1253.00 1238.0 \n",
"575678 51.00 55.00 57.00 57.00 58.00 56.00 58.00 55.0 \n",
"283856 1574.00 1661.00 1735.00 1805.00 1837.00 1854.00 1848.00 1824.0 \n",
"91377 274.00 277.00 287.00 289.00 299.00 300.00 303.00 301.0 \n",
"121691 69.00 74.00 79.00 81.00 83.00 86.00 83.00 86.0 \n",
"137203 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 \n",
"238567 69.70 73.90 77.10 79.90 82.50 83.00 86.20 86.6 \n",
"146527 475.00 503.00 520.00 514.00 511.00 510.00 502.00 469.0 \n",
"219939 198.00 197.00 205.00 206.00 213.00 214.00 215.00 222.0 \n",
"447115 3352.00 3379.00 3380.00 3357.00 3312.00 3257.00 3206.00 3134.0 \n",
"163327 451.00 483.00 538.00 585.00 614.00 630.00 632.00 631.0 \n",
"284220 1604.00 1649.00 1710.00 1727.00 1711.00 1680.00 1628.00 1591.0 \n",
"238931 72.34 74.31 80.62 85.86 89.01 91.63 94.28 94.7 \n",
"122055 77.00 76.00 81.00 84.00 84.00 87.00 90.00 87.0 \n",
"546093 32.00 30.00 34.00 33.00 32.00 32.00 33.00 32.0 \n",
"526007 121.00 125.00 131.00 136.00 139.00 142.00 139.00 137.0 \n",
"322570 924.00 952.00 983.00 1015.00 1038.00 1059.00 1069.00 1071.0 \n",
"543902 86.00 92.00 96.00 96.00 106.00 101.00 106.00 104.0 \n",
"64349 1944.00 2080.00 1996.00 1791.00 1625.00 1541.00 1474.00 1399.0 \n",
"387946 691.00 715.00 734.00 747.00 755.00 752.00 750.00 738.0 \n",
"354710 2822.00 3079.00 3430.00 3764.00 3983.00 4109.00 4196.00 4215.0 \n",
"\n",
" hour16 hour17 hour18 hour19 hour20 hour21 hour22 \\\n",
"63985 1363.00 1380.00 1416.00 1444.00 1551.00 1694.00 1612.00 \n",
"545729 31.00 31.00 32.00 31.00 37.00 39.00 35.00 \n",
"543538 93.00 95.00 98.00 99.00 108.00 105.00 105.00 \n",
"525643 144.00 146.00 148.00 150.00 162.00 169.00 162.00 \n",
"505554 91.00 93.00 92.00 101.00 101.00 105.00 112.00 \n",
"485101 593.00 596.00 608.00 639.00 690.00 688.00 660.00 \n",
"449879 0.00 0.00 0.00 0.00 0.00 0.00 0.00 \n",
"449761 0.00 0.00 0.00 0.00 0.00 0.00 0.00 \n",
"446751 3200.00 3193.00 3191.00 3192.00 3194.00 3313.00 3323.00 \n",
"392330 1154.00 1169.00 1231.00 1280.00 1342.00 1346.00 1282.00 \n",
"387582 842.00 853.00 871.00 888.00 926.00 921.00 887.00 \n",
"322206 1212.00 1208.00 1225.00 1245.00 1302.00 1329.00 1294.00 \n",
"575678 56.00 58.00 61.00 61.00 64.00 67.00 64.00 \n",
"283856 1824.00 1844.00 1862.00 1928.00 2036.00 2011.00 1899.00 \n",
"91377 311.00 325.00 327.00 340.00 351.00 353.00 343.00 \n",
"121691 86.00 83.00 85.00 85.00 98.00 93.00 90.00 \n",
"137203 0.00 0.00 0.00 0.00 0.00 0.00 0.00 \n",
"238567 88.30 89.80 88.60 90.30 97.70 102.00 97.20 \n",
"146527 520.00 521.00 556.00 564.00 628.00 593.00 563.00 \n",
"219939 216.00 220.00 224.00 228.00 249.00 250.00 244.00 \n",
"447115 3092.00 3053.00 3040.00 3051.00 3079.00 3193.00 3208.00 \n",
"163327 626.00 622.00 624.00 627.00 646.00 659.00 625.00 \n",
"284220 1561.00 1531.00 1584.00 1643.00 1820.00 1846.00 1736.00 \n",
"238931 95.77 96.37 97.43 99.79 107.57 109.76 106.17 \n",
"122055 88.00 93.00 92.00 93.00 99.00 96.00 93.00 \n",
"546093 30.00 33.00 33.00 33.00 38.00 38.00 37.00 \n",
"526007 136.00 137.00 141.00 146.00 156.00 158.00 151.00 \n",
"322570 1066.00 1093.00 1130.00 1139.00 1167.00 1162.00 1121.00 \n",
"543902 104.00 107.00 104.00 108.00 113.00 109.00 111.00 \n",
"64349 1353.00 1353.00 1381.00 1417.00 1557.00 1717.00 1647.00 \n",
"387946 732.00 733.00 741.00 751.00 822.00 836.00 804.00 \n",
"354710 4243.00 4254.00 4242.00 4206.00 4297.00 4352.00 4113.00 \n",
"\n",
" hour23 hour24 hour25 timezone_f hour01_f hour02_f hour03_f \\\n",
"63985 1449.00 1272.00 NaN 0 0 0 0 \n",
"545729 34.00 32.00 NaN 0 0 0 0 \n",
"543538 98.00 89.00 NaN 0 0 0 0 \n",
"525643 152.00 141.00 NaN 0 0 0 0 \n",
"505554 106.00 95.00 NaN 0 0 0 0 \n",
"485101 612.00 572.00 NaN 0 0 0 0 \n",
"449879 0.00 0.00 NaN 0 0 0 0 \n",
"449761 0.00 0.00 NaN 0 0 0 0 \n",
"446751 3286.00 3177.00 NaN 0 0 0 0 \n",
"392330 1176.00 1070.00 NaN 0 0 0 0 \n",
"387582 828.00 765.00 NaN 0 0 0 0 \n",
"322206 1217.00 1137.00 NaN 0 0 0 0 \n",
"575678 54.00 45.00 NaN 0 0 0 0 \n",
"283856 1765.00 1639.00 NaN 0 0 0 0 \n",
"91377 324.00 302.00 NaN 0 0 0 0 \n",
"121691 89.00 80.00 NaN 0 0 0 0 \n",
"137203 0.00 0.00 NaN 0 0 0 0 \n",
"238567 88.70 79.20 NaN 0 0 0 0 \n",
"146527 544.00 507.00 NaN 0 0 0 0 \n",
"219939 243.00 231.00 NaN 0 0 0 0 \n",
"447115 3148.00 3043.00 NaN 0 0 0 0 \n",
"163327 574.00 512.00 NaN 0 0 0 0 \n",
"284220 1602.00 1452.00 NaN 0 0 0 0 \n",
"238931 97.21 88.15 NaN 0 0 0 0 \n",
"122055 90.00 77.00 NaN 0 0 0 0 \n",
"546093 35.00 29.00 NaN 0 0 0 0 \n",
"526007 142.00 131.00 NaN 0 0 0 0 \n",
"322570 1045.00 968.00 NaN 0 0 0 0 \n",
"543902 98.00 97.00 NaN 0 0 0 0 \n",
"64349 1525.00 1367.00 NaN 0 0 0 0 \n",
"387946 746.00 688.00 NaN 0 0 0 0 \n",
"354710 3766.00 3315.00 NaN 0 0 0 0 \n",
"\n",
" hour04_f hour05_f hour06_f hour07_f hour08_f hour09_f hour10_f \\\n",
"63985 0 0 0 0 0 0 0 \n",
"545729 0 0 0 0 0 0 0 \n",
"543538 0 0 0 0 0 0 0 \n",
"525643 0 0 0 0 0 0 0 \n",
"505554 0 0 0 0 0 0 0 \n",
"485101 0 0 0 0 0 0 0 \n",
"449879 0 0 0 0 0 0 0 \n",
"449761 0 0 0 0 0 0 0 \n",
"446751 0 0 0 0 0 0 0 \n",
"392330 0 0 0 0 0 0 0 \n",
"387582 0 0 0 0 0 0 0 \n",
"322206 0 0 0 0 0 0 0 \n",
"575678 0 0 0 0 0 0 0 \n",
"283856 0 0 0 0 0 0 0 \n",
"91377 0 0 0 0 0 0 0 \n",
"121691 0 0 0 0 0 0 0 \n",
"137203 0 0 0 0 0 0 0 \n",
"238567 0 0 0 0 0 0 0 \n",
"146527 0 0 0 0 0 0 0 \n",
"219939 0 0 0 0 0 0 0 \n",
"447115 0 0 0 0 0 0 0 \n",
"163327 0 0 0 0 0 0 0 \n",
"284220 0 0 0 0 0 0 0 \n",
"238931 0 0 0 0 0 0 0 \n",
"122055 0 0 0 0 0 0 0 \n",
"546093 0 0 0 0 0 0 0 \n",
"526007 0 0 0 0 0 0 0 \n",
"322570 0 0 0 0 0 0 0 \n",
"543902 0 0 0 0 0 0 0 \n",
"64349 0 0 0 0 0 0 0 \n",
"387946 0 0 0 0 0 0 0 \n",
"354710 0 0 0 0 0 0 0 \n",
"\n",
" hour11_f hour12_f hour13_f hour14_f hour15_f hour16_f hour17_f \\\n",
"63985 0 0 0 0 0 0 0 \n",
"545729 0 0 0 0 0 0 0 \n",
"543538 0 0 0 0 0 0 0 \n",
"525643 0 0 0 0 0 0 0 \n",
"505554 0 0 0 0 0 0 0 \n",
"485101 0 0 0 0 0 0 0 \n",
"449879 0 0 0 0 0 0 0 \n",
"449761 0 0 0 0 0 0 0 \n",
"446751 0 0 0 0 0 0 0 \n",
"392330 0 0 0 0 0 0 0 \n",
"387582 0 0 0 0 0 0 0 \n",
"322206 0 0 0 0 0 0 0 \n",
"575678 0 0 0 0 0 0 0 \n",
"283856 0 0 0 0 0 0 0 \n",
"91377 0 0 0 0 0 0 0 \n",
"121691 0 0 0 0 0 0 0 \n",
"137203 0 0 0 0 0 0 0 \n",
"238567 0 0 0 0 0 0 0 \n",
"146527 0 0 0 0 0 0 0 \n",
"219939 0 0 0 0 0 0 0 \n",
"447115 0 0 0 0 0 0 0 \n",
"163327 0 0 0 0 0 0 0 \n",
"284220 0 0 0 0 0 0 0 \n",
"238931 0 0 0 0 0 0 0 \n",
"122055 0 0 0 0 0 0 0 \n",
"546093 0 0 0 0 0 0 0 \n",
"526007 0 0 0 0 0 0 0 \n",
"322570 0 0 0 0 0 0 0 \n",
"543902 0 0 0 0 0 0 0 \n",
"64349 0 0 0 0 0 0 0 \n",
"387946 0 0 0 0 0 0 0 \n",
"354710 0 0 0 0 0 0 0 \n",
"\n",
" hour18_f hour19_f hour20_f hour21_f hour22_f hour23_f hour24_f \\\n",
"63985 0 0 0 0 0 0 0 \n",
"545729 0 0 0 0 0 0 0 \n",
"543538 0 0 0 0 0 0 0 \n",
"525643 0 0 0 0 0 0 0 \n",
"505554 0 0 0 0 0 0 0 \n",
"485101 0 0 0 0 0 0 0 \n",
"449879 0 0 0 0 0 0 0 \n",
"449761 0 0 0 0 0 0 0 \n",
"446751 0 0 0 0 0 0 0 \n",
"392330 0 0 0 0 0 0 0 \n",
"387582 0 0 0 0 0 0 0 \n",
"322206 0 0 0 0 0 0 0 \n",
"575678 0 0 0 0 0 0 0 \n",
"283856 0 0 0 0 0 0 0 \n",
"91377 0 0 0 0 0 0 0 \n",
"121691 0 0 0 0 0 0 0 \n",
"137203 0 0 0 0 0 0 0 \n",
"238567 0 0 0 0 0 0 0 \n",
"146527 0 0 0 0 0 0 0 \n",
"219939 0 0 0 0 0 0 0 \n",
"447115 0 0 0 0 0 0 0 \n",
"163327 0 0 0 0 0 0 0 \n",
"284220 0 0 0 0 0 0 0 \n",
"238931 0 0 0 0 0 0 0 \n",
"122055 0 0 0 0 0 0 0 \n",
"546093 0 0 0 0 0 0 0 \n",
"526007 0 0 0 0 0 0 0 \n",
"322570 0 0 0 0 0 0 0 \n",
"543902 0 0 0 0 0 0 0 \n",
"64349 0 0 0 0 0 0 0 \n",
"387946 0 0 0 0 0 0 0 \n",
"354710 0 0 0 0 0 0 0 \n",
"\n",
" hour25_f hour_25_valid hour24_25_ratio \n",
"63985 0 False inf \n",
"545729 0 False inf \n",
"543538 0 False inf \n",
"525643 0 False inf \n",
"505554 0 False inf \n",
"485101 0 False inf \n",
"449879 0 False NaN \n",
"449761 0 False NaN \n",
"446751 0 False inf \n",
"392330 0 False inf \n",
"387582 0 False inf \n",
"322206 0 False inf \n",
"575678 0 False inf \n",
"283856 0 False inf \n",
"91377 0 False inf \n",
"121691 0 False inf \n",
"137203 0 False NaN \n",
"238567 0 False inf \n",
"146527 0 False inf \n",
"219939 0 False inf \n",
"447115 0 False inf \n",
"163327 0 False inf \n",
"284220 0 False inf \n",
"238931 0 False inf \n",
"122055 0 False inf \n",
"546093 0 False inf \n",
"526007 0 False inf \n",
"322570 0 False inf \n",
"543902 0 False inf \n",
"64349 0 False inf \n",
"387946 0 False inf \n",
"354710 0 False inf "
]
},
"metadata": {
"tags": []
},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "2DBvKfmZhJnu",
"colab_type": "code",
"colab": {}
},
"source": [
"def timezone_to_tz(timezone):\n",
" return 'Etc/GMT{:+}'.format(-timezone)\n",
"\n",
"def powerset(iterable):\n",
" \"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)\"\n",
" s = list(iterable)\n",
" return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "6clbqBQ5hJcS",
"colab_type": "code",
"outputId": "9179e1f6-9f65-4430-ebe6-a07bf712dd60",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 153
}
},
"source": [
"hourcols = ['hour{:02.0f}'.format(i) for i in range(1,26)]\n",
"\n",
"# These are my best guesses for all of the timezone values that BAs listed\n",
"tz_offset = {\n",
" '1 ': -5,\n",
" \" \": -7,\n",
" \"AKS\": -9,\n",
" \"CST\": -6,\n",
" \"CPT\": -6,\n",
" \"MST\": -7,\n",
" \"PST\": -8,\n",
" \"PDT\": -8,\n",
" \"mst\": -7,\n",
" \"EST\": -5,\n",
" \"EDT\": -5,\n",
" \" CS\": -6,\n",
" \"HST\": -10,\n",
" \"Est\": -5,\n",
" \"PPT\": -8,\n",
" \"MPT\": -7,\n",
" \"EPT\": -5, \n",
" \"MPP\": -7,\n",
"}\n",
"tz_ba = {key: timezone_to_tz(offset) for key, offset in tz_offset.items()}\n",
"error_dfs = {}\n",
"good_dfs = {}\n",
"years = [2011, 2012]\n",
"year_hours = {\n",
" 2011: 8760,\n",
" 2012: 8784\n",
"}\n",
"\n",
"for r in respondents:\n",
" # Not all respondents have data for all years\n",
" r_all_years = corrected_df.loc[\n",
" (corrected_df.respondent_id == r) & (corrected_df.report_yr.isin(years)),\n",
" :\n",
" ]\n",
" # Only proceed if there is positive demand over all years (skip if all 0)\n",
" if r_all_years[hourcols].sum().sum() > 0:\n",
" valid_years = sorted(r_all_years[\"report_yr\"].unique().tolist())\n",
" tz = r_all_years[\"timezone\"].values[0]\n",
" dt = pd.date_range(\n",
" f\"{valid_years[0]}-01-01\", \n",
" f\"{valid_years[-1] + 1}-01-01\",\n",
" freq=\"H\",\n",
" closed=\"left\",\n",
" tz=tz_ba[tz]\n",
" )\n",
"\n",
" df_list = []\n",
" for year in valid_years:\n",
" r_single_year = r_all_years.loc[r_all_years.report_yr == year, :]\n",
"\n",
" # Try to drop March DST changeover values if there are more valid hours\n",
" # (not nan) than hours in the year.\n",
" # I found that hour02, hour03, and hour24 all had 0 values for at\n",
" # least one respondent.\n",
" # Set the 0 values to np.nan so they can be dropped after melting\n",
" if r_single_year[hourcols].count().sum() > year_hours[year]:\n",
" r_single_year.loc[\n",
" (r_single_year[\"hour02\"] == 0)\n",
" & (r_single_year[\"hour01\"] != 0)\n",
" & (r_single_year[\"plan_date\"].dt.month == 3),\n",
" \"hour02\"\n",
" ] = np.nan\n",
" r_single_year.loc[\n",
" (r_single_year[\"hour03\"] == 0)\n",
" & (r_single_year[\"hour01\"] != 0)\n",
" & (r_single_year[\"plan_date\"].dt.month == 3),\n",
" \"hour03\"\n",
" ] = np.nan\n",
" r_single_year.loc[\n",
" (r_single_year[\"hour24\"] == 0)\n",
" & (r_single_year[\"hour01\"] != 0)\n",
" & (r_single_year[\"plan_date\"].dt.month == 3),\n",
" \"hour24\"\n",
" ] = np.nan\n",
" \n",
" tidy_df = pd.melt(r_single_year, id_vars='plan_date', value_vars=hourcols, \n",
" var_name='hour', value_name='demand_MW')\n",
" tidy_df = tidy_df.sort_values([\"plan_date\", \"hour\"])\n",
"\n",
" tidy_df = tidy_df.dropna()\n",
" tidy_df[\"hour\"] = tidy_df[\"hour\"].str[-2:].astype(int)\n",
" tidy_df[\"respondent_id\"] = r\n",
"\n",
" df_list.append(tidy_df)\n",
"\n",
" # Concat the years together\n",
" r_df = pd.concat(df_list)\n",
" r_df = r_df.reset_index(drop=True)\n",
"\n",
" # If the length of one year, all years, or some combination of years\n",
" if len(r_df) in [sum(x) for x in powerset(year_hours.values())]:\n",
"\n",
" r_df[\"date_time\"] = dt\n",
" columns = [\"date_time\", \"demand_MW\", \"respondent_id\"]\n",
" r_df = r_df.loc[:, columns]\n",
" good_dfs[r] = r_df\n",
"\n",
" else:\n",
" print(r, len(r_df))\n",
" error_dfs[r] = r_df\n"
],
"execution_count": 13,
"outputs": [
{
"output_type": "stream",
"text": [
"/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py:494: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" self.obj[item] = s\n"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"225 17545\n",
"311 9149\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BIFVTZb8r2Tc",
"colab_type": "text"
},
"source": [
"From the code above it looks like ~120 BAs have data clean enough that I'm able to extract the correct number of hours. \n",
"\n",
"\\#225 has 2 days with demand in hour25 (daylight savings) but only one of the years has an hour in March with 0 demand. I suppose I could just remove an hour from that day?\n",
"\n",
"\\# 311 has demand in hour25 every day (or just about). Interestingly, hour01 and hour02 seem to always have the same values. Maybe just drop hour01 and shift everything over?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6ZyhiKg9EbOW",
"colab_type": "text"
},
"source": [
"## Check the demand data for anomalies\n",
"\n",
"The anomaly checking functions and parameter values below are all from [a notebook](https://github.com/truggles/EIA_Cleaned_Hourly_Electricity_Demand_Code) by Tyler Ruggles, which he developed to screen [hourly demand data from EIA-931](https://www.eia.gov/realtime_grid/#/status?end=20200325T07). I've modified some functions to speed them up. EIA's hourly data only goes back to mid-2015, which is why I'm using FERC 714. Fortunately it looks like the FERC data — once properly extracted — doesn't have many of the anomalies that Tyler found in the EIA data. Tyler also [developed a method](https://github.com/truggles/EIA_Cleaned_Hourly_Electricity_Demand_Code/blob/master/MICE_step.Rmd) to impute missing or anomalous data but at first glance it looks like the FERC demand data can largely be used as-is without too many issues. Let me know if you disagree!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZXjdiqakbxcQ",
"colab_type": "code",
"colab": {}
},
"source": [
"short_hour_window = 24 # 48 hour moving median (M_{t,48hr})\n",
"iqr_hours = 24*5 # width in hours of IQR values of relative deviations from diurnal cycle template (IQR_{dem,t})\n",
"nDays = 10 # Used for normalized hourly demand template (h_{t,diurnal}) and 480 hour moving median (M_{t,480hr})\n",
"global_dem_cut = 10 # threshold selection for global demand filter\n",
"local_dem_cut_up = 3.5 # upwards threshold for local demand filter\n",
"local_dem_cut_down = 2.5 # downwards threshold for local demand filter\n",
"delta_multiplier = 2 # selection threshold for double-sided delta filter\n",
"delta_single_multiplier = 5 # selection threshold for single-sided delta filter\n",
"rel_multiplier = 15 # other selection threshold for single-sided delta filter\n",
"anomalous_regions_width = 24 # width in hours of anomalous region filter\n",
"anomalous_pct = .85 # required pct of good data in anomalous region filter"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jRQ3Zq5nEeVf",
"colab_type": "code",
"outputId": "b827b69f-bb14-47f2-d489-154ccfe1fa30",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
}
},
"source": [
"name_df_list = Parallel(n_jobs=-1, verbose=10)(delayed(screen_timeseries)(\n",
" name=name, \n",
" df=df, \n",
" short_hour_window=short_hour_window,\n",
" iqr_hours=iqr_hours,\n",
" nDays=nDays,\n",
" global_dem_cut=global_dem_cut,\n",
" local_dem_cut_up=local_dem_cut_up,\n",
" local_dem_cut_down=local_dem_cut_down,\n",
" delta_multiplier=delta_multiplier,\n",
" delta_single_multiplier=delta_single_multiplier,\n",
" rel_multiplier=rel_multiplier,\n",
" anomalous_regions_width=rel_multiplier,\n",
" anomalous_pct=anomalous_pct\n",
" ) for name, df in good_dfs.items()\n",
")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"[Parallel(n_jobs=-1)]: Using backend LokyBackend with 2 concurrent workers.\n",
"[Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 7.0s\n",
"[Parallel(n_jobs=-1)]: Done 4 tasks | elapsed: 19.1s\n",
"[Parallel(n_jobs=-1)]: Done 9 tasks | elapsed: 31.7s\n",
"[Parallel(n_jobs=-1)]: Done 14 tasks | elapsed: 49.5s\n",
"[Parallel(n_jobs=-1)]: Done 21 tasks | elapsed: 1.1min\n",
"[Parallel(n_jobs=-1)]: Done 28 tasks | elapsed: 1.5min\n",
"[Parallel(n_jobs=-1)]: Done 37 tasks | elapsed: 2.0min\n",
"[Parallel(n_jobs=-1)]: Done 46 tasks | elapsed: 2.4min\n",
"[Parallel(n_jobs=-1)]: Done 57 tasks | elapsed: 3.0min\n",
"[Parallel(n_jobs=-1)]: Done 68 tasks | elapsed: 3.6min\n",
"[Parallel(n_jobs=-1)]: Done 81 tasks | elapsed: 4.3min\n",
"[Parallel(n_jobs=-1)]: Done 94 tasks | elapsed: 5.1min\n",
"[Parallel(n_jobs=-1)]: Done 109 tasks | elapsed: 5.9min\n",
"[Parallel(n_jobs=-1)]: Done 120 out of 120 | elapsed: 6.5min finished\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "5WcZMeOTEkaf",
"colab_type": "code",
"outputId": "e8f8e35f-6c81-4be0-a134-f0b4437c7d99",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 439
}
},
"source": [
"anomaly_dict = {name: df for name, df in name_df_list}\n",
"summary_df = make_anomaly_summary(anomaly_dict)\n",
"summary_df"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>median_demand</th>\n",
" <th>OKAY</th>\n",
" <th>MISSING</th>\n",
" <th>NEG_OR_ZERO</th>\n",
" <th>GLOBAL_DEM</th>\n",
" <th>GLOBAL_DEM_PLUS_MINUS</th>\n",
" <th>LOCAL_DEM_UP</th>\n",
" <th>LOCAL_DEM_DOWN</th>\n",
" <th>DELTA</th>\n",
" <th>SINGLE_DELTA</th>\n",
" <th>IDENTICAL_RUN</th>\n",
" <th>ANOMALOUS_REGION</th>\n",
" <th>pct_OKAY</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>101</td>\n",
" <td>879.07</td>\n",
" <td>17538</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0.999658</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>102</td>\n",
" <td>6753.00</td>\n",
" <td>17543</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0.999943</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>110</td>\n",
" <td>4747.00</td>\n",
" <td>17544</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>115</td>\n",
" <td>78.00</td>\n",
" <td>14613</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>897</td>\n",
" <td>2032</td>\n",
" <td>0.832934</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>116</td>\n",
" <td>3286.00</td>\n",
" <td>17544</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>115</th>\n",
" <td>298</td>\n",
" <td>1261.00</td>\n",
" <td>17543</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0.999943</td>\n",
" </tr>\n",
" <tr>\n",
" <th>116</th>\n",
" <td>299</td>\n",
" <td>348.00</td>\n",
" <td>17518</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>26</td>\n",
" <td>0</td>\n",
" <td>0.998518</td>\n",
" </tr>\n",
" <tr>\n",
" <th>117</th>\n",
" <td>307</td>\n",
" <td>6728.60</td>\n",
" <td>17544</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>118</th>\n",
" <td>308</td>\n",
" <td>62.00</td>\n",
" <td>17371</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>171</td>\n",
" <td>0</td>\n",
" <td>0.990139</td>\n",
" </tr>\n",
" <tr>\n",
" <th>119</th>\n",
" <td>321</td>\n",
" <td>58622.00</td>\n",
" <td>17544</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>120 rows × 14 columns</p>\n",
"</div>"
],
"text/plain": [
" name median_demand OKAY MISSING NEG_OR_ZERO GLOBAL_DEM \\\n",
"0 101 879.07 17538 0 1 0 \n",
"1 102 6753.00 17543 0 0 0 \n",
"2 110 4747.00 17544 0 0 0 \n",
"3 115 78.00 14613 0 0 0 \n",
"4 116 3286.00 17544 0 0 0 \n",
".. ... ... ... ... ... ... \n",
"115 298 1261.00 17543 0 0 0 \n",
"116 299 348.00 17518 0 0 0 \n",
"117 307 6728.60 17544 0 0 0 \n",
"118 308 62.00 17371 0 1 0 \n",
"119 321 58622.00 17544 0 0 0 \n",
"\n",
" GLOBAL_DEM_PLUS_MINUS LOCAL_DEM_UP LOCAL_DEM_DOWN DELTA SINGLE_DELTA \\\n",
"0 0 0 1 4 0 \n",
"1 0 0 0 1 0 \n",
"2 0 0 0 0 0 \n",
"3 0 0 2 0 0 \n",
"4 0 0 0 0 0 \n",
".. ... ... ... ... ... \n",
"115 0 0 0 1 0 \n",
"116 0 0 0 0 0 \n",
"117 0 0 0 0 0 \n",
"118 0 0 0 1 0 \n",
"119 0 0 0 0 0 \n",
"\n",
" IDENTICAL_RUN ANOMALOUS_REGION pct_OKAY \n",
"0 0 0 0.999658 \n",
"1 0 0 0.999943 \n",
"2 0 0 1.000000 \n",
"3 897 2032 0.832934 \n",
"4 0 0 1.000000 \n",
".. ... ... ... \n",
"115 0 0 0.999943 \n",
"116 26 0 0.998518 \n",
"117 0 0 1.000000 \n",
"118 171 0 0.990139 \n",
"119 0 0 1.000000 \n",
"\n",
"[120 rows x 14 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 18
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "D-XSSs6AE18r",
"colab_type": "code",
"outputId": "759d5cb4-8f81-48ba-a128-e9aa6cd22996",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 317
}
},
"source": [
"summary_df.describe()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>median_demand</th>\n",
" <th>OKAY</th>\n",
" <th>MISSING</th>\n",
" <th>NEG_OR_ZERO</th>\n",
" <th>GLOBAL_DEM</th>\n",
" <th>GLOBAL_DEM_PLUS_MINUS</th>\n",
" <th>LOCAL_DEM_UP</th>\n",
" <th>LOCAL_DEM_DOWN</th>\n",
" <th>DELTA</th>\n",
" <th>SINGLE_DELTA</th>\n",
" <th>IDENTICAL_RUN</th>\n",
" <th>ANOMALOUS_REGION</th>\n",
" <th>pct_OKAY</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>120.000000</td>\n",
" <td>118.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.0</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.0</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" <td>120.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>210.433333</td>\n",
" <td>4074.016356</td>\n",
" <td>16993.791667</td>\n",
" <td>0.0</td>\n",
" <td>50.033333</td>\n",
" <td>0.050000</td>\n",
" <td>0.0</td>\n",
" <td>4.191667</td>\n",
" <td>16.291667</td>\n",
" <td>4.891667</td>\n",
" <td>2.250000</td>\n",
" <td>303.150000</td>\n",
" <td>96.150000</td>\n",
" <td>0.972809</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>55.879633</td>\n",
" <td>10646.382812</td>\n",
" <td>2758.183088</td>\n",
" <td>0.0</td>\n",
" <td>445.498568</td>\n",
" <td>0.547723</td>\n",
" <td>0.0</td>\n",
" <td>24.807933</td>\n",
" <td>163.768867</td>\n",
" <td>21.980815</td>\n",
" <td>14.088705</td>\n",
" <td>1965.903862</td>\n",
" <td>855.626338</td>\n",
" <td>0.151178</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>101.000000</td>\n",
" <td>44.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>164.750000</td>\n",
" <td>350.250000</td>\n",
" <td>17510.750000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.998176</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>213.500000</td>\n",
" <td>1136.500000</td>\n",
" <td>17537.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>3.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.999629</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>257.500000</td>\n",
" <td>2765.000000</td>\n",
" <td>17542.250000</td>\n",
" <td>0.0</td>\n",
" <td>1.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>2.000000</td>\n",
" <td>0.000000</td>\n",
" <td>26.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.999900</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>321.000000</td>\n",
" <td>85017.500000</td>\n",
" <td>17544.000000</td>\n",
" <td>0.0</td>\n",
" <td>4750.000000</td>\n",
" <td>6.000000</td>\n",
" <td>0.0</td>\n",
" <td>225.000000</td>\n",
" <td>1794.000000</td>\n",
" <td>224.000000</td>\n",
" <td>142.000000</td>\n",
" <td>17288.000000</td>\n",
" <td>9169.000000</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name median_demand OKAY MISSING NEG_OR_ZERO \\\n",
"count 120.000000 118.000000 120.000000 120.0 120.000000 \n",
"mean 210.433333 4074.016356 16993.791667 0.0 50.033333 \n",
"std 55.879633 10646.382812 2758.183088 0.0 445.498568 \n",
"min 101.000000 44.000000 0.000000 0.0 0.000000 \n",
"25% 164.750000 350.250000 17510.750000 0.0 0.000000 \n",
"50% 213.500000 1136.500000 17537.000000 0.0 0.000000 \n",
"75% 257.500000 2765.000000 17542.250000 0.0 1.000000 \n",
"max 321.000000 85017.500000 17544.000000 0.0 4750.000000 \n",
"\n",
" GLOBAL_DEM GLOBAL_DEM_PLUS_MINUS LOCAL_DEM_UP LOCAL_DEM_DOWN \\\n",
"count 120.000000 120.0 120.000000 120.000000 \n",
"mean 0.050000 0.0 4.191667 16.291667 \n",
"std 0.547723 0.0 24.807933 163.768867 \n",
"min 0.000000 0.0 0.000000 0.000000 \n",
"25% 0.000000 0.0 0.000000 0.000000 \n",
"50% 0.000000 0.0 0.000000 0.000000 \n",
"75% 0.000000 0.0 0.000000 0.000000 \n",
"max 6.000000 0.0 225.000000 1794.000000 \n",
"\n",
" DELTA SINGLE_DELTA IDENTICAL_RUN ANOMALOUS_REGION pct_OKAY \n",
"count 120.000000 120.000000 120.000000 120.000000 120.000000 \n",
"mean 4.891667 2.250000 303.150000 96.150000 0.972809 \n",
"std 21.980815 14.088705 1965.903862 855.626338 0.151178 \n",
"min 0.000000 0.000000 0.000000 0.000000 0.000000 \n",
"25% 0.000000 0.000000 0.000000 0.000000 0.998176 \n",
"50% 0.000000 0.000000 3.000000 0.000000 0.999629 \n",
"75% 2.000000 0.000000 26.000000 0.000000 0.999900 \n",
"max 224.000000 142.000000 17288.000000 9169.000000 1.000000 "
]
},
"metadata": {
"tags": []
},
"execution_count": 19
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PFDpZLG6E4cH",
"colab_type": "text"
},
"source": [
"## Matching FERC respondents to EIA-816 utilities\n",
"\n",
"The SI of [Auffhammer et al](https://www.pnas.org/content/pnas/suppl/2017/02/01/1613193114.DCSupplemental/pnas.1613193114.sapp.pdf) describes how they derived geographic coverage of FERC respondents using EIA-861 and the crosswalk between FERC respondents and EIA utilities/BAs. I'm including a little code to start exploring that here.\n",
"\n",
"This is very exploratory! I'm trying to match codes between FERC and 861 but haven't figured out yet if they should be matched to EIA BAs or Utilities. At a minimum it looks like there are 84 FERC respondents that don't have a match in either category of the 2012 861.\n",
"\n",
"A few extra resources that might be helpful:\n",
"- SPP [historical hourly load](https://marketplace.spp.org/pages/hourly-load#) back through 2011. The company acronyms (at least for 2011/2012 data) are described in [this document](https://www.nerc.com/pa/rrm/Resources/Monitoring_and_Situational_Awareness_Conference2/2%20Testing%20Your%20Sensitivity%20To%20Loss%20of%20Data_T%20Miller.pdf).\n",
"- MISO has [archivled historical hourly load](https://www.misoenergy.org/markets-and-operations/real-time--market-data/market-reports/market-report-archives/#nt=%2FMarketReportType%3ASummary%2FMarketReportName%3AArchived%20Historical%20Regional%20Forecast%20and%20Actual%20Load%20%20(zip)&t=10&p=0&s=MarketReportPublished&sd=desc) for 2007-2014 and [historical hourly load](https://www.misoenergy.org/markets-and-operations/real-time--market-data/market-reports/#nt=%2FMarketReportType%3ASummary%2FMarketReportName%3AHistorical%20Regional%20Forecast%20and%20Actual%20Load%20(xls)&t=10&p=0&s=MarketReportPublished&sd=desc) back through 2013 for Central, East, and West regions.\n",
"- ERCOT has [historical hourly load](http://www.ercot.com/gridinfo/load/load_hist/) in each of the 8 weather zones. (NOTE: the IPM region ERC_PHDL has no demand, so all weather zones are split into ERC_WEST and ERC_REST)\n",
"- PJM requires a login and API key to access their [historical metered data](https://dataminer2.pjm.com/feed/hrl_load_metered/definition) through dataminer (back to 1993).\n",
"- NYISO has hourly load by zone (which look like they group well into IPM regions) back through 2001. I need to figure out the difference between [real time actual load](http://mis.nyiso.com/public/P-58Blist.htm) and [integrated real time actual load](http://mis.nyiso.com/public/P-58Clist.htm).\n",
"- ISONE has [hourly load by zone](https://www.iso-ne.com/isoexpress/web/reports/load-and-demand/-/tree/zone-info) back through 2011. Unfortunately, from a quick glance at 2011 it looks like while there are only 8760 hourly demand values the 2am on March 13 (DST) has demand of 0. Oh, but the 2am value on Nov. 6 is twice the surrounding hours, so they appear to add two hours together on one line.... (Need to remember this as something other places might do)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "PN2ouEdRE3-1",
"colab_type": "code",
"colab": {}
},
"source": [
"# Download the 2012 EIA-861 data to a temp folder\n",
"url = 'https://www.eia.gov/electricity/data/eia861/archive/zip/f8612012.zip'\n",
"save_folder = cwd / \"EIA861\"\n",
"save_folder.mkdir(parents=True, exist_ok=True)\n",
"urllib.request.urlretrieve(url, save_folder / 'f8612012.zip')\n",
"### Unzip it\n",
"data_path = save_folder / \"f8612012\"\n",
"with zipfile.ZipFile(save_folder / 'f8612012.zip', 'r') as zfile:\n",
" zfile.extractall(data_path)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "OIsdXi04E9dr",
"colab_type": "code",
"outputId": "7fd86d1b-8fe3-4a2a-8697-008f8f09366a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
}
},
"source": [
"ferc_eia_map = pd.read_csv(cwd / \"FERC\" / \"form714-database\" / \"Respondent IDs.csv\", index_col=0)\n",
"\n",
"# Only keep the respondents that are in the years of data we're looking at\n",
"ferc_eia_map = ferc_eia_map.loc[respondents, :]\n",
"ferc_eia_map.head()"
],
"execution_count": 16,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_name</th>\n",
" <th>eia_code</th>\n",
" </tr>\n",
" <tr>\n",
" <th>respondent_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>101</th>\n",
" <td>PowerSouth Energy Cooperative (Alabama Electri...</td>\n",
" <td>189</td>\n",
" </tr>\n",
" <tr>\n",
" <th>102</th>\n",
" <td>Alabama Power Company ...</td>\n",
" <td>195</td>\n",
" </tr>\n",
" <tr>\n",
" <th>110</th>\n",
" <td>American Electric Power Company, Inc. ...</td>\n",
" <td>829</td>\n",
" </tr>\n",
" <tr>\n",
" <th>115</th>\n",
" <td>Arizona Electric Power Cooperative, Inc. ...</td>\n",
" <td>796</td>\n",
" </tr>\n",
" <tr>\n",
" <th>116</th>\n",
" <td>Arizona Public Service Company ...</td>\n",
" <td>803</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_name eia_code\n",
"respondent_id \n",
"101 PowerSouth Energy Cooperative (Alabama Electri... 189\n",
"102 Alabama Power Company ... 195\n",
"110 American Electric Power Company, Inc. ... 829\n",
"115 Arizona Electric Power Cooperative, Inc. ... 796\n",
"116 Arizona Public Service Company ... 803"
]
},
"metadata": {
"tags": []
},
"execution_count": 16
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Sf_Ov6x6FBrW",
"colab_type": "code",
"colab": {}
},
"source": [
"# 307 is PacifiCorp - Part II Sch 2 (East & West combined), which looks to be\n",
"# EIA utility number 14354\n",
"summary_df[\"eia_code\"] = summary_df[\"name\"].map(ferc_eia_map[\"eia_code\"])\n",
"summary_df.query(\"~(eia_code > 0)\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "WFPHvT4uFH-c",
"colab_type": "code",
"outputId": "3841e718-4e8d-4858-f578-f74289489b8c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
}
},
"source": [
"eia8612012_territory = pd.read_excel(cwd / \"EIA861\" / \"f8612012\" / \"service_territory_2012.xls\")\n",
"eia8612012_territory.head()"
],
"execution_count": 18,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Data Year</th>\n",
" <th>Utility Number</th>\n",
" <th>Utility Name</th>\n",
" <th>State</th>\n",
" <th>County</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2012</td>\n",
" <td>34</td>\n",
" <td>City of Abbeville - (SC)</td>\n",
" <td>SC</td>\n",
" <td>Abbeville</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2012</td>\n",
" <td>55</td>\n",
" <td>City of Aberdeen - (MS)</td>\n",
" <td>MS</td>\n",
" <td>Monroe</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2012</td>\n",
" <td>59</td>\n",
" <td>City of Abbeville - (LA)</td>\n",
" <td>LA</td>\n",
" <td>Vermilion</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2012</td>\n",
" <td>84</td>\n",
" <td>A &amp; N Electric Coop</td>\n",
" <td>MD</td>\n",
" <td>Somerset</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2012</td>\n",
" <td>84</td>\n",
" <td>A &amp; N Electric Coop</td>\n",
" <td>VA</td>\n",
" <td>Accomack</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Data Year Utility Number Utility Name State County\n",
"0 2012 34 City of Abbeville - (SC) SC Abbeville\n",
"1 2012 55 City of Aberdeen - (MS) MS Monroe\n",
"2 2012 59 City of Abbeville - (LA) LA Vermilion\n",
"3 2012 84 A & N Electric Coop MD Somerset\n",
"4 2012 84 A & N Electric Coop VA Accomack"
]
},
"metadata": {
"tags": []
},
"execution_count": 18
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "On7uQOayFM_a",
"colab_type": "code",
"colab": {}
},
"source": [
"# Document which ferc respondents can match with a utility. These utilities have\n",
"# a list of all the counties they are active in.\n",
"ferc_eia_map[\"utility_match\"] = False\n",
"ferc_eia_map.loc[\n",
" ferc_eia_map.eia_code.isin(eia8612012_territory['Utility Number']),\n",
" \"utility_match\"\n",
"] = True"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "sjd_p3v7FOpP",
"colab_type": "code",
"outputId": "485654c2-d2bd-4fad-c7ac-675159becea8",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
}
},
"source": [
"eia8612012_bas = pd.read_excel(cwd / \"EIA861\" / \"f8612012\" / \"balancing_authority_2012.xls\")\n",
"eia8612012_bas.head()"
],
"execution_count": 20,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Data Year</th>\n",
" <th>Utility Number</th>\n",
" <th>Utility Name</th>\n",
" <th>BA Code</th>\n",
" <th>Balancing Authority Name</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2012</td>\n",
" <td>34</td>\n",
" <td>City of Abbeville - (SC)</td>\n",
" <td>5416</td>\n",
" <td>Duke Energy Carolinas</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2012</td>\n",
" <td>55</td>\n",
" <td>City of Aberdeen - (MS)</td>\n",
" <td>18642</td>\n",
" <td>Tennessee Valley Authority</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2012</td>\n",
" <td>59</td>\n",
" <td>City of Abbeville - (LA)</td>\n",
" <td>3265</td>\n",
" <td>Cleco Corporation, Inc.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2012</td>\n",
" <td>84</td>\n",
" <td>A &amp; N Electric Coop</td>\n",
" <td>14725</td>\n",
" <td>PJM Interconnection</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2012</td>\n",
" <td>87</td>\n",
" <td>City of Ada</td>\n",
" <td>56669</td>\n",
" <td>Midwest Independent System Operator</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Data Year Utility Number Utility Name BA Code \\\n",
"0 2012 34 City of Abbeville - (SC) 5416 \n",
"1 2012 55 City of Aberdeen - (MS) 18642 \n",
"2 2012 59 City of Abbeville - (LA) 3265 \n",
"3 2012 84 A & N Electric Coop 14725 \n",
"4 2012 87 City of Ada 56669 \n",
"\n",
" Balancing Authority Name \n",
"0 Duke Energy Carolinas \n",
"1 Tennessee Valley Authority \n",
"2 Cleco Corporation, Inc. \n",
"3 PJM Interconnection \n",
"4 Midwest Independent System Operator "
]
},
"metadata": {
"tags": []
},
"execution_count": 20
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "SdwzToM0FTo_",
"colab_type": "code",
"colab": {}
},
"source": [
"ferc_eia_map[\"ba_match\"] = False\n",
"ferc_eia_map.loc[\n",
" ferc_eia_map.eia_code.isin(eia8612012_bas['BA Code']),\n",
" \"ba_match\"\n",
"] = True"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "_vs0kIlCKoIM",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "9kg191UxKqJx",
"colab_type": "text"
},
"source": [
"## Examine which entities we can match to BAs and utilities\n",
"\n",
"It looks like some of the respondents are only BAs, some are only utilities, and some are both. Will need to figure out if any of the utilities that are not a BA are within a BA territory - are there cases where demand from one respondent is a subset of another respondent?"
]
},
{
"cell_type": "code",
"metadata": {
"id": "771mtwQhKUDj",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 450
},
"outputId": "58ffb2c7-77c2-4265-c82e-65be5915dcf0"
},
"source": [
"ferc_eia_map.query(\"ba_match==True\")"
],
"execution_count": 25,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_name</th>\n",
" <th>eia_code</th>\n",
" <th>utility_match</th>\n",
" <th>ba_match</th>\n",
" </tr>\n",
" <tr>\n",
" <th>respondent_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>101</th>\n",
" <td>PowerSouth Energy Cooperative (Alabama Electri...</td>\n",
" <td>189</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>116</th>\n",
" <td>Arizona Public Service Company ...</td>\n",
" <td>803</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>118</th>\n",
" <td>Associated Electric Cooperative, Inc. ...</td>\n",
" <td>924</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>119</th>\n",
" <td>Avista Corporation ...</td>\n",
" <td>20169</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>122</th>\n",
" <td>Bonneville Power Administration, USDOE ...</td>\n",
" <td>1738</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>273</th>\n",
" <td>Western Area Power Administration - Colorado-M...</td>\n",
" <td>28503</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>274</th>\n",
" <td>Western Area Power Administration - Lower Colo...</td>\n",
" <td>19610</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>275</th>\n",
" <td>Western Area Power Administration - Upper Miss...</td>\n",
" <td>25471</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>277</th>\n",
" <td>Western Farmers Electric Cooperative ...</td>\n",
" <td>20447</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>321</th>\n",
" <td>MISO ...</td>\n",
" <td>56669</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>67 rows × 4 columns</p>\n",
"</div>"
],
"text/plain": [
" respondent_name eia_code \\\n",
"respondent_id \n",
"101 PowerSouth Energy Cooperative (Alabama Electri... 189 \n",
"116 Arizona Public Service Company ... 803 \n",
"118 Associated Electric Cooperative, Inc. ... 924 \n",
"119 Avista Corporation ... 20169 \n",
"122 Bonneville Power Administration, USDOE ... 1738 \n",
"... ... ... \n",
"273 Western Area Power Administration - Colorado-M... 28503 \n",
"274 Western Area Power Administration - Lower Colo... 19610 \n",
"275 Western Area Power Administration - Upper Miss... 25471 \n",
"277 Western Farmers Electric Cooperative ... 20447 \n",
"321 MISO ... 56669 \n",
"\n",
" utility_match ba_match \n",
"respondent_id \n",
"101 False True \n",
"116 True True \n",
"118 False True \n",
"119 True True \n",
"122 False True \n",
"... ... ... \n",
"273 False True \n",
"274 False True \n",
"275 False True \n",
"277 False True \n",
"321 False True \n",
"\n",
"[67 rows x 4 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 25
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "PGcpGgxZFVJW",
"colab_type": "code",
"outputId": "42ec3fed-04dd-4e0b-bf20-9b0eb0fb8df3",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"ferc_eia_map.query(\"utility_match==False & ba_match==False\")"
],
"execution_count": 22,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_name</th>\n",
" <th>eia_code</th>\n",
" <th>utility_match</th>\n",
" <th>ba_match</th>\n",
" </tr>\n",
" <tr>\n",
" <th>respondent_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>110</th>\n",
" <td>American Electric Power Company, Inc. ...</td>\n",
" <td>829</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>115</th>\n",
" <td>Arizona Electric Power Cooperative, Inc. ...</td>\n",
" <td>796</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>124</th>\n",
" <td>Buckeye Power, Inc. ...</td>\n",
" <td>7004</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>128</th>\n",
" <td>Central Electric Power Cooperative, Inc. ...</td>\n",
" <td>40218</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>169</th>\n",
" <td>Florida Municipal Power Agency ...</td>\n",
" <td>6567</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>173</th>\n",
" <td>Golden Spread Electric Cooperative, Inc. ...</td>\n",
" <td>7349</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>183</th>\n",
" <td>Indiana Municipal Power Agency ...</td>\n",
" <td>9234</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>190</th>\n",
" <td>Westar Energy (KPL) ...</td>\n",
" <td>10015</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>199</th>\n",
" <td>Massachusetts Municipal Wholesale ...</td>\n",
" <td>11806</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>201</th>\n",
" <td>Metropolitan Water District of Southern Califo...</td>\n",
" <td>12397</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>213</th>\n",
" <td>Northeast Texas Electric Cooperative ...</td>\n",
" <td>13670</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>214</th>\n",
" <td>Northeast Utilities Service Company ...</td>\n",
" <td>13356</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>218</th>\n",
" <td>Oglethorpe Power Company ...</td>\n",
" <td>13994</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>221</th>\n",
" <td>Oklahoma Municipal Power Authority ...</td>\n",
" <td>14077</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>231</th>\n",
" <td>Platte River Power Authority ...</td>\n",
" <td>15143</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>245</th>\n",
" <td>Sam Rayburn G&amp;T Electric Coop. ...</td>\n",
" <td>40233</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>257</th>\n",
" <td>Southwest Power Pool (SPP) ...</td>\n",
" <td>17690</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>260</th>\n",
" <td>Square Butte Electric Coop ...</td>\n",
" <td>17858</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>264</th>\n",
" <td>Tex-La Electric Cooperative of Texas, Inc. ...</td>\n",
" <td>18679</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>265</th>\n",
" <td>Tri-State G &amp; T Assn., Inc. ...</td>\n",
" <td>30151</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>271</th>\n",
" <td>Wabash Valley Power Association, Inc. ...</td>\n",
" <td>40211</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>272</th>\n",
" <td>Western Area Power Admin - Upper Missouri-East...</td>\n",
" <td>28502</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>283</th>\n",
" <td>Minnesota Municipal Power Agency ...</td>\n",
" <td>12667</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>284</th>\n",
" <td>Missouri River Energy Services ...</td>\n",
" <td>12710</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>285</th>\n",
" <td>Minnkota Power Cooperative, Inc. ...</td>\n",
" <td>12658</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>287</th>\n",
" <td>Basin Electric Power Cooperative ...</td>\n",
" <td>1307</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>292</th>\n",
" <td>City of West Memphis ...</td>\n",
" <td>20383</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>299</th>\n",
" <td>Southern Power Company ...</td>\n",
" <td>16687</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>307</th>\n",
" <td>PacifiCorp - Part II Sch 2 (East &amp; West combin...</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>308</th>\n",
" <td>City of St. Cloud ...</td>\n",
" <td>17867</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>311</th>\n",
" <td>Arkansas Electric Cooperative Corporation ...</td>\n",
" <td>807</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>328</th>\n",
" <td>Municipal Electric Authority of Georgia ...</td>\n",
" <td>13100</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>329</th>\n",
" <td>East Texas Electric Cooperative ...</td>\n",
" <td>0</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_name eia_code \\\n",
"respondent_id \n",
"110 American Electric Power Company, Inc. ... 829 \n",
"115 Arizona Electric Power Cooperative, Inc. ... 796 \n",
"124 Buckeye Power, Inc. ... 7004 \n",
"128 Central Electric Power Cooperative, Inc. ... 40218 \n",
"169 Florida Municipal Power Agency ... 6567 \n",
"173 Golden Spread Electric Cooperative, Inc. ... 7349 \n",
"183 Indiana Municipal Power Agency ... 9234 \n",
"190 Westar Energy (KPL) ... 10015 \n",
"199 Massachusetts Municipal Wholesale ... 11806 \n",
"201 Metropolitan Water District of Southern Califo... 12397 \n",
"213 Northeast Texas Electric Cooperative ... 13670 \n",
"214 Northeast Utilities Service Company ... 13356 \n",
"218 Oglethorpe Power Company ... 13994 \n",
"221 Oklahoma Municipal Power Authority ... 14077 \n",
"231 Platte River Power Authority ... 15143 \n",
"245 Sam Rayburn G&T Electric Coop. ... 40233 \n",
"257 Southwest Power Pool (SPP) ... 17690 \n",
"260 Square Butte Electric Coop ... 17858 \n",
"264 Tex-La Electric Cooperative of Texas, Inc. ... 18679 \n",
"265 Tri-State G & T Assn., Inc. ... 30151 \n",
"271 Wabash Valley Power Association, Inc. ... 40211 \n",
"272 Western Area Power Admin - Upper Missouri-East... 28502 \n",
"283 Minnesota Municipal Power Agency ... 12667 \n",
"284 Missouri River Energy Services ... 12710 \n",
"285 Minnkota Power Cooperative, Inc. ... 12658 \n",
"287 Basin Electric Power Cooperative ... 1307 \n",
"292 City of West Memphis ... 20383 \n",
"299 Southern Power Company ... 16687 \n",
"307 PacifiCorp - Part II Sch 2 (East & West combin... 0 \n",
"308 City of St. Cloud ... 17867 \n",
"311 Arkansas Electric Cooperative Corporation ... 807 \n",
"328 Municipal Electric Authority of Georgia ... 13100 \n",
"329 East Texas Electric Cooperative ... 0 \n",
"\n",
" utility_match ba_match \n",
"respondent_id \n",
"110 False False \n",
"115 False False \n",
"124 False False \n",
"128 False False \n",
"169 False False \n",
"173 False False \n",
"183 False False \n",
"190 False False \n",
"199 False False \n",
"201 False False \n",
"213 False False \n",
"214 False False \n",
"218 False False \n",
"221 False False \n",
"231 False False \n",
"245 False False \n",
"257 False False \n",
"260 False False \n",
"264 False False \n",
"265 False False \n",
"271 False False \n",
"272 False False \n",
"283 False False \n",
"284 False False \n",
"285 False False \n",
"287 False False \n",
"292 False False \n",
"299 False False \n",
"307 False False \n",
"308 False False \n",
"311 False False \n",
"328 False False \n",
"329 False False "
]
},
"metadata": {
"tags": []
},
"execution_count": 22
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "pCiP34U6FXNA",
"colab_type": "code",
"outputId": "33f7149c-eaf5-4ce9-8ed1-ff8b86392272",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"# Some entities are both a BA and a utility\n",
"ferc_eia_map.query(\"utility_match==True & ba_match==True\")"
],
"execution_count": 23,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_name</th>\n",
" <th>eia_code</th>\n",
" <th>utility_match</th>\n",
" <th>ba_match</th>\n",
" </tr>\n",
" <tr>\n",
" <th>respondent_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>116</th>\n",
" <td>Arizona Public Service Company ...</td>\n",
" <td>803</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>119</th>\n",
" <td>Avista Corporation ...</td>\n",
" <td>20169</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>133</th>\n",
" <td>Chugach Electric Association, Inc. ...</td>\n",
" <td>3522</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>138</th>\n",
" <td>City of Lafayette Utilities System ...</td>\n",
" <td>9096</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>139</th>\n",
" <td>City of Tacoma, Dept. of Public Utilities ...</td>\n",
" <td>18429</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>140</th>\n",
" <td>City of Tallahassee ...</td>\n",
" <td>18445</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>142</th>\n",
" <td>Cleco Corporation ...</td>\n",
" <td>3265</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>156</th>\n",
" <td>City of North Little Rock ...</td>\n",
" <td>13718</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>157</th>\n",
" <td>Duke Energy Carolinas, LLC ...</td>\n",
" <td>5416</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>159</th>\n",
" <td>East Kentucky Power Cooperative ...</td>\n",
" <td>5580</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>160</th>\n",
" <td>El Paso Electric Company ...</td>\n",
" <td>5701</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>161</th>\n",
" <td>Electric Energy, Inc. ...</td>\n",
" <td>5748</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>163</th>\n",
" <td>Empire District Electric Company (the) ...</td>\n",
" <td>5860</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>171</th>\n",
" <td>Florida Power &amp; Light Company ...</td>\n",
" <td>6452</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>172</th>\n",
" <td>Gainesville Regional Utilities ...</td>\n",
" <td>6909</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>174</th>\n",
" <td>Grand River Dam Authority ...</td>\n",
" <td>7490</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>178</th>\n",
" <td>Hawaiian Electric Company, Inc ...</td>\n",
" <td>19547</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>180</th>\n",
" <td>Idaho Power Company ...</td>\n",
" <td>9191</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>182</th>\n",
" <td>Imperial Irrigation District ...</td>\n",
" <td>9216</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>186</th>\n",
" <td>JEA ...</td>\n",
" <td>9617</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>187</th>\n",
" <td>Kansas City Board of Public Utilities &amp; Wyando...</td>\n",
" <td>9996</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>193</th>\n",
" <td>Lincoln Electric System ...</td>\n",
" <td>11018</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>194</th>\n",
" <td>Los Angeles Department of Water and Power ...</td>\n",
" <td>11208</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>209</th>\n",
" <td>Nebraska Public Power District ...</td>\n",
" <td>13337</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>210</th>\n",
" <td>Nevada Power Company ...</td>\n",
" <td>13407</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>217</th>\n",
" <td>NorthWestern Energy ...</td>\n",
" <td>12825</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>220</th>\n",
" <td>Oklahoma Gas &amp; Electric Company ...</td>\n",
" <td>14063</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>223</th>\n",
" <td>Omaha Public Power District ...</td>\n",
" <td>14127</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>225</th>\n",
" <td>Orlando Utilities Commission ...</td>\n",
" <td>14610</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>232</th>\n",
" <td>Portland General Electric Company ...</td>\n",
" <td>15248</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>233</th>\n",
" <td>Progress Energy (Carolina Power &amp; Light Compan...</td>\n",
" <td>3046</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>234</th>\n",
" <td>Progress Energy (Florida Power Corp.) ...</td>\n",
" <td>6455</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>235</th>\n",
" <td>Public Service Company of Colorado ...</td>\n",
" <td>15466</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>236</th>\n",
" <td>Public Service Company of New Mexico ...</td>\n",
" <td>15473</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>237</th>\n",
" <td>PUD No. 1 of Chelan County ...</td>\n",
" <td>3413</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>238</th>\n",
" <td>PUD No. 1 of Douglas County ...</td>\n",
" <td>5326</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>239</th>\n",
" <td>PUD No. 2 of Grant County ...</td>\n",
" <td>14624</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>240</th>\n",
" <td>Puget Sound Energy, Inc. ...</td>\n",
" <td>15500</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>243</th>\n",
" <td>Sacramento Municipal Utility District (&amp; City ...</td>\n",
" <td>16534</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>244</th>\n",
" <td>Salt River Project ...</td>\n",
" <td>16572</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>247</th>\n",
" <td>Seattle City Light ...</td>\n",
" <td>16868</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>249</th>\n",
" <td>Sierra Pacific Resources ...</td>\n",
" <td>17166</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>250</th>\n",
" <td>South Carolina Electric &amp; Gas ...</td>\n",
" <td>17539</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>251</th>\n",
" <td>South Carolina Public Service Authority ...</td>\n",
" <td>17543</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>259</th>\n",
" <td>Southwestern Public Service Company (Xcel) ...</td>\n",
" <td>17718</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>262</th>\n",
" <td>Tampa Electric Company ...</td>\n",
" <td>18454</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>266</th>\n",
" <td>Tucson Electric Power Company ...</td>\n",
" <td>24211</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>267</th>\n",
" <td>Turlock Irrigation District ...</td>\n",
" <td>19281</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_name eia_code \\\n",
"respondent_id \n",
"116 Arizona Public Service Company ... 803 \n",
"119 Avista Corporation ... 20169 \n",
"133 Chugach Electric Association, Inc. ... 3522 \n",
"138 City of Lafayette Utilities System ... 9096 \n",
"139 City of Tacoma, Dept. of Public Utilities ... 18429 \n",
"140 City of Tallahassee ... 18445 \n",
"142 Cleco Corporation ... 3265 \n",
"156 City of North Little Rock ... 13718 \n",
"157 Duke Energy Carolinas, LLC ... 5416 \n",
"159 East Kentucky Power Cooperative ... 5580 \n",
"160 El Paso Electric Company ... 5701 \n",
"161 Electric Energy, Inc. ... 5748 \n",
"163 Empire District Electric Company (the) ... 5860 \n",
"171 Florida Power & Light Company ... 6452 \n",
"172 Gainesville Regional Utilities ... 6909 \n",
"174 Grand River Dam Authority ... 7490 \n",
"178 Hawaiian Electric Company, Inc ... 19547 \n",
"180 Idaho Power Company ... 9191 \n",
"182 Imperial Irrigation District ... 9216 \n",
"186 JEA ... 9617 \n",
"187 Kansas City Board of Public Utilities & Wyando... 9996 \n",
"193 Lincoln Electric System ... 11018 \n",
"194 Los Angeles Department of Water and Power ... 11208 \n",
"209 Nebraska Public Power District ... 13337 \n",
"210 Nevada Power Company ... 13407 \n",
"217 NorthWestern Energy ... 12825 \n",
"220 Oklahoma Gas & Electric Company ... 14063 \n",
"223 Omaha Public Power District ... 14127 \n",
"225 Orlando Utilities Commission ... 14610 \n",
"232 Portland General Electric Company ... 15248 \n",
"233 Progress Energy (Carolina Power & Light Compan... 3046 \n",
"234 Progress Energy (Florida Power Corp.) ... 6455 \n",
"235 Public Service Company of Colorado ... 15466 \n",
"236 Public Service Company of New Mexico ... 15473 \n",
"237 PUD No. 1 of Chelan County ... 3413 \n",
"238 PUD No. 1 of Douglas County ... 5326 \n",
"239 PUD No. 2 of Grant County ... 14624 \n",
"240 Puget Sound Energy, Inc. ... 15500 \n",
"243 Sacramento Municipal Utility District (& City ... 16534 \n",
"244 Salt River Project ... 16572 \n",
"247 Seattle City Light ... 16868 \n",
"249 Sierra Pacific Resources ... 17166 \n",
"250 South Carolina Electric & Gas ... 17539 \n",
"251 South Carolina Public Service Authority ... 17543 \n",
"259 Southwestern Public Service Company (Xcel) ... 17718 \n",
"262 Tampa Electric Company ... 18454 \n",
"266 Tucson Electric Power Company ... 24211 \n",
"267 Turlock Irrigation District ... 19281 \n",
"\n",
" utility_match ba_match \n",
"respondent_id \n",
"116 True True \n",
"119 True True \n",
"133 True True \n",
"138 True True \n",
"139 True True \n",
"140 True True \n",
"142 True True \n",
"156 True True \n",
"157 True True \n",
"159 True True \n",
"160 True True \n",
"161 True True \n",
"163 True True \n",
"171 True True \n",
"172 True True \n",
"174 True True \n",
"178 True True \n",
"180 True True \n",
"182 True True \n",
"186 True True \n",
"187 True True \n",
"193 True True \n",
"194 True True \n",
"209 True True \n",
"210 True True \n",
"217 True True \n",
"220 True True \n",
"223 True True \n",
"225 True True \n",
"232 True True \n",
"233 True True \n",
"234 True True \n",
"235 True True \n",
"236 True True \n",
"237 True True \n",
"238 True True \n",
"239 True True \n",
"240 True True \n",
"243 True True \n",
"244 True True \n",
"247 True True \n",
"249 True True \n",
"250 True True \n",
"251 True True \n",
"259 True True \n",
"262 True True \n",
"266 True True \n",
"267 True True "
]
},
"metadata": {
"tags": []
},
"execution_count": 23
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "OLdiLqsUFkCS",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 855
},
"outputId": "756d3525-559f-45cf-c061-dabc7bb0e81b"
},
"source": [
"ferc_eia_map.query(\"utility_match==True & ba_match==False\")"
],
"execution_count": 24,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>respondent_name</th>\n",
" <th>eia_code</th>\n",
" <th>utility_match</th>\n",
" <th>ba_match</th>\n",
" </tr>\n",
" <tr>\n",
" <th>respondent_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>102</th>\n",
" <td>Alabama Power Company ...</td>\n",
" <td>195</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>121</th>\n",
" <td>Black Hills Corporation ...</td>\n",
" <td>19545</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>125</th>\n",
" <td>California Independent System Operator ...</td>\n",
" <td>229</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>135</th>\n",
" <td>City of Burbank ...</td>\n",
" <td>2507</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>141</th>\n",
" <td>City Utilities of Springfield, MO ...</td>\n",
" <td>17833</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>143</th>\n",
" <td>Colorado Springs Utilities ...</td>\n",
" <td>3989</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>150</th>\n",
" <td>Dayton Power &amp; Light Company, The ...</td>\n",
" <td>4922</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151</th>\n",
" <td>Decatur Utilities ...</td>\n",
" <td>4958</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>162</th>\n",
" <td>Electric Power Board of Chattanooga ...</td>\n",
" <td>3408</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>166</th>\n",
" <td>Eugene Water &amp; Electric Board ...</td>\n",
" <td>6022</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>177</th>\n",
" <td>Greenville Utilities Commission ...</td>\n",
" <td>7639</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>191</th>\n",
" <td>Lakeland Electric ...</td>\n",
" <td>10623</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>197</th>\n",
" <td>Louisville Gas &amp; Electric and Kentucky Utiliti...</td>\n",
" <td>11249</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>200</th>\n",
" <td>Memphis Light, Gas and Water ...</td>\n",
" <td>12293</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>206</th>\n",
" <td>Modesto Irrigation District ...</td>\n",
" <td>12745</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>212</th>\n",
" <td>New York State Electric &amp; Gas Corporation ...</td>\n",
" <td>13511</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>241</th>\n",
" <td>Rochester Gas and Electric Corporation ...</td>\n",
" <td>16183</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>246</th>\n",
" <td>San Diego Gas &amp; Electric Company ...</td>\n",
" <td>16609</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>268</th>\n",
" <td>United Illuminating Company ...</td>\n",
" <td>19497</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>280</th>\n",
" <td>Wolverine Power Supply Coop., Inc. ...</td>\n",
" <td>20910</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>282</th>\n",
" <td>Wisconsin Public Power Inc. ...</td>\n",
" <td>20858</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>289</th>\n",
" <td>City of Conway ...</td>\n",
" <td>4280</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>296</th>\n",
" <td>Georgia Power Company ...</td>\n",
" <td>7140</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>297</th>\n",
" <td>Mississippi Power Company ...</td>\n",
" <td>12686</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>298</th>\n",
" <td>Gulf Power Company ...</td>\n",
" <td>7801</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" respondent_name eia_code \\\n",
"respondent_id \n",
"102 Alabama Power Company ... 195 \n",
"121 Black Hills Corporation ... 19545 \n",
"125 California Independent System Operator ... 229 \n",
"135 City of Burbank ... 2507 \n",
"141 City Utilities of Springfield, MO ... 17833 \n",
"143 Colorado Springs Utilities ... 3989 \n",
"150 Dayton Power & Light Company, The ... 4922 \n",
"151 Decatur Utilities ... 4958 \n",
"162 Electric Power Board of Chattanooga ... 3408 \n",
"166 Eugene Water & Electric Board ... 6022 \n",
"177 Greenville Utilities Commission ... 7639 \n",
"191 Lakeland Electric ... 10623 \n",
"197 Louisville Gas & Electric and Kentucky Utiliti... 11249 \n",
"200 Memphis Light, Gas and Water ... 12293 \n",
"206 Modesto Irrigation District ... 12745 \n",
"212 New York State Electric & Gas Corporation ... 13511 \n",
"241 Rochester Gas and Electric Corporation ... 16183 \n",
"246 San Diego Gas & Electric Company ... 16609 \n",
"268 United Illuminating Company ... 19497 \n",
"280 Wolverine Power Supply Coop., Inc. ... 20910 \n",
"282 Wisconsin Public Power Inc. ... 20858 \n",
"289 City of Conway ... 4280 \n",
"296 Georgia Power Company ... 7140 \n",
"297 Mississippi Power Company ... 12686 \n",
"298 Gulf Power Company ... 7801 \n",
"\n",
" utility_match ba_match \n",
"respondent_id \n",
"102 True False \n",
"121 True False \n",
"125 True False \n",
"135 True False \n",
"141 True False \n",
"143 True False \n",
"150 True False \n",
"151 True False \n",
"162 True False \n",
"166 True False \n",
"177 True False \n",
"191 True False \n",
"197 True False \n",
"200 True False \n",
"206 True False \n",
"212 True False \n",
"241 True False \n",
"246 True False \n",
"268 True False \n",
"280 True False \n",
"282 True False \n",
"289 True False \n",
"296 True False \n",
"297 True False \n",
"298 True False "
]
},
"metadata": {
"tags": []
},
"execution_count": 24
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "s6layIR6KJ_C",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 0,
"outputs": []
}
]
}
@truggles
Copy link

This is great work Greg! I think the summary stats from the cleaning look reasonable and inline with what we saw from EIA-930. You find 97.3% of the demand values appear good (looking at output from summary_df.describe()). We find 2.2% of values are missing in the EIA-930 database and 0.5% are anomalous = 97.3% good values.

In your summary stats, you have 0.0% missing, that is impressive.

If you are using the values for creating average profiles, I think this should be fine. We imputed in our work because we need continuous time series for use in models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment