monocongo

Configure Python development environment

Install pip, virtualenv, and virtualenvwrapper

$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python3 get-pip.py
$ rm get-pip.py
$ sudo pip install virtualenv virtualenvwrapper

Update .bashrc

monocongo

Create and activate a new conda environment:

$ conda create -n myenv python=3 --yes
$ conda activate myenv

Add the conda-forge channel into the conda configuration:

$ conda config --add channels conda-forge

Add packages into the environment using both conda and pip:

monocongo

27 INFO: PyInstaller: 3.4
27 INFO: Python: 3.6.8
28 INFO: Platform: Linux-4.18.0-22-generic-x86_64-with-Ubuntu-18.04-bionic
28 INFO: wrote /home/james/git/deep_monitor/deep_monitor.spec
28 DEBUG: Testing for UPX ...
29 INFO: UPX is not available.
30 DEBUG: script: /home/james/git/deep_monitor/deep_monitor.py
30 INFO: Extending PYTHONPATH with paths
['/home/james/git/deep_monitor', '/home/james/git/deep_monitor']
30 INFO: checking Analysis

monocongo

Find the Docker container ID (from localhost bash):

$ export DOCKER_CONTAINER=`sudo docker ps -a --no-trunc -q`

If more than a single Docker container is running then get the PyTorch container ID by grepping pytorch from the docker ps -a output, then using the final value of the output (the container's name) we can get the container ID via docker inspect:

$ sudo docker ps -a | grep pytorch

monocongo

import pandas as pd

def transform_to_supervised(df,
                            previous_steps=1, 
                            forecast_steps=1,
                            dropnan=True):

    """
    Transforms a DataFrame containing time series data into a DataFrame
    containing data suitable for use as a supervised learning problem.

monocongo

"""## Pull data files from Google Drive

Install PyDrive which will be used to access Google Drive and kick off the process to authorize the notebook running in the Google Colaboratory environment to touch our Drive files. When this cell executes it'll provide a link to authenticate into a Google Drive account to instatiate a PyDrive client. The Drive account that is selected should be one which has access to our all variables dataset file that we'll use for training/testing our model.
"""

#!pip install -U -q PyDrive
 
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth

monocongo

import numba
import numpy as np

@numba.jit
def _pdsi_from_zindex(Z):

    ## INITIALIZE PDSI AND PHDI CALCULATIONS
    
    # V is the sum of the Uw (Ud) values for the current and previous months of an
    # established dry (wet) spell and is used in calculating the Pe value for a month.

monocongo

import numba
import numpy as np

@numba.jit
def _pdsi_from_zindex(Z):

    ## INITIALIZE PDSI AND PHDI CALCULATIONS
    
    # V is the sum of the Uw (Ud) values for the current and previous months of an
    # established dry (wet) spell and is used in calculating the Pe value for a month.

monocongo

    # for each file in the list we concatenate it to the base dataset
    output_file_name = root_file_name + '.v7_plus_first_guess.nc'
    with xarray.open_dataset(base_file_name) as output_dataset:
        
        # loop over the first guess files, concatenating (appending?) each to the base (output) dataset
        for first_guess_file in first_guess_files:
    
            # open the first guess dataset, concatenate it with the base (output) dataset 
            with xarray.open_dataset(first_guess_file, decode_times=False) as first_guess_dataset:
                output_dataset = xarray.concat([output_dataset.precip[:, :, :], first_guess_dataset.p[0]], dim='time, lat, lon')

monocongo

from __future__ import division
import logging
import numpy as np
import sys
import xarray

# set up a basic, global logger
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)s %(message)s',
                    datefmt='%Y-%m-%d  %H:%M:%S')