ddbs

Requirements

• no upfront installation/agents on remote/slave machines - ssh should be enough
• application components should use third-party software, e.g. HDFS, Spark's cluster, deployed separately
• configuration templating
• environment requires/asserts, i.e. we need a JVM in a given version before doing deployment
• deployment process run from Jenkins

Solution

ddbs

# Install SSH server
sudo apt-get install openssh-server

# Update CRAN mirror for R
echo "deb https://cran.rstudio.com/bin/linux/ubuntu trusty/" | sudo tee -a /etc/apt/sources.list

## Secure CRAN mirror and upgrade
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
sudo apt-get update && sudo apt-get -y upgrade

ddbs

## Comparing execution times between OpenCPU (RApache) and FastRWeb (Rserve and HTTP).
## Same requests and same code. Note that FastRWeb is up to 13 times faster on the 2nd call.

# OpenCPU: summarize 2 layers across districts for the whole of SSA, return json of summary table
# and then json of all pixel values for mapping
$ time curl http://127.0.0.1/ocpu/library/hcapi3/R/getLayer/json \
-d '{"var" : ["whea_h", "AEZ16_CLAS"], "by" : "ADM2_NAME_ALT"}' -X POST -H "Content-Type:application/json"

[...]
    {

ddbs

# ipak function: install and load multiple R packages.
# check to see if packages are installed. Install them if they are not, then load them into the R session.

ipak <- function(pkg){
    new.pkg <- pkg[!(pkg %in% installed.packages()[, "Package"])]
    if (length(new.pkg)) 
        install.packages(new.pkg, dependencies = TRUE)
    sapply(pkg, require, character.only = TRUE)
}

ddbs

# Instructions to build a Data Science VM based on Ubuntu Xenial

######################################
# Pre-requisites
######################################

# Install Virtual Box 5.1+
# https://www.virtualbox.org/

# Install Ubuntu 16.04.1 LTS on VirtualBox

ddbs

require(caret)

#load some data
data(USArrests)

### Prepare Data (postive observations)
# add a column to be the strata.  In this case it is states, it can be sites, or other locations
# the original data has 50 rows, so this adds a state label to 10 consecutive observations
USArrests$state <- c(rep(c("PA","MD","DE","NY","NJ"), each = 5))
# this replaces the existing rownames (states) with a simple numerical index

ddbs

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=merged.pdf *.pdf

ddbs

Create a directory for the project

mkdir flask_hello_world

Move into the project directory

cd flask_hello_world

Create a new virtualenv

ddbs

// Use Gists to store code you would like to remember later on
console.log(window); // log the "window" object to the console

ddbs

Python & SQLAlchemy

Sessions

To get started, create a new project in a directory, and cd into it. Then initialize and activate a virtual environment:

sudo apt-get update
sudo apt-get install python3.4-venv
python3 -m venv env
source env/bin/activate