hughdbrown

(data2) C:\Users\hughdbrown>pip install graphlab-create==1.8
Collecting graphlab-create==1.8
  Could not find a version that satisfies the requirement graphlab-create==1.8 (from versions: 2.1)
No matching distribution found for graphlab-create==1.8

(data2) C:\Users\hughdbrown>pip install graphlab-create==1.8.3
Collecting graphlab-create==1.8.3
  Could not find a version that satisfies the requirement graphlab-create==1.8.3 (from versions: 2.1)
No matching distribution found for graphlab-create==1.8.3

hughdbrown

#!/usr/bin/env bash -e

export image_version="2.0.1"
export image_name="datafellas/distributed-pipeline-quotes:${image_version}"

sudo docker pull ${image_name}

sudo docker run --rm -it \
    --memory=8g \
    --cpuset-cpus="0-3" \

hughdbrown

def partial_soln(solns, coins, target):
    solns[target] = []
    for i in coins:
        x = target - i
        if x in solns:
            for xx in solns[x]:
                new_item = tuple(sorted(list(xx) + [i]))
                solns[target].append(new_item)

def solution(coins, target):

hughdbrown

Suppose you have these line-item and snapshot/lookup tables that you want to relate:

Contracts

ID,Type,Date,Amt
1,PO,1/20,$100
1,PO,2/20,$200
1,PO,2/23,$300
2,PO,2/10,$1000
2,PO,2/15,$2000

hughdbrown

# An occasional python interview question I have seen is:
#   "How would you make this series unique while preserving order?"
# The standard code looks like this:

def unique2(series):
    result = []
    seen = set()
    for s in series:
        if s not in seen:
            seen.add(s)

hughdbrown

def cartesian(lol):
    if not lol:
        yield []
    else:
        left, right = lol[0], lol[1:]
        for item in left:
            for result in cartesian(right):
                yield [item] + result

>>> data = [

hughdbrown

from itertools import tee
from random import shuffle
from datetime import datetime


def issorted(series):
    s1, s2 = tee(series)
    next(s2)
    return all(elem1 <= elem2 for elem1, elem2 in zip(s1, s2))

hughdbrown

• http://sparktutorials.net/spark-clusters-on-aws-ec2---reading-and-writing-s3-data---predicting-flight-delays-with-spark-part-1
• https://golog.co/blog/article/Setting_up_Apache_Spark_and_IPython
• http://www.slideshare.net/jonathandinu/the-data-scientists-guide-to-apache-spark

hughdbrown

import numpy as np

def downsample(data, labels):
    """
    >>> data = np.arange(100)
    >>> label = np.array([1] * 95 + [0] * 5)
    >>> print downsample(data, label)
    """
    zero_index = np.array([i for i, val in enumerate(labels) if val == 0])
    one_index = np.array([i for i, val in enumerate(labels) if val == 1])

hughdbrown

Initialize spark in python

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local").setAppName("My App")
sc = SparkContext(conf=conf)

Load data