Atlas AtlasPilotPuppy
AtlasPilotPuppy
/ whiteman_vehicledata.py
Created
September 16, 2014 20:48
Whiteman data joined with Vehicle data(2008- 2013)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pandas as pd | |
| import numpy as np | |
| import statsmodels.formula.api as sm | |
| import matplotlib.pyplot as plt | |
| from matplotlib.backends.backend_pdf import PdfPages | |
| whiteman_pems = pd.read_csv('whiteman_pems.csv') | |
| whiteman_cleaned = whiteman_pems.fillna(0) |
AtlasPilotPuppy
/ SparkHbaseALS.scala
Last active
August 29, 2015 14:03
Uses values in hbase tables to train and test ALS model in MLib.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.rdd.NewHadoopRDD | |
| import org.apache.hadoop.hbase.mapreduce.TableInputFormat | |
| import org.apache.hadoop.hbase.HBaseConfiguration | |
| import org.apache.hadoop.hbase.client.Result | |
| import org.apache.hadoop.hbase.io.ImmutableBytesWritable | |
| import scala.collection.JavaConversions._ | |
| import scala.collection.JavaConverters._ | |
| import org.apache.spark.mllib.recommendation.ALS | |
| import org.apache.spark.mllib.recommendation.Rating | |
| import scala.collection.mutable.ArrayBuffer |
AtlasPilotPuppy
/ hbase_rdd.scala
Last active
August 3, 2016 14:21
Accessing Hbase from Apache Spark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.rdd.NewHadoopRDD | |
| import org.apache.hadoop.hbase.mapreduce.TableInputFormat | |
| import org.apache.hadoop.hbase.HBaseConfiguration | |
| import org.apache.hadoop.hbase.client.Result | |
| import org.apache.hadoop.hbase.io.ImmutableBytesWritable | |
| import scala.collection.JavaConversions._ | |
| import scala.collection.JavaConverters._ | |
| val sc = new SparkContext("local", "Simple App") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pylab import * | |
| #create the function q(t) based on the question | |
| def q(t): | |
| if t < 0: | |
| return 0 | |
| if t <= 10: | |
| return 5 * t | |
| if t <= 60: | |
| return 60 - t |
NewerOlder