Christopher Matta cjmatta

##Building and using Spark Notebook for MapR

The spark-notebook is a useful browser-based REPL that can be used to explore data and build visualizations. This guide will illustrate the MapR-specific requirements for building, and using the spark notebook on MapR clusters.

###Building Checkout the source:

$ git clone https://github.com/andypetrella/spark-notebook.git

Download json:

http://reddit.com/r/all.json?sort=hot&limit=100

Save to directories named after unix timestamp.

Create View

CREATE OR REPLACE VIEW reddit_view AS SELECT 
 `id`,

Getting weather data

Use this python script to download weather data: https://github.com/cjmatta/wundergroundloader

Drill config

Ensure that the csv type for drill has "extractHeader": true, set in the filesystem plugin, and that you haven't used the --strip-headers option in the wundergroundloader script.

Create View

CREATE OR REPLACE VIEW maprfs.cmatta.`weather_view` AS
SELECT`dir0` AS `airportcode`,

Philly crime database - Drill Exploration

source: https://www.opendataphilly.org/dataset/philadelphia-police-part-one-crime-incidents

Download json files and need to do small transforms:

$ mkdir phillycrime
$ cd phillycrime
$ mkdir 2006

Keybase proof

I hereby claim:

I am cjmatta on github.
I am cmatta (https://keybase.io/cmatta) on keybase.
I have a public key whose fingerprint is C469 2213 66F0 371D CD16 0AE2 9885 1CEF B447 06DB

To claim this, I am signing this object:

Drill Demo - MapR

Retail data

Show logs flat JSON:

head -n 5 /mapr/*/data/flat/logs/2012/1/log.json

Show how it can be queried:

select * from mfs.flat.`logs` limit 10;

	import pyodbc
	import re

	# make sure the Drill ODBC driver is installed
	# this is for Mac
	MY_DSN = """
	Driver = /opt/mapr/drillodbc/lib/universal/libmaprdrillodbc.dylib
	ConnectionType = Zookeeper
	ZKQuorum = node10:5181,node11:5181,node12:5181
	ZKClusterID = se1-drillbits

	#!/bin/bash
	set -o nounset
	set -o errexit

	if [[ ! -f $1 ]];
	then
	echo "Query file ${1} not found, exiting.";
	exit 1;
	fi

	#!/bin/bash
	# Copyright (C) 2013 by Teradata Corporation.
	# All Rights Reserved.
	#
	# This script installs tdch for Oozie transfers with Hadoop
	#
	# Version : $Id$

	# MapR Notes
	# Since MapR doesn't need a nameNode, we've removed it from this script