Timothy Spann tspannhw

LAB

This lab is part of a 'Sql on Hadoop' webinar. The recording and slides can be found here

Purpose

How/when to use Hive vs Phoenix vs SparkSQL

Hbase indexing to solr in HDP 2.3

Background:

The HBase Indexer provides the ability to stream events from HBase to Solr for near real time searching. The HBase indexer is included with HDPSearch as an additional service. The indexer works by acting as an HBase replication sink. As updates are written to HBase, the events are asynchronously replicated to the HBase Indexer processes, which in turn creates Solr documents and pushes them to Solr.

References:
- http://doc.lucidworks.com/hdpsearch23/JobJarGuide.html#_hbase_indexer
- https://github.com/NGDATA/hbase-indexer/wiki/Tutorial

Phoenix/Spark demo

These demo is part of a webinar on 'HBase for Mission Critical Applications'
The webinar recording /slides are available at http://hortonworks.com/partners/learn/#hbase2
The Ambari service for OpenTSDB is available at https://github.com/abajwa-hw/opentsdb-service
The full list of available workshops is available at http://hortonworks.com/partners/learn

Option 1: prebuilt VM

There is a prebuilt Centos 6.5 VM with the below components installed:

HDP 2.3.0.0-1754
Spark 1.3.1

Future of Big Data: Philadelphia

These are notes for following along on the talk I am giving.

This builds on the gist: https://gist.github.com/epugh/5729071c3b8aab81636d422c391aa716, but is meant to be stand alone!

This gist is using the latest version of Zeppelin. Replace the ip address 192.168.99.100 with the your docker machine ip. Get it by running docker-machine ip.
Fire up Zeppelin + Spark Master and a Spark Worker via: docker run -d --name zeppelin -p 8080:8080 dylanmei/zeppelin
If it doesnt' work, go back to the specific "stable" version of Zeppelin. There is a 1 GB layer in there, watch out!

	#Install R and then following packages
	#repr failed to create
	yum install R-*
	install.packages("evaluate", dependencies = TRUE)
	install.packages("base64enc", dependencies = TRUE)
	install.packages("devtools", dependencies = TRUE)
	install_github('IRkernel/repr')
	install.packages("dplyr", dependencies = TRUE)
	install.packages("caret", dependencies = TRUE)
	install.packages("repr", dependencies = TRUE)

	<!DOCTYPE html>
	<meta charset="utf-8">
	<link rel="stylesheet" href="http://cmx.io/v/0.1/cmx.css">
	<script src="http://cmx.io/v/0.1/cmx.js" charset="utf-8"></script>
	<style>.cmx-user-scene4 .cmx-text-border .cmx-path {stroke: orange}</style>
	<body>
	<div style="max-width:900px; -webkit-transform:rotate(0deg)">

	<scene id="scene1">
	<label t="translate(0,346)">


	import java.nio.charset.StandardCharsets
	import org.apache.commons.io.IOUtils
	import org.apache.nifi.processor.io.StreamCallback
	import groovy.transform.ToString
	import com.vader.SentimentAnalyzer
	import groovy.json.JsonBuilder
	import groovy.json.JsonOutput
	import groovy.json.JsonSlurper

	Flow Controller:
	name: MiNiFi S2S temp and humidity
	comment: ''
	Core Properties:
	flow controller graceful shutdown period: 10 sec
	flow service write delay interval: 500 ms
	administrative yield duration: 30 sec
	bored yield duration: 10 millis
	max concurrent threads: 1
	FlowFile Repository: