Sean Roberts seanorama

PySpark on YARN in self-contained environments

Author: https://github.com/seanorama

Note: This was tested on HDP 3.1. It may not work with other Spark/YARN distributions.

Reference:

This is an example of how to use Terraform AWS registry modules with Terragrunt.

Notes:

source has to be full git URL and not Terraform Registry open issue #311
File main_providers.tf is named so, because it will be copied to another local directory and merged with module's code. If such file exists in the module already then it will overwrite the one provided by the module.

Mac OS X 10.10 Yosemite

Custom recipe to get OS X 10.10 Yosemite running from scratch, setup applications and developer environment. I use this gist to keep track of the important software and steps required to have a functioning system after a semi-annual fresh install. On average, I reinstall each computer from scratch every 6 months, and I do not perform upgrades between distros.

This keeps the system performing at top speeds, clean of trojans, spyware, and ensures that I maintain good organizational practices for my content and backups. I highly recommend this.

You are encouraged to fork this and modify it to your heart's content to match your own needs.

Install Software

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

-E: will describe the underlaying queries of the \ commands (cool for learning!)
-l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

	### variables

	variable "env" {}
	variable "riak_count" {}
	variable "elastic_count" {}

	### hostname.tpl
	${name}-${env}-${format("%02s",index)} ${extra}

	### ansible.tpl

	# Script to install HDP 2.5, Nifi 1.0, Zeppein with Truckin demo and Nifi-Twitter demo. Run below command to kick off:
	#curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw \| sudo -E sh

	#To run on multinode: run below on non-ambariserver nodes first
	#export ambari_server=<FQDN of host running ambari-server>; export ambari_version=2.4.1.0; curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh \| sudo -E sh ;

	bootstrap_home=/tmp
	rm -rf ${bootstrap_home}/ambari-bootstrap
	set -e -x
	export hdp_ver=${hdp_ver:-2.5} #version of HDP

	#!/bin/sh
	# This script checks if the current branch needs to pull, push, if it has
	# diverged, or if it is up-to-date with the remote branch.

	# Credits to: http://stackoverflow.com/questions/3258243/check-if-pull-needed-in-git

	# Check if git is installed.
	hash git 2>/dev/null \|\| { echo >&2 "This script requires git to be installed."; exit 1; }

	# Update remote branch to fetch latest remote commit SHA.

	# See https://pip.pypa.io/en/latest/installing.html for how to install pip, then:
	# pip install simple-salesforce
	from simple_salesforce import Salesforce
	import string

	# To get your SFDC token, see https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
	# salesforce_login.txt should contain the following, one per line:
	# Your Name
	# your_email@sfdc_account.com
	# your_password

	FROM dockerfile/nodejs
	MAINTAINER Marvin

	WORKDIR /root
	RUN npm install -g yo generator-hubot

	RUN useradd -ms /bin/bash marvin
	ENV HOME /home/marvin
	#Place here variables needed by hubot scripts
	ADD env-vars.sh /home/marvin/.profile

	USER='admin'
	PASS='admin'
	CLUSTER='dev'
	HOST=$(hostname -f):8080

	function start(){
	curl -u $USER:$PASS -i -H 'X-Requested-By: ambari' -X PUT -d \
	'{"RequestInfo": {"context" :"Start '"$1"' via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' \
	http://$HOST/api/v1/clusters/$CLUSTER/services/$1
	}