Daniel daniel-cortez-stevenson

AWS EMR bootstrap to install R packages from CRAN

This bootstrap is useful if you want to deploy SparkR applications that run arbitrary code on the EMR cluster's workers. The R code will need to have its dependencies already installed on each of the workers, and will fail otherwise. This is the case if you use functions such as gapply or dapply.

How to use the bootstrap

You will first have to download the gist to a file and then upload it to S3 in a bucket of your choice.
Using the AWS EMR Console create a cluster and choose advanced options.
In Step 3 you can configure your bootstraps. Choose to Configure and add a Custom action

	#!/usr/bin/env bash
	set -x -e

	JUPYTER_PASSWORD=${1:-"myJupyterPassword"}
	NOTEBOOK_DIR=${2:-"s3://myS3Bucket/notebooks/"}

	# home backup
	if [ ! -d /mnt/home_backup ]; then
	sudo mkdir /mnt/home_backup
	sudo cp -a /home/* /mnt/home_backup

	#!/bin/bash

	# These variables can be overwritten using the arguments below
	VERSION="1.1.463"
	# drwho is listed as user in YARN's Resource Manager UI.
	USER="drwho"
	# Depending on where the EMR cluster lives, you might have to change this to avoid security issues.
	# To change the default password (and user), use the arguments bellow.
	# If the cluster is not visible on the Internet, you can just leave the defaults for convenience.
	PASS="tardis"

	sudo: required #is required to use docker service in travis

	language: php #can be any language, just php for example

	services:
	- docker # required, but travis uses older version of docker :(

	install:
	- echo "install nothing!" # put your normal pre-testing installs here

	// Show how to handle Ctrl+C in Node.js

	var zmq = require('zmq')
	, socket = zmq.createSocket('rep');

	socket.on('message', function(buf) {
	// echo request back
	socket.send(buf);
	});

	docker logs nginx 2>&1 \| grep "127."

	# ref: http://stackoverflow.com/questions/34724980/finding-a-string-in-docker-logs-of-container