Dat Tran datitran

Spark on Cloud Foundry

This document describes one means of running a simple Apache Spark cluster on Cloud Foundry. It makes heavy use of Cloud Foundry's container networking features.

You can see an example running at http://spark-ui-proxy.184.73.108.92.xip.io.

Deploy BOSH-Lite on AWS

This cluster was deployed using BOSH-Lite on AWS. Note, this Director cannot be targetted with the new BOSH CLI (see cloudfoundry-attic/bosh-lite#424), but you can use the "old" Ruby CLI just fine. You can use the new CLI for local workflows like manifest interpolation, and then the "old" CLI for remote workflows like deploying and SSH.

	#!/bin/sh

	sudo yum -y install autoconf automake gcc gcc-c++ make boost-devel zlib-devel ncurses-devel protobuf-devel openssl-devel
	cd /usr/local/src
	sudo wget http://mosh.mit.edu/mosh-1.2.4.tar.gz
	sudo tar xvf mosh-1.2.4.tar.gz
	cd mosh-1.2.4
	sudo ./autogen.sh
	sudo ./configure
	sudo make

	from pyspark import SparkConf, SparkContext
	from sklearn.datasets import make_classification
	from sklearn.ensemble import ExtraTreesClassifier
	import pandas as pd
	import numpy as np

	conf = (SparkConf()
	.setMaster("local[*]")
	.setAppName("My app")
	.set("spark.executor.memory", "1g"))