John Robenalt robenalt

Building Spark for PySpark use on top of YARN

Build Spark on local machine (only if using PySpark; otherwise, remote machine works) (http://spark.apache.org/docs/latest/building-with-maven.html)

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

Copy the assembly/target/scala-2.10/...jar to the corresponding directory on the cluster node and also into a location in HDFS.

AWS S3 bucket and IAM policy recipes

Recipes

People

`:bowtie:`	😄 `:smile:`	😆 `:laughing:`
😊 `:blush:`	😃 `:smiley:`	☺️ `:relaxed:`
😏 `:smirk:`	😍 `:heart_eyes:`	😘 `:kissing_heart:`
😚 `:kissing_closed_eyes:`	😳 `:flushed:`	😌 `:relieved:`
😆 `:satisfied:`	😁 `:grin:`	😉 `:wink:`
😜 `:stuck_out_tongue_winking_eye:`	😝 `:stuck_out_tongue_closed_eyes:`	😀 `:grinning:`
😗 `:kissing:`	😙 `:kissing_smiling_eyes:`	😛 `:stuck_out_tongue:`

	package example

	import org.apache.avro.Schema.Parser
	import java.io.{DataInput, DataOutput, File}
	import org.apache.avro.generic.GenericData.Record
	import org.apache.avro.generic.{GenericRecord, GenericDatumWriter}
	import org.apache.avro.file.DataFileWriter
	import org.apache.spark.SparkContext
	import org.apache.avro.mapreduce.AvroKeyInputFormat
	import org.apache.avro.mapred.AvroKey

	#!/bin/bash

	version=1.0.1
	versionDate="2014-02-14"

	function showHelp() {
	echo "watchfile - monitor file(s)/command and perform action when changed

	Possible ways of usage
	----------------------------------------

	#!/usr/bin/env python
	# -- coding: utf-8 --

	'''

	This script just show the basic workflow to compute TF-IDF similarity matrix with Gensim


	OUTPUT :

	#!/bin/bash
	#
	# convert-video.sh
	#
	# Copyright (c) 2013-2014 Don Melton
	#

	about() {
	cat <<EOF
	$program 2.0 of December 3, 2014

	#!/bin/bash
	#
	# detect-crop.sh
	#
	# Copyright (c) 2013-2015 Don Melton
	#

	about() {
	cat <<EOF
	$program 3.3 of January 22, 2015

	#!/bin/bash
	#
	# transcode-video.sh
	#
	# Copyright (c) 2013-2015 Don Melton
	#

	about() {
	cat <<EOF
	$program 5.13 of April 8, 2015

	Getting afni working on Mac OSX 10.9 Mavericks, using Homebrew

	Briefly: parts of afni (e.g. uber_subject.py) require the pyqt library ... and pyqt requires qt ... but the homebrew install of qt is currently broken on osx mavericks 10.9 ... but there is a patch ... but pyqt also requires sip ... and there is a problem where pyqt doesn't like the version of sip, in particular with a newer haswell CPU macbook pro ... but there is a fix ... and we need to add lines to our .bash_profile for afni ... and be careful if you have other python things installed since afni expects to find python things in certain locations

	Here is a step-by-step:

	1. install Xcode command line tools

	xcode-select --install