Nazarii nmeln

Applied Functional Programming with Scala - Notes

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x

Standalone Spark 2.0.0 with s3

###Tested with:

Spark 2.0.0 pre-built for Hadoop 2.7
Mac OS X 10.11
Python 3.5.2

Goal

Use s3 within pyspark with minimal hassle.

Intro

TCL-Expect scripts are an amazingly easy way to script out laborious tasks in the shell when you need to be interactive with the console. Think of them as a "macro" or way to programmaticly step through a process you would run by hand. They are similar to shell scripts but utilize the .tcl extension and a different #! call.

Setup Your Script

The first step, similar to writing a bash script, is to tell the script what it's executing under. For expect we use the following:

#!/usr/bin/expect

	/**
	* Pretty prints a Scala value similar to its source represention.
	* Particularly useful for case classes.
	* @param a - The value to pretty print.
	* @param indentSize - Number of spaces for each indent.
	* @param maxElementWidth - Largest element size before wrapping.
	* @param depth - Initial depth to pretty print indents.
	* @return
	*/
	private def prettyPrint(a: Any, indentSize: Int = 2, maxElementWidth: Int = 30, depth: Int = 0): String = {