Francisco Lopez fran0x

What this?

So one of the painful points of using docker on OS X is that you need to run a virtualbox VM, which often suffers from performance issues. With xhyve, a OS X virtualization system, and docker-machine-xhyve you can now have docker use the native OS X hypervisor to run containers.

No more dealing with virtualbox shenanigans!

In this script, I've also set up a way to autoconfigure terminal sessions to load docker's environment vars (dependent on docker-machine) so you do not have to run eval $(docker-machine env whatever) every time you open a new terminal window.

Requirements

brew install python3

pip install cookiecutter # command-line utility to create projects from cookiecutters / project templates
pip install numpy # vectors and matrices in Python
pip install scipy # other scientific computing tools
pip install pandas # dataframes in Python!
pip install jupyter # notebook

Cheat Sheet SBT

To install sbt in OS X run brew install sbt (requires the almighty Homebrew installed first).

Basic commands are the following:

Command	Action
Deletes all generated files (in the `target` directory)	`clean`
Compiles the main sources (in `src/main/scala and src/main/java` directories)	`compile`

Action	Command
Show Screen	Fn + F11
Spotlight Search	Command (⌘) + Space bar
Generate	Command (⌘) + N
Go to	Command (⌘) + O
Take Screenshot	Command (⌘) + Shift + 4
Forward Delete	Fn + Delete

To encrypt: openssl -e des3 -salt -in secrets.txt -out secrets.xxx

Cheat Sheet IntelliJ IDEA CE

To install IntelliJ IDEA CE in OS X run brew cask install intellij-idea-ce (requires the almighty Homebrew installed first, and then tap caskroom/versions).

Action	Command
Refactor	`Ctrl + Shift + Alt + T`
Rename Refactor	`Shift + F6`
Find Action	`Command + Shift + d`
Find Class	`Ctrl + N`

How much of machine learning is statistics and vice versa?

Learning using https://www.coursera.org/learn/machine-learning/home/welcome

machine learning = teaching a computer to learn concepts using data — without being explicitly programmed.
Supervised learning = "right answers" given
Regression problem
- continuous valued output
- deduce the function for a given data set and predict other values
"in regression problems, we are taking input variables and trying to map the output onto a continuous expected result function."

Introducting Apache Spark

What use cases are a good fit for Apache Spark? How to work with Spark?
- create RDDs, transform them, and execute actions to get result of a computation
- All computations in memory = "memory is cheap" (we do need enough of memory to fit all the data in)
  - the less disk operations, the faster (you do know it, don't you?)
- You develop such computation flows or pipelines using a programming language - Scala, Python or Java <-- that's where ability to write code is paramount
- Data is usually on a distributed file system like Hadoop HDFS or NoSQL databases like Cassandra
- Data mining = analysis / insights / analytics
log mining

From http://stackoverflow.com/a/32393044/1305344:

object size extends App {
  (1 to 1000000).map(i => ("foo"+i, ()))
  val input = readLine("prompt> ")
}

Run it with sbt 'runMain size' and then use jps (to know the pids), jstat -gc pid (to query for gc) and jmap (similar to jstat) to analise resource allocation.

From https://github.com/spark-jobserver/spark-jobserver#getting-started-with-spark-job-server:

The easiest way to get started is to try the Docker container which prepackages a Spark distribution with the job server and lets you start and deploy it.

➜  spark-jobserver git:(master) docker-machine version
docker-machine version 0.7.0, build a650a40

// https://gist.github.com/radekg/ec5a1575c450a48e5cba

	import akka.actor.ActorSystem
	import akka.http.scaladsl.Http
	import akka.http.scaladsl.model.{StatusCodes, _}
	import akka.http.scaladsl.unmarshalling.Unmarshal
	import akka.stream.ActorMaterializer

	import scala.concurrent.Future
	import scala.util.{Failure, Success, Try}

	object Http extends App {