aseigneurin

object Conway extends App {

  var grid = Grid(3, 3)

  while (true) {
    grid.print
    grid = grid.nextGrid
    Thread.sleep(2000)
  }

aseigneurin

Set up the VM with automating tools

• Use Vagrant
• Use Ansible

Properly export the VM

Shrink the disk (from https://www.maketecheasier.com/shrink-your-virtualbox-vm/):

• Reboot in Recovery mode (hold the Shift key while the VM is booting)

aseigneurin

// http://www.cs.princeton.edu/~chazelle/courses/BIB/pagerank.htm
object MainIanRogers {

  case class Edge(src: Int, dest: Int)

  def main(args: Array[String]) {

    val edges = Seq(
      Edge(0, 1),
      Edge(0, 2),

aseigneurin

case class PendingResult(k1: Long, k2: String, futureResults: ResultSetFuture)

val pendingResults = ArrayBuffer.empty[PendingResult]
for (i <- 1 to iterations) {
  val k1 = ...
  val k2 = ...
  val futureResults = session.executeAsync(s"SELECT * FROM ${tableName} WHERE k1=${k1} AND k2='${k2}'")
  pendingResults += PendingResult(k1, k2, futureResults)
}

aseigneurin

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.sample</groupId>
    <artifactId>Spark_Kafka_Streaming</artifactId>
    <packaging>jar</packaging>
    <version>0.0.1-SNAPSHOT</version>

    <properties>

aseigneurin

#!/bin/bash -e

if [ ! -d data/wikipedia-pagecounts-hours ]; then
  mkdir -p data/wikipedia-pagecounts-hours
fi
cd data/wikipedia-pagecounts-hours

yyyy=2014
MM=06
dd=19

aseigneurin

Spark - High availability

Components in play

As a reminder, here are the components in play to run an application:

• The cluster:
  • Spark Master: coordinates the resources
  • Spark Workers: offer resources to run the applications
• The application:

aseigneurin

Spark - File formats and storage options

In this document, I'm using a data file containing 40 million records. The file is a text file with one record per line.

The following Scala code is run in a spark-shell:

val filename = "<path to the file>"
val file = sc.textFile(filename)
file.count()

aseigneurin

Spark - Parquet files

Basic file formats - such as CSV, JSON or other text formats - can be useful when exchanging data between applications. When it comes to storing intermediate data between steps of an application, Parquet can provide more advanced capabilities:

• Support for complex types, as opposed to string-based types (CSV) or a limited type system (JSON only supports strings, basic numbers, booleans).
• Columnar storage - more efficient when not all the columns are used or when filtering the data.
• Partitioning - files are partitioned out of the box
• Compression - pages can be compressed with Snappy or Gzip (this preserves the partitioning)

The tests here are performed with Spark 2.0.1 on a cluster with 3 workers (c4.4xlarge, 16 vCPU and 30 GB each).

aseigneurin

PROMPT=$'%{$fg_bold[red]%}%D{%K:%M:%S}%{$reset_color%} %{$fg[cyan]%}%n%{$fg[grey]%}@%{$fg[green]%}%M%{$fg[grey]%}:%{$fg_bold[yellow]%}%d%{$fg[grey]%}$(git_prompt_info) $ %{$reset_color%}'

ZSH_THEME_GIT_PROMPT_PREFIX=" %{$fg_bold[white]%}git:("
ZSH_THEME_GIT_PROMPT_SUFFIX="%{$fg[white]%})%{$reset_color%}"
ZSH_THEME_GIT_PROMPT_DIRTY="%{$fg[red]%}*"
ZSH_THEME_GIT_PROMPT_CLEAN=""