林煒清(Lin Wei-Ching) superChing

Experimentation on Spark's SaveMode

Experiment on the effect of different SaveMode and Cassandra starting from a populated table

Summary

If the cassandra table that spark targets exists then

SaveMode.Append will update it
SaveMode.Overwrite will truncate and insert (but it requires option "confirm.truncate" -> "true")
SaveMode.Ignore will not perform any action on existing table
SaveMode.ErrorIfExists (default) will throw the following exception: https://github.com/datastax/spark-cassandra-connector/blob/v2.0.6/spark-cassandra-connector/src/main/scala/org/apache/spark/sql/cassandra/DefaultSource.scala#L93-L96

Build And Deploy Docker Containers

Linux Alpine docker image is based on super tiny Linux kernel BusyBox and features and package manager, apk, making easy to install build dependencies. This is particularly important for static compile languages like haskell and go as example but also for java and scala.

Scala is also a statically compiled language for the jvm and although we think of it as requiring just the jdk the typical way to build a scala application is with sbt which requires the jvm but the build also downloads a considerable number of sbt core dependencies. Hence, the safe and reliable way for reproducing build a project anywhere is to start from equal sbt installation. The best way to ensure that is to start from a pre-loaded sbt docker image.

Ideally we shouldn't deploy the build system but just the artifact in a reliable and reproducible way. Here, I'll describe two ways to do just that.

Sbt Base Image

Install PostgreSQL 10 on Ubuntu

This is a quick guide to install PostgreSQL 10 - tested on Ubuntu 16.04 but likely can be used for Ubuntu 14.04 and 17.04 as well, with one minor modification detailed below.

(Optional) Uninstall other versions of postgres

To make life simple, remove all other versions of Postgres. Obviously not required, but again, makes life simple. If you have data in your previous version of postgres that you'd like to retain, then this is not recommended. Instead, you'll have to use pg_upgrade or pg_upgradecluster.

Thread Pools

Thread pools on the JVM should usually be divided into the following three categories:

CPU-bound
Blocking IO
Non-blocking IO polling

Each of these categories has a different optimal configuration and usage pattern.

If you want to learn Scala, I’d suggest:

Install IntelliJ + Scala Plugin
Don’t do the Coursera courses yet.
Don’t do the “red book” Functional Programming in Scala yet.
Do: http://underscore.io/books/
- Essential Scala
- Essential Play
- Essential Slick
Do Scala for the Impatient: https://www.amazon.com/Scala-Impatient-Cay-S-Horstmann/dp/0321774094
The Neophyte’s Guide to Scala http://danielwestheide.com/scala/neophytes.html

Kubectl output options

Let's look at some basic kubectl output options.

Our intention is to list nodes (with their AWS InstanceId) and Pods (sorted by node).

We can start with:

kubectl get no

	* Cluster Setup for Google Container Engine *

	0/ Install and configure local gcloud and kubectl: https://cloud.google.com/sdk/docs/
	> gcloud components install kubectl

	1/ Configure Google Cloud account:
	> gcloud config set account YOUR_EMAIL_ADDRESS
	> gcloud config set project YOUR_PROJECT_ID
	> gcloud config set compute/zone us-west1-a
	> gcloud config set container/cluster example

	/*
	There are times when you need to find out of the IP address your domain maps to in DNS is the same
	as the IP address of your web server's outgoing interface. This is particularly useful when dealing
	with APIs such as eNOM. The following just performs a simple query agains http://ipecho.net/plain to
	optain your server's outgoing IP. This can be modified with any number of similar services.
	*/

	$realIP = file_get_contents("http://ipecho.net/plain");
	echo $realIP;

	package com.google.functions.examples.scalajs

	import scala.scalajs.js.annotation._
	import scala.scalajs.js
	import js.DynamicImplicits._

	@js.native
	trait Request extends js.Any {
	val body: js.Dictionary[Any] = js.native
	}

	"""
	Add mypy type-checking cell magic to jupyter/ipython.

	Save this script to your ipython profile's startup directory.

	IPython's directories can be found via `ipython locate [profile]` to find the current ipython directory and ipython profile directory, respectively.

	For example, this file could exist on a path like this on mac:

	/Users/yourusername/.ipython/profile_default/startup/typecheck.py