bradkarels’s gists

bradkarels / kafkaCmds.txt

Last active December 2, 2015 18:50

Crude notes on setting up a kafka cluster node with embedded file notes.

	$ wget http://apache.mirrors.tds.net/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
	$ apt-get install -y software-properties-common unzip
	$ add-apt-repository ppa:openjdk-r/ppa
	$ apt-get update
	$ apt-get install -y openjdk-8-jdk
	$ wget http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.tgz
	$ tar xzf scala-2.11.7.tgz
	$ mkdir -p /usr/local/share/scala && mv scala-2.11.7 /usr/local/share/scala
	$ touch /etc/profile.d/scalahome.sh
	$ vim /etc/profile.d/scalahome.sh

bradkarels / zkCmds.txt

Created December 1, 2015 22:06

Setting up a zookeeper cluster - crude commands with embedded file notes

	$ wget http://www.trieuvan.com/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz
	$ tar xvf zookeeper-3.4.6.tar.gz
	$ cd zookeeper-3.4.6/
	$ apt-get install -y software-properties-common unzip
	$ add-apt-repository ppa:openjdk-r/ppa
	$ apt-get update
	$ apt-get install -y openjdk-8-jdk
	$ touch /etc/profile.d/javahome.sh
	$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/" > /etc/profile.d/javahome.sh
	$ cat /etc/profile.d/javahome.sh

bradkarels / doAvro.txt

Last active August 31, 2015 19:05

Fuddling about with avro and the scala repl...

	bkarels@ubuntu:~/dev/avro/thing$ cat thing.avsc
	{
	"namespace": "com.rbh.avro.pops.thing",
	"type": "record",
	"name": "Thing",
	"fields": [
	{
	"name": "name",
	"type": "string"
	},

bradkarels / mapUnion.scala

Created July 14, 2015 14:41

Creating a simple union of two Maps where values in 'core' will be overwritten by the 'overlay' and values in overlay that do not exist in core will be added to the resulitng Map

	val key0 = ("0","0")
	val key1 = ("1","0")
	val key2 = ("2","0")
	val key3 = ("3","0")

	val core:Map[(String,String),Option[String]] = Map(key0 -> Some("a"), key1 -> Some("b"), key2 -> Some("c"))
	val overlay:Map[(String,String),Option[String]] = Map(key2 -> Some("y"), key3 -> Some("z"))

	//val expected = Map(key0 -> Some("a"), key1 -> Some("b"), key2 -> Some("y"), key3 -> Some("z"))

bradkarels / writeFile.scala

Created June 2, 2015 13:01

Write list of Strings to text file on filesystem using java.nio in Scala (2.10.5) REPL

	import java.nio.file.Files
	import java.nio.charset.Charset
	import java.nio.charset.StandardCharsets
	import java.nio.file.Paths
	import java.nio.file.StandardOpenOption
	import collection.JavaConverters._

	val utf8:Charset = StandardCharsets.UTF_8

	Files.write(Paths.get("foo.txt"), "foo".getBytes(utf8))

bradkarels / tcpdump example

Created May 14, 2015 18:10

Using tcpdump on ubuntu local to access Kafka remote on OpenStack - just saving off the command.

sudo tcpdump -nn -i eth0 port 6667

bradkarels / kryoChill.scala

Created April 30, 2015 19:59

Fuddling with Kryo/chill on repl with Spark 1.2.1

	// scala> :cp lib/chill_2.10-0.5.2.jar

	//bkarels@ahimsa:~/spark/current$ ./bin/spark-shell --master local[*] --jars lib/mongo-java-driver-3.0.0.jar,lib/mongo-hadoop-core-1.3.2.jar,lib/chill_2.10-0.5.2.jar

	import com.esotericsoftware.kryo.io.{Input, Output}
	import com.twitter.chill.ScalaKryoInstantiator
	import java.io.ByteArrayOutputStream
	class Person(val name:String) extends Serializable
	val p0:Person = new Person("p0")
	val p1:Person = new Person("p1")

bradkarels / chill.scala

Created April 30, 2015 19:34

Spark Chill Example

	import java.io.ByteArrayOutputStream
	import java.io.ObjectOutputStream
	import java.io.Serializable
	import com.twitter.chill.{Input, Output, ScalaKryoInstantiator}

	class Person extends Serializable {
	var name: String = ""

	def this(name:String) {
	this()

bradkarels / gist:18038cbbab539b426b50

Created March 23, 2015 17:32

YARN Capacity Scheduler with ACLs added - example - Experimental

	yarn.scheduler.capacity.maximum-am-resource-percent=0.2
	yarn.scheduler.capacity.maximum-applications=10000
	yarn.scheduler.capacity.node-locality-delay=40
	yarn.scheduler.capacity.root.acl_administer_queue=bkarels hdpAdmins
	yarn.scheduler.capacity.root.capacity=100
	yarn.scheduler.capacity.root.ds.acl_administer_jobs=dsAdmin,bkarels,nadelman dsAdmins
	yarn.scheduler.capacity.root.ds.acl_submit_applications=dsAdmin,dsUser0,dsUser1 dsAdmins,mlGroup,analyticsGroup
	yarn.scheduler.capacity.root.ds.capacity=40
	yarn.scheduler.capacity.root.ds.maximum-capacity=50
	yarn.scheduler.capacity.root.eng.acl_administer_jobs=bkarels hdpAdmins

bradkarels / gist:b874a0159b5aafa37528

Created March 23, 2015 14:28

YARN capacity scheduler - working example: 1

	yarn.scheduler.capacity.maximum-am-resource-percent=0.2
	yarn.scheduler.capacity.maximum-applications=10000
	yarn.scheduler.capacity.node-locality-delay=40
	yarn.scheduler.capacity.root.acl_administer_queue=*
	yarn.scheduler.capacity.root.capacity=100
	yarn.scheduler.capacity.root.ds.acl_administer_jobs=*
	yarn.scheduler.capacity.root.ds.acl_submit_applications=*
	yarn.scheduler.capacity.root.ds.capacity=40
	yarn.scheduler.capacity.root.ds.maximum-capacity=50
	yarn.scheduler.capacity.root.eng.acl_administer_jobs=*

Brad Karels bradkarels