squito’s gists

squito / GroupedRDD.scala

Last active August 29, 2015 14:17

GroupedRDD

	import java.io.{IOException, ObjectOutputStream}

	import scala.language.existentials
	import scala.reflect.ClassTag

	import org.apache.spark._
	import org.apache.spark.rdd.RDD


	case class GroupedRDDPartition(

squito / AccumulatorListener.scala

Last active March 15, 2019 06:34

Accumulator Examples

	import scala.collection.mutable.Map

	import org.apache.spark.{Accumulator, AccumulatorParam, SparkContext}
	import org.apache.spark.scheduler.{SparkListenerStageCompleted, SparkListener}
	import org.apache.spark.SparkContext._


	/**
	* just print out the values for all accumulators from the stage.
	* you will only get updates from named accumulators, though

squito / gist:242f82ad6345e3f85a5b

Last active August 29, 2015 14:16 — forked from ryanlecompte/gist:6313683

	import scala.concurrent.{Future, Promise}
	import scala.concurrent.duration.{Duration, FiniteDuration}
	import scala.concurrent.{Await, Promise}
	import scala.util.{Failure, Success}
	import scala.concurrent.ExecutionContext.Implicits.global


	def performSequentially[A](items: Seq[A])(f: A => Future[Unit]): Future[Unit] = {
	items.headOption match {
	case Some(nextItem) =>

squito / test-time-parse.scala

Created January 14, 2015 20:50

parsing the scalatest timing output from sbt

	case class TestTime(suite: String, test: String, time: Int)

	def parse(input: Iterator[String]): Seq[TestTime] = {
	val TimingPattern = """(.)$((\d+) seconds?, )?(.) milliseconds?$""".r
	val x = input.filter{line => line.startsWith("[info]") && !line.contains("!!! IGNORED !!!") && !line.startsWith("[info] +")}.map{_.substring("[info] ".length)}
	var suiteName: String = null
	var result = IndexedSeq[TestTime]()
	while(x.hasNext) {
	val line = x.next
	if(line.endsWith(":"))

squito / .screenrc

Created December 3, 2014 21:50

	logfile screenlogs/log.%n
	deflog on
	defscrollback 10000
	screen 1
	#other settings
	hardstatus alwayslastline
	hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'
	vbell off
	autodetach on
	#Binds

squito / 0_reuse_code.js

Last active August 29, 2015 14:08

Here are some things you can do with Gists in GistBox.

	// Use Gists to store code you would like to remember later on
	console.log(window); // log the "window" object to the console

squito / Build.scala

Created October 29, 2014 13:09

avoid sbt assemblies

	...

	// define your projects, with a custom "settings"
	lazy val common = Project(
	id = "common",
	base = file("common"),
	settings = baseSettings ++ Seq(
	libraryDependencies ++= commonLibraryDependencies
	)
	)

squito / mylib.R

Last active June 13, 2019 19:09

searchable source dirs for R (aka, a "classpath" for R)

	## this should get loaded by your ~/.Rprofile
	## either just stick all these definitions right in ~/.Rprofile, or
	## have ~/.Rprofile source this file, etc.

	source.dirs <- c(
	## add any "base" locations here. this is like a "classpath" in java
	## my default is c("~/myRUtils/src","~/companyRUtils/src","~/publicRUtils"),
	## but you can use whatever you like
	paste(Sys.getenv("HOME"),"myRUtils","src", sep="/"),
	paste(Sys.getenv("HOME"),"companyRUtils","src", sep="/"),

squito / binomial_confidence.R

Created June 17, 2014 14:56

Some hacky R code to explore confidence interval estimation for binomial rates, and the difference between two binomial rates.


	alpha = 0.8
	z <- 1.28

	normal.theta.conf <- function(x, n) {
	# normal approximation to a binomial. Generally agreed to be horrible
	phat <- x / n
	t <- z * sqrt(phat * (1 - phat)) / sqrt(n)
	c(max(0,phat - t), min(1, phat + t))
	}

squito / TaggedTypes.java

Created May 7, 2014 22:41

	➜ tmp javap -c -private TaggedTypes$
	Compiled from "TaggedTypes.scala"
	public final class TaggedTypes$ {
	public static final TaggedTypes$ MODULE$;

	public static {};
	Code:
	0: new #2 // class TaggedTypes$
	3: invokespecial #12 // Method "<init>":()V
	6: return

	case class TestTime(suite: String, test: String, time: Int)

	def parse(input: Iterator[String]): Seq[TestTime] = {
	val TimingPattern = """(.)\(((\d+) seconds?, )?(.) milliseconds?\)""".r
	val x = input.filter{line => line.startsWith("[info]") && !line.contains("!!! IGNORED !!!") && !line.startsWith("[info] +")}.map{_.substring("[info] ".length)}
	var suiteName: String = null
	var result = IndexedSeq[TestTime]()
	while(x.hasNext) {
	val line = x.next
	if(line.endsWith(":"))

Imran Rashid squito