Suhail Shergill suhailshergill

This is in relation to the IBM Bluemix talk given at https://www.meetup.com/Toronto-Apache-Spark/events/232329359/ on July 27, 2016.

The main utility being highlighted by IBM folks was their ability to provide fair and efficient allocation for workflows. The key element needed for them to be able to deliver their solution was to be able to be able to schedule at a hierarchy of levels. A thing to note is that Apache Mesos/DCOS allows you to write your own customized allocation modules and/or tune the default hierachical allocator.

I would be interested to see a comparison/benchmark between a properly tuned allocation module (customized hierachical DRF or otherwise) for Mesos with IBM bluemix.

Mo kobrosli 1:41 PM

Sent from Toronto Apache Spark

Hi Suhail. Sorry. Yesterday was nuts. Answers below! Didn't fit inline on the comments. over the max :( Didn't feel like breaking it all up.

	/**
	* a sans-shapeless solution to the problem described and solved here:
	* <https://github.com/AlecZorab/solid-disco/blob/master/ShapelessIds.ipynb>. note
	* that the solution below uses the tagless final approach to bake in
	* extensibility for multiple interpretations (which adds to the complexity),
	* but it's not strictly needed for the problem at hand.
	*
	* per aleczorab: "if I have multiple different classes with a field called
	* (say) `id` and a `String => T` constructor, how do I make a `(T, String) =>
	* T` whilst minimising the amount of boilerplate needed for each different

	[ 5 of 25] Compiling Control.Eff1 ( src/Control/Eff1.hs, dist/build/Control/Eff1.o )

	src/Control/Eff1.hs:492:10:
	Could not deduce (Data.OpenUnion51.FindElem Choose r)
	arising from the superclasses of an instance declaration
	from the context (Member NdetEff r)
	bound by the instance declaration at src/Control/Eff1.hs:492:10-46
	In the instance declaration for ‘MonadPlus (Eff r)’

	src/Control/Eff1.hs:527:34:

	;; by conditioning on A, note that we could also condition on B first if desired
	(define (ab)
	(define a (flip 0.8))
	(define b (if a (flip 0.5) (flip 0.3)))
	(list a b)
	)
	(hist (repeat 1000 ab) "joint AB by conditioning on A")

	;; alternative solution
	(define (joint-ab)

	(define (take-sample)
	(rejection-query

	(define A (if (flip) 1 0))
	(define B (if (flip) 1 0))
	(define C (if (flip) 1 0))
	(define D (+ A B C))

	A

	object RDDasMonadPlus {
	import org.apache.spark.{ SparkContext }
	import org.apache.spark.rdd.RDD
	import scalaz._
	import Scalaz._
	import scala.reflect.ClassTag

	// RDDMPlus is the type for which we will define the Monad instance. it can be
	// constructed from an RDD using the RDDClassTag constructor. this
	// implementation is based on insights from

	#!/bin/bash
	export GDK_NATIVE_WINDOWS=1
	exec emacsclient --alternate-editor="" -s ~/.emacs.d/server/socket -c "$@"

	(dolist (pkg '(s))
	(message "%s" pkg) ;; this will print "s"
	(eval `(use-package ,pkg)))

Suhail Shergill suhailshergill

Sessions/Users