Tyler Weir tjweir

Internet Scale Services Checklist

A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."

http://mvdirona.com/jrh/talksandpapers/jamesrh_lisa.pdf

Basic tenets

Does the design expect failures to happen regularly and handle them gracefully?
Have we kept things as simple as possible?

I'm putting this list together as a sort of reading plan for myself in order to learn more about general cluster scheduling/utilization and various ways of generically programming to them. Lists of direct links to PDFs here in the order I think makes some sense from skimming reference sections.

Happy to here of any additions that might be sensible.

The Basics

Google File System since everything references it and data locality is a thing.
Google MapReduce because it's one of the earlier well-known functional approaches to programming against a cluster.
Dryad for a more general (iterative?) programming model.
Quincy for a different take on scheduling.
[Delay Scheduling](h

The introduction to Reactive Programming you've been missing

(by @andrestaltz)

So you're curious in learning this new thing called (Functional) Reactive Programming (FRP).

Learning it is hard, even harder by the lack of good material. When I started, I tried looking for tutorials. I found only a handful of practical guides, but they just scratched the surface and never tackled the challenge of building the whole architecture around it. Library documentations often don't help when you're trying to understand some function. I mean, honestly, look at this:

Rx.Observable.prototype.flatMapLatest(selector, [thisArg])

Projects each element of an observable sequence into a new sequence of observable sequences by incorporating the element's index and then transforms an observable sequence of observable sequences into an observable sequence producing values only from the most recent observable sequence.

	So, let’s flip a coin: if it’s heads, play a drum, if it’s tails, play a cymbal. Easy. We can emulate a coin flip with our one_in function (introduced in the section on randomness) specifying a probability of 1 in 2: one_in(2). We can then use the result of this to decide between two pieces of code, the code to play the drum and the code to play the cymbal:

	loop do

	if one_in(2)
	sample :drum_heavy_kick
	else
	sample :drum_cymbal_closed
	end

	{-# LANGUAGE RecordWildCards, Arrows #-}

	import Numeric
	import Data.Char

	import Control.Monad
	import Data.Monoid ((<>))
	import Data.List (nub, sort, reverse)

	data RepeatBounds = RB

	{-# language KindSignatures #-}
	{-# language PolyKinds #-}
	{-# language DataKinds #-}
	{-# language TypeFamilies #-}
	{-# language RankNTypes #-}
	{-# language NoImplicitPrelude #-}
	{-# language FlexibleContexts #-}
	{-# language MultiParamTypeClasses #-}
	{-# language GADTs #-}
	{-# language ConstraintKinds #-}

	-- in response to https://twitter.com/jfischoff/status/948768178070470656
	{-# LANGUAGE TemplateHaskell #-}
	module Main where
	import Control.Concurrent (threadDelay)
	import Control.Monad.IO.Class (liftIO)
	import Data.Traversable (for)
	import Text.Printf (printf)

	import Control.Distributed.Process (Process, NodeId, spawn)
	import Control.Distributed.Process (SendPort, newChan, sendChan, receiveChan)

	{-
	This is from:
	https://www.reddit.com/r/haskell/comments/73jj32/snapservant_servant_scotty_snap_spockthe_sweet/dnsz25v/
	Basically:
	Domain doesn't know about persistence
	Persistence knows about domain, but doesn't know Routing
	Routing knows about domain, but doesn't know Persistence
	Main know all of them and tie them all together
	If you want to expose the application via command line, you can just create a "CommandLine.hs" that basically parse command line args into domain types. Domain + Persistence need not to change.
	-}

	sealed trait Interact[A]

	case class Ask(prompt: String)
	extends Interact[String]

	case class Tell(msg: String)
	extends Interact[Unit]

	trait Monad[M[_]] {
	def pure[A](a: A): M[A]

	import java.io._
	import java.nio.file.{Path, Files}
	import org.specs2.io._
	import org.specs2.Specification
	import scalaz._, Scalaz._, stream._
	import scalaz.stream.nio._

	class ScalazStreamSpec extends Specification { def is = s2"""
	prepare ${step(createFile)}
	with io $ioProcess