darkseed

darkseed

Advanced Functional Programming with Scala - Notes

Copyright © 2017 Fantasyland Institute of Learning. All rights reserved.

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x

darkseed

def sigmoid(x: Double) = 1.0 / (1.0 + math.exp(-x))

def f1measure(TP: Double, TN: Double, FP: Double, FN: Double, alpha: Double = 1) = {
    val P = precision(TP, FP)
    val R = recall(TP, FN)
    (2.0 * P * R) / (P + R)
}

def precision(TP: Double, FP: Double) = TP / (FP + TP)

darkseed

from __future__ import print_function

import sys
from math import sqrt
import argparse
from collections import defaultdict
from random import randint

from pyspark import SparkContext

darkseed

def knapsack_aux(x: (Int, Int), is: List[Int]): List[Int] = {
  for {
    w <- is.zip(is.take(x._1) ::: is.take(is.size - x._1).map(_ + x._2))
  } yield math.max(w._1, w._2)
}

def knapsack_rec(xs: List[(Int, Int)], is: List[Int]): List[List[Int]] = {
  xs match {
    case x :: xs => knapsack_aux(x, is) :: knapsack_rec(xs, knapsack_aux(x, is))
    case _ => Nil

darkseed

sealed trait Tensor[V] {
  val n, m: Int
  def apply(vs: List[V], vds: List[V => Double]): Double
}
 
case class TUnit[V](v: V) extends Tensor[V] {
  val n = 1
  val m = 0
  def apply(vs: List[V], vds: List[V => Double]): Double = vds head(v)
}

darkseed

import java.nio.file.Files
import java.nio.file.Paths

import scala.Array.canBuildFrom
import scala.collection.JavaConverters.iterableAsScalaIterableConverter
import scala.concurrent.ExecutionContext.Implicits.global

import org.elasticsearch.client.transport.TransportClient
import org.elasticsearch.common.settings.ImmutableSettings
import org.elasticsearch.common.transport.InetSocketTransportAddress

darkseed

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

• Introduce Gizzard of Twitter - 트위터의 새로운 분산 관리 라이브러리 Gizzard를 소개합니다.
• The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds
• Scaling Twitter: Making Twitter 10000 Percent Faster

darkseed

package org.apache.spark.examples

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.rdd.RDD
import java.util.Random
import scala.collection.mutable
import org.apache.spark.serializer.KryoRegistrator
import com.esotericsoftware.kryo.Kryo

darkseed

import spark.SparkContext
import SparkContext._

/**
 * A port of [[http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/]]
 * to Spark.
 * Uses movie ratings data from MovieLens 100k dataset found at [[http://www.grouplens.org/node/73]]
 */
object MovieSimilarities {