Skip to content

Instantly share code, notes, and snippets.

@atamborrino
atamborrino / redisStream.scala
Last active April 25, 2017 14:14
Parallel SCAN per node on a Redis Cluster as an Akka Stream Source
import akka.NotUsed
import akka.stream.scaladsl.Source
import redis.clients.jedis.{Jedis, JedisCluster, ScanParams}
import scala.collection.JavaConverters._
import scala.concurrent.Future
class RedisStream(maxNodeParallelism: Int)
(implicit jedisCluster: JedisCluster,
blockingEC: RedisBlockingEC) {
@atamborrino
atamborrino / speculativeRetries.scala
Last active June 6, 2018 14:28
Speculative retries for Scala's Futures, inspired from Cassandra Java driver http://docs.datastax.com/en/developer/java-driver/3.1/manual/speculative_execution/
import akka.actor.Scheduler
import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.concurrent.duration.FiniteDuration
object FutureHelper {
private val log = ScalaLogger.get(this.getClass)
/**
* Retry concurrently the async execution ''f'' every ''delay'' while no execution completes.
* Number of retries is bounded by ''maxSpeculativeExecutions''.
@atamborrino
atamborrino / ExecutionContextMonitor.scala
Last active April 14, 2022 09:10
Monitor Scala's ExecutionContext / Akka Dispatcher lag (number of tasks in waiting queues)
import java.util.concurrent._
import akka.dispatch.{Dispatcher, ExecutorServiceDelegate}
import config.Config
import helpers.ScalaLogger
class ExecutionContextMonitor()(implicit metricsService: MetricsClient, config: Config) {
private val log = ScalaLogger.get(this.getClass)
private val scheduler = Executors.newSingleThreadScheduledExecutor()
@atamborrino
atamborrino / inmemoryMleap.scala
Last active July 6, 2017 07:09
In-memory use of MLeap, a serialization format for Spark ML (PoC style, no error handling)
import ml.combust.mleap.runtime.{LeapFrame, LocalDataset, Row}
import ml.combust.mleap.runtime.types.{DoubleType, StructField, StructType, TensorType}
import ml.combust.mleap.tensor.Tensor
import resource._
import ml.combust.mleap.runtime.MleapSupport._
import scala.collection.JavaConverters._
import com.google.common.jimfs.{Configuration, Jimfs}
import ml.combust.bundle.BundleFile
import java.nio.file.{FileSystems, Files}
@atamborrino
atamborrino / typeSafeEquality.scala
Last active July 13, 2017 13:11
Simple type safe equality on arbitrary case class with Cats + Kittens
import cats.derived._, eq._, legacy._
import cats.implicits._
case class Foo(a: String, b: Int)
case class Bar(c: Boolean)
Foo("a", 0) === Foo("b", 0) // compile
Foo("a", 0) === Bar(true) // does not compile
Foo("a", 0) == Bar(true) // compile with warning
@atamborrino
atamborrino / timedTypeClass.scala
Last active September 7, 2017 10:05
Timed type class + implicit derivation to Order type class via Natural Transformation
import java.time.Instant
import shapeless._
import simulacrum._
import scala.concurrent.duration.FiniteDuration
import scala.language.implicitConversions
import scala.concurrent.duration._
@typeclass
trait Timed[A] {
def getInstant(a: A): Instant
@atamborrino
atamborrino / keybase.md
Last active February 5, 2018 10:06
keybase.md

Keybase proof

I hereby claim:

  • I am atamborrino on github.
  • I am atamborrino (https://keybase.io/atamborrino) on keybase.
  • I have a public key ASBtyXFwZvKC4C4JCyiktqVzas0I6gsAnKjZp2OodH54Ogo

To claim this, I am signing this object:

@atamborrino
atamborrino / tf_test.ipynb
Last active June 18, 2018 08:03
Example of use of TensorFlow Core API to train a linear regression model with standard normalization of data baked into the TF Computational Graph
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@atamborrino
atamborrino / gist:4f66b4a47a1785f150e7b813cae7041e
Created August 24, 2018 12:32
Churn prediction / Timeseries classification links

Classic ML:

Class imbalance:

DL:

# src: https://www.kaggle.com/aglotero/another-iou-metric
def iou_metric(y_true_in, y_pred_in, print_table=False):
labels = y_true_in
y_pred = y_pred_in
true_objects = 2
pred_objects = 2
intersection = np.histogram2d(labels.flatten(), y_pred.flatten(), bins=(true_objects, pred_objects))[0]