Skip to content

Instantly share code, notes, and snippets.

View mserranom's full-sized avatar

Miguel Serrano mserranom

  • London
View GitHub Profile
// THIS IS NOT WORKING!
type A = {
type: "A";
aprop: number;
};
type B = {
type: "B";
otherprops: number;
@mserranom
mserranom / feature_store_mvp_backend.md
Last active May 31, 2018 16:39
feature_store_mvp_backend.md

Feature Store MVP Backend

Data

Entities and cardinality:

                                                  1 +------------+
                                             +------+ stream_dtc |
+-------------+ 1     n +-------------+ 1    |      +------------+
@mserranom
mserranom / jonthebeach.md
Last active May 29, 2018 07:39
jonthebeach.md

jonthebeach

Infrastructure as Code with Terraform

Mitchell Hashimoto - @mitchellh

The problem - history

  • From bash scripts to virtualization empowers automation
  • Virtualization AAS: the cloud
  • Cloud architectures are super powerful but complex -> Capability breeds complexity
  • Terraform (TF) describes the infrastructure as code

Feature Store

Uber Michelangelo

https://eng.uber.com/michelangelo/

Finding good features is often the hardest part of machine learning and we have found that building and managing data pipelines is typically one of the most costly pieces of a complete machine learning solution.

A platform should provide standard tools for building data pipelines to generate feature and label data sets for training (and re-training) and feature-only data sets for predicting. These tools should have deep integration with the company’s data lake or warehouses and with the company’s online data serving systems. The pipelines need to be scalable and performant, incorporate integrated monitoring for data flow and data quality, and support both online and offline training and predicting. Ideally, they should also generate the features in a way that is shareable across teams to reduce duplicate work and increase data quality. They should also provide strong guard rails and controls to encourage and empower users to adop

@mserranom
mserranom / dtc_stream.yml
Last active May 7, 2018 16:08
Example for a Blurr transformation schema that uses a Python convention-over-configuration DSL-ish instead of YAML
Type: Blurr:Transform:Streaming
Version: '2018-03-01'
Description: New York Store Exchange Transformations
Name: nyse
Import:
- { Module: datetime, Identifiers: [ datetime ] }
Identity: source.symbol
@mserranom
mserranom / nyse.ipynb
Last active May 2, 2018 16:51
nyse.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@mserranom
mserranom / postprocess.sh
Created March 29, 2018 09:11
Example documentation postproccessing
echo "postprocessing documentation..."
PACKAGE=`cat blurr/PACKAGE` # PACKAGE and VERSION are generated during pypy package build
VERSION=`cat blurr/VERSION`
BRANCH=`git branch | sed -n -e 's/^\* \(.*\)/\1/p'`
sed -e "s/\@BRANCH@/$BRANCH/" binder/README-template.md > binder/README.md
sed -e "s/\@PACKAGE@/$PACKAGE/" -e "s/\@VERSION@/$VERSION/" binder/requirements-template.txt > binder/requirements.txt
@mserranom
mserranom / blurr_goals.md
Created March 13, 2018 14:02
Blurr Goals

1. what is it

I't a serverless tool that extend traditional event logging introducing support advance Real Time Processing and ML Training scenarios.

2. why should I use it

Because it enables advanced data engineering and ML scenarios for your project in a lightweight and affordable fashion

3. why should I contribute to it

Because we're open to colaborate in your proposal and build the features you need.

@mserranom
mserranom / moving_house.md
Last active February 7, 2018 13:24
Stuff to do when moving house

Stuff to do when moving house

  • Change HMRC address
  • Cancel/Change gas and electricity contracts
  • Notify address change if waiting for Permanent Residency approval / Citizenship / Passport
  • Change Address in banks
  • Notify and cancel Council Tax
  • GP notify / change
  • If temporarily, notify RoyalMail to forward email somewhere else
@mserranom
mserranom / ciudad_del_paraiso.md
Last active January 16, 2018 16:02
Malaga, Ciudad del Paraíso

Ciudad del Paraíso

Vicente Aleixandre

A mi ciudad de Málaga

Siempre te ven mis ojos, ciudad de mis días marinos.

Colgada del imponente monte, apenas detenida

en tu vertical caída a las ondas azules,