Skip to content

Instantly share code, notes, and snippets.

@mcmoe

mcmoe/strata-data-18-sessions.md

Last active February 10, 2025 08:13
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mcmoe/3359f08a79bfd619ff08c2d477131b73 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mcmoe/3359f08a79bfd619ff08c2d477131b73 to your computer and use it in GitHub Desktop.
Download ZIP
The various Tuesday and Wednesday Sessions of Strata Data Conference 2018
Raw
strata-data-18-sessions.md

TODO

  • Kafka Schema Registry
  • LINQ SQL style API
  • How does a "table" like TaxiRides get created in the Flink example?
    • My question would be if the info is coming from differnt sources how do we create a join out of them?

The future of ETL isn’t what it used to be.

  • 11:20am-12:00pm, Sep 12 / 1A 23/24
  • Topics: Data engineering and architecture
  • Description:
    • Gwen Shapira shares design and architecture patterns that are used to modernize data engineering.
    • You'll learn how modern engineering organizations use Apache Kafka, microservices, and event streams to efficiently build data pipelines that are scalable, reliable, and built to evolve.
  • Speaker: Gwen Shapira
  • Company: Confluent

Notes

  • In the 60s you could imagine the 2000s will have robots and flying cars but nothing like Wikipedia :D

  • So as humans, we extrapolate the trends that we see

  • With Moore's law, this works quite well (so well I see it in every conference!)

  • We'll take a look at the future we imagined in the pst, the trends of today, the future they indicate (5 yrs), suggestions and thoughts

  • TODO: Check the book - Kafka The Definitive Guide

  • She speaks about the Data Warehouse bible book

  • Our efforts went into data modelling - which she compares to hostae negotiation

  • The conforming data problem - documented in the big kimble bible

  • You had to get everyone in the company to agree on the model - this was long and not fun

  • The entire ETL process took a long time

  • The book says quick was to get a report every single day

  • Main pain points of data integration

  • So we went towards Hadoop - it promised it will handle a lot of various data sources

  • One big target like the Warehouse

  • Mayn ETL tools were kind of confusing

  • The big thing though is "Schema on Read"

  • We don't have to get everyone to agree on the data model!

  • Just copy files from one place to another

  • Now there are poor data scientists that have to make sense of this mess

  • We took the hardest most annoying part of our job and put it on someone else's lap (the poor data scientist)

  • So we still have the same problems! (now the data scientists are spending their times in meetings trying to understand the fields)

  • Reports are still running once a day!

  • Things became incredbily messy as well since people did not completely move their arch since change is hard

  • Changes in the world since then

    • Cloud
      • ETL in the cloud - data sources could be in opaque systems
      • cloud data could be in different vendors
      • Cloud native idea came up
      • So our ETL process should be able to understand that machines can come and leave
      • ETL should be resilient to these
    • Microservices
      • They have their own data (probably a schema in a big db)
      • you have some 30K schemas now but still need to create a consolidated report
      • ETL should be able to deal with this
      • Microservices are a very good design pattern that we can use to implement our ETL tool
      • cloud and microservices drove more changes
    • Software engineers now do everything
      • Git, CI, all the kinds of tests, push to production, use API's
      • Some people don't even know they're doing ETL
      • Thinking about this ... a full stack would do an ETL
    • Serverless
      • Your microservices are really really micro, basically have one function
      • and you don't know anything about where and how it is deployed
      • ETL would probably need to be possible using these serverless archs

So what now?

1- Move to stream based architectures

  • Imagine all your data as a stream
  • An event is a good unit for our model
  • You can join them, aggregate them, send them here or there ...
  • This is basically the vision behind Apache Kafka
  • process them, store them and send them somewhere else when needed

2- Move fast, don't break compatibility

  • Schema registry!
  • This is a way to model your events
  • it can have a gradle verify step to ensure that everything matches the schema

3- Integrate database and applications at scale

  • Do you think that's a table you are querying? :D
  • Tables are an illusion
  • The only source of truth of a database is a transaction log
  • A DB recovers using its transaction log
  • This is a stream of events
  • This brings us to the Stream/Table Duality
  • Full streaming can be achieved with Kafka Connect and Kafka Streams
  • Goes through an intersting example

Is ETL dead?

  • No it's not - it's changing a lot and very fast
  • If a caterpillar become a butterfly, it's not dead :)

Resources

Why and how to leverage the power and simplicity of SQL on Apache Flink

  • 1:15pm-1:55pm, Sep 12 / 1E 07/08
  • Topics: Streaming systems & real-time applications
  • Description:
    • Fabian Hueske discusses why SQL is a great approach to unify batch and stream processing.
    • He gives an update on Apache Flink's SQL support and shares some interesting use cases from large-scale production deployments.
    • Finally, Fabian presents Flink's new query service that enables users and applications to submit streaming and batch SQL queries and retrieve low-latency updated results.
  • Speaker: Fabian Hueske
  • Company: Apache Flink project

Notes

  • Flink data stream processor

  • Uber, Netflixm, Alibaba, ING

  • Uber uses it for stream SQL

  • High-level Analytics API (SQL/Table API (dynamic tables)

  • Provides for Stream and Batch Data Processing its DataStream API (streams, windows)

  • Stateful Event-Driven Applications - it provides Process Functions (events, state, time)

  • We will focus on the High-level analytics API

  • two relational apis

  • ANSI SQL

  • and LINQ-style Table API

    •  tableEnvironment
   .scan("clicks")
   .groupBy('user')
   .select(...)

  • Query translation with Apache Calcite

  • The query is a continuous one

Database systems run queries on streams

  • Materialized views (MV) are similar to regular views, but persisted to disk or memort
  • MV maintance is very similar to SQL on streams

Continuous queries in flink

  • Core conecpt is a "Dynamic Table" - they change over time
  • Queries on dynamic tables reutrn results that do not end
  • Three types od stream <-> dunamic table conversions
    • append conversions - only inserted no updated or deletes
    • upsert conversions - data must have a unique key
      • records are upserted or deleted based on that key
    • changelog conversions
      • most generic
      • records inserted or deleted
      • update = delete old + insert new

Flink (1.6.0) SQL feature set

  • SELECT FROM WHERE
  • GROUP BY + HABING
  • JOIN
    • Windowed INNER + OUTER JOIN
    • Non-windowed INNER + OUTER JOIN
  • Scalar, aggregation, table-valued UDFs (so can join a stream to a function)
  • ...

What can I build with this

  • Data pipelines
  • Low-latency ETL
  • Stream and Batch Analytics
  • Power live dashboards

Sounds great, how can I use it?

  • Embed SQL in Flink app
  • Run SQL via CLI client
    • The SQL CLI cient optimizes the query before sending it to the server

More to come

  • FLIP-24 - an SQL query service

    • REST service to submit and manage SQL queries
    • Integration with external catalogs

  • Uses cases

    • Data exploration with notebooks like Apache Zeppelin
    • Access to real-time data from applications
    • Easy data routing/ETL from management consoles

/!\ Couldn't make it into the 3rd session on Tuesday

Real-time analytics and BI with data lakes and data warehouses using Kudu, HBase, Spark, and Kafka: Lessons learned

  • 2:55pm–3:35pm Wednesday, 09/12/2018
  • Location: 1A 23/24
  • Topics: Data engineering and architecture
  • Secondary topics: Data Integration and Data Pipelines
  • Description:
    • Explore a blueprint and tips from the trenches for creating a streaming data warehouse and data lake
    • Mauricio Aristizabal shares lessons learned from migrating Impact’s traditional ETL platform to a real-time platform on Hadoop (leveraging the full Cloudera EDH stack).
    • Mauricio also discusses the company’s data lake in HBase, Spark Streaming jobs (with Spark SQL), using Kudu for “fast data” BI queries, and using Kafka’s data bus for loose coupling between components.
  • Speaker: Mauricio Aristizabal
  • Company: Impact

Notes

  • Their new Stack

    • Kafka - Distributed Log, managed schemas
    • HBase - 3D Schema-less KB Store
    • SPark efficient in-memory data processing, batch or streaming (micro-bathces)
    • Kudu - fast analytics on fast data + KV
      • It's columnar
      • Perfect for OLAP and BI
      • Fast data - data goes in and is available immediately
    • Impala - Analytic query engine (SQL/JDBC) over HDFS (parquet), HBase, Kudu

  • Use Kafka as their bus

  • Components don't talk to each other, they wrie and read from Kafka

  • Stores raw data events in Kafka!

  • Then build DW out of it

  • If DW needs to change, no problem since they always have the full data in Kafka

  • Spark SQL on DW for continuous querying i guess - right?

  • So they have quadrants of data

    • Raw Stream
    • Raw Store
    • Structured Stream
    • Structured Store

  • /!\ Too fast must take look at video if available - I hope it is!

  • Used open-replicator but not mysql-bind...?? cdc

  • Uses Avro

  • Mac decided to restart ... :/

Setting up a lightweight distributed caching layer using Apache Arrow

  • 4:35pm–5:15pm Wednesday, 09/12/2018
  • Topics: Data engineering and architecture
  • Location: 1A 10
  • Description:
    • Explore Apache Arrow, from its design and architecture to using it in applications
    • Learn how to use Arrow to achieve various objectives for performance, governance, and access
  • Speaker: Jacques Nadeau
  • Company: Dremio

Notes

  • Apache Arrow, Calcite, Parquet

  • easy extension customization and enterprise flexibility

  • exec, input and output are all built on native arrow

  • a data as a service platform

  • What is arrow?

    • started almost three years ago
    • how to represent data to process it efficiently
    • allows you to stictch together a set of data systems
    • Arrow is trying to make the whole serialization/deserialization overhead less
    • supports both relational and complex data
    • focused on columnar in-memory data

  • its two things

    • for processing
    • and for export

  • The transport is not the problem, the problem is all these custom change in representation in data rather than the transport itelf!

  • you're spending too much time transforming the data!

  • so let's come up with a representation and algorithms that can process very well on them - arrow

  • It's both a transport format and for effective processing

  • Arrow as said is in memory columnar format

  • Adopted by

    • dremio
    • nvida for GPU analytics
    • Spark to transport data to python
    • closely related to parquet

  • Data types

    • Scalars
      • boolean, int, decimal, float, double, date, time, timestamp, string, binary
    • ...

  • Multi language libraries

  • Arrow building blocks (in project)

    • Plasma - shared memory caching layer (originally created in Ray)
      • I can take data in arrow and store it in Plasma
    • Feather
      • fast ephemeral format for movement of data between R/Python
      • drop it to disk and re-read it somewhere else
    • Arrow Floght
      • RPC/IPC interchange library
    • Gandiva initiative
      • high performance compiler for Arrow processing of arbitrary expressions

  • What Arrow is not

    • its not an installable system
    • not a memory frig or in-memory cache
    • its not designed for streaming or other single record operations
      • too much overhead in this case

  • Arrow Flight

    • Network protocol to move arrow from one node to another in an efficient manner
    • RPC between two different systems
    • Arrow Flight is a set of tools that allows you to build a data microservice

  • Information on how to use Arrow Flight ...

  • Flight is built on top of GRPC

  • Uses protobuf as a its serialization format

  • But won't use the "data" to avoid copying and take advantage of the zero copy definition that arrow has... (?)

  • Security

    • Authenticate to get request token
    • Arbitrary back and forth as part of negotiation
    • JWT like??

  • Back pressure and stream management

    • Arrow Flight builds on HTTP/2 Streams specs
    • Each stream can be individually throttled by client consumption
    • Allows a multi-tenant application to use a single Arrow connection for many users
    • So one socket to an Arrow Flight service for multi users! So won't care about things like pooling :)

  • Extended GRPC to better work with Arrow Streams

  • Getting rid of the protobuf copy gains some 25% in throughput

Examples of data service components

Apache Kafka and the four challenges of production machine learning systems

  • 5:25pm–6:05pm Wednesday, 09/12/2018
  • Topics: Data engineering and architecture
  • Secondary topics: Model lifecycle management
  • Location: 1A 21/22
  • Description:
    • Learn how to use Apache Kafka and stream processing to make it easier to build machine learning systems
    • Jay Kreps explores some of the difficulties of building production machine learning systems and explains how Apache Kafka and stream processing can help.
  • Speaker: Jay Kreps
  • Company: Confluent

Notes

  • Challenges

    • How do you deploy ML to production
    • How do you have both model builders and system builders work together
    • How do you test ML? How do you do QA on ML?!
    • Diverse data dependencies

  • The Risk Formula

    • Badness = P(bad thing) * impact * duration

  • To decrease the duration, historically we said ok we ship out smaller and smaller things so we can rollback

  • But to assess the impact of your ML models, you need to rollout, and collect metrics and use that to assess them

  • so the duration thing is much harder to control now

  • With so many data feeds, when they break your ML features break - so it makes it hard

  • Google paper: The high interest credit card of technical debt

  • What do we do?

  • Open question :)

  • Hmmmmmm!

Kafka

  • Solve this data pipelining and data integration problem at LinkedIn

  • It's about streams or commit logs od data

  • It's like a DW or Hadoop cluster, but instead of a static table, it's these continuous streams

  • You can publish to other sources or read off it and derive other streams as well

  • 3 quick ways to think about Kafka

    • messaging done right - an evolution of messaging queues
      • Persistent record of what happened
    • ETL and data integration as a platform
      • evolution of it ...
    • Hadoop made fast
      • evolution of it from static data to real time sreams
      • KSQL allows you to do continuous processing on streams of Kafka

  • So what does this have to do with ML?

    • Kafka can decouple the model building portion from the serving portion
    • Kafka acts as the pipeline for instrumentation
    • Kafka helps tame diverse data dependencies
      • Different than an ETL approach
      • We won't consume from your DB, you as DB owner should publish based on this contract
      • So instead of having 100 dependencies, now you have 100 contract
      • /!\ ETL does not provide a contact, as if the source changes, you have to change your ETL, while if you formalize a contract and have the source publish events, then it can simplify the process

  • Schema for the data

    • Many solutions for that
      • Confluent produces a global schema registry that uses Avro
        • It allows you to document all the fields that you're expecting
        • It has a concise binary format
        • It allows you think about compatibility
          • What are the things consumers need to accept, etc...
          • Can set constraints to enforce what can be changed
          • And can enforce that back up to the producer

Why the name - Kafka?

  • They had a theme of naming things after writers
  • and the way it came together it was very kafa-esque :)

Wednesday

The Vega project: Building an ecosystem of tools for interactive visualization

  • 11:20am–12:00pm Thursday, 09/13/2018
  • Location: 1A 12/14
  • Topics: Data science and machine learning, Visualization and user experience
  • Description:
    • Explore Vega and Vega-Lite—emerging visualization tools useful for both analysis practice and the development of interactive data systems
  • Speaker: Jeffrey Heer
  • Company: Trifacta | University of Washington

Notes

  • An upgrade to D3?

  • BI tools like Tableau might be a bit rigid when it comes to interaction ...

  • Vega aims to provide rapid exploaration ...

  • Sweet spot: Visual Analysis Grammars: tabluea VizQL

  • Sweet spot: Visualization Grammars: Protovis D3

  • Component Architectures Prefuse, Flare, Improvide, VTK

  • Vega stems from Visualization Grammar

  • Similar to SQL but for describing visualizatios

  • Like an SQL engine, it will be a graph

  • Vega-Lite - a Visual Analysis Grammar

  • It has a grammar for interactive graphics

  • A strip plot - a tick with x field

  • Vega-Lite is portable JSON representation

  • Very interesting this Vega-Lite

  • has smart defaults and additional processing steps that the language takes on our behalf

  • Of course we can override these defaults

  • Wow can breakd down easily by "weather" for example to split a stacked chart into multiple charts

  • Has composition operators: Facet, Repeat, Layer - can be applied recursively (+ VConcat and HConcat)

  • Has some nice interaction between charts like cross-filtering

  • Vega uses a lot of things from D3 and then Vega-Lite simplifies Vega

  • Voyager is an interactive graphical tool that uses Vega-Lite

  • Interesting way of creating visual explorations via suggested "related views"

  • Add Categorical Field, Alternative Encoding

  • Very good for early profiling of exploration

  • Lyra: Interactive environment for creating chart images - that uses Vega + Vega-Lite

  • Wikipedia uses Vega

  • Next:

    • Automated Design
    • Visualization recommender engines
    • Natural Language UI

  • All are open source @ https://vega.github.io

  • More

  • There is a similar tool https://plot.ly/

  • There is a knowledge base for representing design knowledge about effective visualization design as a collection of constraints. https://uwdata.github.io/draco/

    • could help for recommendation engine?

A deep dive into Kafka controller

  • 1:10pm–1:50pm Thursday, 09/13/2018
  • Location: 1E 07/08
  • Topics: Streaming systems & real-time applications
  • Description:
    • ...
  • Speaker: Jun Rao
  • Company: Confluent

Notes

  • Kafka started as a pub/sub system

  • How do we deliver data in a performant way

  • This started with the "Controller"

  • Connect is another component that allows us to get all the data from these different places into Kafka

  • Connect is for integration

  • Streams - once you have the data in one place, in real time, you want to process them, like transform them for ex.

  • You might want to do some enrichment

  • you might want to do some aggregation

  • Streams is for processing

  • So not only can you store the data in real time you can process it as wekk

  • Kafka is now everywhere

    • 7 of top 10 global banks use it
    • 9 of the top 10 telecom companies
    • 8 of the top 10 insurance companies
    • 6 of the top 10 travel companies

  • Kafka Replication

    • The data has to be durable and always available
    • You have the option of latency/durability to ACK on producer only when leader gets it vs when all get it

  • What's Controller?

    • One broker in a cluster acts as a controlller
    • It monitors the liveness of brokers
    • Elects new leaders on broker failure
    • Data topology of partitions and which brokers are their leaders is stored in Zookeeper
    • This allows the state of leaders/followers to be re-read by a new controller on controller failure

  • Ques: What if the controller fails????

    • Controller failover - Zookeeper KV for /controller is ephemeral
    • So once it dies, the KV is cleaned by Zookeeper
    • Then the other brokers will try to create the path themselves, and the first to succeed will be the new controller :)

  • pre 1.1

    • a lot of the R/W to ZK were serial which would impact performance as number of topics increase
    • Zombie old controller can impact inconsistency (Where old controller did not really die)

  • 1.1 improvements

    • Controller uses async ZK api for R/W
    • Controller communicates new leaders to brokers in batches

  • Fencing zombie controller

    • ZK session expiration
    • Controller epoch (v2.1)

  • More details in JITA task KAFKA-5027 :)

  • Future work in controller

    • further improvement on controller failover - using a standby controller

  • Better handling of quick broker restart (KAFKA-1120)

    • Broker generation

  • 2.1 will be released in October (ish)

High-performance messaging with Apache Pulsar

  • 2:00pm–2:40pm Thursday, 09/13/2018
  • Location: 1E 07/08
  • Topics: Emerging technologies & case studies
  • Description:
    • ...
  • Speaker: Karthik Ramasamy, Matteo Merli
  • Company: Streamlio

... Bad sound - not great ... left at beginning.

Went to

Stories beat statistics: How to master the art and science of data storytelling

  • 2:00pm–2:40pm Thursday, 09/13/2018
  • Topics: Data science and machine learning, Visualization and user experience
  • Location: 1A 12/14
  • Description:
    • Explore the psychology behind why stories beat statistics
  • Learn how to create a powerful narrative for your key insights and better visualize the insights to communicate them more effectively
  • Speaker: Brent Dykes
  • Company: Domo

Notes

  • None - in my head :)

Kafka at PayPal: Enabling 400 billion messages a day

  • Speaker: Kevin Lu, MAULIN VASAVADA, Na Yang
  • 3:30pm–4:10pm Thursday, 09/13/2018
  • Topcis: Streaming systems & real-time applications
  • Location: 1E 09
  • Secondary topics: Data Integration and Data Pipelines, Data Platforms, Financial Services

Missed out on the Paypal Kafka talk - too many people, could not make it in ... :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment