-
Created a general-purpose join service using Apache Flink and Scala to extract, filter, denormalize, sort, and summarize structured data for Elasticsearch indexing, which led to an over 50% improvement in data freshness for the Campus Labs Insight product. Typical queries processed hundreds of millions of source rows from over one-hundred tables across several databases multiple times per day. Service runs on Kubernetes and provides a GraphQL-inspired API for describing the relationships between data sources and the desired shape of the results. Supports automatic RDBMS schema discovery, parallel extraction, throttling, Vault-based credential management, and recursive joins using delta-iteration.
-
Built a scheduled ETL process using Flink and Scala to transform and load over 1 billion rows daily into a Data Vault-based data warehouse. Implemented keyset table slicing to parallelize extracti
Joshua Griffith hadronzoo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [package] | |
| name = "triggering" | |
| version = "0.1.0" | |
| authors = ["Joshua Griffith <[email protected]>"] | |
| edition = "2018" | |
| [dependencies] | |
| rand = "0.7" | |
| [dev-dependencies] |
OlderNewer