Also notebooks here https://github.com/databricks-academy/data-engineering-with-databricks-english
Lakehouse is a modern open architecture, combining the best elements of data lakes and data warehouses.
Data warehouses: structured data schema enforcement and governance transactions for integrity and consistent reads responsive BI workloads
Data lakes: open file formats like parquet compute separate from storage - multiple compute options machine learning, batch processing, end-to-end streaming semi-structured data like JSON and XML unstructured data like audio, video and images
Lakehouse supports everything that data warehouses provide AND lowers the barrier so every business can use the same sophisticated tools & processes as the most advanced data-driven organisations. Lakehouse supports diverse workloads: data science, machine learning, SQL and analytics One platform is simpler and cheaper to manage Less data movement Open architecture and open source tech: Delta lake and MLFlow, avoids vendor lock-in Future compatible
Delta Lake Metadata and versioning on top of parquet files ACID transaction logs in the storage layer Unifies batch and streaming support view of history, mechanisms to restore or vacuum optimization commands repair partitions purge soft-deleted data (like dropped columns) file compaction
Transaction Isolation modes "Optimistic concurrency control" concurrency level "append-only writes" snapshot isolation even with concurrent reads (and append-only writes)
default transaction isolation level "write serializable" ensures writes are serializable "serializable" ensures both reads and writes are serializable serializable: allowed where there exists a serial sequence of executing them one-at-a-time that generates thje same outcome as that seen in the table.
-
data file sizes-maxRecordsPerFile, less skew in parallel reads and quicker data scans for better performance -
optimized writes- partition data collected on node before writing default target size of 128MB files reduce number of files written Traditionally, writes are direct from nodes to partitions which creates more files. Some cost of shuffling which could pay off in better write throughput or at least subsquent reads are better. "Adaptive shuffle" means you can removecoalesceandrepartition -
autocompaction- compacts to 128MB files -
optimize- compacts to 1GB files -
auto optimize- default for merge, update, delete. minNumFiles default 50, could be good (data skipping, minimize file re-writes) or could be bad (overpartitioning) -
Z-order indexesbest for data skipping reduce data read Colocates data data skipping (partition pruning) less skew in parallel reads quicker scans Uses the z-order curve algorithm to map multidimensional data into a single dimension for easy traversal in a binary tree or skip list. good sorting index across multiple columns when used in any order
What if you z-order twice? Only new rows will be sorted. Relies on having column stats so must be within delta.dataSkippingNumIndexedCols = default 32 Z-order on a high-cardinality column that appears in predicates - used in where clauses or joins
Auto optimize DOES NOT support z-ordering, need to optimize and then z-order periodically for large (10TB+) tables .
Vacuumfile retention default 7 days ... default values and when would you change them
- cloudFiles source
- python & SQL within DLT
- parquet, ORC, csv, avro, json, text, binaryfile
- key-value checkpoints in rocksDB in checkpoint location
- resume with exactly-once guarantees, fault tolerance all managed for you
- dir list vs file notify mode
- dir list is optimised by running in parallel, can do lexical/incremental on azure/gcp
- notification service + queue - (S3 + SQS queue + SNS subscription) better for scaling, more config and permissions required
- trigger regular backfills to deal with cloud provider lack of latency SLA on file events
- extends spark structured streaming - abstraction layer provides simplicity expectations static live table workflows / multi-task jobs
automated prediction and decisions streaming & ML
Data warehouses are good for their traditional business use case of BI and Dashboards. Those are useful outcomes but are much lower down the curve of maturity when compared to high value competitive advantage provided by automated prediction and decisions, which are enabled by modern data processing paradigms like realtime streaming and Machine Learning with live inferences. The Lakehouse supports everything the DW-centric solution supports as well as enabling the same sophisticated tooling and approaches used by the most advanced data-driven organisations.
Streaming, then a scheduled optimize instead of continuously
Depending on the nature of your work, you are also expected to have a well-rounded awareness of 2 out of the three topics below.
- user management & metastore in the account layer, not the workspace layer
- Single definition, standards compliant, built-in auditing
- Metastores must be attached to workspaces
metastore | catalogs | schemas |- external tables, managed tables, views ( materialized?)
How do you set up and provide access for a recipient Choose metastore - enable delta sharing, configure recipient token lifetime,
Control plane -> Data plane Account with multiple workspaces Secure clusters - no public IP ( except for pypi libs) Customer KMS keys
DBFS in customer side Data plane Web console, notebooks, jobs & queries, cluster mgt in Control plane logs in control plane unless configured to write to logging bucket in data plane
Private link options for front & backend
What is the Bronze / Silver / Gold architecture? Streaming What are Auto-loaders? What is the value? What are Delta Live Tables? What is the value? What are Multi-task Jobs? What is the value? How to orchestrate jobs from external orchestrators e.g. Airflow? How does Delta Change Feed make pipeline building simpler and faster? How to handle logging, testing and CI/CD?
Why is this feature important? Queries and Dashboards Integrations with external tools like PowerBI & Tableau Serverless vs Classic Endpoints
-
Classic runs on client infra
-
Pro
-
Serverless How to fine-tune performance and handle endpoint sizing?
-
40 concurrent users
-
scaling by clusters (can still be multiple nodes)
ML Runtimes Feature Stores AutoML Model serving - serverless MLFlow - use-cases and how it could benefit
- Tracking - log params, code versions, metrics and artifacts
- Projects - packaging reusable DS code, a dir or git repo, project version can be the commit, can be chained into workflows
- Models - dir with files and a descriptor file. tracking remebers which project models are in
- Registry - model store, set of APIs for lifecycle. model lineage for experiments, model versions, stage transitions (e.g. to prod)
A delta table plus additional metadata Must have PK Tracks its lineage Online & offline store (publish to online store for low latency and realtime model inference) supports streaming Training set can be extracted from the feature store Doesn't work with unity catalog, must use default hive metastore
Delta sharing - provider and recipient, a token & URL with table-level access - uses short-lived URLs
deep clone - disaster recovery & backup