Skip to content

Instantly share code, notes, and snippets.

@Anshuman-02905

Anshuman-02905/Awa_pillers_data_engineering.md

Created October 30, 2025 10:34
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Anshuman-02905/bf28fa279db42dffc01a4d4f6a31c0a9 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Anshuman-02905/bf28fa279db42dffc01a4d4f6a31c0a9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Awa_pillers_data_engineering.md
Layer Purpose Example AWS Services Azure Equivalent
Ingestion Bring data from SaaS, databases, or streams DMS, AppFlow, Kinesis ADF, Azure DMS, Event Hubs
Transformation Clean, enrich, and process raw data Glue, EMR Data Factory (Mapping Data Flows), Databricks
Storage / Modeling Persist structured and semi-structured data S3, Redshift, Lake Formation ADLS Gen2, Synapse, Purview
Consumption Analytics, BI, ML Athena, QuickSight, SageMaker Synapse, Power BI, Azure ML
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment