| Glue Feature | How EMR Uses It |
|---|---|
| 🗂️ Glue Data Catalog | Acts as EMR's Hive Metastore - so Spark SQL and Hive queries can discover schema & tables stored in S3. |
| 🧩 Glue Crawlers | Automatically infer schema from S3 data → EMR Spark can read these tables instantly. |
| 🪄 Glue ETL Jobs | You can mix Glue ETL jobs (serverless) with EMR PySpark workflows - e.g., Glue for light jobs, EMR for heavy transformations. |
| 🔄 Glue Job Bookmarks | Track incremental load progress (used in both Glue and EMR workflows). |
| 🔐 **Glue + L |
Anshuman Mandal Anshuman-02905
Anshuman-02905
/ Glue_features.md
Created
November 4, 2025 08:01
Anshuman-02905
/ Kinesis.md
Created
November 3, 2025 12:03
| Service | Tagline | Core Role |
|---|---|---|
| KDS | 🧩 "Ingest Everything" | Real-time event ingestion & buffering |
| KDA | ⚙️ "Process It in Motion" | Real-time analytics & enrichment |
| KDF | 🚚 "Deliver Everything" | Reliable managed delivery to storage |
| KVS | 🎥 "See Everything" | Real-time video/audio ingestion & analytics |
Anshuman-02905
/ KDS_KDF.md
Created
November 3, 2025 07:38
| Feature | Kinesis Data Streams (KDS) | Kinesis Data Firehose (KDF) |
|---|---|---|
| Purpose | Custom, scalable real-time stream processing | Managed delivery of streaming data |
| Control | Developer-managed (shards, consumers, scaling) | Fully managed (auto-scaling, no shards) |
| Replay Support | Yes (1–7 days retention) | No (fire-and-forget delivery) |
| Processing | Consumer applications process in near real-time | Lambda transformations or format conversions |
| Destinations | Any consumer (custom apps, Firehose, Analytics) | S3, Redshift, OpenSearch, HTTP |
| Ideal For | Real-time computation, analytics, custom logic | Near-real-time ingestion, ETL, data lake loading |
Anshuman-02905
/ EMRvsDatabricka.md
Last active
November 4, 2025 08:04
| Feature | Amazon EMR | Azure Databricks |
|---|---|---|
| Engine | Spark, Hive, Hadoop, Presto | Spark |
| Cluster Management | Fully managed, autoscaling | Managed via workspace |
| Cost Optimization | Spot + transient clusters | Job clusters, autoscaling |
| Integration | S3, Glue Catalog, Redshift | ADLS, Synapse |
| Customization | Root access, bootstrap scripts | Limited (managed runtime) |
Anshuman-02905
/ Glue_azure.md
Created
October 31, 2025 08:04
Anshuman-02905
/ Kinesis Data streams.md
Last active
November 3, 2025 07:21
| Feature | Kinesis Data Streams (AWS) | Azure Event Hubs |
|---|---|---|
| Purpose | Collect and store real-time event data for custom processing | Same — ingest event, log, or telemetry data for downstream processing |
| Data Retention | 1–7 days (default 24 hrs, extendable to 7) — optionally up to 365 days with extended retention | 1–7 days (standard), up to 90 days with premium tiers |
| Scalability Unit | Shards (~1 MB/s in, 2 MB/s out each) | Throughput Units (1 MB/s in, 2 MB/s out) |
| Consumption Model | Multiple consumers can read the same stream independently (enhanced fan-out) | Multiple consumer groups can read the same event hub independently |
| Integration | Feeds into Firehose, Lambda, KDA (Flink), or custom Spark apps | Feeds into Stream Analytics, Functions, Databricks, or custom apps |
| Use Case | Real-time ingestion for IoT, logs, clickstreams, metrics, CDC pipelines | Same — telemetry, log ing |
Anshuman-02905
/ AppFlow.md
Last active
November 3, 2025 06:36
| Feature | AppFlow | Azure Data Factory |
|---|---|---|
| Focus | SaaS integrations | Enterprise-scale ETL/ELT orchestration |
| Control | Minimal setup, low code | Full control, dynamic pipelines |
| Connectors | Deep SaaS partnerships | Broad hybrid connector library |
| Transformations | Basic (mapping, filters) | Complex data flows (Mapping Data Flows) |
| Orchestration | ❌ Not supported | ✅ Triggers, branching, retries |
| Best For | Plug-and-play SaaS → S3/Redshift | Metadata-driven ETL pipelines |
Anshuman-02905
/ Aws_DMS_azure_DMS2.md
Last active
October 30, 2025 10:40
| Feature | AWS DMS | Azure DMS |
|---|---|---|
| Migration Mode | Full + Continuous (CDC) | Full + Continuous (CDC) |
| Engine Support | Heterogeneous (Oracle → PostgreSQL) | Heterogeneous (Oracle → Azure SQL) |
| Monitoring | CloudWatch | Azure Portal |
| Security | IAM Roles, VPC | ExpressRoute, VPN |
| Integration | S3, Redshift, Kinesis | ADLS, Synapse |
Anshuman-02905
/ Awa_pillers_data_engineering.md
Created
October 30, 2025 10:34
| Layer | Purpose | Example AWS Services | Azure Equivalent |
|---|---|---|---|
| Ingestion | Bring data from SaaS, databases, or streams | DMS, AppFlow, Kinesis | ADF, Azure DMS, Event Hubs |
| Transformation | Clean, enrich, and process raw data | Glue, EMR | Data Factory (Mapping Data Flows), Databricks |
| Storage / Modeling | Persist structured and semi-structured data | S3, Redshift, Lake Formation | ADLS Gen2, Synapse, Purview |
| Consumption | Analytics, BI, ML | Athena, QuickSight, SageMaker | Synapse, Power BI, Azure ML |
Anshuman-02905
/ Kinesis.md
Created
October 30, 2025 10:12
| Service | Purpose | Characteristics |
|---|---|---|
| Kinesis Data Streams (KDS) | Raw event ingestion | Flexible schema, developer-managed |
| Kinesis Data Firehose | Reliable delivery to storage | Schema-bound, delivery-optimized |
| Kinesis Data Analytics (KDA) | Stream processing | SQL or Apache Flink runtime |
| Kinesis Video Streams (KVS) | Unstructured media streaming | Handles live video/audio data |