| Field | Purpose |
|---|---|
| lock | Mutex used for protecting span and memory operations. |
| free | mTreap of non-scavenged spans, i.e., available memory that hasn’t been returned to the OS. |
| scav | mTreap of free and scavenged spans, i.e., memory ready to be reused or returned to the OS. |
| sweepgen | Tracks the current sweep generation for the garbage collector. |
| sweepdone | Indicates whether all spans in a generation have been swept. |
| sweepers | Number of active sweepone calls running concurrently. |
Anshuman Mandal Anshuman-02905
Anshuman-02905
/ Sweep_internals.md
Created
September 24, 2025 04:40
Anshuman-02905
/ memory_Segmet.md
Created
September 24, 2025 04:43
| Metric | Description |
|---|---|
| VSZ (Virtual Memory Size) | Total virtual memory requested by a process, including all mapped libraries, heap, stack, and reserved address space. Can appear very large, even if not all memory is used. |
| RSS (Resident Set Size) | Actual memory currently residing in RAM for the process. Reflects the real memory footprint that affects system performance. |
Anshuman-02905
/ UserspaceVskernelspace.md
Created
September 24, 2025 05:52
| Memory Area | Description |
|---|---|
| User Space | Memory accessible by applications, including heap, stack, and loaded libraries. Each process has its own isolated user space. |
| Kernel Space | Protected memory used by the operating system for core routines, drivers, and managing processes. Shared among all processes but inaccessible to user applications. |
Anshuman-02905
/ Pillers_of_data_Engineering.md
Created
October 30, 2025 08:03
| Stage | AWS Services | Purpose |
| ------------------ | -------------------------------------- | ------------------------------------------------------ |
| Ingestion | DMS, Kinesis, AppFlow, Transfer Family | Bringing data from on-prem, SaaS, or streaming sources |
| Transformation | Glue, EMR, Kinesis Data Analytics | Cleaning, enriching, and standardizing data |
| Orchestration | Glue Workflows, Step Functions, Lambda | Scheduling, dependency handling, and automation |
Anshuman-02905
/ Aws_DMS_AZURE_DMS.md
Created
October 30, 2025 08:12
| Feature | Amazon DMS | Azure DMS |
| -------------------------- | ------------------------------------ | -------------------------------------------------------- |
| Primary Use | Database migration & CDC replication | Database migration & CDC replication |
| Supported Targets | Aurora, RDS, Redshift, S3 | Azure SQL, PostgreSQL, MySQL, SQL MI |
| CDC / Incremental Load | Yes | Yes |
Anshuman-02905
/ Amazon_DMS_vs_Azure_DMS
Created
October 30, 2025 08:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| | **Feature** | **Amazon DMS** | **Azure DMS** | | |
| |----------------------------|--------------------------------------------------|----------------------------------------------------------| | |
| | **Primary Use** | Database migration & CDC replication | Database migration & CDC replication | | |
| | **Supported Targets** | Aurora, RDS, Redshift, S3 | Azure SQL, PostgreSQL, MySQL, SQL Managed Instance | | |
| | **CDC / Incremental Load** | Yes | Yes | | |
| | **Serverless Mode** | Yes (DMS Serverless) | Yes (DMS Standard with Azure Migrate Integration) | | |
| | **Monitoring** | CloudWatch | Azure Monitor / Azure Portal | | |
| | **Schema Conversion** | With AW |
Anshuman-02905
/ Aws_DMS_AZURE.md
Created
October 30, 2025 08:17
| Feature | Amazon DMS | Azure DMS |
|---|---|---|
| Primary Use | Database migration & CDC replication | Database migration & CDC replication |
| Supported Targets | Aurora, RDS, Redshift, S3 | Azure SQL, PostgreSQL, MySQL, SQL Managed Instance |
| CDC / Incremental Load | Yes | Yes |
| Serverless Mode | Yes (DMS Serverless) | Yes (DMS Standard with Azure Migrate Integration) |
| Monitoring | CloudWatch | Azure Monitor / Azure Portal |
| Schema Conversion | With AW |
Anshuman-02905
/ Kinesis.md
Created
October 30, 2025 10:12
| Service | Purpose | Characteristics |
|---|---|---|
| Kinesis Data Streams (KDS) | Raw event ingestion | Flexible schema, developer-managed |
| Kinesis Data Firehose | Reliable delivery to storage | Schema-bound, delivery-optimized |
| Kinesis Data Analytics (KDA) | Stream processing | SQL or Apache Flink runtime |
| Kinesis Video Streams (KVS) | Unstructured media streaming | Handles live video/audio data |
Anshuman-02905
/ Awa_pillers_data_engineering.md
Created
October 30, 2025 10:34
| Layer | Purpose | Example AWS Services | Azure Equivalent |
|---|---|---|---|
| Ingestion | Bring data from SaaS, databases, or streams | DMS, AppFlow, Kinesis | ADF, Azure DMS, Event Hubs |
| Transformation | Clean, enrich, and process raw data | Glue, EMR | Data Factory (Mapping Data Flows), Databricks |
| Storage / Modeling | Persist structured and semi-structured data | S3, Redshift, Lake Formation | ADLS Gen2, Synapse, Purview |
| Consumption | Analytics, BI, ML | Athena, QuickSight, SageMaker | Synapse, Power BI, Azure ML |
Anshuman-02905
/ Aws_DMS_azure_DMS2.md
Last active
October 30, 2025 10:40
| Feature | AWS DMS | Azure DMS |
|---|---|---|
| Migration Mode | Full + Continuous (CDC) | Full + Continuous (CDC) |
| Engine Support | Heterogeneous (Oracle → PostgreSQL) | Heterogeneous (Oracle → Azure SQL) |
| Monitoring | CloudWatch | Azure Portal |
| Security | IAM Roles, VPC | ExpressRoute, VPN |
| Integration | S3, Redshift, Kinesis | ADLS, Synapse |