Imran Ismail imranismail

Batch No More, Stream Galore: Zero ETL Pipelines from Day 0

In the early stages, many startups rely on traditional batch-processing systems that perform adequately when data volumes are low. However, as organizations grow, these batch pipelines often become inefficient - resulting in slow processing times and complex troubleshooting when issues arise. In many cases, organizations face debates over selecting the optimal batch processing tool, which can consume valuable resources without resolving the underlying performance challenges. Moreover, when failures occur, the process of tracking down issues within a complex network of scheduled jobs can be arduous, sometimes necessitating the reprocessing of entire data batches and causing operational delays.

Recent advancements in the field of Extract, Transform, Load (ETL) have led to the emergence of Zero-ETL strategies. With these developments, streaming data processing is increasingly recognized as the preferred approach from the outset, rather than as a lat

Motivation

The terraform helm provider is rather noisy printing unnecessary metadata in the plan output making it hard to review changes. This awk script redacts unnecessary field from the output to make it easier to review.

Usage

terraform plan -out plan.tfplan | awk -f supress_helm_release.awk
terraform show -no-color plan.tfplan | awk -f supress_helm_release.awk

flowchart BT
    Jedan[Jedan\nChristmas] -->|KomentarA| Dva[Neštonešto]
    Dva --> Tri{Let me think}
    Tri -->|Broj4| Četiri[LeKomentar]
    Tri -->|PremaDva| Pet[Tekst5]
    Tri -->|Tri| Šest[fa:fa-car Car]

Pet[<img src='https://iconscout.com/ms-icon-310x310.png' width='40' /> \nasd]

Start BFF container

$ docker run --rm -it -v $(pwd)/config.yml:/srv/config.yml ghcr.io/imranismail/bff:v0.5.0

Send a request using the `httpbin` anything endpoint

$ curl 'http://localhost:5000/params/1' -H 'access-token: this-is-an-access-token'

token_endpoint_auth_method == client_secret_post

method: POST
path: /oauth2/token
accept: application/json
content-type: application/x-www-form-urlencoded
---
grant_type: string
code: string

Keybase proof

I hereby claim:

I am imranismail on github.
I am imranismail (https://keybase.io/imranismail) on keybase.
I have a public key ASBLClmsBnZA7O-DyJ_LpLeIud7cHMMK153ZGO-Hp_dHiwo

To claim this, I am signing this object:

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: myapp
	spec:
	selector:
	matchLabels:
	app: myapp
	template:
	metadata:

	#! /usr/bin/env bash

	# This script is used to patch all argocd apps in a cluster
	# the patch will add a new annotation to all apps
	# the annotation will be used to prevent the app from being refreshed unnecessarily
	# the following annotation will be added to all apps
	# argocd.argoproj.io/manifest-generate-paths: .

	set -euo pipefail

	# Allow neighbor cache entries to expire even when the cache is not full
	net.ipv4.neigh.default.gc_thresh1 = 0
	net.ipv6.neigh.default.gc_thresh1 = 0

	# Avoid neighbor table contention in large subnets
	net.ipv4.neigh.default.gc_thresh2 = 15360
	net.ipv6.neigh.default.gc_thresh2 = 15360
	net.ipv4.neigh.default.gc_thresh3 = 16384
	net.ipv6.neigh.default.gc_thresh3 = 16384

	class DataManager {
	_listeners = [];

	constructor(initialData) {
	this._data = initialData;
	}

	dispatch = (data) => {
	this._data = data;