Skip to content

Instantly share code, notes, and snippets.

View DennisFederico's full-sized avatar

Dennis Federico DennisFederico

View GitHub Profile

Event Deduplication in Kafka Stream processing using ksqlDB (SIMPLIFIED)

Introduction

Event deduplication emiting the very first message and filtering the rest of the duplicates within a tumbling window or a session window, using ksqlDB.

The ID to deduplicate is the eventId field inside the record payload, thus the first step is to re-key the stream by this field.

The deduplication is done by counting the number of times the eventId appears in the stream within a window, and only emitting the first event that has a count of 1.

@dcode
dcode / GitHub Flavored Asciidoc (GFA).adoc
Last active March 20, 2025 13:24
Demo of some useful tips for using Asciidoc on GitHub

GitHub Flavored Asciidoc (GFA)