Skip to content

Instantly share code, notes, and snippets.

View markcode's full-sized avatar

Mark Ashcroft markcode

21 followers · 29 following
View GitHub Profile
@markcode
markcode / notes.md
Last active June 29, 2020 23:55
Scaling Event Logs and Hash Partitioning Function

Scaling Event Logs and Hash Partitioning Function

Setup

A COLLECTION is created in a CLUSTER (of BROKERS) with a defined and fixed number of potential PARTITIONS. For example: 12 data log files (AKA PARTITIONS), each with 256 VIRTUAL-PARTITIONS within; totalling 3072 potential PARTITIONS.

ERA is an incrementing and synchronous COLLECTION of the PARTITIONS. The starting ERA is 1, and the next will be 2; and so on.

Each BROKER is a member of a CLUSTER; whenever a BROKER joins or leaves a CLUSTER a new ERA is declared across all BROKERS within the CLUSTER. The old ERA’s PARTITIONS are closed from writes and new PARTITIONS are created for writing.