Skip to content

Instantly share code, notes, and snippets.

@bb01100100
bb01100100 / simple-topic-counter.sh
Last active October 9, 2023 17:27
Count Kafka topic messages using unix cli tools
#!/usr/bin/env bash
# Author: Kel Graham
# Date: 2019-12-04
# Purpose: Let's use Kafka CLI tools to get topic counts.
# This should be usable on any Kafka install since it uses
# only the kafka-consumer-groups utility and some standard
# unix tools to sum up.
@bb01100100
bb01100100 / kafka-change-replication-factor
Last active October 7, 2024 04:47
Change the replication factor for an existing Kafka topic by nominating a new set of replicas
#!/bin/bash
# Author: Kel Graham
# Date: 2022-05-19
# Purpose: Increase the replication factor of one or more topics, using only
# a connection to the Kafka broker (no Zookeeper, REST APIs etc)
# and the standard kafka-topics, kafka-reassign-partitions scripts
# that come with Kafka.
#
@bb01100100
bb01100100 / traffic-control.sh
Created March 17, 2021 23:59
Introduce networking delay, packet loss, or restrict bandwidth to specified destinations on a linux host.
#!/usr/bin/env bash
# Author: Kel Graham
# Date: 2021-03-17
# Purpose:
# This script uses the linux traffic control `tc` command to add delay, drop packets, etc
# which is useful when we want to test software behaviour under stress conditions, e.g.
# such as spiking latency between Zookeeper or Kafka nodes, simulating a data centre outage
# or restricting bandwidth between components to simulate inter-DC resource contention.
@bb01100100
bb01100100 / cp-curl.sh
Last active March 30, 2021 01:40
Simplify repetitive curl args on Confluent Platform ansible deployed components
#!/usr/bin/env bash
# Author: Kel Graham
# Date: 2021-03-30
# Purpose:
# This script wraps curl with caroot, cert and key files obtained from the standard
# /var/ssl/private folder used by cp-ansible for deployed Confluent Platform components.
# The idea is to allow you to get on with hitting the Schema Registry REST endpoing
# without faffing around providing repetitive cert args.
# Does some super basic checks if we are registering Schemas on SR and provides the
@bb01100100
bb01100100 / zk-4lw-from-bash.sh
Created April 10, 2022 23:59
Use bash's tcp pseudo device for fun and profit
#!/usr/bin/env bash
# We set up a file descriptor to redirect input/output from bash's tcp pseudodevice
# This allows us to, for example, call Zookeeper 4lws without needing netcat
# Works everywhere that bash works (proven on RHEL 7.x, AIX, MacOS bash 3.2)
ZKHOST="$1"
ZKPORT=2181
ZKCMD="$2"
@bb01100100
bb01100100 / build.sh
Last active September 2, 2022 03:38
Build kcat on RHEL/Centos
# Confirmed on Centos 8.5 (in Sept 2022)
sudo yum group install -y "Development Tools"
sudo yum install -y git openssl-devel cmake libcurl-devel
git clone https://github.com/edenhill/kcat.git
cd kcat
./bootstrap.sh
@bb01100100
bb01100100 / summarise-acls.jq
Created March 30, 2023 04:18
jq - summarise Confluent Cloud service account topic access
(INDEX($sa[] ; .id) | map_values(.name)) as $dict
| [(.[] | select(.principal[:7] == "User:sa"))]
| group_by(.principal)
| map(
{
(first.principal[5:] as $p
| ($dict[$p] // ("ACL-for-non-existant-service-account: " + $p))
): map(
((.resource_type | ascii_downcase) + "/"+ .resource_name)
)