I've been working with Apache Kafka for over 7 years. I inevitably find myself doing the same set of activities while I'm developing or working with someone else's system. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now. 🔥
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- The goal of this script is to trim messages that have been processed by | |
-- all extant groups from the a given Redis stream. It returns the number | |
-- of messages that were deleted from the stream, if any. I make no | |
-- guarantees about its performance, particularly if the stream is large | |
-- and not fully processed (so a simple XTRIM isn't possible). | |
-- First off, bail out early if the stream doesn't exist. | |
if redis.call("EXISTS", KEYS[1]) == 0 then | |
return false | |
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env bash | |
set -e | |
set -u | |
set -o pipefail | |
show_help() { | |
cat << EOF | |
Usage: $(basename "$0") <options> | |
-h, --help Display help |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function kubernetes_status | |
[ -z "$KUBECTL_PROMPT_SEPARATOR" ]; and set -l KUBECTL_PROMPT_SEPARATOR "/" | |
set -l config $KUBECONFIG | |
[ -z "$config" ]; and set -l config "$HOME/.kube/config" | |
if [ ! -f $config ] | |
echo (set_color red)$KUBECTL_PROMPT_ICON" "(set_color white)"no config" | |
return | |
end | |
set -l ctx (kubectl config current-context 2>/dev/null) |
- boto3 initializes a session using the specified profile, for which it assumes a role as configured in your
~/.aws/config
- Python script with above session initialization prints out shell-compatible environment variables of the temporary credentials
- Wrapper script sets these a la eval
- By the time Ansible runs, the
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
andAWS_SECURITY_TOKEN
(for boto2) /AWS_SESSION_TOKEN
(for boto3) are all set, and are consumed by boto2 in the inventory script and other boto2-based modules
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
set -eu | |
type aws > /dev/null 2>&1 && type jq > /dev/null 2>&1 || { | |
echo 'Required aws-cli and jq' | |
exit 1 | |
} | |
repositories=$(aws ecr describe-repositories --query 'repositories[*].repositoryName' | tr -d "[\",\t\n\r\"]") |
This script can help you find and remove unused AWS snapshots and volumes.
There is hardcoded list of regions that it searches, adjust the value to suit your needs.
Use snapshot.py snapshot-report
to generate report.csv
containing information about all snapshots.
snapshot.py snapshot-cleanup
lets you interactively delete snapshot if it finds it is referencing unexisting resources.
./snapshots.py --help
btauth --user USER_OR_EMAIL --pass PASSWORD # To Authenticate with the BT WiFi
btmaintain --user USER_OR_EMAIL --pass PASSWORD # To maintain a connection and re-authenticate automatically
nohup btmaintain --user USER_OR_EMAIL --pass PASSWORD 2>&1 >/dev/null & # Same as above but ran in background