Mohan Balasundaram tuxfight3r

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

## Get private key fingerprint for AWS generated key ( will match the id in AWS Console)
$ openssl pkcs8 -in test_key.pem -inform PEM -outform DER -topk8 -nocrypt | openssl sha1 -c
(stdin)= f6:38:9c:53:9c:64:a8:b3:40:23:9f:6c:ed:0e:3d:bf:bf:16:bb:1c

## Get fingerpint from Openssh public key only md5 / sha1
$ ssh-keygen -f test_key.pub -e -m PKCS8 | openssl pkey -pubin -outform DER | openssl md5 -c
(stdin)= 55:b6:90:db:34:29:da:60:68:c9:08:24:f1:9f:c8:4b

(or)

use kubectl to patch configmap via json patch operations (add/remove/replace/test)

## Add a new key to the existing configmap
$ kubectl patch cm demo-app-config --type json --patch '[{ "op": "add", "path": "/data/TEST_VALUE", "value": "test_key" }]'
configmap/demo-app-config patched

$ kubectl get cm demo-app-config -o json | jq .data.TEST_VALUE
"test_key"

Kafka Cheat Sheet

Display Topic Information

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic beacon
Topic:beacon	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: beacon	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: beacon	Partition: 1	Leader: 1	Replicas: 1	Isr: 1

Utilities you'll care about

All these are already installed on epyc.

kafkacat (conda install -c conda-forge kafkacat)
kt (grab it from https://github.com/fgeller/kt/releases)
kafka-* (come with kafka, if you yum install if from Confluent's repo, or via Docker if you're so inclined). Warning -- JVM based and dreadfully slow.
jq (conda install -c conda-forge jq or use your favorite package manager)

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

Virtual Hosts on nginx (CSC309)

When hosting our web applications, we often have one public IP address (i.e., an IP address visible to the outside world) using which we want to host multiple web apps. For example, one may wants to host three different web apps respectively for example1.com, example2.com, and example1.com/images on the same machine using a single IP address.

How can we do that? Well, the good news is Internet browsers

	#!/bin/bash

	HEIGHT=10
	WIDTH=40
	CHOICE_HEIGHT=4
	BACKTITLE="Cluster Options"
	TITLE="Select Cluster"
	MENU="Choose one of the following options:"

	ENV="${ENV:-dev}"

	#code: https://github.com/miguno/avro-cli-examples
	# get schema
	java -jar ../avro-tools-1.10.1.jar getschema twitter.avro

	# get meta
	java -jar ../avro-tools-1.10.1.jar getmeta twitter.avro


	#create avro
	java -jar ../avro-tools-1.10.1.jar fromjson --schema-file twitter.avsc twitter.json > twitter.avro


	# generate a key if you don't have one already
	gpg --full-gen-key

	# list the keys in long format and copy the key id after sec rsa4096/<KEY>
	gpg --list-secret-keys --keyid-format LONG

	# export the public key and post into your git profile (ui)
	gpg --armor --export 127CAA89CEA6BE66