Dimitris Theodorou dtheodor

Kafka producer configuration

linger.ms, alias: queue.buffering.max.ms, default: 5
how long to wait to form message batches. basic latency vs throughput
batch.size, default: 1000000
how big the batches can be. can be exceeded when a single message is bigger
batch.num.messages, default: 10000
how many messages per batch size
message.max.bytes, default: 1000000
maximum Kafka protocol request message size, 1MB

Apt

Missing pubkey

After a error such as

Err:4 https://apt.releases.hashicorp.com focal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701

x509

certificate info

openssl x509 -in <file.pem> -text -noout

certificate subject

confluent-kafka-go Consumer.Close() hangs

Consumer.Close() blocks indefinately when having multiple consumers in the same process that are consuming from the same topic and belong to the same consumer group

The go source code contains a minimal test case to reproduce

This is run with

Debian GNU/Linux 9.4 (stretch)
go version go1.10.1 linux/amd64

Following instructions at

Install

sudo yum -y install boost expect perl perl-DBI openssl zlib snappy libaio
wget https://downloads.mariadb.com/ColumnStore/1.0.8/centos/x86_64/7/mariadb-columnstore-1.0.8-1-centos7.x86_64.rpm.tar.gz

Following instructions at https://mariadb.com/kb/en/mariadb/preparing-for-columnstore-installation/

Install

sudo apt-get -y install libboost-all-dev expect perl openssl file sudo libdbi-perl libreadline-dev libdbd-mysql-perl
wget https://downloads.mariadb.com/ColumnStore/1.0.7/ubuntu/dists/xenial/main/binary_amd64/mariadb-columnstore-1.0.7-1-xenial.x86_64.deb.tar.gz
tar zxf mariadb-columnstore-1.0.7-1-xenial.x86_64.deb.tar.gz
sudo dpkg -i *.deb
sudo /usr/local/mariadb/columnstore/bin/postConfigure

Following instructions at https://mariadb.com/kb/en/mariadb/preparing-for-columnstore-installation/ (There's also https://mariadb.com/kb/en/mariadb/installing-mariadb-deb-files/, but I didn't follow that one.)

Install

sudo apt-get -y install libboost-all-dev expect perl openssl file sudo libdbi-perl libreadline-dev libdbd-mysql-perl
cd ~
wget http://downloads.mariadb.com/MariaDB/mariadb-10.1.22/repo/ubuntu/mariadb-10.1.22-ubuntu-trusty-amd64-debs.tar
cd /opt
sudo tar -xf ~/mariadb-columnstore-tars/mariadb-10.1.22-ubuntu-trusty-amd64-debs.tar
sudo ./mariadb-10.1.22-ubuntu-trusty-amd64-debs/setup_repository

	CREATE PROCEDURE profile_table_varchar_lengths @table nvarchar(256)
	AS BEGIN
	IF OBJECT_ID(@table, 'U') IS NULL
	RAISERROR ('Table %s does not exist', 16, 1, @table) WITH NOWAIT;
	DECLARE @sql nvarchar(max);

	with varchar_cols as (
	SELECT c.name, t.name as type, c.max_length
	from sys.columns c
	join sys.types t on t.system_type_id = c.system_type_id

	import requests
	from requests.adapters import HTTPAdapter
	from urllib3.util.retry import Retry


	def create_session(num_hosts: int, num_workers: int, retry: int) -> requests.Session:
	session = requests.Session()
	adapter = HTTPAdapter(
	pool_connections=num_hosts, # number of different hosts we will be talking to
	pool_maxsize=num_workers, # number of maximum concurrent requests the session will be doing (i.e. with multiple threads that share it)

	from decorator import decorator

	def retry(retries, delay, backoff, should_retry_fn, jitter=0, max_delay=None,
	on_retry_fn=None, logger=_logger, label=None):

	@decorator
	def _retry(f, args, *kwargs):
	retries_left, current_delay = retries, delay

	while retries_left: