Davide Icardi davideicardi

Package a Python Lambda for AWS

Create Python virtual env

python3 -m venv v-env

Activate

aws kinesis get-shard-iterator --stream-name YOUR_STREAM_NAME --shard-id 0 --shard-iterator-type TRIM_HORIZON

aws kinesis get-records --shard-iterator YOUR_SHARD_ITERATOR

Add the following library:

libraryDependencies += "org.bouncycastle" % "bcprov-jdk16" % "1.46"

There are 4 possible serialization format when using avro:

Avro Json encoding
Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

Allow the default service account to read pods data, by adding a role binding:

TODO: Now the yaml works only for the dev1 namespace, how can I make it more generic?

kubectl apply -f ./add-read-pods-binding.yaml

Assume that you have the following directory structure:

	# https://github.com/wurstmeister/kafka-docker
	# https://medium.com/big-data-engineering/hello-kafka-world-the-complete-guide-to-kafka-with-docker-and-python-f788e2588cfc

	version: '3'
	services:

	zookeeper:
	image: wurstmeister/zookeeper:3.4.6
	ports:
	- "2181:2181"

	/*
	Calculate project's version using GIT tags and commits.

	Main idea is to avoid having a version written in the source code but instead link it to the GIT repository.
	It is similar to sbt-git plugin but increment the PATCH part when there are additional commits.

	The version is calculated using the last tag and the last commit (using `git describe` output).
	If there is at least one commit after the last tag the PATCH part is incremented by 1
	and `-SNAPSHOT` is added as a suffix.
	Otherwise the tag is used as is.

	#!/bin/bash
	set -e

	NEW_SOURCE=1
	DEFAULT_SOURCE=`pacmd list-sources \| grep '* index' \| cut -f5 -d' '`
	if [ $DEFAULT_SOURCE -eq 1 ]
	then
	NEW_SOURCE=0
	fi

	const fs = require("fs");
	const { Transform, PassThrough } = require('stream');

	function concatStreams(streams) {
	let pass = new PassThrough();
	let waiting = streams.length;
	for (let stream of streams) {
	pass = stream.pipe(pass, {end: false});
	stream.once('end', () => --waiting === 0 && pass.emit('end'));
	}