Mark Ashcroft markcode

Scaling Event Logs and Hash Partitioning Function

Setup

A COLLECTION is created in a CLUSTER (of BROKERS) with a defined and fixed number of potential PARTITIONS. For example: 12 data log files (AKA PARTITIONS), each with 256 VIRTUAL-PARTITIONS within; totalling 3072 potential PARTITIONS.

ERA is an incrementing and synchronous COLLECTION of the PARTITIONS. The starting ERA is 1, and the next will be 2; and so on.

Each BROKER is a member of a CLUSTER; whenever a BROKER joins or leaves a CLUSTER a new ERA is declared across all BROKERS within the CLUSTER. The old ERA’s PARTITIONS are closed from writes and new PARTITIONS are created for writing.

Cryptographic Best Practices

Putting cryptographic primitives together is a lot like putting a jigsaw puzzle together, where all the pieces are cut exactly the same way, but there is only one correct solution. Thankfully, there are some projects out there that are working hard to make sure developers are getting it right.

The following advice comes from years of research from leading security researchers, developers, and cryptographers. This Gist was [forked from Thomas Ptacek's Gist][1] to be more readable. Additions have been added from

Disclaimer

I'm still very new to Kafka, eventsourcing, stream processing, etc. I'm in the middle of building my first production system with this stuff and am writing this at the request of a few folks on Twitter. So if you do have experience, please do me and anyone else reading this a favor by pointing out things I get wrong :)

Inspirations

The Log — http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Turning the database inside out — http://www.confluent.io/blog/2015/03/04/turning-the-database-inside-out-with-apache-samza/
Why local state is a fundamental primitive in stream processing — http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html
Samza
Various functional systems been exposed to over past year or so, React, Flux, RX, etc.

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

	recaman <- function(n, s=c()) {
	a <- 0

	if (n > 0) {
	val <- recaman(n - 1, s)

	a <- val$a
	s <- c(val$s, a)

	an1 <- a - n

	'use strict';

	const crypto = require('crypto');

	const ENCRYPTION_KEY = process.env.ENCRYPTION_KEY; // Must be 256 bits (32 characters)
	const IV_LENGTH = 16; // For AES, this is always 16

	function encrypt(text) {
	let iv = crypto.randomBytes(IV_LENGTH);
	let cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(ENCRYPTION_KEY), iv);

	#!/usr/bin/env python3

	# This program prints all hcn (highly composite numbers) <= MAXN (=10**18)
	#
	# The value of MAXN can be changed arbitrarily. When MAXN = 10**100, the
	# program needs less than one second to generate the list of hcn.
	from math import log

	MAXN = 10**18

	// How to connect 5 publishers with 5 subscribers
	// over TCP using ZeroMQ's XPUB/XSUB proxy.

	// sub (connect)
	// <-8701->
	// (bind) xpub <---> xsub (bind)
	// <-8700->
	// (connect) pub

	var zmq = require('zmq');

	// Require node modules.
	var http = require( "http" );
	var fileSystem = require( "fs" );
	var stream = require( "stream" );

	// Create an instance of our http server.
	var httpServer = http.createServer(
	function handleRequest( request, response ) {

	// Create our file-input stream.

	// Part of https://github.com/chris-rock/node-crypto-examples

	var crypto = require('crypto'),
	algorithm = 'aes-256-ctr',
	password = 'd6F3Efeq';

	function encrypt(buffer){
	var cipher = crypto.createCipher(algorithm,password)
	var crypted = Buffer.concat([cipher.update(buffer),cipher.final()]);
	return crypted;