Trevor Hartman devth

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

Producers send messages to brokers
Consumers read messages from brokers
Messages are sent to a topic

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000

Next Steps

Measure time spend on index, flush, refresh, merge, query, etc. (TD - done)
Take hot threads snapshots under read+write, read-only, write-only (TD - done)
Adjust refresh time to 10s (from 1s) and see how load changes (TD)
Measure time of a rolling restart doing disable_flush and disable_recovery (TD)
Specify routing on query -- make it choose same node for each shard each time (MD)
GC new generation size (TD)
Warmers

measure before/after of client query time with and without warmers (MD)

	;; see http://stackoverflow.com/questions/16386793/datomic-query-function-that-filters-and-binds

	(require '[datomic.api :as d])
	(def uri "datomic:mem://test")
	(d/create-database uri)
	(def conn (d/connect uri))

	;; sample attribute definiton
	(d/transact conn [{:db.install/_attribute :db.part/db
	:db/id #db/id[:db.part/db]

	Teacup::Stylesheet.new :main do
	style :my_style,
	constraints: [
	constrain_left(0),
	constrain_width(100),
	constrain_top(0),
	constrain(:bottom).equals(:superview, :bottom),
	]
	end

	/*
	Copyright 2018 Viktor Klang

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software

	# download
	wget http://downloads.datomic.com/0.8.3848/datomic-pro-0.8.3848.zip
	unzip datomic-pro-X.X.zip & cd datomic-pro-X.X

	# install as a maven artifact
	./bin/maven-install

	# configure transactor.properties
	cp config/samples/sql-transactor-template.properties transactor.properties
	# protocol=sql

	(require
	'[pallet.crate.git :refer [git clone]]
	'[pallet.crate.java :refer [java]]
	'[pallet.crate.lein :refer [lein leiningen]])

	(def repo "git://github.com/jcrossley3/random-apps-of-kindness.git")

	(def demo-directory "random-apps-of-kindness/demo")

	(defplan setup-machine

	class MP

	class << self

	def init_mixpanel
	@mixpanel = Mixpanel.sharedInstance
	self.identify(@mixpanel)
	@mixpanel.people.set({"$last_login" => Time.now})
	MP.update_person(@mixpanel)
	end

	(set
	(mapcat #(...
	...


	;; AND


	(into {}
	(map #(vector ...