Phillip Calvin pnc

Kinesis Firehose Survival Guide

Avoid regular Kinesis if you can

It's every bit as complicated to build software against as Apache Kafka and reasoning about repartioning, retry, and failover is a huge pain and only matters if you have TB/s of data. Kinesis Firehose will happily push many, many MB/s or maybe even GB/s without the complexity.

Kinesis Firehose is pay-by-the-drink (price per byte) and pretty darn cheap. Real Kinesis (price per hour) gets expensive fast, since it involves provisioned infrastructure. Ditto Kinesis Data Analytics. One more reason not to use it.

If you find yourself writing a Kinesis Firehose consumer, run screaming into the night. Here is the first half of the steps to connect to Kinesis and consume data:

The KCL acts as an intermediary between your record processing logic and Kinesis Data Streams. The KCL performs the following tasks:

This one is a real challenge! The wide design of the Grand Army Plaza circle, combined with fairly long light cycles, permits drivers to feel safe going 40-50mph, and brings them off the circle onto Vanderbilt (and into this intersection) at speed. This intersection marks the start of significant bicycle (rare and unexpected to motorists two-way bike traffic on Plaza St E) and pedestrian (four crosswalks) activity not present on the stretch of Grand Army Plaza circle from the last light until this light. This shift from parkway/highway-like design in the traffic circle to residential street design surprises drivers, and they often have no time to adjust their speed appropriately when the light is green.

Additional issues at this intersection:

Drivers at speed going northbound cannot see pedestrians or cyclists entering the intersection from the east side due to the advertising portion of the MTA bus stop and the allocation of the corner parking spot. Physically blocking this with a low planter or parking

Prepare to spend a little cash on a wild ride. You are about to go from an expensive, Jet-A-enriched glass-and-steel shopping mall to an acoustic-tile-ceilinged underground train pit with a bowling alley on top.

Load up Google Maps and pick the result that has you taking AirTrain to Jamaica, and then a cluster of colorful names including Babylon, Hempstead, Long Beach, Port Jefferson, and Ronkonkoma to Penn Station.

🍹 True fact: Ronkonkoma isn't a real place!

Once in the baggage area of any of the terminals follow the signs for Ground Transportation and then AirTrain. Board (ticketless) a Jamaica Station-bound AirTrain. Do not board a Howard Beach-bound AirTrain. Once at Jamaica Station, swarm with the others to purchase a MetroCard of at least $5 value. I recommend just getting a $20 (plus $1.4536̅6̅ bonus!!!) card for your time here. The UI will try to create a false distinction between an "AirTrain MetroCard" and a normal MetroCard. Do not be fooled. You simply need a card with at least $5 v

Just about the only thing that'll crash the Erlang VM is running out of memory. Although processes themselves are cheap, a good way to run out of memory is to start an unbounded number of processes from an external signal (such as one per web request), and then make each of those processes hold onto some more expensive resource. If you're not careful about cleanup, you can also leak processes, which compounds the problem. Anyway, nobody's perfect, so to avoid these cases, we use the poolboy library in a bunch of places to handle process pools that handle cleanup and limit the total amount of concurrency to some reasonable amount.

When:

all a pool's workers are checked out (by actual organic load/concurrency)
some process is leaking pool workers by not checking them back in (rude)
the pool supervisor isn't responding (it's deadlocked, bugs, etc.)

you'll see "checkout" timeouts (waited longer than the timeout to check out a worker from the pool):

	-- run with `sqlite3 -readonly ~/Library/Messages/chat.db`
	with top as (
	select
	chat.display_name,
	associated_message_guid,
	associated_message_type,
	count() -- count()
	from
	message
	left outer join handle on handle_id = handle.rowid

	# Simulate a fairly realistic "bad network" (e.g. bufferbloat,
	# flaky Wi-Fi, LTE from a brick building) to help find bugs
	# in code that never expects network responses to arrive out of order.

	# Run with `sudo ruby network-chaos.rb` and stop with control-C.
	# Test with:
	#
	# $ ping 1.1.1.1
	# PING 1.1.1.1 (1.1.1.1): 56 data bytes
	# Request timeout for icmp_seq 0

	npm install --save --build-from-source canvas on master

	> [email protected] install /Users/phil/Development/creator/node_modules/canvas
	> node-pre-gyp install --fallback-to-build

	node-pre-gyp WARN Using needle for node-pre-gyp https download
	gyp ERR! configure error
	gyp ERR! stack Error: Command failed: /usr/local/opt/python/libexec/bin/python -c import sys; print "%s.%s.%s" % sys.version_info[:3];
	gyp ERR! stack File "<string>", line 1
	gyp ERR! stack import sys; print "%s.%s.%s" % sys.version_info[:3];

	#!/bin/bash

	set -e

	CONTENTS=$(tesseract -c language_model_penalty_non_dict_word=0.8 --tessdata-dir /usr/local/share/tessdata/ "$1" stdout -l eng \| xml esc)

	hex=$((cat <<EOF
	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
	<plist version="1.0">

	phil@Supernova > java -jar lib/swagger-codegen-cli.jar version
	3.0.0-SNAPSHOT
	phil@Supernova > java -jar lib/swagger-codegen-cli.jar generate -v -i specs/client/petstore.yaml -l swagger -o output/client/petstore
	[main] INFO io.swagger.codegen.config.CodegenConfigurator -
	VERBOSE MODE: ON. Additional debug options are injected
	- [debugSwagger] prints the swagger specification as interpreted by the codegen
	- [debugModels] prints models passed to the template engine
	- [debugOperations] prints operations passed to the template engine
	- [debugSupportingFiles] prints additional data passed to the template engine
	[main] INFO io.swagger.parser.Swagger20Parser - reading from specs/client/petstore.yaml

	### Keybase proof

	I hereby claim:

	* I am pnc on github.
	* I am pnc (https://keybase.io/pnc) on keybase.
	* I have a public key whose fingerprint is CAE3 A08A 3CFB 5042 82BD 80B8 37B7 21F0 DAE6 961F

	To claim this, I am signing this object: