RedisConf Notes Part I

gistfile1.txt

Redisconf Notes:

"We will begin this morning with a performance"
	"Is this a key store..." Bohemian Rhapsody Key/Value Store ballad.  Epic Win!
	"Any way the data flows doesn't really matter to me... flushdb..."
	"Lua killed a db by writing after calling a nondeterministic function..."
	"Got to leave SQL behind and face the truth..."
	"Redis, I don't want to join, sometimes I wish I'd never described a query..."
	"The ZSET has some data set aside for me, for me, for MEEEEEEE"

"Redis in perspective" - Antirez.
	- Can't be here because his baby is imminent.
	- Past, present and future of Redis. "Humble project", humble because it is supposed to be small and solve one problem.
	- Redis Manifesto, written 1.5 years ago, still valid.
	- Hetzner.de - dedicated 64GB RAM for 106 euro / month.
	- Will be in memory, always, but will replicate to disk.
	- "Don't touch the code too much, don't abuse the code."
	- "Fight against complexity" - "Virtual Memory was removed"
	- Redis codebase is less than 30k LOC, including Hiredis
	- Simpler systems are easier to understand [in their failure modes]
	- Two APIs for two different worlds
		- Some commands are tricky in distributed systems
		- Many K/V dbs that are distributed are simple get/put keys
		- With set intersection, etc, it is hard to do this in distriubted way.
		- So, one is extended API for single-server, one subset that is implemented in Distributed way.
	- "Optimize for joy. Redis is simple to get started, to understand, to use."
		- "I right much of the documentation myself because I think the documentation is at least as important as the code; for the users and for me...[because it helps to identify deficiencies in the design]"
		- "Open-Source is optimized for joy"

"Two things Redis is not (and likely never will be)" - Pieter Noordhuis
	- "Redis does a few things well, and a few things not so good... and this is something people seem to forget."
	- "It is SUPER easy to get started.. and that is both a good and a bad thing."
	- "I work at VMWare where I split my time between Redis and CloudFoundry...started contributing in 2010..."
	- "I also maintain Hireds, but I'm not a python, ruby or node guy.. send pull requests if stuff doesn't work"
	- Doesn't know where we are in the Software Hype cycle
	- "1 million ops/second on a commodity box... but there are tradeoffs... and that is important to talk about"
	- "Redis is NOT an available system." (CAP theorom, it is a CP system.)
	- Half of audience uses redis master with multiple slaves, a quarter uses single-node.
	- Aware of cumatative distributed datastructures, but says it does not apply to Redis
	- Describes Sentinel and consensus wrt promoting master
	- Redis can be "highly available", but data loss still available due to replication lag
	- Responding to my question, "Right now, nothing to mitigate master dying while writing out buffer, leading some slaves to have some data that other slaves don't, so you can get inconsistent data across slaves.  In the future, there may be a command count so we can pick the slave with the highest count.."
	- "Redis does NOT grow beyond RAM"  Because it is single threaded, if you have to wait for kernel to page in memory, everything blocks.

Using Redis to build analytics - Jeff Seibert from Crashalytics
	- Crashalytics reports back crash data from the line of code that your app crashed on.
	- On a quarter-billion devices right now
	- Strings for Activity Tracking. Hashes for event tracking.
	- We don't care about 2 years ago, just last few months.
	- Bitmap users to bits and mark bit when user is active in a string for a time period; one key per time period
	- Do weekly active by bitop OR'ing your days for the week, then do bitcount to sum
	- % Active last week vs this week: bitwise OR the two weeks
	- Churn: OR last N months and then NOT that from the current set.
	- Counts not worth using bitmasks, use HINCRBY, with a hash per app, key per period. Harder to expire, but easy to read (HMGET all keys for time period)
	- For grouped things, put time componant in keyname, put groups as keys in the hashes
	- "And then there's MongoDB... and that always adds a wrinkle", wants an auto-incr primary key, just like mysql
	- Uses zadd + zcard to store mongodbid->auto-incr primary key, using lua scripting to do so atomically.
	- "How do you handle persistence if it is only in redis?" - "Internal reporting, slaves accross AZs.  For production data, we have two-tiered...[speed layer, batch layer]..we're also persisting all raw data...into cassandandra [if we lost all of it, we could restore from cassandra]"
	- "How do you prune the data if it is embeded?"- "[Good question]... so far, we haven't addressed that problem..."

“He’s just a poor boy, from a poor company, spare him his life, from this MongoDB…”

The slideshow from Jeff (Crashlytics): http://www.slideshare.net/crashlytics/crashlytics-on-redis-analytics