Notes for My talk at RedisConf

gistfile1.txt

Hi!
















@aaronblohowiak
[email protected]
These notes will be / were published right after the talk.
10/22/2012




















WHOAMI
	Conributed to redis-doc: fdc12e2a214f044cb966652ccfcce5641361c41f :D
	Founder of u2601.com, "Best cloud provider."
	Sponsor of the.endoftimezones.com, "Most important political movement ever"
	Instigator of EVALSHA.com, "An OSS repo of public domain Redis scripts"
	64 public repos on github, most are useless ;D
	Independent contractor.




















EVAL, EVALSHA and EVALSHA.com
	* Introduction to EVAL & EVALSHA
	* Lua in Redis
	* Examples of Lua Scripting
	* Caveats
	* EVALSHA.com

























Introduction to EVAL & EVALSHA:
	* Lua _in_ redis-server
	* Uninteruptable kind of atomic
	* Cached procedures vs Stored procedures
	* Good client support already
	* Interesting OSS projects
		- http://evalsha.com
		- https://github.com/andyet/redis-lua-example
		- https://github.com/Shopify/wolverine

	* Resources:
		- http://redis.io/commands/eval !!! This is the best
		- http://evalsha.com/documentation
		- http://redis.io/commands#scripting
		- http://tjholowaychuk.com/post/19321054250/redis-lua-scripting-is-badass
		- http://blog.vishalshah.org/post/22175246910/redis-lua-for-processing-json-values
		- http://dan.lecocq.us/wordpress/2012/03/11/redis-and-lua-for-robust-portable-libraries/



























Lua in Redis:
	* Easy to call redis:
		- redis.call(CMD, [...ARGS...]) (raise)
		- redis.pcall(CMD, [...ARGS...]) (error table)
		- redis.log(loglevel,message)
	* No Globals
		- variable declaration: 'local myvar = redis.call("GET", KEYS[1])'
	* JSON + MSGPACK
	* Datatypes auto-converteted
		- Lua Number is like JS Number
		- Redis converts Number to Int! (use tostring(num))
		- Lua Array-likes => Multi-Bulk
		- Lua Hash-likes => :(
	* The *script* is replicated, not its write commands
		- BW - preserving, but complicates usage
	* Deterministic OR Readonly
		- math.random always the same sequence
		- math.randomseed through ARGV
		- TIME / SRANDOMKEY / SRANDMEMBER: Readonly!
	* Develop using ruby client:
		r.eval(File.read(filename), keys, args)


















Thoonk from &Yet:
	Convention-based event routing and dispatching system.
	Gives you a pattern for publishing events and for managing subscriptions to them.
	In your app, you define your event schema and the lua scripts that run in redis.. your object then "Lives in Redis."
	If you use an object with these keys with these patterns, you get things for free.
	RPC System, where you are remotely executing a locally defined script for each method on an object.
	You should check it out!
		- http://github.com/andyet/thoonk.js
		- http://github.com/andyet/thoonk.py
		- AndBang is implemented using it
		- Questions? Ask @lancestout (py) @fritzy (js)


















Why not Watch/Multi/Exec:

	* Optimistic: while(fail?) retry
	* Fewer network roundtrips
	* Continues even if there are errors.
		redis> multi
		OK
		redis> set hi 'hello'
		QUEUED
		redis> rpop hi
		QUEUED
		redis> set okay 'done'
		QUEUED
		redis> exec
		1. OK
		2. (error) ERR Operation against a key holding the wrong kind of value
		3. OK
		redis> get okay
		"'done'"
		redis> get hi
		"'hello'"
		redis>


























Example: ZPOP WME vs EVAL:
	* ZPOP is LPOP for ZSET
	* (Remove lowest item and return it)

	WME: (from redis.io)
		WATCH zset
		element = ZRANGE zset 0 0
		MULTI
		ZREM zset element
		EXEC

	EVAL: http://evalsha.com/scripts/00c37234520b7295684ccc128636b2413132684a
		local elm = redis.call("ZRANGE", KEYS[1], 0, 0)[1]
		if elm then
			redis.call("ZREM", KEYS[1], elm)
		end
		return elm

























Example: CAS
	* Antirez thinks its dumb in redis https://groups.google.com/forum/?fromgroups=#!msg/redis-db/a4zK2k1Lefo/OU9ewOJro5MJ
	* But it is a simple example
	* http://evalsha.com/scripts/4298aecc807ec90f8bd4ec7f7006563cee4f7a1b
		local current = redis.call('GET', KEYS[1])
		if current == ARGV[1]
		then
			return redis.call('SET', KEYS[1], ARGV[2])
		end
		return false




























Example: Publish to multiple channels
	* publish a message to multiple channels
	* avoid resending the message to redis
	* http://evalsha.com/scripts/d639c8a2713f88d88d3ad6900e05a64a99aa3ead
		local clients=0
		for i,v in ipairs(ARGV) do 
			if i > 1 then
				clients = clients + redis.call("PUBLISH", v, ARGV[1])
			end
		end
		return clients


























Example: Disconnected Subscriptions (mailboxes)
	* Support connected & disconnected subscriptions
	* Publish to channel & to mailboxes
	* Single-Server only.
	Lua script: http://evalsha.com/scripts/67e165f02e2830b2450acba0fae439a704edb515
		local chan = KEYS[1]
		local result = {}
		result[1] = "clients"
		result[2] = redis.call("PUBLISH", KEYS[1], ARGV[1])
		result[3] = "boxes"

		local boxes = redis.call("LRANGE", KEYS[1], 0, -1)
		local boxCount = 0
		for i,v in ipairs(boxes) do
			redis.call("RPUSH", v, ARGV[1])
			boxCount = boxCount + 1
		end
		result[4] = boxCount
		return result
	Results:
		redis 127.0.0.1:6379> EVALSHA 7e992b1c3e547703c84e3901c4847b6ca92e84ce 1 News "My news!!!"
		1) "clients"
		2) (integer) 0
		3) "boxes"
		4) (integer) 2
	Clients:
		LTRIM listname 0 -1 to poll and/or BLPOP in a loop

















Example: HMGETALL (like Crashalytics' Jeff wants):
	- Get all keys for all hashes
	Lua:
		local result = {}
		local hshValue = nil
		for i,v in ipairs(KEYS) do
			hshValue = redis.call("HGETALL", v)
			result[i] = hshValue
		end
		return result

	Example in Use:
		redis 127.0.0.1:6379> HMSET hsh1 clicks 10 crashes 0
		OK
		redis 127.0.0.1:6379> HMSET hsh2 clicks 100 crashes 10
		OK
		redis 127.0.0.1:6379> HMSET hsh4 clicks 300 crashes 10
		OK
		1.9.2p320 :030 > r.eval(File.read('mhgetall.lua'), %w{hsh1 hsh2 hsh3 hsh4})
		 => [["clicks", "10", "crashes", "0"], ["clicks", "100", "crashes", "10"], [], ["clicks", "300", "crashes", "10"]] 
















Caveats. AKA :(
	* In Lua a[1]..a[N]
	* Full script sent to slave/AOF (currently) scripting.c:867
	* No Blocking operations!
	* Single-server issues
	* All scripts kept forever by default.
	* No include/require: &Yet uses the c preprocessor to include multiple functions
	* Testing this stuff is hard
	* Error reporting =(
	* Implicit sort:
		- SMEMBERS
	* What if script loops forever? ( BUSY )
		- If readonly:  SCRIPT KILL
		- else: SHUTDOWN NOSAVE
	* Logging is very expensive, churns fds
		- redis.c:272
		- fp = (server.logfile == NULL) ? stdout : fopen(server.logfile,"a");
		- Don't log a lot in production!
	* ASCIIZ-style arrays: First "nil" in Lua Array marks its end =(
		- 1.9.2p320 :033 > r.eval("return {1,2,nil,3}", [])
		=> [1, 2]

























EVALSHA.com:
	* Free/Open-Source repo of public domain scripts
	* Based on redis.io code from CitrusByte
	* Search using elasticsearch
	* Needs your help!
		- github.com/aaronblohowiak/evalsha.com
	* What would you like to see added to this?
	* Add other-licenses?

















Questions?