gistfile1.txt

import socket
import sys
import pickle
import redis

"""
Basically, this bad boy needs to receive 500k pickled metrics of the type [name, (timestamp, 
value)] every 10 seconds. It needs to then unpickle them and pop them in Redis. When you fire 
it up, it works for about five seconds (just listening - not even Redising yet), and then it 
stops, with a weird exception in gen_unpickle:
'\x00'
'\x00'
'\x93'
'\x0b'

So I have no idea what's going on.
"""
def gen_unpickle(infile):
    while True:
        try:
            item = pickle.load(infile)
            yield item
        except Exception as e:
            print e

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind((socket.gethostname(), 2014))
s.listen(5)

r = redis.StrictRedis(host='localhost', port=6379, db=0)
r.flushall()

while True:
    (conn, address) = s.accept()

    try:
        print >>sys.stderr, 'connection from', address

        while True:
            for item in gen_unpickle(conn.makefile()):
            	for metric in item:
            	    name = "metric." + metric[0]
            	    print name
            	    #r.rpush(name, metric[1])

    except Exception as e:
        print e
    finally:
        conn.close()

we're passing it a makefile, so it can continually grow while in the gen_unpickle function. these are streams, remember

right I get that, I just haven't seen it used outside of an iterable before. I figured the buffer would take care of that rather than using yield. anyways, way over my head. good luck!