gistfile1.txt

import socket
import sys
import pickle
import redis

"""
Basically, this bad boy needs to receive 500k pickled metrics of the type [name, (timestamp, 
value)] every 10 seconds. It needs to then unpickle them and pop them in Redis. When you fire 
it up, it works for about five seconds (just listening - not even Redising yet), and then it 
stops, with a weird exception in gen_unpickle:
'\x00'
'\x00'
'\x93'
'\x0b'

So I have no idea what's going on.
"""
def gen_unpickle(infile):
    while True:
        try:
            item = pickle.load(infile)
            yield item
        except Exception as e:
            print e

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind((socket.gethostname(), 2014))
s.listen(5)

r = redis.StrictRedis(host='localhost', port=6379, db=0)
r.flushall()

while True:
    (conn, address) = s.accept()

    try:
        print >>sys.stderr, 'connection from', address

        while True:
            for item in gen_unpickle(conn.makefile()):
            	for metric in item:
            	    name = "metric." + metric[0]
            	    print name
            	    #r.rpush(name, metric[1])

    except Exception as e:
        print e
    finally:
        conn.close()

I'm not certain, but I think

 for item in gen_unpickle(conn.makefile()): # (ln 38)

makefile needs to be on the socket itself, not the connection?

 for item in gen_unpickle(s.makefile()): # (ln 38)

ha! totally ln 41 not 38, need moar coffee

yeah the unpickling works for a hot second, and it prints out all the metrics. but then it fucks up

I'm suspicious of "yield" here (ln 22), does return work? or why are you using yield (so I can understand)

we're passing it a makefile, so it can continually grow while in the gen_unpickle function. these are streams, remember

right I get that, I just haven't seen it used outside of an iterable before. I figured the buffer would take care of that rather than using yield. anyways, way over my head. good luck!