Rollbar log observer for Scrapy / Python

log.py

import rollbar
import logging

# Needed if you use rollbar.report_exc_info
import sys

class RollbarLogObserver(object):
    def __init__(self, rollbar_access_token, env="development", level=logging.INFO):
        self.level = level
        rollbar.init(rollbar_access_token, env)

    def emit(self, event={}):
        if event['isError']:
            if 'failure' in event:
                text = event['failure'].getTraceback()
            else:
                text = " ".join([str(m) for m in event["message"]]) + "\n"

            # You could also add more infor and exec info like this.
            # rollbar.report_exc_info(sys.exc_info(), {})

            extra_data = {}

            if event['spider']:
                spider = event['spider']
                
                # Some usefull data like class, name and 'why', useful for debugging.
                extra_data = {
                    'class': str(spider.__class__.__name__),
                    'name': spider.name,
                    'why': event['why']
                }

            rollbar.report_message(text,
                                   level=self.level,
                                   extra_data=extra_data)


# To add Rollbar to your Scrapy project add this to settings.py
# ... settings ...
from scrapy import log
from collector.log import RollbarLogObserver # You could also save it somewhere else
log.log.addObserver(RollbarLogObserver("token goes here", env="production", level=LOG_LEVEL).emit)

Hey! Brian from Rollbar here -- I found this via a Google alert.

A quick suggestion: for the case where you do have a traceback, use the rollbar.report_exc_info() form. This way you'll send a nicely structured traceback to Rollbar, and that will make a lot of things work better (grouping, UI, etc.).

Thanks for such good receipt, also I've tried rollbar.report_exc_info() approach but it crashes:

File "/tmp/eggs-19WeYZ/rollbar.egg/rollbar/init.py", line 716, in _add_locals_data
args.append(_local_repr(local_vars[named_arg]))
TypeError: unhashable type: 'list'