Last active
August 29, 2015 14:27
-
-
Save Markbnj/c4ecefbc5a6f248a192b to your computer and use it in GitHub Desktop.
User ipdb to debug a running spider in scrapyd
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # The scrapyd daemon adds a lot of power and flexibility to scrapy, but it also adds a layer of indirection | |
| # (another process) that can make debugging a running spider troublesome. Use this code to launch scrapyd | |
| # in ipdb (or pdb). You'll be able to breakpoint and catch exceptions inside the spiders it launches. | |
| import scrapyd.runner | |
| import ipdb | |
| ipdb.runcall(scrapyd.runner.main) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment