My first shot at fixing this was in tasks.py file below.
But then that just gave a "unhandled error in deferred", so I went on to use CrawlRunner.
That showed no output at all anymore, and didn't run as expected.
Eventually, I just settled on CELERY_WORKER_MAX_TASKS_PER_CHILD=1=1
in settings.py
Note: CELERY_WORKER_MAX_TASKS_PER_CHILD=1
is for django. Celery without django probably drops the CELERY_
prefix