SQLite as a Background Job Queue.md

Features
- Schedule jobs
- Schedule jobs in the future
- Explicit failure: An exception was thrown
- Implicit failure: A job was picked up and not finished in a certain amount of time
- Retry jobs that failed (explicitly or implicitly)
- After a certain number of retries, fail the job
- Have a way to force the worker to run immediately
- On graceful termination finish existing jobs
Questions
- Where do we implement this?
  - node
    - Con: Needs to be aware of sqlite and server
      - Circular dependency, because server already depends on node
  - sqlite
    - Con: Needs to bring node in, which has its own signal handler stuff
  - Separate package
    - Con: A lot of fragmentation
- Separate database (better performance) or same database (ease of management)?
- How do we run the migration to create the schema?
  - On migrate()
  - On backgroundJob()
- How to distribute work within a backgroundJob process
  - Different backgroundJob() per type
    - Pro: Simpler to implement
    - Con: Too many jobs of different kinds may lead to thrashing
    - Pro: If a certain type of job is too heavy, it doesn’t overwhelm a single process
  - Poll of workers
    - Pro: May work on multiple jobs of the same type, increasing throughput
    - Con: You’d have to poll too often
Future
- Force a worker to run immediately
  - Watch for filesystem changes
  - ZeroMQ to communicate between workers
  - Redis or something like that
References

leafac/SQLite as a Background Job Queue.md