Software Engineering :: Database :: Redis :: Solutions :: Job Queue :: Example :: A Job Queuing Platform in Redis

readme.md

Software Engineering :: Database :: Redis :: Solutions :: Job Queue :: Example :: A Job Queuing Platform in Redis

⪼ Made with 💜 by Polyglot.

• ..

assistant

• BullMQ Redis Data Inspection (ChatGPT)
• Redis Job Scheduling (ChatGPT)
• Redis Delayed Tasks Implementation (ChatGPT)

related

• Software Engineering :: Database :: Redis :: Solutions :: Job Queue
• Software Engineering :: Database :: Redis :: Solutions :: Job Queue :: About :: Redis Queue

source

• wilmoore/job (GitHub)

A Job Queuing Platform in Redis

• grafana.json
• HSET

setup metrics indexing for gists

Important

The Redisearch index ensures fast lookups for schedules due at a specific time or belonging to a specific queue.

application="gistwiz"
namespace="${application}:scheduled"
scheduleIndexName="${namespace}::metrics"

redis-cli FLUSHALL

redis-cli FT.CREATE "${scheduleIndexName}" ON HASH PREFIX 1 "${namespace}:" SCHEMA \
  id TEXT \
  queue TAG \
  ms NUMERIC SORTABLE \
  next NUMERIC SORTABLE

setup schedule gists queue

Note

Custom content is stored in the content field for each schedule. This is flexible and can include JSON strings for queue-specific payloads.

queue="gists"
scheduleId=$(uuidgen | tr -d "-")
scheduleKey="${namespace}:${queue}:${scheduleId}"

redis-cli HSET "${scheduleKey}" \
  id "${scheduleId}" \
  queue "${queue}" \
  ms 100 \
  next $(date -u +%s) \
  content '{ "token": "ghp_demo_token", "username": "wilmoore", "plan": "10s" }'

redis-cli FT.SEARCH "${scheduleIndexName}" "@next:[-inf $(date -u +%s)] @queue:{gists}" NOCONTENT
redis-cli HGETALL $(redis-cli --raw FT.SEARCH "${scheduleIndexName}" "@next:[-inf $(date -u +%s)] @queue:{gists}" NOCONTENT | tail -1)

response

1) (integer) 1
2) "gistwiz:scheduled:gists:180E8590F14D47D9BD49FE066A0403FA"
 1) "ms"
 2) "100"
 3) "next"
 4) "1733854122"
 5) "id"
 6) "180E8590F14D47D9BD49FE066A0403FA"
 7) "queue"
 8) "gists"
 9) "content"
10) "{ \"token\": \"ghp_demo_token\", \"username\": \"wilmoore\", \"plan\": \"10s\" }"

setup schedule for email queue

Tip

Filtering by queue: The use of @queue:{gists} or @queue:{email} enables visualization of schedules for different queues in Grafana.

queue="email"
scheduleId=$(uuidgen | tr -d "-")
scheduleKey="${namespace}:${queue}:${scheduleId}"

redis-cli HSET "${scheduleKey}" \
  id "${scheduleId}" \
  queue "${queue}" \
  ms 100 \
  next $(date -u +%s) \
  content '{ "to": "[email protected]", "subject": "GisWiz Email Job Queue", "body": "Testing..." }'

redis-cli FT.SEARCH "${scheduleIndexName}" "@next:[-inf $(date -u +%s)] @queue:{email}" NOCONTENT
redis-cli HGETALL $(redis-cli --raw FT.SEARCH "${scheduleIndexName}" "@next:[-inf $(date -u +%s)] @queue:{email}" NOCONTENT | tail -1)

response

1) (integer) 1
2) "gistwiz:scheduled:email:438B4A3157F742F480ACBDA03FC0F545"
 1) "ms"
 2) "100"
 3) "next"
 4) "1733854431"
 5) "id"
 6) "438B4A3157F742F480ACBDA03FC0F545"
 7) "queue"
 8) "email"
 9) "content"
10) "{ \"to\": \"[email protected]\", \"subject\": \"GisWiz Email Job Queue\", \"body\": \"Testing...\" }"

setup metrics indexing for jobs

jobNamespace="${application}:scheduled::job"
jobIndexName="${jobNamespace}::metrics"

redis-cli FT.CREATE "${jobIndexName}" ON HASH PREFIX 1 "${jobNamespace}:" SCHEMA \
  id TEXT \
  queue TAG \
  schedule TEXT \
  status TAG \
  created_at NUMERIC SORTABLE

response

OK

add job

job_ns="${application}:scheduled::job"
job_id=$(uuidgen | tr -d "-")
job_key="${job_ns}:${queue}:{job_id}"

## create job
redis-cli HSET "${job_key}" \
  id "${job_id}" \
  queue "${queue}" \
  schedule "${scheduleKey}" \
  status "pending" \
  created_at "" \
  content '{ "token": "ghp_demo_token", "username": "wilmoore", "plan": "10s" }'

## add job to queue
redis-cli LPUSH "${job_ns}" "${job_key}"

...

❯ redis-cli HSET "gistwiz:scheduled:gists:12345" \
  id "gistwiz:scheduled:gists:12345" \
  queue "gists" \
  interval 10 \
  next_run $(date -u +%s)
(integer) 4


❯ redis-cli HSET "gistwiz:scheduled:email:67890" \
  id "gistwiz:scheduled:email:67890" \
  queue "email" \
  interval 30 \
  next_run $(date -u +%s)

(integer) 4


❯ redis-cli FT.CREATE "gistwiz:schedule:metrics" ON HASH PREFIX 1 "gistwiz:scheduled:" SCHEMA \
  id TEXT \
  queue TAG \
  interval NUMERIC SORTABLE \
  next_run NUMERIC SORTABLE

OK


❯ redis-cli FT.SEARCH "gistwiz:schedule:metrics" "*" NOCONTENT

1) (integer) 2
2) "gistwiz:scheduled:email:67890"
3) "gistwiz:scheduled:gists:12345"


❯ redis-cli FT.SEARCH "gistwiz:schedule:metrics" "@queue:{gists}" NOCONTENT

1) (integer) 1
2) "gistwiz:scheduled:gists:12345"


❯ redis-cli FT.SEARCH "gistwiz:schedule:metrics" "@queue:{email}" NOCONTENT

1) (integer) 1
2) "gistwiz:scheduled:email:67890"


❯ redis-cli FT.SEARCH "gistwiz:schedule:metrics" "*" LIMIT 0 10

1) (integer) 2
2) "gistwiz:scheduled:email:67890"
3) 1) "id"
   2) "gistwiz:scheduled:email:67890"
   3) "queue"
   4) "email"
   5) "interval"
   6) "30"
   7) "next_run"
   8) "1733823358"
4) "gistwiz:scheduled:gists:12345"
5) 1) "id"
   2) "gistwiz:scheduled:gists:12345"
   3) "queue"
   4) "gists"
   5) "interval"
   6) "10"
   7) "next_run"
   8) "1733823337"

❯ redis-cli FT.SEARCH "gistwiz:schedule:metrics" "@next_run:[-inf $(date -u +%s)]" NOCONTENT

1) (integer) 2
2) "gistwiz:scheduled:email:67890"
3) "gistwiz:scheduled:gists:12345"

---