redis-elasticsearch sync

notes.md

• upsert / merge (data) is called

  • validate data with schema
  • create transaction
  • set new value
  • add key to 'sync-queue' list : [{key, value}]
  • publish 'trigger-sync' event
  • execute transaction

• worker event listener receives an 'entity-update' event

  • if current < limit, spawn a 'sync-worker' instance
  • worker executes lua script (STEP A)
    • temp = sync-queue.lpop
    • sync-processing.rpush (temp)
    • return data
  • worker updates elasticsearch
  • IF elasticsearc update fails
    • execute script to remove item from sync-processing, then push it back to sync-queue
  • worker executes script
    • sync-queue.lrem item
  • worker publishes event 'sync-complete' for affected keys / transaction
  • worker attempts goes back to STEP A
  • if no data was fetched in STEP A, we kill this worker instance, reduce current

• lua script examples https://medium.com/@stockholmux/save-node-js-headaches-with-lua-and-redis-165df4a620b5

• we should replace transactions with lua scripts for performance, I assume?

  • reduces function calls
  • works with javascript string literals ${varGoesHere}

mandatory change:

• we need to use lists instead of sets since we have to tolerate same entity changes at almost the same time
• we have to use UUID's since same entity updates in span of 1 sec could reflect different transactions.
• the queue actually serves as a chronological snapshot

in node, it's redis.eval(script, 0, args). in lua script, it's redis.call(method, args)