Skip to content

Instantly share code, notes, and snippets.

@dhana-git

dhana-git/Guide on Microservices: Backups and Consistency.md

Created March 29, 2019 09:28
Show Gist options
  • Save dhana-git/3dda5326b3bd15a93d3389a6c30d3000 to your computer and use it in GitHub Desktop.
Save dhana-git/3dda5326b3bd15a93d3389a6c30d3000 to your computer and use it in GitHub Desktop.
Download ZIP
Guide on Microservices: Backups and Consistency | Consistent Disaster Recovery for Microservices - the BAC Theorem
Raw
Guide on Microservices: Backups and Consistency.md

Guide on Microservices: Backups and Consistency | Consistent Disaster Recovery for Microservices - the BAC Theorem

The BAC (Backup Availability Consistency) Theorem

When backing up an entire microservices architecture, it is not possible to have both availability and consistency.

  • Eventual Inconsistency with Full Availability (High Availability)

    • When making an independent backup of the state (the event log) of each service, it is possible that one service will be backed up before the event is added to the log, while the other service will be backed up after the corresponding event is added to its log. When restoring the services from backup, only one of the two services will recover its complete log. The global state of the microservice architecture will thus become inconsistent after restoring it from backup.
    • Possible problems,
    • Broken link
    • Orphan state

  • Consistent Backups with Limited Availability (Eventual Consistency)

    • While the backup is running, no events can be added to the microservice logs, effectively reducing the availability of the microservices for clients that need to perform some state transitions/write operations.

  • No Backup but Availablity and Consistency (Eventual IN-Consistency)

Eventual Consistency into Eventual IN-Consistency

  • Let's start simple and take 2 microservices. We'll call them MS1 and MS2 for short.
  • Suppose MS2 crashes and you restore its database from a backup. How recent is this backup? Does it contain all the updates that correspond to the latest data in MS1? Who knows... Perhaps not?
  • Let's make it more intriguing still: suppose both microservices crash. Then you restore both MS1 and MS2 from their respective backups. Were those backups taken at the same time (so to speak) - so they each contain data that is consistent with the other?
  • As you may have guessed by now, in general there is no such guarantee. You can't have autonomous persistence (or polyglot persistence, for that matter) and have consistency after recovery from a backup.
  • The implications are huge. Imagine "eventual consistency" as it is implemented today: it does not work. If two systems are independent and count on changes to ripple through, then what about the backups? Do they change at the same rate? Probably not.
  • If you want consistency in your microservices, then back them up at once. In practice (so far) this probably means you need one shared data store for all of them. OR: you can learn about a new solution we've been working on to have consistent backups across separate data stores.

Credit: Dr. Guy Pardon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment