mjallday/gist:f07618417d1920f4e20e

Created March 15, 2015 22:14

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/mjallday/f07618417d1920f4e20e.js"></script>
Save mjallday/f07618417d1920f4e20e to your computer and use it in GitHub Desktop.

Balanced Partial Outage Post Mortem - 2015-03-15

Raw

Balanced experienced a partial outage that affected 25% of card processing transactions between 8:40AM and 9:42AM this morning due to a degraded machine which was not correctly removed from the load balancer.

The core of the issue was in our secure vault system, which handles storage and retrieval of sensitive card data. One of the machines stopped sending messages, which cause some requests to be queued up but not processed but our automated health checks did not flag the machine as unhealthy.

We found the root cause of the issue and have resolved the problem to ensure that it will not re-occur.

We're dedicated to providing high-quality support and a culture of openness during the migration process. If you continue to experience issues, please email us at [email protected] or tweet at @balancedstatus.

taylorbrooks commented Mar 16, 2015

@mjallday Does the dashboard need to be kicked? I'm not seeing transaction data past 11:35am CST today.

matin commented Mar 16, 2015

@taylorbrooks: Are you still experiencing any issues? The dashboard should be working fine. I just checked on your marketplace in particular to verify.

msherry commented Mar 16, 2015

Transactions created after the end of the outage showed up correctly. I am manually reingesting any transactions created during the outage, as some of them didn't show up correctly in the dashboard.