As we increase reliability cost increases so risk of unavailability has to be balanced with goal of innovation and efficient servbice operations
Cost has 2 aspects
- cost of redundant compute
- opportunity cost
Service should be reliable enough but no more reliable than we need to make it.
- Measure acceptable level of unplanned downtime
availability = uptime / (uptime + downtime)
| Availability | Downtime allowance |
|---|---|
| 5 9s | 5.26 mins |
| 4 9s | 52.6 mins |
| 99.95% | 4.38 hours |
| 3 9s | 8.76 hours |
In a distributed system time based availability is not very meaningful because given fault tolerance we will be partially available at all times. A better metric is request success rate
availability = successful requests / total requests
In a typical application not all requests are equal - a new sign up failing is different from failing for polling new email in background
Set quarterly availability targets for a service and track our performance against those targets on a weekly basis.
- work with product owners to turn business goals into explicit objectives
- In absence of a product team the following questions can help assess the risk tolerance of a services
- what level of availabiltiy is required
- do different types of failures have different effects on a service?
- how can we use the service cost to help locate a service on the risk continuum
- what other service metrics are important to take into account?
- what level of service will the users accept?
- does this service directly tie to revenuw?
- Is this paid service or is it free?
- if there are competitors in the market what level of service do those provide?
- is this service targeted at busineses or consumers?
- how resilient is our business to service downtime?
- Which is worse for service - cosntant low rate of failures or an occasional full site outage?
- If we were to build and operate this system at one more nine of availability what would our incremental increase in revenue be?
- Does this additional revenue offset the cost of reaching that level of reliability?
- a shared piece of infra may serve 2 use cases one needs high latency other more throughput, one of the ways to achieve that is to partition the infra according to se case and offer it at multiple levels of service for ex low latency cluster vs high throughput cluster
- splitting levels of service in this way offers clients the choice to select level of service that fits in with thier cost
Ask the following questions to determine a risk / effort boundary
- Software fault tolernace
- Testing
- Push Frequency
- Canary duration and size
- Product management defines a SLO for quarter
- actual uptime is measured by a third party - eg monitoring
- difference between 2 is the how much error budget is remaining
- as long as it is remaining - new changes can be pushed.