Checklist - Ideal Ops.md

Checklist - Ideal Ops

In a perfect world, where things are done well, not just quickly, I would expect to find the following when joining the company:

Documentation

• Accurate / up-to-date systems architecture diagram
• Accurate / up-to-date network diagram
• Out-of-hours support plan
• Incident management plan
• Change management plan
• Application documentation

Metric collection

• comprehensive system metrics (eg. cpu, load, mem, disk, network, etc)
• application metrics instrumented in code (eg. queue length, time to post new job) [statsd]
• business metrics instrumented in code as well (eg. registrations) [statsd]
• include network devices (eg. firewall, loadbalancers, switches, vpns, vpc)
• include storage (eg. netapp)
• include database
• include cron jobs
• include CD pipeline systems/applications (e.g., jenkins, chef, build / test farm)
• majority of monitoring from internal systems
• also monitor from external systems (e.g., Nimsoft/Watchmouse)
• retrieve external monitoring data into internal collection for correlation

Alert system:

• alert off data collected (passive)
• alert on checks (active)
• call-out on important alerts
• email, irc/chat, sms, mobile escalation
• call-out rotation, escalation plans

Dashboards:

• Real-time dashboards of all services
• Real-time dashboard of what is being viewed on the site, where traffic is coming from
• Dashboards to include event / deploy lines
• Anyone can create/share dashboards
• No passwords to access dashboards
• Key dashboards visible in the office on screen
• Dashboard of environments - what's deployed
• Cost dashboard (IaaS, SaaS)

Correlation / Investigation

• Graphing system which allows ad-hoc metric correlation (eg. Graphite)
• Centralized logging with search (eg. Logstash, Greylog)
• Record of everything that has changed, by whom, when, and what the change was
• Access to all relevant systems

Infrastructure as Code

• Infrastructure DB with API (Chef server)
• All infra changes tracked, done via configuration management

Security

• Automated view of what needs to be patched/updated
• Regular vulnerability scans with recorded history
• ssh-key as only authentication
• segregated environments (dev, test, prod)
• data anonymisation for performance testing

Performance testing

• Prod-like environment to test in
• Good performance test, with assumptions and approximations documented
• Record of all previous test results
• Automated running of test
• Automated comparison of test results with previous tests

Communications

• whole company using the same instant messaging / chat system
• task/kanbansystem for giving work to systems engineers / infrastructure developers
• ops twitter
• ops status (eg. etsystatus.com; stashboard; amazon status)

Deployment

• single-click deploy
• rollback-able
• performed by developer
• dashboard/KPI used to validate release
• zero-down time
• dark-launches
• feature flags can be turned on/off via webui

Standards

• Published standards of web systems requirements

Process

• Light-weight post-mortem process, blame-free
• Daily operations review
• Monthly/quarterly architecture summit
• Daily stand-ups
• Iteration planning/review
• Regular capacity planning /cost optimization

Meta-metrics

• MTTD
• MTTR
• Availability
• Service degradation (Slow versus broken; features disabled to protect site)
• CD Pipeline Availability
• Release tracking (type, success/failure, success rate, length of incident)