chapter5.md

Book club

Chapter (5)

If you can't measure it, you can't improve it.

Good metrics

1. they help you focus on the right things

• Define a metric associated with your goal - whether that metric is weekly active rate, responsive time, and so on
• Measure the change's impact

2. Visualized over time, good metrics help guard against future regressions

• Want to have a dashboard of signup rates, application latency, and other useful metrics

3. Good metrics can drive forward progress

• performance ratcheting technology - system-level improvement to not make it downward

4. Good metrics let you measure your effectiveness

• 1% by a week

Ask yourself

• Is there some way to measure the progress of what I'm doing?
• If a task I'm working on doesn't move a core metric, is it worth doing? Or is there a missing key metric?

Pick the right metric

• Hours worked per week vs. productivity per week JAPAN!
• Click-through rates vs. long click-through rates What number you measure
• Average response times vs. 95th or 99th percentile response times What to improve by measuring what
• Bugs fixed vs. bugs outstanding Tracking outstanding bugs - what is outstanding bugs? The easy to fix ones?
• Registered users vs. weekly growth rate of registered users
• Weekly active users vs. weekly active rate by age of cohort

Be skeptical about data integrity

• Log data liberally, in case it turns out to be useful later on
• Build tools to iterate on data accuracy sooner
• Write end-to-end integration tests to validate your entire analytics pipeline
• Examine collected data sooner
• Cross-validate data accuracy by computing the same metrics in multiple ways
• When a number does look off, dig in to it early

Key Takeaways

• Measure your progress

• Carefully choose your top-level metric

• Instrument your system

• Know your numbers

• Prioritize data integrity

• What the key metrics should be

• Write own perspective

  • Define our goals with metrics
  • What are we tracking?
  • To attract a potential client, citizens' feedback and a number of users are useful
  • Why does a client stop renewing the contract?
  • What do we want to measure? performance? end user feedback? client's feedback?
  • Good to know the business direction