Skip to content

Instantly share code, notes, and snippets.

@technolo-g

technolo-g/0_keynotes.md

Last active August 29, 2015 14:26
Show Gist options
  • Save technolo-g/8ba5533cde48b731e0ca to your computer and use it in GitHub Desktop.
Save technolo-g/8ba5533cde48b731e0ca to your computer and use it in GitHub Desktop.
Download ZIP
OSCON 2015 Notes
Raw
designing_for_failure.md

Desigining for Failure: How Uber Scaled its Realtime Platform - Matt Ranney @ Uber (@mranney)

Our software affects people's daily lives. What we think of as a bug could be the bane of someone's existence if they need to deal with it everyday.

Notes

  • If failures rarely happen, the mechanisms to handle them become rusty. It is very important to practice this frequently.
  • This includes the database!
  • Optimize for teams working independently.
  • Having (really) slow requests, but not conistently?
    • Send the request to two (equal) endpoints
    • The first endpoint to begin processing sends a cancel message to the second endpoint
    • This always allows the faster request to be returned
    • This doesn't work if all requests are slow
  • Pure HTTP may not be the best for your app. Check out the RPC services in the links below
  • Uber's take on CAP theorem is that availability and partition tolerance are what should be targetted
    • A customer is often happier with a response containing some stale data as long as they get a response
    • When the service is down, they just go to the competetor (lost forever)
  • Uber uses their clients to store backup state
    • During a trip, the client (a cell phone) is constantly sending updates to the datacenter
    • Occasionaly the datacenter will respond with an encrypted digest of the current state
    • If a datacenter failover happens and the client begins communicating with an endpoint that does not know about its existing state, it will request the digest
    • That digest is then used to rehydrate the client's state and things continue where they left off
    • This gives you a distributed backup for the data that actually matters
    • This architecture also allows you to kill any service at any time without affecting users

Links

Raw
refactoring_systems_with_confidence.md

Refactoring Systems with Confidence - Jesse Toth (@jesseplusplus) + Nathan Witmer (@zerowidth) @ GitHub

Notes

  • Before refactoring, you need test coverage on all important (to your customers) behavior
    • Need to cover anything that should not change
    • Users may be using your bugs as a feature
    • Test suites do not cover Production though so it is hard to get metrics on that aspect
  • When GitHub refactored its authentication system, they needed more confidence than just tests passing
    • This lead to the creation of Scientist: A Ruby library for carefully refactoring critical paths
  • Scientist provides a method for you to confirm that your refactor conforms to the old spec
    • Wraps your legacy code and refactor
    • Sends the incoming request to both code paths
    • Compares the responses
    • If responses don't match, it is logged/graphed and the user is returned the proper (old) response
    • If they do match, the new response is returned
    • Can be turned on incrementally until all code is new path only
  • This writes to both the old and new database simutaneously
    • Populates new data as it is created
    • Use a script to backfill existing data
  • Make sure to validate the data and fix when backfilling!
  • Production data is the real test. Nothing finds edge cases like your users do!
  • Data quality is paramount. If you have bad data (and most of us do), all bets are off
    • Data grows organically and over the course of years of development, some gets inserted that is bad
    • Clean this up when you can!

Links

Raw
the_paradox_of_tech_choice.md

The Paradox of Technology Choice - Michelle Brush @ Cerner Corp (@michellebrush)

Notes

  • Avoid building things outside of your competency (when possible)
  • Incentive is made up of both value + visibility. This can lead to re-inventing the wheel for reasons other than pure value
  • Software decision making process:
    • Determine criteria (persistent software attributes / 'ilities')
    • Prioritize criteria (balance business and consumer needs)
    • Discover options
    • Evaluate options (Unpacking Technical Decisions by Sarah Mei)
    • Choose option
    • Evaluate choice
  • Choice architecture: How choices are presented will affect the choice being made
  • Make the right thing to do the easy thing to do
    • Choosing existing and already implemented technologies should be the easiest
    • Can reuse existing patterns, tools, monitoring, etc..
    • Intentional sane defaults help
  • New choices and introductions of technology need to meet the same bar as existing technologies
    • The onus of ramping a technology up with monitoring, patterns, and tooling is on the team who introduces it
    • Makes sense to meet this bar before a new technology makes it into production
    • Define guidelines for projects
      • deployment/automation
      • monitoring
      • metrics
      • documentation
      • reusable patterns
  • Adoption and reuse of a choice within a company may require some internal 'marketing' to have other teams pick it up
    • Peer pressure can work as well
  • If your environment doesn't support the 'make things easier' route, there is always the option to make the wrong choice harder.

Links etc..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment