Presentation on Scale

scale.md

Scale

Scale is a buzzword for growth in the tech industry, but it has so many meanings. Let's dig into them to see what scale really means!

Scaling means you are increasing the responsibility or load of a system.

How do we, as web developers, handle the increased load?

1. Increase Concurrency
2. Distribute load across mulitple servers
3. Caching
4. Asynchronous processing
5. Break up applications

Finally, we'll discuss the ramifications of these changes.

Concurrency

TODO

Multi-Threaded

The first answer to this problem was to increase threads from a single to multiple. In web systems, we have this nice property of independent requests that (most of the time) don't need to communicate with each other. This means we can spin up as many threads as we can, with each handling the a request at a time.

See: Rails, ColdFusion, PHP, Django

Thread-level Concurrency

Then someone realized that web apps (again, most of the time) are IO-bound. (Definition: A process is waiting for inputs or sending outputs more often than running calculations in the CPU. Opposite of CPU-bound). So why not run more than one request on a thread at a time?

See: node.js, Go, http-kit, Twisted

Multi-server deployments

Going from a single to multiple servers is a bigger leap than single thread to multiple threads. No longer can we rely on in-memory or same-box solutions for data storage. We have to establish canonical sources of data in our system: Databases.

We also now have multiple sources of the same service, so we need a way to tell people about the different sources. Two ways to do this: service discovery (i.e. DNS) or routing (i.e. load balancing)

Databases

TODO

Load Balancing

TODO

Sockets and Messaging Infrastructure

TODO

Caching

Caching is an amazing way to increase the scalability of a system. Basically, we take a snapshot of content at a canonical address and replay it to anyone who asks for that content later. That way, we only ask the canonical source once (or every so often), and only have to do the hard processing rarely. Unfortunately, it introduces one of the two hardest problems in tech: cache invalidation (the other being naming things).

Cache Invalidation

Lets say we want /timeline/jon. Its a cached endpoint and we can receive his timeline very quickly. When must this cache be invalidated (i.e. updated)?

• Anyone jon follows tweeting, updating a tweet, changing their name, or changing their icon
• Any tweet on the top page of his timeline being retweeted or favorited.

Walk through in your head this action from start to finish and imagine the algorithm.

Cache Invalidation is hard.

Caching in a new light

We can still cache, but we must be very careful when we do.

• Timed cache invalidations, so something non-important may be T minutes old.
• Cache static data
• Change the identifier when the data changes
  • npm caches all package content because they make the assumption version x.y.z never changes
  • So to change content, you must update the package number.

Asynchronous Processing

Work Queues are awesome!

TODO

Monolithic Applications

Monolithic Applications are bad!

• Small apps are easy to change (like, when you have bugs)
• Small apps are easier to reason about (i.e. know how they'll handle load)
• Small apps are easy to understand (for all those new employees you'll need)
• Small apps are individually scalable (add servers where its needed)

Scaling Teams

OK, so you've scaled you application to 100 servers serving 10 apps, using load balancers, caching, and distributed work queues. But now no new employee understands the system!

• 1 developer (unfunded startup)
• 5 developers (funded startup)
• 25 developers (successful startup)
• 100 developers (exited startup)
• 50,000 developers (google startup)

How do we make scalable software for scalable teams?

Some possibly right answers

• Use multiple apps: many reasons
  • Distribute responsbility (read: Amazon)
  • Easier for newbies to make progress on small apps
  • Easy to change the whole system one app at a time (rewrites are inevitable)
• Establish consistent paradigms at an above-team level
  • Let teams do as they please but when teams communicate, be consistent about it.
  • Use the same deployment infrastructure for all apps (special cases aside)
  • Use the same distributed work queue for all apps (up to a point)
• REFACTOR REFACTOR REFACTOR: Its not done til its rewritten three times (or more).