David Pennington xeoncross

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

Clarify and agree on the scope of the system

User cases (description of sequences of events that, taken together, lead to a system doing something useful)
- Who is going to use it?
- How are they going to use it?

Rate Limiting Redis with multiples of a time.Duration

In a distributed system, you want the API layer to be stateless. So the app can offload rate limiting to a redis instance. This way, you can use the INCR to both increment and return the number of requests a given IP/Client/API Key has made across any number of App / serverless instances.

However, what do you do if the limit isn't roundable to the current second/minute/hour/day? How does each instance agree which key to use? The answer is to modulo the previous and future keys to find the right one.

For example, 3 requests every 5 seconds is the desired limit. That means we can use -max-1:+max-1 as the range of values to scan to find the key we should use.

max := 5

Intel iMac

i7 32GB RAM AMD Radeon R9 M295X 4 GB

Native Mac Steam Lib

Decent FPS (50-100FPS), but stalls and lags at random times.

The future of fast MVP's using Go and React

As Go matures, no-code (if you squint) solutions are making their way into the ecosystem. This means faster MVP's and less errors writing out the same CRUD project-after-project.

Starting with just your sql statements, you use sqlc to generate the models and entities
Write your actual service/business logic using the entities and interfaces sqlc generated (wish list: skip this step)
Use goa to generate your OpenAPI HTTP or gRPC webserver
Have openapi-generator produce the Typescript SDK you're UI will use to call it
Build out your interface
Publish (web, app store, etc..)

Common Crawl Corpus

Why crawl the web when someone already does for us?

Demo walking through the common crawl data: https://tech.marksblogg.com/petabytes-of-website-data-spark-emr.html

Steps

Multiple times a year, there is a new crawl of the "whole" web completed.

	package main

	import "fmt"

	func BenchmarkMapIteration(b *testing.B) {

	size := 1000
	ranked := make(map[int]uint8, size)
	for i := 1; i < size; i++ {
	ranked[i] = uint8(i % 8)

	package main

	import (
	"encoding/binary"
	"fmt"
	)

	func main() {
	// subtract 1 from each answer since we're counting zero also
	fmt.Printf("%d bits = %8b = %d\n", 2, byte(1<<2-1), 1<<2)

	package iiexperiments

	import (
	"fmt"
	"testing"
	)

	// How much space will using a delta encoding save?

	// If we assume we have 1 byte (0-255) to tell us how many bytes the delta is contained in, we can then only store the diff between the last and next int values. This difference is the "delta".

	package main

	import "fmt"

	type A string

	func (a A) Foo() {
	fmt.Printf("Foo: %q\n", a)
	}

	MYSQL_HOST=mysqldb
	MYSQL_USER=design
	MYSQL_PASS=design
	MYSQL_NAME=design
	MYSQL_PORT=3306