These are quick notes from making my own Formula and Tap.
gobuild.sh
#!/bin/bash| Latency Comparison Numbers (~2012) | |
| ---------------------------------- | |
| L1 cache reference 0.5 ns | |
| Branch mispredict 5 ns | |
| L2 cache reference 7 ns 14x L1 cache | |
| Mutex lock/unlock 25 ns | |
| Main memory reference 100 ns 20x L2 cache, 200x L1 cache | |
| Compress 1K bytes with Zippy 3,000 ns 3 us | |
| Send 1K bytes over 1 Gbps network 10,000 ns 10 us | |
| Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD |
| { | |
| "name": "my-app", | |
| "version": "1.0.0", | |
| "description": "My test app", | |
| "main": "src/js/index.js", | |
| "scripts": { | |
| "jshint:dist": "jshint src/js/*.js", | |
| "jshint": "npm run jshint:dist", | |
| "jscs": "jscs src/*.js", | |
| "browserify": "browserify -s Validating -o ./dist/js/build.js ./lib/index.js", |
| echo "$(git tag --sort=committerdate | grep -o 'v[0-9]*\.[0-9]*\.[0-9]*' | tail -n1)" |
| # https://stackoverflow.com/questions/6543519/undoing-accidental-git-stash-pop | |
| # https://stackoverflow.com/questions/89332/how-to-recover-a-dropped-stash-in-git | |
| accepted | |
| If you have only just popped it and the terminal is still open, you will still have the hash value printed by git stash pop on screen (thanks, Dolda). | |
| Otherwise, you can find it using this for Linux and Unix: | |
| git fsck --no-reflog | awk '/dangling commit/ {print $3}' | |
| and for Windows: |
| type Valuable<T> = { [K in keyof T as T[K] extends null | undefined ? never : K]: T[K] }; | |
| function getValuable< | |
| // eslint-disable-next-line @typescript-eslint/ban-types | |
| T extends {}, | |
| V = Valuable<T>, | |
| >(obj: T): V { | |
| return Object.fromEntries( | |
| Object.entries(obj).filter( | |
| ([, v]) => |
(00:00:00) – How batch size affects token cost and speed
(00:32:09) – How MoE models are laid out across a GPU racks
(00:47:12) – How pipeline parallelism moves model layers across racks
(01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.”
(01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal
(01:33:02) – Deducing long context memory costs from API pricing
(02:04:02) – Convergent evolution between neural nets and cryptography