| """ | |
| The most atomic way to train and run inference for a GPT in pure, dependency-free Python. | |
| This file is the complete algorithm. | |
| Everything else is just efficiency. | |
| @karpathy | |
| """ | |
| import os # os.path.exists | |
| import math # math.log, math.exp |
They basically all suggest that apparent improvements to the state of the art in ML and related fields are often not real, or at least the result of factors other than what the authors claim.
The state of sparsity in deep neural networks
What is the state of neural network pruning?
On the State of the Art of Evaluation in Neural Language Models
Do Transformer Modifications Transfer Across Implementations and Applications?
I recently happened upon a very interesting implementation of popen() (different API, same idea) called popen-noshell using clone(2), and so I opened an issue requesting use of vfork(2) or posix_spawn() for portability. It turns out that on Linux there's an important advantage to using clone(2). I think I should capture the things I wrote there in a better place. A gist, a blog, whatever.
This is not a paper. I assume reader familiarity with
fork()in particular and Unix in general, though, of course, I link to relevant wiki pages, so if the unfamiliar reader is willing to go down the rabbit hole, they should be able to come ou
