- "Not All Language Model Features Are One-Dimensionally Linear"
- [Applications and Broader Implications (Interpretability, Safety, etc.)](https://gist.github.com/ajcwebdev/545701f96
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pygame | |
import sys | |
import random | |
pygame.init() | |
# -- Window size -- | |
SCREEN_WIDTH, SCREEN_HEIGHT = 800, 600 | |
screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT)) | |
pygame.display.set_caption("Two-Player Side-Scroller with Platform Collision & Knockback") |
From the paper Transformer decoding in fifty lines of pseudocode by Bob Carpenter at the Flatiron Institute. I copied it by hand and it almost certainly has a mistake or two in it.
DECODE(tok: int<lower=1, upper=T>[N],
alpha: matrix(T, V),
betas: { query: matrix(V, K),
key: matrix(V, K),
value: matrix(V, V) }[A],
gammas: { 1: vector(L), 2: matrix(L, V),
3: vector(V), 4: matrix(V, L) }[A],