Skip to content

Instantly share code, notes, and snippets.

Agrippa
(A Book of The Dead)
by William Gibson
I hesitated
before untying the bow
that bound this book together.
A black book:
ALBUMS CA. AGRIPPA
@segyges
segyges / gist:6c861962e98e0026f89782a31a63ee4f
Created November 2, 2023 22:50
impact_depth_width_results.csv
params,d_model,d_ff,layers,C4 val. PPL,COGS,COGS-vf,GeoQuery Standard,English Passivization
41M,512,4779,1,45.7,12.4,25.7,68.2,0
41M,512,2048,2,31.1,58.2,78.3,76.4,9.88
41M,512,1138,3,29.3,63.1,80.8,79.6,26.2
41M,512,682,4,28.8,68.5,82.5,78.6,28
41M,512,409,5,28.8,63.4,82.5,76.8,89.9
41M,512,227,6,29.1,68.4,82.6,77.5,74.1
41M,512,97,7,29.6,72.3,83,77.1,78.3
134M,768,36000,1,33.6,19.4,26.3,72.5,0
134M,768,17000,2,22.3,65.5,83,81.4,29.9
@segyges
segyges / ff_modifications.py
Last active November 17, 2023 00:06
ff_modifications.py
import torch
import torch.nn as nn
class FeedForwardLayer(nn.Module):
def __init__(self, d_model, nonlinearity):
super(FeedForwardLayer, self).__init__()
self.linear1 = nn.Linear(d_model, d_model*6)
self.linear2 = nn.Linear(d_model, d_model)
self.nonlinearity = nonlinearity() # literally any function that is six-to-one
@segyges
segyges / ff_modifications_2.py
Last active November 17, 2023 05:18
ff_modifications_2.py
# The last one was getting too long
import torch
import torch.nn as nn
class FeedForwardLayer(nn.Module):
def __init__(self, d_model, nonlinearity):
super(FeedForwardLayer, self).__init__()
self.linear1 = nn.Linear(d_model, d_model*6)
self.linear2 = nn.Linear(d_model, d_model)
@segyges
segyges / ubuntu-nvidia-container-setup.sh
Created December 31, 2023 19:18
Script for (re)-installing nvidia drivers, docker, and nvidia-container-toolkit
#!/bin/sh
# you may need to bounce the machine after this before it will work
# success is tested by running: sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
# drivers
sudo ubuntu-drivers install
# install docker
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
sudo apt-get update
@segyges
segyges / lfs_parity_checker.sh
Last active March 3, 2024 01:30
lfs_parity_checker.sh
#!/bin/bash
lfs_files_long=$(git lfs ls-files --long)
checked=0
bad=0
while IFS= read -r line; do
# Read each part of the line into separate variables
read -r hash separator filename <<< "$line"
@segyges
segyges / hf_byte_tokenizer.json
Created March 31, 2024 19:45
Should be good hf tokenizer file for a pure byte tokenizer
{
"version": "1.0",
"truncation": null,
"padding": null,
"added_tokens": [
{
"id": 0,
"content": "<|endoftext|>",
"single_word": false,
"lstrip": false,
177MF LLC PITTS MODEL 12
2007 SAVAGE AIR LLC EPIC LT
2021FX3 LLC CCX-2000
3XTRIM 450 ULTRA
5 RIVERS LLC SQ-2
737 800
777 FF2
781569 INC FX 210
A VAN NIEROP RISEN 915 IS
A. SCHLEICHER GMBH & CO. ASW 27-18
Fyodorov, the Russian librarian, was a man of profound thought and even more profound ideas. His days were spent surrounded by the wisdom held within the pages of countless books in the Moscow library. He wasn't content with the mundane tasks of cataloging and shelving; Fyodorov harbored a grander vision, a destiny that transcended the earthly confines of Russia and the 宇宙 (yǔzhòu) [universe, cosmos, space].
His central philosophy, which he developed over many years, was called Cosmism, though it wasn't always given that specific name during his lifetime. It was a radical 融合 (rónghé) [fusion, blend, integration] of science, philosophy, and religion. Fyodorov believed that death wasn't an inevitable part of existence, but rather a 限制 (xiànzhì) [limitation, restriction, constraint] that humanity could and should overcome. He envisioned a future where scientific and technological advancements allowed humans to achieve physical immortality, not just for themselves, but potentially for their ancestors as well.
T
# Originally by https://jerryxio.ng/
class MultiPositionRotary(nn.Module):
def __init__(
self,
head_dim: int,
pos_dim: int,
min_freq: float,
max_freq: float,
frozen: bool = True,