Ido Rosen ido

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

Pre-Transformer Models

Key-Value Memory Networks for Directly Reading Documents

Introduction

Knowledge Bases (KBs) are effective tools for Question Answering (QA) but are often too restrictive (due to fixed schema) and too sparse (due to limitations of Information Extraction (IE) systems).
The paper proposes Key-Value Memory Networks, a neural network architecture based on Memory Networks that can leverage both KBs and raw data for QA.
The paper also introduces MOVIEQA, a new QA dataset that can be answered by a perfect KB, by Wikipedia pages and by an imperfect KB obtained using IE techniques thereby allowing a comparison between systems using any of the three sources.
Link to the paper.

Related Work

📂 Persistent "pipes" in Linux

In a project I'm working on I ran into the requirement of having some sort of persistent FIFO buffer or pipe in Linux, i.e. something file-like that could accept writes from a process and persist it to disk until a second process reads (and acknowledges) it. The persistence should be both across process restarts as well as OS restarts.

AFAICT unfortunately in the Linux world such a primitive does not exist (named pipes/FIFOs do not persist

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

a simple git branching model (written in 2013)

This is a very simple git workflow. It (and variants) is in use by many people. I settled on it after using it very effectively at Athena. GitHub does something similar; Zach Holman mentioned it in this talk.

Update: Woah, thanks for all the attention. Didn't expect this simple rant to get popular.

	import os
	import pickle
	import warnings

	import numpy as np
	import pandas as pd
	from sklearn.model_selection import train_test_split
	from tensorflow.keras.callbacks import EarlyStopping
	from tensorflow.keras.layers import Dense
	from tensorflow.keras.layers import Dropout

 # Download latest archlinux bootstrap package, see https://www.archlinux.org/download/
 wget 'ftp://ftp.nluug.nl/pub/os/Linux/distr/archlinux/iso/latest/archlinux-bootstrap-*-x86_64.tar.gz'
 # Make sure you'll have enough entropy for pacman-key later.
 apt-get install haveged
 # Install the arch bootstrap image in a tmpfs.
 mount -t tmpfs none /mnt
 cd /mnt
 tar xvf ~/archlinux-bootstrap-*-x86_64.tar.gz --strip-components=1

	#!/bin/bash

	set -eux

	status=0

	# Create Ubuntu's PostgreSQL socket dir and relax permissions.
	sudo mkdir -p /var/run/postgresql
	sudo chown -R `whoami` /var/run/postgresql

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	#!/usr/bin/env sh

	##
	# This is script with usefull tips taken from:
	# https://github.com/mathiasbynens/dotfiles/blob/master/.osx
	#
	# install it:
	# curl -sL https://raw.github.com/gist/2108403/hack.sh \| sh
	#