Seb Insua sebinsua

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

How React Fast Refresh works

Implementations:

Additional reading:

A half-hour to learn Zig

This is inspired by https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/

Basics

the command zig run my_code.zig will compile and immediately run your Zig program. Each of these cells contains a zig program that you can try to run (some of them contain compile-time errors that you can comment out to play with)

	import math
	import uuid


	class BTree:
	def __init__(self, order=3):
	self.root = BTreeNode(order)

	def insert(self, toinsert):
	all_elements = self.list()

	type _Tuple<
	T,
	N extends number,
	R extends readonly T[] = []
	> = R["length"] extends N ? R : _Tuple<T, N, readonly [T, ...R]>;

	type Tuple<T, N extends number> = _Tuple<T, N> & {
	readonly length: N;
	[I: number]: T;
	[Symbol.iterator]: () => IterableIterator<T>;

	// A JavaScript `Date` is fundamentally specified as the number of milliseconds that have elapsed since the ECMAScript epoch,
	// which is defined as January 1, 1970, UTC (equivalent to the UNIX epoch).
	//
	// However, depending on how you construct a `Date` the date passed in can be parsed as a date in your local time zone or a UTC date.
	// For instance `"2022-08-01"` will be treated as a UTC value, but `"2022-08-01T00:00:00.000"` will be treated as if it
	// is a localised date and offset before setting the underlying value of the `Date`. Similarly, there is a difference
	// between passing in `(2022, 7, 1, 0, 0, 0, 0)` which is assumed to be localised and `Date.UTC(2022, 7, 1, 0, 0, 0, 0)`
	// which is a unix timestamp (in milliseconds).

	// Firstly, let's print our local timezone and the current timezone offset:

	/// benchmark.js
	async function benchmark() {
	let namePromises = [];
	let namePromises2 = [];
	for (let i=0;i<100000;++i){
	namePromises.push(utils.getRandomName()); // you'll need to make your own
	namePromises2.push(utils.getRandomName()); // same here
	}

	let names = await Promise.all(namePromises);

	Permission is hereby granted, free of charge, to any person obtaining a copy
	of this software and associated documentation files (the "Software"), to deal
	in the Software without restriction, including without limitation the rights
	to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	copies of the Software, and to permit persons to whom the Software is
	furnished to do so, subject to the following conditions:

	The above copyright notice and this permission notice shall be included in
	all copies or substantial portions of the Software.

	#!/bin/bash

	gdb -p "$1" -batch -ex 'set {short}$rip = 0x050f' -ex 'set $rax=231' -ex 'set $rdi=0' -ex 'cont'


	async function avery() {
	await null;
	for (let i = 0; i < 10; i += 1) {
	console.log('await');
	await null;
	}
	}

	function blake() {