Jason Yee jwsy

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

Flask cookies, when stored client-side, are .-delimited, often zlib-deflated, and Base64-encoded with _ subbed in for / and - subbed in for + (URL-safe encoding).

Let's take an example cookie from https://snowball2.kringlecastle.com/:

.eJy1kT8PgjAQxb9L5w7UIv9mNJoYB3UjDI1WJQE0IENj_O6WZxFQYhw0l7z23l1yvV-vJEz2-2RbpRdFAouSSS4ztRDqVF1IEEUWfYuYapf90WXdgDv8BqvX23WHe_tubHZdH5NzWa8aPaaNoBxqQ8da9aFPH5kHdaGOqXBkNnQMz0WHqztqbTyOzI7r8dMikfku_YJ2E_1Nf0v71W1ndml_dj_SbtZtgQ-jppFjgHsGF4f3QMxRYYDLAJcZuC3-9mNcfJajtX7BpipyEpBzKpQsCDWXeai9WXU4rkRenrKwUOVOpPJZX4pM6g7f9rjv-CNObncEAMTx.YaTaAA.SGRJEzFnk9BCtFwX-SmuSHrKofM

We can take the first piece (ignoring the signature) and store it as a variable in python3:

	for node in node1 node2 node3;do
	multipass launch -n $node
	done

	# Init cluster on node1
	multipass exec node1 -- bash -c "curl -sfL https://get.k3s.io \| sh -"

	# Get node1's IP
	IP=$(multipass info node1 \| grep IPv4 \| awk '{print $2}')

	#!/bin/bash

	# requires https://stedolan.github.io/jq/download/

	# config
	KEYCLOAK_URL=http://localhost:8080/auth
	KEYCLOAK_REALM=realm
	KEYCLOAK_CLIENT_ID=clientId
	KEYCLOAK_CLIENT_SECRET=clientSecret
	USER_ID=userId

	function join($path)
	{
	$files = Get-ChildItem -Path "$path.*.part" \| Sort-Object -Property @{Expression={
	$shortName = [System.IO.Path]::GetFileNameWithoutExtension($_.Name)
	$extension = [System.IO.Path]::GetExtension($shortName)
	if ($extension -ne $null -and $extension -ne '')
	{
	$extension = $extension.Substring(1)
	}
	[System.Convert]::ToInt32($extension)

	#!/bin/sh

	# Adapted from https://gist.github.com/westurner/3196564

	# Package versions as of 9/14/2013, assumes that you've installed a virtualenv

	setup_brew () {
	if [ ! -f "/usr/local/bin/brew" ]; then
	/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"
	fi

	#!/bin/sh

	setup_brew () {
	if ![-f "/usr/local/bin/brew"]; then
	/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"
	fi
	}

	setup_ipython () {
	brew install readline