yoavg

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

veekaybee

ChatGPT Resources

Context

ChatGPT appeared like an explosion on all my social media timelines in early December 2022. While I keep up with machine learning as an industry, I wasn't focused so much on this particular corner, and all the screenshots seemed like they came out of nowhere. What was this model? How did the chat prompting work? What was the context of OpenAI doing this work and collecting my prompts for training data?

I decided to do a quick investigation. Here's all the information I've found so far. I'm aggregating and synthesizing it as I go, so it's currently changing pretty frequently.

Model Architecture

kacperlukawski

import multiprocessing
import tempfile
import time
from pathlib import Path
from urllib.request import urlretrieve

import h5py
from qdrant_client import QdrantClient
from qdrant_client.conversions.common_types import VectorParams
from qdrant_client.http.models import Distance, SearchRequest

moyix

Using Linear Algebra to Convert a Large Code Model

Background

The SalesForce CodeGen models are a family of large language models trained on a large amount of natural language data and then fine-tuned on specialized datasets of code. Models of size 350M, 2B, 6B, and 16B parameters are provided in three flavors:

• nl, the base model trained on The Pile, a large natural language dataset compiled by EleutherAI
• multi, which is fine-tuned from the nl model on a dataset of code in multiple languages, scraped from GitHub, and
• mono, which is fine-tuned from the multi model on Python code only.

SkyyySi

NONE OF THESE CLIENTS ARE VERIFIED BY ME FOR SECURITY OR ANYTHING ELSE! USE AT YOUR OWN RISK!

---

Update 2025-07-16: Use either Revanced, NewPipe or Firefox with add-ons. Revanced is my personal pick, due to it being a mod of the official App like Vanced, but better (e.g. it has integration of Sponsor Block, Return YouTube Disklike and more).

I wouldn't recommend Kiwi Browser anymore as it has been discontinued. AFAIK it has been accuired by Microsoft and the extension support has been merged into Edge (Canary). But you'd have to use Edge. Yuck.

---

yotamorimoto

Server.supernova;
s.reboot;
// use your own sound instead:
b = Buffer.read(s,"/Users/yota/Desktop/tail.wav");
b = Buffer.read(s,"/Users/yota/Music/am_cpb/maison/mk_nuit1-.wav");
~pos = Bus.control;
(
SynthDef(\play, { |rate=1, pos=0, dur=1, pan=0|
	var sig, env;
	sig = PlayBuf.ar(2, b, rate, 1, ~pos.kr*SampleRate.ir);

dendisuhubdy

#! /usr/bin/python3
import tweepy
import time


def limit_handled(cursor):
    while True:
        try:
            yield next(cursor)
        except tweepy.errors.TooManyRequests as e:

daefresh

tkuhn

Knowledge Space

https://w3id.org/knowledge-space/

version 1.0, 27 April 2026

by Tobias Kuhn

Cite as: Tobias Kuhn. Knowledge Space. https://w3id.org/knowledge-space/. Version 1.0. 2026.

DanielDe

require 'set'

def read_most_common_words
  File.read('100_000-english.txt').split("\n")
end

def read_etymologies
  File.read('etymwn-20130208/etymologies.tsv').split("\n").map do |line|
    parts = line.downcase.split("\t")
    [parts[0], parts[2]]