hwu dawu76

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

Pre-Transformer Models

Fine-tuning LLaMA-7B on ~12GB VRAM with QLoRA, 4-bit quantization

nvidia-smi said this required 11181MiB, at least to train on the sequence lengths of prompt that occurred initially in the alpaca dataset (~337 token long prompts).
You can get this down to about 10.9GB if (by modifying qlora.py) you run torch.cuda.empty_cache() after PEFT has been applied to your loaded model and before you begin training.

Setup

All instructions are written assuming your command-line shell is bash.

Clone repository:

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

	#!/usr/bin/env python3
	# /// script
	# requires-python = ">=3.9"
	# dependencies = [
	# "httpx",
	# "typer",
	# "rich",
	# ]
	# ///
	"""

	# /// script
	# requires-python = ">=3.11"
	# dependencies = [
	# "pip",
	# "edsl @ git+https://github.com/expectedparrot/edsl.git@improved_report",
	# ]
	# ///

	from textwrap import dedent
	from edsl import QuestionFreeText, QuestionList, Agent

	# train_grpo.py
	#
	# See https://github.com/willccbb/verifiers for ongoing developments
	#
	"""
	citation:

	@misc{brown2025grpodemo,
	title={Granular Format Rewards for Eliciting Mathematical Reasoning Capabilities in Small Language Models},
	author={Brown, William},

	import tiktoken
	import langdetect
	T = tiktoken.get_encoding("o200k_base")

	length_dict = {}

	for i in range(T.n_vocab):
	try:
	length_dict[i] = len(T.decode([i]))
	except:

	# coding=utf-8
	# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software

	import openai
	import openai_function

	@openai_function
	def sum(a:int, b:int) -> int:
	"""Sum description adds a + b"""
	return a + b

	completion = openai.ChatCompletion.create(
	model="gpt-3.5-turbo-0613",

	from langchain.document_loaders import YoutubeLoader
	from langchain.indexes import VectorstoreIndexCreator
	urls = [
	("https://www.youtube.com/watch?v=fP6vRNkNEt0", "Prompt Injection"),
	("https://www.youtube.com/watch?v=qWv2vyOX0tk", "Low Code-No Code"),
	("https://www.youtube.com/watch?v=k8GNCCs16F4", "Agents In Production"),
	("https://www.youtube.com/watch?v=1gRlCjy18m4", "Agents"),
	("https://www.youtube.com/watch?v=fLn-WqliEQU", "Output Parsing"),
	("https://www.youtube.com/watch?v=ywT-5yKDtDg", "Document QA"),
	("https://www.youtube.com/watch?v=GrCFyyyAxCU", "SQL"),