Xuemin Zhao xmzhao

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Waifu Diffusion 1.3 Release Notes

HuggingFace Page for model download: https://huggingface.co/hakurei/waifu-diffusion-v1-3

Skip-Thought Vectors

Introduction

The paper describes an unsupervised approach to train a generic, distributed sentence encoder.
It also describes a vocabulary expansion method to encode words not seen at training time.
Link to the paper

Skip-Thoughts

Multiprocess Task Broker with REST API

This gist shows and example or an asynchronous multiprocess task broker which can take job requests and report on running jobs via a minimal REST API.

Adapted from https://gist.github.com/nitaku/10d0662536f37a087e1b

All of the caveats from the original author still apply.

	def remove_empty_elements(d):
	"""recursively remove empty lists, empty dicts, or None elements from a dictionary"""

	def empty(x):
	return x is None or x == {} or x == []

	if not isinstance(d, (dict, list)):
	return d
	elif isinstance(d, list):
	return [v for v in (remove_empty_elements(v) for v in d) if not empty(v)]

	#include <chrono>
	#include <iomanip>
	#include <iostream>

	string getTimestamp() {
	// get a precise timestamp as a string
	const auto now = std::chrono::system_clock::now();
	const auto nowAsTimeT = std::chrono::system_clock::to_time_t(now);
	const auto nowMs = std::chrono::duration_cast<std::chrono::milliseconds>(
	now.time_since_epoch()) % 1000;

	// This is a sample MPI program in Go.
	//
	// To build and run it, we need to install MPI. I downloaded and built
	// Open MPI 1.8.8:
	//
	// wget https://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.8.tar.bz2
	// tar xjf openmpi-1.8.8.tar.bz2
	// cd openmpi-1.8.8
	// ./configure --prefix=/home/yi/openmpi
	// make -j2 install


	covariateShift <- function(data, resla, riskfac, ssize=10000){
	## importance sampling approach
	## when different distributions for the
	## training and test data

	require(plyr)

	Natsal.riskfac.table <- DistnTable(data, riskfac)
	Natsal.riskfac.table <- colNameReplace(Natsal.riskfac.table, "(all)", "Natsalfreq")

	from gevent import monkey
	monkey.patch_socket()
	import logging

	import gevent
	from gevent.queue import Queue
	import pymysql as db

	logging.basicConfig(level=logging.DEBUG)
	LOGGER = logging.getLogger("connection_pool")