Matthew Hollings mattholl

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Frontend Development

Attention: the list was moved to

https://github.com/dypsilon/frontend-dev-bookmarks

This page is not maintained anymore, please update your bookmarks.

High level style in javascript.

Opinions are like assholes, every one has got one.

This one is mine.

Punctuation: who cares?

Punctuation is a bikeshed. Put your semicolons, whitespace, and commas where you like them.

	# coding=utf-8
	# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software

	var url = require('url')
	var websocket = require('websocket-stream')
	var engine = require('voxel-engine')
	var duplexEmitter = require('duplex-emitter')
	console.log('starting', 1)
	var socket = websocket('ws://' + url.parse(window.location.href).host)


	var emitter = duplexEmitter(socket)

	## Filter Table

	*filter
	:INPUT ACCEPT [971:197590]
	:FORWARD ACCEPT [0:0]
	:OUTPUT ACCEPT [95:9682]
	-A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
	-A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
	COMMIT

	window.addEventListener('load', function() {
	var section = document.querySelector('div.main'),
	args = document.querySelector('div.arguments'),
	image = args.querySelector('img'),
	video = args.querySelector('video'),
	log = $('log');

	//check support
	if (!supportsWebGL()) {
	log.innerHTML = '<p class=\'error\'>Your browser doesn\'t seem to support WebGL. More info <a href=\'http://get.webgl.org/\'>here</a>.</p>';

	//Learning WebGL for Plask Lesson16
	//Learing WebGL : http://learningwebgl.com/blog/?page_id=1217
	//Github : https://github.com/gpjt/webgl-lessons (this script use 'earth.jpg' , 'crate.gif' and 'macbook.json')
	//
	//Plask : http://www.plask.org/
	//
	//俺俺でMat3とMat4.toInverseMat3()あたりを追記したものを使ってやりました。
	//元のplask.jsと区別つけるために同一フォルダから読み込むようにしてます。
	//尚、binary版とgithubの最新版で微妙にplask.jsの変更があるみたいなので、
	//plask_.js binary版用、plask.js github用としてます。

	//Learning WebGL for Plask Lesson14
	//Learing WebGL : http://learningwebgl.com/blog/?page_id=1217
	//Github : https://github.com/gpjt/webgl-lessons (this script use 'earth.jpg' and 'earth-specular.gif')
	//
	//KeyMap
	//zoom : qe,
	//lighting : t,
	//colorMap : y
	//useColorMap : u,
	//reset : r,