Quan Dang 18520339

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Chat GPT "DAN" (and other "Jailbreaks")

Clone the repo.
Use git rebase -i --root
vim will open. Select the commits you want to modify by changing pick to edit. If you would like to change all the commits, perform the following replace: :%s/^pick/edit/g. This command changes all instances of "pick" at the start of lines to "edit".
You will now be shown all the selected commits one by one. Each commit message will be displayed. You have two options:
- If you would like to keep the commit author details the same, do a git rebase --continue.
- If you would like to change it to a different name/email, do git commit --amend --reset-author. If --reset-author is specified, it will use the details from your git config. (If you need to specify an alternate name/email, you can do so with --author="John Doe <[email protected]>". If you would like to change the time to a previous date, you can do so with --date "2 days ago".)
Do the same for all the commits and finish the rebase.
Perform git push -f origin master to

Prevent creation of vmmem files in VMware (Windows)

Issue

VMWare creates .vmem files to back the guest RAM. On the host this causes disk thrashing especially during powering on and off the guest.

Solution

Add the following lines to the .vmx file to prevent creation of .vmem files. This will reduce disk IO and VM performance will improve especially on non-SSD disks.

Conventional Commit Messages

See how a minor change to your commit message style can make a difference.

git commit -m"<type>(<optional scope>): <description>" \
  -m"<optional body>" \
  -m"<optional footer>"

	# -----------------------------------------------------------------------------
	# AI-powered Git Commit Function
	# Copy paste this gist into your ~/.bashrc or ~/.zshrc to gain the `gcm` command. It:
	# 1) gets the current staged changed diff
	# 2) sends them to an LLM to write the git commit message
	# 3) allows you to easily accept, edit, regenerate, cancel
	# But - just read and edit the code however you like
	# the `llm` CLI util is awesome, can get it here: https://llm.datasette.io/en/stable/

	gcm() {

	const fs = require('fs');
	const axios = require('axios');
	const FormData = require('form-data');

	// Function to read the image file as a buffer
	const readImageFile = (filePath) => {
	return new Promise((resolve, reject) => {
	fs.readFile(filePath, (err, data) => {
	if (err) {
	reject(err);

	#######################################
	### -------- Load libraries ------- ###

	# Load Huggingface transformers
	from transformers import TFBertModel, BertConfig, BertTokenizerFast

	# Then what you need from tensorflow.keras
	from tensorflow.keras.layers import Input, Dropout, Dense
	from tensorflow.keras.models import Model
	from tensorflow.keras.optimizers import Adam

	# -- coding: utf-8 --

	import regex as re

	uniChars = "àáảãạâầấẩẫậăằắẳẵặèéẻẽẹêềếểễệđìíỉĩịòóỏõọôồốổỗộơờớởỡợùúủũụưừứửữựỳýỷỹỵÀÁẢÃẠÂẦẤẨẪẬĂẰẮẲẴẶÈÉẺẼẸÊỀẾỂỄỆĐÌÍỈĨỊÒÓỎÕỌÔỒỐỔỖỘƠỜỚỞỠỢÙÚỦŨỤƯỪỨỬỮỰỲÝỶỸỴÂĂĐÔƠƯ"
	unsignChars = "aaaaaaaaaaaaaaaaaeeeeeeeeeeediiiiiooooooooooooooooouuuuuuuuuuuyyyyyAAAAAAAAAAAAAAAAAEEEEEEEEEEEDIIIOOOOOOOOOOOOOOOOOOOUUUUUUUUUUUYYYYYAADOOU"


	def loaddicchar():
	dic = {}

	function removeVietnameseTones(str) {
	str = str.replace(/à\|á\|ạ\|ả\|ã\|â\|ầ\|ấ\|ậ\|ẩ\|ẫ\|ă\|ằ\|ắ\|ặ\|ẳ\|ẵ/g,"a");
	str = str.replace(/è\|é\|ẹ\|ẻ\|ẽ\|ê\|ề\|ế\|ệ\|ể\|ễ/g,"e");
	str = str.replace(/ì\|í\|ị\|ỉ\|ĩ/g,"i");
	str = str.replace(/ò\|ó\|ọ\|ỏ\|õ\|ô\|ồ\|ố\|ộ\|ổ\|ỗ\|ơ\|ờ\|ớ\|ợ\|ở\|ỡ/g,"o");
	str = str.replace(/ù\|ú\|ụ\|ủ\|ũ\|ư\|ừ\|ứ\|ự\|ử\|ữ/g,"u");
	str = str.replace(/ỳ\|ý\|ỵ\|ỷ\|ỹ/g,"y");
	str = str.replace(/đ/g,"d");
	str = str.replace(/À\|Á\|Ạ\|Ả\|Ã\|Â\|Ầ\|Ấ\|Ậ\|Ẩ\|Ẫ\|Ă\|Ằ\|Ắ\|Ặ\|Ẳ\|Ẵ/g, "A");
	str = str.replace(/È\|É\|Ẹ\|Ẻ\|Ẽ\|Ê\|Ề\|Ế\|Ệ\|Ể\|Ễ/g, "E");