Yuekai Zhang yuekaizhang

Maybe you've heard about this technique but you haven't completely understood it, especially the PPO part. This explanation might help.

We will focus on text-to-text language models 📝, such as GPT-3, BLOOM, and T5. Models like BERT, which are encoder-only, are not addressed.

Reinforcement Learning from Human Feedback (RLHF) has been successfully applied in ChatGPT, hence its major increase in popularity. 📈

RLHF is especially useful in two scenarios 🌟:

You can’t create a good loss function
- Example: how do you calculate a metric to measure if the model’s output was funny?
You want to train with production data, but you can’t easily label your production data

create an env:

python3.8 -m venv .env
source .env/bin/activate
pip install -U pip
pip3 install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu113

fix the paths in simple.sh

Notes on `gdb` (GNU debugger)

Intel for display, NVIDIA for computing

This guide will show you how to use Intel graphics for rendering display and NVIDIA graphics for CUDA computing on Ubuntu 18.04 / 20.04 desktop.

I made this work on an ordinary gaming PC with two graphics devices, an Intel UHD Graphics 630 plus an NVIDIA GeForce GTX 1080 Ti. Both of them can be shown via lspci | grep VGA.

00:02.0 VGA compatible controller: Intel Corporation Device 3e92
01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

GDB for the Uninitiated

Overview

You're working on a C project for CS 2200. There's 2 days left before the assignment is due and for some reason, you're getting a segmentation fault after writing your last function. You've looked over your code 23 times but you just can't figure out where the bug is! Giving up and calling it a day, you decide to go to office hours and ask a TA for help.

If it sounds like this might be you in the near future, then allow me to run through a typical scenario for what happens when you come in and ask for help. You'll sit down, wait for 30 minutes while the TAs get through the queue of students asking questions, and finally notice that it's your turn next. The TA will walk over and ask you what's up. You'll explain that your code is seg-faulting and for the life of you, you can't figure out why. So the TA will take a quick peak at your code for any obvious errors, and then ask you the following question:

Did you run it through GDB?

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname

	import asyncio
	import aiohttp
	import json
	import wave
	import webrtcvad
	import contextlib
	import time
	from typing import List, Dict, Union, Optional
	from pathlib import Path
	import tempfile

	// Website you intended to retrieve for users.
	const upstream = 'api.openai.com'

	// Custom pathname for the upstream website.
	const upstream_path = '/'

	// Website you intended to retrieve for users using mobile devices.
	const upstream_mobile = upstream

	// Countries and regions where you wish to suspend your service.

	import re, collections

	def get_stats(vocab):
	pairs = collections.defaultdict(int)
	for word, freq in vocab.items():
	symbols = word.split()
	for i in range(len(symbols)-1):
	pairs[symbols[i],symbols[i+1]] += freq
	return pairs