Shaohao Zeng GregoryZeng

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Headers and Caching

Caching is a big subject and it can get pretty complicated. Luckily, the browsers have done all the really hard work for us and we just need to #useThePlatform.

There are three types of caches you'll likely deal with in Remix applications:

Application-level caches that you implement in your own code.
Browser caches you can control through the Cache-Control header.
CDN caches you also can control through the Cache-Control headers.

A Deep Dive into the Wonderful World of Preprocessing in NLP

https://mlexplained.com/2019/11/06/a-deep-dive-into-the-wonderful-world-of-preprocessing-in-nlp/

Many thanks to the author for writing in such depth and taking the time to explaining these topics.

Normalization(Cleaning) → Segmentation(tokenization) → Numericalization(vocab matching)

features

Python relative imports in AWS Lambda fail with attempted relative import with no known parent package

The Problem

In AWS Lambda if I attempt an explicit relative import like this

.
├── lambda_file.py
└── example.py

	import SwiftUI

	struct DuolingoButtonStyle: ButtonStyle {
	func makeBody(configuration: Configuration) -> some View {
	configuration.label
	.textCase(.uppercase)
	.fontWeight(.bold)
	.fontDesign(.rounded)
	.frame(maxWidth: .infinity)
	.padding()

	import { parse } from 'cache-control-parser';

	export default {
	async fetch(request: Request, env: {}, ctx: ExecutionContext): Promise<Response> {
	try {
	const cache = await caches.default;
	const cachedResponse = await cache.match(request);
	if (cachedResponse) {
	console.log('Cache: HIT');
	if (shouldRevalidate(cachedResponse)) {

	import torch
	import numpy as np
	import k_diffusion as K

	from PIL import Image
	from torch import autocast
	from einops import rearrange, repeat

	def pil_img_to_torch(pil_img, half=False):
	image = np.array(pil_img).astype(np.float32) / 255.0

	// Author: SwiftUI-Lab (swiftui-lab.com)
	// Description: Implementation of the showSizes() debugging modifier
	// blog article: https://swiftui-lab.com/layout-protocol-part-2

	import SwiftUI

	struct MeasureExample: View {
	var body: some View {

	VStack {

	import Foundation
	import SwiftUI

	// MARK: Demo

	struct Example: RawRepresentable, Codable {
	let id: UUID
	}

	// This is functionally the same as above, but we have a neater type without creating a new protocol.

	0.
	基于tf2.0
	1.
	把已有的模型导出成pb格式
	保存的时候的INFO:tensorflow:Unsupported signature for serialization貌似不用管
	python code:

	import os
	os.environ['TF_KERAS'] = '1'
	import numpy as np