Ruocheng Guo rguo12

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Chat GPT "DAN" (and other "Jailbreaks")

海外兔

##VGG16 model for Keras

This is the Keras model of the 16-layer network used by the VGG team in the ILSVRC-2014 competition.

It has been obtained by directly converting the Caffe model provived by the authors.

Details about the network architecture can be found in the following arXiv paper:

Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, A. Zisserman

	# NOTE:
	# You can find an updated, more robust and feature-rich implementation
	# in Zeno Build
	# - Zeno Build: https://github.com/zeno-ml/zeno-build/
	# - Implementation: https://github.com/zeno-ml/zeno-build/blob/main/zeno_build/models/providers/openai_utils.py

	import openai
	import asyncio
	from typing import Any

	import torch
	import torch.nn as nn
	import torch.nn.functional as F


	class SpatialSoftArgmax(nn.Module):
	"""Spatial softmax as defined in [1].

	Concretely, the spatial softmax of each feature
	map is used to compute a weighted mean of the pixel

	import torch
	from torchvision import datasets

	class ImageFolderWithPaths(datasets.ImageFolder):
	"""Custom dataset that includes image file paths. Extends
	torchvision.datasets.ImageFolder
	"""

	# override the __getitem__ method. this is the method that dataloader calls
	def __getitem__(self, index):

	# (C) Mathieu Blondel, November 2013
	# License: BSD 3 clause

	import numpy as np


	def ranking_precision_score(y_true, y_score, k=10):
	"""Precision at rank k

	Parameters

	import itertools

	import numpy as np

	from sklearn.linear_model import SGDClassifier, SGDRanking
	from sklearn import metrics
	from minirank.compat import RankSVM as MinirankSVM
	from scipy import stats