Jon Durbin jondurbin

Trained on 10x a6000 GPUs on runpod.io.

I actually ran many fine-tunes, including multiple full-finetunes, fp16 loras, and qloras, and the below qlora actually did best in my testing.

dataset: https://hf.co/datasets/jondurbin/airoboros-3.1 (plus a few unpublished de-censoring instructions)

training script: https://github.com/jondurbin/qlora specifically commit 8cd269bf9bd7753c92164934269019e12f23314f

export BASE_DIR=/workspace

My fork of qlora (but using the --full-finetune option), commit b5771f3caa9a5ea3ec397526a09720e957dc03d0 (main branch as of 2023-10-02):

6x 80GB a100s on runpod.io

Dataset: https://hf.co/datasets/jondurbin/airoboros-3.0 (private currently, until fine-tunes are finished)

Script:

7x h100 servers on https://runpod.io/

export BASE_DIR=/workspace

Trained on 7x 80gb a100 nodes in runpod.

Merged with qmerge.py from my fork of qlora, similar to:

python qlora/qmerge.py --base llama-2-70b-hf --peft spicyboros-70b-2.2-checkpoints/checkpoint-1173/model_adapter --out spicyboros-2.2

Trained on 8x 80gb a100 nodes in runpod.

Note: the final selected checkpoint used to merge the model was checkpoint-750!

Merged with qmerge.py from my fork of qlora, similar to:

Specifically, the instructions-clean.jsonl file.

This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)

export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]

Specifically, the instructions-clean.jsonl file.

8x 80gb a100s

This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)

Specifically, the instructions.jsonl file.

8x 80gb a100s

This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)

	import aiohttp
	import asyncio
	import os
	import re
	from loguru import logger
	from playwright.async_api import async_playwright


	async def check_site(browser, domain, status):
	page = await browser.new_page()

	import re
	import gc
	import os
	import glob
	import json
	from copy import deepcopy
	from datasets import concatenate_datasets, Dataset
	from transformers import AutoTokenizer
	from huggingface_hub import snapshot_download