Jon Durbin jondurbin

Trained on 7x 80gb a100 nodes in runpod.

Merged with qmerge.py from my fork of qlora, similar to:

python qlora/qmerge.py --base llama-2-70b-hf --peft spicyboros-70b-2.2-checkpoints/checkpoint-1173/model_adapter --out spicyboros-2.2

7x h100 servers on https://runpod.io/

export BASE_DIR=/workspace

My fork of qlora (but using the --full-finetune option), commit b5771f3caa9a5ea3ec397526a09720e957dc03d0 (main branch as of 2023-10-02):

6x 80GB a100s on runpod.io

Dataset: https://hf.co/datasets/jondurbin/airoboros-3.0 (private currently, until fine-tunes are finished)

Script:

Trained on 10x a6000 GPUs on runpod.io.

I actually ran many fine-tunes, including multiple full-finetunes, fp16 loras, and qloras, and the below qlora actually did best in my testing.

dataset: https://hf.co/datasets/jondurbin/airoboros-3.1 (plus a few unpublished de-censoring instructions)

training script: https://github.com/jondurbin/qlora specifically commit 8cd269bf9bd7753c92164934269019e12f23314f

export BASE_DIR=/workspace

python3 -m venv chutes-venv
source chutes-venv/bin/activate
pip install chutes 'bittensor<8'

If you don't already have a coldkey/hotkey (replace chutes/chuteshk with your desired coldkey/hotkey names)

	import re
	import gc
	import os
	import glob
	import json
	from copy import deepcopy
	from datasets import concatenate_datasets, Dataset
	from transformers import AutoTokenizer
	from huggingface_hub import snapshot_download

	import aiohttp
	import asyncio
	import os
	import re
	from loguru import logger
	from playwright.async_api import async_playwright


	async def check_site(browser, domain, status):
	page = await browser.new_page()

	{
	"id": "27ab0d1289814bb28c7c30e38a98df8d",
	"object": "chat.completion",
	"created": 1742109451,
	"model": "cognitivecomputations/Dolphin3.0-Mistral-24B",
	"choices": [
	{
	"index": 0,
	"message": {
	"role": "assistant",

	import os
	import requests
	import base64

	audio = base64.b64encode(open("test.wav", "rb").read()).decode()
	result = requests.post(
	"https://chutes-csm-1b.chutes.ai/speak",
	json={
	"speaker": 1,
	"context": [

	import os
	import requests
	import base64

	audio = base64.b64encode(open("test.wav", "rb").read()).decode()
	result = requests.post(
	"https://chutes-spark-tts.chutes.ai/speak",
	json={
	"text": "How much wood would a woodchuck chuck if a woodchuck could chuck wood?",
	"sample_audio_b64": audio,