My fork of qlora (but using the --full-finetune option), commit b5771f3caa9a5ea3ec397526a09720e957dc03d0 (main branch as of 2023-10-02):
6x 80GB a100s on runpod.io
Dataset: https://hf.co/datasets/jondurbin/airoboros-3.0 (private currently, until fine-tunes are finished)
Script:
Using my fork of qlora at this commit: https://github.com/jondurbin/qlora/tree/ef708769c9365eb86cf173aa9fc7c37e2d772773
Airoboros dataset 3.0: https://hf.co/datasets/jondurbin/airoboros-3.0
7x h100 servers on https://runpod.io/
export BASE_DIR=/workspace
Trained on 7x 80gb a100 nodes in runpod.
Dataset: https://hf.co/datasets/jondurbin/airoboros-2.2 (specifically, instructions.jsonl)
My fork of qlora: https://github.com/jondurbin/qlora
Merged with qmerge.py from my fork of qlora, similar to:
python qlora/qmerge.py --base llama-2-70b-hf --peft spicyboros-70b-2.2-checkpoints/checkpoint-1173/model_adapter --out spicyboros-2.2
Trained on 8x 80gb a100 nodes in runpod.
Dataset: https://hf.co/datasets/jondurbin/airoboros-2.2 (specifically, instructions.jsonl)
My fork of qlora: https://github.com/jondurbin/qlora
Note: the final selected checkpoint used to merge the model was checkpoint-750!
Merged with qmerge.py from my fork of qlora, similar to:
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
This was a qlora fine-tune of llama-30b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
I used my fork of qlora: https://github.com/jondurbin/qlora which has support for airoboros dataset format, updated prompt format, etc.
This was a full fine-tune of llama-2-7b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re