https://github.com/jondurbin/airoboros
pip install --upgrade airoboros==2.0.13
Jon Durbin jondurbin
jondurbin
/ airoboros-gpt4-2.0.md
Last active
November 2, 2023 06:30
jondurbin
/ airoboros-l2-70b-gpt4-1.4.1-qlora.md
Created
July 26, 2023 12:39
Fork of qlora: https://github.com/jondurbin/qlora
Make sure to change dataset format, and dataset path to your file, along with model/output paths.
If you want to modify the prompt format, edit this: https://github.com/jondurbin/qlora/blob/main/qlora.py#L433
Args used:
python qlora.py \
jondurbin
/ airoboros-l2-13b-gpt4-2.0-tuning.md
Last active
September 12, 2023 05:42
This was a full fine-tune of llama-2-13b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re
jondurbin
/ airoboros-l2-7b-gpt4-2.0-tuning.md
Last active
March 24, 2024 09:04
This was a full fine-tune of llama-2-7b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re
jondurbin
/ airoboros-33b-gpt4-2.0-tuning.md
Last active
July 31, 2023 12:57
This was a qlora fine-tune of llama-30b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
I used my fork of qlora: https://github.com/jondurbin/qlora which has support for airoboros dataset format, updated prompt format, etc.
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
jondurbin
/ spicyboros-13b-2.2-tuning.md
Last active
September 10, 2023 11:33
spicyboros-13b-2.2-tuning
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
jondurbin
/ airoboros-l2-13b-2.2-tuning.md
Created
September 10, 2023 11:32
airoboros-l2-13b-2.2-tuning
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
jondurbin
/ airoboros-l2-7b-2.2-tuning.md
Last active
September 10, 2023 11:41
airoboros-l2-7b-2.2-tuning
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
jondurbin
/ spicyboros-70b-2.2-tuning.md
Last active
September 18, 2024 18:42
spicyboros-70b-2.2-tuning
Trained on 8x 80gb a100 nodes in runpod.
Dataset: https://hf.co/datasets/jondurbin/airoboros-2.2 (specifically, instructions.jsonl)
My fork of qlora: https://github.com/jondurbin/qlora
Note: the final selected checkpoint used to merge the model was checkpoint-750!
Merged with qmerge.py from my fork of qlora, similar to:
OlderNewer