Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions-clean.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
8x 80gb a100s
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
Dataset used: https://huggingface.co/datasets/jondurbin/airoboros-2.2
Specifically, the instructions.jsonl file.
Fine-tuned with my fork of qlora: https://github.com/jondurbin/qlora
This was a full fine-tune (yes, the script is called qlora, but I used the --full_finetune option)
export BASE_DIR=/workspace
export WANDB_API_KEY=[redacted]
This was a qlora fine-tune of llama-30b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
I used my fork of qlora: https://github.com/jondurbin/qlora which has support for airoboros dataset format, updated prompt format, etc.
This was a full fine-tune of llama-2-7b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re
This was a full fine-tune of llama-2-13b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re
Fork of qlora: https://github.com/jondurbin/qlora
Make sure to change dataset format, and dataset path to your file, along with model/output paths.
If you want to modify the prompt format, edit this: https://github.com/jondurbin/qlora/blob/main/qlora.py#L433
Args used:
python qlora.py \
https://github.com/jondurbin/airoboros
pip install --upgrade airoboros==2.0.13