This was a full fine-tune of llama-2-13b-hf using dataset https://huggingface.co/datasets/jondurbin/airoboros-gpt4-2.0
Convert the JSONL (newline delimeted JSON strings) into conversational format that FastChat expects:
import re
Jon Durbin jondurbin
jondurbin
/ airoboros-l2-13b-gpt4-2.0-tuning.md
Last active
September 12, 2023 05:42
jondurbin
/ airoboros-l2-70b-gpt4-1.4.1-qlora.md
Created
July 26, 2023 12:39
Fork of qlora: https://github.com/jondurbin/qlora
Make sure to change dataset format, and dataset path to your file, along with model/output paths.
If you want to modify the prompt format, edit this: https://github.com/jondurbin/qlora/blob/main/qlora.py#L433
Args used:
python qlora.py \
jondurbin
/ airoboros-gpt4-2.0.md
Last active
November 2, 2023 06:30
https://github.com/jondurbin/airoboros
pip install --upgrade airoboros==2.0.13NewerOlder