-
Star
(338)
You must be signed in to star a gist -
Fork
(45)
You must be signed in to fork a gist
-
-
Save adrienbrault/b76631c56c736def9bc1bc2167b5d129 to your computer and use it in GitHub Desktop.
# Clone llama.cpp | |
git clone https://github.com/ggerganov/llama.cpp.git | |
cd llama.cpp | |
# Build it | |
make clean | |
LLAMA_METAL=1 make | |
# Download model | |
export MODEL=llama-2-13b-chat.ggmlv3.q4_0.bin | |
wget "https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolve/main/${MODEL}" | |
# Run | |
echo "Prompt: " \ | |
&& read PROMPT \ | |
&& ./main \ | |
--threads 8 \ | |
--n-gpu-layers 1 \ | |
--model ${MODEL} \ | |
--color \ | |
--ctx-size 2048 \ | |
--temp 0.7 \ | |
--repeat_penalty 1.1 \ | |
--n-predict -1 \ | |
--prompt "[INST] ${PROMPT} [/INST]" |
Thanks for above.
I was running into an error:
error loading model: failed to open --color: No such file or directory
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '--color'
main: error: unable to load model
Deleted everything and then ran:
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git reset --hard 1aa18ef
Then ran the rest of gist and it worked again.
Yeah, latest llama.cpp is no longer compatible with GGML models. The new model format, GGUF, was merged recently. As far as llama.cpp is concerned, GGML is now dead
https://huggingface.co/TheBloke/vicuna-13B-v1.5-16K-GGML/discussions/6#64e5ba63a9a5eabaa6fd4a04
Replacing the GGML model with a GGUF model
https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF/blob/main/llama-2-7b-chat.Q8_0.gguf
You can check if it works:
PROMPT> ./main -m models/llama-2-7b-chat.Q8_0.gguf --random-prompt
snip lots of info
response to the prompt
After years of hard work and dedication, a high school teacher in Texas has been recognized for her outstanding contributions to education.
Ms. Rodriguez, a mathematics teacher at...
Does anybody know how to adjust the prompt input to include multiple lines of input before submitting the prompt?
Thanks for above.
I was running into an error:
error loading model: failed to open --color: No such file or directory llama_load_model_from_file: failed to load model llama_init_from_gpt_params: error: failed to load model '--color' main: error: unable to load model
Deleted everything and then ran:
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git reset --hard 1aa18ef
Then ran the rest of gist and it worked again.
Seems to have worked once but now continues to fail. Any ideas why @smart-patrol ?
Prompt:
How large is the sun?
main: build = 904 (1aa18ef)
main: seed = 1700587479
error loading model: failed to open --color: No such file or directory
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '--color'
main: error: unable to load model
same issue
There's a similar error reported in the Python bindings for llama.cpp. Sounds like need to wait for a new model format to be available. In the meantime, a temporary workaround is to checkout an older release of llama.cpp, for example:
Which is for this release from Jul 25.
Then run the build again.