Skip to content

Instantly share code, notes, and snippets.

@akhenakh

akhenakh/llama.sh

Last active November 30, 2023 22:16
Show Gist options
  • Save akhenakh/12bbb8e1a438c53b2f94d347a5053031 to your computer and use it in GitHub Desktop.
Save akhenakh/12bbb8e1a438c53b2f94d347a5053031 to your computer and use it in GitHub Desktop.
Download ZIP
Use a llama model on your mac
Raw
llama.sh
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j
pip3 install huggingface-hub
huggingface-cli download TheBloke/zephyr-7B-alpha-GGUF zephyr-7b-alpha.Q5_K_M.gguf --local-dir ./models --local-dir-use-symlinks False
export MODEL=./models/zephyr-7b-alpha.Q5_K_M.gguf
./examples/server-llama2-13B.sh
# go to http://localhost:8080
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment