Skip to content

Instantly share code, notes, and snippets.

@a1678991

a1678991/Containerfile

Created June 18, 2026 06:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save a1678991/bd99e4d61a3bef683623b3cee272edb0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save a1678991/bd99e4d61a3bef683623b3cee272edb0 to your computer and use it in GitHub Desktop.
Download ZIP
llama.cpp rpc-server container
Raw
Containerfile
FROM nvidia/cuda:13.3.0-devel-ubuntu24.04
RUN apt-get update && apt-get install -y git cmake openssl-dev npm
RUN useradd -m user
USER user
WORKDIR /home/user
RUN git clone https://github.com/ggml-org/llama.cpp
WORKDIR /home/user/llama.cpp/build-rpc-cuda
RUN cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
RUN cmake --build . --config Release --parallel 10
ENTRYPOINT ["/home/user/llama.cpp/build-rpc-cuda/bin/rpc-server", "--host 0.0.0.0"]
@a1678991

a1678991 commented Jun 19, 2026

Copy link
Copy Markdown
Author

ビルド時にGPUがないと問題が起きるので実行時にした方が良い
実行時に毎回ビルドするのは現実的ではないのでいい感じのキーを考えてキャッシュを入れる

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment