carlosroman

Simple test to show TLS over UDS

To run:

go test -v -timeout 1s

The third test hangs (ClientUsesTLS) at the moment. The output does look like the client has encrypted the data as it got sent to the listner.

carlosroman

FROM python:3.11

WORKDIR /usr/src/app

COPY requirements.txt ./

RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

carlosroman

Setup llama.cpp

Clone git clone [email protected]:ggml-org/llama.cpp.git and then cd into llama.cpp and run the following:

uv venv --python 3.12
uv pip install --index-url https://repo.amd.com/rocm/whl/gfx1151/ "rocm[libraries,devel]"
source ./.venv/bin/activate
rocm-sdk init
deactivate

carlosroman

model	size	params	backend	ngl	n_batch	n_ubatch	type_k	type_v	fa	mmap	test	t/s
qwen35moe 122B.A10B Q5_K - Medium	82.16 GiB	122.11 B	ROCm	999	2048	1024	q8_0	q8_0	1	0	pp512 @ d16384	96.71 ± 0.00
qwen35moe 122B.A10B Q5_K - Medium	82.16 GiB	122.11 B	ROCm	999	2048	1024	q8_0	q8_0	1	0	tg128 @ d16384	15.90 ± 0.00
qwen35moe 122B.A10B Q5_K - Medium	82.16 GiB	122.11 B	ROCm	999	2048	1024	q8_0	q8_0	1	0	pp512 @ d32768	57.69 ± 0.00
qwen35moe 122B.A10B Q5_K - Medium	82.16 GiB	122.11 B	ROCm	999	2048	1024	q8_0	q8_0	1	0	tg128 @ d32768