120B model serve runbook (gx10-83fb)

120B_SERVE_RUNBOOK.md

120B model serve runbook (gx10-83fb)

Host: gx10-83fb (Tailscale: gx10-83fb.tail3dac72.ts.net)
Port for 120B: 8002 (single active 120B service at a time)
Recommended stack: Qwen3.5 122B A10B + llama.cpp (Option D)

Authoritative artifacts to read first: docs/GX10_PORT_ASSIGNMENT.md, docs/evidence/gx10_runtime_baseline.json, ontology/configuration_management.ttl.

Required first step: Refresh runtime evidence with python3 scripts/capture_gx10_runtime_baseline.py, then run ./scripts/gx10_config_guard.py [--live] before any change. If it fails, fix the reported issues first.

---

1. Prerequisites on gx10

• Ubuntu 24.04 aarch64, NVIDIA driver 580.x, CUDA 13.
• Enough disk for GGUF (~70–80 GB for Qwen3.5-122B-A10B Q4_K_M).
• User with GPU access (e.g. in video group or session with GPU).

---

2. Install llama.cpp and OpenAI-compatible server

On gx10:

# Clone and build with CUDA (OpenAI-compatible server)
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_CUBLAS=ON -DGGML_CUDA=ON
cmake --build . --config Release -j

# Binary: ./bin/llama-server (or server in older builds)

Install or symlink so the service can run it (e.g. /opt/llama.cpp or $HOME/llama.cpp). Adjust paths in the systemd unit below.

---

3. Download Qwen3.5 122B A10B (Q4_K_M GGUF)

Example (Hugging Face):

# Create model dir (adjust to your layout)
mkdir -p /path/to/models
cd /path/to/models

# Install huggingface-hub if needed: pip install huggingface-hub
huggingface-cli download Qwen/Qwen3.5-122B-A10B-Instruct-GGUF \
  qwen3.5-122b-a10b-instruct-q4_k_m.gguf --local-dir . --local-dir-use-symlinks False

Or download the Q4_K_M GGUF file directly from Hugging Face Qwen3.5-122B-A10B-Instruct-GGUF. Use the Q4_K_M variant for ~64–72 GB VRAM on 128 GB GPU.

---

4. Systemd units (one per 120B model)

Only one of these services should be running at a time (same port 8002). Use the swap script (Section 6) to switch.

• qwen-122b.service — Qwen3.5 122B A10B on port 8002.
• nemotron-120b.service — (Optional) Nemotron 120B when added; also binds to 8002.

Copy the unit file from this repo’s deploy/ (or create from the template in Section 7). Install:

sudo cp deploy/qwen-122b.service /etc/systemd/system/
sudo systemctl daemon-reload
# Do not enable both; only one runs at a time
sudo systemctl start qwen-122b

Edit the unit to set:

• User=, Group= (user that has GPU access).
• ExecStart= paths: path to llama-server and path to the GGUF file.

---

5. Start command (reference)

Example for Qwen 122B (adjust paths):

/path/to/llama-server \
  --model /path/to/models/qwen3.5-122b-a10b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 \
  --port 8002 \
  --n-gpu-layers -1

For MoE, if needed: --n-cpu-moe N or tensor-split options per llama.cpp docs. Confirm with a quick test run.

---

6. Swap script (stop one 120B, start the other)

Use the script in deploy/swap-llm.sh (or equivalent on gx10). It:

1. Stops the currently running 120B service (e.g. qwen-122b or nemotron-120b).
2. Starts the requested service (e.g. qwen-122b or nemotron-120b).

Usage:

# Deploy script to gx10, then:
./swap-llm.sh qwen122b    # Stop others, start Qwen 122B on 8002
./swap-llm.sh nemotron120b   # Stop others, start Nemotron 120B (when added)

After swapping, verify with the healthcheck below.

---

7. Healthcheck

From gx10 or any machine with Tailscale to gx10:

# List models
curl -s http://gx10-83fb.tail3dac72.ts.net:8002/v1/models

# Chat completion (replace MODEL_ID with value from /v1/models, e.g. qwen3.5-122b-a10b-instruct)
curl -s -X POST http://gx10-83fb.tail3dac72.ts.net:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Hi"}],"max_tokens":32}'

Confirm HTTP 200 and non-empty choices[0].message.content. Use the reported id from /v1/models as the model in Goose’s custom provider.

---

8. Streaming / stream decode (keepalive)

Issue: Goose’s stream decoder expects specific event types. If the backend sends {"type":"keepalive"} or other unknown events, the decoder can abort (stream decode error).

Current mitigation: The 120B Goose provider uses supports_streaming: false, so Goose sends non-streaming requests and this mismatch is avoided.

To re-enable streaming later, apply one of:

1. Server/proxy fix (preferred): Disable keepalive on the 120B server (or proxy), or emit keepalive in a format Goose accepts.
2. Client tolerance: Change Goose’s streaming parser to ignore unknown event types (e.g. treat keepalive as no-op).
3. Schema alignment: Make the server emit exactly the SSE/event schema Goose expects.

See GOOSE_LLM_GX10_ACCESS.md for the 120B endpoint and streaming note.

---

9. Firewall

Ensure port 8002 is not blocked (e.g. no iptables DROP). If the server binds to 0.0.0.0:8002, Tailscale and LAN can reach it.

---

10. Summary

Item	Value
120B port	8002
Swap	./swap-llm.sh qwen122b or nemotron120b
Healthcheck	curl ... :8002/v1/models and /v1/chat/completions
Streaming	Off until a remediation is applied; see Section 8.

---

Verification evidence to capture

After deploy or change: (1) run python3 scripts/capture_gx10_runtime_baseline.py, (2) ./scripts/gx10_config_guard.py [--live] exits 0, (3) curl -s http://127.0.0.1:8002/v1/models returns JSON, (4) ss -tlnp on gx10 shows port 8002 listening, and (5) update the last-known-good state record only if the evidence is complete.