Skip to content

Instantly share code, notes, and snippets.

@louspringer

louspringer/GOOSE_LLM_GX10_ACCESS.md

Created March 17, 2026 21:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save louspringer/ba9376cd87a2221c8753cb2e4276d8f8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save louspringer/ba9376cd87a2221c8753cb2e4276d8f8 to your computer and use it in GitHub Desktop.
Download ZIP
Goose: accessing the LLM on gx10
Raw
GOOSE_LLM_GX10_ACCESS.md

Goose configuration: accessing the LLM on gx10

Interrogation date: 2026-03-13

How to check status and whether the model is coming up: See LLM_STATUS_AND_HEALTH.md.

Goose config locations

File Purpose
~/.config/goose/config.yaml Main config: GOOSE_PROVIDER, GOOSE_MODEL, MCPs, extensions
~/.config/goose/profiles.yaml LLM provider profiles (engine, model)
~/.config/goose/custom_providers/*.json Custom OpenAI-compatible endpoints (e.g. gx10)

Current setup

  • Default profile (profiles.yaml): engine: openai, model: Qwen2.5-Coder-7B-Instruct.
    Comment: "OpenAI provider uses OPENAI_HOST/OPENAI_API_KEY from env (set by Launch Agent at login)."
  • Custom provider for gx10: ~/.config/goose/custom_providers/local_qwen.json:
{
  "name": "local_qwen",
  "engine": "openai",
  "display_name": "Local Qwen (GX10)",
  "api_key_env": "OPENAI_API_KEY",
  "base_url": "http://gx10-83fb.tail3dac72.ts.net:8000",
  "models": ["Qwen2.5-Coder-7B-Instruct"]
}

So the LLM on gx10 is already configured in Goose as the Local Qwen (GX10) custom provider, pointing at:

  • URL: http://gx10-83fb.tail3dac72.ts.net:8000
    (Tailscale DNS for gx10; port 8000 = trtllm-serve (7B) or LLM proxy from this repo when deployed. See GX10_PORT_ASSIGNMENT.md.)

Port 8000 status: As of the last probe, 8000 is still serving stub responses (/v1/models → stubbed list; /v1/responses → “GX10 stub is up.”). Goose cannot use 8000 for the real 7B until the proxy is what’s on 8000 and points at the real backend. See LLM_PROXY_8000_RUNBOOK.md — “Current state” and “Remediation.”

  • Auth: OPENAI_API_KEY from environment (same env var name as OpenAI).

Goose UI checklist (select the model that is up)

  • Two providers: You need both Local Qwen (GX10) (7B, port 8000) and Qwen 122B (GX10) (120B, port 8002) in Switch models → Use other provider. See GOOSE_CUSTOM_PROVIDERS_PROMPT.md for exact JSON.
  • Select the one that is up: Only one of 7B or 120B runs at a time (single GPU). Before using Goose, check http://gx10-83fb.tail3dac72.ts.net:8000/status: if 8002 is "up", use Qwen 122B (GX10); if 8003 is "up", use Local Qwen (GX10). If you pick the wrong one you get 503 (7B down) or connection refused (120B down).
  • Both providers must have supports_streaming: false or you get: "Stream decode error: unknown variant keepalive". Edit each provider JSON under ~/.config/goose/custom_providers/ and set "supports_streaming": false, then restart Goose.

How to access the LLM on gx10 via Goose

  1. Switch models in Goose

    • Open Switch models and click "Use other provider" — the gx10 custom providers (e.g. Local Qwen (GX10), Qwen 122B (GX10)) appear there, not in the main provider list.
    • Or set GOOSE_PROVIDER and GOOSE_MODEL in ~/.config/goose/config.yaml (e.g. custom_qwen_122b and the model id), then restart Goose.
    • Ensure OPENAI_API_KEY is set in the environment where Goose runs (value can be a placeholder if the gx10 proxy/stub does not validate it).

  2. If you run Goose on gx10 itself

    • The same provider works (Tailscale hostname resolves to gx10).
    • Alternatively you can point at localhost by adding or editing a custom provider, e.g.:
      • llm_proxy (full stack): http://localhost:8000
      • gx10_stub only: http://localhost:8001

  3. If you want the default profile to use gx10

    • Set env before starting Goose:
      • OPENAI_BASE_URL or OPENAI_HOST (check Goose docs for the exact name) = http://gx10-83fb.tail3dac72.ts.net:8000 (or http://localhost:8000 on gx10).
      • OPENAI_API_KEY = your key or placeholder.
    • Keep default profile as engine: openai, model: Qwen2.5-Coder-7B-Instruct.

What is on gx10 (canonical: see docs/GX10_PORT_ASSIGNMENT.md)

  • Port 8000: Currently trtllm-serve (7B), harmony-only; not Goose-compatible. Optional fix: deploy the LLM proxy (this repo) on 8000 so it accepts standard Responses; the proxy must forward to a backend on a port other than 8001 (e.g. 8003). Port 8001 is assigned to gx10_stub; do not use it as the proxy backend.
  • Port 8001: gx10_stub (eudorus) — minimal OpenAI-style /v1/models and /v1/chat/completions. Do not reassign. Eudorus llm_proxy (when used) calls gx10_stub at 8001.
  • Port 8002: 120B (Qwen 122B or Nemotron); one at a time via swap script. See 120B_SERVE_RUNBOOK.md.

Port 8000 — root cause and fix: The service on gx10:8000 (trtllm-serve) rejects standard Responses API and only accepts harmony format. This repo owns the fix: run the LLM proxy on 8000 so it accepts standard Responses and forwards to a backend that speaks Chat Completions. The backend must not be on 8001 (8001 is gx10_stub). Use another port (e.g. 8003). See LLM_PROXY_8000_RUNBOOK.md and GX10_PORT_ASSIGNMENT.md.

When the chosen model is not available (single GPU)

On gx10 only one of 7B or 120B can run at a time (single GPU). Availability can change after you start or stop a model.

  • You picked Local Qwen (7B) but the 7B backend is not running
    The proxy on 8000 is up, but the backend on 8003 is down (e.g. 120B is using the GPU). Goose will get 503 with a JSON error whose message explains that the model is not available and that only one of 7B or 120B can run at a time. What to do: On gx10 run ./deploy/easy.sh 7b (or ./deploy/remediate_8000_proxy_and_backend.sh), or switch in Goose to the 120B provider if you want to use the 120B.

  • You picked Qwen 122B (120B) but the 120B service is not running
    Nothing is listening on 8002. Goose will get connection refused (no HTTP response). What to do: On gx10 run ./deploy/easy.sh 120b (or start the 120B service and stop the 7B backend to free the GPU). See 120B_SERVE_RUNBOOK.md.

Until there is more capacity (e.g. a second GPU or cloud), treat 503 or connection refused as “the other model is using the GPU or this model isn’t started.”

Streaming mismatch (word-by-word or one chunk per line)

Symptom: Replies show up as many short lines, each with a timestamp and "Copy" (e.g. "Deployment", "resolution", "implemented", "for", "each", "service" as separate bubbles).

Cause: Goose was requesting streaming (stream: true) while the eudorus backend (llm_proxy → gx10_stub, and optionally OpenAI post-processor) returns a single completion (non-streaming). The client then misparsed or rendered the response as separate chunks.

  • llm_proxy (port 8000): Already rejects stream: true with 400 streaming_not_supported.
  • gx10_stub (port 8001): Now rejects stream: true with 400 so clients must use stream: false.

Fix:

  1. Force non-streaming in the custom provider
    Set "supports_streaming": false in ~/.config/goose/custom_providers/local_qwen.json so Goose sends stream: false and gets one JSON completion (already applied to Local Qwen (GX10)). Restart Goose after editing. If your provider config has a streaming toggle in the UI, turn it off for this provider.

  2. If you see a 400 after the change
    The backend is rejecting stream: true. With supports_streaming: false in the custom provider, Goose should send stream: false; restart Goose and try again.

120B-class endpoint (Qwen 122B or Nemotron 120B)

A separate custom provider is used for the 120B model(s) on gx10:

  • Port: 8002 (single active 120B service at a time; swap via script on gx10).
  • Base URL: http://gx10-83fb.tail3dac72.ts.net:8002 (or http://localhost:8002 when Goose runs on gx10).
  • Custom provider file: ~/.config/goose/custom_providers/qwen_122b.json (for Qwen 122B); use the same port for Nemotron when added.
  • Model id: Set to the value returned by the server’s GET /v1/models (e.g. qwen3.5-122b-a10b-instruct or as reported). Restart Goose after editing.
  • Streaming: Use supports_streaming: false for the 120B provider. Re-enable only after applying a streaming remediation (see “Streaming / stream decode” below).
  • Swap: To change which 120B model is live, on gx10 run the swap script (e.g. ./swap-llm.sh qwen122b or ./swap-llm.sh nemotron120b). See 120B_SERVE_RUNBOOK.md.

Streaming / stream decode (keepalive)

Issue: Goose’s stream decoder expects a fixed set of event types. If the backend (eudorus proxy or 120B server) sends {"type":"keepalive"} or other unknown event types, the decoder can abort with a stream decode error. This is a protocol mismatch, not a transient network fault.

Fix for "unknown variant keepalive": In both provider JSON files under ~/.config/goose/custom_providers/ (Local Qwen and Qwen 122B), set "supports_streaming": false. Restart Goose. Goose will then send non-streaming requests and will not see keepalive events.

Current mitigation: Both the Local Qwen (GX10) and the 120B custom provider use supports_streaming: false, so Goose sends non-streaming requests and the mismatch is avoided.

Proxy fix (port 8000): The LLM proxy (deploy/llm_proxy_8000.py) now strips keepalive events from any SSE stream it passes through. When Goose hits 8000 and the backend returns a stream, the proxy drops {"type":"keepalive",...} events before forwarding so Goose only sees the response.* and error events it expects. Applies to GET or POST pass-through with Content-Type: text/event-stream. For 8002 (120B), keep supports_streaming: false unless the server stops sending keepalive or a filter is placed in front of it.

To re-enable streaming later, apply one of:

  1. Server/proxy fix (preferred): Disable keepalive events, or emit them in a format Goose accepts (proxy on 8000 now filters them).
  2. Client tolerance: Change Goose’s streaming parser to ignore unknown event types (e.g. treat keepalive as no-op).
  3. Schema alignment: Make the server/proxy emit exactly the SSE/event schema Goose expects.

Until the 120B server is fixed or filtered, keep supports_streaming: false for the provider that talks to port 8002; streaming through the proxy on 8000 is now safe.

Model notes (retrieval / accuracy)

Benchmarks indicate that Qwen 3.5 (and the Qwen family in this size class) is particularly strong on accurate retrieval and related tasks — often competitive with or ahead of frontier and larger open models on those workloads despite smaller parameter count. The 7B backend on gx10 (Qwen2.5-Coder-7B-Instruct, and any Qwen 3.x variants we run) is therefore a good fit when the goal is retrieval quality or accuracy over raw scale. Prefer routing retrieval/QA-style requests to the Local Qwen (GX10) provider when that is the priority. See: issue #2 (proxy routing by task so retrieval requests hit the Qwen backend) and issue #3 (Qwen 3.5 retrieval/accuracy benchmarks summary and links).

Summary

  • To use the LLM on gx10 from Goose: Select the Local Qwen (GX10) provider in Goose and set OPENAI_API_KEY in the environment.
  • Endpoint (7B / proxy): http://gx10-83fb.tail3dac72.ts.net:8000 (from other machines) or http://localhost:8000 (when running on gx10).
  • Endpoint (120B): http://gx10-83fb.tail3dac72.ts.net:8002; use the Qwen 122B (GX10) (or equivalent) custom provider; swap model on gx10 via runbook script.
  • Model names: 7B = Qwen2.5-Coder-7B-Instruct; 120B = as reported by GET :8002/v1/models.
  • Streaming: Use non-streaming for both providers (supports_streaming: false). See “Streaming / stream decode” above for re-enabling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment