Skip to content

Instantly share code, notes, and snippets.

@mattpetters

mattpetters/runpod_codex_36.md

Forked from goodalexander/runpod_codex_36.md
Last active May 16, 2026 09:20
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mattpetters/52199263c65b536aebd5810a4bf19eff to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mattpetters/52199263c65b536aebd5810a4bf19eff to your computer and use it in GitHub Desktop.
Download ZIP
how to get Qwen3.6 going on a runpod
Raw
runpod_codex_36.md

Self-Hosted Qwen3.6-27B on RunPod + OpenAI Codex CLI

End-to-end guide for running Qwen3.6-27B as a private, self-hosted coding agent accessible from your Mac via SSH tunnel and OpenAI Codex CLI. Hardware-tested on RTX PRO 6000 Blackwell. ~$1.91/hr.

Prerequisites

  • RunPod account with credit loaded
  • SSH key on your Mac (~/.ssh/id_ed25519 and .pub)
  • Node.js 18+ on your Mac (for Codex CLI)
  • Hugging Face account + read token (avoids rate limits on downloads)

Architecture

Mac (Codex CLI)
    ↓
SSH tunnel localhost:8000 → pod:8000
    ↓
Ollama server on RunPod pod
    ↓
Qwen3.6-27B (Q4_K_XL GGUF) on RTX PRO 6000 Blackwell

Step 1: Register your SSH key with RunPod

On your Mac, print your public key:

cat ~/.ssh/id_ed25519.pub

If it prints nothing, generate one:

ssh-keygen -t ed25519 -C "runpod" -f ~/.ssh/id_ed25519 -N ""
cat ~/.ssh/id_ed25519.pub

In RunPod: Settings → SSH Public Keys → paste the key → Update public key.

Step 2: Deploy the pod

RunPod → Pods → Deploy → GPU Pod. Select:

  • GPU: RTX PRO 6000 (96GB VRAM, ~$1.89/hr) — good balance of price and VRAM headroom
  • Template: runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04 (or similar RunPod PyTorch template)
  • Container disk: 100GB
  • Volume disk: 150GB (critical — the model is 17GB quantized but you need room for caches and workspace)
  • Volume mount path: /workspace
  • Expose TCP port: 22 (for direct SSH)
  • Start Jupyter Notebook: yes

Click Deploy On-Demand. Wait ~2 minutes for the pod to come up.

Step 3: Open a JupyterLab terminal on the pod

In RunPod → your pod → Connect → Jupyter Lab. In JupyterLab: File → New → Terminal.

Verify environment:

nvidia-smi
df -h /workspace

You should see the GPU free (0 MiB used) and /workspace with ~150GB available.

Step 4: Install dependencies and download the model

In the pod terminal:

cd /workspace
export HF_HOME=/workspace/huggingface
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=hf_your_token_here    # replace with your real token

pip install -q -U huggingface_hub hf_transfer # root action ignore if this fails

Download the Q4_K_XL quant (~17GB, ~1 min on RunPod's network):

hf download unsloth/Qwen3.6-27B-GGUF \
  --include "*UD-Q4_K_XL*" \
  --local-dir /workspace/gguf

Verify:

ls -lh /workspace/gguf/

You should see Qwen3.6-27B-UD-Q4_K_XL.gguf (~17GB).

Step 5: Install Ollama

Why Ollama instead of vLLM or llama.cpp: vLLM's prebuilt CUDA kernels don't support Blackwell's sm_120 for Qwen3.6 yet (CUDA PTX errors). llama.cpp works for basic inference but hits a known bug with OpenAI Codex CLI v0.88+ ('type' of tool must be 'function'). Ollama handles both Blackwell and Codex tool calling cleanly.

curl -fsSL https://ollama.com/install.sh | sh

Ignore the "systemd is not running" and "Unable to detect NVIDIA GPU" warnings — they're harmless in RunPod containers. Ollama auto-detects the GPU at runtime.

Step 6: Start Ollama server

Run Ollama on port 8000 (matches the tunnel we'll set up), with its model registry on the persistent volume:

OLLAMA_MODELS=/workspace/ollama-models OLLAMA_HOST=0.0.0.0:8000 nohup ollama serve > /workspace/ollama.log 2>&1 &

Wait 2 seconds and verify:

sleep 2
curl http://localhost:8000/api/tags

Should return {"models":[]}.

Step 7: Pull the tool-enabled Qwen3.6 from Ollama's registry

Do not try to ollama create from the local GGUF — the resulting model won't advertise tool-calling capability, and Codex will reject it. Pull Ollama's official build instead (includes proper chat template with tool calling):

OLLAMA_HOST=0.0.0.0:8000 ollama pull qwen3.6:27b

~17GB, 1-2 minutes. When it finishes, verify capabilities:

OLLAMA_HOST=0.0.0.0:8000 ollama show qwen3.6:27b

Under Capabilities you should see: completion, vision, tools, thinking. All four matter. If tools is missing, Codex will fail.

Since Ollama re-downloaded, you can optionally delete the original GGUF to save space:

rm -rf /workspace/gguf

Step 8: Set up SSH tunnel from your Mac

In RunPod → your pod → Connect. Find the "SSH over exposed TCP" section (has a numeric IP, not ssh.runpod.io). It looks like:

ssh root@198.13.252.15 -p 12570 -i ~/.ssh/id_ed25519

If the pod's authorized_keys is missing your key (rare, but happens when keys are added after pod creation), first SSH in via the proxy (ssh <pod-id>@ssh.runpod.io) or use JupyterLab's terminal to fix it:

mkdir -p /root/.ssh
echo "$(cat ~/.ssh/id_ed25519.pub | head -c 1000)" >> /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys

(Paste your actual public key string in place of the $(...) — or just paste the key literally.)

Back on your Mac, open the tunnel:

ssh -L 8000:localhost:8000 \
    -o ServerAliveInterval=60 \
    -o ServerAliveCountMax=10 \
    -o StrictHostKeyChecking=no \
    root@198.13.252.15 -p 12570 -i ~/.ssh/id_ed25519

Replace IP and port with yours. You'll land in the pod shell. Leave this terminal open — closing it closes the tunnel.

Step 9: Verify the tunnel

In a new Mac terminal window:

curl http://localhost:8000/v1/models

Should return JSON listing qwen3.6:27b. If it does, the tunnel works and Ollama is reachable.

Step 10: Install and configure Codex CLI

On your Mac:

npm install -g @openai/codex@latest
codex --version

You need v0.122+ (the Ollama setup above works with the latest Codex; earlier llama.cpp-based setups required v0.87 due to a bug).

Create the Codex config:

mkdir -p ~/.codex
cat > ~/.codex/config.toml << 'EOF'
model_provider = "runpod"
model = "qwen3.6:27b"
model_context_window = 32768
model_max_output_tokens = 4096

[model_providers.runpod]
name = "RunPod Ollama"
base_url = "http://localhost:8000/v1"
wire_api = "responses"
stream_idle_timeout_ms = 1800000

[profiles.runpod]
model = "qwen3.6:27b"
model_provider = "runpod"
EOF

Key settings explained:

  • wire_api = "responses" — Codex dropped chat-completions support in v0.118
  • stream_idle_timeout_ms = 1800000 — 30-min timeout, needed because Qwen3.6 thinking blocks are slow
  • model_context_window = 32768 — matches what Ollama advertises after our pull

Set a dummy API key (required even for local):

echo 'export OPENAI_API_KEY="dummy"' >> ~/.zshrc
export OPENAI_API_KEY="dummy"

Step 11: Launch Codex

codex --profile runpod

Test it with a simple prompt:

› say hi

You may see ⚠ Model metadata for 'qwen3.6:27b' not found — that's cosmetic, Codex just doesn't have a preset for this model name. Ignore it.

Test tool calling:

› list the files in this directory

If that returns file names without a 'type' of tool must be 'function' error, your full stack is working.

Day-to-day usage

Starting a session

  1. Open pod in RunPod console → Start (if stopped)
  2. Open Mac terminal → SSH tunnel: 
    ssh -L 8000:localhost:8000 root@<IP> -p <PORT> -i ~/.ssh/id_ed25519
  3. In pod shell, verify Ollama is running: 
    curl http://localhost:8000/v1/models
    If empty response, restart Ollama: 
    OLLAMA_MODELS=/workspace/ollama-models OLLAMA_HOST=0.0.0.0:8000 nohup ollama serve > /workspace/ollama.log 2>&1 &
  4. New Mac terminal → cd into your project → codex --profile runpod

Stopping a session

Pod console → Stop (NOT Terminate). Stopped pods retain the volume ($0.015/hr vs $1.91/hr running) and resume in ~30 seconds. Terminating wipes /workspace and you lose the model download.

Performance expectations

  • Raw generation speed: ~56 tok/s on RTX PRO 6000, ~90 tok/s on H100
  • First Codex request in a session: 30-90 seconds (Codex's ~27K-token system prompt prefill)
  • Subsequent chat-only prompts: 5-20 seconds
  • Tool-calling agent loops (file edits, multi-step): 1-5 minutes per cycle due to thinking overhead

Qwen3.6 is a thinking model — every response starts with a 500-1500 token <think> block before the actual answer. This makes it noticeably slower than Claude Code via Anthropic's API for agentic work. It's best for sensitive IP work where privacy outweighs speed.

Troubleshooting

Permission denied (publickey) on SSH

Your public key isn't in the pod's authorized_keys. SSH in via ssh <pod-id>@ssh.runpod.io (the proxy — your key works there) or use JupyterLab's terminal and manually add:

echo "ssh-ed25519 AAAA... youremail@example.com" >> /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys

Connection reset by peer on curl from Mac

Ollama isn't running or the tunnel dropped. Check:

  1. On Mac, is the SSH tunnel terminal still alive? If not, re-open it.
  2. On pod, is Ollama running? curl http://localhost:8000/api/tags from the pod.

'type' of tool must be 'function' in Codex

You're running llama.cpp instead of Ollama, or the Ollama model you're using doesn't have tools capability. Run ollama show <model> and verify tools appears under Capabilities. If not, pull the official Qwen3.6 build: ollama pull qwen3.6:27b.

does not support tools

Same root cause — your Ollama model doesn't advertise tool support. Same fix.

CUDA / PTX errors during vLLM startup

vLLM 0.19.x doesn't have Blackwell (sm_120) kernels for Qwen3.6. Don't use vLLM for this model on RTX PRO 6000. Use Ollama (this guide).

Ollama says GPU not detected on install

Cosmetic warning during curl ollama.com/install.sh | sh. Ollama auto-detects at runtime from the CUDA driver, which works fine in RunPod containers.

Codex hangs on first prompt for minutes

Normal. Prefill of ~27K system prompt tokens takes 30-90 sec on this hardware. If it exceeds 5 minutes, check the Ollama log in the pod: tail -f /workspace/ollama.log.

Cost estimate

  • Active use: $1.91/hr (GPU + storage)
  • Stopped pod with storage: ~$0.015/hr (just the 150GB volume)
  • Cost per hour of actual coding: $1.91
  • Monthly estimate at 4 hrs/day, 20 days: ~$153 + negligible storage when stopped

For comparison: Claude Max subscription is $200/mo, and you're trading privacy for that.

When to use what

Use this setup (local Qwen3.6) for:

  • IP-sensitive code you don't want in training data
  • Offline / limited-connectivity situations (if you host on a local machine instead)
  • Learning how agentic coding stacks work under the hood

Use Claude Code or Codex with frontier models for:

  • Multi-step refactors where speed matters
  • Complex reasoning that benefits from Opus/Sonnet-class models
  • Anything where 5× faster tool calls beat privacy

Many builders run both — local for sensitive work, hosted for speed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment