LLM local AI for coding on DGX Spark

README.md

This Gist explains how to run Qwen3.6-35B-A3B on an NVIDIA DGX Spark with a 128 KB context window, which is necessary (in my case) to do anything useful.

Chat-only works fine with PhpStorm, but successfully working with Visual Studio Code for macOS and extension Cline (saoudrizwan.claude-dev) using following configuration:

• API provider: OpenAI Compatible
• Base URL: http://{{DGX-SPARK-IP}}:8001/v1
• OpenAI Compatible API Key: placeholder
• Model ID: Qwen/Qwen3.6-35B-A3B
• Model Configuration
  • Context Window Size: 131072
  • Max Output Tokens: 4096

docker-compose.yml

services:
  sglang-backend:
    image: lmsysorg/sglang:latest
    container_name: sglang-backend
    restart: unless-stopped
    ipc: host
    ports:
      - "8000:8000"
    environment:
      - HF_TOKEN
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      sglang serve
      --model-path Qwen/Qwen3.6-35B-A3B
      --served-model-name "Qwen/Qwen3.6-35B-A3B"
      --host 0.0.0.0
      --port 8000
      --context-length 131072
      --kv-cache-dtype fp8_e4m3
      --mem-fraction-static 0.85
      --chat-template chatml
      --chunked-prefill-size 4096
      --max-prefill-tokens 4096
      --cuda-graph-max-bs-decode 4
      --attention-backend flashinfer

  litellm-gateway:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm-gateway
    restart: unless-stopped
    ports:
      - "8001:4000"
    environment:
      - OPENAI_API_KEY=placeholder
    volumes:
      - ./litellm.yaml:/app/config.yaml
    depends_on:
      - sglang-backend
    command: --config /app/config.yaml

litellm.yaml

model_list:
  - model_name: Qwen/Qwen3.6-35B-A3B
    litellm_params:
      model: openai/Qwen/Qwen3.6-35B-A3B
      api_base: http://{{DGX-SPARK-IP}}:8000/v1
      max_tokens: 4096
      timeout: 600
      model_info:
        supports_reasoning: true
        preserve_thinking: true

Start the containers with:

export HF_TOKEN={{YOUR-HUGGINGFACE-TOKEN}}
docker compose up -d