Skip to content

Instantly share code, notes, and snippets.

@G36maid

G36maid/dspark-deployment-report.md

Created August 17, 2026 09:26
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save G36maid/e071fbbeee41efe0a20f72ea854c6c7b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save G36maid/e071fbbeee41efe0a20f72ea854c6c7b to your computer and use it in GitHub Desktop.
Download ZIP
DeepSeek-V4-Flash-0731 on 2x NVIDIA DGX Spark (TP=2) — deployment report
Raw
dspark-deployment-report.md

DeepSeek-V4-Flash-0731 Deployment Report — 2× NVIDIA DGX Spark (TP=2)

Date: 2026-08-17 Recipe: MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark Status: 🟢 Production-ready — API live, smoke test 6/6 passed, inference verified

1. Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                       Tailscale (remote mgmt)                       │
│   Laptop (archfw13) ── 100.85.81.82                                 │
│        │ ProxyJump                                                  │
│   spark-037c ── 100.102.84.36  (same tailnet as laptop)             │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────── 200Gb RoCE Interconnect ───────────────────────────────┐
│                                                                                       │
│  MASTER (rank 0)                     WORKER (rank 1)                                  │
│  spark-037c                          spark-3343                                        │
│  ├─ enP7s7         192.168.1.191    ├─ enP7s7         192.168.1.186                  │
│  ├─ enp1s0f0np0    192.168.100.11 ◄─┼─► enp1s0f0np0    192.168.100.10  (NCCL fabric) │
│  └─ enP2p1s0f0np0  192.168.101.11 ◄─┼─► enP2p1s0f0np0  192.168.101.10                │
│                                                                                       │
└───────────────────────────────────────────────────────────────────────────────────────┘
Role Host NCCL HCA NCCL if GID index VLLM_HOST_IP
Master (rank 0) spark-037c rocep1s0f0 enp1s0f0np0 3 192.168.100.11
Worker (rank 1) spark-3343 rocep1s0f0 enp1s0f0np0 3 192.168.100.10

Fabric: 2 × 200 Gb/s RoCEv2 direct connections between nodes (loopback point-to-point, no switch).

2. Node Prerequisites

Check spark-037c spark-3343
OS / Kernel DGX Spark 7.5.0 / 6.17.0-1029-nvidia identical
GPU NVIDIA GB10 (SM121, 128 GB unified) identical
Driver / CUDA 580.173.02 / 13.0 identical
Docker 29.2.1 identical
Docker Compose v5.0.2 identical
NVIDIA Container Toolkit 1.19.1 (GPU passthrough verified) identical
Disk (/) 3.7 TB NVMe, 3.4 TB free identical
Memory 121 GiB identical

Note: Docker access required adding user to docker group via sudo usermod -aG docker g36maid on both nodes.

3. Model

Property Value
HF repo deepseek-ai/DeepSeek-V4-Flash-0731
Snapshot (resolved) 7872f01b1d1fe23eabc4c98b48bffcef5a386062 (latest main)
Recipe-pinned revision 9e165c30... — no longer exists on HF hub (repo README out of date)
Size 167 GB total / 155.43 GiB checkpoint / 158 GiB on disk
Shards 48× safetensors (3.57–3.69 GB each) + index + config + encoding
Quantization FP8 weights, expert_dtype fp4, E8M0 DeepGEMM, UE8M0 enabled
Encoding encoding/encoding_dsv4.py (auto-installed into vLLM by compose)

Download strategy (bandwidth-optimized):

  1. prepare-dspark-model-cache.sh on master (037c) — 1 GbE uplink, ~16 min after resume
  2. Manually interrupted once at 117G (user request) → resumed seamlessly (HF hub resumable)
  3. rsync over 200 Gb interconnect to worker: 158G @ ~330–370 MB/s in ~7 min
  4. Offline verify on BOTH nodes: snapshot_download(local_files_only=True) → 48/48 shards present

⚠️ One rsync warning: trees/*.json (root-owned 600 metadata, generated inside container) failed to copy with Permission denied. Model weights unaffected; vLLM loads from its local HF cache without this file.

4. Runtime Profile (.env.dspark)

DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1
DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731
SERVED_MODEL_NAME=deepseek-v4-flash-0731

MAX_MODEL_LEN=1048576          # 1M tokens
MAX_NUM_SEQS=6
MAX_NUM_BATCHED_TOKENS=8192
GPU_MEMORY_UTILIZATION=0.80
MTP_NUM_TOKENS=5               # dspark_block_size min = 5
DEFAULT_THINKING=low           # off/low/high/max

KV_CACHE_DTYPE=nvfp4_ds_mla     # experimental NVFP4 deep-sparse MLA path
BLOCK_SIZE=256
MOE_BACKEND=flashinfer_b12x
VLLM_USE_FLASHINFER_SAMPLER=1
VLLM_USE_BREAKABLE_CUDAGRAPH=0  # regular CUDA graphs (faster)
HF_HUB_OFFLINE=1                # both nodes have full local hub cache

Full rendered vLLM command (validated via validate-dspark-config.sh):

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --served-model-name deepseek-v4-flash-0731 \
  --host 0.0.0.0 --port 8888 --trust-remote-code \
  --tensor-parallel-size 2 --pipeline-parallel-size 1 \
  --kv-cache-dtype nvfp4_ds_mla --block-size 256 \
  --max-model-len 1048576 --max-num-seqs 6 \
  --max-num-batched-tokens 8192 --max-cudagraph-capture-size 36 \
  --gpu-memory-utilization 0.80 --enable-prefix-caching \
  --enable-prompt-tokens-details --async-scheduling \
  --enable-chunked-prefill \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 --distributed-executor-backend mp \
  --moe-backend flashinfer_b12x \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":" thinking","reasoning_end_str":" response"}' \
  --default-chat-template-kwargs '{"thinking":true,"reasoning_effort":"low"}' \
  --generation-config vllm --enable-flashinfer-autotune \
  --nnodes 2 --node-rank 0 --master-addr 192.168.100.11 --master-port 25000

5. Boot Timeline (observed)

Time Event
T+0s Both containers created (worker first, then head — worker-first start script)
T+0s NCCL distributed init: world_size=2, backend=nccl, tcp://192.168.100.11:25000
T+0s DeepGEMM E8M0 enabled, B12X_MXFP4 MoE backend, FP8 indexer cache, Eagle3 aux layers (41,42,43)
T+1m Loading safetensors checkpoint shards — 48 shards, ~155 GiB
T+2m38s 48/48 shards loaded (159.04 s)
T+4m TileLang kernel JIT compilation (mhc_pre_big_fuse, hc_head_fuse, …)
T+5m FlashInfer SM120 sparse MLA DSv4 autotune + warmup
T+5m Available KV cache memory: 12.66 GiB per rank
T+8m API live — /v1/models HTTP 200, max_model_len=1048576

6. Verification Results

6.1 API endpoint

GET http://<master>:8888/v1/models
→ {"data":[{"id":"deepseek-v4-flash-0731", "max_model_len":1048576, ...}]}

6.2 Smoke test — 6-way concurrency

Running 6-way smoke test against http://127.0.0.1:8888/v1/chat/completions
Smoke test passed: 6/6 requests succeeded.

6.3 Direct inference checks

Minimal (temperature 0):

usage: {prompt_tokens: 14, completion_tokens: 42}
content: OK
reasoning: None          ← correctly separated

Reasoning task (17 * 23 - 5):

usage: {prompt_tokens: 23, completion_tokens: 68}
reasoning: (emitted first, properly separated)
content: 17 × 23 = 391
         391 − 5 = 386
         **Final answer: 386**

✅ Arithmetic correct, reasoning/answer streams split correctly, no output garbling.

7. Networking & Connectivity Decisions

  1. Tailscale topology:

    • spark-037c, laptop → same tailnet (miku65434@)
    • spark-3343 → different tailnet (justinwu6900942@) → shows offline on our side, but L3-reachable
    • Workaround: laptop reaches 3343 via ProxyJump spark-037c; master→worker SSH runs over the 200 Gb interconnect (192.168.100.x)

  2. mDNS (.local) is unreliable across nodes — .env.dspark uses explicit interconnect IPs for MASTER_ADDR / VLLM_HOST_IP / WORKER_VLLM_HOST_IP, with WORKER_HOST=spark-3343.local resolvable from master to 192.168.100.10.

  3. RoCEv2 GID resolution: both nodes only expose IPv4-mapped RoCEv2 GID at index 3 (rocep1s0f0); the start script's sysfs auto-resolve found head=3, worker=3 correctly.

  4. SSH trust chain:

    • laptop → 037c: ~/.ssh/spark-037c_ed25519
    • laptop → 3343: ~/.ssh/spark-3343_ed25519 via ProxyJump
    • 037c → 3343 (fabric): 037c's id_ed25519 authorized on 3343

8. Files & Commands (master: /home/g36maid/Code/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/)

Command Purpose
./start-deepseek-v4-flash-dspark.sh Worker-first launch + wait + minimal chat probe (FINAL_EXIT=0)
./stop-deepseek-v4-flash-dspark.sh Stop both ranks
./status-deepseek-v4-flash-dspark.sh Show status
./smoke-deepseek-v4-flash-dspark.sh N-way concurrency smoke test
./validate-dspark-config.sh Print resolved profile + rendered vLLM command
./prepare-dspark-model-cache.sh Download + offline-verify HF cache (master or worker)

Quick curl:

curl http://100.102.84.36:8888/v1/models
curl http://100.102.84.36:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-v4-flash-0731","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'

9. Notes / Caveats

  • Recipe pinned revision is stale — the documented 9e165c30... fails 404 on HF; we deployed latest main (7872f01b). If exact reproducibility matters, re-pin the README.
  • trees/*.json metadata not synced (permission artifact) — cosmetic only.
  • VLLM_BUILD_ unknown-env warnings* are image build metadata, harmless.
  • KV cache 12.66 GiB/rank at utilization 0.80 (recipe's own benchmark used 18.08 GiB at 0.835 with seqs=4 — per-rank KV pool can be enlarged by adjusting GPU_MEMORY_UTILIZATION / MAX_NUM_SEQS).
  • max_num_scheduled_tokens warning about 8168 vs batched 8192 is expected under speculative decoding; negligible.
  • Model is text-only per recipe — multimodal requests need a sidecar.

10. Summary

Metric Result
Deployment time (prereq → live API) ~40 min (incl. model transfer)
Model readiness 48/48 shards verified both nodes
API 0.0.0.0:8888 on spark-037c
Context window 1,048,576 tokens
Concurrency 6 sequences / 6-way smoke pass
Speculative decode DSpark γ=5 probabilistic
KV cache NVFP4 deep-sparse MLA (12.66 GiB/rank)
Inference quality Correct output + separate reasoning stream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment