llama.cpp on the SpacemiT K3 Pico-ITX (RISC-V) — Reproduction Guide

Tested 2026-07-11 on: SpacemiT K3 pico-ITX, 32GB RAM SKU, Bianbu 4.0.1 (Resolute Raccoon), kernel 6.18.3, GCC 15.2. 16 cores (8× perf @2.2GHz, 8× @1.8GHz). End result: Qwen3-30B-A3B MoE served at ~38 tok/s prompt processing / ~8-11 tok/s generation, CPU-only, via the SpacemiT IME (integrated matrix engine) instructions.

TL;DR

sudo apt-get install -y cmake build-essential libcurl4-openssl-dev
git clone --depth 1 --branch v0.1.6 https://github.com/spacemit-com/llama.cpp.git spacemit-llama-src
cd spacemit-llama-src
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CPU_RISCV64_SPACEMIT=ON -DGGML_RV_ZBA=ON
cmake --build build --config Release -j8
build/bin/llama-server -m model.gguf -fa on -c 40960 -fit off \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.0 \
  --host 0.0.0.0 --port 8080 -t 8 -tb 8

Do NOT: use the Bianbu apt package, use --cache-type-k/v q8_0, run hybrid-SSM models (Qwen3.6 family), or run two model instances at once. Details below.

1. Why build from source

Bianbu ships a stale vendor package (llama.cpp-tools-spacemit, build 17ce6aa) that predates newer model architectures — Qwen3.6-family GGUFs fail with missing tensor 'blk.N.ssm_conv1d.weight' even when the file is perfectly good.
SpacemiT's prebuilt release tarballs (github.com/spacemit-com/llama.cpp/releases) crash on this OS (std::unexpected / terminate during load) — apparent libstdc++ ABI mismatch against Bianbu's GCC 15.2 userland. Build on the box; it's ~15 min on 8 cores.

2. Build

sudo apt-get update
sudo apt-get install -y cmake build-essential libcurl4-openssl-dev
git clone --depth 1 --branch v0.1.6 https://github.com/spacemit-com/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release \
      -DGGML_CPU_RISCV64_SPACEMIT=ON \
      -DGGML_RV_ZBA=ON
cmake --build build --config Release -j8

-DGGML_CPU_RISCV64_SPACEMIT=ON enables the IME1/IME2 kernels (vmadot int4/int8 matrix ops). Verify configure prints RISCV64_SPACEMIT_IME_SPEC: ...IME1;...IME2 and the march string contains xsmtvdotii.
-DGGML_RV_ZBA=ON is required even though it's not a declared option in v0.1.6's CMake — ime.cpp hard-errors (#error riscv zba extension not enabled) without it under GCC 15. The K1/K3 CPU has zba; the build system just forgets to pass it.
Build via cmake --build ... 2>&1 | tail hides errors — tail's exit code masks the build's. Log to a file and grep for error: if something looks off.

Expected startup banner (harmless warnings included):

CPU_RISCV64_SPACEMIT: ... use_ime2: 1, mem_backend: HPAGE ...
CPU_RISCV64_SPACEMIT: alloc_chunk: open(/dev/tcm_sync_mem) failed, errno=2   <- harmless
CPU_RISCV64_SPACEMIT: failed to allocate init_barrier from shared mem, falling back to heap

3. Choosing a model — this matters more than anything else

The IME acceleration hooks only static weight matmuls (MUL_MAT / MUL_MAT_ID, via load-time repack into IME tile layout). Everything else runs generic RVV or scalar code. Consequences:

Model type	Verdict	Why
Plain-transformer MoE (e.g. Qwen3-30B-A3B)	✅ best	Low active params + full IME coverage. ~38 pp / ~8-11 tg tok/s at Q4_K_XL
Plain dense (Qwen2.5, Phi, etc.)	✅ fine	Full IME coverage; speed scales inversely with size
Hybrid GDN/SSM (Qwen3.6-27B, Qwen3.6-35B-A3B)	❌ avoid	`ssm_conv`/`ssm_scan` have ZERO IME/RVV coverage → scalar fallback. Measured: 1-3.5 tok/s tg. Also forces full-prompt reprocessing on cold cache (llama.cpp hybrid-memory checkpoint limitation)

Quant format: Q4_0/Q4_K/Q5_K/Q6_K/Q8_0 all have registered IME tensor_traits (repacked to the engine's q8 tiles at load). Q4_K_XL (Unsloth dynamic) worked well.

Sizing: 32GB RAM, no swap. Weights (mmap) + KV cache + prompt-cache must fit or the board OOM-deadlocks. A 17.7GB model + f16 KV @ 40960 ctx (~4GB) is comfortable. Never start a second instance while one is loaded — instant OOM hang.

4. Critical runtime flags

-fit off — the fork's automatic memory-fit estimator wildly underestimates (claimed 553MiB for a model needing ~20GB) and hangs/breaks loading. Always disable.
NO --cache-type-k q8_0 --cache-type-v q8_0 — THE big trap. There is no vectorized attention-over-quantized-KV kernel in this backend; it falls to scalar code whose cost grows with context. Invisible on short prompts (~18% hit at 19 tokens), catastrophic on real ones: at a 2k-token prompt, pp collapses 39.8 → 2.7 tok/s (14.5x) and tg 8.25 → 1.04 tok/s. f16 KV (the default) is fast. This flag pair is commonly carried over from GPU configs where quantized-KV kernels are first-class — drop it here.
-fa on — fine (with f16 KV). -ub 256 — fine. Large -c — fine.
-t 8 -tb 8 — matches the 8-core perf cluster the SpacemiT backend pins to (mask ff00).
Sampling for Qwen3-30B-A3B (thinking model, loops badly on greedy/default sampling): --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.0. If a client sends no sampling params, these server defaults rule.
--cache-ram N is in MiB — a leftover --cache-ram 2 from a GPU config caps the prompt cache at 2MiB and silently defeats checkpoint reuse. Use e.g. 4096.

5. Binary quirks

llama-cli crashes at exit without a TTY (abort in console-history destructor during __cxa_finalize) — cosmetic but alarming. Use llama-completion for scripted runs (-no-cnv was removed from llama-cli) and llama-server for serving.
llama-bench aborts when running multiple tests in one process (second context init trips the IME barrier state). Run one test per invocation.
First load of a big model is disk-bound: ~85MB/s → 1.5-4 min for 17-25GB. Client timeouts (e.g. a 30s HTTP timeout in an agent tool) will fire while the server is still legitimately prefilling; raise them.

6. Platform gotchas (Bianbu userland)

The CPU has vector-crypto (zvkned/zvknha/zvksh) but half the userland wasn't built to use it:

wget pegs a core on HTTPS — it links GnuTLS→nettle, and Bianbu's nettle was compiled without zvkned (scalar T-table AES). Use curl (links OpenSSL, which runtime-detects the extensions; ~1.9GB/s AES-GCM).
sha256sum is scalar coreutils code — minutes for a 20GB file. Use openssl dgst -sha256 instead (~1.5GB/s, 17s for 17.7GB).
Passwordless sudo: Bianbu ships /etc/sudoers.d/10-installer containing %sudo ALL=(ALL) ALL, which overrides a NOPASSWD rule via last-match-wins. Remove it.
SSH sessions drop (exit 255) under heavy load/saturated link — retry before assuming the remote command failed (it often succeeded).

7. systemd service (final working config)

/etc/systemd/system/llama-server.service:

[Unit]
Description=llama.cpp server (Qwen3-30B-A3B Q4_K_XL, SpacemiT IME)
After=network.target

[Service]
Type=simple
User=aaaaaa
ExecStart=/home/aaaaaa/spacemit-llama-src/build/bin/llama-server \
        -m /home/aaaaaa/Qwen3-30B-A3B-UD-Q4_K_XL.gguf \
        -fa on -c 40960 -fit off \
        --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
        --presence-penalty 1.0 \
        --parallel 1 \
        --ubatch-size 256 \
        --no-context-shift \
        --ctx-checkpoints 8 \
        --cache-ram 4096 \
        --jinja -lv 4 \
        --host 0.0.0.0 --port 8080 -t 8 -tb 8

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp llama-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
curl -s http://localhost:8080/health          # {"status":"ok"} once loaded (~2 min)
curl -s http://localhost:8080/props | jq .default_generation_settings.params  # verify sampling

8. Verifying performance

# One test per invocation (see §5). Expect ~65 tok/s pp2048 on a 3B, ~40 on 30B-A3B.
build/bin/llama-bench -m model.gguf -p 2048 -n 0 -r 1 -t 8

If long-prompt pp is an order of magnitude slower than llama-bench predicts, you've re-introduced one of the §4 traps (quantized KV being the usual suspect). Bench at realistic prompt lengths — every trap here is invisible at short context.

9. Known-unfixed / future work

SSM ops (ssm_conv/ssm_scan) unaccelerated → hybrid models (Qwen3.6+) stay slow until someone writes RVV/IME kernels for them.
Attention over quantized KV unaccelerated (same class of gap).
/dev/tcm_sync_mem missing → IME uses heap fallback instead of TCM; possibly a driver/permissions item with some performance left on the table.
-fit estimator broken for these backends.
Speculative decoding (--spec-type draft-mtp) untested on this fork.

mischief/spacemit-llama.md

Select an option

No results found