README-omlx.md

oMLX local configuration

Captured 2026-09-02. This is a redacted reconstruction reference: it records explicit local settings, storage paths, installed-model metadata, and profiles. It intentionally omits credentials and request statistics.

System settings

Server and lifecycle

Setting	Value
Bind address / port	127.0.0.1:8000
Log level	info
CORS origins	*
Server aliases	localhost, 127.0.0.1, two private-LAN addresses, 10.42.0.0, 10.43.0.0
SSE keepalive	chunk
Automatic start	auto_start_on_launch: true
Decode mode	burst_decode_mode: aggressive
Preserve mid-system cache	true
Distributed inference	false
Audio upload limit	100MB

Model discovery, scheduling, and memory

Setting	Value
Model directories	~/.omlx/models, then ~/.lmstudio/models
Primary model directory	~/.omlx/models
Model fallback / helper visibility	false / false
Concurrent requests / embedding batch	12 / 32
Chunked prefill / prefill priority / decode fairness	false / context / false
Memory guard tier / ceiling	aggressive / 100GB
Soft / hard threshold	.85 / .95
Prefill safe-zone / minimum chunk	.8 / 32 tokens

Cache and global sampling

Setting	Value
SSD cache directory / limit	~/.omlx/cache / 185GB
Hot cache limit	32GB
GDN SSD split / snapshot storage / sidecar precision	true / auto / fp32
GDN pending maximum / initial blocks	512MB / 256
Hot-only / write-through / ANE compile cache	false / false / false
Maximum context / output	131072 / 32768
Temperature / top-p / top-k / repetition penalty	1 / .95 / 0 / 1
Context-window policy	null

Remaining system settings

• logging: retention 7 days; explicit directory null.
• mcp: tools exposed; configuration path null.
• huggingface: cache enabled; endpoint empty. modelscope.endpoint empty.
• network: HTTP/HTTPS/no-proxy and CA-bundle fields empty.
• claude_code: mode cloud; model fields null.
• integrations: Codex, OpenCode, OpenClaw, Hermes, Pi, and Copilot models null; OpenClaw tools profile coding. MarkItDown enabled but not exposed as a model, 25 MB/file, five files/request, engine markitdown. DDGS search with three results, snippet mode, 20,000 characters.
• ui.language: en; idle timeout null.
• Authentication material is configured locally and omitted here. API-key verification is skipped. Brave search key is empty.

GUI/app-support preferences

Setting	Value
Base path	~/.omlx
Port	8000
Model directory	~/.lmstudio/models
Launch at login	false
Start server on launch	false

Shared model settings

These values are stored for every configured model unless a model/profile table explicitly gives a different value.

Group	Settings
General	force_sampling: false; model_type_override: llm; guided_grammar_enabled: false; trust_remote_code: false
Acceleration	turboquant_kv_enabled: false; turboquant_skip_last: true; specprefill_enabled: false; mtp_enabled: false; vlm_mtp_enabled: false
ANE prefill	disabled; sequence 2048; fraction .53; max layers 64; dual ANE and GDN true; GDN fraction .5; GDN max layers 48
CPU prefill	disabled; fraction .135; eight threads; shared resource true
DFlash	disabled; RAM cache enabled with four entries and 8 GiB; SSD cache disabled with 20 GiB; sink size 0

Qwen3.8-27B-8bit

Package: ~/.omlx/models/mlx-community/Qwen3.8-27B-8bit. Metadata: Qwen 3.5 conditional-generation architecture; affine quantization, 8 bits, group size 64. A bundled chat template is present.

Model settings

Setting	Value
Context / output	150000 / 4096
Sampling	T .5, top-p .95, top-k 20, min-p 0, repetition 1, presence 0
Thinking / budget / parser	true / enabled 1024 / qwen_3_5
Template argument	reasoning_effort: low
TurboQuant KV bits	4
DFlash helper	~/.omlx/models/jfan/Qwen3.8-27B-heretic-dflash
Pin / primary selection	false / false

Profiles

ID	Display / API name	Explicit profile fields
p-mstx8kiz-y23u	Non thinking / non-thinking	Context 150000; T .2; top-p .9; repetition 1.05; thinking false; TurboQuant 4; ANE false; DFlash false; RAM cache true/four entries; SSD false; verification adaptive; speculative/MTP/VLM-MTP false.
p-msvvugnp-a95p	Thinking / thinking	Context/output 150000/4096; T .5; top-p .95; top-k 20; min-p 0; repetition 1; presence 0; thinking/budget true/1024; parser qwen_3_5; reasoning_effort: low; TurboQuant 8; DFlash false; draft quant true at weight 4, activation 16, group 64; RAM cache true/four entries; SSD true; verification adaptive.

Both profiles set expose_as_model: false.

Qwen3.8-27B-MLX-6bit

Package: ~/.omlx/models/lmstudio-community/Qwen3.8-27B-MLX-6bit. Metadata: Qwen 3.5 conditional-generation architecture; affine quantization, 6 bits, group size 64. A bundled chat template is present.

Model settings

Setting	Value
Context / output	150000 / 4096
Sampling	T .5, top-p .95, top-k 20, min-p 0, repetition 1, presence 0
Thinking / budget / parser	true / enabled 1024 / qwen_3_5
Template argument / forced argument	reasoning_effort: low / reasoning_effort
TurboQuant KV bits	4
Speculative helper / keep percentage	~/.omlx/models/mlx-community/Qwen3.5-0.8B-8bit / .2
DFlash verification	adaptive
Pin / primary selection / active profile	true / true / p-msunjyp1-lbxv

Profiles

ID	Display / API name	Explicit profile fields
p-msunjyp1-lbxv	Qwen 3.8 27b Q6 Thinking / qwen-3-8-27b-q6-thinking	Context/output 150000/4096; T .5; top-p .95; top-k 20; min-p 0; repetition 1; presence 0; thinking/budget true/1024; parser qwen_3_5; reasoning_effort: low forced; TurboQuant 4; ANE/CPU/speculative/DFlash/MTP/VLM-MTP false; RAM cache four entries/8 GiB; SSD false/20 GiB; verification adaptive.
p-msunkkss-nvaq	Qwen 3.8 27b Q6 Non Thinking / qwen-3-8-27b-q6-non-thinking	Context/output 150000/4096; T .5; top-p .95; top-k 20; min-p 0; repetition 1; presence 0; thinking false; TurboQuant 4; ANE/speculative/DFlash/MTP/VLM-MTP false; RAM cache four entries; SSD false; verification adaptive.

Both profiles set expose_as_model: false.

Qwen3.8-27B-MLX-4bit

Package: ~/.omlx/models/lmstudio-community/Qwen3.8-27B-MLX-4bit. Metadata: Qwen 3.5 conditional-generation architecture; affine quantization, 4 bits, group size 64. A bundled chat template is present.

Model settings

Setting	Value
Context / output	150000 / 4096
Sampling	T .5, top-p .95, top-k 20, min-p 0, repetition 1, presence 0
Thinking field / budget / parser	absent / enabled 1024 / qwen_3_5
Template argument / forced argument	reasoning_effort: low / reasoning_effort
TurboQuant KV bits	4
Pin / primary selection	false / false

Profile

ID	Display / API name	Explicit profile fields
p-mtbqthnr-1b6s	extract / extract	Context/output 150000/4096; T .5; top-p .95; top-k 20; min-p 0; repetition 1; presence 0; budget true/1024; parser qwen_3_5; reasoning_effort: low forced; TurboQuant 4; ANE/speculative/DFlash/MTP/VLM-MTP false; RAM cache four entries; SSD false; verification adaptive.

expose_as_model: false.

Qwen3.6-35B-A3B-MLX-mixed-4bit

Package: ~/.omlx/models/EigenLabs/Qwen3.6-35B-A3B-MLX-mixed-4bit. Metadata: Qwen 3.5 MoE conditional-generation architecture. Base affine quantization is 4 bits/group 64; selected embedding, head, attention, and expert tensors use 5-6 bits.

Model settings

Setting	Value
Context / output	64000 / 4096
Sampling	T .2, top-p .9, top-k 20, min-p 0, repetition 1, presence 0
Thinking / budget	false / disabled
TurboQuant KV bits	4
Pin / primary selection	true / false

No profiles are stored for this model.

oMLX & Opencode Findings.md

Here is everything the knowledge base holds on your oMLX + Qwen setup, as of 2026-09-03.

The rig

• M5 Max Mac, 128GB unified memory. oMLX (omlx.ai) serving an OpenAI/Anthropic-compatible API on localhost:8000. Client is OpenCode, used for code and log/text analysis only, never images.
• Orchestrator / daily driver: lmstudio-community/Qwen3.8-27B-MLX-6bit, ~20.6GB loaded. Qwen3.8-27B is a VLM with hybrid attention (gated DeltaNet linear + gated full attention, arch tag qwen35). The 6-bit build engages oMLX's custom Metal kernel: omlx.custom_kernels.qwen35_prefill.fast - Qwen qmm NAX dispatch enabled.
• Worker: EigenLabs/Qwen3.6-35B-A3B-MLX-mixed-4bit, 35B total / ~3B active, ~20GB. Mixed quant protects shared experts, attention/DeltaNet projections, embeddings and output at 5-6bit/BF16, which is why tool-call JSON stays reliable.
• Sampling (settled 2026-08-19): temperature 0.5, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repeat_penalty 1.0. enable_thinking true with chat_template_kwargs {"reasoning_effort": "low"}. preserve_thinking left true, since false mutates the prefix and kills the cache. Context 150k in oMLX. max_tokens 4096, set in OpenCode, not oMLX, because OpenCode overrides the oMLX per-model value.

Measured performance

• 27B 6-bit: 15.5-17.8 tok/s decode, ~487-600 tok/s prefill, sagging toward ~400 at long context.
• 35B-A3B 4-bit worker (measured on your machine 2026-09-03): 82-123 tok/s decode, ~1,450 tok/s prefill, and perfect prefix reuse on consecutive turns (45,056 of 45,056, then 49,152 of 49,152). That confirms the ~7.5x decode advantage previously predicted from MoE sparsity analysis.
• Cold start at ~131k context is 5+ minutes. Not a regression, just 131,072 tokens at ~487 tok/s plus quadratic sag.
• oMLX logs report tok/s as decode-only while the accompanying wall-clock includes prefill.

Model verdicts

• Rejected avlp12/Qwen3.8-27B-Alis-MLX-8bit (27.9GB, 333 vision tensors + MTP head): 14.6-15.1 tok/s at 62-66k vs the 6-bit's 17.0-17.8 at 78k, and the MTP head never fired. MLX 6-bit beats 8-bit on this architecture; 8-bit vs BF16 perplexity differs by only ~0.14%.
• Ruled out DeepSeek-V4-Flash: 103GB resident, cannot coexist with the orchestrator, only 33 tok/s.
• Qwen3.8-Flash-Next-oQ4e: third-party benchmark only, unknown hardware. Wins prefill 1.65-2.15x with the gap widening, but decode crosses under the 27B past ~16k and it costs ~28GB more peak. The sweep stops at 32k, far short of your working context.
• Quality tradeoff accepted: SWE-bench Verified 73.4 (35B-A3B) vs 77.2 (27B dense). Roughly 4 points for 7.5x speed, which is why the orchestrator stays dense.

MTP and quantization

• MTP gates: real mtp.* weights in the checkpoint (metadata alone is silently skipped), process-wide attach flag on, supported arch, batch size 1. Verify by log line, never the model card.
• Depth 1 only. Depth 2 and 3 give nothing, because single-stream decode is ~95% backbone-compute-bound. Same reason Q8 to Q4 only buys ~1.4x.
• MTP degrades with batching, so parallel subagent fan-out trades it away. On the A3B it is largely moot anyway (+12% vs +75% on the dense 27B).
• 8-bit KV cache: no help. Prefill is compute-bound, K is quantization-sensitive, and it risks dropping off the custom Metal kernel. MTP is decode-only and cannot touch initial load.

Prefix cache (the big lever)

• Blocks are 256 tokens, content-addressed by SHA-256 chain hash, stored as .safetensors under ~/.omlx/cache. Index is rebuilt on startup, so restart survival is intended. Sessions sharing a system prompt and tool schemas share blocks automatically.
• oMLX auto-enlarges block size 256 to 2048 for ArraysCache hybrid models (both your models), overriding any manual smaller value. The old advice to try 512 does not apply.
• boundary_snapshot_unavailable skips are benign on hybrid models. With block_size 2048 the cache advances in a 2048-token staircase; the worst case loss is the trailing partial block. Do not chase those log lines.
• Gotchas: the hot RAM tier is write-back and only flushes on clean shutdown, so a kill -9 loses everything (and on 128GB, memory pressure may never trigger a flush). One differing byte in the first 256 tokens invalidates all downstream blocks, so static content goes at the head and date/cwd/git-branch/session-id at the tail. hot_cache_only disables SSD persistence entirely. Enabling TurboQuant invalidates the entire on-disk cache.
• Diagnostic fork via /admin stats: saves ~0 means write-path failure (unclean shutdown); saves high but loads ~0 means restore rejection.
• Root cause of your slow loads (2026-08-31): prompts diverge from cache at ~token 4332, inside the OpenCode tool-schema block. Prime suspect is nondeterministic tool ordering from MCP servers, then date/cwd/git in the system prompt, then a file tree. Fix is to diff two tokenized prompts around 4200-4500 and move all session-varying content to the tail. Capacity is not the constraint (evictions are 0), so raising hot_cache_max_size to 40-48GB is second priority.
• compaction.prune: true in OpenCode is a prefix-cache killer and should be off. It rewrites the middle of the conversation and forces a full re-prefill, roughly 3 minutes at 100k. You had it on as of 2026-09-03, alongside preserve_recent_tokens: 8192 and reserved: 10000. General rule: compacting less often beats keeping context small.

Where settings live

• Per-model settings: /admin dashboard, persisted to model_settings.json in the oMLX base path. Not ~/.omlx/settings.json, which is server/admin config.
• No dedicated reasoning-level control. For Qwen3.8, set reasoning_effort (xhigh/medium/low) inside chat_template_kwargs, and do not also set enable_thinking in both places.
• Profiles are named setting bundles exposed as separate model IDs (e.g. qwen3.8-27b:think-low). They share the base model's memory and apply per request without a reload, so they are the way to A/B without paying a cold prefill.
• Config precedence: defaults, then ~/.omlx/settings.json, then OMLX_* env, then CLI. Most cache settings need a restart.

OpenCode side

• Agent architecture at github.com/Gipetto/opencode-config: orchestrator and plan on the 27B 6-bit (edit denied), explore/implement/verify on the A3B, review back on the 27B. subagent_depth 1, small_model and compaction model on the A3B. Verification is split so the A3B does mechanical checks (~7s) and the 27B only judges the green path (~25s).
• Context overflow was solved 2026-08-28: base system prompt was 64,193 tokens, of which ~55-58k was tool JSON from ~146 tools across 7 MCP servers. Fix was top-level tools with server-prefixed globs set to false, re-enabled per agent. permission: deny does not reclaim MCP context. After the fix: orchestrator ~60-70k, explore ~18-22k, implement/verify ~14-16k.
• Corollary that reframes everything: your 62-80k "typical working prompts" were mostly tool schemas, with only 0-15k of actual content. Trimming tools cuts prefill linearly.
• permission.task takes subagent-name globs, not paths. {"*": "deny"} blocks built-ins too.
• Tool-call churn was resolved 2026-08-19. Cause not definitively attributed among repeat_penalty 1.1 to 1.0, max_tokens 8192 to 4096, reverting to 6-bit, and adding the kindex policy plugin.
• Known contradiction found 2026-09-03: the shared AGENTS.md was written for a single do-everything agent and fights the role split, both on "prefer editing files directly" and on whether the orchestrator may consult kindex.

Open items

• Whether oMLX holds two models resident concurrently or swaps on demand. Still listed as an unverified blocker for automated delegation, though the 2026-09-03 A3B measurements suggest the routing is running in practice.
• Whether a Qwen3.8-generation A3B exists with MLX 4-bit quants. All verified A3B builds are 3.6 series, and mixed generations mean different tokenizers, ruling out shared-drafter tricks.
• Whether the model reliably consults kindex before grepping.
• Compaction settings should be retuned now that the orchestrator base prompt has dropped.