Nemotron 3.5 ASR Streaming 0.6B — On-Device Feasibility (Android/iOS/Web)

Model: https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b

Model facts

Architecture: Cache-Aware FastConformer encoder (24 layers) + RNNT decoder, 600M params — streaming with configurable 80ms–1120ms chunk latency
Formats: safetensors, PyTorch, GGUF (q8_0) via NeMo-Speech.cpp (C++ runtime)
Official hardware/OS support: NVIDIA GPU archs only (Ampere/Blackwell/Hopper/Jetson/Lovelace/Turing/Volta), Linux / Linux4Tegra. No ONNX, Core ML, or TFLite export path officially provided. No mobile or web runtime published.
40 language-locales, license OpenMDW-1.1
No roadmap/future plans stated on the model card

Feasibility by platform

Web — best-documented, works today

khawjaahmad/nemotron-asr-webgpu: this exact model, INT4 ONNX, running fully in-browser via onnxruntime-web. Encoder on WebGPU (~690MB), decoder/joiner on WASM (hybrid split to avoid per-token GPU sync overhead). Total download ~750MB, cached via Cache API.
WebGPU is mandatory on mobile — the encoder cannot run on WASM/CPU alone on phones.
Browser support: Chrome/Edge desktop, Chrome Android, Safari 18+ iOS, Firefox behind a flag. Requires HTTPS/secure context.
Backing weights: onnx-community/nemotron-3.5-asr-streaming-0.6b-onnx-int4
Predecessor (Parakeet.js, ysdede): same hybrid WebGPU-encoder/WASM-decoder approach, output "nearly identical" to native NeMo/ONNX, some precision loss noted, works even on Intel integrated graphics per a community tester.

iOS — shipping products exist, no published methodology

Hedy AI added Nemotron as an on-device engine (iPhone 12+, iPad, Mac), replacing Parakeet — "a clear step up in local accuracy," real-time diarization on iPhone/Mac. No conversion path, latency, RTF, memory, or battery numbers disclosed.
"On-Device AI: Private Workspace" (App Store) added Nemotron Speech + Parakeet + Qwen3-ASR support — no technical detail public.
No CoreML conversion of the 3.5 checkpoint itself exists yet — only the older Parakeet does (FluidInference/parakeet-tdt-0.6b-v2-coreml: ~110x RTF on M4 Pro, ~800MB peak memory, macOS 14+/iOS 17+). An open HF discussion (#31) explicitly requests Core ML docs for Nemotron 3.5 — unanswered.
HN thread on a C++/Metal port (parakeet.cpp): maintainer said iOS support is "theoretically yes, untested."

Android — thinnest evidence

soniqo/speech-android: on-device Android SDK using ONNX Runtime, targets Parakeet TDT (25 or 114-language variant), encoder + decoder-joint both INT8 quantized — not confirmed to include Nemotron 3.5 yet.
Voxtype (Linux-focused) uses Parakeet as a backend option, primarily desktop/Linux framing.
No Android-specific benchmarks, NNAPI compatibility reports, or known blockers found for this checkpoint.

Conversion/export paths observed in the ecosystem

ONNX export (int4/int8) — dominant path. onnx-community and istupakov/onnx-asr maintain conversions supporting NeMo Conformer/FastConformer/Parakeet/Canary with CTC/RNNT/TDT/Transformer decoders.
sherpa-onnx — initially unsupported while NVIDIA tuned WER; a community member shipped an independent ONNX export/Rust wrapper before official support landed; maintainer merged official support ~June 12. No cache/state or tokenizer blockers reported.
GGML/native C++ (parakeet.cpp, also supports Nemotron) — RTF 0.0066 on RTX 5090 (327s audio in 2.17s), 3.18% WER, Q8 quantization cut VRAM 5125MiB→4028MiB. Pushback in comments: RTF alone doesn't capture streaming UX (revision rates matter more).
CoreML — only documented for predecessor Parakeet, not yet for Nemotron 3.5.
Microsoft CoreAI research paper (arXiv 2604.14493) — third-party, rigorous CPU-only benchmark on AMD EPYC (32 cores, no GPU): compressed 2.47GB→0.67GB (73% smaller) via calibration-free PTQ; int4 k-quant gave 8.20% WER vs 8.03% FP32 (0.17% degradation, "effectively lossless"); RTFx >6x; ~0.56s algorithmic + ~0.7s effective latency. Scoped as CPU/edge, not phone-tested; excludes diarization/code-switching/custom vocab.

Known blockers / issues

Tokenizer/prompt handling bug (this exact checkpoint): transcribe() throws ValueError: Unknown prompt key: 'None' when target_lang isn't explicitly passed. Open NeMo GitHub issue, unresolved.
Language coverage gaps: Malayalam/Tamil tokenization problems reported; feature requests open for Farsi, Indonesian, Bangla, Azerbaijani.
Misc environment friction: Python 3.10 incompatibility, Windows path errors, from_pretrained() import error — general robustness issues, not mobile-specific.
Cautionary real-world data point (predecessor Parakeet, Mac not mobile): one user reported 22GB unified memory use on M4 Max, an efficiency core pinned at 100%, and 3x slower than faster-whisper on their file — contradicting vendor benchmark numbers; they reverted to Whisper.
No battery-life data exists anywhere — not from NVIDIA, not from any third party.
Cache/streaming state handling was NOT reported as a broken/blocking issue anywhere (sherpa-onnx and GGML ports both worked without cache-export problems) — but this hasn't been stress-tested on mobile-class hardware specifically.

Comparison to what people actually ship today

whisper.cpp: still the incumbent for local/offline ASR. Parakeet/Nemotron benchmarks faster on Apple Silicon (Parakeet ~103x realtime vs Whisper Turbo ~14.3x on M4 Pro in one benchmark), but Whisper wins on language coverage (99 vs ~25-40) and is viewed as more reliable across non-Apple hardware.
Moonshine: praised as the efficient edge specialist (245M params, competitive with 6x larger models), increasingly filling the niche Vosk used to occupy.
Vosk: called out as lightweight but lower accuracy than both Parakeet and Moonshine — losing relevance.
Apple Speech framework / WhisperKit: the native-iOS baseline that new engines (Nemotron/Parakeet-via-CoreML) are being layered alongside/compared to in apps like Hedy and "On-Device AI."

What is NOT yet documented (gaps)

No first-party NVIDIA statement or benchmark of Nemotron 3.5 specifically on a phone (Android or iOS), in a browser, or via TFLite.
No confirmed CoreML conversion of the 3.5 checkpoint itself.
No Reddit threads (r/LocalLLaMA, r/MachineLearning) found — discussion is concentrated on HF discussions, GitHub, HN, and blogs instead.
No TFLite/Android-NNAPI-specific benchmark or known-incompatibility report for this checkpoint.
No battery-life data anywhere, mobile or desktop.

Future plans

Nothing official from NVIDIA — no roadmap stated on the model card, no mobile/web SDK announced. The only forward signal is community demand: the unanswered CoreML request on HF discussions, and independent third parties (onnx-community, sherpa-onnx maintainers, FluidInference) doing the mobile/web enablement work NVIDIA hasn't published itself. If this ships on-device, it'll be via the community ONNX/GGML ecosystem, not an NVIDIA-provided SDK.

Bottom line

Web (WebGPU): real working code, most feasible path today.
iOS: possible — people ship it — but no published performance numbers for this exact checkpoint.
Android: least proven; would likely be first to try Nemotron 3.5 specifically here.

qcam/nemotron-3.5-asr-on-device.md

Select an option

No results found