CohereLabs/cohere-transcribe-03-2026 — Working OpenVINO Inference Instructions (companion to omega#12)

README.md

CohereLabs/cohere-transcribe-03-2026 — Working OpenVINO Inference Instructions

Companion to openvinotoolkit/omega#12 and huggingface/optimum-intel#1788.

This gist documents the verified-working way to run this model on OpenVINO today, and the known limitation that is not yet fixable without new upstream work.

TL;DR

Approach	Status
optimum.intel.OVModelForSpeechSeq2Seq + HF pipeline	✅ Works (fixed in optimum-intel PR #1788, commit f13d947e)
openvino_genai.WhisperPipeline	❌ Not supported — genuine architecture mismatch, documented, not a quick fix

1. Export (do this first, always)

The exact command matters. Passing an explicit --task does not auto-append -with-past (a real usability trap — see root cause section below), so you must include it yourself:

pip install "git+https://github.com/huggingface/optimum-intel.git@fix/cohere-asr-with-tests"
# (until PR #1788 merges to main; afterwards use the released version)

optimum-cli export openvino \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --task automatic-speech-recognition-with-past \
  --weight-format fp16 \
  --trust-remote-code \
  ./cohere-transcribe_ov

Verify the export is correct before proceeding:

python -c "
import openvino as ov
enc = ov.Core().read_model('./cohere-transcribe_ov/openvino_encoder_model.xml')
dec = ov.Core().read_model('./cohere-transcribe_ov/openvino_decoder_model.xml')
print('encoder inputs:', [i.get_any_name() for i in enc.inputs])   # expect input_features + length
print('decoder inputs:', [i.get_any_name() for i in dec.inputs])   # expect beam_idx present
"

If length is missing from the encoder or beam_idx is missing from the decoder, you exported without -with-past or with a stale/older optimum-intel — re-check the command above.

2. Inference — Optimum-Intel (recommended, working path)

from transformers import AutoProcessor, pipeline
from optimum.intel import OVModelForSpeechSeq2Seq
import librosa

model_path = "./cohere-transcribe_ov"
model = OVModelForSpeechSeq2Seq.from_pretrained(model_path, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

speech_recognition = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
)

audio, sr = librosa.load("jfk.wav", sr=16000)
result = speech_recognition(audio)
print(result["text"])

Device selection: pass device="GPU.0" / "GPU.1" / "CPU" to from_pretrained(..., ov_config={"CACHE_DIR": "..."}) or set model.to("GPU.0") — verified working on CPU, iGPU (GPU.0) and discrete Arc Pro B60 (GPU.1) with 3s/7s/11s audio clips, no shape errors, no cache errors.

Known, pre-existing, unrelated caveat: on the current CohereLabs/cohere-transcribe-03-2026 Hub revision, generate() output quality itself is currently degraded (garbled/repetitive text) — this reproduces identically on eager PyTorch (no OpenVINO involved at all), so it is not caused by this fix or by OpenVINO/optimum-intel. It appears to be Hub-side remote-code drift: an older revision's feature extractor supported decoder_input_ids/language-conditioning that the current revision's CohereAsrFeatureExtractor no longer honors. If you need production-quality WER, track this separately — it is out of scope for the export/inference-contract bugs fixed here. The originally reported reference WER numbers (1.20% CPU / 1.25% reference) were measured against an older Hub revision and can no longer be reproduced as-is.

3. Inference — OpenVINO GenAI WhisperPipeline (NOT SUPPORTED)

import openvino_genai as ov_genai
pipe = ov_genai.WhisperPipeline("./cohere-transcribe_ov", "CPU")

This will not work, even against the fixed export above. It is not a config issue and not a one-line fix. Root causes (three independent, structural mismatches):

1. generation_config.json fields: WhisperGenerationConfig requires a Whisper-style max_length (and other Whisper-only fields like lang_to_id) that Cohere-ASR's config doesn't have and can't gain without misrepresenting the model. Without it, max_new_tokens underflows to SIZE_MAX and pipe.generate() raises ValueError: vector::reserve.
2. Missing length-tensor threading: whisper.cpp's encode() only ever sets the input_features tensor; it has no concept of the new dynamic length input this model's encoder requires. Even bypassing failure #1, the Conformer's internal length-based masking collapses a dimension to 0 and OpenVINO throws a Broadcast shape-inference error.
3. Incompatible mel-feature-extraction algorithm: ov::genai::WhisperFeatureExtractor is a fixed-30s-chunk, non-length-aware, log10 mel implementation matching OpenAI Whisper exactly. CohereAsrFeatureExtractor is a NeMo-style FilterbankFeatures extractor with length-masked per-feature (per-mel-bin) CMVN normalization, different log compression, dither, and optional frame splicing — there is no way to numerically reproduce this inside the current WhisperFeatureExtractor implementation.

Supporting this model in openvino_genai would require a new, dedicated Conformer/NeMo-ASR pipeline type (own feature extractor, own generation-config handling, length-tensor threading) — genuinely new-pipeline-type-scope work, not a bugfix. This has been documented directly in the openvino.genai docs (site/docs/supported-models/index.mdx) so it isn't silently reattempted; a full technical gap analysis with exact file/line references is attached in this gist (genai_gap_analysis.md) for whoever picks up that follow-up.

Summary of what changed upstream

• optimum-intel PR #1788 (branch fix/cohere-asr-with-tests, commit f13d947e): fixed the encoder's static-3000-frame dummy-input contract (now dynamic + length-aware), and fixed the decoder's broken with-past export (trace-time-constant cache-length bug in the Hub remote code, patched for trace duration only). New regression tests added (tests/openvino/test_seq2seq.py).
• openvino.genai: no code changed. cohere_asr / NeMo-Conformer-style ASR architectures documented as explicitly unsupported by WhisperPipeline, with a link back to this issue.
• Both changes validated end-to-end on real hardware: CPU (Core Ultra 9 285K), GPU.0 (Arrow Lake-U iGPU), GPU.1 (Arc Pro B60 dGPU).

References

• Issue: https://github.com/openvinotoolkit/omega/issues/12
• Fix PR: huggingface/optimum-intel#1788

genai_gap_analysis.md

GenAI Gap Analysis — cohere_asr / CohereAsrForConditionalGeneration

Model: CohereLabs/cohere-transcribe-03-2026 Ticket: https://github.com/openvinotoolkit/omega/issues/12 Related fix: huggingface/optimum-intel#1788 (commit f13d947e, branch fix/cohere-asr-with-tests) Conclusion: Not supportable by WhisperPipeline without new pipeline-level C++ work. No openvino_genai code was changed. Documentation was updated to mark this architecture family explicitly unsupported (site/docs/supported-models/index.mdx).

---

1. Re-test against the fixed optimum-intel export

Re-exported a fresh IR from fix/cohere-asr-with-tests (commit f13d947e) using the tiny-random cohere_asr model already available at dev/.../tiny-random-cohere-asr (same architecture, same config/preprocessor conventions as the real 2B model — used here instead of the full model purely for turnaround speed; the shape/contract findings below are architecture-level, not size-dependent):

optimum-cli export openvino --model tiny-random-cohere-asr \
  --task automatic-speech-recognition-with-past --trust-remote-code ./tiny-cohere-asr-ov

Confirmed the two optimum-intel bugs are genuinely fixed:

Port	Before (PR #1788 original)	After (f13d947e)
encoder input_features	[1, 128, 3000] (static)	[?, ?, ?] (dynamic)
encoder length	(absent)	[?] (new)
decoder beam_idx	(absent — not stateful)	[?] (present — stateful)

The original beam_idx port-not-found error from the ticket is gone, as expected.

2. New failure mode #1 — WhisperGenerationConfig assumes Whisper-specific fields

pipe = ov_genai.WhisperPipeline("./tiny-cohere-asr-ov", "CPU")
config = pipe.get_generation_config()
config.return_timestamps = False
result = pipe.generate(audio, config)

ValueError: vector::reserve

Root cause (traced via src/cpp/src/generation_config.cpp:253-259 and whisper.cpp:302-304): GenerationConfig::get_max_new_tokens() falls back to max_length - prompt_length when max_new_tokens is unset. max_length defaults to SIZE_MAX unless overridden by generation_config.json. Whisper's own generation_config.json always ships a Whisper-specific max_length (typically 448). Cohere-ASR's generation_config.json (both the tiny-random fixture and the real model, produced by the Hub remote code / optimum-intel export) only contains bos_token_id / eos_token_id / pad_token_id / decoder_start_token_id — no max_length, no lang_to_id, no is_multilingual. With max_length left at SIZE_MAX, max_new_tokens becomes astronomically large, and the .reserve() calls in whisper_generate() (whisper.cpp:302-304) throw std::length_error, surfaced to Python as ValueError: vector::reserve.

This is not a one-line default fix: it demonstrates that WhisperGenerationConfig is built around Whisper's generation_config.json conventions (max_length, lang_to_id, is_multilingual, forced decoder ids) which this architecture's export pipeline has no reason to produce, since Cohere-ASR's own HF generate() contract does not use them either.

3. New failure mode #2 — no length tensor is ever set, Conformer masking breaks

Setting config.max_new_tokens manually to bypass finding #2 above reaches the actual encoder/decoder inference and immediately fails inside the encoder graph:

RuntimeError: Exception from .../infer_request.cpp:224:
Exception from .../node.cpp:787:
[CPU] Broadcast node with name '__module.pre_encode.conv/aten::mul/Multiply' ...
Input shape dimension equal 0 cannot be broadcasted (numpy mode) to 1.

Root cause: encode() in whisper.cpp:217-248 only calls request.set_tensor("input_features", ...). It never sets the encoder's new length input at all (confirmed by reading the full function body — no reference to "length" anywhere in whisper.cpp). The length port is therefore left as whatever OpenVINO default-initializes an unset required input to (an all-zero/empty tensor), which the Conformer's internal ConvSubsampling/masking (pre_encode.conv) uses to build a broadcast mask — a zero length collapses a dimension to 0, which cannot broadcast to 1, hence the shape-inference failure.

This confirms the analysis in the architecture report: there is no length-threading anywhere in whisper.cpp's encode/decode call chain. Even if this one broadcast were somehow patched around, the same problem exists at every other point in the pipeline that assumes the encoder output has no notion of "how many of these frames are real vs. padding" (is_shortform chunk logic, nb_max_frames-based windowing in whisper.cpp:310+, feature_extractor.cpp's get_data_with_offset chunk logic) — all Whisper-specific 30-second-chunk conventions this model was never designed around.

4. Feature-extractor algorithm mismatch (independent of the above; blocks even a "single 30s

chunk with hand-set length" workaround)

Compared ov::genai::WhisperFeatureExtractor (src/cpp/src/whisper/feature_extractor.cpp) against the Hub CohereAsrFeatureExtractor (processing_cohere_asr.py, NeMo-style FilterbankFeatures) by reading both implementations line-by-line (a full numeric run against real audio was not needed — the algorithms diverge structurally, not just parametrically):

Aspect	WhisperFeatureExtractor (genai C++)	CohereAsrFeatureExtractor (Hub remote code)
Log compression	log10(max(x, 1e-10)) (fixed floor)	torch.log(x + log_zero_guard_value), natural log, configurable guard (2**-24, "add"/"clamp" mode)
Post-log normalization	none	per-feature (per-mel-bin) mean/variance normalization over the valid (length-masked) time steps — this step is itself length-dependent, so it cannot even be computed correctly without the real frame count
Dither	not applied	applied per-sample, seeded by valid waveform length (_apply_dither)
Frame splicing	not supported	supported (frame_splicing param, splice_frames)
Windowing	Whisper hann, n_fft-sized window	NeMo hann, separate n_window_size(320)/n_fft handling, exact_pad option
Mel filter construction	Whisper's own mel_filter_bank() (slaney-style triangular filters + enorm) — hardcoded, not driven by preprocessor_config.json's _fb_config block at all	NeMo FilterbankFeatures mel filter with configurable mel_norm ("slaney"), nfilt, lowfreq/highfreq, stft_conv

Even where surface-level parameters coincide (feature_size: 128, mel_norm: "slaney"), the per-feature CMVN normalization step has no equivalent at all in the genai C++ implementation, and it is fundamentally length-aware (it only normalizes over real, non-padded frames per sample). This alone would produce numerically wrong encoder inputs even if every shape/tensor plumbing issue above were fixed. This is a different, NeMo-derived feature-extraction algorithm, not a Whisper-preprocessor-with-different-constants — implementing it would require a new FeatureExtractor implementation in openvino.genai, not a config tweak.

5. Was the "pad to 3000 + internal masking" approach (issue #12 history) viable here?

No — re-examined the issue #12 history's original manual workaround (pad input_features to 3000 frames + supply length, with a masked-softmax fix for NaN propagation from the Conv/attention layers on the zero-padded rows). That approach assumed a Whisper-shaped encoder contract from the start (fixed nb_max_frames, Whisper's own log-mel with no length-dependent normalization). It does not resolve findings #2 (WhisperGenerationConfig field assumptions) or #4 (fundamentally different, length-aware normalization algorithm) above, both of which are structural, not shape-related. Formalizing that workaround into WhisperPipeline would still leave the pipeline numerically wrong for this architecture and is not recommended.

6. What a proper fix would require (scope estimate for a future ticket)

A genuine cohere_asr / NeMo-Conformer-ASR pipeline needs:

1. A new feature extractor implementing NeMo FilterbankFeatures semantics: natural-log compression with configurable zero-guard, dither, optional frame splicing, and — critically — length-aware per-feature mean/variance normalization. This cannot reuse WhisperFeatureExtractor; it should be a new class (e.g. ConformerFeatureExtractor) reading the _fb_config block from preprocessor_config.json.
2. length-tensor threading through the encoder call in the new pipeline's encode() equivalent, and through any decoder cross-attention masking that depends on true encoder sequence length (verify whether the decoder needs an explicit encoder-attention-mask input, or whether it relies on the encoder's own internal masking only — not yet investigated, out of scope for this analysis).
3. A generation-config path that does not assume Whisper conventions: no lang_to_id, is_multilingual, or Whisper-specific max_length defaulting; max_new_tokens/max_length validation should not silently fall back to SIZE_MAX-derived arithmetic for models whose generation_config.json doesn't define these fields (arguably a robustness bug in GenerationConfig validation worth a separate, narrower ticket regardless of cohere_asr).
4. Chunking/streaming policy: whether variable-length (non-30s-chunked) architectures need their own single-shot (no long-form chunking) code path, since Cohere-ASR's own remote code does not implement Whisper's 30s sliding-window convention at all.
5. Tiny-random model + C++ functional test + Python test per contribution guidelines, once a pipeline type is designed.

This is new-pipeline-type-scale work, not a bug fix, and is out of scope for this task.

Evidence artifacts

• Fresh IR: dev/CohereLabs-cohere-transcribe-03-2026/tiny-cohere-asr-ov/ (encoder confirmed input_features [?,?,?] + length [?]; decoder confirmed beam_idx [?], stateful).
• Repro script output captured in this document (§2, §3).
• Source reading: openvino.genai/src/cpp/src/whisper/{whisper.cpp,feature_extractor.cpp,generation_config.cpp}, tiny-random-cohere-asr/processing_cohere_asr.py.