Skip to content

Instantly share code, notes, and snippets.

@mlukasze

mlukasze/gist_summary.md

Created July 15, 2026 07:16
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mlukasze/3dd2060d09025b8a9116858d150da6d6 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mlukasze/3dd2060d09025b8a9116858d150da6d6 to your computer and use it in GitHub Desktop.
Download ZIP
cohere_asr OpenVINO follow-up: two bug reports investigated (PR#1788 / omega#12)
Raw
gist_summary.md

cohere_asr OpenVINO follow-up — bug investigation & fix (2026-07 iteration)

Related: optimum-intel PR #1788, omega issue #12, prior investigation gist.

Task

A user testing PR #1788 (adds cohere_asr / CohereLabs/cohere-transcribe-03-2026 OpenVINO export+inference support) reported two problems:

  1. Export failure with --task automatic-speech-recognition-with-past: MatMul dimension mismatch (1024 vs 1280) in encoder_decoder_proj.
  2. After upgrading optimum to 2.2.0.dev0, export succeeded, but inference via OVModelForSpeechSeq2Seq through the transformers ASR pipeline crashed with AttributeError: 'NoneType' object has no attribute 'shape'.

Mandate: reproduce both in an isolated venv, determine root cause and ownership (our bug vs. user error), fix properly with tests if it's ours, validate across CPU/GPU/NPU, and report back via PR/issue/gist.

Environment

Isolated venv at ~/tmp/cohere-asr-repro/venv (Python 3.12), matching the user's stack:

  • optimum-intel from git+https://github.com/mlukasze/optimum-intel.git@fix/cohere-asr-with-tests (PR #1788 fork branch)
  • optimum 2.2.0.dev0
  • transformers 5.5.4
  • openvino / openvino-genai 2026.2.1

Machine has CPU, 2×GPU (GPU.0 iGPU, GPU.1 dGPU), and an NPU — used for cross-device validation.

Bug #1 — export MatMul mismatch

Verdict: already fixed, not reproducible on current PR HEAD.

Ran optimum-cli export openvino --model CohereLabs/cohere-transcribe-03-2026 --task automatic-speech-recognition-with-past --weight-format fp16 --trust-remote-code against the current branch head (bc5ea4a at time of testing) — export succeeds. The fix landed earlier in the same PR, commit 28d8dbd3 (CohereAsrDummySeq2SeqDecoderTextInputGenerator, which derives the dummy decoder hidden-size input from the encoder's own d_model instead of a hardcoded/mismatched value). The user was very likely testing an intermediate revision of the PR (before that commit) after only upgrading optimum, not optimum-intel, itself.

No code change needed. No further action beyond confirming.

Bug #2 — inference AttributeError

Verdict: our bug (pre-existing, generic gap in optimum-intel), not user error. Fixed.

Reproduction

Direct model.generate(**inputs) calls work fine. The crash only reproduces when going through transformers.pipeline("automatic-speech-recognition", model=ov_model, tokenizer=..., feature_extractor=...) — which is exactly what the user's script does (speech_recognition(audio)). Reproduced the identical traceback, down to the same line (modeling_seq2seq.py:1006, self.next_beam_idx = np.arange(input_ids.shape[0], ...) — input_ids is None).

Root cause

transformers.pipelines.automatic_speech_recognition.AutomaticSpeechRecognitionPipeline.__init__ selects its internal execution mode with:

if model.config.model_type == "whisper":
    self.type = "seq2seq_whisper"
elif model.__class__.__name__ in MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES.values():
    self.type = "seq2seq"
else:
    self.type = "ctc"

MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES only ever contains native PyTorch class names (e.g. CohereAsrForConditionalGeneration), never the optimum-intel OpenVINO wrapper class OVModelForSpeechSeq2Seq. Since cohere_asr's model_type isn't "whisper" (the only architecture special-cased), the pipeline silently falls through to the "ctc" branch — which calls model(**inputs) directly instead of model.generate(**inputs). That's the wrong calling contract for a stateful, generate()-only decoder: decoder_input_ids is never constructed, ends up None, and the crash follows deep inside the OV model's forward pass with an unrelated-looking AttributeError.

This is not specific to cohere_asr — it affects any non-Whisper OVModelForSpeechSeq2Seq architecture (confirmed the same mapping gap also affects qwen3_asr). It equally affects optimum.intel.pipelines.pipeline(), since that convenience factory just delegates to transformers.pipeline().

Fix

Pushed to mlukasze/optimum-intel@fix/cohere-asr-with-tests (commit a665334): register OVModelForSpeechSeq2Seq's class name into MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES once, idempotently, at module import time in optimum/intel/openvino/modeling_seq2seq.py:

def _register_ov_speech_seq2seq_for_pipeline_autodetection():
    try:
        from transformers.models.auto.modeling_auto import MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES
    except ImportError:
        return
    if OVModelForSpeechSeq2Seq.__name__ not in MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES.values():
        MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES["_optimum_intel_ov_speech_seq2seq"] = (
            OVModelForSpeechSeq2Seq.__name__
        )

_register_ov_speech_seq2seq_for_pipeline_autodetection()

This is deliberately generic rather than a cohere_asr-specific patch: it's a single registration of the (architecture-agnostic) OV wrapper class, benefiting every current and future non-Whisper OpenVINO ASR model that goes through OVModelForSpeechSeq2Seq, consistent with how the component is meant to be used across the whole export→inference pipeline (not just cohere_asr).

TDD

Added test_pipeline_autodetection_registers_non_whisper_architectures to OVModelForSpeechSeq2SeqIntegrationTest in tests/openvino/test_seq2seq.py — fast, no model download required. Confirmed RED before the fix (via git stash), GREEN after.

Validation

  • New regression test: RED → GREEN confirmed.
  • Existing Whisper regression tests (test_compare_to_transformers): pass, no regression.
  • End-to-end, real gated model (CohereLabs/cohere-transcribe-03-2026), full pipeline (optimum-cli export → transformers.pipeline("automatic-speech-recognition", ...)):
    • CPU: pass — pipeline.type == "seq2seq", generation completes.
    • GPU.0 (iGPU): pass.
    • GPU.1 (dGPU): pass.
    • NPU: model compilation fails (Upper bounds are not specified for node ... / non-broadcastable dims in the conv pre-encode frontend) — a known OpenVINO NPU compiler limitation with fully-dynamic input shapes, unrelated to this bug or fix (fails before the Python pipeline code even runs). Would need static/bounded-shape reshaping of the encoder to target NPU; tracked as a separate follow-up, not a blocker per the task's own NPU-is-best-effort guidance.
  • openvino_genai.WhisperPipeline remains not supportable for cohere_asr (structural NeMo-Conformer vs. Whisper architecture mismatch) — reconfirmed conclusion from the prior iteration, no code changes expected or made here.

Side discoveries (flagged, not acted on this iteration)

  1. Native transformers cohere_asr support: transformers 5.5.4 now ships a native (non-remote-code) implementation, transformers.models.cohere_asr.modeling_cohere_asr.CohereAsrForConditionalGeneration, registered in CONFIG_MAPPING_NAMES / MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES. It has a different forward contract (attention_mask/output_attention_mask) than the Hub's bespoke remote code (length-based), confirmed by testing export without --trust-remote-code — fails with a dummy-inputs-not-a-subset error against the native config's expected inputs. Our CohereAsrOpenVINOConfig/dummy-input generators target the Hub's remote-code interface, so --trust-remote-code is still required for the export instructions in PR #1788 — no doc changes needed there yet, but worth revisiting once/if the native implementation stabilizes and matches the released model weights.
  2. Missing tiny-random test fixture: the existing cohere_asr tests (test_pipeline, test_compare_to_transformers) reference mlukasze/tiny-random-cohere-asr, which was never uploaded to the Hub (404) — meaning those tests have never actually executed in CI; they silently no-op on a download failure. Attempted to build a tiny fixture; hit a state-dict key-naming mismatch on save/reload (first_sub_layer/second_sub_layer custom decoder naming vs. standard HF self_attn/mlp naming), likely tied to the native-vs-remote-code resolution ambiguity above. Deprioritized as out of scope for these two specific bug reports; recommend a small dedicated follow-up ticket to fix the fixture-generation script and upload the model so these tests actually run in CI, instead of leaving this gap unaddressed indefinitely.
  3. Confirmed the RuntimeError: Float did not match BFloat16 failure seen when substituting the real gated model for the (missing) tiny fixture in test_compare_to_transformers is pre-existing and unrelated to this fix (reproduces identically with the fix stashed out) — an artifact of testing a bf16 model against a test written for a float32 tiny fixture, not a real bug.

Links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment