Michal Lukaszewski mlukasze

CohereLabs/cohere-transcribe-03-2026 — Working OpenVINO Inference Instructions

Companion to openvinotoolkit/omega#12 and huggingface/optimum-intel#1788.

This gist documents the verified-working way to run this model on OpenVINO today, and the known limitation that is not yet fixable without new upstream work.

TL;DR

cohere_asr OpenVINO follow-up — bug investigation & fix (2026-07 iteration)

Related: optimum-intel PR #1788, omega issue #12, prior investigation gist.

Task

A user testing PR #1788 (adds cohere_asr / CohereLabs/cohere-transcribe-03-2026 OpenVINO export+inference support) reported two problems:

Export failure with --task automatic-speech-recognition-with-past: MatMul dimension mismatch (1024 vs 1280) in encoder_decoder_proj.
After upgrading optimum to 2.2.0.dev0, export succeeded, but inference via OVModelForSpeechSeq2Seq through the transformers ASR pipeline crashed with AttributeError: 'NoneType' object has no attribute 'shape'.

GPU Performance Analysis: INT8 vs INT4 on Xe2 (Arc Pro B60) for Bonsai-27B / Qwen3.6-27B

Ticket: openvinotoolkit/omega#66 Reported symptom: INT8 weight-compressed inference of Qwen/Qwen3.6-27B (base model for prism-ml/Bonsai-27B-gguf) on GPU.1 (Intel Arc Pro B60, discrete, Xe2/"Battlemage") is ~75x slower (~3.35 s/token) than INT4 on the same device (~44.75 ms/token), despite both being functionally correct (WWB similarity 0.962 for INT8, 0.911 for INT4, both ≥ 0.9 threshold). CPU and GPU.0 (integrated Xe-LPG) show no such anomaly.

Bottom line up front: the dominant root cause is **GPU memory (VRAM) capacity

	{
	"model_id": "nomic-ai/nomic-embed-text-v1.5",
	"canonical_type": "genai",
	"primary_metric": "similarity",
	"runs": [
	{
	"run_id": 1,
	"timestamp": "2026-07-13T17:11:03.405866Z",
	"mode": "full",
	"triggered_by": "optimum-genai",