mlukasze

GPU Performance Analysis: INT8 vs INT4 on Xe2 (Arc Pro B60) for Bonsai-27B / Qwen3.6-27B

Ticket: openvinotoolkit/omega#66 Reported symptom: INT8 weight-compressed inference of Qwen/Qwen3.6-27B (base model for prism-ml/Bonsai-27B-gguf) on GPU.1 (Intel Arc Pro B60, discrete, Xe2/"Battlemage") is ~75x slower (~3.35 s/token) than INT4 on the same device (~44.75 ms/token), despite both being functionally correct (WWB similarity 0.962 for INT8, 0.911 for INT4, both ≥ 0.9 threshold). CPU and GPU.0 (integrated Xe-LPG) show no such anomaly.

Bottom line up front: the dominant root cause is **GPU memory (VRAM) capacity

mlukasze

cohere_asr OpenVINO follow-up — bug investigation & fix (2026-07 iteration)

Related: optimum-intel PR #1788, omega issue #12, prior investigation gist.

Task

A user testing PR #1788 (adds cohere_asr / CohereLabs/cohere-transcribe-03-2026 OpenVINO export+inference support) reported two problems:

1. Export failure with --task automatic-speech-recognition-with-past: MatMul dimension mismatch (1024 vs 1280) in encoder_decoder_proj.
2. After upgrading optimum to 2.2.0.dev0, export succeeded, but inference via OVModelForSpeechSeq2Seq through the transformers ASR pipeline crashed with AttributeError: 'NoneType' object has no attribute 'shape'.

mlukasze

CohereLabs/cohere-transcribe-03-2026 — Working OpenVINO Inference Instructions

Companion to openvinotoolkit/omega#12 and huggingface/optimum-intel#1788.

This gist documents the verified-working way to run this model on OpenVINO today, and the known limitation that is not yet fixable without new upstream work.

TL;DR

mlukasze

{
  "model_id": "nomic-ai/nomic-embed-text-v1.5",
  "canonical_type": "genai",
  "primary_metric": "similarity",
  "runs": [
    {
      "run_id": 1,
      "timestamp": "2026-07-13T17:11:03.405866Z",
      "mode": "full",
      "triggered_by": "optimum-genai",

mlukasze

Intriguing properties of neural networks (2013)

Explaining and harnessing adversarial examples (2014)

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples (2016)

Towards evaluating the robustness of neural networks (2017)

Synthesizing robust adversarial examples (2018)