anthonyalayo

Qwen3.8-27B on RTX 5090 — vLLM vs NInfer

Client-side benchmark of two inference engines serving the same model (Qwen3.8-27B, 262 K context) on a single NVIDIA GeForce RTX 5090 32 GB, measured with tool-eval-bench (v2.5.1.dev30+gded5b8f04, Python 3.14.7): a llama-bench-style pp/tg throughput sweep (depths 0–32 K, concurrency 1/2/4) and a 69-scenario tool-call quality suite.

Setup

anthonyalayo

import csv

from context import Instagram # pylint: disable=no-name-in-module
from time import sleep

# login strings
my_user = 'changeme'
my_pass = 'changeme'

# log in