Notes from tuning a headless RTX 5090 on a Proxmox host running ollama in an LXC container. Includes a small NVML tool for applying clock offsets without an X server, and measured results showing where the gains actually are.
Short version: memory overclocking bought ~0.7%. Core offset bought ~3%. Undervolting by a specified millivolt figure is not possible on Linux/GeForce.
| GPU | NVIDIA GeForce RTX 5090, 32 GB GDDR7, 512-bit, 28 Gbps stock |
| Driver | 595.45.04, CUDA 13.2 |
| Host | Debian 13 (trixie), Proxmox, headless — no X server |
| Workload | ollama in LXC CT, qwen3.8:27b (dense 27.3B, Q4_K_M, 17.5 GB, fully in VRAM) |
| Stock clocks | core 2648 MHz sustained / 3105 max, memory 14001 MHz |
| Power | min 400 W / default 600 W / max 600 W |
This is the big one, and it is not well documented.
nvidia-smi -q -d VOLTAGEreturns empty on GeForce. You cannot even read core voltage.- NVML exports no voltage setter. The full list of relevant symbols is:
All clocks. No volts.nvmlDeviceGetGpcClkVfOffset nvmlDeviceSetGpcClkVfOffset nvmlDeviceGetMemClkVfOffset nvmlDeviceSetMemClkVfOffset nvmlDeviceGetClockOffsets nvmlDeviceSetClockOffsets nvmlDeviceSetGpuLockedClocks nvmlDeviceResetGpuLockedClocks nvmlDeviceGetMinMaxClockOfPState nvmlDeviceGetPerformanceModes
MSI Afterburner on Windows edits V/F curve points directly. The Linux driver does not expose that surface at all. So "undervolt to 0.95 V" or "-100 mV" is simply not expressible.
What you can do instead: shift the whole V/F curve with a positive core offset. Any given clock is then reached at a lower point on the curve, i.e. lower voltage. Optionally add a ceiling clock lock so the card spends the shift on reduced voltage rather than on boosting higher.
How to verify it, since voltage is unreadable: measure power at constant clock. Same MHz + fewer watts = undervolt. See the caveat in Finding 4 — this is harder than it sounds.
On driver 595, nvidia-smi clocks -h only offers sync-boost and sparse-operation
mode. No offset subcommand. But nvmlDeviceSetClockOffsets exists in
libnvidia-ml.so.1 and works fine headless.
The struct (NVML 555+):
typedef struct {
unsigned int version; // sizeof(struct) | (1 << 24)
nvmlClockType_t type; // GRAPHICS=0 SM=1 MEM=2 VIDEO=3
nvmlPstates_t pstate; // 0 for P0
int clockOffsetMHz;
int minClockOffsetMHz; // filled by Get
int maxClockOffsetMHz; // filled by Get
} nvmlClockOffset_v1_t;Permitted ranges on this card, queried at P0:
graphics P0: allowed [-1000 .. +1000]
mem P0: allowed [-2000 .. +6000]
Pure-ctypes Python, no pynvml needed. Install to /usr/local/sbin/nv-oc.
#!/usr/bin/env python3
"""Query and apply NVIDIA GPU clock offsets through NVML (headless-safe)."""
import argparse, ctypes, sys, time
NVML_SUCCESS = 0
CLOCK_TYPES = {"graphics": 0, "sm": 1, "mem": 2, "video": 3}
class ClockOffset(ctypes.Structure):
_fields_ = [("version", ctypes.c_uint), ("type", ctypes.c_int),
("pstate", ctypes.c_int), ("clockOffsetMHz", ctypes.c_int),
("minClockOffsetMHz", ctypes.c_int),
("maxClockOffsetMHz", ctypes.c_int)]
STRUCT_VERSION = ctypes.sizeof(ClockOffset) | (1 << 24)
def nvml_init(wait=0):
"""Retry nvmlInit for `wait` seconds so this can run early at boot."""
lib = ctypes.CDLL("libnvidia-ml.so.1")
deadline = time.monotonic() + wait
while True:
if lib.nvmlInit_v2() == NVML_SUCCESS:
return lib
if time.monotonic() >= deadline:
raise RuntimeError("nvmlInit_v2 failed")
time.sleep(2)
def get_offset(lib, dev, clock, pstate):
o = ClockOffset(version=STRUCT_VERSION, type=CLOCK_TYPES[clock], pstate=pstate)
if lib.nvmlDeviceGetClockOffsets(dev, ctypes.byref(o)) != NVML_SUCCESS:
raise RuntimeError("get %s offset failed" % clock)
return o
def set_offset(lib, dev, clock, pstate, mhz):
cur = get_offset(lib, dev, clock, pstate)
if not cur.minClockOffsetMHz <= mhz <= cur.maxClockOffsetMHz:
raise RuntimeError("%s offset %+d out of range [%+d..%+d]" % (
clock, mhz, cur.minClockOffsetMHz, cur.maxClockOffsetMHz))
o = ClockOffset(version=STRUCT_VERSION, type=CLOCK_TYPES[clock],
pstate=pstate, clockOffsetMHz=mhz)
if lib.nvmlDeviceSetClockOffsets(dev, ctypes.byref(o)) != NVML_SUCCESS:
raise RuntimeError("set %s offset failed (root?)" % clock)
return cur.clockOffsetMHzFull version adds status / show / apply / reset subcommands and a
one-line status readout.
Two things are required, and the second one is easy to miss:
- A systemd unit to re-apply at boot.
- Persistence mode (
nvidia-smi -pm 1). Offsets are dropped whenever the driver deinitialises, which happens any time no CUDA client is attached. Without persistence mode your overclock silently vanishes every time the workload stops.
[Unit]
Description=Persistent NVIDIA clock offsets (nv-oc)
After=systemd-modules-load.service local-fs.target
Before=ollama.service
[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/default/nvidia-oc
ExecStartPre=-/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/local/sbin/nv-oc apply --mem ${NV_MEM_OFFSET} --gpu ${NV_GPU_OFFSET} --pstates ${NV_PSTATES} --wait 60
ExecStop=/usr/local/sbin/nv-oc reset --pstates ${NV_PSTATES}
[Install]
WantedBy=multi-user.targetNVML memory offsets are in transfer-rate MHz — double the clock that
nvidia-smi reports. An offset of +1400 moves the reported memory clock by
+700. Verify empirically rather than assuming; a wrapper that applies, measures
clocks.max.memory, and self-corrects removes the guesswork.
Naive before/after is useless here — the effect sizes are ~1% and session-to-session drift was larger than the effect (stock measured 141.52 tok/s in one session and 140.26 in another).
What worked:
- Interleave arms within each rep (A/B/A/B), never all-A-then-all-B, so thermal drift and background load hit every arm equally.
- Drive ollama's
/api/generatewithstream:falseand readeval_count/eval_durationfor exact decode tok/s. Don't time the CLI. - Fix
num_predict,temperature: 0,seed. Pin the model in VRAM withkeep_alive. - Warm up once first, to settle model + context config.
- Report the within-arm spread alongside the delta. A delta smaller than that spread is not a result.
- Sample
clocks.current.smandpower.drawat 200 ms vianvidia-smi -lms 200during the run; average only samples above a load threshold.
Three arms, n=4, interleaved:
| memory offset | clock | decode tok/s | vs stock |
|---|---|---|---|
| +0 | 13801 MHz | 140.26 | — |
| +1400 (+5%) | 14501 MHz | 141.19 | +0.66% |
| +2800 (+10%) | 15201 MHz | 141.37 | +0.79% |
Largest within-arm spread: 0.63 tok/s. So +5% is real (arms separated cleanly), but +10% is not distinguishable from +5%. Doubling the overclock added +0.13%.
Prefill was flat (~360 tok/s) across all arms, as expected — it's compute-bound.
The model is dense 27.3B, 17.5 GB of weights (confirmed — no expert.* keys in
/api/show, so not MoE).
- Stock bandwidth roofline, one full weight pass per token: 1792 GB/s ÷ 17.5 GB = 102 tok/s
- Measured: 142 tok/s — 39% above that ceiling
That's only possible because llama.cpp is running MTP speculative decoding
(--spec-type draft-mtp --spec-draft-n-max 4). Multiple tokens land per weight
read, so bandwidth demand per output token drops and the bottleneck moves to
compute and per-step latency.
Fitting Amdahl to the two overclock points: only ~8–14% of decode time is memory-bandwidth-bound. Even infinite VRAM bandwidth caps out near +10% on this workload.
If you run speculative decoding, memory overclocking is close to pointless.
Unlocked core offsets, memory held at +5%, n=2, interleaved:
| core offset | sustained clock | power | decode tok/s | tok/s per W |
|---|---|---|---|---|
| +0 | 2648 MHz | 531.5 W | 143.95 | 0.2708 |
| +150 | 2765 MHz | 510.2 W | 145.03 | 0.2843 |
| +300 | 2780 MHz | 519.3 W | 148.45 | 0.2859 |
+300 gives +3.1% throughput while drawing 12 W less than stock. Efficiency +5.6%.
That combination — 132 MHz faster on less power — is the undervolt effect, reached from the clock side. It just cannot be labelled in millivolts.
The plan: pin the clock with nvidia-smi -lgc X,X so frequency is constant, sweep
the offset, and read the power drop as the voltage drop.
| arm (pin 2800) | actual clock | power | tok/s |
|---|---|---|---|
| +0 | 2592 MHz | 531.3 W | 144.38 |
| +150 | 2666 MHz | 491.4 W | 146.67 |
| +300 | 2536 MHz | 475.1 W | 142.84 |
| +450 | 2507 MHz | 452.6 W | 142.53 |
It doesn't work. The pin holds at idle (2797 MHz — note 2800 snaps to the nearest supported step) but the card sags below it under load, by a different amount per arm. Clock was never constant, so the power deltas conflate frequency and voltage and cannot be converted to millivolts. Any mV figure derived this way is fiction.
Two useful negatives did come out of it:
- Pinning
min=maxcosts ~95 W at idle vs 17 W unpinned. Never persist that. Use a ceiling (-lgc 0,X) if you lock at all. - At
+450pinned, clock sagged while power fell — voltage too low to hold frequency. The usable limit is below there.
Unlocked +300 beat every pinned configuration anyway (148.45 vs best pinned 146.67).
$ nvidia-smi -pl 650
Provided power limit 650.00 W is not a valid power limit which should be
between 400.00 W and 600.00 W
620 W is refused too. min 400 / default 600 / max 600, set by vBIOS. No flag, NVML call, or config moves it.
Worth noting 600 W is also exactly the 12V-2x6 connector's rating, so going past it means running that connector — the one with a melting track record on 4090/5090 — out of spec.
For this workload it's moot: peak draw was 447–530 W of 600 W, so it was never power-limited.
# /etc/default/nvidia-oc
NV_MEM_OFFSET=1680 # +6% memory -> 14841 MHz
NV_GPU_OFFSET=200 # +200 MHz core -> ~2752 MHz sustained
NV_PSTATES=0Measured at this setting: 146.50 tok/s, 2752 MHz, 520 W.
(+300 measured slightly better at 148.34 tok/s / 525 W; +200 was chosen as the
more conservative setting.)
Zero Xid errors across roughly 25 minutes of accumulated load.
- No voltage API on Linux/GeForce. Not readable, not writable. Millivolt undervolting is a Windows-only capability.
- Enable persistence mode or your offsets disappear when the workload stops.
- NVML memory offsets are transfer-rate MHz, 2x the reported clock.
-lgc X,Xcosts ~80 W at idle. Use-lgc 0,Xif you need a ceiling.- Interleave your benchmark arms. Session drift here exceeded the effect size.
- Speculative decoding changes which knob matters — it moves you off the bandwidth roofline, making memory overclocks near-useless and core offsets the real lever.
clocks.max.graphicsdoes not reflect an applied-lgcceiling; it keeps reporting the hardware max.
nv-oc show # permitted offset ranges per pstate
nv-oc status # one-line clocks / power / offsets
oc_nv.sh 6 -g 200 -p # +6% memory, +200 MHz core, persist to boot
oc_nv.sh --status
oc_nv.sh --reset # back to stock immediately
oc_bench.sh 0 1400 2800 # interleaved memory sweep
uv_bench.sh 0:0 0:150 0:300 # interleaved core sweep w/ power samplingLogin banner via /etc/profile.d/, guarded so it doesn't pollute
non-interactive SSH (which would break scp/rsync):
case $- in *i*) ;; *) return 0 ;; esac
[ -x /usr/local/sbin/nv-oc ] || return 0
[ -e /dev/nvidiactl ] || return 0
timeout 5 /usr/local/sbin/nv-oc status 2>/dev/nullGPU0 RTX 5090 SM 2872/3300 MHz MEM 14641/14841 MHz oc[gra+200,mem+1680] 61°C 548/600 W 6% util
- Core sweeps are n=2 per arm; memory sweeps are n=4. The core numbers are thinner evidence.
- No long soak test. ~25 min of accumulated load, zero Xid. Core offsets are the most likely setting to destabilise, and undervolt-style failures can hard-hang the GPU rather than degrade gracefully.
- Results are specific to speculative-decoding LLM inference. A gaming or training workload sits at a different point on the roofline and would likely favour the memory overclock far more.
- Single GPU sample. Silicon varies.