Skip to content

Instantly share code, notes, and snippets.

@costis-t

costis-t/logs.md

Last active August 5, 2026 02:18
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save costis-t/9ce04a2e009b77a30d789155a765442d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save costis-t/9ce04a2e009b77a30d789155a765442d to your computer and use it in GitHub Desktop.
Download ZIP
nVidia 595.84 and 610.57.04: hibernation fails
Raw
logs.md

AI-generated supporting logs for a comment on NVIDIA/open-gpu-kernel-modules#1226.

Disk and LUKS UUIDs in the kernel command line have been replaced with <UUID-A> / <UUID-B>.

The complete matrix

595.84 610.57.04
A — NVIDIA modules in the initramfs (force_drivers) restore leg: pci_pm_freeze returns -5 restore leg: pci_pm_freeze returns -5
B — modules not in the initramfs write leg: NULL deref in nvEvoDisableVblankSemControl write leg: NULL deref in nvEvoDisableVblankSemControl

The configuration determines which leg fails. The driver version changes nothing. No combination completes a hibernate round trip.

System
GPU NVIDIA Blackwell-arch GB203, 10de:2c05 rev a1, subsystem 1458:4183 (Gigabyte)
Distro Gentoo, systemd
Driver x11-drivers/nvidia-drivers-595.84 (runs 1-2) and 610.57.04 (runs 3-4), open kernel modules, built from source with GCC 15.3.0
Swap 70 GiB btrfs swapfile on a LUKS2 + btrfs root, resume=/resume_offset= on the kernel command line at the time of all four runs

Hibernation target is a btrfs swapfile inside the encrypted root, unlocked in the initramfs by systemd-cryptsetup. The image is written and read through dm-crypt at full NVMe speed (1765.88 MB/s measured, see run 1), so the storage path is not implicated in either failure.

Configuration

/etc/modprobe.d/nvidia.conf (relevant parts, Gentoo's default)

options nvidia \
	NVreg_UseKernelSuspendNotifiers=1 \
	NVreg_TemporaryFilePath=/var/tmp

options nvidia \
	NVreg_DeviceFileGID=27 \
	NVreg_DeviceFileMode=432 \
	NVreg_DeviceFileUID=0 \
	NVreg_ModifyDeviceFiles=1

blacklist nouveau
blacklist nova_core

NVreg_UseKernelSuspendNotifiers=1 implies NVreg_PreserveVideoMemoryAllocations=1. /proc/driver/nvidia/suspend does not exist on this system, and x11-drivers/nvidia-drivers-595.84 installs no nvidia-suspend.service / nvidia-hibernate.service / nvidia-resume.service — only drop-ins setting SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=false on the systemd sleep units. Gentoo's own comment in that file states that the former systemd suspend services are not needed with kernel notifiers.

/etc/dracut.conf.d/10-gentoo.conf

hostonly="yes"
hostonly_cmdline="no"
compress="zstd"
early_microcode="yes"
add_dracutmodules+=" crypt btrfs systemd systemd-cryptsetup dm rootfs-block resume "
omit_dracutmodules+=" plymouth network iscsi nfs nbd "
use_fstab="yes"

/etc/dracut.conf.d/20-nvidia.conf

Configuration A (run 1):

force_drivers+=" nvidia nvidia_modeset nvidia_uvm nvidia_drm "
install_items+=" /usr/bin/nvidia-modprobe "

Configuration B (run 2) — the force_drivers line commented out, nothing else changed:

#force_drivers+=" nvidia nvidia_modeset nvidia_uvm nvidia_drm "
install_items+=" /usr/bin/nvidia-modprobe "

Switching between A and B regenerates the initramfs and Unified Kernel Image (emerge --config sys-kernel/gentoo-kernel-bin). Verified on each switch that nvidia.ko, nvidia_modeset.ko, nvidia_uvm.ko and nvidia_drm.ko were absent from (B) or present in (A) the initramfs, and that rd.driver.pre= entries followed accordingly.

Kernel command line (all runs)

root=UUID=<UUID-A> rootflags=subvol=@,discard=async rootfstype=btrfs
rd.luks.uuid=<UUID-B> rd.luks.name=<UUID-B>=cryptroot
rd.luks.options=<UUID-B>=discard,no-read-workqueue,no-write-workqueue
resume=UUID=<UUID-A> resume_offset=533760 zswap.enabled=1 rw loglevel=4
nvidia-drm.modeset=1 nvidia-drm.fbdev=1

Reproduction

systemctl hibernate

100% reproducible in all four combinations of configuration and driver version. Nothing else is running that touches the GPU beyond the Plasma Wayland session itself.

Run 1 — configuration A (modules in the initramfs): restore leg fails, pci_pm_freeze returns -5

The image is written correctly and the machine powers off. The next boot finds the image, reads all 22 GB back, and then the device-freeze pass that must precede handing control to the restored image is refused by the NVIDIA driver.

Aug 04 19:34:34 host kernel: ACPI: PM: Registering ACPI NVS region [mem 0x0a200000-0x0a20dfff] (57344 bytes)
Aug 04 19:34:34 host kernel: ACPI: PM: Registering ACPI NVS region [mem 0xca5ed000-0xca9d5fff] (4100096 bytes)
Aug 04 19:34:34 host kernel: PM: RTC time: 17:34:30, date: 2026-08-04
Aug 04 19:34:34 host kernel: ACPI: PM: (supports S0 S3 S4 S5)
Aug 04 19:34:34 host kernel: clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Aug 04 19:34:34 host kernel: PM:   Magic number: 6:146:592
Aug 04 19:34:34 host kernel: PM: genpd: Disabling unused power domains
Aug 04 19:34:34 host systemd-hibernate-resume-generator[409]: Reported hibernation image: ID=gentoo VERSION_ID=2.18 kernel=6.18.41-gentoo-dist-bin UUID=<UUID-A> offset=533760
Aug 04 19:34:34 host kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  595.84  Release Build  (portage@localhost)  Tue Aug  4 09:06:37 AM CEST 2026
Aug 04 19:34:34 host kernel: Loading firmware: nvidia/595.84/gsp_ga10x.bin
Aug 04 19:34:47 host kernel: PM: Image signature found, resuming
Aug 04 19:34:47 host kernel: PM: hibernation: resume from hibernation
Aug 04 19:34:47 host kernel: Loading firmware: nvidia/595.84/gsp_ga10x.bin
Aug 04 19:35:01 host kernel: PM: hibernation: Basic memory bitmaps created
Aug 04 19:35:01 host kernel: PM: Using 3 thread(s) for lzo decompression
Aug 04 19:35:01 host kernel: PM: Loading and decompressing image data (5553706 pages)...
Aug 04 19:35:01 host kernel: PM: Image loading progress:   0%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  10%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  20%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  30%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  40%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  50%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  60%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  70%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  80%
Aug 04 19:35:01 host kernel: PM: Image loading progress:  90%
Aug 04 19:35:01 host kernel: PM: Image loading progress: 100%
Aug 04 19:35:01 host kernel: PM: Image loading done
Aug 04 19:35:01 host kernel: PM: hibernation: Read 22214824 kbytes in 12.58 seconds (1765.88 MB/s)
Aug 04 19:35:01 host kernel: PM: Image successfully loaded
Aug 04 19:35:01 host kernel: NVRM: GPU 0000:08:00.0: PreserveVideoMemoryAllocations module parameter is set. System Power Management attempted without driver procfs suspend interface. Please refer to the 'Configuring Power Management Support' section in the driver README.
Aug 04 19:35:01 host kernel: nvidia 0000:08:00.0: PM: pci_pm_freeze(): nvidia_isr_kthread_bh [nvidia] returns -5
Aug 04 19:35:01 host kernel: nvidia 0000:08:00.0: PM: dpm_run_callback(): pci_pm_freeze returns -5
Aug 04 19:35:01 host kernel: nvidia 0000:08:00.0: PM: failed to quiesce async: error -5
Aug 04 19:35:01 host kernel: PM: hibernation: Failed to load image, recovering.
Aug 04 19:35:01 host kernel: PM: hibernation: Basic memory bitmaps freed
Aug 04 19:35:01 host kernel: PM: hibernation: resume failed (-5)
Aug 04 19:35:01 host kernel: BTRFS info (device dm-0): first mount of filesystem <UUID-A>
Aug 04 19:35:02 host systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67

Read top to bottom, almost everything passes: resume=/resume_offset= resolve, the image signature is found, 22214824 kbytes are read back off the encrypted root at 1765.88 MB/s, and PM: Image successfully loaded. Only then does pci_pm_freeze return -EIO. The kernel recovers cleanly and continues as an ordinary cold boot; nothing is corrupted.

Note the BTRFS info (device dm-0): first mount line lands after the failure, confirming the resume was attempted from the initramfs before the real root was mounted.

Run 2 — configuration B (modules NOT in the initramfs): write leg fails, kernel oops

With force_drivers removed and nothing else changed, the failure moves to the other leg. The snapshot is taken successfully, CPUs are offlined and brought back, and the oops lands in the device-resume pass that has to run before the image can be written out. The image is therefore never written, and the next boot reports PM: Image not found (code -22).

Hibernation entry through the failed device-resume pass

Abridged: per-CPU offline/online lines, unrelated USB/SATA resume chatter and firewall log lines removed. Nothing between the retained lines bears on the GPU.

Aug 04 23:44:14 host systemd-sleep[3046]: User sessions remain unfrozen on explicit request ($SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=0).
Aug 04 23:44:14 host systemd-sleep[3046]: This is not recommended, and might result in unexpected behavior, particularly
Aug 04 23:44:14 host systemd-sleep[3046]: in suspend-then-hibernate operations or setups with encrypted home directories.
Aug 04 23:44:14 host systemd-sleep[3046]: Performing sleep operation 'hibernate'...
Aug 04 23:44:14 host kernel: PM: hibernation: hibernation entry
Aug 04 23:44:14 host kernel: Loading firmware: nvidia/595.84/gsp_ga10x.bin
Aug 04 23:45:11 host kernel: Freezing user space processes
Aug 04 23:45:11 host kernel: Freezing user space processes completed (elapsed 0.001 seconds)
Aug 04 23:45:11 host kernel: OOM killer disabled.
Aug 04 23:45:11 host kernel: PM: hibernation: Basic memory bitmaps created
Aug 04 23:45:11 host kernel: PM: hibernation: Preallocating image memory
Aug 04 23:45:11 host kernel: PM: hibernation: Allocated 1606643 pages for snapshot
Aug 04 23:45:11 host kernel: PM: hibernation: Allocated 6426572 kbytes in 1.93 seconds (3329.83 MB/s)
Aug 04 23:45:11 host kernel: Freezing remaining freezable tasks
Aug 04 23:45:11 host kernel: Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
Aug 04 23:45:11 host kernel: ACPI: PM: Preparing to enter system sleep state S4
Aug 04 23:45:11 host kernel: ACPI: PM: Saving platform NVS memory
Aug 04 23:45:11 host kernel: PM: hibernation: Creating image:
Aug 04 23:45:11 host kernel: PM: hibernation: Need to copy 1509603 pages
Aug 04 23:45:11 host kernel: PM: hibernation: Normal pages needed: 1509603 + 1024, available pages: 15242423
Aug 04 23:45:11 host kernel: ACPI: PM: Restoring platform NVS memory
Aug 04 23:45:11 host kernel: ACPI: PM: Waking up from system sleep state S4
Aug 04 23:45:11 host kernel: ata5: failed to resume link (SControl 0)
Aug 04 23:45:11 host kernel: PM: hibernation: Basic memory bitmaps freed
Aug 04 23:45:11 host kernel: OOM killer enabled.
Aug 04 23:45:11 host kernel: efivarfs: resyncing variable state
Aug 04 23:45:11 host kernel: efivarfs: removing variable HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67
Aug 04 23:45:11 host kernel: efivarfs: finished resyncing variable state
Aug 04 23:45:16 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010026; hClass=0x0000ca7e; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 04 23:45:24 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010021; hClass=0x0000ca7b; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 04 23:45:32 host kernel: note: systemd-sleep[3046] exited with irqs disabled

Points to note in that sequence:

  • 57 seconds pass between hibernation entry and Freezing user space processes. GSP firmware is reloaded at the start of that window. This gap does not appear in configuration A.
  • The snapshot itself is fine: 1509603 pages copied, NVS saved and restored, Waking up from system sleep state S4 is the normal post-snapshot return that precedes writing the image.
  • efivarfs: removing variable HibernateLocation then fires — i.e. the location record is torn down before any image is written.
  • Two GspRmAlloc timeouts follow, 8 seconds apart, on different objects (hClass=0x0000ca7e, then hClass=0x0000ca7b), before the oops at 23:45:32.
  • note: systemd-sleep[3046] exited with irqs disabled.
  • No PM: Writing image data / image-saved lines appear at all: the image is never written.

GSP timeout, then the oops

Aug 04 23:45:24 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010021; hClass=0x0000ca7b; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 04 23:45:24 host kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Aug 04 23:45:24 host kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine window channels
Aug 04 23:45:32 host kernel: BUG: unable to handle page fault for address: 00000000000026b0
Aug 04 23:45:32 host kernel: #PF: supervisor read access in kernel mode
Aug 04 23:45:32 host kernel: #PF: error_code(0x0000) - not-present page
Aug 04 23:45:32 host kernel: PGD 0 P4D 0 
Aug 04 23:45:32 host kernel: Oops: Oops: 0000 [#1] SMP NOPTI
Aug 04 23:45:32 host kernel: CPU: 3 UID: 0 PID: 3046 Comm: systemd-sleep Tainted: G           OE       6.18.41-gentoo-dist-bin #1 PREEMPT(lazy) 
Aug 04 23:45:32 host kernel: Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Aug 04 23:45:32 host kernel: Hardware name: ASUS System Product Name/ROG STRIX B550-F GAMING WIFI II, BIOS 3641 06/23/2026
Aug 04 23:45:32 host kernel: RIP: 0010:nvEvoDisableVblankSemControl+0x34/0xd0 [nvidia_modeset]
Aug 04 23:45:32 host kernel: Code: 49 89 ff 41 54 55 53 49 89 f4 48 83 ec 08 8b 06 48 8d 5e 10 4c 8b b4 c7 d0 02 00 00 49 8d ae d8 26 00 00 4d 8d ae d8 27 00 00 <48> 8b 55 d8 48 8b 43 20 8b 08 48 89 50 10 f0 48 83 0c 24 00 89 48
Aug 04 23:45:32 host kernel: RSP: 0018:ffffd273063e7968 EFLAGS: 00010292
Aug 04 23:45:32 host kernel: RAX: 0000000000000000 RBX: ffff89f9c881bb18 RCX: ffff89f9c881bb08
Aug 04 23:45:32 host kernel: RDX: 0000000000000000 RSI: ffff89f9c881bb08 RDI: ffffd27301bd9008
Aug 04 23:45:32 host kernel: RBP: 00000000000026d8 R08: ffff89fa0606f1f0 R09: ffffd273063e79b4
Aug 04 23:45:32 host kernel: R10: ffff89f9d36fc000 R11: ffff89f9c004fc00 R12: ffff89f9c881bb08
Aug 04 23:45:32 host kernel: R13: 00000000000027d8 R14: 0000000000000000 R15: ffffd27301bd9008
Aug 04 23:45:32 host kernel: FS:  00007f1528b1b840(0000) GS:ffff8a0917449000(0000) knlGS:0000000000000000
Aug 04 23:45:32 host kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 04 23:45:32 host kernel: CR2: 00000000000026b0 CR3: 000000010b9b6000 CR4: 0000000000f50ef0
Aug 04 23:45:32 host kernel: PKRU: 55555554
Aug 04 23:45:32 host kernel: Call Trace:
Aug 04 23:45:32 host kernel:  <TASK>
Aug 04 23:45:32 host kernel:  nvFreePerOpenDev+0x13d/0x240 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  nvRevokeDevice+0x94/0x150 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  nvFreePerOpenDev+0x229/0x240 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  nvRevokeDevice+0x94/0x150 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  nvResumeDevEvo+0x1f7/0x200 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? up+0x53/0xa0
Aug 04 23:45:32 host kernel:  nvKmsResume+0x5c/0xd0 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  ? down+0x1e/0x60
Aug 04 23:45:32 host kernel:  nvKmsKapiUI32ToF32+0x7f/0x830 [nvidia_modeset]
Aug 04 23:45:32 host kernel:  nvidia_isr_kthread_bh+0xdf1/0xf20 [nvidia]
Aug 04 23:45:32 host kernel:  nvidia_isr_kthread_bh+0xeb5/0xf20 [nvidia]
Aug 04 23:45:32 host kernel:  notifier_call_chain+0x5d/0xd0
Aug 04 23:45:32 host kernel:  blocking_notifier_call_chain+0x42/0x70
Aug 04 23:45:32 host kernel:  hibernate.cold+0x3b/0x449
Aug 04 23:45:32 host kernel:  state_store+0x45/0x60
Aug 04 23:45:32 host kernel:  kernfs_fop_write_iter+0x14d/0x200
Aug 04 23:45:32 host kernel:  vfs_write+0x25d/0x480
Aug 04 23:45:32 host kernel:  ksys_write+0x73/0xf0
Aug 04 23:45:32 host kernel:  do_syscall_64+0x82/0x7d0
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? do_syscall_64+0xba/0x7d0
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? do_syscall_64+0xba/0x7d0
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 04 23:45:32 host kernel:  ? irqentry_exit_to_user_mode+0x2c/0x210
Aug 04 23:45:32 host kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 04 23:45:32 host kernel: RIP: 0033:0x7f15284a08fa
Aug 04 23:45:32 host kernel: Code: 08 03 00 00 59 5e 48 83 f8 fc 75 1e 83 e2 39 83 fa 08 75 16 e8 17 ff ff ff 0f 1f 80 00 00 00 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 83 c4 18 c3 90 f3 0f 1e fa 48 83 ec 10 ff 74 24 18 e8 5f ff ff
Aug 04 23:45:32 host kernel: RSP: 002b:00007ffc1c6011d0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
Aug 04 23:45:32 host kernel: RAX: ffffffffffffffda RBX: 000055cf61875010 RCX: 00007f15284a08fa
Aug 04 23:45:32 host kernel: RDX: 0000000000000005 RSI: 000055cf6187e6c0 RDI: 0000000000000005
Aug 04 23:45:32 host kernel: RBP: 0000000000000005 R08: 0000000000000000 R09: 0000000000000000
Aug 04 23:45:32 host kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000005
Aug 04 23:45:32 host kernel: R13: 000055cf6187e6c0 R14: 0000000000000000 R15: 00007ffc1c601380
Aug 04 23:45:32 host kernel:  </TASK>
Aug 04 23:45:32 host kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq des3_ede_x86_64 des_generic libdes md4 vfat fat nvidia_drm(OE) nvidia_modeset(OE) mt7921e mt7921_common uvcvideo mt792x_lib snd_hda_codec_alc882 uvc amd_atl intel_rapl_msr mt76_connac_lib videobuf2_vmalloc snd_hda_codec_realtek_lib snd_hda_codec_nvhdmi intel_rapl_common videobuf2_memops mt76 ee1004 snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel btusb videobuf2_v4l2 btmtk snd_usb_audio snd_hda_codec videobuf2_common mac80211 btrtl snd_hda_core snd_usbmidi_lib btbcm videodev nvidia(OE) edac_mce_amd btintel snd_ump snd_intel_dspcfg joydev asus_nb_wmi snd_intel_sdw_acpi snd_rawmidi bluetooth mc asus_wmi snd_seq_device rapl snd_hwdep sparse_keymap drm_ttm_helper cfg80211 platform_profile wmi_bmof snd_pcm ttm snd_timer drm_client_lib rfkill pcspkr k10temp libarc4 drm_kms_helper snd igc video gpio_amdpt soundcore gpio_generic i2c_piix4 i2c_smbus binfmt_misc ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog
Aug 04 23:45:32 host kernel:  xt_comment xt_multiport xt_limit xt_addrtype xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables tun fuse nfnetlink dm_crypt uas usb_storage polyval_clmulni ghash_clmulni_intel sp5100_tco wmi pkcs8_key_parser
Aug 04 23:45:32 host kernel: CR2: 00000000000026b0
Aug 04 23:45:32 host kernel: ---[ end trace 0000000000000000 ]---

systemd's view

Aug 04 23:44:14 host systemd-logind[1949]: The system will hibernate now!
Aug 04 23:44:14 host systemd-sleep[3046]: User sessions remain unfrozen on explicit request ($SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=0).
Aug 04 23:44:14 host systemd-sleep[3046]: This is not recommended, and might result in unexpected behavior, particularly
Aug 04 23:44:14 host systemd-sleep[3046]: in suspend-then-hibernate operations or setups with encrypted home directories.
Aug 04 23:44:14 host systemd-sleep[3046]: Performing sleep operation 'hibernate'...
Aug 04 23:45:32 host kernel: CPU: 3 UID: 0 PID: 3046 Comm: systemd-sleep Tainted: G           OE       6.18.41-gentoo-dist-bin #1 PREEMPT(lazy) 
Aug 04 23:45:32 host kernel: note: systemd-sleep[3046] exited with irqs disabled
Aug 04 23:45:32 host systemd[1]: systemd-hibernate.service: Main process exited, code=killed, status=9/KILL
Aug 04 23:45:32 host systemd[1]: systemd-hibernate.service: Failed with result 'signal'.
Aug 04 23:45:32 host systemd[1]: systemd-hibernate.service: Consumed 24.877s CPU time over 31.192s wall clock time, 970.2M memory peak.
Aug 04 23:45:32 host systemd-logind[1949]: Operation 'hibernate' finished.

Next boot

Aug 04 23:47:23 host kernel: PM: Image not found (code -22)

Run 3 — configuration B on driver 610.57.04: identical crash

x11-drivers/nvidia-drivers-610.57.04 (open), released 2026-08-03. Same kernel 6.18.41, configuration B, nothing else changed. The crash reproduces exactly.

Entry through the failed device-resume pass

Aug 05 00:56:46 host systemd-sleep[2892]: User sessions remain unfrozen on explicit request ($SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=0).
Aug 05 00:56:46 host systemd-sleep[2892]: This is not recommended, and might result in unexpected behavior, particularly
Aug 05 00:56:46 host systemd-sleep[2892]: in suspend-then-hibernate operations or setups with encrypted home directories.
Aug 05 00:56:46 host systemd-sleep[2892]: Performing sleep operation 'hibernate'...
Aug 05 00:56:46 host kernel: PM: hibernation: hibernation entry
Aug 05 00:56:46 host kernel: Loading firmware: nvidia/610.57.04/gsp_ga10x.bin
Aug 05 00:58:03 host kernel: Freezing user space processes
Aug 05 00:58:03 host kernel: Freezing user space processes completed (elapsed 0.001 seconds)
Aug 05 00:58:03 host kernel: OOM killer disabled.
Aug 05 00:58:03 host kernel: PM: hibernation: Basic memory bitmaps created
Aug 05 00:58:03 host kernel: PM: hibernation: Preallocating image memory
Aug 05 00:58:03 host kernel: PM: hibernation: Allocated 1629237 pages for snapshot
Aug 05 00:58:03 host kernel: PM: hibernation: Allocated 6516948 kbytes in 1.94 seconds (3359.25 MB/s)
Aug 05 00:58:03 host kernel: Freezing remaining freezable tasks
Aug 05 00:58:03 host kernel: Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
Aug 05 00:58:03 host kernel: ACPI: PM: Preparing to enter system sleep state S4
Aug 05 00:58:03 host kernel: ACPI: PM: Saving platform NVS memory
Aug 05 00:58:03 host kernel: PM: hibernation: Creating image:
Aug 05 00:58:03 host kernel: PM: hibernation: Need to copy 1542694 pages
Aug 05 00:58:03 host kernel: PM: hibernation: Normal pages needed: 1542694 + 1024, available pages: 15209330
Aug 05 00:58:03 host kernel: ACPI: PM: Restoring platform NVS memory
Aug 05 00:58:03 host kernel: ACPI: PM: Waking up from system sleep state S4
Aug 05 00:58:03 host kernel: PM: hibernation: Basic memory bitmaps freed
Aug 05 00:58:03 host kernel: OOM killer enabled.
Aug 05 00:58:03 host kernel: efivarfs: resyncing variable state
Aug 05 00:58:03 host kernel: efivarfs: removing variable HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67
Aug 05 00:58:03 host kernel: efivarfs: finished resyncing variable state
Aug 05 00:58:08 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010026; hClass=0x0000ca7e; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 05 00:58:08 host kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Aug 05 00:58:08 host kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine window channels
Aug 05 00:58:16 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010021; hClass=0x0000ca7b; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 05 00:58:16 host kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Aug 05 00:58:16 host kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine window channels
Aug 05 00:58:24 host kernel: note: systemd-sleep[2892] exited with irqs disabled

The oops

Aug 05 00:58:08 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010026; hClass=0x0000ca7e; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 05 00:58:08 host kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Aug 05 00:58:08 host kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine window channels
Aug 05 00:58:16 host kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d00001; hParent=0x00010012; hObject=0x00010021; hClass=0x0000ca7b; paramsSize=0x00000028; paramsStatus=0x00000065; status=0x00000065
Aug 05 00:58:16 host kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Aug 05 00:58:16 host kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine window channels
Aug 05 00:58:24 host kernel: BUG: unable to handle page fault for address: 00000000000025f0
Aug 05 00:58:24 host kernel: #PF: supervisor read access in kernel mode
Aug 05 00:58:24 host kernel: #PF: error_code(0x0000) - not-present page
Aug 05 00:58:24 host kernel: PGD 0 P4D 0 
Aug 05 00:58:24 host kernel: Oops: Oops: 0000 [#1] SMP NOPTI
Aug 05 00:58:24 host kernel: CPU: 1 UID: 0 PID: 2892 Comm: systemd-sleep Tainted: G           OE       6.18.41-gentoo-dist-bin #1 PREEMPT(lazy) 
Aug 05 00:58:24 host kernel: Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Aug 05 00:58:24 host kernel: Hardware name: ASUS System Product Name/ROG STRIX B550-F GAMING WIFI II, BIOS 3641 06/23/2026
Aug 05 00:58:24 host kernel: RIP: 0010:nvEvoDisableVblankSemControl+0x34/0xd0 [nvidia_modeset]
Aug 05 00:58:24 host kernel: Code: 49 89 ff 41 54 55 53 49 89 f4 48 83 ec 08 8b 06 48 8d 5e 10 4c 8b b4 c7 c8 02 00 00 49 8d ae 18 26 00 00 4d 8d ae 18 27 00 00 <48> 8b 55 d8 48 8b 43 20 8b 08 48 89 50 10 f0 48 83 0c 24 00 89 48
Aug 05 00:58:24 host kernel: RSP: 0018:ffffd2234146b938 EFLAGS: 00010292
Aug 05 00:58:24 host kernel: RAX: 0000000000000000 RBX: ffff8de841ab1758 RCX: ffff8de841ab1748
Aug 05 00:58:24 host kernel: RDX: 0000000000000000 RSI: ffff8de841ab1748 RDI: ffffd22341c9d008
Aug 05 00:58:24 host kernel: RBP: 0000000000002618 R08: ffff8de8bef361f0 R09: ffffd2234146b984
Aug 05 00:58:24 host kernel: R10: ffff8de857c3d000 R11: ffff8de84004a800 R12: ffff8de841ab1748
Aug 05 00:58:24 host kernel: R13: 0000000000002718 R14: 0000000000000000 R15: ffffd22341c9d008
Aug 05 00:58:24 host kernel: FS:  00007ff4805a6840(0000) GS:ffff8df78c149000(0000) knlGS:0000000000000000
Aug 05 00:58:24 host kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 05 00:58:24 host kernel: CR2: 00000000000025f0 CR3: 0000000170360000 CR4: 0000000000f50ef0
Aug 05 00:58:24 host kernel: PKRU: 55555554
Aug 05 00:58:24 host kernel: Call Trace:
Aug 05 00:58:24 host kernel:  <TASK>
Aug 05 00:58:24 host kernel:  nvFreePerOpenDev+0x13d/0x290 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  nvRevokeDevice+0x94/0x1e0 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  nvFreePerOpenDev+0x229/0x290 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  nvRevokeDevice+0x94/0x1e0 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  nvResumeDevEvo+0x1f4/0x2b0 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? up+0x53/0xa0
Aug 05 00:58:24 host kernel:  nvKmsResume+0x5c/0xd0 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  ? down+0x1e/0x60
Aug 05 00:58:24 host kernel:  nvKmsKapiUI32ToF32+0x7f/0x830 [nvidia_modeset]
Aug 05 00:58:24 host kernel:  nvidia_isr_kthread_bh+0xeb1/0xfe0 [nvidia]
Aug 05 00:58:24 host kernel:  nvidia_isr_kthread_bh+0xf75/0xfe0 [nvidia]
Aug 05 00:58:24 host kernel:  notifier_call_chain+0x5d/0xd0
Aug 05 00:58:24 host kernel:  blocking_notifier_call_chain+0x42/0x70
Aug 05 00:58:24 host kernel:  hibernate.cold+0x3b/0x449
Aug 05 00:58:24 host kernel:  state_store+0x45/0x60
Aug 05 00:58:24 host kernel:  kernfs_fop_write_iter+0x14d/0x200
Aug 05 00:58:24 host kernel:  vfs_write+0x25d/0x480
Aug 05 00:58:24 host kernel:  ksys_write+0x73/0xf0
Aug 05 00:58:24 host kernel:  do_syscall_64+0x82/0x7d0
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? do_sys_openat2+0xa2/0xe0
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? __x64_sys_openat+0x61/0xa0
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? do_syscall_64+0xba/0x7d0
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Aug 05 00:58:24 host kernel:  ? irqentry_exit_to_user_mode+0x2c/0x210
Aug 05 00:58:24 host kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 05 00:58:24 host kernel: RIP: 0033:0x7ff47fea08fa
Aug 05 00:58:24 host kernel: Code: 08 03 00 00 59 5e 48 83 f8 fc 75 1e 83 e2 39 83 fa 08 75 16 e8 17 ff ff ff 0f 1f 80 00 00 00 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 83 c4 18 c3 90 f3 0f 1e fa 48 83 ec 10 ff 74 24 18 e8 5f ff ff
Aug 05 00:58:24 host kernel: RSP: 002b:00007ffd2587a010 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
Aug 05 00:58:24 host kernel: RAX: ffffffffffffffda RBX: 000055e0af604010 RCX: 00007ff47fea08fa
Aug 05 00:58:24 host kernel: RDX: 0000000000000005 RSI: 000055e0af60db10 RDI: 0000000000000005
Aug 05 00:58:24 host kernel: RBP: 0000000000000005 R08: 0000000000000000 R09: 0000000000000000
Aug 05 00:58:24 host kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000005
Aug 05 00:58:24 host kernel: R13: 000055e0af60db10 R14: 0000000000000000 R15: 00007ffd2587a1c0
Aug 05 00:58:24 host kernel:  </TASK>
Aug 05 00:58:24 host kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq des3_ede_x86_64 des_generic libdes md4 vfat fat mt7921e mt7921_common nvidia_drm(OE) snd_hda_codec_alc882 mt792x_lib snd_hda_codec_realtek_lib snd_hda_codec_nvhdmi snd_hda_codec_generic snd_hda_codec_hdmi mt76_connac_lib uvcvideo amd_atl intel_rapl_msr nvidia_modeset(OE) intel_rapl_common snd_hda_intel uvc mt76 ee1004 videobuf2_vmalloc nvidia(OE) snd_hda_codec videobuf2_memops snd_usb_audio btusb videobuf2_v4l2 snd_hda_core btmtk mac80211 videobuf2_common snd_usbmidi_lib btrtl snd_ump snd_intel_dspcfg btbcm eeepc_wmi snd_rawmidi snd_intel_sdw_acpi videodev btintel asus_wmi snd_seq_device snd_hwdep drm_ttm_helper edac_mce_amd joydev bluetooth rapl snd_pcm mc ttm sparse_keymap cfg80211 platform_profile wmi_bmof drm_client_lib snd_timer drm_kms_helper rfkill i2c_piix4 pcspkr libarc4 snd video igc k10temp soundcore i2c_smbus gpio_amdpt gpio_generic binfmt_misc ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog xt_comment
Aug 05 00:58:24 host kernel:  xt_multiport xt_limit xt_addrtype xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables tun fuse nfnetlink dm_crypt uas usb_storage polyval_clmulni ghash_clmulni_intel sp5100_tco wmi pkcs8_key_parser
Aug 05 00:58:24 host kernel: CR2: 00000000000025f0
Aug 05 00:58:24 host kernel: ---[ end trace 0000000000000000 ]---

What changed between 595.84 and 610.57.04

Nothing that matters, and the way it differs is informative:

595.84     Code: … 49 8d ae d8 26 00 00 …   lea rbp,[r14+0x26d8]   RBP=0x26d8  R14=0x0  CR2=0x26b0
610.57.04  Code: … 49 8d ae 18 26 00 00 …   lea rbp,[r14+0x2618]   RBP=0x2618  R14=0x0  CR2=0x25f0

The faulting instruction is <48> 8b 55 d8 = mov rdx,[rbp-0x28] in both, and 0x2618 - 0x28 = 0x25f0 / 0x26d8 - 0x28 = 0x26b0 match CR2 exactly. The displacement moved by 0xC0 (192 bytes) because a struct grew between releases; R14 is NULL in both dumps. The call-site offsets shifted with recompilation (nvFreePerOpenDev+0x13d/0x240 → +0x13d/0x290, nvRevokeDevice+0x94/0x150 → +0x94/0x1e0, nvResumeDevEvo+0x1f7/0x200 → +0x1f4/0x2b0) but the sequence is identical.

The two GspRmAlloc timeouts precede the oops on both drivers, on the same objects (hClass=0x0000ca7e then hClass=0x0000ca7b), 8 seconds apart.

Run 4 — configuration A on driver 610.57.04: the -5 restore failure, unchanged

The last cell of the matrix. force_drivers restored, driver 610.57.04, nothing else changed. The write leg works again — the image is written and the machine powers off cleanly in about 60 seconds — and the restore leg fails exactly as it did on 595.84.

Aug 05 01:13:51 host systemd-hibernate-resume-generator[409]: Reported hibernation image: ID=gentoo VERSION_ID=2.18 kernel=6.18.41-gentoo-dist-bin UUID=<UUID-A> offset=533760
Aug 05 01:13:51 host kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.57.04  Release Build  (portage@localhost)  Wed Aug  5 12:04:50 AM CEST 2026
Aug 05 01:13:52 host kernel: Loading firmware: nvidia/610.57.04/gsp_ga10x.bin
Aug 05 01:14:06 host kernel: PM: Image signature found, resuming
Aug 05 01:14:06 host kernel: PM: hibernation: resume from hibernation
Aug 05 01:14:06 host kernel: Loading firmware: nvidia/610.57.04/gsp_ga10x.bin
Aug 05 01:14:09 host kernel: PM: hibernation: Basic memory bitmaps created
Aug 05 01:14:09 host kernel: PM: Using 3 thread(s) for lzo decompression
Aug 05 01:14:09 host kernel: PM: Loading and decompressing image data (1033512 pages)...
Aug 05 01:14:09 host kernel: PM: Image loading done
Aug 05 01:14:09 host kernel: PM: hibernation: Read 4134048 kbytes in 2.38 seconds (1736.99 MB/s)
Aug 05 01:14:09 host kernel: PM: Image successfully loaded
Aug 05 01:14:09 host kernel: NVRM: GPU 0000:08:00.0: PreserveVideoMemoryAllocations module parameter is set. System Power Management attempted without driver procfs suspend interface. Please refer to the 'Configuring Power Management Support' section in the driver README.
Aug 05 01:14:09 host kernel: nvidia 0000:08:00.0: PM: pci_pm_freeze(): nvidia_isr_kthread_bh [nvidia] returns -5
Aug 05 01:14:09 host kernel: nvidia 0000:08:00.0: PM: dpm_run_callback(): pci_pm_freeze returns -5
Aug 05 01:14:09 host kernel: nvidia 0000:08:00.0: PM: failed to quiesce async: error -5
Aug 05 01:14:09 host kernel: PM: hibernation: Failed to load image, recovering.
Aug 05 01:14:09 host kernel: PM: hibernation: Basic memory bitmaps freed
Aug 05 01:14:09 host kernel: PM: hibernation: resume failed (-5)
Aug 05 01:14:09 host kernel: BTRFS info (device dm-0): first mount of filesystem <UUID-A>
Aug 05 01:14:10 host systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67

Observations

  1. The faulting symbol is reached from two different call paths. #1226 reports nv_pm_notifier → hibernate+0x10d; this report has nvKmsResume → nvResumeDevEvo → nvRevokeDevice → nvFreePerOpenDev → nvEvoDisableVblankSemControl, from hibernate.cold+0x3b. Same symbol, same +0x34/0xd0 offset.

  2. The oops is preceded by two GSP RPC timeouts, 8 seconds apart, on hClass=0x0000ca7e then hClass=0x0000ca7b, identically on both driver versions. GspRmAlloc failed with status=0x65 (NV_ERR_TIMEOUT), then Display engine push buffer channel allocation failed, then Failed to allocate display engine window channels. The subsequent teardown appears to walk a device whose display engine channels were never allocated. If so, the bug may be an unchecked pointer in the failure path rather than in normal teardown — and the GSP timeout is the trigger rather than the defect.

  3. GSP firmware is reloaded during hibernation entry, immediately after PM: hibernation: hibernation entry and roughly 70 seconds before the first timeout.

  4. 57-77 seconds elapse between hibernation entry and the freeze pass in configuration B, which is presumably the video-memory save to NVreg_TemporaryFilePath (/var/tmp, on the encrypted btrfs root). The equivalent gap does not appear in configuration A.

  5. Neither configuration completes a round trip, and the two fail on opposite legs. With the modules in the initramfs the write leg works and the restore leg fails; without them the write leg fails. This is the same pattern reported independently at basecamp/omarchy#5554 for Blackwell + open 595, where the suggested mechanism is that early KMS denies the driver access to NVreg_TemporaryFilePath at restore time.

  6. Plain suspend-to-RAM (S3) works correctly in every configuration tested, verified the same evening: suspended at 19:15:38, resumed at 19:15:59, Wayland session and GPU intact. Only hibernation is affected.

  7. Nothing is corrupted by either failure. In configuration A the kernel takes the documented recovery path and continues as a cold boot. In configuration B systemd-hibernate.service is SIGKILLed after 78 seconds; the machine had to be power-cycled, and came back with clean filesystems.

  8. 610.57.04 does not fix it. The crash reproduces with byte-identical structure on the current feature branch. The fault address moves only because a struct grew by 192 bytes; the base pointer (R14) is NULL in both. See "What changed between 595.84 and 610.57.04" above.

What was ruled out

  • Storage / swapfile / resume_offset. Configuration A read a 22 GB image back through LUKS at 1765.88 MB/s and printed PM: Image successfully loaded. The resume target is correct.
  • resume= / resume_offset= on the kernel command line. Same — the image signature is found and the offset resolves.
  • systemd-hibernate.service / sleep configuration. systemd starts the operation correctly in both runs; the failure is in the kernel PM path.
  • Memory pressure. The runs had different working sets (5553706 / 1509603 / 1542694 pages) and each failed deterministically for its configuration.
  • Driver version. Reproduced on 595.84 (production branch) and 610.57.04 (feature branch, released 2026-08-03).
  • NVreg_PreserveVideoMemoryAllocations=0. Not tested, and deliberately so: it is required for a clean Wayland resume from plain suspend-to-RAM on this machine, which currently works. omarchy#5554 reports it does not fix Blackwell + open 595 anyway (their Test 2: Xid cascade during resume).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment