Skip to content

Instantly share code, notes, and snippets.

@kzall0c

kzall0c/PREEMPT_RT-perf-soft-lockup.log

Created July 25, 2025 00:00
Show Gist options
  • Save kzall0c/99349ae1e2bcd05c50f3732da2d3ade1 to your computer and use it in GitHub Desktop.
Save kzall0c/99349ae1e2bcd05c50f3732da2d3ade1 to your computer and use it in GitHub Desktop.
Download ZIP
https://lore.kernel.org/linux-rt-devel/[email protected]/ ``` $ git remote add rt-devel git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git $ git checkout v6.16-rc1-rt1 ``` Arm64 Server - RADXA Orion6 - Armv9.2 Architecture - Quad Big Cortex®-A720 Up to 2.6GHz - Quad Medium Cortex®-A720 Up to 2.4GHz - Quad Lit…
Raw
gistfile1.txt
watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [systemd-rfkill:7789]
CPU#2 Utilization every 4s during lockup:
#1: 98% system, 0% softirq, 1% hardirq, 0% idle
#2: 100% system, 0% softirq, 0% hardirq, 0% idle
#3: 100% system, 0% softirq, 0% hardirq, 0% idle
#4: 100% system, 0% softirq, 1% hardirq, 0% idle
#5: 100% system, 0% softirq, 0% hardirq, 0% idle
Modules linked in:
CPU: 2 UID: 0 PID: 7789 Comm: systemd-rfkill Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
Tainted: [W]=WARN
Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
pstate: 83400005 (Nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
pc : __arch_counter_get_cntvct arch/arm64/include/asm/arch_timer.h:208 [inline]
pc : arch_counter_get_cntvct+0xc/0x1c drivers/clocksource/arm_arch_timer.c:229
lr : arch_counter_read+0x1c/0x28 drivers/clocksource/arm_arch_timer.c:243
sp : ffff800098f872b0
x29: ffff800098f872b0 x28: 000000000000bc70 x27: ffff80008e3fc000
x26: dfff800000000000 x25: ffff80008e3fcb40 x24: dfff800000000000
x23: 1ffff00011c7f969 x22: ffff800090718a40 x21: ffff80008e3fcb48
x20: ffff0001a126f3e8 x19: 00000038cc6f2570 x18: ffff80009b69d059
x17: 6e6974696177202c x16: ffff80008a158744 x15: ffff700011c5999c
x14: 1ffff00011c5999c x13: 0000000000000004 x12: ffffffffffffffff
x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
x8 : ffff80008a158ddc x7 : 0000000000000001 x6 : 0000000000000001
x5 : ffff800098f868d8 x4 : ffff80008e2fcfa0 x3 : ffff8000804dc7dc
x2 : 0000000000000001 x1 : 000000012a05f200 x0 : 00000039a1ab950c
Call trace:
__arch_counter_get_cntvct arch/arm64/include/asm/arch_timer.h:204 [inline] (P)
arch_counter_get_cntvct+0xc/0x1c drivers/clocksource/arm_arch_timer.c:229 (P)
tk_clock_read kernel/time/timekeeping.c:251 [inline]
timekeeping_get_ns kernel/time/timekeeping.c:360 [inline]
__ktime_get_fast_ns kernel/time/timekeeping.c:408 [inline]
ktime_get_mono_fast_ns+0x9c/0x2d0 kernel/time/timekeeping.c:448
csd_lock_wait_toolong kernel/smp.c:249 [inline]
__csd_lock_wait+0xc4/0xa88 kernel/smp.c:326
csd_lock_wait kernel/smp.c:336 [inline]
smp_call_function_many_cond+0xa84/0xc54 kernel/smp.c:885
smp_call_function_many kernel/smp.c:909 [inline]
smp_call_function kernel/smp.c:931 [inline]
kick_all_cpus_sync+0x40/0xa0 kernel/smp.c:1076
flush_icache_range arch/arm64/include/asm/cacheflush.h:103 [inline]
__text_poke+0xd4/0xf4 arch/arm64/kernel/patching.c:130
aarch64_insn_copy+0x3c/0x50 arch/arm64/kernel/patching.c:161
bpf_arch_text_copy+0x34/0x54 arch/arm64/net/bpf_jit_comp.c:2099
bpf_jit_binary_pack_finalize+0x44/0xf0 kernel/bpf/core.c:1186
bpf_int_jit_compile+0x9ec/0xf5c arch/arm64/net/bpf_jit_comp.c:2042
bpf_prog_select_runtime+0x3d0/0x6b0 kernel/bpf/core.c:2499
bpf_migrate_filter net/core/filter.c:1315 [inline]
bpf_prepare_filter+0xd30/0xe84 net/core/filter.c:1363
__get_filter+0x210/0x318 net/core/filter.c:1532
sk_attach_filter+0x2c/0x140 net/core/filter.c:1547
sk_setsockopt+0x1944/0x25e4 net/core/sock.c:1470
sock_setsockopt+0x68/0x80 net/core/sock.c:1683
do_sock_setsockopt+0x19c/0x328 net/socket.c:2292
__sys_setsockopt net/socket.c:2321 [inline]
__do_sys_setsockopt net/socket.c:2327 [inline]
__se_sys_setsockopt net/socket.c:2324 [inline]
__arm64_sys_setsockopt+0x170/0x1e0 net/socket.c:2324
__invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
el0_svc+0x40/0x13c arch/arm64/kernel/entry-common.c:767
el0t_64_sync_handler+0x78/0x108 arch/arm64/kernel/entry-common.c:786
el0t_64_sync+0x1ac/0x1b0 arch/arm64/kernel/entry.S:600
[ 218.781092][ T42] perf: interrupt took too long (2573 > 2500), lowering kernel.perf_event_max_sample_rate to 77700
[ 218.781379][ T42] perf: interrupt took too long (3264 > 3216), lowering kernel.perf_event_max_sample_rate to 61200
[ 218.781676][ T42] perf: interrupt took too long (4301 > 4080), lowering kernel.perf_event_max_sample_rate to 46500
[ 218.782025][ T42] perf: interrupt took too long (5469 > 5376), lowering kernel.perf_event_max_sample_rate to 36500
[ 218.782529][ T42] perf: interrupt took too long (6870 > 6836), lowering kernel.perf_event_max_sample_rate to 29100
[ 218.783433][ T42] perf: interrupt took too long (8629 > 8587), lowering kernel.perf_event_max_sample_rate to 23100
[ 218.784048][ T42] perf: interrupt took too long (11433 > 10786), lowering kernel.perf_event_max_sample_rate to 17400
[ 219.149372][T15651] binder: 15648:15651 ioctl c06864ce 0 returned -22
[ 219.400143][T15678] binder: 15675:15678 ioctl c06864ce 0 returned -22
[ 220.902914][T15796] binder: 15788:15796 ioctl c06864ce 0 returned -22
[ 220.961173][T15804] binder: 15793:15804 ioctl c06864ce 0 returned -22
[ 225.961105][ T4020] smp: csd: Detected non-responsive CSD lock (#1) on CPU#1, waiting 5000000020 ns for CPU#00 do_nothing+0x0/0x1c(0x0).
[ 225.961174][ T4020] smp: csd: CSD lock (#1) unresponsive.
[ 225.961182][ T4020] Sending NMI from CPU 1 to CPUs 0:
[ 226.246360][ T7789] smp: csd: Detected non-responsive CSD lock (#2) on CPU#2, waiting 5000000080 ns for CPU#00 do_nothing+0x0/0x1c(0x0).
[ 226.246494][ T7789] smp: csd: CSD lock (#2) unresponsive.
[ 226.246517][ T7789] smp: csd: Re-sending CSD lock (#2) IPI from CPU#02 to CPU#00
[ 226.246585][ T7789] CPU: 2 UID: 0 PID: 7789 Comm: systemd-rfkill Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 226.246633][ T7789] Tainted: [W]=WARN
[ 226.246648][ T7789] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 226.246669][ T7789] Call trace:
[ 226.246684][ T7789] show_stack+0x2c/0x3c (C)
[ 226.246746][ T7789] __dump_stack+0x30/0x40
[ 226.246839][ T7789] dump_stack_lvl+0x148/0x1d8
[ 226.246900][ T7789] dump_stack+0x1c/0x3c
[ 226.246952][ T7789] __csd_lock_wait+0x648/0xa88
[ 226.246989][ T7789] smp_call_function_many_cond+0xa84/0xc54
[ 226.247026][ T7789] kick_all_cpus_sync+0x40/0xa0
[ 226.247062][ T7789] __text_poke+0xd4/0xf4
[ 226.247114][ T7789] aarch64_insn_copy+0x3c/0x50
[ 226.247164][ T7789] bpf_arch_text_copy+0x34/0x54
[ 226.247207][ T7789] bpf_jit_binary_pack_finalize+0x44/0xf0
[ 226.247257][ T7789] bpf_int_jit_compile+0x9ec/0xf5c
[ 226.247298][ T7789] bpf_prog_select_runtime+0x3d0/0x6b0
[ 226.247345][ T7789] bpf_prepare_filter+0xd30/0xe84
[ 226.247395][ T7789] __get_filter+0x210/0x318
[ 226.247440][ T7789] sk_attach_filter+0x2c/0x140
[ 226.247484][ T7789] sk_setsockopt+0x1944/0x25e4
[ 226.247528][ T7789] sock_setsockopt+0x68/0x80
[ 226.247569][ T7789] do_sock_setsockopt+0x19c/0x328
[ 226.247604][ T7789] __arm64_sys_setsockopt+0x170/0x1e0
[ 226.247639][ T7789] invoke_syscall+0x98/0x2b8
[ 226.247690][ T7789] el0_svc_common+0x130/0x23c
[ 226.247739][ T7789] do_el0_svc+0x48/0x58
[ 226.247786][ T7789] el0_svc+0x40/0x13c
[ 226.247871][ T7789] el0t_64_sync_handler+0x78/0x108
[ 226.247913][ T7789] el0t_64_sync+0x1ac/0x1b0
[ 238.016768][ T4020] smp: csd: Re-sending CSD lock (#1) IPI from CPU#01 to CPU#00
[ 238.016810][ T4020] CPU: 1 UID: 0 PID: 4020 Comm: kworker/u16:18 Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 238.016823][ T4020] Tainted: [W]=WARN
[ 238.016827][ T4020] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 238.016833][ T4020] Workqueue: events_unbound toggle_allocation_gate
[ 238.016865][ T4020] Call trace:
[ 238.016869][ T4020] show_stack+0x2c/0x3c (C)
[ 238.016895][ T4020] __dump_stack+0x30/0x40
[ 238.016916][ T4020] dump_stack_lvl+0x148/0x1d8
[ 238.016926][ T4020] dump_stack+0x1c/0x3c
[ 238.016936][ T4020] __csd_lock_wait+0x648/0xa88
[ 238.016946][ T4020] smp_call_function_many_cond+0xa84/0xc54
[ 238.016954][ T4020] kick_all_cpus_sync+0x40/0xa0
[ 238.016961][ T4020] arch_jump_label_transform_apply+0x14/0x20
[ 238.016972][ T4020] __jump_label_update+0x30c/0x334
[ 238.016988][ T4020] jump_label_update+0x2c8/0x364
[ 238.016996][ T4020] static_key_enable_cpuslocked+0x118/0x21c
[ 238.017004][ T4020] static_key_enable+0x24/0x38
[ 238.017012][ T4020] toggle_allocation_gate+0xbc/0x240
[ 238.017022][ T4020] process_scheduled_works+0x678/0xd18
[ 238.017045][ T4020] worker_thread+0x834/0xc00
[ 238.017053][ T4020] kthread+0x5e8/0x734
[ 238.017070][ T4020] ret_from_fork+0x10/0x20
[ 240.030454][ C3] watchdog: CPU3: Watchdog detected hard LOCKUP on cpu 0
[ 240.030475][ C3] Modules linked in:
[ 240.030484][ C3] Sending NMI from CPU 3 to CPUs 0:
[ 244.030451][ C2] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [systemd-rfkill:7789]
[ 244.030477][ C2] CPU#2 Utilization every 4s during lockup:
[ 244.030482][ C2] #1: 98% system, 0% softirq, 1% hardirq, 0% idle
[ 244.030489][ C2] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 244.030497][ C2] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 244.030503][ C2] #4: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 244.030509][ C2] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 244.030516][ C2] Modules linked in:
[ 244.030531][ C2] CPU: 2 UID: 0 PID: 7789 Comm: systemd-rfkill Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 244.030544][ C2] Tainted: [W]=WARN
[ 244.030547][ C2] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 244.030553][ C2] pstate: 83400005 (Nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[ 244.030561][ C2] pc : arch_counter_get_cntvct+0xc/0x1c
[ 244.030615][ C2] lr : arch_counter_read+0x1c/0x28
[ 244.030637][ C2] sp : ffff800098f872b0
[ 244.030640][ C2] x29: ffff800098f872b0 x28: 000000000000bc70 x27: ffff80008e3fc000
[ 244.030654][ C2] x26: dfff800000000000 x25: ffff80008e3fcb40 x24: dfff800000000000
[ 244.030666][ C2] x23: 1ffff00011c7f969 x22: ffff800090718a40 x21: ffff80008e3fcb48
[ 244.030677][ C2] x20: ffff0001a126f3e8 x19: 00000038cc6f2570 x18: ffff80009b69d059
[ 244.030689][ C2] x17: 6e6974696177202c x16: ffff80008a158744 x15: ffff700011c5999c
[ 244.030700][ C2] x14: 1ffff00011c5999c x13: 0000000000000004 x12: ffffffffffffffff
[ 244.030711][ C2] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 244.030721][ C2] x8 : ffff80008a158ddc x7 : 0000000000000001 x6 : 0000000000000001
[ 244.030732][ C2] x5 : ffff800098f868d8 x4 : ffff80008e2fcfa0 x3 : ffff8000804dc7dc
[ 244.030743][ C2] x2 : 0000000000000001 x1 : 000000012a05f200 x0 : 00000039a1ab950c
[ 244.030754][ C2] Call trace:
[ 244.030757][ C2] arch_counter_get_cntvct+0xc/0x1c (P)
[ 244.030768][ C2] ktime_get_mono_fast_ns+0x9c/0x2d0
[ 244.030782][ C2] __csd_lock_wait+0xc4/0xa88
[ 244.030791][ C2] smp_call_function_many_cond+0xa84/0xc54
[ 244.030799][ C2] kick_all_cpus_sync+0x40/0xa0
[ 244.030806][ C2] __text_poke+0xd4/0xf4
[ 244.030829][ C2] aarch64_insn_copy+0x3c/0x50
[ 244.030837][ C2] bpf_arch_text_copy+0x34/0x54
[ 244.030851][ C2] bpf_jit_binary_pack_finalize+0x44/0xf0
[ 244.030867][ C2] bpf_int_jit_compile+0x9ec/0xf5c
[ 244.030875][ C2] bpf_prog_select_runtime+0x3d0/0x6b0
[ 244.030884][ C2] bpf_prepare_filter+0xd30/0xe84
[ 244.030901][ C2] __get_filter+0x210/0x318
[ 244.030911][ C2] sk_attach_filter+0x2c/0x140
[ 244.030919][ C2] sk_setsockopt+0x1944/0x25e4
[ 244.030934][ C2] sock_setsockopt+0x68/0x80
[ 244.030942][ C2] do_sock_setsockopt+0x19c/0x328
[ 244.030950][ C2] __arm64_sys_setsockopt+0x170/0x1e0
[ 244.030957][ C2] invoke_syscall+0x98/0x2b8
[ 244.030967][ C2] el0_svc_common+0x130/0x23c
[ 244.030977][ C2] do_el0_svc+0x48/0x58
[ 244.030986][ C2] el0_svc+0x40/0x13c
[ 244.030994][ C2] el0t_64_sync_handler+0x78/0x108
[ 244.031001][ C2] el0t_64_sync+0x1ac/0x1b0
[ 244.031016][ C2] Kernel panic - not syncing: softlockup: hung tasks
[ 244.164780][ C2] CPU: 2 UID: 0 PID: 7789 Comm: systemd-rfkill Tainted: G W L 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 244.168401][ C2] Tainted: [W]=WARN, [L]=SOFTLOCKUP
[ 244.168927][ C2] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 244.170129][ C2] Call trace:
[ 244.178932][ C2] show_stack+0x2c/0x3c (C)
[ 244.179411][ C2] __dump_stack+0x30/0x40
[ 244.179859][ C2] dump_stack_lvl+0x34/0x1d8
[ 244.180307][ C2] dump_stack+0x1c/0x3c
[ 244.190727][ C2] panic+0x2a0/0x81c
[ 244.191147][ C2] watchdog_timer_fn+0xccc/0xd0c
[ 244.197778][ C2] __hrtimer_run_queues+0x2a8/0x70c
[ 244.198337][ C2] hrtimer_interrupt+0x2bc/0xb5c
[ 244.198841][ C2] arch_timer_handler_virt+0x74/0x88
[ 244.199379][ C2] handle_percpu_devid_irq+0x168/0x2ec
[ 244.199937][ C2] generic_handle_domain_irq+0xe0/0x140
[ 244.200480][ C2] gic_handle_irq+0x4ec/0x930
[ 244.215071][ C2] call_on_irq_stack+0x24/0x30
[ 244.215591][ C2] do_interrupt_handler+0xd4/0x138
[ 244.216102][ C2] el1_interrupt+0x44/0x90
[ 244.216542][ C2] el1h_64_irq_handler+0x18/0x24
[ 244.217032][ C2] el1h_64_irq+0x80/0x84
[ 244.217451][ C2] arch_counter_get_cntvct+0xc/0x1c (P)
[ 244.229257][ C2] ktime_get_mono_fast_ns+0x9c/0x2d0
[ 244.229826][ C2] __csd_lock_wait+0xc4/0xa88
[ 244.230286][ C2] smp_call_function_many_cond+0xa84/0xc54
[ 244.230866][ C2] kick_all_cpus_sync+0x40/0xa0
[ 244.231342][ C2] __text_poke+0xd4/0xf4
[ 244.241889][ C2] aarch64_insn_copy+0x3c/0x50
[ 244.242400][ C2] bpf_arch_text_copy+0x34/0x54
[ 244.242902][ C2] bpf_jit_binary_pack_finalize+0x44/0xf0
[ 244.243473][ C2] bpf_int_jit_compile+0x9ec/0xf5c
[ 244.243984][ C2] bpf_prog_select_runtime+0x3d0/0x6b0
[ 244.244519][ C2] bpf_prepare_filter+0xd30/0xe84
[ 244.248412][ C2] __get_filter+0x210/0x318
[ 244.249327][ C2] sk_attach_filter+0x2c/0x140
[ 244.249822][ C2] sk_setsockopt+0x1944/0x25e4
[ 244.250290][ C2] sock_setsockopt+0x68/0x80
[ 244.253421][ C2] do_sock_setsockopt+0x19c/0x328
[ 244.254802][ C2] __arm64_sys_setsockopt+0x170/0x1e0
[ 244.255354][ C2] invoke_syscall+0x98/0x2b8
[ 244.255841][ C2] el0_svc_common+0x130/0x23c
[ 244.256312][ C2] do_el0_svc+0x48/0x58
[ 244.256776][ C2] el0_svc+0x40/0x13c
[ 244.257178][ C2] el0t_64_sync_handler+0x78/0x108
[ 244.257691][ C2] el0t_64_sync+0x1ac/0x1b0
[ 244.258144][ C2] SMP: stopping secondary CPUs
[ 246.409795][ C2] SMP: failed to stop secondary CPUs 0,3
[ 246.412376][ C2] Kernel Offset: disabled
[ 246.413592][ C2] CPU features: 0x2000,000003c0,02934cb1,357ff667
[ 246.414245][ C2] Memory Limit: none
[ 252.885517][ C2] Rebooting in 86400 seconds..
Raw
PREEMPT_RT-perf-soft-lockup.log
perf lockup
watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [syz.1.915:10188]
CPU#2 Utilization every 4s during lockup:
#1: 100% system, 0% softirq, 0% hardirq, 0% idle
#2: 100% system, 0% softirq, 0% hardirq, 0% idle
#3: 100% system, 0% softirq, 1% hardirq, 0% idle
#4: 100% system, 0% softirq, 0% hardirq, 0% idle
#5: 100% system, 0% softirq, 0% hardirq, 0% idle
Modules linked in:
CPU: 2 UID: 0 PID: 10188 Comm: syz.1.915 Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
Tainted: [W]=WARN
Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
pstate: 83400005 (Nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
pc : tk_clock_read kernel/time/timekeeping.c:251 [inline]
pc : timekeeping_get_ns kernel/time/timekeeping.c:360 [inline]
pc : __ktime_get_fast_ns kernel/time/timekeeping.c:408 [inline]
pc : ktime_get_mono_fast_ns+0x84/0x2d0 kernel/time/timekeeping.c:448
lr : ktime_get_mono_fast_ns+0x2c/0x2d0 kernel/time/timekeeping.c:447
sp : ffff80009d9d77a0
x29: ffff80009d9d77a0 x28: 000000000000701e x27: ffff80008e3fc000
x26: dfff800000000000 x25: ffff80008e3fcb40 x24: dfff800000000000
x23: 1ffff00011c7f969 x22: ffff800090718a40 x21: ffff80008e3fcb48
x20: ffff80009d9d7948 x19: 000000216e795376 x18: 0000000000000000
x17: 0000000000000000 x16: ffff80008a158744 x15: ffff700011c5999c
x14: 1ffff00011c5999c x13: 0000000000000004 x12: ffffffffffffffff
x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000038
x8 : 0000000000000000 x7 : 0000000000000001 x6 : 0000000000000001
x5 : ffff80009d9d6db8 x4 : ffff80008e2fcfa0 x3 : ffff8000804dc7dc
x2 : 0000000000000001 x1 : 000000012a05f200 x0 : ffff80008e3fcb70
Call trace:
tk_clock_read kernel/time/timekeeping.c:251 [inline] (P)
timekeeping_get_ns kernel/time/timekeeping.c:360 [inline] (P)
__ktime_get_fast_ns kernel/time/timekeeping.c:408 [inline] (P)
ktime_get_mono_fast_ns+0x84/0x2d0 kernel/time/timekeeping.c:448 (P)
csd_lock_wait_toolong kernel/smp.c:249 [inline]
__csd_lock_wait+0xc4/0xa88 kernel/smp.c:326
csd_lock_wait kernel/smp.c:336 [inline]
smp_call_function_single+0x58c/0x5ac kernel/smp.c:679
task_function_call kernel/events/core.c:121 [inline]
event_function_call+0x210/0x7f4 kernel/events/core.c:328
perf_remove_from_context+0xa4/0x150 kernel/events/core.c:2562
perf_event_release_kernel+0x120/0x384 kernel/events/core.c:5777
perf_release+0x40/0x54 kernel/events/core.c:5852
__fput+0x340/0x75c fs/file_table.c:465
____fput+0x20/0x58 fs/file_table.c:493
task_work_run+0x1dc/0x260 kernel/task_work.c:227
resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
do_notify_resume+0x160/0x244 arch/arm64/kernel/entry-common.c:151
exit_to_user_mode_prepare arch/arm64/kernel/entry-common.c:169 [inline]
exit_to_user_mode arch/arm64/kernel/entry-common.c:178 [inline]
el0_svc+0x74/0x13c arch/arm64/kernel/entry-common.c:768
el0t_64_sync_handler+0x78/0x108 arch/arm64/kernel/entry-common.c:786
el0t_64_sync+0x1ac/0x1b0 arch/arm64/kernel/entry.S:600
2900
[ 120.026312][ T23] perf: interrupt took too long (85819 > 85810), lowering kernel.perf_event_max_sample_rate to 2300
[ 120.048478][ T23] perf: interrupt took too long (107364 > 107273), lowering kernel.perf_event_max_sample_rate to 1800
[ 120.078812][ T23] perf: interrupt took too long (134981 > 134205), lowering kernel.perf_event_max_sample_rate to 1400
[ 120.125049][ C3] vkms_vblank_simulate: vblank timer overrun
[ 125.722441][ T12] smp: csd: Detected non-responsive CSD lock (#1) on CPU#3, waiting 5000000030 ns for CPU#01 do_nothing+0x0/0x1c(0x0).
[ 125.722510][ T12] smp: csd: CSD lock (#1) unresponsive.
[ 125.722518][ T12] Sending NMI from CPU 3 to CPUs 1:
[ 126.517828][T10188] smp: csd: Detected non-responsive CSD lock (#2) on CPU#2, waiting 5000000050 ns for CPU#01 remote_function+0x0/0x180(0xffff80009d9d7a80).
[ 126.517887][T10188] smp: csd: CSD lock (#2) unresponsive.
[ 126.517892][T10188] smp: csd: Re-sending CSD lock (#2) IPI from CPU#02 to CPU#01
[ 126.517916][T10188] CPU: 2 UID: 0 PID: 10188 Comm: syz.1.915 Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 126.517929][T10188] Tainted: [W]=WARN
[ 126.517932][T10188] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 126.517938][T10188] Call trace:
[ 126.517941][T10188] show_stack+0x2c/0x3c (C)
[ 126.517967][T10188] __dump_stack+0x30/0x40
[ 126.518010][T10188] dump_stack_lvl+0x148/0x1d8
[ 126.518021][T10188] dump_stack+0x1c/0x3c
[ 126.518031][T10188] __csd_lock_wait+0x648/0xa88
[ 126.518041][T10188] smp_call_function_single+0x58c/0x5ac
[ 126.518049][T10188] event_function_call+0x210/0x7f4
[ 126.518059][T10188] perf_remove_from_context+0xa4/0x150
[ 126.518067][T10188] perf_event_release_kernel+0x120/0x384
[ 126.518074][T10188] perf_release+0x40/0x54
[ 126.518088][T10188] __fput+0x340/0x75c
[ 126.518106][T10188] ____fput+0x20/0x58
[ 126.518115][T10188] task_work_run+0x1dc/0x260
[ 126.518129][T10188] do_notify_resume+0x160/0x244
[ 126.518137][T10188] el0_svc+0x74/0x13c
[ 126.518149][T10188] el0t_64_sync_handler+0x78/0x108
[ 126.518157][T10188] el0t_64_sync+0x1ac/0x1b0
[ 137.166815][ T12] smp: csd: Re-sending CSD lock (#1) IPI from CPU#03 to CPU#01
[ 137.166924][ T12] CPU: 3 UID: 0 PID: 12 Comm: kworker/u16:0 Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 137.167015][ T12] Tainted: [W]=WARN
[ 137.167031][ T12] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 137.167054][ T12] Workqueue: events_unbound toggle_allocation_gate
[ 137.167135][ T12] Call trace:
[ 137.167153][ T12] show_stack+0x2c/0x3c (C)
[ 137.167219][ T12] __dump_stack+0x30/0x40
[ 137.167276][ T12] dump_stack_lvl+0x148/0x1d8
[ 137.167330][ T12] dump_stack+0x1c/0x3c
[ 137.167382][ T12] __csd_lock_wait+0x648/0xa88
[ 137.167423][ T12] smp_call_function_many_cond+0xa84/0xc54
[ 137.167463][ T12] kick_all_cpus_sync+0x40/0xa0
[ 137.167500][ T12] arch_jump_label_transform_apply+0x14/0x20
[ 137.167545][ T12] __jump_label_update+0x30c/0x334
[ 137.167594][ T12] jump_label_update+0x2c8/0x364
[ 137.167638][ T12] static_key_enable_cpuslocked+0x118/0x21c
[ 137.167685][ T12] static_key_enable+0x24/0x38
[ 137.167728][ T12] toggle_allocation_gate+0xbc/0x240
[ 137.167779][ T12] process_scheduled_works+0x678/0xd18
[ 137.167838][ T12] worker_thread+0x834/0xc00
[ 137.167874][ T12] kthread+0x5e8/0x734
[ 137.167921][ T12] ret_from_fork+0x10/0x20
[ 144.022284][ C0] watchdog: CPU0: Watchdog detected hard LOCKUP on cpu 1
[ 144.022350][ C0] Modules linked in:
[ 144.022386][ C0] Sending NMI from CPU 0 to CPUs 1:
[ 144.032161][ C2] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [syz.1.915:10188]
[ 144.032180][ C2] CPU#2 Utilization every 4s during lockup:
[ 144.032185][ C2] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 144.032192][ C2] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 144.032198][ C2] #3: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 144.032203][ C2] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 144.032209][ C2] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 144.032215][ C2] Modules linked in:
[ 144.032229][ C2] CPU: 2 UID: 0 PID: 10188 Comm: syz.1.915 Tainted: G W 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 144.032240][ C2] Tainted: [W]=WARN
[ 144.032243][ C2] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 144.032248][ C2] pstate: 83400005 (Nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[ 144.032256][ C2] pc : ktime_get_mono_fast_ns+0x84/0x2d0
[ 144.032276][ C2] lr : ktime_get_mono_fast_ns+0x2c/0x2d0
[ 144.032282][ C2] sp : ffff80009d9d77a0
[ 144.032285][ C2] x29: ffff80009d9d77a0 x28: 000000000000701e x27: ffff80008e3fc000
[ 144.032297][ C2] x26: dfff800000000000 x25: ffff80008e3fcb40 x24: dfff800000000000
[ 144.032308][ C2] x23: 1ffff00011c7f969 x22: ffff800090718a40 x21: ffff80008e3fcb48
[ 144.032318][ C2] x20: ffff80009d9d7948 x19: 000000216e795376 x18: 0000000000000000
[ 144.032328][ C2] x17: 0000000000000000 x16: ffff80008a158744 x15: ffff700011c5999c
[ 144.032338][ C2] x14: 1ffff00011c5999c x13: 0000000000000004 x12: ffffffffffffffff
[ 144.032348][ C2] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000038
[ 144.032358][ C2] x8 : 0000000000000000 x7 : 0000000000000001 x6 : 0000000000000001
[ 144.032367][ C2] x5 : ffff80009d9d6db8 x4 : ffff80008e2fcfa0 x3 : ffff8000804dc7dc
[ 144.032377][ C2] x2 : 0000000000000001 x1 : 000000012a05f200 x0 : ffff80008e3fcb70
[ 144.032388][ C2] Call trace:
[ 144.032391][ C2] ktime_get_mono_fast_ns+0x84/0x2d0 (P)
[ 144.032401][ C2] __csd_lock_wait+0xc4/0xa88
[ 144.032410][ C2] smp_call_function_single+0x58c/0x5ac
[ 144.032417][ C2] event_function_call+0x210/0x7f4
[ 144.032434][ C2] perf_remove_from_context+0xa4/0x150
[ 144.032442][ C2] perf_event_release_kernel+0x120/0x384
[ 144.032449][ C2] perf_release+0x40/0x54
[ 144.032462][ C2] __fput+0x340/0x75c
[ 144.032479][ C2] ____fput+0x20/0x58
[ 144.032488][ C2] task_work_run+0x1dc/0x260
[ 144.032500][ C2] do_notify_resume+0x160/0x244
[ 144.032514][ C2] el0_svc+0x74/0x13c
[ 144.032525][ C2] el0t_64_sync_handler+0x78/0x108
[ 144.032532][ C2] el0t_64_sync+0x1ac/0x1b0
[ 144.032546][ C2] Kernel panic - not syncing: softlockup: hung tasks
[ 144.089905][ C2] CPU: 2 UID: 0 PID: 10188 Comm: syz.1.915 Tainted: G W L 6.16.0-rc1-rt1 #11 PREEMPT_RT
[ 144.091095][ C2] Tainted: [W]=WARN, [L]=SOFTLOCKUP
[ 144.091640][ C2] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8 05/13/2025
[ 144.093592][ C2] Call trace:
[ 144.093972][ C2] show_stack+0x2c/0x3c (C)
[ 144.094470][ C2] __dump_stack+0x30/0x40
[ 144.094971][ C2] dump_stack_lvl+0x34/0x1d8
[ 144.095461][ C2] dump_stack+0x1c/0x3c
[ 144.095925][ C2] panic+0x2a0/0x81c
[ 144.096358][ C2] watchdog_timer_fn+0xccc/0xd0c
[ 144.096915][ C2] __hrtimer_run_queues+0x2a8/0x70c
[ 144.097463][ C2] hrtimer_interrupt+0x2bc/0xb5c
[ 144.098067][ C2] arch_timer_handler_virt+0x74/0x88
[ 144.098586][ C2] handle_percpu_devid_irq+0x168/0x2ec
[ 144.099125][ C2] generic_handle_domain_irq+0xe0/0x140
[ 144.099642][ C2] gic_handle_irq+0x4ec/0x930
[ 144.100108][ C2] call_on_irq_stack+0x24/0x30
[ 144.100611][ C2] do_interrupt_handler+0xd4/0x138
[ 144.101173][ C2] el1_interrupt+0x44/0x90
[ 144.101601][ C2] el1h_64_irq_handler+0x18/0x24
[ 144.102162][ C2] el1h_64_irq+0x80/0x84
[ 144.102594][ C2] ktime_get_mono_fast_ns+0x84/0x2d0 (P)
[ 144.103190][ C2] __csd_lock_wait+0xc4/0xa88
[ 144.103654][ C2] smp_call_function_single+0x58c/0x5ac
[ 144.104255][ C2] event_function_call+0x210/0x7f4
[ 144.104744][ C2] perf_remove_from_context+0xa4/0x150
[ 144.105270][ C2] perf_event_release_kernel+0x120/0x384
[ 144.105802][ C2] perf_release+0x40/0x54
[ 144.106213][ C2] __fput+0x340/0x75c
[ 144.106607][ C2] ____fput+0x20/0x58
[ 144.106982][ C2] task_work_run+0x1dc/0x260
[ 144.107421][ C2] do_notify_resume+0x160/0x244
[ 144.107878][ C2] el0_svc+0x74/0x13c
[ 144.108264][ C2] el0t_64_sync_handler+0x78/0x108
[ 144.108767][ C2] el0t_64_sync+0x1ac/0x1b0
[ 144.109195][ C2] SMP: stopping secondary CPUs
[ 145.807031][ C2] SMP: failed to stop secondary CPUs 0-1
[ 145.808044][ C2] Kernel Offset: disabled
[ 145.808479][ C2] CPU features: 0x6000,000003c0,02934cb1,357ff667
[ 145.809139][ C2] Memory Limit: none
[ 150.343076][ C2] Rebooting in 86400 seconds..
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment