Created
August 5, 2013 19:12
-
-
Save troyready/6158563 to your computer and use it in GitHub Desktop.
storage1 lockup, 2013-08-01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Aug 1 17:19:34 storage1 kernel: [2263513.341374] BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341736] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341811] CPU 2 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341813] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341868] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341871] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341875] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341885] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341887] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341889] RDX: 0000000000000002 RSI: 000000000000004f RDI: 0000000000000380 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341891] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341893] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341895] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341898] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341900] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341902] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341906] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341909] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341910] Stack: | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341912] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341918] 00080f913951789d ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341923] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341927] Call Trace: | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341929] <IRQ> | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341932] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341941] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341948] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341952] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341956] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341961] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341963] <EOI> | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341965] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341973] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341976] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341980] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341994] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342005] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342015] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342049] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342069] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342075] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342079] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342085] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342089] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342093] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342097] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342101] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342105] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342108] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342109] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.274676] BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275076] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275137] CPU 2 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275138] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275172] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275174] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275176] RIP: 0010:[<ffffffff8133eef3>] [<ffffffff8133eef3>] __write_lock_failed+0x13/0x20 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275183] RSP: 0018:ffff880c2a451ab0 EFLAGS: 00000287 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275184] RAX: ffff880634001300 RBX: 0000000000000000 RCX: ffff880c2a451ea0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275186] RDX: 0000000000000e72 RSI: 0000000000000001 RDI: ffffffff81c41b40 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275187] RBP: ffff880c2a451ab0 R08: ffff8814ae65f340 R09: 0000000100400029 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275188] R10: 00000000ae65f801 R11: 0000000000000006 R12: 0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275189] R13: ffff880c3fc36e70 R14: ffff88186fffbd68 R15: ffffffff81688a4f | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275191] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275193] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275197] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275198] Stack: | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275199] ffff880c2a451ac0 ffffffff8169cf17 ffff880c2a451ae0 ffffffff8115e764 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275203] ffffc90c96a45000 ffff88182bf03840 ffff880c2a451b10 ffffffff8115e86d | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275206] ffff881561ad0200 ffffc90c96a86000 ffff88182bf03840 ffff88182bf03000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275209] Call Trace: | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275214] [<ffffffff8169cf17>] _raw_write_lock+0x17/0x20 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275218] [<ffffffff8115e764>] remove_vm_area+0x64/0xa0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275221] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275223] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275225] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275236] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275242] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275249] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275276] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275289] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275293] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275296] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275299] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275302] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275304] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275307] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275310] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275313] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275315] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275316] Code: 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 55 48 89 e5 f0 81 07 00 00 10 00 f3 90 81 3f 00 00 10 00 <75> f6 f0 81 2f 00 00 10 00 75 e6 5d c3 55 48 89 e5 f0 ff 07 f3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932386] INFO: rcu_sched self-detected stall on CPU { 2} (t=15000 jiffies) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932879] sending NMI to all CPUs: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932938] NMI backtrace for cpu 0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932943] CPU 0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932944] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933022] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933025] Pid: 0, comm: swapper/0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933030] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933039] RSP: 0018:ffffffff81c01e08 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933041] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933043] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933045] RBP: ffffffff81c01e68 R08: 00000000fffffffc R09: 000000000001b517 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933047] R10: 00000000000000d2 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933049] R13: ffff880c3fc1a070 R14: 0000000000000020 R15: 1316699177b54b0c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933052] FS: 0000000000000000(0000) GS:ffff880c3fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933054] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933056] CR2: 00007f484d4e4fd0 CR3: 0000000001c0b000 CR4: 00000000000007f0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933063] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c13460) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933064] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933066] 0000000000000000 0000000000000f6e 0000000000000000 00000000066e562a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933071] ffffffff81c01e78 0000000081535457 0000000000000000 ffff880c3fc1a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933076] 00000000fffffff0 0000000000000003 0000000000000000 000000000008c800 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933081] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933088] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933092] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933097] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933103] [<ffffffff8166080e>] rest_init+0x72/0x74 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933109] [<ffffffff81cf2c4d>] start_kernel+0x3cf/0x3dc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933113] [<ffffffff81cf27ff>] ? pass_bootoption.constprop.3+0xd3/0xd3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933118] [<ffffffff81cf2397>] x86_64_start_reservations+0x131/0x135 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933122] [<ffffffff81cf2120>] ? early_idt_handlers+0x120/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933126] [<ffffffff81cf2468>] x86_64_start_kernel+0xcd/0xdc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933128] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933167] NMI backtrace for cpu 1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933171] CPU 1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933173] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933233] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933236] Pid: 0, comm: swapper/1 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933240] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933247] RSP: 0018:ffff880c2d535e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933249] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933251] RDX: 0000000000000000 RSI: ffff880c2d535fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933253] RBP: ffff880c2d535e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933254] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933256] R13: ffff88186fc1a070 R14: 0000000000000020 R15: 13166991773d5ebb | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933258] FS: 0000000000000000(0000) GS:ffff88186fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933260] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933262] CR2: 00007f55c4e4e980 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933268] Process swapper/1 (pid: 0, threadinfo ffff880c2d534000, task ffff880c2d52ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933269] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933270] 0000000000000001 ffff88186fc0eb80 0000000000000000 00000000016a735e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933275] ffff880c2d535ea8 0000000181535457 0000000000000000 ffff88186fc1a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933279] 00000000fffffff0 0000000000000003 0000000000000001 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933283] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933289] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933293] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933298] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933304] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933305] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933338] NMI backtrace for cpu 2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933341] CPU 2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933342] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933402] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933404] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933408] RIP: 0010:[<ffffffff81035d71>] [<ffffffff81035d71>] default_send_IPI_mask_sequence_phys+0x71/0xe0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933416] RSP: 0018:ffff880c3fc23d38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933418] RAX: ffff88186fc20000 RBX: 0000000000000002 RCX: 0000000000000022 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933420] RDX: 0000000000000003 RSI: 0000000000000100 RDI: 0000000000000100 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933422] RBP: ffff880c3fc23d78 R08: ffffffff81cd5680 R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933424] R10: 0000000000000000 R11: 0000000000026810 R12: 000000000000dc82 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933426] R13: ffffffff81cd5680 R14: 0000000000000400 R15: 0000000000000092 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933428] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933431] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933433] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933435] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933439] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933441] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933442] 0000000000000002 0000000000000002 ffff880c3fc23d98 0000000000002710 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933447] ffffffff81c387c0 0000000000000002 0000000000000002 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933452] ffff880c3fc23d88 ffffffff8103a2e7 ffff880c3fc23da8 ffffffff81035f61 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933456] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933458] <IRQ> | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933460] [<ffffffff8103a2e7>] physflat_send_IPI_all+0x17/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933469] [<ffffffff81035f61>] arch_trigger_all_cpu_backtrace+0x61/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933476] [<ffffffff810e9b59>] print_cpu_stall+0x79/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933480] [<ffffffff810eb83a>] __rcu_pending+0xea/0x210 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933484] [<ffffffff810eb98c>] rcu_pending+0x2c/0x60 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933489] [<ffffffff810ec151>] rcu_check_callbacks+0xa1/0x160 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933495] [<ffffffff81065698>] update_process_times+0x48/0x90 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933500] [<ffffffff810ab914>] tick_sched_timer+0x64/0xc0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933505] [<ffffffff8107c238>] __run_hrtimer+0x78/0x1f0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933509] [<ffffffff810ab8b0>] ? tick_nohz_handler+0x100/0x100 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933514] [<ffffffff8105ba7d>] ? __do_softirq+0xfd/0x210 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933518] [<ffffffff8107cac7>] hrtimer_interrupt+0xf7/0x240 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933523] [<ffffffff816a6a5c>] ? call_softirq+0x1c/0x30 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933527] [<ffffffff816a73b9>] smp_apic_timer_interrupt+0x69/0x99 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933533] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933534] <EOI> | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933536] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933544] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933548] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933551] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933565] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933575] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933586] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933620] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933641] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933646] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933651] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933656] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933660] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933664] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933668] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933672] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933675] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933679] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933680] Code: 00 00 4c 89 ef 48 63 d2 e8 8d d2 30 00 89 c2 48 63 05 e8 0a ca 00 48 39 c2 73 41 48 8b 04 d5 60 49 cd 81 83 fb 02 41 0f b7 0c 04 <75> 07 eb 46 0f 1f 00 f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933779] NMI backtrace for cpu 3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933784] CPU 3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933786] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933847] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933850] Pid: 0, comm: swapper/3 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933854] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933860] RSP: 0018:ffff880c2d601e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933862] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933864] RDX: 0000000000000000 RSI: ffff880c2d601fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933867] RBP: ffff880c2d601e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933868] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933870] R13: ffff88186fc3a070 R14: 0000000000000020 R15: 13166991773d7296 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933873] FS: 0000000000000000(0000) GS:ffff88186fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933875] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933877] CR2: 00000000006e1dc8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933881] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933884] Process swapper/3 (pid: 0, threadinfo ffff880c2d600000, task ffff880c2d52dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933885] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933887] 0000000000000003 ffff88186fc2eb80 0000000000000000 00000000012d5695 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933892] ffff880c2d601ea8 0000000381535457 0000000000000000 ffff88186fc3a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933897] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933901] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933906] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933910] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933914] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933918] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933920] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933960] NMI backtrace for cpu 4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933963] CPU 4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933965] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934026] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934028] Pid: 0, comm: swapper/4 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934032] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934038] RSP: 0018:ffff880c2d603e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934040] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934042] RDX: 0000000000000000 RSI: ffff880c2d603fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934044] RBP: ffff880c2d603e98 R08: 00000000fffffffc R09: 0000000000005ce8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934046] R10: 0000000000001548 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934048] R13: ffff880c3fc5a070 R14: 0000000000000020 R15: 13166991773a9cd8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934051] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934053] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934055] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934062] Process swapper/4 (pid: 0, threadinfo ffff880c2d602000, task ffff880c2d608000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934063] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934065] 0000000000000082 ffff880c3fc4eb80 0000000000000000 00000000016aed8c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934070] ffff880c2d603ea8 0000000481535457 0000000000000000 ffff880c3fc5a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934075] 00000000fffffff0 0000000000000003 0000000000000004 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934079] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934084] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934088] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934092] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934097] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934099] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934139] NMI backtrace for cpu 5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934142] CPU 5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934144] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934205] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934208] Pid: 0, comm: swapper/5 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934211] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934218] RSP: 0018:ffff880c2d605e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934220] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934222] RDX: 0000000000000000 RSI: ffff880c2d605fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934224] RBP: ffff880c2d605e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934226] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934228] R13: ffff88186fc5a070 R14: 0000000000000020 R15: 131669917778d5a7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934230] FS: 0000000000000000(0000) GS:ffff88186fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934233] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934235] CR2: 00007ff33b9f5000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934239] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934241] Process swapper/5 (pid: 0, threadinfo ffff880c2d604000, task ffff880c2d609700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934243] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934244] 0000000000000005 ffff88186fc4eb80 0000000000000000 00000000016c0248 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934250] ffff880c2d605ea8 0000000581535457 0000000000000000 ffff88186fc5a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934254] 00000000fffffff0 0000000000000003 0000000000000005 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934259] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934264] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934268] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934272] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934276] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934278] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934317] NMI backtrace for cpu 6 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934320] CPU 6 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934322] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934374] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934377] Pid: 0, comm: swapper/6 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934380] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934385] RSP: 0018:ffff880c2d607e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934387] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934389] RDX: 0000000000000000 RSI: ffff880c2d607fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934391] RBP: ffff880c2d607e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934392] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934394] R13: ffff880c3fc7a070 R14: 0000000000000020 R15: 13166991773d1333 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934396] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934398] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934400] CR2: 00007f484b9517d8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934405] Process swapper/6 (pid: 0, threadinfo ffff880c2d606000, task ffff880c2d60ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934407] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934408] 0000000000000006 ffff880c3fc6eb80 0000000000000000 00000000016abb88 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934413] ffff880c2d607ea8 0000000681535457 0000000000000000 ffff880c3fc7a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934417] 00000000fffffff0 0000000000000003 0000000000000006 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934421] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934425] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934428] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934432] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934435] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934437] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934471] NMI backtrace for cpu 7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934475] CPU 7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934477] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934537] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934540] Pid: 0, comm: swapper/7 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934544] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934550] RSP: 0018:ffff880c2d611e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934552] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934554] RDX: 0000000000000000 RSI: ffff880c2d611fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934556] RBP: ffff880c2d611e98 R08: 00000000fffffffc R09: 0000000000005cdc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934558] R10: 0000000000000dc3 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934560] R13: ffff88186fc7a070 R14: 0000000000000020 R15: 13166991773ab18a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934563] FS: 0000000000000000(0000) GS:ffff88186fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934565] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934567] CR2: 00007f484b9517d8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934571] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934574] Process swapper/7 (pid: 0, threadinfo ffff880c2d610000, task ffff880c2d60c500) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934578] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934580] 0000000000000082 ffff88186fc6eb80 0000000000000000 00000000016abbcc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934584] ffff880c2d611ea8 0000000781535457 0000000000000000 ffff88186fc7a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934588] 00000000fffffff0 0000000000000003 0000000000000007 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934592] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934597] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934600] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934603] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934607] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934608] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934643] NMI backtrace for cpu 8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934646] CPU 8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934648] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934709] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934711] Pid: 0, comm: swapper/8 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934715] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934721] RSP: 0018:ffff880c2d615e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934723] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934725] RDX: 0000000000000000 RSI: ffff880c2d615fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934727] RBP: ffff880c2d615e98 R08: 00000000fffffffc R09: 000000000006a9e9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934729] R10: 000000000000002c R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934731] R13: ffff880c3fc9a070 R14: 0000000000000020 R15: 1316699177f91da1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934734] FS: 0000000000000000(0000) GS:ffff880c3fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934736] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934738] CR2: 00007f86145c6be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934742] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934745] Process swapper/8 (pid: 0, threadinfo ffff880c2d614000, task ffff880c2d60dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934746] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934748] 0000000000000000 000000000000118b 0000000000000000 000000001a07b9f1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934753] ffff880c2d615ea8 0000000881535457 0000000000000000 ffff880c3fc9a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934758] 00000000fffffff0 0000000000000003 0000000000000008 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934762] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934767] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934771] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934775] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934779] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934781] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934820] NMI backtrace for cpu 9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934824] CPU 9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934825] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934878] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934880] Pid: 0, comm: swapper/9 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934883] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934889] RSP: 0018:ffff880c2d617e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934890] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934892] RDX: 0000000000000000 RSI: ffff880c2d617fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934894] RBP: ffff880c2d617e98 R08: 00000000fffffffc R09: 0000000000005cc7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934896] R10: 0000000000001012 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934897] R13: ffff88186fc9a070 R14: 0000000000000020 R15: 1316699177fab29b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934900] FS: 0000000000000000(0000) GS:ffff88186fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934902] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934903] CR2: 00007f55c4de5c39 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934907] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934909] Process swapper/9 (pid: 0, threadinfo ffff880c2d616000, task ffff880c2d618000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934910] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934911] 0000000000000082 ffff88186fc8eb80 0000000000000000 00000000016a6ca1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934916] ffff880c2d617ea8 0000000981535457 0000000000000000 ffff88186fc9a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934920] 00000000fffffff0 0000000000000003 0000000000000009 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934924] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934928] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934932] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934935] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934939] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934940] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934974] NMI backtrace for cpu 10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934978] CPU 10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934980] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935059] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935062] Pid: 0, comm: swapper/10 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935067] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935073] RSP: 0018:ffff880c2d621e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935076] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935079] RDX: 0000000000000000 RSI: ffff880c2d621fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935082] RBP: ffff880c2d621e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935085] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935088] R13: ffff880c3fcba070 R14: 0000000000000020 R15: 13166991773d3013 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935091] FS: 0000000000000000(0000) GS:ffff880c3fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935097] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935103] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935106] Process swapper/10 (pid: 0, threadinfo ffff880c2d620000, task ffff880c2d619700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935108] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935110] 000000000000000a ffff880c3fcaeb80 0000000000000000 00000000016aa105 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935117] ffff880c2d621ea8 0000000a81535457 0000000000000000 ffff880c3fcba070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935124] 00000000fffffff0 0000000000000003 000000000000000a 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935130] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935136] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935141] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935146] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935152] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935154] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935208] NMI backtrace for cpu 11 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935211] CPU 11 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935212] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935265] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935267] Pid: 0, comm: swapper/11 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935271] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935276] RSP: 0018:ffff880c2d623e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935278] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935279] RDX: 0000000000000000 RSI: ffff880c2d623fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935281] RBP: ffff880c2d623e98 R08: 00000000fffffffc R09: 0000000000005cc2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935283] R10: 0000000000001366 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935285] R13: ffff88186fcba070 R14: 0000000000000020 R15: 131669917806059e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935287] FS: 0000000000000000(0000) GS:ffff88186fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935289] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935291] CR2: 00007f6964a633dd CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935294] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935296] Process swapper/11 (pid: 0, threadinfo ffff880c2d622000, task ffff880c2d61ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935297] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935299] 0000000000000082 ffff88186fcaeb80 0000000000000000 00000000016a5749 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935303] ffff880c2d623ea8 0000000b81535457 0000000000000000 ffff88186fcba070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935307] 00000000fffffff0 0000000000000003 000000000000000b 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935311] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935315] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935319] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935322] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935326] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935327] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935362] NMI backtrace for cpu 12 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935365] CPU 12 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935367] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935428] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935431] Pid: 0, comm: swapper/12 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935435] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935440] RSP: 0018:ffff880c2d625e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935443] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935445] RDX: 0000000000000000 RSI: ffff880c2d625fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935447] RBP: ffff880c2d625e98 R08: 00000000fffffffc R09: 0000000000005cd5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935449] R10: 00000000000012b2 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935451] R13: ffff880c3fcda070 R14: 0000000000000020 R15: 131669917807810e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935453] FS: 0000000000000000(0000) GS:ffff880c3fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935456] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935458] CR2: 0000000000f155b8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935464] Process swapper/12 (pid: 0, threadinfo ffff880c2d624000, task ffff880c2d61c500) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935466] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935467] 0000000000000082 ffff880c3fcceb80 0000000000000000 00000000016aa20a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935472] ffff880c2d625ea8 0000000c81535457 0000000000000000 ffff880c3fcda070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935477] 00000000fffffff0 0000000000000003 000000000000000c 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935481] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935486] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935490] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935494] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935498] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935500] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935540] NMI backtrace for cpu 13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935543] CPU 13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935545] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935606] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935609] Pid: 0, comm: swapper/13 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935613] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935619] RSP: 0018:ffff880c2d627e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935621] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935623] RDX: 0000000000000000 RSI: ffff880c2d627fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935625] RBP: ffff880c2d627e98 R08: 00000000fffffffc R09: 0000000000005cd1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935627] R10: 0000000000000cb2 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935629] R13: ffff88186fcda070 R14: 0000000000000020 R15: 13166991780909a4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935632] FS: 0000000000000000(0000) GS:ffff88186fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935634] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935636] CR2: 00007f55c4e4df50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935640] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935642] Process swapper/13 (pid: 0, threadinfo ffff880c2d626000, task ffff880c2d61dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935644] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935645] 0000000000000082 ffff88186fcceb80 0000000000000000 00000000016a92f7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935651] ffff880c2d627ea8 0000000d81535457 0000000000000000 ffff88186fcda070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935655] 00000000fffffff0 0000000000000003 000000000000000d 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935660] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935665] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935668] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935672] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935677] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935678] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935718] NMI backtrace for cpu 14 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935722] CPU 14 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935723] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935784] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935787] Pid: 0, comm: swapper/14 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935791] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935797] RSP: 0018:ffff880c2d639e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935799] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935801] RDX: 0000000000000000 RSI: ffff880c2d639fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935803] RBP: ffff880c2d639e98 R08: 00000000fffffffc R09: 000000000007e16c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935805] R10: 0000000000000046 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935807] R13: ffff880c3fcfa070 R14: 0000000000000020 R15: 13166991780a9b61 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935810] FS: 0000000000000000(0000) GS:ffff880c3fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935814] CR2: 00007f9dd6730be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935818] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935821] Process swapper/14 (pid: 0, threadinfo ffff880c2d638000, task ffff880c2d630000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935822] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935824] 0000000000000000 000000000000353e 0000000000000000 000000001ec891a8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935829] ffff880c2d639ea8 0000000e81535457 0000000000000000 ffff880c3fcfa070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935833] 00000000fffffff0 0000000000000003 000000000000000e 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935838] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935843] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935846] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935850] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935855] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935857] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935896] NMI backtrace for cpu 15 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935899] CPU 15 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935901] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935962] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935965] Pid: 0, comm: swapper/15 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935969] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935975] RSP: 0018:ffff880c2d63be38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935977] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935979] RDX: 0000000000000000 RSI: ffff880c2d63bfd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935981] RBP: ffff880c2d63be98 R08: 00000000fffffffc R09: 0000000000005ccf | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935983] R10: 0000000000000cb9 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935985] R13: ffff88186fcfa070 R14: 0000000000000020 R15: 13166991780c1a5f | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935988] FS: 0000000000000000(0000) GS:ffff88186fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935990] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935992] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935999] Process swapper/15 (pid: 0, threadinfo ffff880c2d63a000, task ffff880c2d631700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936000] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936002] 0000000000000082 ffff88186fceeb80 0000000000000000 00000000016a8a77 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936007] ffff880c2d63bea8 0000000f81535457 0000000000000000 ffff88186fcfa070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936012] 00000000fffffff0 0000000000000003 000000000000000f 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936016] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936021] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936025] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936029] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936033] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936035] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198526] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198893] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198962] CPU 2 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198964] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199019] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199022] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199026] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199035] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199037] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199039] RDX: 0000000000000002 RSI: 000000000000004b RDI: 0000000000000380 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199041] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199043] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199045] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199048] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199050] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199052] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199058] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199060] Stack: | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199062] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199067] 00080f9f3198d15c ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199072] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199076] Call Trace: | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199079] <IRQ> | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199081] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199090] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199097] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199100] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199105] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199110] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199111] <EOI> | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199113] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199121] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199125] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199129] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199143] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199153] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199164] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199198] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199219] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199224] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199229] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199234] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199238] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199242] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199246] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199250] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199254] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199257] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199259] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.131835] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132196] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132271] CPU 2 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132273] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132327] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132330] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132334] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132343] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132345] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132347] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132349] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132351] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132353] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132356] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132358] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132360] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132364] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132367] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132368] Stack: | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132370] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132376] 00080fa5b686e8a4 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132380] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132385] Call Trace: | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132387] <IRQ> | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132389] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132399] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132405] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132409] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132413] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132418] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132420] <EOI> | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132422] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132430] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132434] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132437] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132452] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132462] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132472] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132507] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132527] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132533] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132537] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132543] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132547] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132551] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132555] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132559] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132563] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132566] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132568] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065186] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065548] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065623] CPU 2 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065625] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065680] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065683] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065687] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065696] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065698] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065700] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065702] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065704] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065706] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065708] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065711] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065713] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065719] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065721] Stack: | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065723] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065728] 00080fac3b750090 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065733] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065738] Call Trace: | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065740] <IRQ> | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065742] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065752] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065758] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065762] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065766] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065771] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065773] <EOI> | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065775] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065783] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065787] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065790] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065805] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065815] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065825] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065859] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065880] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065886] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065890] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065895] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065899] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065903] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065908] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065911] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065915] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065918] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065920] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998497] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998879] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998954] CPU 2 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998956] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999011] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999014] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999018] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999027] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999029] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999031] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999033] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999035] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999037] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999039] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999042] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999044] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999050] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999052] Stack: | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999054] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999059] 00080fb2c06318a8 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999064] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999069] Call Trace: | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999071] <IRQ> | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999073] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999083] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999089] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999093] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999097] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999102] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999104] <EOI> | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999106] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999114] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999117] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999121] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999135] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999145] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999156] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999189] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999209] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999215] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999219] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999224] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999228] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999232] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999237] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999241] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999244] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999248] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999249] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650356] BUG: soft lockup - CPU#6 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650880] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650931] CPU 6 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650932] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650965] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650967] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650970] RIP: 0010:[<ffffffff8115e786>] [<ffffffff8115e786>] remove_vm_area+0x86/0xa0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650977] RSP: 0018:ffff880c2a451ad0 EFLAGS: 00000287 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650978] RAX: ffff880b6ff39080 RBX: ffff880ae3791340 RCX: ffff880c2a451ea0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650979] RDX: ffff880b6ff395c0 RSI: 0000000000000001 RDI: ffffffff81c41b40 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650980] RBP: ffff880c2a451ae0 R08: ffff880ae3791340 R09: 0000000100400004 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650982] R10: 00000000e3791e01 R11: 0000000000000001 R12: 00000000e3791e01 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650983] R13: 0000000000000001 R14: 0000000000000000 R15: ffff880c2a451ab0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650984] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650986] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650987] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650989] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650991] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650992] Stack: | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650993] ffffc90f2de10000 ffff88182bf03840 ffff880c2a451b10 ffffffff8115e86d | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650997] 0000000000000000 ffffc90f27db8000 ffff88182bf03840 ffff88182bf03000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650999] ffff880c2a451b20 ffffffff8115e92b ffff880c2a451b30 ffffffff8115ea5a | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651002] Call Trace: | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651006] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651008] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651010] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651021] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651027] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651034] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651062] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651074] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651080] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651083] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651086] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651088] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651091] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651094] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651097] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651100] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651102] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651102] Code: 40 1b c4 81 e8 9c e7 53 00 48 8b 05 8d 6d d9 00 48 c7 c2 f8 54 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 a4 33 ae 00 00 00 10 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.583730] BUG: soft lockup - CPU#6 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584091] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584165] CPU 6 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584167] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584222] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584225] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584229] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584238] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584241] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584243] RDX: 0000000000000002 RSI: 0000000000000070 RDI: 0000000000000380 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584245] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584247] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584249] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584251] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584253] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584255] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584260] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584262] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584264] Stack: | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584265] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584271] 00080fe065ff3588 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584276] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584280] Call Trace: | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584282] <IRQ> | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584285] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584294] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584301] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584305] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584309] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584314] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584316] <EOI> | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584318] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584326] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584329] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584333] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584347] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584357] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584368] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584402] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584423] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584429] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584433] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584438] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584443] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584447] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584451] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584455] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584459] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584462] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584463] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974032] INFO: rcu_sched self-detected stall on CPU { 6} (t=15000 jiffies) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974556] sending NMI to all CPUs: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974618] NMI backtrace for cpu 0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974622] CPU 0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974624] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974702] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974705] Pid: 0, comm: swapper/0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974709] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974718] RSP: 0018:ffffffff81c01e08 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974724] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974726] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974727] RBP: ffffffff81c01e68 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974729] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974731] R13: ffff880c3fc1a070 R14: 0000000000000020 R15: 131669da0fba3be4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974733] FS: 0000000000000000(0000) GS:ffff880c3fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974736] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007f0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974742] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c13460) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974743] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974745] ffffffff81c01e18 ffffffff8101be59 0000000000000000 000000000039c33a | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974750] ffffffff81c01e78 0000000081535457 0000000000000000 ffff880c3fc1a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974754] 00000000fffffff0 0000000000000003 0000000000000000 000000000008c800 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974758] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974764] [<ffffffff8101be59>] ? sched_clock+0x9/0x10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974769] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974773] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974776] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974782] [<ffffffff8166080e>] rest_init+0x72/0x74 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974787] [<ffffffff81cf2c4d>] start_kernel+0x3cf/0x3dc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974791] [<ffffffff81cf27ff>] ? pass_bootoption.constprop.3+0xd3/0xd3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974795] [<ffffffff81cf2397>] x86_64_start_reservations+0x131/0x135 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974798] [<ffffffff81cf2120>] ? early_idt_handlers+0x120/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974801] [<ffffffff81cf2468>] x86_64_start_kernel+0xcd/0xdc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974803] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974837] NMI backtrace for cpu 1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974841] CPU 1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974843] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974913] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974916] Pid: 0, comm: swapper/1 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974920] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974928] RSP: 0018:ffff880c2d535e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974931] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974933] RDX: 0000000000000000 RSI: ffff880c2d535fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974935] RBP: ffff880c2d535e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974937] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974939] R13: ffff88186fc1a070 R14: 0000000000000020 R15: 131669da0f02b06f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974941] FS: 0000000000000000(0000) GS:ffff88186fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974943] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974945] CR2: 00007fff884dc9a8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974950] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974952] Process swapper/1 (pid: 0, threadinfo ffff880c2d534000, task ffff880c2d52ae00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974953] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974955] 0000000000000001 ffff88186fc0eb80 0000000000000000 00000000016b6961 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974961] ffff880c2d535ea8 0000000181535457 0000000000000000 ffff88186fc1a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974965] 00000000fffffff0 0000000000000003 0000000000000001 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974970] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974977] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974980] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974986] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974992] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974994] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975034] NMI backtrace for cpu 2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975037] CPU 2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975038] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975090] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975093] Pid: 0, comm: swapper/2 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975096] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975101] RSP: 0018:ffff880c2d537e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975103] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975105] RDX: 0000000000000000 RSI: ffff880c2d537fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975107] RBP: ffff880c2d537e98 R08: 00000000fffffffc R09: 0000000000005d07 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975109] R10: 0000000000000f40 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975110] R13: ffff880c3fc3a070 R14: 0000000000000020 R15: 131669da0fb7fe4f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975113] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975114] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975116] CR2: 00007fc26618af50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975122] Process swapper/2 (pid: 0, threadinfo ffff880c2d536000, task ffff880c2d52c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975123] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975124] 0000000000000082 ffff880c3fc2eb80 0000000000000000 00000000016b6710 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975129] ffff880c2d537ea8 0000000281535457 0000000000000000 ffff880c3fc3a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975133] 00000000fffffff0 0000000000000003 0000000000000002 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975137] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975141] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975144] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975148] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975153] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975154] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975189] NMI backtrace for cpu 3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975192] CPU 3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975194] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975252] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975254] Pid: 0, comm: swapper/3 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975258] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975263] RSP: 0018:ffff880c2d601e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975265] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975267] RDX: 0000000000000000 RSI: ffff880c2d601fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975268] RBP: ffff880c2d601e98 R08: 00000000fffffffc R09: 0000000000005d14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975270] R10: 0000000000001363 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975272] R13: ffff88186fc3a070 R14: 0000000000000020 R15: 131669da0f7ad27c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975274] FS: 0000000000000000(0000) GS:ffff88186fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975276] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975278] CR2: 00007ff33b9f5000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975283] Process swapper/3 (pid: 0, threadinfo ffff880c2d600000, task ffff880c2d52dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975285] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975286] 0000000000000082 ffff88186fc2eb80 0000000000000000 00000000016b9893 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975290] ffff880c2d601ea8 0000000381535457 0000000000000000 ffff88186fc3a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975294] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975298] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975303] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975306] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975309] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975313] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975314] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975349] NMI backtrace for cpu 4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975352] CPU 4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975354] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975415] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975417] Pid: 0, comm: swapper/4 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975421] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975427] RSP: 0018:ffff880c2d603e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975429] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975431] RDX: 0000000000000000 RSI: ffff880c2d603fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975433] RBP: ffff880c2d603e98 R08: 00000000fffffffc R09: 0000000000005d26 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975435] R10: 0000000000001015 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975437] R13: ffff880c3fc5a070 R14: 0000000000000020 R15: 131669da0f7ad7a9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975440] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975442] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975444] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975451] Process swapper/4 (pid: 0, threadinfo ffff880c2d602000, task ffff880c2d608000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975452] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975454] 0000000000000082 ffff880c3fc4eb80 0000000000000000 00000000016bdefa | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975459] ffff880c2d603ea8 0000000481535457 0000000000000000 ffff880c3fc5a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975464] 00000000fffffff0 0000000000000003 0000000000000004 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975468] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975473] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975477] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975481] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975485] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975487] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975526] NMI backtrace for cpu 5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975529] CPU 5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975531] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975583] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975586] Pid: 0, comm: swapper/5 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975589] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975594] RSP: 0018:ffff880c2d605e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975596] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975598] RDX: 0000000000000000 RSI: ffff880c2d605fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975600] RBP: ffff880c2d605e98 R08: 00000000fffffffc R09: 0000000000005cfd | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975602] R10: 00000000000011e7 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975603] R13: ffff88186fc5a070 R14: 0000000000000020 R15: 131669da0fb7bf3b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975605] FS: 0000000000000000(0000) GS:ffff88186fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975609] CR2: 0000000000c28b18 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975615] Process swapper/5 (pid: 0, threadinfo ffff880c2d604000, task ffff880c2d609700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975616] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975617] 0000000000000082 ffff88186fc4eb80 0000000000000000 00000000016b3d44 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975622] ffff880c2d605ea8 0000000581535457 0000000000000000 ffff88186fc5a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975626] 00000000fffffff0 0000000000000003 0000000000000005 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975630] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975634] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975637] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975641] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975644] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975646] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975679] NMI backtrace for cpu 6 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975681] CPU 6 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975682] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975742] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975744] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975748] RIP: 0010:[<ffffffff81035d71>] [<ffffffff81035d71>] default_send_IPI_mask_sequence_phys+0x71/0xe0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975755] RSP: 0018:ffff880c3fc63d38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975758] RAX: ffff88186fc60000 RBX: 0000000000000002 RCX: 0000000000000034 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975760] RDX: 0000000000000007 RSI: 0000000000000100 RDI: 0000000000000100 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975762] RBP: ffff880c3fc63d78 R08: ffffffff81cd5680 R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975764] R10: 0000000000000000 R11: 0000000000012564 R12: 000000000000dc82 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975765] R13: ffffffff81cd5680 R14: 0000000000000400 R15: 0000000000000092 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975768] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975770] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975772] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975779] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975780] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975782] 0000000000000014 0000000000000006 ffff880c3fc63d98 0000000000002710 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975787] ffffffff81c387c0 0000000000000006 0000000000000006 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975791] ffff880c3fc63d88 ffffffff8103a2e7 ffff880c3fc63da8 ffffffff81035f61 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975796] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975797] <IRQ> | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975800] [<ffffffff8103a2e7>] physflat_send_IPI_all+0x17/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975808] [<ffffffff81035f61>] arch_trigger_all_cpu_backtrace+0x61/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975815] [<ffffffff810e9b59>] print_cpu_stall+0x79/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975819] [<ffffffff810eb83a>] __rcu_pending+0xea/0x210 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975823] [<ffffffff810eb98c>] rcu_pending+0x2c/0x60 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975828] [<ffffffff810ec151>] rcu_check_callbacks+0xa1/0x160 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975834] [<ffffffff81065698>] update_process_times+0x48/0x90 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975839] [<ffffffff810ab914>] tick_sched_timer+0x64/0xc0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975844] [<ffffffff8107c238>] __run_hrtimer+0x78/0x1f0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975848] [<ffffffff810ab8b0>] ? tick_nohz_handler+0x100/0x100 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975853] [<ffffffff8105ba7d>] ? __do_softirq+0xfd/0x210 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975858] [<ffffffff8107cac7>] hrtimer_interrupt+0xf7/0x240 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975862] [<ffffffff816a6a5c>] ? call_softirq+0x1c/0x30 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975866] [<ffffffff816a73b9>] smp_apic_timer_interrupt+0x69/0x99 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975871] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975873] <EOI> | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975875] [<ffffffff8115e780>] ? remove_vm_area+0x80/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975883] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975887] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975890] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975904] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975914] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975925] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975958] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975978] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975984] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975988] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975993] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975998] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976001] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976005] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976009] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976013] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976016] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976017] Code: 00 00 4c 89 ef 48 63 d2 e8 8d d2 30 00 89 c2 48 63 05 e8 0a ca 00 48 39 c2 73 41 48 8b 04 d5 60 49 cd 81 83 fb 02 41 0f b7 0c 04 <75> 07 eb 46 0f 1f 00 f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976119] NMI backtrace for cpu 7 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976124] CPU 7 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976126] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976187] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976190] Pid: 0, comm: swapper/7 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976194] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976200] RSP: 0018:ffff880c2d611e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976202] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976204] RDX: 0000000000000000 RSI: ffff880c2d611fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976206] RBP: ffff880c2d611e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976208] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976210] R13: ffff88186fc7a070 R14: 0000000000000020 R15: 131669da0f035e80 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976213] FS: 0000000000000000(0000) GS:ffff88186fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976215] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976217] CR2: 00007f1101569000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976219] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976223] Process swapper/7 (pid: 0, threadinfo ffff880c2d610000, task ffff880c2d60c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976225] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976226] 0000000000000007 ffff88186fc6eb80 0000000000000000 00000000016abb8c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976232] ffff880c2d611ea8 0000000781535457 0000000000000000 ffff88186fc7a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976237] 00000000fffffff0 0000000000000003 0000000000000007 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976241] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976246] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976250] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976254] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976258] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976260] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976300] NMI backtrace for cpu 8 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976303] CPU 8 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976305] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976366] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976368] Pid: 0, comm: swapper/8 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976372] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976378] RSP: 0018:ffff880c2d615e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976380] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976382] RDX: 0000000000000000 RSI: ffff880c2d615fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976384] RBP: ffff880c2d615e98 R08: 00000000fffffffc R09: 000000000021df98 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976386] R10: 0000000000000008 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976388] R13: ffff880c3fc9a070 R14: 0000000000000020 R15: 131669da0fbfed5c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976391] FS: 0000000000000000(0000) GS:ffff880c3fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976393] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976395] CR2: 00007f9f4e7803dd CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976399] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976402] Process swapper/8 (pid: 0, threadinfo ffff880c2d614000, task ffff880c2d60dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976403] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976404] 0000000000000000 000000000000020e 0000000000000002 000000000d1bd608 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976410] ffff880c2d615ea8 0000000881535457 0000000000000002 ffff880c3fc9a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976415] 00000000fffffff0 0000000000000003 0000000000000008 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976419] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976424] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976428] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976432] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976436] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976438] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976477] NMI backtrace for cpu 9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976481] CPU 9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976483] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976544] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976546] Pid: 0, comm: swapper/9 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976550] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976556] RSP: 0018:ffff880c2d617e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976559] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976561] RDX: 0000000000000000 RSI: ffff880c2d617fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976563] RBP: ffff880c2d617e98 R08: 00000000fffffffc R09: 0000000000005d13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976565] R10: 0000000000000f5f R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976567] R13: ffff88186fc9a070 R14: 0000000000000020 R15: 131669da0fc1666b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976569] FS: 0000000000000000(0000) GS:ffff88186fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976572] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976574] CR2: 00007fe303c95f69 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976578] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976580] Process swapper/9 (pid: 0, threadinfo ffff880c2d616000, task ffff880c2d618000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976582] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976583] 0000000000000082 ffff88186fc8eb80 0000000000000000 00000000016b941c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976589] ffff880c2d617ea8 0000000981535457 0000000000000000 ffff88186fc9a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976593] 00000000fffffff0 0000000000000003 0000000000000009 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976598] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976603] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976607] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976611] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976615] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976617] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976657] NMI backtrace for cpu 10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976660] CPU 10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976662] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976722] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976725] Pid: 0, comm: swapper/10 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976729] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976735] RSP: 0018:ffff880c2d621e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976737] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976739] RDX: 0000000000000000 RSI: ffff880c2d621fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976741] RBP: ffff880c2d621e98 R08: 00000000fffffffc R09: 0000000000005d03 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976743] R10: 0000000000001049 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976745] R13: ffff880c3fcba070 R14: 0000000000000020 R15: 131669da0fc2f2fe | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976748] FS: 0000000000000000(0000) GS:ffff880c3fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976750] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976752] CR2: 00007fe303d82a00 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976756] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976759] Process swapper/10 (pid: 0, threadinfo ffff880c2d620000, task ffff880c2d619700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976760] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976762] 0000000000000082 ffff880c3fcaeb80 0000000000000000 00000000016b5688 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976767] ffff880c2d621ea8 0000000a81535457 0000000000000000 ffff880c3fcba070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976771] 00000000fffffff0 0000000000000003 000000000000000a 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976776] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976781] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976785] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976789] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976793] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976795] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976835] NMI backtrace for cpu 11 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976838] CPU 11 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976840] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976901] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976904] Pid: 0, comm: swapper/11 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976908] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976914] RSP: 0018:ffff880c2d623e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976916] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976918] RDX: 0000000000000000 RSI: ffff880c2d623fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976920] RBP: ffff880c2d623e98 R08: 00000000fffffffc R09: 0000000000005cf9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976922] R10: 000000000000123d R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976924] R13: ffff88186fcba070 R14: 0000000000000020 R15: 131669da0fc48091 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976927] FS: 0000000000000000(0000) GS:ffff88186fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976929] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976931] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976937] Process swapper/11 (pid: 0, threadinfo ffff880c2d622000, task ffff880c2d61ae00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976939] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976940] 0000000000000082 ffff88186fcaeb80 0000000000000000 00000000016b2ee4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976946] ffff880c2d623ea8 0000000b81535457 0000000000000000 ffff88186fcba070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976950] 00000000fffffff0 0000000000000003 000000000000000b 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976955] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976960] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976963] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976967] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976972] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976973] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977013] NMI backtrace for cpu 12 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977017] CPU 12 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977019] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977079] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977082] Pid: 0, comm: swapper/12 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977086] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977092] RSP: 0018:ffff880c2d625e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977094] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977096] RDX: 0000000000000000 RSI: ffff880c2d625fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977098] RBP: ffff880c2d625e98 R08: 00000000fffffffc R09: 0000000000005cf1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977100] R10: 0000000000000eb5 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977102] R13: ffff880c3fcda070 R14: 0000000000000020 R15: 131669da0fc60c05 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977105] FS: 0000000000000000(0000) GS:ffff880c3fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977109] CR2: 00007f32d7b8af50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977113] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977115] Process swapper/12 (pid: 0, threadinfo ffff880c2d624000, task ffff880c2d61c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977117] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977118] 0000000000000082 ffff880c3fcceb80 0000000000000000 00000000016b0f1d | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977124] ffff880c2d625ea8 0000000c81535457 0000000000000000 ffff880c3fcda070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977128] 00000000fffffff0 0000000000000003 000000000000000c 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977133] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977138] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977141] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977145] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977150] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977152] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977191] NMI backtrace for cpu 13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977195] CPU 13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977197] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977258] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977260] Pid: 0, comm: swapper/13 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977264] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977270] RSP: 0018:ffff880c2d627e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977273] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977275] RDX: 0000000000000000 RSI: ffff880c2d627fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977277] RBP: ffff880c2d627e98 R08: 00000000fffffffc R09: 0000000000005d0c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977279] R10: 0000000000001039 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977281] R13: ffff88186fcda070 R14: 0000000000000020 R15: 131669da0fc75c56 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977283] FS: 0000000000000000(0000) GS:ffff88186fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977286] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977288] CR2: 00007fbb1b727c62 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977292] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977294] Process swapper/13 (pid: 0, threadinfo ffff880c2d626000, task ffff880c2d61dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977296] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977297] 0000000000000082 ffff88186fcceb80 0000000000000000 00000000016b771f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977302] ffff880c2d627ea8 0000000d81535457 0000000000000000 ffff88186fcda070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977307] 00000000fffffff0 0000000000000003 000000000000000d 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977312] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977317] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977320] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977324] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977328] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977330] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977369] NMI backtrace for cpu 14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977373] CPU 14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977375] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977454] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977457] Pid: 0, comm: swapper/14 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977462] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977469] RSP: 0018:ffff880c2d639e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977472] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977474] RDX: 0000000000000000 RSI: ffff880c2d639fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977477] RBP: ffff880c2d639e98 R08: 00000000fffffffc R09: 00000000000000fc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977480] R10: 0000000000001cc3 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977483] R13: ffff880c3fcfa070 R14: 0000000000000020 R15: 131669d94b0671d4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977486] FS: 0000000000000000(0000) GS:ffff880c3fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977489] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977492] CR2: 00007f8ee7fabbe0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977498] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977501] Process swapper/14 (pid: 0, threadinfo ffff880c2d638000, task ffff880c2d630000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977503] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977506] ffff880c2d639e48 ffff880c3fcee980 0000000000000003 000000003b99af6d | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977512] ffff880c2d639ea8 0000000e81535457 0000000000000003 ffff880c3fcfa070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977519] 00000000fffffff0 0000000000000003 000000000000000e 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977525] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977531] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977536] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977541] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977547] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977549] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977603] NMI backtrace for cpu 15 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977606] CPU 15 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977608] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977669] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977672] Pid: 0, comm: swapper/15 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977676] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977682] RSP: 0018:ffff880c2d63be38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977684] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977686] RDX: 0000000000000000 RSI: ffff880c2d63bfd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977688] RBP: ffff880c2d63be98 R08: 00000000fffffffc R09: 0000000000005cf0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977690] R10: 0000000000001f3c R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977692] R13: ffff88186fcfa070 R14: 0000000000000020 R15: 131669da0fd6ee67 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977695] FS: 0000000000000000(0000) GS:ffff88186fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977697] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977699] CR2: 00007ff8e4688be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977701] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977703] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977706] Process swapper/15 (pid: 0, threadinfo ffff880c2d63a000, task ffff880c2d631700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977707] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977709] 0000000000000082 ffff88186fceeb80 0000000000000000 00000000016b0a13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977714] ffff880c2d63bea8 0000000f81535457 0000000000000000 ffff88186fcfa070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977719] 00000000fffffff0 0000000000000003 000000000000000f 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977723] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977728] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977732] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977736] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977740] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977742] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507530] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507900] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507969] CPU 6 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507971] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508026] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508028] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508032] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508042] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508044] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508046] RDX: 0000000000000002 RSI: 000000000000006f RDI: 0000000000000380 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508048] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508050] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508052] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508054] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508057] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508059] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508065] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508067] Stack: | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508068] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508074] 00080fe7d95876b7 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508078] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508083] Call Trace: | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508085] <IRQ> | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508087] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508097] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508103] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508107] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508111] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508116] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508118] <EOI> | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508120] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508128] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508131] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508135] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508149] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508159] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508170] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508204] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508224] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508230] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508234] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508240] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508244] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508247] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508252] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508256] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508260] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508263] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508264] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.440825] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441298] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441373] CPU 6 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441374] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441429] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441432] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441436] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441445] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441448] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441450] RDX: 0000000000000002 RSI: 000000000000006d RDI: 0000000000000380 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441451] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441453] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441455] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441458] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441460] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441462] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441466] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441469] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441470] Stack: | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441472] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441477] 00080fee5e468e6a ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441482] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441486] Call Trace: | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441489] <IRQ> | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441491] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441500] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441507] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441511] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441515] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441520] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441521] <EOI> | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441523] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441532] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441535] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441539] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441553] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441563] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441573] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441606] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441627] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441632] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441637] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441642] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441646] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441650] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441655] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441658] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441662] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441665] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441667] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.374184] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386693] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386769] CPU 6 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386770] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386825] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386828] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386832] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386841] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386844] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386846] RDX: 0000000000000002 RSI: 0000000000000070 RDI: 0000000000000380 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386847] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386849] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386851] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386854] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386856] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386858] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386865] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386866] Stack: | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386868] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386873] 00080ff4e334a596 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386878] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386883] Call Trace: | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386885] <IRQ> | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386887] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386896] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386903] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386907] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386911] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386916] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386918] <EOI> | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386920] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386928] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386932] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386935] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386949] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386960] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386970] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387004] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387024] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387030] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387034] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387040] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387044] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387048] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387052] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387056] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387060] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387063] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387064] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.307506] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320178] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320253] CPU 6 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320254] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320309] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320312] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320316] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320326] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320328] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320330] RDX: 0000000000000002 RSI: 000000000000006e RDI: 0000000000000380 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320332] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320334] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320336] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320338] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320341] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320343] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320347] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320349] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320351] Stack: | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320353] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320358] 00080ffb6822bdaf ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320363] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320367] Call Trace: | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320369] <IRQ> | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320372] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320381] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320388] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320391] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320396] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320401] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320402] <EOI> | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320404] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320412] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320416] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320420] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320434] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320444] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320454] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320488] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320509] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320515] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320519] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320524] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320528] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320532] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320537] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320540] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320544] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320547] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320549] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.717589] BUG: soft lockup - CPU#4 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733215] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733290] CPU 4 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733291] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733347] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733350] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733354] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733363] RSP: 0018:ffff880c3fc43ee0 EFLAGS: 00000206 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733365] RAX: 0000000000000004 RBX: ffff880c3fc4e980 RCX: 0000000000000020 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733368] RDX: 0000000000000002 RSI: 0000000000000063 RDI: 0000000000000380 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733369] RBP: ffff880c3fc43f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733371] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc43e58 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733373] R13: ffffffff816a610a R14: ffff880c3fc43f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733376] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733378] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733380] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733384] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733387] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733388] Stack: | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733390] ffffffff81c18f00 ffff880c2a451fd8 000000040000000a ffff880c2a451fd8 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733396] 000810325929b636 ffff880c3fc43f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733400] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc905d3ab7000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733405] Call Trace: | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733407] <IRQ> | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733410] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733419] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733426] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733430] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733434] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733439] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733441] <EOI> | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733443] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733451] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733455] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733458] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733472] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733483] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733493] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733529] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733561] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733572] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733578] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733586] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733592] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733599] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733605] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733611] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733617] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733621] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733623] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:31:34 storage1 kernel: [2264231.650941] BUG: soft lockup - CPU#4 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663177] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663268] CPU 4 | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663270] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663325] | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663328] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663332] RIP: 0010:[<ffffffff8105b9e0>] Aug 1 18:03:08 storage1 kernel: imklog 5.8.6, log source = /proc/kmsg started. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Aug 1 17:19:34 storage1 kernel: [2263513.341374] BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341736] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341811] CPU 2 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341813] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341868] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341871] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341875] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341885] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341887] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341889] RDX: 0000000000000002 RSI: 000000000000004f RDI: 0000000000000380 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341891] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341893] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341895] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341898] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341900] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341902] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341906] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341909] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341910] Stack: | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341912] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341918] 00080f913951789d ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341923] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341927] Call Trace: | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341929] <IRQ> | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341932] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341941] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341948] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341952] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341956] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341961] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341963] <EOI> | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341965] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341973] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341976] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341980] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.341994] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342005] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342015] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342049] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342069] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342075] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342079] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342085] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342089] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342093] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342097] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342101] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342105] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342108] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:19:34 storage1 kernel: [2263513.342109] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.274676] BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275076] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275137] CPU 2 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275138] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275172] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275174] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275176] RIP: 0010:[<ffffffff8133eef3>] [<ffffffff8133eef3>] __write_lock_failed+0x13/0x20 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275183] RSP: 0018:ffff880c2a451ab0 EFLAGS: 00000287 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275184] RAX: ffff880634001300 RBX: 0000000000000000 RCX: ffff880c2a451ea0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275186] RDX: 0000000000000e72 RSI: 0000000000000001 RDI: ffffffff81c41b40 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275187] RBP: ffff880c2a451ab0 R08: ffff8814ae65f340 R09: 0000000100400029 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275188] R10: 00000000ae65f801 R11: 0000000000000006 R12: 0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275189] R13: ffff880c3fc36e70 R14: ffff88186fffbd68 R15: ffffffff81688a4f | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275191] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275193] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275197] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275198] Stack: | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275199] ffff880c2a451ac0 ffffffff8169cf17 ffff880c2a451ae0 ffffffff8115e764 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275203] ffffc90c96a45000 ffff88182bf03840 ffff880c2a451b10 ffffffff8115e86d | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275206] ffff881561ad0200 ffffc90c96a86000 ffff88182bf03840 ffff88182bf03000 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275209] Call Trace: | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275214] [<ffffffff8169cf17>] _raw_write_lock+0x17/0x20 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275218] [<ffffffff8115e764>] remove_vm_area+0x64/0xa0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275221] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275223] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275225] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275236] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275242] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275249] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275276] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275289] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275293] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275296] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275299] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275302] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275304] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275307] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275310] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275313] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275315] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:02 storage1 kernel: [2263541.275316] Code: 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 55 48 89 e5 f0 81 07 00 00 10 00 f3 90 81 3f 00 00 10 00 <75> f6 f0 81 2f 00 00 10 00 75 e6 5d c3 55 48 89 e5 f0 ff 07 f3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932386] INFO: rcu_sched self-detected stall on CPU { 2} (t=15000 jiffies) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932879] sending NMI to all CPUs: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932938] NMI backtrace for cpu 0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932943] CPU 0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.932944] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933022] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933025] Pid: 0, comm: swapper/0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933030] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933039] RSP: 0018:ffffffff81c01e08 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933041] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933043] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933045] RBP: ffffffff81c01e68 R08: 00000000fffffffc R09: 000000000001b517 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933047] R10: 00000000000000d2 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933049] R13: ffff880c3fc1a070 R14: 0000000000000020 R15: 1316699177b54b0c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933052] FS: 0000000000000000(0000) GS:ffff880c3fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933054] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933056] CR2: 00007f484d4e4fd0 CR3: 0000000001c0b000 CR4: 00000000000007f0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933063] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c13460) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933064] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933066] 0000000000000000 0000000000000f6e 0000000000000000 00000000066e562a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933071] ffffffff81c01e78 0000000081535457 0000000000000000 ffff880c3fc1a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933076] 00000000fffffff0 0000000000000003 0000000000000000 000000000008c800 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933081] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933088] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933092] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933097] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933103] [<ffffffff8166080e>] rest_init+0x72/0x74 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933109] [<ffffffff81cf2c4d>] start_kernel+0x3cf/0x3dc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933113] [<ffffffff81cf27ff>] ? pass_bootoption.constprop.3+0xd3/0xd3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933118] [<ffffffff81cf2397>] x86_64_start_reservations+0x131/0x135 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933122] [<ffffffff81cf2120>] ? early_idt_handlers+0x120/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933126] [<ffffffff81cf2468>] x86_64_start_kernel+0xcd/0xdc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933128] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933167] NMI backtrace for cpu 1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933171] CPU 1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933173] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933233] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933236] Pid: 0, comm: swapper/1 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933240] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933247] RSP: 0018:ffff880c2d535e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933249] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933251] RDX: 0000000000000000 RSI: ffff880c2d535fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933253] RBP: ffff880c2d535e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933254] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933256] R13: ffff88186fc1a070 R14: 0000000000000020 R15: 13166991773d5ebb | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933258] FS: 0000000000000000(0000) GS:ffff88186fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933260] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933262] CR2: 00007f55c4e4e980 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933268] Process swapper/1 (pid: 0, threadinfo ffff880c2d534000, task ffff880c2d52ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933269] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933270] 0000000000000001 ffff88186fc0eb80 0000000000000000 00000000016a735e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933275] ffff880c2d535ea8 0000000181535457 0000000000000000 ffff88186fc1a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933279] 00000000fffffff0 0000000000000003 0000000000000001 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933283] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933289] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933293] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933298] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933304] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933305] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933338] NMI backtrace for cpu 2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933341] CPU 2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933342] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933402] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933404] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933408] RIP: 0010:[<ffffffff81035d71>] [<ffffffff81035d71>] default_send_IPI_mask_sequence_phys+0x71/0xe0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933416] RSP: 0018:ffff880c3fc23d38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933418] RAX: ffff88186fc20000 RBX: 0000000000000002 RCX: 0000000000000022 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933420] RDX: 0000000000000003 RSI: 0000000000000100 RDI: 0000000000000100 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933422] RBP: ffff880c3fc23d78 R08: ffffffff81cd5680 R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933424] R10: 0000000000000000 R11: 0000000000026810 R12: 000000000000dc82 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933426] R13: ffffffff81cd5680 R14: 0000000000000400 R15: 0000000000000092 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933428] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933431] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933433] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933435] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933439] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933441] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933442] 0000000000000002 0000000000000002 ffff880c3fc23d98 0000000000002710 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933447] ffffffff81c387c0 0000000000000002 0000000000000002 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933452] ffff880c3fc23d88 ffffffff8103a2e7 ffff880c3fc23da8 ffffffff81035f61 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933456] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933458] <IRQ> | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933460] [<ffffffff8103a2e7>] physflat_send_IPI_all+0x17/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933469] [<ffffffff81035f61>] arch_trigger_all_cpu_backtrace+0x61/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933476] [<ffffffff810e9b59>] print_cpu_stall+0x79/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933480] [<ffffffff810eb83a>] __rcu_pending+0xea/0x210 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933484] [<ffffffff810eb98c>] rcu_pending+0x2c/0x60 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933489] [<ffffffff810ec151>] rcu_check_callbacks+0xa1/0x160 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933495] [<ffffffff81065698>] update_process_times+0x48/0x90 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933500] [<ffffffff810ab914>] tick_sched_timer+0x64/0xc0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933505] [<ffffffff8107c238>] __run_hrtimer+0x78/0x1f0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933509] [<ffffffff810ab8b0>] ? tick_nohz_handler+0x100/0x100 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933514] [<ffffffff8105ba7d>] ? __do_softirq+0xfd/0x210 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933518] [<ffffffff8107cac7>] hrtimer_interrupt+0xf7/0x240 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933523] [<ffffffff816a6a5c>] ? call_softirq+0x1c/0x30 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933527] [<ffffffff816a73b9>] smp_apic_timer_interrupt+0x69/0x99 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933533] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933534] <EOI> | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933536] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933544] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933548] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933551] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933565] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933575] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933586] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933620] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933641] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933646] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933651] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933656] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933660] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933664] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933668] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933672] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933675] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933679] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933680] Code: 00 00 4c 89 ef 48 63 d2 e8 8d d2 30 00 89 c2 48 63 05 e8 0a ca 00 48 39 c2 73 41 48 8b 04 d5 60 49 cd 81 83 fb 02 41 0f b7 0c 04 <75> 07 eb 46 0f 1f 00 f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933779] NMI backtrace for cpu 3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933784] CPU 3 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933786] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933847] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933850] Pid: 0, comm: swapper/3 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933854] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933860] RSP: 0018:ffff880c2d601e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933862] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933864] RDX: 0000000000000000 RSI: ffff880c2d601fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933867] RBP: ffff880c2d601e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933868] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933870] R13: ffff88186fc3a070 R14: 0000000000000020 R15: 13166991773d7296 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933873] FS: 0000000000000000(0000) GS:ffff88186fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933875] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933877] CR2: 00000000006e1dc8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933881] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933884] Process swapper/3 (pid: 0, threadinfo ffff880c2d600000, task ffff880c2d52dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933885] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933887] 0000000000000003 ffff88186fc2eb80 0000000000000000 00000000012d5695 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933892] ffff880c2d601ea8 0000000381535457 0000000000000000 ffff88186fc3a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933897] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933901] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933906] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933910] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933914] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933918] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933920] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933960] NMI backtrace for cpu 4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933963] CPU 4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.933965] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934026] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934028] Pid: 0, comm: swapper/4 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934032] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934038] RSP: 0018:ffff880c2d603e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934040] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934042] RDX: 0000000000000000 RSI: ffff880c2d603fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934044] RBP: ffff880c2d603e98 R08: 00000000fffffffc R09: 0000000000005ce8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934046] R10: 0000000000001548 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934048] R13: ffff880c3fc5a070 R14: 0000000000000020 R15: 13166991773a9cd8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934051] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934053] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934055] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934062] Process swapper/4 (pid: 0, threadinfo ffff880c2d602000, task ffff880c2d608000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934063] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934065] 0000000000000082 ffff880c3fc4eb80 0000000000000000 00000000016aed8c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934070] ffff880c2d603ea8 0000000481535457 0000000000000000 ffff880c3fc5a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934075] 00000000fffffff0 0000000000000003 0000000000000004 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934079] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934084] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934088] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934092] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934097] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934099] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934139] NMI backtrace for cpu 5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934142] CPU 5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934144] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934205] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934208] Pid: 0, comm: swapper/5 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934211] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934218] RSP: 0018:ffff880c2d605e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934220] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934222] RDX: 0000000000000000 RSI: ffff880c2d605fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934224] RBP: ffff880c2d605e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934226] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934228] R13: ffff88186fc5a070 R14: 0000000000000020 R15: 131669917778d5a7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934230] FS: 0000000000000000(0000) GS:ffff88186fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934233] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934235] CR2: 00007ff33b9f5000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934239] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934241] Process swapper/5 (pid: 0, threadinfo ffff880c2d604000, task ffff880c2d609700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934243] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934244] 0000000000000005 ffff88186fc4eb80 0000000000000000 00000000016c0248 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934250] ffff880c2d605ea8 0000000581535457 0000000000000000 ffff88186fc5a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934254] 00000000fffffff0 0000000000000003 0000000000000005 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934259] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934264] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934268] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934272] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934276] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934278] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934317] NMI backtrace for cpu 6 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934320] CPU 6 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934322] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934374] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934377] Pid: 0, comm: swapper/6 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934380] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934385] RSP: 0018:ffff880c2d607e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934387] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934389] RDX: 0000000000000000 RSI: ffff880c2d607fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934391] RBP: ffff880c2d607e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934392] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934394] R13: ffff880c3fc7a070 R14: 0000000000000020 R15: 13166991773d1333 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934396] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934398] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934400] CR2: 00007f484b9517d8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934405] Process swapper/6 (pid: 0, threadinfo ffff880c2d606000, task ffff880c2d60ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934407] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934408] 0000000000000006 ffff880c3fc6eb80 0000000000000000 00000000016abb88 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934413] ffff880c2d607ea8 0000000681535457 0000000000000000 ffff880c3fc7a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934417] 00000000fffffff0 0000000000000003 0000000000000006 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934421] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934425] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934428] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934432] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934435] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934437] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934471] NMI backtrace for cpu 7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934475] CPU 7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934477] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934537] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934540] Pid: 0, comm: swapper/7 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934544] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934550] RSP: 0018:ffff880c2d611e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934552] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934554] RDX: 0000000000000000 RSI: ffff880c2d611fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934556] RBP: ffff880c2d611e98 R08: 00000000fffffffc R09: 0000000000005cdc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934558] R10: 0000000000000dc3 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934560] R13: ffff88186fc7a070 R14: 0000000000000020 R15: 13166991773ab18a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934563] FS: 0000000000000000(0000) GS:ffff88186fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934565] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934567] CR2: 00007f484b9517d8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934571] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934574] Process swapper/7 (pid: 0, threadinfo ffff880c2d610000, task ffff880c2d60c500) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934578] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934580] 0000000000000082 ffff88186fc6eb80 0000000000000000 00000000016abbcc | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934584] ffff880c2d611ea8 0000000781535457 0000000000000000 ffff88186fc7a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934588] 00000000fffffff0 0000000000000003 0000000000000007 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934592] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934597] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934600] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934603] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934607] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934608] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934643] NMI backtrace for cpu 8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934646] CPU 8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934648] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934709] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934711] Pid: 0, comm: swapper/8 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934715] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934721] RSP: 0018:ffff880c2d615e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934723] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934725] RDX: 0000000000000000 RSI: ffff880c2d615fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934727] RBP: ffff880c2d615e98 R08: 00000000fffffffc R09: 000000000006a9e9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934729] R10: 000000000000002c R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934731] R13: ffff880c3fc9a070 R14: 0000000000000020 R15: 1316699177f91da1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934734] FS: 0000000000000000(0000) GS:ffff880c3fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934736] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934738] CR2: 00007f86145c6be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934742] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934745] Process swapper/8 (pid: 0, threadinfo ffff880c2d614000, task ffff880c2d60dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934746] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934748] 0000000000000000 000000000000118b 0000000000000000 000000001a07b9f1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934753] ffff880c2d615ea8 0000000881535457 0000000000000000 ffff880c3fc9a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934758] 00000000fffffff0 0000000000000003 0000000000000008 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934762] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934767] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934771] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934775] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934779] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934781] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934820] NMI backtrace for cpu 9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934824] CPU 9 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934825] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934878] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934880] Pid: 0, comm: swapper/9 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934883] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934889] RSP: 0018:ffff880c2d617e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934890] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934892] RDX: 0000000000000000 RSI: ffff880c2d617fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934894] RBP: ffff880c2d617e98 R08: 00000000fffffffc R09: 0000000000005cc7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934896] R10: 0000000000001012 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934897] R13: ffff88186fc9a070 R14: 0000000000000020 R15: 1316699177fab29b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934900] FS: 0000000000000000(0000) GS:ffff88186fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934902] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934903] CR2: 00007f55c4de5c39 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934907] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934909] Process swapper/9 (pid: 0, threadinfo ffff880c2d616000, task ffff880c2d618000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934910] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934911] 0000000000000082 ffff88186fc8eb80 0000000000000000 00000000016a6ca1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934916] ffff880c2d617ea8 0000000981535457 0000000000000000 ffff88186fc9a070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934920] 00000000fffffff0 0000000000000003 0000000000000009 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934924] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934928] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934932] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934935] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934939] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934940] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934974] NMI backtrace for cpu 10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934978] CPU 10 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.934980] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935059] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935062] Pid: 0, comm: swapper/10 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935067] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935073] RSP: 0018:ffff880c2d621e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935076] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935079] RDX: 0000000000000000 RSI: ffff880c2d621fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935082] RBP: ffff880c2d621e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935085] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935088] R13: ffff880c3fcba070 R14: 0000000000000020 R15: 13166991773d3013 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935091] FS: 0000000000000000(0000) GS:ffff880c3fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935097] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935103] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935106] Process swapper/10 (pid: 0, threadinfo ffff880c2d620000, task ffff880c2d619700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935108] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935110] 000000000000000a ffff880c3fcaeb80 0000000000000000 00000000016aa105 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935117] ffff880c2d621ea8 0000000a81535457 0000000000000000 ffff880c3fcba070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935124] 00000000fffffff0 0000000000000003 000000000000000a 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935130] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935136] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935141] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935146] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935152] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935154] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935208] NMI backtrace for cpu 11 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935211] CPU 11 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935212] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935265] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935267] Pid: 0, comm: swapper/11 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935271] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935276] RSP: 0018:ffff880c2d623e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935278] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935279] RDX: 0000000000000000 RSI: ffff880c2d623fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935281] RBP: ffff880c2d623e98 R08: 00000000fffffffc R09: 0000000000005cc2 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935283] R10: 0000000000001366 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935285] R13: ffff88186fcba070 R14: 0000000000000020 R15: 131669917806059e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935287] FS: 0000000000000000(0000) GS:ffff88186fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935289] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935291] CR2: 00007f6964a633dd CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935294] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935296] Process swapper/11 (pid: 0, threadinfo ffff880c2d622000, task ffff880c2d61ae00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935297] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935299] 0000000000000082 ffff88186fcaeb80 0000000000000000 00000000016a5749 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935303] ffff880c2d623ea8 0000000b81535457 0000000000000000 ffff88186fcba070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935307] 00000000fffffff0 0000000000000003 000000000000000b 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935311] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935315] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935319] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935322] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935326] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935327] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935362] NMI backtrace for cpu 12 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935365] CPU 12 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935367] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935428] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935431] Pid: 0, comm: swapper/12 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935435] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935440] RSP: 0018:ffff880c2d625e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935443] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935445] RDX: 0000000000000000 RSI: ffff880c2d625fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935447] RBP: ffff880c2d625e98 R08: 00000000fffffffc R09: 0000000000005cd5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935449] R10: 00000000000012b2 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935451] R13: ffff880c3fcda070 R14: 0000000000000020 R15: 131669917807810e | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935453] FS: 0000000000000000(0000) GS:ffff880c3fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935456] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935458] CR2: 0000000000f155b8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935464] Process swapper/12 (pid: 0, threadinfo ffff880c2d624000, task ffff880c2d61c500) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935466] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935467] 0000000000000082 ffff880c3fcceb80 0000000000000000 00000000016aa20a | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935472] ffff880c2d625ea8 0000000c81535457 0000000000000000 ffff880c3fcda070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935477] 00000000fffffff0 0000000000000003 000000000000000c 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935481] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935486] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935490] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935494] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935498] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935500] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935540] NMI backtrace for cpu 13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935543] CPU 13 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935545] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935606] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935609] Pid: 0, comm: swapper/13 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935613] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935619] RSP: 0018:ffff880c2d627e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935621] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935623] RDX: 0000000000000000 RSI: ffff880c2d627fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935625] RBP: ffff880c2d627e98 R08: 00000000fffffffc R09: 0000000000005cd1 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935627] R10: 0000000000000cb2 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935629] R13: ffff88186fcda070 R14: 0000000000000020 R15: 13166991780909a4 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935632] FS: 0000000000000000(0000) GS:ffff88186fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935634] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935636] CR2: 00007f55c4e4df50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935640] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935642] Process swapper/13 (pid: 0, threadinfo ffff880c2d626000, task ffff880c2d61dc00) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935644] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935645] 0000000000000082 ffff88186fcceb80 0000000000000000 00000000016a92f7 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935651] ffff880c2d627ea8 0000000d81535457 0000000000000000 ffff88186fcda070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935655] 00000000fffffff0 0000000000000003 000000000000000d 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935660] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935665] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935668] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935672] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935677] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935678] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935718] NMI backtrace for cpu 14 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935722] CPU 14 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935723] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935784] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935787] Pid: 0, comm: swapper/14 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935791] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935797] RSP: 0018:ffff880c2d639e38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935799] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935801] RDX: 0000000000000000 RSI: ffff880c2d639fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935803] RBP: ffff880c2d639e98 R08: 00000000fffffffc R09: 000000000007e16c | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935805] R10: 0000000000000046 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935807] R13: ffff880c3fcfa070 R14: 0000000000000020 R15: 13166991780a9b61 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935810] FS: 0000000000000000(0000) GS:ffff880c3fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935814] CR2: 00007f9dd6730be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935818] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935821] Process swapper/14 (pid: 0, threadinfo ffff880c2d638000, task ffff880c2d630000) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935822] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935824] 0000000000000000 000000000000353e 0000000000000000 000000001ec891a8 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935829] ffff880c2d639ea8 0000000e81535457 0000000000000000 ffff880c3fcfa070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935833] 00000000fffffff0 0000000000000003 000000000000000e 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935838] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935843] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935846] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935850] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935855] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935857] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935896] NMI backtrace for cpu 15 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935899] CPU 15 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935901] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935962] | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935965] Pid: 0, comm: swapper/15 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935969] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935975] RSP: 0018:ffff880c2d63be38 EFLAGS: 00000046 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935977] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935979] RDX: 0000000000000000 RSI: ffff880c2d63bfd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935981] RBP: ffff880c2d63be98 R08: 00000000fffffffc R09: 0000000000005ccf | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935983] R10: 0000000000000cb9 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935985] R13: ffff88186fcfa070 R14: 0000000000000020 R15: 13166991780c1a5f | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935988] FS: 0000000000000000(0000) GS:ffff88186fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935990] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935992] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.935999] Process swapper/15 (pid: 0, threadinfo ffff880c2d63a000, task ffff880c2d631700) | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936000] Stack: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936002] 0000000000000082 ffff88186fceeb80 0000000000000000 00000000016a8a77 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936007] ffff880c2d63bea8 0000000f81535457 0000000000000000 ffff88186fcfa070 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936012] 00000000fffffff0 0000000000000003 000000000000000f 0000000000000000 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936016] Call Trace: | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936021] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936025] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936029] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936033] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:20:09 storage1 kernel: [2263548.936035] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198526] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198893] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198962] CPU 2 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.198964] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199019] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199022] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199026] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199035] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199037] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199039] RDX: 0000000000000002 RSI: 000000000000004b RDI: 0000000000000380 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199041] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199043] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199045] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199048] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199050] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199052] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199058] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199060] Stack: | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199062] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199067] 00080f9f3198d15c ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199072] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199076] Call Trace: | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199079] <IRQ> | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199081] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199090] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199097] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199100] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199105] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199110] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199111] <EOI> | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199113] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199121] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199125] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199129] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199143] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199153] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199164] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199198] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199219] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199224] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199229] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199234] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199238] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199242] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199246] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199250] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199254] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199257] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:20:34 storage1 kernel: [2263573.199259] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.131835] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132196] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132271] CPU 2 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132273] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132327] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132330] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132334] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132343] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132345] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132347] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132349] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132351] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132353] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132356] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132358] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132360] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132364] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132367] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132368] Stack: | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132370] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132376] 00080fa5b686e8a4 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132380] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132385] Call Trace: | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132387] <IRQ> | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132389] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132399] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132405] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132409] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132413] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132418] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132420] <EOI> | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132422] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132430] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132434] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132437] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132452] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132462] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132472] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132507] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132527] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132533] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132537] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132543] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132547] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132551] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132555] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132559] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132563] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132566] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:02 storage1 kernel: [2263601.132568] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065186] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065548] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065623] CPU 2 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065625] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065680] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065683] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065687] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065696] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065698] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065700] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065702] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065704] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065706] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065708] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065711] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065713] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065719] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065721] Stack: | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065723] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065728] 00080fac3b750090 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065733] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065738] Call Trace: | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065740] <IRQ> | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065742] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065752] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065758] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065762] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065766] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065771] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065773] <EOI> | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065775] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065783] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065787] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065790] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065805] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065815] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065825] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065859] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065880] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065886] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065890] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065895] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065899] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065903] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065908] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065911] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065915] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065918] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:30 storage1 kernel: [2263629.065920] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998497] BUG: soft lockup - CPU#2 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998879] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998954] CPU 2 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.998956] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999011] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999014] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999018] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999027] RSP: 0018:ffff880c3fc23ee0 EFLAGS: 00000206 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999029] RAX: 0000000000000002 RBX: ffff880c3fc2e980 RCX: 0000000000000020 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999031] RDX: 0000000000000002 RSI: 0000000000000051 RDI: 0000000000000380 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999033] RBP: ffff880c3fc23f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999035] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc23e58 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999037] R13: ffffffff816a610a R14: ffff880c3fc23f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999039] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999042] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999044] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999050] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999052] Stack: | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999054] ffffffff81c18f00 ffff880c2a451fd8 000000020000000a ffff880c2a451fd8 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999059] 00080fb2c06318a8 ffff880c3fc23f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999064] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc90eddd28000 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999069] Call Trace: | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999071] <IRQ> | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999073] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999083] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999089] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999093] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999097] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999102] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999104] <EOI> | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999106] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999114] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999117] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999121] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999135] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999145] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999156] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999189] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999209] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999215] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999219] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999224] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999228] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999232] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999237] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999241] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999244] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999248] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:21:58 storage1 kernel: [2263656.999249] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650356] BUG: soft lockup - CPU#6 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650880] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650931] CPU 6 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650932] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650965] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650967] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650970] RIP: 0010:[<ffffffff8115e786>] [<ffffffff8115e786>] remove_vm_area+0x86/0xa0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650977] RSP: 0018:ffff880c2a451ad0 EFLAGS: 00000287 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650978] RAX: ffff880b6ff39080 RBX: ffff880ae3791340 RCX: ffff880c2a451ea0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650979] RDX: ffff880b6ff395c0 RSI: 0000000000000001 RDI: ffffffff81c41b40 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650980] RBP: ffff880c2a451ae0 R08: ffff880ae3791340 R09: 0000000100400004 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650982] R10: 00000000e3791e01 R11: 0000000000000001 R12: 00000000e3791e01 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650983] R13: 0000000000000001 R14: 0000000000000000 R15: ffff880c2a451ab0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650984] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650986] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650987] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650989] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650991] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650992] Stack: | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650993] ffffc90f2de10000 ffff88182bf03840 ffff880c2a451b10 ffffffff8115e86d | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650997] 0000000000000000 ffffc90f27db8000 ffff88182bf03840 ffff88182bf03000 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.650999] ffff880c2a451b20 ffffffff8115e92b ffff880c2a451b30 ffffffff8115ea5a | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651002] Call Trace: | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651006] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651008] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651010] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651021] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651027] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651034] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651062] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651074] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651080] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651083] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651086] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651088] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651091] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651094] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651097] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651100] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651102] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:24:46 storage1 kernel: [2263824.651102] Code: 40 1b c4 81 e8 9c e7 53 00 48 8b 05 8d 6d d9 00 48 c7 c2 f8 54 ef 81 48 39 c3 74 14 66 0f 1f 84 00 00 00 00 00 48 89 c2 48 8b 00 <48> 39 c3 75 f5 48 8b 00 48 89 02 f0 81 05 a4 33 ae 00 00 00 10 | |
| Aug 1 17:25:01 storage1 CRON[26602]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) | |
| Aug 1 17:25:14 storage1 kernel: [2263852.583730] BUG: soft lockup - CPU#6 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584091] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584165] CPU 6 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584167] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584222] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584225] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584229] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584238] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584241] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584243] RDX: 0000000000000002 RSI: 0000000000000070 RDI: 0000000000000380 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584245] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584247] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584249] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584251] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584253] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584255] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584260] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584262] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584264] Stack: | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584265] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584271] 00080fe065ff3588 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584276] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584280] Call Trace: | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584282] <IRQ> | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584285] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584294] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584301] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584305] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584309] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584314] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584316] <EOI> | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584318] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584326] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584329] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584333] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584347] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584357] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584368] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584402] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584423] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584429] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584433] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584438] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584443] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584447] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584451] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584455] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584459] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584462] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:14 storage1 kernel: [2263852.584463] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974032] INFO: rcu_sched self-detected stall on CPU { 6} (t=15000 jiffies) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974556] sending NMI to all CPUs: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974618] NMI backtrace for cpu 0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974622] CPU 0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974624] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974702] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974705] Pid: 0, comm: swapper/0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974709] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974718] RSP: 0018:ffffffff81c01e08 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974724] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974726] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974727] RBP: ffffffff81c01e68 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974729] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974731] R13: ffff880c3fc1a070 R14: 0000000000000020 R15: 131669da0fba3be4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974733] FS: 0000000000000000(0000) GS:ffff880c3fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974736] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007f0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974742] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c13460) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974743] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974745] ffffffff81c01e18 ffffffff8101be59 0000000000000000 000000000039c33a | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974750] ffffffff81c01e78 0000000081535457 0000000000000000 ffff880c3fc1a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974754] 00000000fffffff0 0000000000000003 0000000000000000 000000000008c800 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974758] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974764] [<ffffffff8101be59>] ? sched_clock+0x9/0x10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974769] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974773] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974776] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974782] [<ffffffff8166080e>] rest_init+0x72/0x74 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974787] [<ffffffff81cf2c4d>] start_kernel+0x3cf/0x3dc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974791] [<ffffffff81cf27ff>] ? pass_bootoption.constprop.3+0xd3/0xd3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974795] [<ffffffff81cf2397>] x86_64_start_reservations+0x131/0x135 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974798] [<ffffffff81cf2120>] ? early_idt_handlers+0x120/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974801] [<ffffffff81cf2468>] x86_64_start_kernel+0xcd/0xdc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974803] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974837] NMI backtrace for cpu 1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974841] CPU 1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974843] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974913] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974916] Pid: 0, comm: swapper/1 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974920] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974928] RSP: 0018:ffff880c2d535e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974931] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974933] RDX: 0000000000000000 RSI: ffff880c2d535fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974935] RBP: ffff880c2d535e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974937] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974939] R13: ffff88186fc1a070 R14: 0000000000000020 R15: 131669da0f02b06f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974941] FS: 0000000000000000(0000) GS:ffff88186fc00000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974943] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974945] CR2: 00007fff884dc9a8 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974950] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974952] Process swapper/1 (pid: 0, threadinfo ffff880c2d534000, task ffff880c2d52ae00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974953] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974955] 0000000000000001 ffff88186fc0eb80 0000000000000000 00000000016b6961 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974961] ffff880c2d535ea8 0000000181535457 0000000000000000 ffff88186fc1a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974965] 00000000fffffff0 0000000000000003 0000000000000001 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974970] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974977] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974980] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974986] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974992] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.974994] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975034] NMI backtrace for cpu 2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975037] CPU 2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975038] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975090] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975093] Pid: 0, comm: swapper/2 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975096] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975101] RSP: 0018:ffff880c2d537e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975103] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975105] RDX: 0000000000000000 RSI: ffff880c2d537fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975107] RBP: ffff880c2d537e98 R08: 00000000fffffffc R09: 0000000000005d07 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975109] R10: 0000000000000f40 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975110] R13: ffff880c3fc3a070 R14: 0000000000000020 R15: 131669da0fb7fe4f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975113] FS: 0000000000000000(0000) GS:ffff880c3fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975114] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975116] CR2: 00007fc26618af50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975122] Process swapper/2 (pid: 0, threadinfo ffff880c2d536000, task ffff880c2d52c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975123] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975124] 0000000000000082 ffff880c3fc2eb80 0000000000000000 00000000016b6710 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975129] ffff880c2d537ea8 0000000281535457 0000000000000000 ffff880c3fc3a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975133] 00000000fffffff0 0000000000000003 0000000000000002 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975137] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975141] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975144] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975148] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975153] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975154] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975189] NMI backtrace for cpu 3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975192] CPU 3 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975194] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975252] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975254] Pid: 0, comm: swapper/3 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975258] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975263] RSP: 0018:ffff880c2d601e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975265] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975267] RDX: 0000000000000000 RSI: ffff880c2d601fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975268] RBP: ffff880c2d601e98 R08: 00000000fffffffc R09: 0000000000005d14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975270] R10: 0000000000001363 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975272] R13: ffff88186fc3a070 R14: 0000000000000020 R15: 131669da0f7ad27c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975274] FS: 0000000000000000(0000) GS:ffff88186fc20000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975276] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975278] CR2: 00007ff33b9f5000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975283] Process swapper/3 (pid: 0, threadinfo ffff880c2d600000, task ffff880c2d52dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975285] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975286] 0000000000000082 ffff88186fc2eb80 0000000000000000 00000000016b9893 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975290] ffff880c2d601ea8 0000000381535457 0000000000000000 ffff88186fc3a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975294] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975298] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975303] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975306] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975309] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975313] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975314] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975349] NMI backtrace for cpu 4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975352] CPU 4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975354] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975415] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975417] Pid: 0, comm: swapper/4 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975421] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975427] RSP: 0018:ffff880c2d603e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975429] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975431] RDX: 0000000000000000 RSI: ffff880c2d603fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975433] RBP: ffff880c2d603e98 R08: 00000000fffffffc R09: 0000000000005d26 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975435] R10: 0000000000001015 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975437] R13: ffff880c3fc5a070 R14: 0000000000000020 R15: 131669da0f7ad7a9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975440] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975442] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975444] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975451] Process swapper/4 (pid: 0, threadinfo ffff880c2d602000, task ffff880c2d608000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975452] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975454] 0000000000000082 ffff880c3fc4eb80 0000000000000000 00000000016bdefa | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975459] ffff880c2d603ea8 0000000481535457 0000000000000000 ffff880c3fc5a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975464] 00000000fffffff0 0000000000000003 0000000000000004 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975468] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975473] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975477] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975481] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975485] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975487] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975526] NMI backtrace for cpu 5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975529] CPU 5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975531] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975583] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975586] Pid: 0, comm: swapper/5 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975589] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975594] RSP: 0018:ffff880c2d605e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975596] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975598] RDX: 0000000000000000 RSI: ffff880c2d605fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975600] RBP: ffff880c2d605e98 R08: 00000000fffffffc R09: 0000000000005cfd | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975602] R10: 00000000000011e7 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975603] R13: ffff88186fc5a070 R14: 0000000000000020 R15: 131669da0fb7bf3b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975605] FS: 0000000000000000(0000) GS:ffff88186fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975609] CR2: 0000000000c28b18 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975615] Process swapper/5 (pid: 0, threadinfo ffff880c2d604000, task ffff880c2d609700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975616] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975617] 0000000000000082 ffff88186fc4eb80 0000000000000000 00000000016b3d44 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975622] ffff880c2d605ea8 0000000581535457 0000000000000000 ffff88186fc5a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975626] 00000000fffffff0 0000000000000003 0000000000000005 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975630] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975634] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975637] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975641] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975644] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975646] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975679] NMI backtrace for cpu 6 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975681] CPU 6 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975682] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975742] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975744] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975748] RIP: 0010:[<ffffffff81035d71>] [<ffffffff81035d71>] default_send_IPI_mask_sequence_phys+0x71/0xe0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975755] RSP: 0018:ffff880c3fc63d38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975758] RAX: ffff88186fc60000 RBX: 0000000000000002 RCX: 0000000000000034 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975760] RDX: 0000000000000007 RSI: 0000000000000100 RDI: 0000000000000100 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975762] RBP: ffff880c3fc63d78 R08: ffffffff81cd5680 R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975764] R10: 0000000000000000 R11: 0000000000012564 R12: 000000000000dc82 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975765] R13: ffffffff81cd5680 R14: 0000000000000400 R15: 0000000000000092 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975768] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975770] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975772] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975779] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975780] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975782] 0000000000000014 0000000000000006 ffff880c3fc63d98 0000000000002710 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975787] ffffffff81c387c0 0000000000000006 0000000000000006 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975791] ffff880c3fc63d88 ffffffff8103a2e7 ffff880c3fc63da8 ffffffff81035f61 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975796] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975797] <IRQ> | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975800] [<ffffffff8103a2e7>] physflat_send_IPI_all+0x17/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975808] [<ffffffff81035f61>] arch_trigger_all_cpu_backtrace+0x61/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975815] [<ffffffff810e9b59>] print_cpu_stall+0x79/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975819] [<ffffffff810eb83a>] __rcu_pending+0xea/0x210 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975823] [<ffffffff810eb98c>] rcu_pending+0x2c/0x60 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975828] [<ffffffff810ec151>] rcu_check_callbacks+0xa1/0x160 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975834] [<ffffffff81065698>] update_process_times+0x48/0x90 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975839] [<ffffffff810ab914>] tick_sched_timer+0x64/0xc0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975844] [<ffffffff8107c238>] __run_hrtimer+0x78/0x1f0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975848] [<ffffffff810ab8b0>] ? tick_nohz_handler+0x100/0x100 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975853] [<ffffffff8105ba7d>] ? __do_softirq+0xfd/0x210 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975858] [<ffffffff8107cac7>] hrtimer_interrupt+0xf7/0x240 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975862] [<ffffffff816a6a5c>] ? call_softirq+0x1c/0x30 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975866] [<ffffffff816a73b9>] smp_apic_timer_interrupt+0x69/0x99 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975871] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975873] <EOI> | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975875] [<ffffffff8115e780>] ? remove_vm_area+0x80/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975883] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975887] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975890] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975904] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975914] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975925] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975958] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975978] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975984] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975988] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975993] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.975998] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976001] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976005] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976009] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976013] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976016] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976017] Code: 00 00 4c 89 ef 48 63 d2 e8 8d d2 30 00 89 c2 48 63 05 e8 0a ca 00 48 39 c2 73 41 48 8b 04 d5 60 49 cd 81 83 fb 02 41 0f b7 0c 04 <75> 07 eb 46 0f 1f 00 f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976119] NMI backtrace for cpu 7 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976124] CPU 7 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976126] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976187] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976190] Pid: 0, comm: swapper/7 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976194] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976200] RSP: 0018:ffff880c2d611e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976202] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976204] RDX: 0000000000000000 RSI: ffff880c2d611fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976206] RBP: ffff880c2d611e98 R08: 00000000fffffffc R09: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976208] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976210] R13: ffff88186fc7a070 R14: 0000000000000020 R15: 131669da0f035e80 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976213] FS: 0000000000000000(0000) GS:ffff88186fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976215] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976217] CR2: 00007f1101569000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976219] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976223] Process swapper/7 (pid: 0, threadinfo ffff880c2d610000, task ffff880c2d60c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976225] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976226] 0000000000000007 ffff88186fc6eb80 0000000000000000 00000000016abb8c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976232] ffff880c2d611ea8 0000000781535457 0000000000000000 ffff88186fc7a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976237] 00000000fffffff0 0000000000000003 0000000000000007 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976241] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976246] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976250] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976254] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976258] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976260] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976300] NMI backtrace for cpu 8 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976303] CPU 8 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976305] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976366] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976368] Pid: 0, comm: swapper/8 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976372] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976378] RSP: 0018:ffff880c2d615e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976380] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976382] RDX: 0000000000000000 RSI: ffff880c2d615fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976384] RBP: ffff880c2d615e98 R08: 00000000fffffffc R09: 000000000021df98 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976386] R10: 0000000000000008 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976388] R13: ffff880c3fc9a070 R14: 0000000000000020 R15: 131669da0fbfed5c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976391] FS: 0000000000000000(0000) GS:ffff880c3fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976393] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976395] CR2: 00007f9f4e7803dd CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976399] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976402] Process swapper/8 (pid: 0, threadinfo ffff880c2d614000, task ffff880c2d60dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976403] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976404] 0000000000000000 000000000000020e 0000000000000002 000000000d1bd608 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976410] ffff880c2d615ea8 0000000881535457 0000000000000002 ffff880c3fc9a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976415] 00000000fffffff0 0000000000000003 0000000000000008 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976419] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976424] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976428] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976432] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976436] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976438] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976477] NMI backtrace for cpu 9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976481] CPU 9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976483] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976544] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976546] Pid: 0, comm: swapper/9 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976550] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976556] RSP: 0018:ffff880c2d617e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976559] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976561] RDX: 0000000000000000 RSI: ffff880c2d617fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976563] RBP: ffff880c2d617e98 R08: 00000000fffffffc R09: 0000000000005d13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976565] R10: 0000000000000f5f R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976567] R13: ffff88186fc9a070 R14: 0000000000000020 R15: 131669da0fc1666b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976569] FS: 0000000000000000(0000) GS:ffff88186fc80000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976572] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976574] CR2: 00007fe303c95f69 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976578] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976580] Process swapper/9 (pid: 0, threadinfo ffff880c2d616000, task ffff880c2d618000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976582] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976583] 0000000000000082 ffff88186fc8eb80 0000000000000000 00000000016b941c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976589] ffff880c2d617ea8 0000000981535457 0000000000000000 ffff88186fc9a070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976593] 00000000fffffff0 0000000000000003 0000000000000009 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976598] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976603] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976607] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976611] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976615] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976617] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976657] NMI backtrace for cpu 10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976660] CPU 10 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976662] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976722] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976725] Pid: 0, comm: swapper/10 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976729] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976735] RSP: 0018:ffff880c2d621e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976737] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976739] RDX: 0000000000000000 RSI: ffff880c2d621fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976741] RBP: ffff880c2d621e98 R08: 00000000fffffffc R09: 0000000000005d03 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976743] R10: 0000000000001049 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976745] R13: ffff880c3fcba070 R14: 0000000000000020 R15: 131669da0fc2f2fe | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976748] FS: 0000000000000000(0000) GS:ffff880c3fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976750] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976752] CR2: 00007fe303d82a00 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976756] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976759] Process swapper/10 (pid: 0, threadinfo ffff880c2d620000, task ffff880c2d619700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976760] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976762] 0000000000000082 ffff880c3fcaeb80 0000000000000000 00000000016b5688 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976767] ffff880c2d621ea8 0000000a81535457 0000000000000000 ffff880c3fcba070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976771] 00000000fffffff0 0000000000000003 000000000000000a 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976776] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976781] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976785] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976789] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976793] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976795] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976835] NMI backtrace for cpu 11 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976838] CPU 11 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976840] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976901] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976904] Pid: 0, comm: swapper/11 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976908] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976914] RSP: 0018:ffff880c2d623e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976916] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976918] RDX: 0000000000000000 RSI: ffff880c2d623fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976920] RBP: ffff880c2d623e98 R08: 00000000fffffffc R09: 0000000000005cf9 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976922] R10: 000000000000123d R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976924] R13: ffff88186fcba070 R14: 0000000000000020 R15: 131669da0fc48091 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976927] FS: 0000000000000000(0000) GS:ffff88186fca0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976929] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976931] CR2: 000000000061cd64 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976937] Process swapper/11 (pid: 0, threadinfo ffff880c2d622000, task ffff880c2d61ae00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976939] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976940] 0000000000000082 ffff88186fcaeb80 0000000000000000 00000000016b2ee4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976946] ffff880c2d623ea8 0000000b81535457 0000000000000000 ffff88186fcba070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976950] 00000000fffffff0 0000000000000003 000000000000000b 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976955] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976960] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976963] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976967] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976972] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.976973] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977013] NMI backtrace for cpu 12 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977017] CPU 12 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977019] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977079] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977082] Pid: 0, comm: swapper/12 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977086] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977092] RSP: 0018:ffff880c2d625e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977094] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977096] RDX: 0000000000000000 RSI: ffff880c2d625fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977098] RBP: ffff880c2d625e98 R08: 00000000fffffffc R09: 0000000000005cf1 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977100] R10: 0000000000000eb5 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977102] R13: ffff880c3fcda070 R14: 0000000000000020 R15: 131669da0fc60c05 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977105] FS: 0000000000000000(0000) GS:ffff880c3fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977109] CR2: 00007f32d7b8af50 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977113] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977115] Process swapper/12 (pid: 0, threadinfo ffff880c2d624000, task ffff880c2d61c500) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977117] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977118] 0000000000000082 ffff880c3fcceb80 0000000000000000 00000000016b0f1d | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977124] ffff880c2d625ea8 0000000c81535457 0000000000000000 ffff880c3fcda070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977128] 00000000fffffff0 0000000000000003 000000000000000c 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977133] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977138] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977141] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977145] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977150] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977152] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977191] NMI backtrace for cpu 13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977195] CPU 13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977197] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977258] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977260] Pid: 0, comm: swapper/13 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977264] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977270] RSP: 0018:ffff880c2d627e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977273] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977275] RDX: 0000000000000000 RSI: ffff880c2d627fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977277] RBP: ffff880c2d627e98 R08: 00000000fffffffc R09: 0000000000005d0c | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977279] R10: 0000000000001039 R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977281] R13: ffff88186fcda070 R14: 0000000000000020 R15: 131669da0fc75c56 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977283] FS: 0000000000000000(0000) GS:ffff88186fcc0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977286] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977288] CR2: 00007fbb1b727c62 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977292] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977294] Process swapper/13 (pid: 0, threadinfo ffff880c2d626000, task ffff880c2d61dc00) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977296] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977297] 0000000000000082 ffff88186fcceb80 0000000000000000 00000000016b771f | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977302] ffff880c2d627ea8 0000000d81535457 0000000000000000 ffff88186fcda070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977307] 00000000fffffff0 0000000000000003 000000000000000d 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977312] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977317] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977320] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977324] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977328] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977330] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977369] NMI backtrace for cpu 14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977373] CPU 14 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977375] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977454] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977457] Pid: 0, comm: swapper/14 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977462] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977469] RSP: 0018:ffff880c2d639e38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977472] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977474] RDX: 0000000000000000 RSI: ffff880c2d639fd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977477] RBP: ffff880c2d639e98 R08: 00000000fffffffc R09: 00000000000000fc | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977480] R10: 0000000000001cc3 R11: 0000000000000000 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977483] R13: ffff880c3fcfa070 R14: 0000000000000020 R15: 131669d94b0671d4 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977486] FS: 0000000000000000(0000) GS:ffff880c3fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977489] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977492] CR2: 00007f8ee7fabbe0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977498] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977501] Process swapper/14 (pid: 0, threadinfo ffff880c2d638000, task ffff880c2d630000) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977503] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977506] ffff880c2d639e48 ffff880c3fcee980 0000000000000003 000000003b99af6d | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977512] ffff880c2d639ea8 0000000e81535457 0000000000000003 ffff880c3fcfa070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977519] 00000000fffffff0 0000000000000003 000000000000000e 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977525] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977531] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977536] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977541] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977547] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977549] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977603] NMI backtrace for cpu 15 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977606] CPU 15 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977608] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977669] | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977672] Pid: 0, comm: swapper/15 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977676] RIP: 0010:[<ffffffff8139a64c>] [<ffffffff8139a64c>] intel_idle+0xbc/0x150 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977682] RSP: 0018:ffff880c2d63be38 EFLAGS: 00000046 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977684] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977686] RDX: 0000000000000000 RSI: ffff880c2d63bfd8 RDI: ffffffff81c18f00 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977688] RBP: ffff880c2d63be98 R08: 00000000fffffffc R09: 0000000000005cf0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977690] R10: 0000000000001f3c R11: dead000000200200 R12: 0000000000000003 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977692] R13: ffff88186fcfa070 R14: 0000000000000020 R15: 131669da0fd6ee67 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977695] FS: 0000000000000000(0000) GS:ffff88186fce0000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977697] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977699] CR2: 00007ff8e4688be0 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977701] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977703] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977706] Process swapper/15 (pid: 0, threadinfo ffff880c2d63a000, task ffff880c2d631700) | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977707] Stack: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977709] 0000000000000082 ffff88186fceeb80 0000000000000000 00000000016b0a13 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977714] ffff880c2d63bea8 0000000f81535457 0000000000000000 ffff88186fcfa070 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977719] 00000000fffffff0 0000000000000003 000000000000000f 0000000000000000 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977723] Call Trace: | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977728] [<ffffffff815338c9>] cpuidle_enter+0x19/0x20 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977732] [<ffffffff81533eec>] cpuidle_idle_call+0xac/0x2a0 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977736] [<ffffffff8101d87f>] cpu_idle+0xcf/0x120 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977740] [<ffffffff81679abf>] start_secondary+0xc3/0xc5 | |
| Aug 1 17:25:21 storage1 kernel: [2263859.977742] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 2f 98 d0 ff 4c 29 f8 48 89 c7 e8 74 01 cc ff 4c 69 f0 40 42 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507530] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507900] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507969] CPU 6 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.507971] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508026] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508028] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508032] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508042] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508044] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508046] RDX: 0000000000000002 RSI: 000000000000006f RDI: 0000000000000380 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508048] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508050] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508052] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508054] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508057] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508059] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508065] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508067] Stack: | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508068] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508074] 00080fe7d95876b7 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508078] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508083] Call Trace: | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508085] <IRQ> | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508087] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508097] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508103] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508107] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508111] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508116] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508118] <EOI> | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508120] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508128] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508131] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508135] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508149] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508159] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508170] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508204] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508224] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508230] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508234] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508240] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508244] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508247] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508252] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508256] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508260] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508263] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:25:46 storage1 kernel: [2263884.508264] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.440825] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441298] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441373] CPU 6 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441374] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441429] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441432] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441436] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441445] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441448] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441450] RDX: 0000000000000002 RSI: 000000000000006d RDI: 0000000000000380 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441451] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441453] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441455] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441458] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441460] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441462] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441466] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441469] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441470] Stack: | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441472] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441477] 00080fee5e468e6a ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441482] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441486] Call Trace: | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441489] <IRQ> | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441491] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441500] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441507] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441511] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441515] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441520] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441521] <EOI> | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441523] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441532] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441535] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441539] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441553] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441563] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441573] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441606] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441627] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441632] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441637] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441642] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441646] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441650] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441655] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441658] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441662] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441665] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:26:14 storage1 kernel: [2263912.441667] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.374184] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386693] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386769] CPU 6 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386770] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386825] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386828] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386832] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386841] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386844] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386846] RDX: 0000000000000002 RSI: 0000000000000070 RDI: 0000000000000380 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386847] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386849] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386851] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386854] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386856] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386858] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386865] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386866] Stack: | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386868] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386873] 00080ff4e334a596 ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386878] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386883] Call Trace: | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386885] <IRQ> | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386887] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386896] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386903] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386907] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386911] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386916] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386918] <EOI> | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386920] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386928] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386932] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386935] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386949] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386960] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.386970] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387004] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387024] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387030] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387034] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387040] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387044] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387048] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387052] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387056] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387060] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387063] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:26:42 storage1 kernel: [2263940.387064] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.307506] BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:84] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320178] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320253] CPU 6 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320254] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320309] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320312] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320316] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320326] RSP: 0018:ffff880c3fc63ee0 EFLAGS: 00000206 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320328] RAX: 0000000000000006 RBX: ffff880c3fc6e980 RCX: 0000000000000020 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320330] RDX: 0000000000000002 RSI: 000000000000006e RDI: 0000000000000380 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320332] RBP: ffff880c3fc63f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320334] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc63e58 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320336] R13: ffffffff816a610a R14: ffff880c3fc63f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320338] FS: 0000000000000000(0000) GS:ffff880c3fc60000(0000) knlGS:0000000000000000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320341] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320343] CR2: 00007f539a09b000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320347] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320349] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320351] Stack: | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320353] ffffffff81c18f00 ffff880c2a451fd8 000000060000000a ffff880c2a451fd8 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320358] 00080ffb6822bdaf ffff880c3fc63f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320363] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc902dd730000 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320367] Call Trace: | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320369] <IRQ> | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320372] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320381] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320388] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320391] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320396] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320401] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320402] <EOI> | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320404] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320412] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320416] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320420] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320434] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320444] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320454] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320488] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320509] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320515] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320519] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320524] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320528] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320532] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320537] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320540] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320544] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320547] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:27:10 storage1 kernel: [2263968.320549] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:30:01 storage1 CRON[27052]: (root) CMD (zfs-auto-snapshot -q -g --label=frequent --keep=4 //) | |
| Aug 1 17:30:33 storage1 zfs-auto-snap: @zfs-auto-snap_frequent-2013-08-01-1730, 3 created, 3 destroyed, 0 warnings. | |
| Aug 1 17:31:06 storage1 kernel: [2264203.717589] BUG: soft lockup - CPU#4 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733215] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733290] CPU 4 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733291] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733347] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733350] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733354] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733363] RSP: 0018:ffff880c3fc43ee0 EFLAGS: 00000206 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733365] RAX: 0000000000000004 RBX: ffff880c3fc4e980 RCX: 0000000000000020 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733368] RDX: 0000000000000002 RSI: 0000000000000063 RDI: 0000000000000380 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733369] RBP: ffff880c3fc43f40 R08: 0000000000000001 R09: 0000000000000100 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733371] R10: 0000000000000020 R11: 0000000000000000 R12: ffff880c3fc43e58 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733373] R13: ffffffff816a610a R14: ffff880c3fc43f40 R15: ffff880c2a451fd8 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733376] FS: 0000000000000000(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733378] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733380] CR2: 00007f0ecd411000 CR3: 0000000001c0b000 CR4: 00000000000007e0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733384] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733387] Process kswapd0 (pid: 84, threadinfo ffff880c2a450000, task ffff880c2abd1700) | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733388] Stack: | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733390] ffffffff81c18f00 ffff880c2a451fd8 000000040000000a ffff880c2a451fd8 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733396] 000810325929b636 ffff880c3fc43f18 ffffffff810ab4b4 0000000000000046 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733400] ffff880c2a451fd8 0000000000000001 ffff88182bf03840 ffffc905d3ab7000 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733405] Call Trace: | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733407] <IRQ> | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733410] [<ffffffff810ab4b4>] ? tick_program_event+0x24/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733419] [<ffffffff816a6a5c>] call_softirq+0x1c/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733426] [<ffffffff81016235>] do_softirq+0x65/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733430] [<ffffffff8105be0e>] irq_exit+0x8e/0xb0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733434] [<ffffffff816a73be>] smp_apic_timer_interrupt+0x6e/0x99 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733439] [<ffffffff816a610a>] apic_timer_interrupt+0x6a/0x70 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733441] <EOI> | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733443] [<ffffffff8115e786>] ? remove_vm_area+0x86/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733451] [<ffffffff8115e86d>] __vunmap.part.16+0x1d/0xc0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733455] [<ffffffff8115e92b>] __vunmap+0x1b/0x40 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733458] [<ffffffff8115ea5a>] vfree+0x2a/0x30 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733472] [<ffffffffa014a0c8>] kv_free.isra.5+0x68/0x70 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733483] [<ffffffffa014c7d5>] spl_slab_reclaim+0x335/0x440 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733493] [<ffffffffa014c9ea>] spl_kmem_cache_reap_now+0x10a/0x230 [spl] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733529] [<ffffffffa0330e90>] arc_kmem_reap_now+0x60/0xb0 [zfs] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733561] [<ffffffffa0330f9f>] arc_shrinker_func+0xbf/0x120 [zfs] | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733572] [<ffffffff81139a14>] shrink_slab+0x154/0x300 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733578] [<ffffffff8117f1c8>] ? mem_cgroup_iter+0xe8/0x200 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733586] [<ffffffff8113cb24>] balance_pgdat+0x5a4/0x720 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733592] [<ffffffff8113cdc3>] kswapd+0x123/0x240 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733599] [<ffffffff8113cca0>] ? balance_pgdat+0x720/0x720 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733605] [<ffffffff81077ca3>] kthread+0x93/0xa0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733611] [<ffffffff816a6964>] kernel_thread_helper+0x4/0x10 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733617] [<ffffffff81077c10>] ? flush_kthread_worker+0xb0/0xb0 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733621] [<ffffffff816a6960>] ? gs_change+0x13/0x13 | |
| Aug 1 17:31:06 storage1 kernel: [2264203.733623] Code: 04 25 50 dc 00 00 c7 45 b0 0a 00 00 00 89 45 b4 4c 89 7d b8 4c 89 7d a8 0f 1f 00 65 c7 04 25 40 15 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 40 c0 81 66 0f 1f 44 00 00 41 f6 c5 01 74 | |
| Aug 1 17:31:34 storage1 kernel: [2264231.650941] BUG: soft lockup - CPU#4 stuck for 23s! [kswapd0:84] | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663177] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663268] CPU 4 | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663270] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs iscsi_trgt(O) vesafb nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp kvm_intel kvm ghash_clmulni_intel aesni_intel cryptd xt_hl ip6t_rt aes_x86_64 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG joydev microcode ioatdma i7core_edac lpc_ich edac_core xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter mac_hid ip_tables x_tables lp parport ext2 zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) zlib_deflate ses enclosure hid_generic usbhid hid lpfc scsi_transport_fc scsi_tgt mpt2sas scsi_transport_sas raid_class bnx2x igb libcrc32c mdio dca | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663325] | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663328] Pid: 84, comm: kswapd0 Tainted: P O 3.5.0-34-generic #55~precise1-Ubuntu Intel Corporation S5520UR/S5520UR | |
| Aug 1 17:31:34 storage1 kernel: [2264231.663332] RIP: 0010:[<ffffffff8105b9e0>] [<ffffffff8105b9e0>] __do_softirq+0x60/0x210 | |
| Aug 1 17:31:34 Aug 1 18:03:08 storage1 kernel: imklog 5.8.6, log source = /proc/kmsg started. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment