Created
March 17, 2016 08:55
-
-
Save n-soda/c447137afb1523378bb0 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gfsd: page allocation failure. order:5, mode:0xd0 | |
| Pid: 31349, comm: gfsd Not tainted 2.6.32-279.22.1.el6.x86_64 #1 | |
| Call Trace: | |
| [<ffffffff8112343f>] ? __alloc_pages_nodemask+0x77f/0x940 | |
| [<ffffffff8115d3e2>] ? kmem_getpages+0x62/0x170 | |
| [<ffffffff8115dffa>] ? fallback_alloc+0x1ba/0x270 | |
| [<ffffffff8115da4f>] ? cache_grow+0x2cf/0x320 | |
| [<ffffffff8115dd79>] ? ____cache_alloc_node+0x99/0x160 | |
| [<ffffffff8115ef40>] ? kmem_cache_alloc_node_trace+0x90/0x200 | |
| [<ffffffff8115f15d>] ? __kmalloc_node+0x4d/0x60 | |
| [<ffffffff8141edfd>] ? __alloc_skb+0x6d/0x190 | |
| [<ffffffff8146c941>] ? sk_stream_alloc_skb+0x41/0x110 | |
| [<ffffffff8146cd60>] ? tcp_sendmsg+0x350/0xa10 | |
| [<ffffffff812740ad>] ? copy_user_generic_string+0x2d/0x40 | |
| [<ffffffff81416ee3>] ? sock_sendmsg+0x123/0x150 | |
| [<ffffffff81090be0>] ? autoremove_wake_function+0x0/0x40 | |
| [<ffffffffa0709530>] ? xfs_rw_iunlock+0x20/0x40 [xfs] | |
| [<ffffffffa0709e43>] ? xfs_file_aio_read+0x1b3/0x2d0 [xfs] | |
| [<ffffffff8117660a>] ? do_sync_read+0xfa/0x140 | |
| [<ffffffff814171f9>] ? sys_sendto+0x139/0x190 | |
| [<ffffffff81176f6f>] ? vfs_read+0x12f/0x1a0 | |
| [<ffffffff810d3f47>] ? audit_syscall_entry+0x1d7/0x200 | |
| [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b | |
| Mem-Info: | |
| Node 0 DMA per-cpu: | |
| CPU 0: hi: 0, btch: 1 usd: 0 | |
| CPU 1: hi: 0, btch: 1 usd: 0 | |
| CPU 2: hi: 0, btch: 1 usd: 0 | |
| CPU 3: hi: 0, btch: 1 usd: 0 | |
| CPU 4: hi: 0, btch: 1 usd: 0 | |
| CPU 5: hi: 0, btch: 1 usd: 0 | |
| CPU 6: hi: 0, btch: 1 usd: 0 | |
| CPU 7: hi: 0, btch: 1 usd: 0 | |
| CPU 8: hi: 0, btch: 1 usd: 0 | |
| CPU 9: hi: 0, btch: 1 usd: 0 | |
| CPU 10: hi: 0, btch: 1 usd: 0 | |
| CPU 11: hi: 0, btch: 1 usd: 0 | |
| CPU 12: hi: 0, btch: 1 usd: 0 | |
| CPU 13: hi: 0, btch: 1 usd: 0 | |
| CPU 14: hi: 0, btch: 1 usd: 0 | |
| CPU 15: hi: 0, btch: 1 usd: 0 | |
| CPU 16: hi: 0, btch: 1 usd: 0 | |
| CPU 17: hi: 0, btch: 1 usd: 0 | |
| CPU 18: hi: 0, btch: 1 usd: 0 | |
| CPU 19: hi: 0, btch: 1 usd: 0 | |
| CPU 20: hi: 0, btch: 1 usd: 0 | |
| CPU 21: hi: 0, btch: 1 usd: 0 | |
| CPU 22: hi: 0, btch: 1 usd: 0 | |
| CPU 23: hi: 0, btch: 1 usd: 0 | |
| Node 0 DMA32 per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 0 | |
| CPU 5: hi: 186, btch: 31 usd: 0 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 0 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 0 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 0 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| Node 0 Normal per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 0 | |
| CPU 5: hi: 186, btch: 31 usd: 105 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 0 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 0 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 0 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| Node 1 Normal per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 0 | |
| CPU 5: hi: 186, btch: 31 usd: 48 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 28 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 0 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 1 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| active_anon:41260 inactive_anon:8143 isolated_anon:0 | |
| active_file:401439 inactive_file:11238913 isolated_file:0 | |
| unevictable:1800 dirty:983032 writeback:0 unstable:0 | |
| free:73620 slab_reclaimable:205402 slab_unreclaimable:67901 | |
| mapped:5453 shmem:575 pagetables:5602 bounce:0 | |
| Node 0 DMA free:15740kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes | |
| lowmem_reserve[]: 0 3630 24209 24209 | |
| Node 0 DMA32 free:108616kB min:9816kB low:12268kB high:14724kB active_anon:1236kB inactive_anon:344kB active_file:49564kB inactive_file:2827280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3717472kB mlocked:0kB dirty:195780kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:212272kB slab_unreclaimable:8004kB kernel_stack:64kB pagetables:616kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 20578 20578 | |
| Node 0 Normal free:80292kB min:55652kB low:69564kB high:83476kB active_anon:44484kB inactive_anon:15612kB active_file:673440kB inactive_file:19440588kB unevictable:3284kB isolated(anon):0kB isolated(file):0kB present:21072640kB mlocked:3284kB dirty:1164740kB writeback:0kB mapped:11476kB shmem:264kB slab_reclaimable:121260kB slab_unreclaimable:133116kB kernel_stack:5984kB pagetables:10372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 0 0 | |
| Node 1 Normal free:89832kB min:65556kB low:81944kB high:98332kB active_anon:119320kB inactive_anon:16616kB active_file:882752kB inactive_file:22687784kB unevictable:3916kB isolated(anon):0kB isolated(file):0kB present:24821756kB mlocked:3916kB dirty:2571608kB writeback:0kB mapped:10336kB shmem:2036kB slab_reclaimable:488076kB slab_unreclaimable:130484kB kernel_stack:1400kB pagetables:11420kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 0 0 | |
| Node 0 DMA: 3*4kB 2*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB | |
| Node 0 DMA32: 25951*4kB 362*8kB 16*16kB 0*32kB 1*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 108684kB | |
| Node 0 Normal: 16900*4kB 303*8kB 17*16kB 16*32kB 19*64kB 68*128kB 3*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 82520kB | |
| Node 1 Normal: 6251*4kB 6008*8kB 543*16kB 142*32kB 57*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 90204kB | |
| 11642307 total pagecache pages | |
| 0 pages in swap cache | |
| Swap cache stats: add 0, delete 0, find 0/0 | |
| Free swap = 8385920kB | |
| Total swap = 8385920kB | |
| 12582910 pages RAM | |
| 228766 pages reserved | |
| 3131017 pages shared | |
| 9345221 pages non-shared | |
| gfsd: page allocation failure. order:5, mode:0xd0 | |
| Pid: 3316, comm: gfsd Not tainted 2.6.32-279.22.1.el6.x86_64 #1 | |
| Call Trace: | |
| [<ffffffff8112343f>] ? __alloc_pages_nodemask+0x77f/0x940 | |
| [<ffffffff8115d3e2>] ? kmem_getpages+0x62/0x170 | |
| [<ffffffff8115dffa>] ? fallback_alloc+0x1ba/0x270 | |
| [<ffffffff8115da4f>] ? cache_grow+0x2cf/0x320 | |
| [<ffffffff8115dd79>] ? ____cache_alloc_node+0x99/0x160 | |
| [<ffffffff8115ef40>] ? kmem_cache_alloc_node_trace+0x90/0x200 | |
| [<ffffffff8115f15d>] ? __kmalloc_node+0x4d/0x60 | |
| [<ffffffff8141edfd>] ? __alloc_skb+0x6d/0x190 | |
| [<ffffffff8146c941>] ? sk_stream_alloc_skb+0x41/0x110 | |
| [<ffffffff8146cd60>] ? tcp_sendmsg+0x350/0xa10 | |
| [<ffffffff812740ad>] ? copy_user_generic_string+0x2d/0x40 | |
| [<ffffffff81416ee3>] ? sock_sendmsg+0x123/0x150 | |
| [<ffffffff81090be0>] ? autoremove_wake_function+0x0/0x40 | |
| [<ffffffffa0709530>] ? xfs_rw_iunlock+0x20/0x40 [xfs] | |
| [<ffffffffa0709e43>] ? xfs_file_aio_read+0x1b3/0x2d0 [xfs] | |
| [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20 | |
| [<ffffffff8117660a>] ? do_sync_read+0xfa/0x140 | |
| [<ffffffff814171f9>] ? sys_sendto+0x139/0x190 | |
| [<ffffffff81176f6f>] ? vfs_read+0x12f/0x1a0 | |
| [<ffffffff810d3f47>] ? audit_syscall_entry+0x1d7/0x200 | |
| [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b | |
| Mem-Info: | |
| Node 0 DMA per-cpu: | |
| CPU 0: hi: 0, btch: 1 usd: 0 | |
| CPU 1: hi: 0, btch: 1 usd: 0 | |
| CPU 2: hi: 0, btch: 1 usd: 0 | |
| CPU 3: hi: 0, btch: 1 usd: 0 | |
| CPU 4: hi: 0, btch: 1 usd: 0 | |
| CPU 5: hi: 0, btch: 1 usd: 0 | |
| CPU 6: hi: 0, btch: 1 usd: 0 | |
| CPU 7: hi: 0, btch: 1 usd: 0 | |
| CPU 8: hi: 0, btch: 1 usd: 0 | |
| CPU 9: hi: 0, btch: 1 usd: 0 | |
| CPU 10: hi: 0, btch: 1 usd: 0 | |
| CPU 11: hi: 0, btch: 1 usd: 0 | |
| CPU 12: hi: 0, btch: 1 usd: 0 | |
| CPU 13: hi: 0, btch: 1 usd: 0 | |
| CPU 14: hi: 0, btch: 1 usd: 0 | |
| CPU 15: hi: 0, btch: 1 usd: 0 | |
| CPU 16: hi: 0, btch: 1 usd: 0 | |
| CPU 17: hi: 0, btch: 1 usd: 0 | |
| CPU 18: hi: 0, btch: 1 usd: 0 | |
| CPU 19: hi: 0, btch: 1 usd: 0 | |
| CPU 20: hi: 0, btch: 1 usd: 0 | |
| CPU 21: hi: 0, btch: 1 usd: 0 | |
| CPU 22: hi: 0, btch: 1 usd: 0 | |
| CPU 23: hi: 0, btch: 1 usd: 0 | |
| Node 0 DMA32 per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 0 | |
| CPU 5: hi: 186, btch: 31 usd: 0 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 0 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 0 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 0 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| Node 0 Normal per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 1 | |
| CPU 5: hi: 186, btch: 31 usd: 0 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 0 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 44 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 0 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| Node 1 Normal per-cpu: | |
| CPU 0: hi: 186, btch: 31 usd: 0 | |
| CPU 1: hi: 186, btch: 31 usd: 0 | |
| CPU 2: hi: 186, btch: 31 usd: 0 | |
| CPU 3: hi: 186, btch: 31 usd: 0 | |
| CPU 4: hi: 186, btch: 31 usd: 0 | |
| CPU 5: hi: 186, btch: 31 usd: 0 | |
| CPU 6: hi: 186, btch: 31 usd: 0 | |
| CPU 7: hi: 186, btch: 31 usd: 0 | |
| CPU 8: hi: 186, btch: 31 usd: 0 | |
| CPU 9: hi: 186, btch: 31 usd: 0 | |
| CPU 10: hi: 186, btch: 31 usd: 0 | |
| CPU 11: hi: 186, btch: 31 usd: 0 | |
| CPU 12: hi: 186, btch: 31 usd: 0 | |
| CPU 13: hi: 186, btch: 31 usd: 0 | |
| CPU 14: hi: 186, btch: 31 usd: 0 | |
| CPU 15: hi: 186, btch: 31 usd: 0 | |
| CPU 16: hi: 186, btch: 31 usd: 0 | |
| CPU 17: hi: 186, btch: 31 usd: 0 | |
| CPU 18: hi: 186, btch: 31 usd: 0 | |
| CPU 19: hi: 186, btch: 31 usd: 0 | |
| CPU 20: hi: 186, btch: 31 usd: 0 | |
| CPU 21: hi: 186, btch: 31 usd: 0 | |
| CPU 22: hi: 186, btch: 31 usd: 0 | |
| CPU 23: hi: 186, btch: 31 usd: 0 | |
| active_anon:34600 inactive_anon:8083 isolated_anon:0 | |
| active_file:185672 inactive_file:11428469 isolated_file:0 | |
| unevictable:1802 dirty:272056 writeback:0 unstable:0 | |
| free:91255 slab_reclaimable:227098 slab_unreclaimable:58374 | |
| mapped:5419 shmem:575 pagetables:3532 bounce:0 | |
| Node 0 DMA free:15740kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes | |
| lowmem_reserve[]: 0 3630 24209 24209 | |
| Node 0 DMA32 free:186056kB min:9816kB low:12268kB high:14724kB active_anon:0kB inactive_anon:104kB active_file:16180kB inactive_file:2893752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3717472kB mlocked:0kB dirty:64456kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:78540kB slab_unreclaimable:16404kB kernel_stack:16kB pagetables:40kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 20578 20578 | |
| Node 0 Normal free:74208kB min:55652kB low:69564kB high:83476kB active_anon:34676kB inactive_anon:15612kB active_file:439788kB inactive_file:19710300kB unevictable:3284kB isolated(anon):0kB isolated(file):0kB present:21072640kB mlocked:3284kB dirty:923132kB writeback:0kB mapped:11844kB shmem:264kB slab_reclaimable:89508kB slab_unreclaimable:149296kB kernel_stack:5896kB pagetables:6580kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 0 0 | |
| Node 1 Normal free:89016kB min:65556kB low:81944kB high:98332kB active_anon:103724kB inactive_anon:16616kB active_file:286720kB inactive_file:23109824kB unevictable:3924kB isolated(anon):0kB isolated(file):0kB present:24821756kB mlocked:3924kB dirty:100636kB writeback:0kB mapped:9832kB shmem:2036kB slab_reclaimable:740344kB slab_unreclaimable:67796kB kernel_stack:1280kB pagetables:7508kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no | |
| lowmem_reserve[]: 0 0 0 0 | |
| Node 0 DMA: 3*4kB 2*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB | |
| Node 0 DMA32: 34416*4kB 5853*8kB 40*16kB 11*32kB 5*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 186056kB | |
| Node 0 Normal: 16513*4kB 619*8kB 37*16kB 12*32kB 9*64kB 19*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 76012kB | |
| Node 1 Normal: 18873*4kB 739*8kB 29*16kB 13*32kB 16*64kB 50*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 90476kB | |
| 11616087 total pagecache pages | |
| 0 pages in swap cache | |
| Swap cache stats: add 0, delete 0, find 0/0 | |
| Free swap = 8385920kB | |
| Total swap = 8385920kB | |
| 12582910 pages RAM | |
| 228766 pages reserved | |
| 4377109 pages shared | |
| 7939445 pages non-shared | |
| ---- | |
| sshd: page allocation failure. order:1, mode:0x20 | |
| Pid: 7998, comm: sshd Not tainted 2.6.32-279.22.1.el6.x86_64 #1 | |
| Call Trace: | |
| <IRQ> [<ffffffff8112343f>] ? __alloc_pages_nodemask+0x77f/0x940 | |
| [<ffffffff8115d3e2>] ? kmem_getpages+0x62/0x170 | |
| [<ffffffff8115dffa>] ? fallback_alloc+0x1ba/0x270 | |
| [<ffffffff8115da4f>] ? cache_grow+0x2cf/0x320 | |
| [<ffffffff8115dd79>] ? ____cache_alloc_node+0x99/0x160 | |
| [<ffffffff8115ef40>] ? kmem_cache_alloc_node_trace+0x90/0x200 | |
| [<ffffffff8115f15d>] ? __kmalloc_node+0x4d/0x60 | |
| [<ffffffff8141edfd>] ? __alloc_skb+0x6d/0x190 | |
| [<ffffffff8141ef3d>] ? dev_alloc_skb+0x1d/0x40 | |
| [<ffffffffa056d290>] ? ipoib_cm_alloc_rx_skb+0x30/0x450 [ib_ipoib] | |
| [<ffffffffa056e5cf>] ? ipoib_cm_handle_rx_wc+0x2af/0x7a0 [ib_ipoib] | |
| [<ffffffffa0441b33>] ? mlx4_ib_poll_cq+0x2c3/0x890 [mlx4_ib] | |
| [<ffffffffa056d0b3>] ? ipoib_cm_handle_tx_wc+0x193/0x340 [ib_ipoib] | |
| [<ffffffffa0566365>] ? ipoib_poll+0x115/0x1d0 [ib_ipoib] | |
| [<ffffffff8142dba3>] ? net_rx_action+0x103/0x2f0 | |
| [<ffffffff81072ac1>] ? __do_softirq+0xc1/0x1e0 | |
| [<ffffffff810d8d60>] ? handle_IRQ_event+0x60/0x170 | |
| [<ffffffff81072b1f>] ? __do_softirq+0x11f/0x1e0 | |
| [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30 | |
| [<ffffffff8100de05>] ? do_softirq+0x65/0xa0 | |
| [<ffffffff810728a5>] ? irq_exit+0x85/0x90 | |
| [<ffffffff814f2275>] ? do_IRQ+0x75/0xf0 | |
| [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11 | |
| <EOI> | |
| ---- | |
| ksoftirqd/19: page allocation failure. order:1, mode:0x20 | |
| Pid: 81, comm: ksoftirqd/19 Not tainted 2.6.32-279.22.1.el6.x86_64 #1 | |
| Call Trace: | |
| <IRQ> [<ffffffff8112343f>] ? __alloc_pages_nodemask+0x77f/0x940 | |
| [<ffffffff8115d3e2>] ? kmem_getpages+0x62/0x170 | |
| [<ffffffff8115dffa>] ? fallback_alloc+0x1ba/0x270 | |
| [<ffffffff8115dd79>] ? ____cache_alloc_node+0x99/0x160 | |
| [<ffffffff8115ef40>] ? kmem_cache_alloc_node_trace+0x90/0x200 | |
| [<ffffffff8115f15d>] ? __kmalloc_node+0x4d/0x60 | |
| [<ffffffff8141edfd>] ? __alloc_skb+0x6d/0x190 | |
| [<ffffffff8141ef3d>] ? dev_alloc_skb+0x1d/0x40 | |
| [<ffffffffa0576290>] ? ipoib_cm_alloc_rx_skb+0x30/0x450 [ib_ipoib] | |
| [<ffffffffa05775cf>] ? ipoib_cm_handle_rx_wc+0x2af/0x7a0 [ib_ipoib] | |
| [<ffffffffa056f365>] ? ipoib_poll+0x115/0x1d0 [ib_ipoib] | |
| [<ffffffff8142dba3>] ? net_rx_action+0x103/0x2f0 | |
| [<ffffffff81072ac1>] ? __do_softirq+0xc1/0x1e0 | |
| [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30 | |
| <EOI> [<ffffffff8100de05>] ? do_softirq+0x65/0xa0 | |
| [<ffffffff81072620>] ? ksoftirqd+0x80/0x110 | |
| [<ffffffff810725a0>] ? ksoftirqd+0x0/0x110 | |
| [<ffffffff81090876>] ? kthread+0x96/0xa0 | |
| [<ffffffff8100c0ca>] ? child_rip+0xa/0x20 | |
| [<ffffffff810907e0>] ? kthread+0x0/0xa0 | |
| [<ffffffff8100c0c0>] ? child_rip+0x0/0x20 | |
| ---- | |
| by default, vm.min_free_kbytes=131072 on This CentOS 6 machine. | |
| setting the following sysctl parameters fixed this problem: | |
| vm.min_free_kbytes = 2097152 | |
| vm.zone_reclaim_mode = 1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment