4.4.51-40.58.amzn1.x86_64 ena output

gistfile1.txt

May 17 06:30:26 ip-172-16-15-126 consul[13099]:     2017/05/17 06:30:26 [INFO] serf: attempting reconnect to i-0ffe3ee8de2922539 172.16.14.98:8301
May 17 06:30:46 ip-172-16-15-126 consul[13099]:     2017/05/17 06:30:46 [INFO] agent: Synced node info
May 17 06:31:35 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 109170ms.
May 17 06:31:39 ip-172-16-15-126 consul[13099]:     2017/05/17 06:31:39 [WARN] memberlist: Refuting a suspect message (from: i-09ee29f77300a149d)
May 17 06:32:45 ip-172-16-15-126 consul[13099]:     2017/05/17 06:32:45 [INFO] serf: EventMemberJoin: i-03549fcaa8f4e1636 172.16.15.66
May 17 06:32:49 ip-172-16-15-126 consul[13099]:     2017/05/17 06:32:49 [INFO] memberlist: Marking i-057b08569e51a333d as failed, suspect timeout reached
May 17 06:32:49 ip-172-16-15-126 consul[13099]:     2017/05/17 06:32:49 [INFO] serf: EventMemberLeave: i-057b08569e51a333d 172.16.16.105
May 17 06:32:53 ip-172-16-15-126 consul[13099]:     2017/05/17 06:32:53 [WARN] memberlist: Refuting a suspect message (from: i-025dd16dcdd9a82c7)
May 17 06:33:15 ip-172-16-15-126 consul[13099]:     2017/05/17 06:33:15 [ERR] memberlist: failed to receive: EOF from=172.16.16.104:50079
May 17 06:33:15 ip-172-16-15-126 consul[13099]:     2017/05/17 06:33:15 [WARN] memberlist: Refuting a suspect message (from: i-097afca0ba61c8d02)
May 17 06:33:24 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 116940ms.
May 17 06:33:28 ip-172-16-15-126 consul[13099]:     2017/05/17 06:33:28 [ERR] memberlist: failed to receive: EOF from=172.16.17.196:49281
May 17 06:33:28 ip-172-16-15-126 consul[13099]:     2017/05/17 06:33:28 [WARN] memberlist: Refuting a suspect message (from: i-025dd16dcdd9a82c7)
May 17 06:34:22 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:22 [INFO] memberlist: Suspect i-0773de01abcf24ae9 has failed, no acks received
May 17 06:34:23 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:23 [INFO] memberlist: Suspect i-055f50877f84f0725 has failed, no acks received
May 17 06:34:24 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:24 [INFO] memberlist: Suspect i-0edd0cf9d6f372a8c has failed, no acks received
May 17 06:34:25 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:25 [INFO] memberlist: Suspect i-0a8cace27cc324e0f has failed, no acks received
May 17 06:34:26 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:26 [INFO] memberlist: Suspect i-0bd3ab67982f1fe55 has failed, no acks received
May 17 06:34:27 ip-172-16-15-126 consul[13099]:     2017/05/17 06:34:27 [INFO] memberlist: Suspect i-095ff5d9c84db7aed has failed, no acks received
May 17 06:35:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 128890ms.
May 17 06:37:30 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 110280ms.
May 17 06:39:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 123010ms.
May 17 06:41:24 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 114970ms.
May 17 06:43:19 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 124270ms.
May 17 06:45:23 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 117460ms.
May 17 06:47:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 124890ms.
May 17 06:49:25 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 112970ms.
May 17 06:50:19 ip-172-16-15-126 kernel: [365453.512122] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 613.
May 17 06:50:19 ip-172-16-15-126 kernel: [365453.516180] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 614.
May 17 06:50:19 ip-172-16-15-126 kernel: [365495.521705] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 818.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.492061] INFO: task jbd2/xvda1-8:2324 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.496712]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.501576] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.507470] jbd2/xvda1-8    D ffff881e70553b08     0  2324      2 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.512832]  ffff881e70553b08 ffff881e77ab0000 ffff881e70eb57c0 ffff881e70554000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.519031]  0000000000000000 7fffffffffffffff ffff881efff9a0b8 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.525187]  ffff881e70553b20 ffffffff814ec5f5 ffff881e84695340 ffff881e70553bb8
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.531344] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.533441]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.537239]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.540853]  [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.544944]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.549709]  [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.553585]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.557396]  [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.561784]  [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.565713]  [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.569825]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.573634]  [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.578173]  [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.582901]  [<ffffffff812088c7>] __wait_on_buffer+0x27/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.587066]  [<ffffffffa00cf2f6>] jbd2_journal_commit_transaction+0xf66/0x1bb0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.592915]  [<ffffffffa00d3c90>] kjournald2+0x120/0x340 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.597189]  [<ffffffff810acee0>] ? prepare_to_wait_event+0xf0/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.601767]  [<ffffffffa00d3b70>] ? commit_timeout+0x10/0x10 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.606347]  [<ffffffff8108a7f9>] kthread+0xc9/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.610010]  [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.614071]  [<ffffffff814f02cf>] ret_from_fork+0x3f/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.618088]  [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.622168] INFO: task dmeventd:3049 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.626981]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.631856] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.637761] dmeventd        D ffff881e728db8c8     0  3049      1 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.642896]  ffff881e728db8c8 ffff881e77a99d40 ffff881e73598000 ffff881e728dc000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.648990]  0000000000000000 7fffffffffffffff ffff881efff9a478 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.655237]  ffff881e728db8e0 ffffffff814ec5f5 ffff881e84635340 ffff881e728db978
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.661486] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.663586]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.667459]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.671223]  [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.675580]  [<ffffffff8135f659>] ? notify_remote_via_irq+0x29/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.680217]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.685124]  [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.689046]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.692915]  [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.697311]  [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.701238]  [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.705287]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.709137]  [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.713755]  [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.718642]  [<ffffffffa08abe83>] dm_bufio_write_dirty_buffers+0xe3/0x190 [dm_bufio]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.724344]  [<ffffffffa08bd0bc>] dm_bm_flush+0x1c/0x20 [dm_persistent_data]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.729461]  [<ffffffffa08bffc3>] dm_tm_commit+0x33/0x40 [dm_persistent_data]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.735149]  [<ffffffffa08d8d39>] __commit_transaction+0x359/0x360 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.740735]  [<ffffffffa08d9f47>] dm_pool_commit_metadata+0x37/0x60 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.746227]  [<ffffffffa08d3967>] commit+0x27/0x90 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.750505]  [<ffffffffa08d3cf2>] pool_status+0x322/0x6b0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.755227]  [<ffffffff811cd5c6>] ? __memcg_kmem_charge+0x46/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.759749]  [<ffffffffa0889c96>] retrieve_status+0xa6/0x1b0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.764474]  [<ffffffffa088ae14>] table_status+0x54/0x80 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.768939]  [<ffffffffa088adc0>] ? dm_get_live_or_inactive_table.isra.3+0x30/0x30 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.775238]  [<ffffffffa088aae6>] ctl_ioctl+0x246/0x4b0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.778569]  [<ffffffff8107932b>] ? recalc_sigpending+0x1b/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.781825]  [<ffffffffa088ad63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.785157]  [<ffffffff811e772d>] do_vfs_ioctl+0x2cd/0x4b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.788226]  [<ffffffff8107c27b>] ? __set_current_blocked+0x4b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.791642]  [<ffffffff811e7989>] SyS_ioctl+0x79/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.794518]  [<ffffffff8107c4a7>] ? SyS_rt_sigprocmask+0x67/0xb0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.797936]  [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.801516] INFO: task rs:main Q:Reg:3401 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.805308]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.808942] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.813403] rs:main Q:Reg   D ffff881e6a3bf968     0  3401      1 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.817403]  ffff881e6a3bf968 ffff881e77ab0000 ffff881e71ddba80 ffff881e6a3c0000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.822043]  0000000000000000 7fffffffffffffff ffff881efff92a08 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.840036]  ffff881e6a3bf980 ffffffff814ec5f5 ffff881e84695340 ffff881e6a3bfa20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.844684] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.846326]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.850013]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.853482]  [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.857646]  [<ffffffff8142c315>] ? netif_rx_internal+0x35/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.861887]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.866587]  [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.870317]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.874015]  [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.878223]  [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.881947]  [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.885801]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.889444]  [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.893829]  [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.898380]  [<ffffffffa00ccdb9>] do_get_write_access+0x279/0x4c0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.903081]  [<ffffffff8120918e>] ? __find_get_block+0xae/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.907299]  [<ffffffffa00cd031>] jbd2_journal_get_write_access+0x31/0x60 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.912781]  [<ffffffffa0136511>] __ext4_journal_get_write_access+0x31/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.918323]  [<ffffffffa01080da>] ext4_reserve_inode_write+0x6a/0x90 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.923190]  [<ffffffffa010b888>] ? ext4_dirty_inode+0x48/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.927728]  [<ffffffffa010813f>] ext4_mark_inode_dirty+0x3f/0x1f0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.932491]  [<ffffffffa010b888>] ext4_dirty_inode+0x48/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.936911]  [<ffffffff812009e1>] __mark_inode_dirty+0x181/0x370
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.941163]  [<ffffffff811ee887>] generic_update_time+0x87/0xd0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.945363]  [<ffffffff811edfd9>] file_update_time+0x99/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.949398]  [<ffffffff810dc784>] ? get_futex_key+0x194/0x270
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.953487]  [<ffffffff81163f1c>] __generic_file_write_iter+0x9c/0x1e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.958070]  [<ffffffff810dcd41>] ? futex_wake+0x81/0x150
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.961918]  [<ffffffffa00feb76>] ext4_file_write_iter+0xf6/0x360 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.966561]  [<ffffffff810dec50>] ? do_futex+0x100/0xad0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.970384]  [<ffffffff81411c68>] ? sock_sendmsg+0x38/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.974311]  [<ffffffff814120e9>] ? SYSC_sendto+0x139/0x170
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.978286]  [<ffffffff811d4dba>] __vfs_write+0xaa/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.982034]  [<ffffffff811d53e2>] vfs_write+0xa2/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.985716]  [<ffffffff811d60e6>] SyS_write+0x46/0xa0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.989368]  [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.993848] INFO: task agent:43861 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.998607]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.003736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.009987] agent           D ffff880c4fdf7860     0 43861  13580 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.015615]  ffff880c4fdf7860 ffff881e77ab0000 ffff881d07de57c0 ffff880c4fdf8000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.022129]  ffff881e70ff6558 ffff881e70ff6570 ffff880c4fdf7990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.028637]  ffff880c4fdf7878 ffffffff814ec5f5 ffff881d07de57c0 ffff880c4fdf78c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.035142] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.037302]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.041067]  [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.045637]  [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.050637]  [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.054694]  [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.060370]  [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.065245]  [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.069759]  [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.075500]  [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.080323]  [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.084977]  [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.089021]  [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.094118]  [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.098800]  [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.103485]  [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.108597]  [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.113007]  [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.118027]  [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.122134]  [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.126725]  [<ffffffff810b3501>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.133188]  [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.137660]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.142909]  [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.147324]  [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.151830] INFO: task consul:15336 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.157299]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.162777] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.169393] consul          D ffff881e4f6a3860     0 15336  15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.175451]  ffff881e4f6a3860 ffffffff81a11500 ffff881e49d2ba80 ffff881e4f6a4000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.182413]  ffff881e70ff6558 ffff881e70ff6570 ffff881e4f6a3990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.189346]  ffff881e4f6a3878 ffffffff814ec5f5 ffff881e49d2ba80 ffff881e4f6a38c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.196075] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.198259]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.202177]  [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.206997]  [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.212061]  [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.216155]  [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.221855]  [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.226781]  [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.231363]  [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.237145]  [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.241819]  [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.246294]  [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.250354]  [<ffffffff8117028e>] ? lru_cache_add+0xe/0x10
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.254594]  [<ffffffff81162c1d>] ? add_to_page_cache_lru+0x4d/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.259375]  [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.264526]  [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.269243]  [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.273910]  [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.278920]  [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.283247]  [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.288152]  [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.292114]  [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.296529]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.301583]  [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.306036]  [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.310156]  [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.314101] INFO: task consul:15339 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.318987]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.324015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.330096] consul          D ffff881e4f69bd98     0 15339  15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.335620]  ffff881e4f69bd98 ffff881e77a9ba80 ffff881e51918000 ffff881e4f69c000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.341683]  ffff881e749dbc80 ffff881e749dbc68 ffffffff00000000 ffffffff00000002
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.347735]  ffff881e4f69bdb0 ffffffff814ec5f5 ffff881e51918000 ffff881e4f69be30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.354237] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.356452]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.360397]  [<ffffffff814ee9fa>] rwsem_down_write_failed+0x1da/0x320
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.365314]  [<ffffffff810a03b5>] ? pick_next_entity+0xa5/0x160
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.369767]  [<ffffffff812dac03>] call_rwsem_down_write_failed+0x13/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.374620]  [<ffffffff814ee304>] ? down_write+0x24/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.378673]  [<ffffffff8117d53b>] vm_mmap_pgoff+0x6b/0xc0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.382693]  [<ffffffff811955b3>] SyS_mmap_pgoff+0x103/0x260
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.386914]  [<ffffffff810d240a>] ? ktime_get_ts64+0x4a/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.391291]  [<ffffffff8101b292>] SyS_mmap+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.395186]  [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.400120] INFO: task consul:15340 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.405105]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.410223] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.416474] consul          D ffff881e4f657860     0 15340  15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.422156]  ffff881e4f657860 ffff881e77aeba80 ffff881e69c9ba80 ffff881e4f658000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.428666]  ffff881e70ff6558 ffff881e70ff6570 ffff881e4f657990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.435155]  ffff881e4f657878 ffffffff814ec5f5 ffff881e69c9ba80 ffff881e4f6578c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.441440] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.443518]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.447315]  [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.452132]  [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.457138]  [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.461221]  [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.466932]  [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.471825]  [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.476386]  [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.482138]  [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.486966]  [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.491617]  [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.495686]  [<ffffffff8117028e>] ? lru_cache_add+0xe/0x10
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.499933]  [<ffffffff81162c1d>] ? add_to_page_cache_lru+0x4d/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.504756]  [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.509903]  [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.514668]  [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.519387]  [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.524455]  [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.528782]  [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.533708]  [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.537682]  [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.541997]  [<ffffffff8147670a>] ? tcp_poll+0x6a/0x240
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.545909]  [<ffffffff810b3501>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.552179]  [<ffffffff8121a631>] ? SyS_epoll_ctl+0xd1/0xca0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.556523]  [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.574325]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.579398]  [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.583584]  [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.587630] INFO: task jbd2/dm-10-8:16273 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.593052]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.598254] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.604597] jbd2/dm-10-8    D ffff881df5e17b08     0 16273      2 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.610381]  ffff881df5e17b08 ffffffff81a11500 ffff881e51bcba80 ffff881df5e18000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.617026]  0000000000000000 7fffffffffffffff ffff881efff94478 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.623543]  ffff881df5e17b20 ffffffff814ec5f5 ffff881e84615340 ffff881df5e17bb8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.630097] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.632304]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.636368]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.640199]  [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.644499]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.649548]  [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.653643]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.657688]  [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.662327]  [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.666432]  [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.670655]  [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.674676]  [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.679505]  [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.684492]  [<ffffffff812088c7>] __wait_on_buffer+0x27/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.688881]  [<ffffffffa00cf2f6>] jbd2_journal_commit_transaction+0xf66/0x1bb0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.695040]  [<ffffffffa00d3c90>] kjournald2+0x120/0x340 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.699622]  [<ffffffff810acee0>] ? prepare_to_wait_event+0xf0/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.704476]  [<ffffffffa00d3b70>] ? commit_timeout+0x10/0x10 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.709312]  [<ffffffff8108a7f9>] kthread+0xc9/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.713179]  [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.717631]  [<ffffffff814f02cf>] ret_from_fork+0x3f/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.721865]  [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.726156] INFO: task java:16402 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.731055]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.736209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.742194] java            D ffff881e6bc37e08     0 16402  16351 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.747635]  ffff881e6bc37e08 ffff881e77a99d40 ffff881e61288000 ffff881e6bc38000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.754131]  ffff881e75cd9340 ffff881e75cd9328 ffffffff00000000 ffffffff00000001
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.760631]  ffff881e6bc37e20 ffffffff814ec5f5 ffff881e61288000 ffff881e6bc37ea0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.767086] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.769269]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.773139]  [<ffffffff814ee9fa>] rwsem_down_write_failed+0x1da/0x320
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.778030]  [<ffffffff812dac03>] call_rwsem_down_write_failed+0x13/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.783083]  [<ffffffff814ee304>] ? down_write+0x24/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.787210]  [<ffffffff811987d1>] SyS_mprotect+0xc1/0x240
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.791414]  [<ffffffff810cefd1>] ? posix_ktime_get_ts+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.796010]  [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.800940] INFO: task java:16445 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.805799]       Tainted: G            E   4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.810897] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.817115] java            D ffff881debbd3e10     0 16445  16351 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.822756]  ffff881debbd3e10 ffffffff81a11500 ffff881e6bd2ba80 ffff881debbd4000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.829278]  ffff881e75cd9328 ffff881e75cd9340 ffff881debbd3f58 ffff881e6bd2ba80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.835788]  ffff881debbd3e28 ffffffff814ec5f5 ffff881e6bd2ba80 ffff881debbd3e78
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.842069] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.844150]  [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.847945]  [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.852755]  [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.857772]  [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.861861]  [<ffffffff8105ef39>] __do_page_fault+0x389/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.866301]  [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.871390]  [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.875588]  [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365665.548153] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 858.
May 17 06:50:19 ip-172-16-15-126 kernel: [365703.557562] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 229.
May 17 06:50:19 ip-172-16-15-126 kernel: [365704.561666] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 567.
May 17 06:50:19 ip-172-16-15-126 kernel: [365704.566122] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1021.
May 17 06:50:19 ip-172-16-15-126 kernel: [365705.569628] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 647.
May 17 06:50:19 ip-172-16-15-126 kernel: [365705.573997] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 648.
May 17 06:50:19 ip-172-16-15-126 kernel: [365706.577713] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 230.
May 17 06:50:19 ip-172-16-15-126 kernel: [365707.581763] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 876.
May 17 06:50:19 ip-172-16-15-126 kernel: [365708.585837] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 649.
May 17 06:50:19 ip-172-16-15-126 kernel: [365709.593807] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 809.
May 17 06:50:19 ip-172-16-15-126 kernel: [365709.598182] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 231.
May 17 06:50:19 ip-172-16-15-126 kernel: [365711.601901] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 650.
May 17 06:50:19 ip-172-16-15-126 kernel: [365712.605951] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 810.
May 17 06:50:19 ip-172-16-15-126 kernel: [365714.610007] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 651.
May 17 06:50:19 ip-172-16-15-126 kernel: [365724.614383] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 232.
May 17 06:50:19 ip-172-16-15-126 kernel: [365730.618610] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 87.
May 17 06:50:19 ip-172-16-15-126 kernel: [365737.622898] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 872.
May 17 06:50:19 ip-172-16-15-126 kernel: [365739.626961] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 839.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.631063] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 88.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.635070] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 89.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.639040] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 840.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.643015] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 841.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.647046] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 811.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.651030] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 812.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.655302] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 813.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.659545] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 814.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.663826] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 815.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.668055] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 233.
May 17 06:50:19 ip-172-16-15-126 kernel: [365743.671120] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1022.
May 17 06:50:19 ip-172-16-15-126 kernel: [365743.675192] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1023.
May 17 06:50:19 ip-172-16-15-126 kernel: [365744.683098] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 652.
May 17 06:50:19 ip-172-16-15-126 kernel: [365744.687117] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 653.
May 17 06:50:19 ip-172-16-15-126 kernel: [365746.691242] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 0.
May 17 06:50:19 ip-172-16-15-126 kernel: [365747.695279] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 654.
May 17 06:50:19 ip-172-16-15-126 kernel: [365748.699313] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 842.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.703345] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 568.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.707929] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 569.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.712383] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.716774] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 2.
May 17 06:50:19 ip-172-16-15-126 kernel: [365763.723946] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 816.
May 17 06:50:19 ip-172-16-15-126 kernel: [365894.752829] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 899.
May 17 06:50:19 ip-172-16-15-126 kernel: [366004.772985] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 14.
May 17 06:50:19 ip-172-16-15-126 kernel: [366006.777073] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 849.
May 17 06:50:19 ip-172-16-15-126 kernel: [366007.781113] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 15.
May 17 06:50:19 ip-172-16-15-126 kernel: [366007.785109] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 911.
May 17 06:50:19 ip-172-16-15-126 kernel: [366012.789315] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 850.
May 17 06:50:19 ip-172-16-15-126 kernel: [366013.793349] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 16.
May 17 06:50:19 ip-172-16-15-126 kernel: [366025.797802] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 919.
May 17 06:50:19 ip-172-16-15-126 kernel: [366027.801877] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 248.
May 17 06:50:19 ip-172-16-15-126 kernel: [366030.805996] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 851.
May 17 06:50:19 ip-172-16-15-126 kernel: [366031.810023] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 586.
May 17 06:50:19 ip-172-16-15-126 kernel: [366031.814046] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 587.
May 17 06:50:19 ip-172-16-15-126 kernel: [366044.830516] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 669.
May 17 06:50:19 ip-172-16-15-126 kernel: [366060.835104] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 102.
May 17 06:50:19 ip-172-16-15-126 kernel: [366060.839915] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 822.
May 17 06:50:19 ip-172-16-15-126 kernel: [366061.843146] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 588.
May 17 06:50:19 ip-172-16-15-126 kernel: [366063.847294] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 249.
May 17 06:50:19 ip-172-16-15-126 kernel: [366072.863541] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 823.
May 17 06:50:19 ip-172-16-15-126 kernel: [366074.863683] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 924.
May 17 06:50:19 ip-172-16-15-126 kernel: [366100.868637] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 589.
May 17 06:50:19 ip-172-16-15-126 kernel: [366103.872750] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 17.
May 17 06:50:19 ip-172-16-15-126 kernel: [366103.876745] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 18.
May 17 06:50:19 ip-172-16-15-126 kernel: [366105.880828] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 824.
May 17 06:50:19 ip-172-16-15-126 kernel: [366105.884859] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 825.
May 17 06:50:19 ip-172-16-15-126 kernel: [366109.888993] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 23.
May 17 06:50:19 ip-172-16-15-126 kernel: [366117.893301] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 827.
May 17 06:50:19 ip-172-16-15-126 kernel: [366283.903580] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 962.
May 17 06:50:19 ip-172-16-15-126 kernel: [366305.908408] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 682.
May 17 06:50:19 ip-172-16-15-126 kernel: [366313.912713] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 596.
May 17 06:50:19 ip-172-16-15-126 kernel: [366313.917037] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 597.
May 17 06:50:19 ip-172-16-15-126 kernel: [366314.920749] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 683.