Last active
November 14, 2019 15:17
-
-
Save keynslug/bfc19b275f1831fe809e9413c9d2b2a1 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> Starting Consul agent... | |
| ==> Consul agent running! | |
| Version: 'v1.5.1' | |
| Node ID: 'bd03081c-76cf-7c40-1a3d-e305a4db0a31' | |
| Node name: 'n2' | |
| Datacenter: 'dc1' (Segment: '<all>') | |
| Server: true (Bootstrap: false) | |
| Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) | |
| Cluster Addr: 172.23.0.2 (LAN: 8301, WAN: 8302) | |
| Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false | |
| ==> Log data will now stream in as it occurs: | |
| 2019/07/12 17:08:15 [INFO] raft: Initial configuration (index=0): [] | |
| 2019/07/12 17:08:15 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:08:15 [INFO] serf: EventMemberJoin: n2.dc1 172.23.0.2 | |
| 2019/07/12 17:08:15 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:08:15 [INFO] consul: Handled member-join event for server "n2.dc1" in area "wan" | |
| 2019/07/12 17:08:15 [INFO] agent: (LAN) joined: 0 Err: 1 error occurred: | |
| * Failed to join 172.23.0.10: dial tcp 172.23.0.10:8301: connect: connection refused | |
| 2019/07/12 17:08:15 [WARN] agent: Join LAN failed: <nil>, retrying in 30s | |
| 2019/07/12 17:08:15 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:08:15 [WARN] agent/proxy: running as root, will not start managed proxies | |
| 2019/07/12 17:08:15 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp) | |
| 2019/07/12 17:08:15 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp) | |
| 2019/07/12 17:08:15 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp) | |
| 2019/07/12 17:08:15 [INFO] agent: started state syncer | |
| 2019/07/12 17:08:15 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere | |
| 2019/07/12 17:08:15 [INFO] agent: Joining LAN cluster... | |
| 2019/07/12 17:08:15 [INFO] agent: (LAN) joining: [n1] | |
| 2019/07/12 17:08:16 [WARN] raft: no known peers, aborting election | |
| 2019/07/12 17:08:22 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2019/07/12 17:08:26 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:26 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:29 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.2:34503: connect: connection refused | |
| 2019/07/12 17:08:33 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:33 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:33 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.2:35045: connect: connection refused | |
| 2019/07/12 17:08:40 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:40 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:41 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.2:38477: connect: connection refused | |
| 2019/07/12 17:08:44 [ERR] agent: Coordinate update error: No cluster leader | |
| 2019/07/12 17:08:45 [INFO] agent: (LAN) joining: [n1] | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n1 172.23.0.10 | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] agent: (LAN) joined: 1 Err: <nil> | |
| 2019/07/12 17:08:45 [INFO] agent: Join LAN completed. Synced with 1 initial agents | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n3.dc1 172.23.0.8 | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n3.dc1" in area "wan" | |
| 2019/07/12 17:08:46 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:08:46 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:08:46 [INFO] consul: Existing Raft peers reported by n1, disabling bootstrap mode | |
| 2019/07/12 17:08:46 [WARN] raft: Failed to get previous log: 1 log not found (last: 0) | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:08:49 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:10:25 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:10:29 [INFO] memberlist: Marking n5 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:10:29 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:10:29 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:10:29 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:10:30 [INFO] memberlist: Suspect n4.dc1 has failed, no acks received | |
| 2019/07/12 17:10:33 [INFO] memberlist: Marking n4 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:10:33 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:10:33 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:10:33 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:10:43 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:44 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:44 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [INFO] serf: attempting reconnect to n5 172.23.0.3:8301 | |
| 2019/07/12 17:10:45 [WARN] memberlist: Refuting a suspect message (from: n2) | |
| 2019/07/12 17:10:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:10:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:10:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:46 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:10:46 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:50 [WARN] memberlist: Was able to connect to n4.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:10:50 [WARN] memberlist: Refuting a suspect message (from: n5.dc1) | |
| 2019/07/12 17:11:25 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:11:25 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:26 [WARN] raft: Heartbeat timeout from "172.23.0.10:8300" reached, starting election | |
| 2019/07/12 17:11:26 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 19 | |
| 2019/07/12 17:11:27 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:11:29 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:11:30 [INFO] memberlist: Suspect n5.dc1 has failed, no acks received | |
| 2019/07/12 17:11:31 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:11:31 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 23 | |
| 2019/07/12 17:11:32 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:11:32 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 24 | |
| 2019/07/12 17:11:33 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:52323 | |
| 2019/07/12 17:11:34 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:11:34 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:11:34 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 25 | |
| 2019/07/12 17:11:35 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:32953 | |
| 2019/07/12 17:11:36 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:11:36 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 26 | |
| 2019/07/12 17:11:36 [INFO] memberlist: Marking n3 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:11:36 [INFO] serf: EventMemberFailed: n3 172.23.0.8 | |
| 2019/07/12 17:11:36 [INFO] consul: Removing LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:11:36 [ERR] http: Request PUT /v1/session/renew/f0ae2b77-c2d0-67d3-4078-9c6bcd562780, error: No cluster leader from=127.0.0.1:51001 | |
| 2019/07/12 17:11:36 [INFO] memberlist: Marking n5 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:11:36 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:11:36 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:11:36 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:11:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:11:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:11:37 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:37043 | |
| 2019/07/12 17:11:37 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:11:37 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 27 | |
| 2019/07/12 17:11:37 [WARN] raft: Unable to get address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b, using fallback address 172.23.0.8:8300: Could not find address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b | |
| 2019/07/12 17:11:37 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:11:38 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:11:38 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:36443 | |
| 2019/07/12 17:11:39 [INFO] serf: EventMemberFailed: n1 172.23.0.10 | |
| 2019/07/12 17:11:39 [INFO] consul: Removing LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:11:39 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:11:39 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 28 | |
| 2019/07/12 17:11:39 [WARN] raft: Unable to get address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b, using fallback address 172.23.0.8:8300: Could not find address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b | |
| 2019/07/12 17:11:39 [WARN] raft: Unable to get address for server id 9d284f0f-d265-8dc8-52cf-89d26c0c82f7, using fallback address 172.23.0.10:8300: Could not find address for server id 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 | |
| 2019/07/12 17:11:39 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:11:39 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:43909 | |
| 2019/07/12 17:11:41 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:11:41 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:11:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:11:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:11:41 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:38617 | |
| 2019/07/12 17:11:42 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:11:42 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:11:42 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 30 | |
| 2019/07/12 17:11:42 [WARN] raft: Unable to get address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b, using fallback address 172.23.0.8:8300: Could not find address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b | |
| 2019/07/12 17:11:42 [WARN] raft: Unable to get address for server id 9d284f0f-d265-8dc8-52cf-89d26c0c82f7, using fallback address 172.23.0.10:8300: Could not find address for server id 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 | |
| 2019/07/12 17:11:42 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:11:42 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:11:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:11:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:11:42 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:40847 | |
| 2019/07/12 17:11:43 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:11:44 [ERR] agent: Coordinate update error: No cluster leader | |
| 2019/07/12 17:11:44 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:44853 | |
| 2019/07/12 17:11:44 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:11:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:11:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:11:45 [INFO] serf: attempting reconnect to n5 172.23.0.3:8301 | |
| 2019/07/12 17:11:45 [WARN] memberlist: Refuting a suspect message (from: n2) | |
| 2019/07/12 17:11:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:11:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:11:45 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:11:45 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:11:45 [WARN] raft: Failed to get previous log: 108 log not found (last: 104) | |
| 2019/07/12 17:11:45 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:11:46 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:43425 | |
| 2019/07/12 17:11:46 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:11:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:11:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:11:47 [INFO] serf: EventMemberJoin: n1 172.23.0.10 | |
| 2019/07/12 17:11:47 [INFO] consul: Adding LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:11:50 [WARN] memberlist: Was able to connect to n3.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:11:50 [ERR] raft-net: Failed to flush response: write tcp 172.23.0.2:8300->172.23.0.10:47006: write: connection reset by peer | |
| 2019/07/12 17:11:50 [INFO] agent: Deregistered service "machinegun-consuela" | |
| 2019/07/12 17:11:50 [INFO] agent: Deregistered check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:11:52 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:12:00 [INFO] memberlist: Marking n5.dc1 as failed, suspect timeout reached (0 peer confirmations) | |
| 2019/07/12 17:12:00 [INFO] serf: EventMemberFailed: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:12:00 [INFO] consul: Handled member-failed event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:12:00 [INFO] serf: EventMemberJoin: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:12:00 [INFO] consul: Handled member-join event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:12:01 [INFO] serf: EventMemberFailed: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:12:01 [INFO] consul: Handled member-failed event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:12:01 [INFO] serf: EventMemberJoin: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:12:01 [INFO] consul: Handled member-join event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:12:25 [WARN] raft: Heartbeat timeout from "172.23.0.10:8300" reached, starting election | |
| 2019/07/12 17:12:25 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 34 | |
| 2019/07/12 17:12:25 [INFO] raft: Duplicate RequestVote for same term: 34 | |
| 2019/07/12 17:12:26 [INFO] raft: Duplicate RequestVote for same term: 34 | |
| 2019/07/12 17:12:26 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:12:26 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:26 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 35 | |
| 2019/07/12 17:12:26 [INFO] raft: Election won. Tally: 3 | |
| 2019/07/12 17:12:26 [INFO] raft: Node at 172.23.0.2:8300 [Leader] entering Leader state | |
| 2019/07/12 17:12:26 [INFO] raft: Added peer d07675a8-5f56-3ab5-3fb7-71146988f86b, starting replication | |
| 2019/07/12 17:12:26 [INFO] raft: Added peer 9d284f0f-d265-8dc8-52cf-89d26c0c82f7, starting replication | |
| 2019/07/12 17:12:26 [INFO] raft: Added peer 7c0d0be2-a776-9624-088a-a1f6d840575b, starting replication | |
| 2019/07/12 17:12:26 [INFO] raft: Added peer 6b66d87b-2666-f87c-8b35-66484ce8610f, starting replication | |
| 2019/07/12 17:12:26 [INFO] consul: cluster leadership acquired | |
| 2019/07/12 17:12:26 [INFO] consul: New leader elected: n2 | |
| 2019/07/12 17:12:26 [INFO] raft: pipelining replication to peer {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300} | |
| 2019/07/12 17:12:26 [INFO] raft: pipelining replication to peer {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300} | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 501.596282ms | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 in 501.591345ms | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 942.267851ms | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 in 942.262914ms | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 1.41448869s | |
| 2019/07/12 17:12:27 [WARN] raft: Failed to contact 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 in 1.414483753s | |
| 2019/07/12 17:12:29 [WARN] consul: error getting server health from "n3": context deadline exceeded | |
| 2019/07/12 17:12:29 [WARN] consul: error getting server health from "n1": context deadline exceeded | |
| 2019/07/12 17:12:30 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:12:30 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:30 [WARN] consul: error getting server health from "n3": last request still outstanding | |
| 2019/07/12 17:12:32 [WARN] consul: error getting server health from "n3": last request still outstanding | |
| 2019/07/12 17:12:32 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:34 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:12:34 [INFO] memberlist: Marking n3 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:12:34 [INFO] serf: EventMemberFailed: n3 172.23.0.8 | |
| 2019/07/12 17:12:34 [INFO] consul: Removing LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:12:34 [INFO] consul: member 'n3' failed, marking health critical | |
| 2019/07/12 17:12:34 [WARN] consul: error getting server health from "n3": last request still outstanding | |
| 2019/07/12 17:12:34 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:35 [INFO] memberlist: Suspect n1.dc1 has failed, no acks received | |
| 2019/07/12 17:12:35 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: read tcp 172.23.0.2:59382->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: read tcp 172.23.0.2:53576->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: read tcp 172.23.0.2:59460->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: read tcp 172.23.0.2:53654->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to AppendEntries to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: read tcp 172.23.0.2:53520->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to AppendEntries to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: read tcp 172.23.0.2:59328->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:36 [INFO] autopilot: Attempting removal of failed server node "n3" | |
| 2019/07/12 17:12:36 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:36 [WARN] consul: error getting server health from "n3": last request still outstanding | |
| 2019/07/12 17:12:36 [INFO] serf: EventMemberLeave (forced): n3 172.23.0.8 | |
| 2019/07/12 17:12:36 [INFO] consul: Removing LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:12:36 [INFO] consul: removing server by ID: "d07675a8-5f56-3ab5-3fb7-71146988f86b" | |
| 2019/07/12 17:12:36 [INFO] raft: Updating configuration with RemoveServer (d07675a8-5f56-3ab5-3fb7-71146988f86b, ) to [{Suffrage:Voter ID:9d284f0f-d265-8dc8-52cf-89d26c0c82f7 Address:172.23.0.10:8300} {Suffrage:Voter ID:7c0d0be2-a776-9624-088a-a1f6d840575b Address:172.23.0.11:8300} {Suffrage:Voter ID:bd03081c-76cf-7c40-1a3d-e305a4db0a31 Address:172.23.0.2:8300} {Suffrage:Voter ID:6b66d87b-2666-f87c-8b35-66484ce8610f Address:172.23.0.3:8300}] | |
| 2019/07/12 17:12:36 [WARN] raft: Unable to get address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b, using fallback address 172.23.0.8:8300: Could not find address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b | |
| 2019/07/12 17:12:36 [INFO] raft: Removed peer d07675a8-5f56-3ab5-3fb7-71146988f86b, stopping replication after 150 | |
| 2019/07/12 17:12:36 [INFO] consul: member 'n3' left, deregistering | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to heartbeat to 172.23.0.10:8300: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:36 [ERR] raft: Failed to heartbeat to 172.23.0.8:8300: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:36 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:36 [WARN] raft: Unable to get address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b, using fallback address 172.23.0.8:8300: Could not find address for server id d07675a8-5f56-3ab5-3fb7-71146988f86b | |
| 2019/07/12 17:12:37 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:37 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:12:37 [INFO] memberlist: Marking n1 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:12:37 [INFO] serf: EventMemberFailed: n1 172.23.0.10 | |
| 2019/07/12 17:12:37 [INFO] consul: Removing LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:12:37 [INFO] consul: member 'n1' failed, marking health critical | |
| 2019/07/12 17:12:37 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:38 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:38 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:38 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:38 [WARN] consul: error getting server health from "n3": rpc error getting client: failed to get conn: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:38 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:38 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:39 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:39 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:39 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:40 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:41 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:41 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:41 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:41 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:42 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:42 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:42 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:42 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:42 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:43 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:44 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:44 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:44 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:45 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:45 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:45 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:45 [INFO] serf: EventMemberJoin: n1 172.23.0.10 | |
| 2019/07/12 17:12:45 [INFO] consul: Adding LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:12:45 [INFO] consul: member 'n1' joined, marking health alive | |
| 2019/07/12 17:12:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:45 [WARN] memberlist: Refuting a suspect message (from: n5) | |
| 2019/07/12 17:12:45 [WARN] consul.kvs: Rejecting lock of mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa due to lock-delay until 2019-07-12 17:12:46.579763829 +0000 UTC m=+271.546614183 | |
| 2019/07/12 17:12:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:45 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:12:45 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:12:45 [INFO] raft: Updating configuration with AddNonvoter (d07675a8-5f56-3ab5-3fb7-71146988f86b, 172.23.0.8:8300) to [{Suffrage:Voter ID:9d284f0f-d265-8dc8-52cf-89d26c0c82f7 Address:172.23.0.10:8300} {Suffrage:Voter ID:7c0d0be2-a776-9624-088a-a1f6d840575b Address:172.23.0.11:8300} {Suffrage:Voter ID:bd03081c-76cf-7c40-1a3d-e305a4db0a31 Address:172.23.0.2:8300} {Suffrage:Voter ID:6b66d87b-2666-f87c-8b35-66484ce8610f Address:172.23.0.3:8300} {Suffrage:Nonvoter ID:d07675a8-5f56-3ab5-3fb7-71146988f86b Address:172.23.0.8:8300}] | |
| 2019/07/12 17:12:45 [INFO] raft: Added peer d07675a8-5f56-3ab5-3fb7-71146988f86b, starting replication | |
| 2019/07/12 17:12:45 [ERR] raft: peer {Nonvoter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300} has newer term, stopping replication | |
| 2019/07/12 17:12:45 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:12:45 [INFO] raft: aborting pipeline replication to peer {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300} | |
| 2019/07/12 17:12:45 [INFO] raft: aborting pipeline replication to peer {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300} | |
| 2019/07/12 17:12:45 [ERR] consul: failed to add raft peer: leadership lost while committing log | |
| 2019/07/12 17:12:45 [ERR] consul: failed to reconcile member: {n3 172.23.0.8 8301 map[acls:0 build:1.5.1:40cec984 dc:dc1 expect:5 id:d07675a8-5f56-3ab5-3fb7-71146988f86b port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}: leadership lost while committing log | |
| 2019/07/12 17:12:45 [INFO] consul: cluster leadership lost | |
| 2019/07/12 17:12:46 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since our last term is greater (35, 33) | |
| 2019/07/12 17:12:46 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since our last term is greater (35, 33) | |
| 2019/07/12 17:12:46 [ERR] raft: Failed to AppendEntries to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:46 [ERR] raft: Failed to AppendEntries to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:46 [ERR] raft: peer {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300} has newer term, stopping replication | |
| 2019/07/12 17:12:46 [ERR] raft: peer {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300} has newer term, stopping replication | |
| 2019/07/12 17:12:46 [ERR] raft: Failed to heartbeat to 172.23.0.10:8300: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:12:46 [ERR] raft: Failed to heartbeat to 172.23.0.8:8300: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:12:47 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since our last term is greater (35, 33) | |
| 2019/07/12 17:12:47 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since our last term is greater (35, 33) | |
| 2019/07/12 17:12:47 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:12:47 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 51 | |
| 2019/07/12 17:12:47 [INFO] raft: Election won. Tally: 3 | |
| 2019/07/12 17:12:47 [INFO] raft: Node at 172.23.0.2:8300 [Leader] entering Leader state | |
| 2019/07/12 17:12:47 [INFO] raft: Added peer 9d284f0f-d265-8dc8-52cf-89d26c0c82f7, starting replication | |
| 2019/07/12 17:12:47 [INFO] raft: Added peer 7c0d0be2-a776-9624-088a-a1f6d840575b, starting replication | |
| 2019/07/12 17:12:47 [INFO] raft: Added peer 6b66d87b-2666-f87c-8b35-66484ce8610f, starting replication | |
| 2019/07/12 17:12:47 [INFO] raft: Added peer d07675a8-5f56-3ab5-3fb7-71146988f86b, starting replication | |
| 2019/07/12 17:12:47 [INFO] consul: cluster leadership acquired | |
| 2019/07/12 17:12:47 [INFO] consul: New leader elected: n2 | |
| 2019/07/12 17:12:47 [INFO] raft: pipelining replication to peer {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300} | |
| 2019/07/12 17:12:47 [WARN] raft: AppendEntries to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300} rejected, sending older logs (next: 146) | |
| 2019/07/12 17:12:47 [WARN] raft: AppendEntries to {Nonvoter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300} rejected, sending older logs (next: 146) | |
| 2019/07/12 17:12:47 [INFO] raft: pipelining replication to peer {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300} | |
| 2019/07/12 17:12:47 [INFO] raft: pipelining replication to peer {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300} | |
| 2019/07/12 17:12:47 [INFO] raft: pipelining replication to peer {Nonvoter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300} | |
| 2019/07/12 17:12:47 [INFO] consul: member 'n3' joined, marking health alive | |
| 2019/07/12 17:12:49 [WARN] consul: error getting server health from "n1": last request still outstanding | |
| 2019/07/12 17:12:50 [ERR] raft-net: Failed to flush response: write tcp 172.23.0.2:8300->172.23.0.10:51260: write: broken pipe | |
| 2019/07/12 17:12:55 [WARN] memberlist: Was able to connect to n1.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:12:55 [WARN] memberlist: Refuting a suspect message (from: n3.dc1) | |
| 2019/07/12 17:13:00 [INFO] memberlist: Marking n1.dc1 as failed, suspect timeout reached (0 peer confirmations) | |
| 2019/07/12 17:13:00 [INFO] serf: EventMemberFailed: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:13:00 [INFO] consul: Handled member-failed event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:13:01 [INFO] serf: EventMemberJoin: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:13:01 [INFO] consul: Handled member-join event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:13:07 [INFO] autopilot: Promoting Server (ID: "d07675a8-5f56-3ab5-3fb7-71146988f86b" Address: "172.23.0.8:8300") to voter | |
| 2019/07/12 17:13:07 [INFO] raft: Updating configuration with AddStaging (d07675a8-5f56-3ab5-3fb7-71146988f86b, 172.23.0.8:8300) to [{Suffrage:Voter ID:9d284f0f-d265-8dc8-52cf-89d26c0c82f7 Address:172.23.0.10:8300} {Suffrage:Voter ID:7c0d0be2-a776-9624-088a-a1f6d840575b Address:172.23.0.11:8300} {Suffrage:Voter ID:bd03081c-76cf-7c40-1a3d-e305a4db0a31 Address:172.23.0.2:8300} {Suffrage:Voter ID:6b66d87b-2666-f87c-8b35-66484ce8610f Address:172.23.0.3:8300} {Suffrage:Voter ID:d07675a8-5f56-3ab5-3fb7-71146988f86b Address:172.23.0.8:8300}] | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 500.373868ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 511.471919ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact 7c0d0be2-a776-9624-088a-a1f6d840575b in 500.118689ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact 7c0d0be2-a776-9624-088a-a1f6d840575b in 510.770886ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact 6b66d87b-2666-f87c-8b35-66484ce8610f in 500.385333ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact d07675a8-5f56-3ab5-3fb7-71146988f86b in 522.124116ms | |
| 2019/07/12 17:13:25 [WARN] raft: Failed to contact quorum of nodes, stepping down | |
| 2019/07/12 17:13:25 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:13:25 [INFO] raft: aborting pipeline replication to peer {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300} | |
| 2019/07/12 17:13:25 [INFO] raft: aborting pipeline replication to peer {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300} | |
| 2019/07/12 17:13:25 [INFO] raft: aborting pipeline replication to peer {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300} | |
| 2019/07/12 17:13:25 [INFO] raft: aborting pipeline replication to peer {Nonvoter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300} | |
| 2019/07/12 17:13:25 [INFO] consul: cluster leadership lost | |
| 2019/07/12 17:13:27 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:13:30 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:13:33 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:13:34 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:13:34 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 58 | |
| 2019/07/12 17:13:34 [ERR] raft: Failed to heartbeat to 172.23.0.3:8300: read tcp 172.23.0.2:44072->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:13:34 [ERR] raft: Failed to heartbeat to 172.23.0.8:8300: read tcp 172.23.0.2:33248->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:13:34 [ERR] raft: Failed to heartbeat to 172.23.0.11:8300: read tcp 172.23.0.2:54698->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:13:35 [INFO] memberlist: Suspect n5.dc1 has failed, no acks received | |
| 2019/07/12 17:13:35 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:13:35 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:13:36 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:13:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=172.23.0.11:43006 | |
| 2019/07/12 17:13:36 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:13:36 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 60 | |
| 2019/07/12 17:13:36 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:13:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=172.23.0.3:56420 | |
| 2019/07/12 17:13:37 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:13:37 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:13:37 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:13:38 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:13:38 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 61 | |
| 2019/07/12 17:13:38 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:13:38 [INFO] memberlist: Marking n4 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:13:38 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:13:38 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:13:38 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:13:38 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:13:39 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:13:39 [ERR] http: Request PUT /v1/session/renew/3535972a-dab0-12a3-2819-dcf693776e09, error: No cluster leader from=127.0.0.1:57209 | |
| 2019/07/12 17:13:40 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:13:40 [INFO] raft: Node at 172.23.0.2:8300 [Candidate] entering Candidate state in term 62 | |
| 2019/07/12 17:13:40 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:13:40 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:13:41 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:13:41 [INFO] raft: Node at 172.23.0.2:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:13:41 [INFO] serf: EventMemberFailed: n3 172.23.0.8 | |
| 2019/07/12 17:13:41 [INFO] consul: Removing LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:13:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: dial tcp <nil>->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:13:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:13:44 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:13:45 [INFO] serf: attempting reconnect to n5 172.23.0.3:8301 | |
| 2019/07/12 17:13:45 [WARN] memberlist: Refuting a suspect message (from: n2) | |
| 2019/07/12 17:13:45 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:13:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:13:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:13:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:13:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:13:45 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:13:45 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:13:45 [WARN] raft: Failed to get previous log: 186 log not found (last: 179) | |
| 2019/07/12 17:13:45 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:13:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: dial tcp <nil>->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:13:46 [ERR] raft: Failed to make RequestVote RPC to {Voter d07675a8-5f56-3ab5-3fb7-71146988f86b 172.23.0.8:8300}: dial tcp <nil>->172.23.0.8:8300: i/o timeout | |
| 2019/07/12 17:13:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:13:47 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:53 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:55 [WARN] memberlist: Was able to connect to n3.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:13:56 [WARN] memberlist: Refuting a suspect message (from: n4.dc1) | |
| 2019/07/12 17:14:27 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:14:30 [INFO] memberlist: Suspect n3 has failed, no acks received | |
| 2019/07/12 17:14:31 [INFO] serf: EventMemberFailed: n3 172.23.0.8 | |
| 2019/07/12 17:14:31 [INFO] consul: Removing LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:14:33 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:14:33 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:14:33 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:14:35 [INFO] memberlist: Suspect n3.dc1 has failed, no acks received | |
| 2019/07/12 17:14:44 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:45 [INFO] serf: attempting reconnect to n5 172.23.0.3:8301 | |
| 2019/07/12 17:14:45 [WARN] memberlist: Refuting a suspect message (from: n2) | |
| 2019/07/12 17:14:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:14:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:14:45 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:14:45 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:14:45 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:46 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:14:46 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:48 [WARN] raft: Rejecting vote request from 172.23.0.8:8300 since we have a leader: 172.23.0.11:8300 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> Starting Consul agent... | |
| ==> Consul agent running! | |
| Version: 'v1.5.1' | |
| Node ID: 'd07675a8-5f56-3ab5-3fb7-71146988f86b' | |
| Node name: 'n3' | |
| Datacenter: 'dc1' (Segment: '<all>') | |
| Server: true (Bootstrap: false) | |
| Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) | |
| Cluster Addr: 172.23.0.8 (LAN: 8301, WAN: 8302) | |
| Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false | |
| ==> Log data will now stream in as it occurs: | |
| 2019/07/12 17:08:15 [INFO] raft: Initial configuration (index=0): [] | |
| 2019/07/12 17:08:15 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:08:15 [INFO] serf: EventMemberJoin: n3.dc1 172.23.0.8 | |
| 2019/07/12 17:08:15 [INFO] serf: EventMemberJoin: n3 172.23.0.8 | |
| 2019/07/12 17:08:15 [INFO] consul: Adding LAN server n3 (Addr: tcp/172.23.0.8:8300) (DC: dc1) | |
| 2019/07/12 17:08:15 [INFO] consul: Handled member-join event for server "n3.dc1" in area "wan" | |
| 2019/07/12 17:08:15 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp) | |
| 2019/07/12 17:08:15 [WARN] agent/proxy: running as root, will not start managed proxies | |
| 2019/07/12 17:08:15 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp) | |
| 2019/07/12 17:08:15 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp) | |
| 2019/07/12 17:08:15 [INFO] agent: started state syncer | |
| 2019/07/12 17:08:15 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere | |
| 2019/07/12 17:08:15 [INFO] agent: Joining LAN cluster... | |
| 2019/07/12 17:08:15 [INFO] agent: (LAN) joining: [n1] | |
| 2019/07/12 17:08:15 [INFO] agent: (LAN) joined: 0 Err: 1 error occurred: | |
| * Failed to join 172.23.0.10: dial tcp 172.23.0.10:8301: connect: connection refused | |
| 2019/07/12 17:08:15 [WARN] agent: Join LAN failed: <nil>, retrying in 30s | |
| 2019/07/12 17:08:16 [WARN] raft: no known peers, aborting election | |
| 2019/07/12 17:08:22 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.8:40551: connect: connection refused | |
| 2019/07/12 17:08:22 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2019/07/12 17:08:28 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:28 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:28 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.8:39729: connect: connection refused | |
| 2019/07/12 17:08:35 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:35 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:38 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.8:34511: connect: connection refused | |
| 2019/07/12 17:08:42 [WARN] agent: Syncing service "machinegun-consuela" failed. No cluster leader | |
| 2019/07/12 17:08:42 [ERR] agent: failed to sync changes: No cluster leader | |
| 2019/07/12 17:08:44 [WARN] agent: Check "machinegun-consuela:presence:tcp" socket connection failed: dial tcp 172.23.0.8:38931: connect: connection refused | |
| 2019/07/12 17:08:45 [INFO] agent: (LAN) joining: [n1] | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n1 172.23.0.10 | |
| 2019/07/12 17:08:45 [INFO] agent: (LAN) joined: 1 Err: <nil> | |
| 2019/07/12 17:08:45 [INFO] agent: Join LAN completed. Synced with 1 initial agents | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n2.dc1 172.23.0.2 | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n2.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:08:45 [INFO] serf: EventMemberJoin: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] consul: Handled member-join event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:08:45 [INFO] consul: Found expected number of peers, attempting bootstrap: 172.23.0.8:8300,172.23.0.3:8300,172.23.0.2:8300,172.23.0.11:8300,172.23.0.10:8300 | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:08:45 [INFO] consul: Adding LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:08:46 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:46 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:47 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:08:47 [ERR] http: Request PUT /v1/session/create, error: rpc error making call: Check 'machinegun-consuela:presence:tcp' is in critical state from=127.0.0.1:34623 | |
| 2019/07/12 17:08:47 [INFO] agent: Deregistered service "machinegun-consuela" | |
| 2019/07/12 17:08:47 [INFO] agent: Deregistered check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:08:49 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:10:24 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:10:28 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:10:28 [INFO] memberlist: Marking n5 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:10:28 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:10:28 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:10:30 [INFO] memberlist: Suspect n4.dc1 has failed, no acks received | |
| 2019/07/12 17:10:31 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:10:32 [INFO] memberlist: Marking n4 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:10:32 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:10:32 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:10:34 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:10:43 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:44 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:44 [WARN] memberlist: Refuting a suspect message (from: n4) | |
| 2019/07/12 17:10:44 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:10:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:10:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:10:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:10:45 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:45 [WARN] raft: Heartbeat timeout from "172.23.0.10:8300" reached, starting election | |
| 2019/07/12 17:10:45 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 3 | |
| 2019/07/12 17:10:45 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:10:46 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:10:46 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:10:50 [WARN] memberlist: Was able to connect to n5.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:10:51 [WARN] memberlist: Refuting a suspect message (from: n4.dc1) | |
| 2019/07/12 17:11:26 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:11:29 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:11:29 [INFO] memberlist: Marking n2 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:11:29 [INFO] serf: EventMemberFailed: n2 172.23.0.2 | |
| 2019/07/12 17:11:29 [INFO] consul: Removing LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:11:30 [INFO] memberlist: Suspect n4.dc1 has failed, no acks received | |
| 2019/07/12 17:11:31 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:11:33 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:11:33 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:11:33 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:11:44 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:44 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:44 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:45 [WARN] raft: Heartbeat timeout from "172.23.0.10:8300" reached, starting election | |
| 2019/07/12 17:11:45 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 19 | |
| 2019/07/12 17:11:45 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:11:45 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:11:45 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:11:45 [WARN] memberlist: Refuting a suspect message (from: n5) | |
| 2019/07/12 17:11:45 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:11:45 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:11:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:11:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:11:45 [INFO] consul: New leader elected: n1 | |
| 2019/07/12 17:11:45 [WARN] raft: Rejecting vote request from 172.23.0.2:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:46 [WARN] raft: Rejecting vote request from 172.23.0.2:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:50 [WARN] memberlist: Was able to connect to n2.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:11:52 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:54 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.10:8300 | |
| 2019/07/12 17:11:54 [INFO] agent: Synced check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:11:57 [INFO] agent: Synced check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:12:00 [INFO] serf: EventMemberFailed: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:12:00 [INFO] consul: Handled member-failed event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:12:00 [INFO] serf: EventMemberJoin: n5.dc1 172.23.0.3 | |
| 2019/07/12 17:12:00 [INFO] consul: Handled member-join event for server "n5.dc1" in area "wan" | |
| 2019/07/12 17:12:00 [INFO] memberlist: Marking n4.dc1 as failed, suspect timeout reached (0 peer confirmations) | |
| 2019/07/12 17:12:00 [INFO] serf: EventMemberFailed: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:12:00 [INFO] consul: Handled member-failed event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:12:01 [INFO] serf: EventMemberJoin: n4.dc1 172.23.0.11 | |
| 2019/07/12 17:12:01 [INFO] consul: Handled member-join event for server "n4.dc1" in area "wan" | |
| 2019/07/12 17:12:26 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:12:29 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:12:30 [INFO] memberlist: Suspect n2.dc1 has failed, no acks received | |
| 2019/07/12 17:12:32 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:12:32 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 39 | |
| 2019/07/12 17:12:32 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:12:33 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:12:34 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:12:34 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 41 | |
| 2019/07/12 17:12:35 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:12:36 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:12:37 [ERR] http: Request PUT /v1/session/renew/226c14fb-3320-4e4a-d4ac-9d480cda3ba3, error: No cluster leader from=127.0.0.1:60525 | |
| 2019/07/12 17:12:37 [INFO] memberlist: Marking n4 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:12:37 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:12:37 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:12:37 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:12:37 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 43 | |
| 2019/07/12 17:12:37 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:38 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:38 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 44 | |
| 2019/07/12 17:12:38 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:38 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:12:39 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:39 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 45 | |
| 2019/07/12 17:12:39 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:40 [ERR] agent: Coordinate update error: No cluster leader | |
| 2019/07/12 17:12:41 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:41 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 46 | |
| 2019/07/12 17:12:41 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:41 [INFO] serf: EventMemberFailed: n5 172.23.0.3 | |
| 2019/07/12 17:12:41 [INFO] consul: Removing LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:12:41 [INFO] memberlist: Suspect n5 has failed, no acks received | |
| 2019/07/12 17:12:42 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:42 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 47 | |
| 2019/07/12 17:12:42 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:12:42 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: read tcp 172.23.0.8:52758->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:12:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: read tcp 172.23.0.8:41004->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:12:42 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: read tcp 172.23.0.8:39062->172.23.0.2:8300: i/o timeout | |
| 2019/07/12 17:12:43 [INFO] memberlist: Marking n2 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:12:43 [INFO] serf: EventMemberFailed: n2 172.23.0.2 | |
| 2019/07/12 17:12:43 [INFO] consul: Removing LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:12:44 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:44 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 48 | |
| 2019/07/12 17:12:44 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:12:44 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:12:44 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:44 [WARN] memberlist: Was able to connect to n2 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:12:44 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: dial tcp <nil>->172.23.0.2:8300: i/o timeout | |
| 2019/07/12 17:12:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 6b66d87b-2666-f87c-8b35-66484ce8610f 172.23.0.3:8300}: dial tcp <nil>->172.23.0.3:8300: i/o timeout | |
| 2019/07/12 17:12:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: dial tcp <nil>->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:12:45 [WARN] memberlist: Refuting a suspect message (from: n1) | |
| 2019/07/12 17:12:45 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:45 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 49 | |
| 2019/07/12 17:12:45 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:12:45 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:12:45 [WARN] raft: Unable to get address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f, using fallback address 172.23.0.3:8300: Could not find address for server id 6b66d87b-2666-f87c-8b35-66484ce8610f | |
| 2019/07/12 17:12:45 [INFO] serf: attempting reconnect to n4 172.23.0.11:8301 | |
| 2019/07/12 17:12:45 [INFO] serf: EventMemberJoin: n5 172.23.0.3 | |
| 2019/07/12 17:12:45 [INFO] consul: Adding LAN server n5 (Addr: tcp/172.23.0.3:8300) (DC: dc1) | |
| 2019/07/12 17:12:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:12:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:12:45 [INFO] consul: New leader elected: n2 | |
| 2019/07/12 17:12:45 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:12:45 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:12:47 [WARN] memberlist: Refuting a suspect message (from: n2.dc1) | |
| 2019/07/12 17:12:47 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:12:47 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 50 | |
| 2019/07/12 17:12:47 [WARN] raft: Failed to get previous log: 157 log not found (last: 145) | |
| 2019/07/12 17:12:47 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "172.23.0.2:8300") | |
| 2019/07/12 17:12:47 [INFO] consul: New leader elected: n2 | |
| 2019/07/12 17:12:47 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:12:49 [INFO] agent: Deregistered service "machinegun-consuela" | |
| 2019/07/12 17:12:49 [INFO] agent: Deregistered check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:12:50 [WARN] memberlist: Was able to connect to n5.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:12:51 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:12:52 [INFO] agent: Synced service "machinegun-consuela" | |
| 2019/07/12 17:12:52 [WARN] raft: Rejecting vote request from 172.23.0.11:8300 since we have a leader: 172.23.0.2:8300 | |
| 2019/07/12 17:13:00 [INFO] serf: EventMemberFailed: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:13:00 [INFO] consul: Handled member-failed event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:13:01 [INFO] serf: EventMemberJoin: n1.dc1 172.23.0.10 | |
| 2019/07/12 17:13:01 [INFO] consul: Handled member-join event for server "n1.dc1" in area "wan" | |
| 2019/07/12 17:13:11 [INFO] agent: Synced check "machinegun-consuela:presence:tcp" | |
| 2019/07/12 17:13:26 [WARN] raft: Heartbeat timeout from "172.23.0.2:8300" reached, starting election | |
| 2019/07/12 17:13:26 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 52 | |
| 2019/07/12 17:13:26 [INFO] raft: Duplicate RequestVote for same term: 52 | |
| 2019/07/12 17:13:26 [INFO] raft: Duplicate RequestVote for same term: 52 | |
| 2019/07/12 17:13:27 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:13:27 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:13:27 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:13:30 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:13:30 [INFO] memberlist: Marking n1 as failed, suspect timeout reached (2 peer confirmations) | |
| 2019/07/12 17:13:30 [INFO] serf: EventMemberFailed: n1 172.23.0.10 | |
| 2019/07/12 17:13:30 [INFO] consul: Removing LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:13:31 [INFO] serf: EventMemberFailed: n2 172.23.0.2 | |
| 2019/07/12 17:13:31 [INFO] consul: Removing LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:13:36 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: read tcp 172.23.0.8:42798->172.23.0.2:8300: i/o timeout | |
| 2019/07/12 17:13:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: read tcp 172.23.0.8:54880->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:13:40 [INFO] memberlist: Suspect n1.dc1 has failed, no acks received | |
| 2019/07/12 17:13:45 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:13:45 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:13:45 [WARN] memberlist: Refuting a suspect message (from: n5) | |
| 2019/07/12 17:13:45 [WARN] raft: Rejecting vote request from 172.23.0.2:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:45 [INFO] serf: EventMemberJoin: n1 172.23.0.10 | |
| 2019/07/12 17:13:45 [INFO] consul: Adding LAN server n1 (Addr: tcp/172.23.0.10:8300) (DC: dc1) | |
| 2019/07/12 17:13:45 [WARN] raft: Rejecting vote request from 172.23.0.10:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:45 [WARN] raft: Rejecting vote request from 172.23.0.10:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:45 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:13:47 [WARN] raft: Rejecting vote request from 172.23.0.2:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:48 [WARN] raft: Rejecting vote request from 172.23.0.10:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:50 [ERR] raft-net: Failed to flush response: write tcp 172.23.0.8:8300->172.23.0.2:33200: write: broken pipe | |
| 2019/07/12 17:13:53 [WARN] raft: Rejecting vote request from 172.23.0.10:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:13:55 [WARN] memberlist: Was able to connect to n2.dc1 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:14:26 [WARN] raft: Rejecting vote request from 172.23.0.3:8300 since we have a leader: 172.23.0.11:8300 | |
| 2019/07/12 17:14:26 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:14:26 [WARN] raft: Heartbeat timeout from "172.23.0.11:8300" reached, starting election | |
| 2019/07/12 17:14:26 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 67 | |
| 2019/07/12 17:14:28 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:14:29 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:14:29 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 69 | |
| 2019/07/12 17:14:30 [INFO] memberlist: Suspect n4 has failed, no acks received | |
| 2019/07/12 17:14:30 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:14:34 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:55993 | |
| 2019/07/12 17:14:35 [INFO] memberlist: Suspect n1.dc1 has failed, no acks received | |
| 2019/07/12 17:14:35 [INFO] memberlist: Suspect n1 has failed, no acks received | |
| 2019/07/12 17:14:35 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:56589 | |
| 2019/07/12 17:14:36 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:14:36 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 74 | |
| 2019/07/12 17:14:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:14:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: read tcp 172.23.0.8:56494->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:14:36 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: read tcp 172.23.0.8:42678->172.23.0.2:8300: i/o timeout | |
| 2019/07/12 17:14:37 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:44737 | |
| 2019/07/12 17:14:37 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:14:37 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 75 | |
| 2019/07/12 17:14:37 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:14:37 [ERR] agent: Coordinate update error: rpc error making call: EOF | |
| 2019/07/12 17:14:37 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: rpc error making call: EOF from=127.0.0.1:43179 | |
| 2019/07/12 17:14:38 [INFO] memberlist: Marking n4 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:14:38 [INFO] serf: EventMemberFailed: n4 172.23.0.11 | |
| 2019/07/12 17:14:38 [INFO] consul: Removing LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:14:39 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:53363 | |
| 2019/07/12 17:14:39 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:14:39 [ERR] yamux: keepalive failed: i/o deadline reached | |
| 2019/07/12 17:14:39 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=172.23.0.11:36376 | |
| 2019/07/12 17:14:39 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:14:39 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: read tcp 172.23.0.8:42896->172.23.0.2:8300: i/o timeout | |
| 2019/07/12 17:14:39 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: read tcp 172.23.0.8:56368->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:14:39 [ERR] http: Request PUT /v1/session/renew/00ea2415-93a9-9538-5ca2-8f1f2f9bc96e, error: No cluster leader from=127.0.0.1:36323 | |
| 2019/07/12 17:14:40 [INFO] memberlist: Suspect n2 has failed, no acks received | |
| 2019/07/12 17:14:40 [INFO] memberlist: Marking n2 as failed, suspect timeout reached (1 peer confirmations) | |
| 2019/07/12 17:14:40 [INFO] serf: EventMemberFailed: n2 172.23.0.2 | |
| 2019/07/12 17:14:40 [INFO] consul: Removing LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:14:40 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2019/07/12 17:14:40 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 77 | |
| 2019/07/12 17:14:40 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:14:40 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:14:41 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:53187 | |
| 2019/07/12 17:14:42 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:14:42 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 78 | |
| 2019/07/12 17:14:42 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:14:42 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:14:42 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:47551 | |
| 2019/07/12 17:14:43 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:14:43 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 79 | |
| 2019/07/12 17:14:43 [WARN] raft: Unable to get address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31, using fallback address 172.23.0.2:8300: Could not find address for server id bd03081c-76cf-7c40-1a3d-e305a4db0a31 | |
| 2019/07/12 17:14:43 [WARN] raft: Unable to get address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b, using fallback address 172.23.0.11:8300: Could not find address for server id 7c0d0be2-a776-9624-088a-a1f6d840575b | |
| 2019/07/12 17:14:44 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:47899 | |
| 2019/07/12 17:14:44 [WARN] memberlist: Was able to connect to n2 but other probes failed, network may be misconfigured | |
| 2019/07/12 17:14:45 [WARN] memberlist: Refuting a suspect message (from: n5) | |
| 2019/07/12 17:14:45 [ERR] http: Request GET /v1/kv/mg/jbqEQJD20XJM9RQo42D1Ehs6GRUcp7J5PQExc1QDUPcjra4CxJJ4EFa, error: No cluster leader from=127.0.0.1:38245 | |
| 2019/07/12 17:14:45 [INFO] serf: EventMemberJoin: n2 172.23.0.2 | |
| 2019/07/12 17:14:45 [INFO] consul: Adding LAN server n2 (Addr: tcp/172.23.0.2:8300) (DC: dc1) | |
| 2019/07/12 17:14:45 [INFO] serf: EventMemberJoin: n4 172.23.0.11 | |
| 2019/07/12 17:14:45 [INFO] consul: Adding LAN server n4 (Addr: tcp/172.23.0.11:8300) (DC: dc1) | |
| 2019/07/12 17:14:45 [WARN] raft: Election timeout reached, restarting election | |
| 2019/07/12 17:14:45 [INFO] raft: Node at 172.23.0.8:8300 [Candidate] entering Candidate state in term 80 | |
| 2019/07/12 17:14:46 [INFO] raft: Node at 172.23.0.8:8300 [Follower] entering Follower state (Leader: "") | |
| 2019/07/12 17:14:46 [WARN] raft: Failed to get previous log: 201 log not found (last: 196) | |
| 2019/07/12 17:14:46 [INFO] consul: New leader elected: n4 | |
| 2019/07/12 17:14:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 7c0d0be2-a776-9624-088a-a1f6d840575b 172.23.0.11:8300}: read tcp 172.23.0.8:56594->172.23.0.11:8300: i/o timeout | |
| 2019/07/12 17:14:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 9d284f0f-d265-8dc8-52cf-89d26c0c82f7 172.23.0.10:8300}: dial tcp <nil>->172.23.0.10:8300: i/o timeout | |
| 2019/07/12 17:14:46 [ERR] raft: Failed to make RequestVote RPC to {Voter bd03081c-76cf-7c40-1a3d-e305a4db0a31 172.23.0.2:8300}: dial tcp <nil>->172.23.0.2:8300: i/o timeout |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment