Created
December 1, 2020 16:08
-
-
Save james64/d34241214554a23aebec3cc4838d5552 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Node name: 'consul-server-2' | |
| Datacenter: 'dc1' (Segment: '<all>') | |
| Server: true (Bootstrap: false) | |
| Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) | |
| Cluster Addr: 10.233.68.21 (LAN: 8301, WAN: 8302) | |
| Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false | |
| Version: '1.9.0' | |
| ==> Log data will now stream in as it occurs: | |
| 2020-12-01T14:43:32.561Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers | |
| ==> Starting Consul agent... | |
| Node ID: '48b9b896-6bb2-839f-209f-978dc8c93bfd' | |
| 2020-12-01T14:43:32.569Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers | |
| 2020-12-01T14:43:32.595Z [INFO] agent.server.raft: restored from snapshot: id=220-1458502-1606817805527 | |
| 2020-12-01T14:43:32.742Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader= | |
| 2020-12-01T14:43:32.742Z [INFO] agent.server.raft: initial configuration: index=1462060 servers="[{Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.152:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.53:8300}]" | |
| 2020-12-01T14:43:32.746Z [INFO] agent.router: Initializing LAN area manager | |
| 2020-12-01T14:43:32.746Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21 | |
| 2020-12-01T14:43:32.744Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.233.68.21 | |
| 2020-12-01T14:43:32.747Z [INFO] agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)" | |
| 2020-12-01T14:43:32.747Z [INFO] agent.server: Raft data found, disabling bootstrap mode | |
| 2020-12-01T14:43:32.747Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan | |
| 2020-12-01T14:43:32.750Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp | |
| 2020-12-01T14:43:32.750Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp | |
| 2020-12-01T14:43:32.751Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http | |
| 2020-12-01T14:43:32.751Z [INFO] agent: started state syncer | |
| 2020-12-01T14:43:32.751Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc] | |
| 2020-12-01T14:43:32.751Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them. | |
| ==> Consul agent running! | |
| 2020-12-01T14:43:32.751Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" | |
| 2020-12-01T14:43:32.751Z [INFO] agent: Joining cluster...: cluster=LAN | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200 | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20 | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53 | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152 | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)" | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)" | |
| 2020-12-01T14:43:32.756Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a dead message (from: consul-server-2) | |
| 2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183 | |
| 2020-12-01T14:43:32.758Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a dead message (from: consul-server-2.dc1) | |
| 2020-12-01T14:43:32.758Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.53 | |
| 2020-12-01T14:43:32.758Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.152 | |
| 2020-12-01T14:43:32.758Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan | |
| 2020-12-01T14:43:32.758Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan | |
| 2020-12-01T14:43:32.779Z [WARN] agent.server.raft: failed to get previous log: previous-index=1462073 last-index=1462060 error="log not found" | |
| 2020-12-01T14:43:32.786Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2 | |
| 2020-12-01T14:43:32.786Z [WARN] agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-2.consul-server.consul.svc: lookup consul-server-2.consul-server.consul.svc on 10.233.0.3:53: no such host | |
| 2020-12-01T14:43:32.786Z [INFO] agent: (LAN) joined: number_of_nodes=2 | |
| 2020-12-01T14:43:33.029Z [INFO] agent: Synced node info | |
| 2020-12-01T14:43:41.295Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-server-1.dc1 10.233.69.152 | |
| 2020-12-01T14:43:41.295Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-server-1.dc1 area=wan | |
| 2020-12-01T14:43:45.609Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152 | |
| 2020-12-01T14:43:45.609Z [INFO] agent.server: Removing LAN server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)" | |
| 2020-12-01T14:43:46.725Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.233.69.152:8300 | |
| 2020-12-01T14:43:46.725Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=228 | |
| 2020-12-01T14:43:50.442Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader=10.233.67.53:8300 | |
| 2020-12-01T14:43:50.469Z [INFO] agent.server: New leader elected: payload=consul-server-0 | |
| 2020-12-01T14:44:06.932Z [INFO] agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301 | |
| 2020-12-01T14:44:06.932Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201 | |
| 2020-12-01T14:44:06.933Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)" | |
| 2020-12-01T14:44:07.067Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.201 | |
| 2020-12-01T14:44:07.067Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan | |
| 2020-12-01T14:44:07.067Z [INFO] agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-1.dc1 from 10.233.69.152:8302 to 10.233.69.201:8302 | |
| 2020-12-01T14:44:15.067Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-server-0.dc1 10.233.67.53 | |
| 2020-12-01T14:44:15.067Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-server-0.dc1 area=wan | |
| 2020-12-01T14:44:18.880Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53 | |
| 2020-12-01T14:44:18.880Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)" | |
| 2020-12-01T14:44:20.755Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.233.67.53:8300 | |
| 2020-12-01T14:44:20.755Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=229 | |
| 2020-12-01T14:44:20.762Z [INFO] agent.server.raft: election won: tally=1 | |
| 2020-12-01T14:44:20.762Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]" | |
| 2020-12-01T14:44:20.762Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05 | |
| 2020-12-01T14:44:20.762Z [INFO] agent.server: cluster leadership acquired | |
| 2020-12-01T14:44:20.762Z [INFO] agent.server: New leader elected: payload=consul-server-2 | |
| 2020-12-01T14:44:20.765Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}" | |
| 2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" | |
| 2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="federation state pruning" | |
| 2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" | |
| 2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="CA root pruning" | |
| 2020-12-01T14:44:20.772Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left | |
| 2020-12-01T14:44:20.775Z [INFO] agent.server: member joined, marking health alive: member=consul-server-1 | |
| 2020-12-01T14:44:20.778Z [INFO] agent.server: member joined, marking health alive: member=consul-server-2 | |
| 2020-12-01T14:44:21.200Z [INFO] agent.server: federation state anti-entropy synced | |
| 2020-12-01T14:44:30.771Z [INFO] agent.server.autopilot: Promoting server: id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 address=10.233.69.201:8300 name=consul-server-1 | |
| 2020-12-01T14:44:30.771Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 server-addr=10.233.69.201:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]" | |
| 2020-12-01T14:44:32.517Z [INFO] agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-0.dc1 from 10.233.67.53:8302 to 10.233.67.184:8302 | |
| 2020-12-01T14:44:32.517Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184 | |
| 2020-12-01T14:44:32.517Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan | |
| 2020-12-01T14:44:32.518Z [INFO] agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301 | |
| 2020-12-01T14:44:32.518Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184 | |
| 2020-12-01T14:44:32.518Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T14:44:32.519Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]" | |
| 2020-12-01T14:44:32.521Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T14:44:32.525Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462091 | |
| 2020-12-01T14:44:32.525Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0 | |
| 2020-12-01T14:44:32.550Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T14:44:50.771Z [INFO] agent.server.autopilot: Promoting server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 name=consul-server-0 | |
| 2020-12-01T14:44:50.771Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]" | |
| 2020-12-01T15:01:44.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received | |
| 2020-12-01T15:01:45.293Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.500142563s | |
| 2020-12-01T15:01:45.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:01:47.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:01:47.777Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.984306253s | |
| 2020-12-01T15:01:48.010Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:01:48.011Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s | |
| 2020-12-01T15:01:48.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:01:48.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:01:48.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:01:48.747Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received | |
| 2020-12-01T15:01:48.747Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0 | |
| 2020-12-01T15:01:49.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:01:49.974Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:01:50.004Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:01:50.013Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:01:50.293Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:01:50.293Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:01:50.293Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0 | |
| 2020-12-01T15:01:52.211Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:01:52.211Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s | |
| 2020-12-01T15:02:01.036Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:02:03.020129975 +0000 UTC m=+1110.522970294" | |
| 2020-12-01T15:02:13.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received | |
| 2020-12-01T15:02:13.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:13.782Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.50016888s | |
| 2020-12-01T15:02:15.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:16.262Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.979693266s | |
| 2020-12-01T15:02:17.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received | |
| 2020-12-01T15:02:17.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:02:17.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:17.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:17.747Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0 | |
| 2020-12-01T15:02:17.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:17.892Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:02:17.892Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:02:17.892Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s | |
| 2020-12-01T15:02:18.723Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=7.440531602s | |
| 2020-12-01T15:02:19.749Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received | |
| 2020-12-01T15:02:19.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:21.341Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:02:21.341Z [ERROR] agent.server.raft: failed to pipeline appendEntries: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" error="write tcp 10.233.68.21:43522->10.233.67.184:8300: use of closed network connection" | |
| 2020-12-01T15:02:21.420Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:21.504Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.233.67.184:8300 error="read tcp 10.233.68.21:43524->10.233.67.184:8300: i/o timeout" | |
| 2020-12-01T15:02:21.515Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:21.792Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:21.792Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=c8bb9a79-dcf1-edd3-66b9-851265043986 name=consul-server-0 address=10.233.67.184:8300 | |
| 2020-12-01T15:02:21.798Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:21.799Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:21.799Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr= servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]" | |
| 2020-12-01T15:02:21.810Z [INFO] agent.server.raft: removed peer, stopping replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 last-index=1462385 | |
| 2020-12-01T15:02:21.823Z [INFO] agent.server.autopilot: removed server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:02:21.823Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left | |
| 2020-12-01T15:02:22.412Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:22.495Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1) | |
| 2020-12-01T15:02:22.611Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:02:22.611Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:22.611Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:22.611Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s | |
| 2020-12-01T15:02:22.618Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]" | |
| 2020-12-01T15:02:22.620Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:02:22.623Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0 | |
| 2020-12-01T15:02:23.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:24.498Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462385 | |
| 2020-12-01T15:02:24.572Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:02:26.717Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:02:32.9010502 +0000 UTC m=+1140.403890506" | |
| 2020-12-01T15:02:28.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received | |
| 2020-12-01T15:02:29.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:31.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:32.744Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc1 has failed, no acks received | |
| 2020-12-01T15:02:32.746Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:02:32.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:02:32.747Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s | |
| 2020-12-01T15:02:32.747Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received | |
| 2020-12-01T15:02:33.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:02:35.895Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s) | |
| 2020-12-01T15:02:37.280Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:37.280Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:37.280Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0 | |
| 2020-12-01T15:02:37.286Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:38.001Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:38.516Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:39.512Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:40.324Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:02:40.772Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=c8bb9a79-dcf1-edd3-66b9-851265043986 name=consul-server-0 address=10.233.67.184:8300 | |
| 2020-12-01T15:02:40.772Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:40.772Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr= servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]" | |
| 2020-12-01T15:02:40.772Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:40.776Z [INFO] agent.server.raft: removed peer, stopping replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 last-index=1462404 | |
| 2020-12-01T15:02:40.779Z [INFO] agent.server.autopilot: removed server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:02:40.779Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left | |
| 2020-12-01T15:02:40.776Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:02:40.897Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20 | |
| 2020-12-01T15:02:40.897Z [INFO] agent.server: member failed, marking health critical: member=worker2.k8s | |
| 2020-12-01T15:02:41.086Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20 | |
| 2020-12-01T15:02:41.086Z [INFO] agent.server: member joined, marking health alive: member=worker2.k8s | |
| 2020-12-01T15:02:42.494Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: consul-server-0.dc1) | |
| 2020-12-01T15:02:45.832Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300 | |
| 2020-12-01T15:02:46.888Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) | |
| 2020-12-01T15:02:46.888Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc1 10.233.67.184 | |
| 2020-12-01T15:02:46.888Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc1 area=wan | |
| 2020-12-01T15:02:50.571Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184 | |
| 2020-12-01T15:02:50.571Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan | |
| 2020-12-01T15:02:52.063Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300 | |
| 2020-12-01T15:02:52.480Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:02:52.480Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:02:52.480Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:02:52.480Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s | |
| 2020-12-01T15:02:52.489Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]" | |
| 2020-12-01T15:02:52.493Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:02:52.494Z [ERROR] agent.server: failed to reconcile member: member="{consul-server-0 10.233.67.184 8301 map[acls:0 build:1.9.0:a417fe51 dc:dc1 expect:3 ft_fs:1 ft_si:1 id:c8bb9a79-dcf1-edd3-66b9-851265043986 port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" | |
| 2020-12-01T15:02:52.494Z [INFO] agent.server: cluster leadership lost | |
| 2020-12-01T15:02:52.494Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:02:52.494Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader= | |
| 2020-12-01T15:02:52.494Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}" | |
| 2020-12-01T15:02:52.494Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 error="leadership lost while committing log" | |
| 2020-12-01T15:02:54.224Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader" | |
| 2020-12-01T15:02:58.700Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=230 | |
| 2020-12-01T15:02:58.700Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= | |
| 2020-12-01T15:02:58.726Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]" | |
| 2020-12-01T15:02:58.726Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05 | |
| 2020-12-01T15:02:58.726Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:02:58.726Z [INFO] agent.server.raft: election won: tally=2 | |
| 2020-12-01T15:02:58.727Z [INFO] agent.server: New leader elected: payload=consul-server-2 | |
| 2020-12-01T15:02:58.726Z [INFO] agent.server: cluster leadership acquired | |
| 2020-12-01T15:02:58.727Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}" | |
| 2020-12-01T15:02:58.728Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:02:58.733Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader= | |
| 2020-12-01T15:02:58.733Z [ERROR] agent.server: failed to wait for barrier: error="leadership lost while committing log" | |
| 2020-12-01T15:02:58.733Z [INFO] agent.server: cluster leadership lost | |
| 2020-12-01T15:02:58.733Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}" | |
| 2020-12-01T15:02:59.224Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader" | |
| 2020-12-01T15:03:01.424Z [WARN] agent.server.raft: rejecting vote request since our last term is greater: candidate=10.233.67.184:8300 last-term=230 last-candidate-term=229 | |
| 2020-12-01T15:03:04.502Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=233 | |
| 2020-12-01T15:03:04.502Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= | |
| 2020-12-01T15:03:04.563Z [INFO] agent.server.raft: election won: tally=2 | |
| 2020-12-01T15:03:04.563Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]" | |
| 2020-12-01T15:03:04.563Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05 | |
| 2020-12-01T15:03:04.563Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 | |
| 2020-12-01T15:03:04.563Z [INFO] agent.server: cluster leadership acquired | |
| 2020-12-01T15:03:04.564Z [INFO] agent.server: New leader elected: payload=consul-server-2 | |
| 2020-12-01T15:03:04.564Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}" | |
| 2020-12-01T15:03:04.569Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462405 | |
| 2020-12-01T15:03:04.576Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" | |
| 2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" | |
| 2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="CA root pruning" | |
| 2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="federation state pruning" | |
| 2020-12-01T15:03:04.585Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0 | |
| 2020-12-01T15:03:14.586Z [INFO] agent.server.autopilot: Promoting server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 name=consul-server-0 | |
| 2020-12-01T15:03:14.586Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]" | |
| 2020-12-01T15:04:50.554Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled" | |
| 2020-12-01T15:08:03.134Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.500124934s | |
| 2020-12-01T15:08:03.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:03.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received | |
| 2020-12-01T15:08:05.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:05.596Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.962230093s | |
| 2020-12-01T15:08:07.292Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:08:07.292Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:08:07.292Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s | |
| 2020-12-01T15:08:07.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:07.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received | |
| 2020-12-01T15:08:07.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations) | |
| 2020-12-01T15:08:07.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:08:07.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:08:07.748Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0 | |
| 2020-12-01T15:08:08.087Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=7.452862656s | |
| 2020-12-01T15:08:09.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:10.670Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:08:10.711Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986" | |
| 2020-12-01T15:08:10.734Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184 | |
| 2020-12-01T15:08:10.734Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183 | |
| 2020-12-01T15:08:10.734Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)" | |
| 2020-12-01T15:08:10.735Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0 | |
| 2020-12-01T15:08:10.741Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s | |
| 2020-12-01T15:08:10.808Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201 | |
| 2020-12-01T15:08:10.808Z [INFO] agent.server: member failed, marking health critical: member=consul-server-1 | |
| 2020-12-01T15:08:10.808Z [INFO] agent.server: Removing LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)" | |
| 2020-12-01T15:08:10.816Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 fallback=10.233.69.201:8300 error="Could not find address for server id 3048cdda-bdc9-e93e-54eb-809e8c03ae05" | |
| 2020-12-01T15:08:10.892Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201 | |
| 2020-12-01T15:08:10.892Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)" | |
| 2020-12-01T15:08:10.892Z [INFO] agent.server: member joined, marking health alive: member=consul-server-1 | |
| 2020-12-01T15:08:11.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:11.608Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.233.67.184:8300 error="read tcp 10.233.68.21:37132->10.233.67.184:8300: i/o timeout" | |
| 2020-12-01T15:08:13.460Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" | |
| 2020-12-01T15:08:13.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:15.183Z [ERROR] agent: yamux: keepalive failed: i/o deadline reached | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF" | |
| 2020-12-01T15:08:15.586Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded" | |
| 2020-12-01T15:08:17.494Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: consul-server-0.dc1) | |
| 2020-12-01T15:08:20.728Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:08:22.301795542 +0000 UTC m=+1489.804635841" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment