consul-server-1

           Node ID: '3048cdda-bdc9-e93e-54eb-809e8c03ae05'
         Node name: 'consul-server-1'
      Cluster Addr: 10.233.69.201 (LAN: 8301, WAN: 8302)
==> Starting Consul agent...
           Version: '1.9.0'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
    2020-12-01T14:44:06.692Z [WARN]  agent: bootstrap_expect > 0: expecting 3 servers
    2020-12-01T14:44:06.702Z [WARN]  agent.auto_config: bootstrap_expect > 0: expecting 3 servers
    2020-12-01T14:44:06.726Z [INFO]  agent.server.raft: restored from snapshot: id=222-1458580-1606818035001
    2020-12-01T14:44:06.885Z [INFO]  agent.server.raft: initial configuration: index=1462077 servers="[{Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.53:8300} {Suffrage:Nonvoter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300}]"
    2020-12-01T14:44:06.885Z [INFO]  agent.server.raft: entering follower state: follower="Node at 10.233.69.201:8300 [Follower]" leader=
    2020-12-01T14:44:06.887Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.201
    2020-12-01T14:44:06.891Z [INFO]  agent.router: Initializing LAN area manager
    2020-12-01T14:44:06.892Z [INFO]  agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T14:44:06.892Z [INFO]  agent.server: Raft data found, disabling bootstrap mode
    2020-12-01T14:44:06.891Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T14:44:06.892Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
    2020-12-01T14:44:06.899Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
    2020-12-01T14:44:06.899Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
    2020-12-01T14:44:06.898Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
    2020-12-01T14:44:06.899Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
    2020-12-01T14:44:06.899Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
    2020-12-01T14:44:06.900Z [INFO]  agent: Joining cluster...: cluster=LAN
    2020-12-01T14:44:06.900Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
    2020-12-01T14:44:06.900Z [INFO]  agent: started state syncer
==> Consul agent running!
    2020-12-01T14:44:06.912Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T14:44:06.912Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
    2020-12-01T14:44:06.912Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:44:06.913Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T14:44:06.913Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
    2020-12-01T14:44:06.913Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a dead message (from: consul-server-1)
    2020-12-01T14:44:06.914Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T14:44:06.914Z [INFO]  agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T14:44:06.915Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.233.68.21
    2020-12-01T14:44:06.915Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.53
    2020-12-01T14:44:06.915Z [WARN]  agent.server.memberlist.wan: memberlist: Refuting a dead message (from: consul-server-1.dc1)
    2020-12-01T14:44:06.915Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan
    2020-12-01T14:44:06.915Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
    2020-12-01T14:44:06.925Z [WARN]  agent.server.raft: failed to get previous log: previous-index=1462084 last-index=1462077 error="log not found"
    2020-12-01T14:44:06.928Z [WARN]  agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-1.consul-server.consul.svc: lookup consul-server-1.consul-server.consul.svc on 10.233.0.3:53: no such host
    2020-12-01T14:44:06.933Z [INFO]  agent: (LAN) joined: number_of_nodes=2
    2020-12-01T14:44:06.933Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
    2020-12-01T14:44:07.215Z [INFO]  agent: Synced node info
    2020-12-01T14:44:07.220Z [INFO]  agent: Synced service: service=devportal-be-8080
    2020-12-01T14:44:15.067Z [INFO]  agent.server.serf.wan: serf: EventMemberLeave: consul-server-0.dc1 10.233.67.53
    2020-12-01T14:44:15.067Z [INFO]  agent.server: Handled event for server in area: event=member-leave server=consul-server-0.dc1 area=wan
    2020-12-01T14:44:18.880Z [INFO]  agent.server.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
    2020-12-01T14:44:18.880Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:44:20.892Z [INFO]  agent.server: New leader elected: payload=consul-server-2
    2020-12-01T14:44:21.218Z [INFO]  agent: Synced check: check=service:devportal-be-8080
    2020-12-01T14:44:32.514Z [INFO]  agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
    2020-12-01T14:44:32.514Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T14:44:32.514Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T14:44:32.516Z [INFO]  agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-0.dc1 from 10.233.67.53:8302 to 10.233.67.184:8302
    2020-12-01T14:44:32.516Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
    2020-12-01T14:44:32.516Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
    2020-12-01T15:01:43.894Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:01:47.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:01:47.895Z [INFO]  agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:01:47.895Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:01:48.881Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:01:48.881Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:01:50.251Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:01:50.251Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:01:52.296Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:13.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:02:16.888Z [INFO]  agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc1 has failed, no acks received
    2020-12-01T15:02:17.880Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:02:17.880Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:17.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:02:17.893Z [INFO]  agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:02:17.893Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:02:21.950Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:21.950Z [INFO]  agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
    2020-12-01T15:02:22.433Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:22.433Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:02:22.433Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:29.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:02:32.747Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:02:32.747Z [INFO]  agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:02:34.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:02:36.889Z [WARN]  agent.server.memberlist.wan: memberlist: Was able to connect to consul-server-0.dc1 but other probes failed, network may be misconfigured
    2020-12-01T15:02:37.280Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:02:37.280Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:40.811Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
    2020-12-01T15:02:40.893Z [INFO]  agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
    2020-12-01T15:02:40.893Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:40.896Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
    2020-12-01T15:02:41.096Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T15:02:45.832Z [WARN]  agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
    2020-12-01T15:02:46.889Z [INFO]  agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2020-12-01T15:02:46.889Z [INFO]  agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc1 10.233.67.184
    2020-12-01T15:02:46.889Z [INFO]  agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc1 area=wan
    2020-12-01T15:02:50.745Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
    2020-12-01T15:02:50.745Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
    2020-12-01T15:02:52.063Z [WARN]  agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
    2020-12-01T15:02:52.547Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:52.548Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:02:52.548Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:58.738Z [ERROR] agent.server.raft: failed to flush response: error="write tcp 10.233.69.201:8300->10.233.68.21:37574: write: broken pipe"
    2020-12-01T15:02:58.747Z [INFO]  agent.server: New leader elected: payload=consul-server-2
    2020-12-01T15:03:01.392Z [WARN]  agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
    2020-12-01T15:03:04.747Z [INFO]  agent.server: New leader elected: payload=consul-server-2
    2020-12-01T15:08:01.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:08:05.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:08:07.347Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:08:07.947Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:08:07.947Z [INFO]  agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:08:09.892Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:08:10.747Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:08:10.747Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:08:10.747Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:08:10.747Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"