consul-j54c2

         Node name: 'worker1.k8s'
        Datacenter: 'dc1' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 10.233.69.200 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
    2020-12-01T14:43:21.998Z [WARN]  agent: Node name "worker1.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
==> Starting Consul agent...
           Version: '1.9.0'
           Node ID: '3b32d665-7338-5da0-d080-d2d3c0f1fb08'
    2020-12-01T14:43:22.012Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
    2020-12-01T14:43:22.012Z [WARN]  agent.router.manager: No servers available
    2020-12-01T14:43:22.008Z [WARN]  agent.auto_config: Node name "worker1.k8s" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
    2020-12-01T14:43:22.010Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
    2020-12-01T14:43:22.010Z [INFO]  agent.router: Initializing LAN area manager
    2020-12-01T14:43:22.010Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
    2020-12-01T14:43:22.010Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
    2020-12-01T14:43:22.011Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
    2020-12-01T14:43:22.011Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
    2020-12-01T14:43:22.011Z [INFO]  agent: Joining cluster...: cluster=LAN
    2020-12-01T14:43:22.012Z [INFO]  agent: started state syncer
==> Consul agent running!
    2020-12-01T14:43:22.011Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
    2020-12-01T14:43:22.011Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
    2020-12-01T14:43:22.011Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
    2020-12-01T14:43:22.018Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a dead message (from: worker1.k8s)
    2020-12-01T14:43:22.018Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:43:22.018Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
    2020-12-01T14:43:22.018Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
    2020-12-01T14:43:22.018Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152
    2020-12-01T14:43:22.393Z [INFO]  agent: Synced node info
    2020-12-01T14:43:25.034Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
    2020-12-01T14:43:25.034Z [INFO]  agent: (LAN) joined: number_of_nodes=2
    2020-12-01T14:43:27.798Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T14:43:27.869Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T14:43:32.797Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T14:43:32.798Z [INFO]  agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T14:43:45.669Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152
    2020-12-01T14:43:45.669Z [INFO]  agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
    2020-12-01T14:43:50.669Z [INFO]  agent.client: New leader elected: payload=consul-server-0
    2020-12-01T14:44:06.947Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301
    2020-12-01T14:44:06.947Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T14:44:06.947Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T14:44:18.869Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
    2020-12-01T14:44:18.869Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
    2020-12-01T14:44:20.948Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T14:44:32.547Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T14:44:32.547Z [INFO]  agent.client.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
    2020-12-01T14:44:32.547Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:01:47.010Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:01:47.895Z [INFO]  agent.client.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:01:47.895Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:01:48.748Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:01:48.748Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:01:49.809Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
    2020-12-01T15:01:50.296Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:01:50.296Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:01:51.011Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to consul-server-0 but other probes failed, network may be misconfigured
    2020-12-01T15:01:52.017Z [INFO]  agent.client.serf.lan: serf: attempting reconnect to worker3.k8s 10.233.67.183:8301
    2020-12-01T15:01:52.019Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:14.011Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:02:17.880Z [INFO]  agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:02:17.880Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:02:17.880Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:18.011Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:02:18.092Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:02:21.011Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to worker3.k8s but other probes failed, network may be misconfigured
    2020-12-01T15:02:22.009Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
    2020-12-01T15:02:22.009Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:22.493Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:22.492Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:22.493Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:02:22.615Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-2 10.233.68.21
    2020-12-01T15:02:22.616Z [INFO]  agent.client: removing server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T15:02:22.680Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T15:02:22.680Z [INFO]  agent.client: adding server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T15:02:26.810Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
    2020-12-01T15:02:31.011Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:02:32.748Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:02:35.011Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to consul-server-0 but other probes failed, network may be misconfigured
    2020-12-01T15:02:37.260Z [INFO]  agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:02:37.260Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:02:37.260Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:40.949Z [INFO]  agent.client.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
    2020-12-01T15:02:40.949Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
    2020-12-01T15:02:40.949Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:41.081Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T15:02:52.480Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:52.481Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:02:52.481Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:02:58.809Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T15:03:04.749Z [INFO]  agent.client: New leader elected: payload=consul-server-2
    2020-12-01T15:08:05.010Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
    2020-12-01T15:08:07.347Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:08:08.080Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
    2020-12-01T15:08:08.080Z [INFO]  agent.client: removing server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:08:08.080Z [INFO]  agent.client.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
    2020-12-01T15:08:09.011Z [INFO]  agent.client.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
    2020-12-01T15:08:10.809Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:08:10.809Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
    2020-12-01T15:08:10.809Z [INFO]  agent.client: removing server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T15:08:10.880Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:08:10.880Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T15:08:10.880Z [INFO]  agent.client: adding server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T15:08:10.893Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T15:08:10.893Z [INFO]  agent.client: adding server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"