nomad log

 Started nomad server.
     Loaded configuration from /etc/nomad.d/server/server.hcl
 ==> Starting Nomad agent...
     2018/02/13 06:42:22 [WARN] raft: Rejecting vote request from 10.10.1.119:8300 since we have a leader: 10.10.11.34:8300
 raft: Rejecting vote request from 10.10.1.119:8300 since we have a leader: 10.10.11.34:8300
     2018/02/13 06:42:24 [ERR] raft: peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300} has newer term, stopping replication
 raft: peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300} has newer term, stopping replication
     2018/02/13 06:42:25 [INFO] raft: aborting pipeline replication to peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300}
     2018/02/13 06:42:25 [INFO] raft: Node at 10.10.11.34:8300 [Follower] entering Follower state (Leader: "")
     2018/02/13 06:42:25 [INFO] consul: cluster leadership lost
     2018/02/13 06:42:25 [WARN] memberlist: Refuting a suspect message (from: cluster-spot-worker-03369fa4893ad47ea)
     2018/02/13 06:42:25 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
     2018/02/13 06:42:25 [INFO] raft: aborting pipeline replication to peer {Voter 876f6571-6a39-8ecd-d10c-ae37b37734da 10.10.3.249:8300}
     2018/02/13 06:42:25 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:45200->10.10.6.69:8301: i/o timeout
     2018/02/13 06:42:25 [INFO] memberlist: Suspect cluster-spot-worker-03369fa4893ad47ea has failed, no acks received
 raft: aborting pipeline replication to peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300}
 raft: Node at 10.10.11.34:8300 [Follower] entering Follower state (Leader: "")
 consul: cluster leadership lost
 memberlist: Refuting a suspect message (from: cluster-spot-worker-03369fa4893ad47ea)
 memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
 raft: aborting pipeline replication to peer {Voter 876f6571-6a39-8ecd-d10c-ae37b37734da 10.10.3.249:8300}
 memberlist: Failed fallback ping: write tcp 10.10.11.34:45200->10.10.6.69:8301: i/o timeout
 memberlist: Suspect cluster-spot-worker-03369fa4893ad47ea has failed, no acks received
     2018/02/13 06:42:30 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": context deadline exceeded
     2018/02/13 06:42:31 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
     2018/02/13 06:42:32 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
 consul.acl: Failed to get policy from ACL datacenter: No cluster leader
     2018/02/13 06:42:33 [WARN] consul.coordinate: Batch update failed: node is not the leader
 consul.coordinate: Batch update failed: node is not the leader
     2018/02/13 06:42:33 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
 consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
 2018/02/13 06:42:34 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:53248
     2018/02/13 06:42:34 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": last request still outstanding
     2018/02/13 06:42:34 [WARN] consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-094858a83da874c6a": last request still outstanding
 consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
     2018/02/13 06:42:34 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:50910->10.10.3.249:8301: i/o timeout
     2018/02/13 06:42:35 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-094858a83da874c6a)
     2018/02/13 06:42:35 [INFO] consul: New leader elected: live-cluster-admin-06bb5a1bedeb9b2a2
 memberlist: Refuting a suspect message (from: live-cluster-admin-094858a83da874c6a)
 consul: New leader elected: live-cluster-admin-06bb5a1bedeb9b2a2
     2018/02/13 06:42:35 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
 consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
     2018/02/13 06:42:35 [INFO] memberlist: Suspect live-cluster-admin-06ddc0342365da23b has failed, no acks received
 memberlist: Suspect live-cluster-admin-06ddc0342365da23b has failed, no acks received
     2018/02/13 06:42:36 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
     2018/02/13 06:42:39 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:38696->10.10.3.249:8302: i/o timeout
 memberlist: Failed fallback ping: write tcp 10.10.11.34:38696->10.10.3.249:8302: i/o timeout
     2018/02/13 06:42:43 [INFO] memberlist: Suspect live-cluster-admin-06ddc0342365da23b.us-east-1 has failed, no acks received
     2018/02/13 06:42:40 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
 2018/02/13 06:42:46 [ERR] raft-net: Failed to flush response: write tcp 10.10.11.34:8300->10.10.1.119:59380: write: broken pipe
     2018/02/13 06:42:46 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
     2018/02/13 06:42:47 [WARN] consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
     2018/02/13 06:42:49 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:55990->10.10.5.157:8301: i/o timeout
     2018/02/13 06:42:51 [INFO] memberlist: Suspect cluster-spot-worker-001782337f6a961d4 has failed, no acks received
     2018/02/13 06:42:55 [INFO] memberlist: Marking live-cluster-admin-06ddc0342365da23b as failed, suspect timeout reached (1 peer confirmations)
 2018/02/13 06:43:00 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:44746
     2018/02/13 06:43:01 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
 consul.acl: Failed to get policy from ACL datacenter: No cluster leader
     2018/02/13 06:42:59 [ERR] agent: failed to sync remote state: No cluster leader
 agent: failed to sync remote state: No cluster leader
     2018/02/13 06:43:02 [INFO] raft: Duplicate RequestVote for same term: 63
 raft: Duplicate RequestVote for same term: 63
     2018/02/13 06:43:00 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
     2018/02/13 06:43:01 [INFO] serf: EventMemberFailed: live-cluster-admin-06ddc0342365da23b 10.10.3.249
     2018/02/13 06:43:01 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
     2018/02/13 06:43:02 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
 2018/02/13 06:43:04 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:48998
     2018/02/13 06:43:03 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
     2018/02/13 06:43:03 [INFO] consul: Removing LAN server live-cluster-admin-06ddc0342365da23b (Addr: tcp/10.10.3.249:8300) (DC: us-east-1)
     2018/02/13 06:43:03 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
 memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
     2018/02/13 06:43:05 [WARN] raft: Failed to get previous log: 5376652 log not found (last: 5376650)
 raft: Failed to get previous log: 5376652 log not found (last: 5376650)
     2018/02/13 06:43:05 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
 consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
     2018/02/13 06:43:08 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:44680->10.10.1.119:8302: i/o timeout
 memberlist: Failed fallback ping: write tcp 10.10.11.34:44680->10.10.1.119:8302: i/o timeout
 ....