Skip to content

Instantly share code, notes, and snippets.

@bgandon

bgandon/consul_z1-0

Created April 14, 2016 14:32
Show Gist options
  • Save bgandon/08707466324be7c9a093a56fd95a64e4 to your computer and use it in GitHub Desktop.
Save bgandon/08707466324be7c9a093a56fd95a64e4 to your computer and use it in GitHub Desktop.
Download ZIP
Sync failures with 3-nodes consul cluster in cf v231
Raw
consul_z1-0
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:49:24] info : 'consul_agent' trying to restart
[UTC Apr 14 13:49:24] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:49:56] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:49:56] info : monit daemon at 101 awakened
[UTC Apr 14 13:50:24] error : 'consul_agent' failed to start
[UTC Apr 14 13:50:24] info : Awakened by User defined signal 1
[UTC Apr 14 13:50:24] info : 'consul_agent' stop action done
[UTC Apr 14 13:53:57] info : start service 'consul_agent' on user request
[UTC Apr 14 13:53:57] info : monit daemon at 101 awakened
[UTC Apr 14 13:53:57] info : Awakened by User defined signal 1
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460642037.420836210","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460642037.420992851","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z1-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.0.54","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460642037.421272516","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460642037.421312332","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460642037.421325207","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460642037.421336174","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460642037.421345949","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460642037.421418190","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460642037.422872782","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460642037.422965527","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460642037.422995806","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642037.424357414","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:53:57] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460642038.427438498","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642038.430511475","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z1-0'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.0.54 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: consul-z1-0.dc1 10.244.0.54
2016/04/14 13:53:58 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 13:53:58 [INFO] serf: Attempting re-join to previously known node: cell-z3-0: 10.244.20.6:8301
2016/04/14 13:53:58 [WARN] serf: Failed to re-join any previously known node
2016/04/14 13:53:58 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:53:58 [INFO] agent: Joining cluster...
2016/04/14 13:53:58 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 13:53:58 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 13:53:58 [INFO] serf: Re-joined to previously known node: cell-z3-0: 10.244.20.6:8301
2016/04/14 13:53:58 [INFO] agent: (LAN) joined: 1 Err: <nil>
2016/04/14 13:53:58 [INFO] agent: Join completed. Synced with 1 initial agents
{"timestamp":"1460642039.441989183","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642039.806461811","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.0.138","10.244.0.154","10.244.0.54","10.244.20.6","10.244.18.2","10.244.16.2","10.244.18.22","10.244.18.6","10.244.0.134","10.244.16.6","10.244.18.10","10.244.18.18","10.244.16.10","10.244.2.134","10.244.16.22","10.244.16.14","10.244.2.130","10.244.18.14","10.244.16.18","10.244.0.22","10.244.2.22","10.244.2.138","10.244.0.130","10.244.20.2","10.244.0.34"],"wan":false}}
{"timestamp":"1460642039.808006048","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460642039.808039188","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 13:53:59 [INFO] agent.rpc: Accepted client: 127.0.0.1:38959
{"timestamp":"1460642039.825457335","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}}
{"timestamp":"1460642039.825728178","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642040.026663542","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.0.54","10.244.20.6","10.244.18.2","10.244.16.2","10.244.18.22","10.244.18.6","10.244.0.138","10.244.0.154","10.244.0.134","10.244.16.6","10.244.18.10","10.244.18.18","10.244.16.10","10.244.16.14","10.244.2.130","10.244.18.14","10.244.16.18","10.244.0.22","10.244.2.22","10.244.2.134","10.244.16.22","10.244.2.138","10.244.0.130","10.244.20.2","10.244.0.34"],"wan":false}}
{"timestamp":"1460642040.026713371","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":1,"expected_members_count":3,"is_last_node":false}}
{"timestamp":"1460642040.026733398","source":"confab","message":"confab.controller.configure-server.set-keys","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460642040.026747227","source":"confab","message":"confab.agent-client.set-keys.list-keys.request","log_level":1,"data":{}}
2016/04/14 13:54:00 [INFO] serf: Received list-keys query
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 13:54:00 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:00 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:00 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:03 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: dial tcp 10.244.2.54:8300: connection refused
2016/04/14 13:54:05 [INFO] memberlist: Suspect api-worker-z2-0 has failed, no acks received
2016/04/14 13:54:05 [INFO] serf: Received list-keys query
2016/04/14 13:54:05 [WARN] raft: Rejecting vote from 10.244.0.58:8300 since our last term is greater (90, 88)
2016/04/14 13:54:05 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 13:54:06 [WARN] memberlist: Refuting a suspect message (from: cell-z3-0)
2016/04/14 13:54:06 [INFO] memberlist: Suspect consul-z1-1 has failed, no acks received
2016/04/14 13:54:07 [INFO] serf: Received list-keys query
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 13:54:08 [INFO] memberlist: Suspect database-z3-0 has failed, no acks received
2016/04/14 13:54:08 [WARN] raft: Rejecting vote from 10.244.0.58:8300 since our last term is greater (90, 88)
2016/04/14 13:54:08 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:08 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:08 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:09 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
{"timestamp":"1460642050.941734552","source":"confab","message":"confab.agent-client.set-keys.list-keys.response","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460642051.013489485","source":"confab","message":"confab.agent-client.set-keys.install-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 13:54:11 [INFO] serf: Received install-key query
2016/04/14 13:54:11 [INFO] memberlist: Suspect api-z2-0 has failed, no acks received
2016/04/14 13:54:11 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:11 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:11 [WARN] memberlist: Refuting a suspect message (from: router-z2-0)
2016/04/14 13:54:12 [INFO] memberlist: Suspect access-z1-0 has failed, no acks received
2016/04/14 13:54:13 [INFO] raft: Duplicate RequestVote for same term: 282
2016/04/14 13:54:13 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:13 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:13 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 13:54:14 [INFO] raft: Duplicate RequestVote for same term: 283
2016/04/14 13:54:16 [INFO] memberlist: Suspect database-z2-0 has failed, no acks received
2016/04/14 13:54:16 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:16 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:17 [INFO] memberlist: Suspect cell-z3-0 has failed, no acks received
2016/04/14 13:54:19 [INFO] memberlist: Suspect uaa-z1-0 has failed, no acks received
2016/04/14 13:54:19 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:19 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:20 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received
2016/04/14 13:54:22 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:22 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:22 [INFO] memberlist: Suspect brain-z2-0 has failed, no acks received
2016/04/14 13:54:23 [INFO] memberlist: Marking cc-bridge-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:23 [INFO] serf: EventMemberFailed: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:23 [INFO] memberlist: Marking consul-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:23 [INFO] serf: EventMemberFailed: consul-z2-0 10.244.2.54
2016/04/14 13:54:23 [INFO] consul: removing server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:23 [INFO] memberlist: Suspect cc-bridge-z2-0 has failed, no acks received
2016/04/14 13:54:24 [INFO] memberlist: Marking blobstore-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:24 [INFO] serf: EventMemberFailed: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:24 [INFO] memberlist: Marking consul-z1-1 as failed, suspect timeout reached
2016/04/14 13:54:24 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 13:54:24 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:25 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:25 [WARN] raft: Rejecting vote from 10.244.0.58:8300 since our last term is greater (90, 88) 2016/04/14 13:54:25 [INFO] memberlist: Suspect route-emitter-z1-0 has failed, no acks received
2016/04/14 13:54:25 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 13:54:26 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:26 [INFO] memberlist: Suspect route-emitter-z2-0 has failed, no acks received
2016/04/14 13:54:26 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:26 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:28 [INFO] memberlist: Suspect uaa-z2-0 has failed, no acks received
2016/04/14 13:54:28 [INFO] serf: attempting reconnect to consul-z2-0 10.244.2.54:8301
2016/04/14 13:54:29 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: read tcp 10.244.2.54:8300: i/o timeout
2016/04/14 13:54:29 [INFO] memberlist: Marking api-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:29 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 13:54:29 [INFO] memberlist: Marking api-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:29 [INFO] serf: EventMemberFailed: api-z1-0 10.244.0.138
2016/04/14 13:54:29 [INFO] memberlist: Marking uaa-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:29 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 13:54:29 [INFO] memberlist: Suspect cell-z1-0 has failed, no acks received
2016/04/14 13:54:29 [INFO] raft: Duplicate RequestVote for same term: 288
2016/04/14 13:54:29 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
2016/04/14 13:54:30 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
{"timestamp":"1460642070.810367823","source":"confab","message":"confab.agent-client.set-keys.install-key.request.failed","log_level":2,"data":{"error":"0/26 nodes reported success","key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460642070.837576151","source":"confab","message":"confab.controller.configure-server.set-keys.failed","log_level":2,"data":{"error":"0/26 nodes reported success","keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
2016/04/14 13:54:31 [INFO] memberlist: Suspect router-z2-0 has failed, no acks received
2016/04/14 13:54:31 [INFO] agent.rpc: Accepted client: 127.0.0.1:40003
{"timestamp":"1460642071.925284624","source":"confab","message":"confab.controller.stop-agent.leave","log_level":1,"data":{}}
{"timestamp":"1460642072.315482616","source":"confab","message":"confab.agent-client.leave.leave.request","log_level":1,"data":{}}
2016/04/14 13:54:33 [INFO] agent.rpc: Graceful leave triggered
2016/04/14 13:54:33 [INFO] consul: server starting leave 2016/04/14 13:54:33 [WARN] memberlist: Refuting a suspect message (from: database-z1-0)
2016/04/14 13:54:33 [INFO] serf: EventMemberLeave: consul-z1-0.dc1 10.244.0.54
2016/04/14 13:54:33 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:33 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:34 [INFO] consul: removing server consul-z1-0.dc1 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 13:54:33 [INFO] memberlist: Suspect cc-bridge-z1-0 has failed, no acks received
2016/04/14 13:54:34 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:34 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:34 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
2016/04/14 13:54:34 [INFO] serf: EventMemberLeave: consul-z1-0 10.244.0.54
2016/04/14 13:54:34 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 13:54:34 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:35 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:35 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 13:54:35 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 13:54:35 [INFO] memberlist: Marking database-z3-0 as failed, suspect timeout reached
2016/04/14 13:54:35 [INFO] serf: EventMemberFailed: database-z3-0 10.244.20.2
2016/04/14 13:54:36 [INFO] raft: Duplicate RequestVote for same term: 289
2016/04/14 13:54:36 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received 2016/04/14 13:54:36 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2 2016/04/14 13:54:36 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:36 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:36 [ERR] yamux: Failed to write header: use of closed network connection
2016/04/14 13:54:36 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:36 [INFO] serf: Received install-key query
2016/04/14 13:54:36 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
2016/04/14 13:54:37 [INFO] memberlist: Suspect access-z2-0 has failed, no acks received
2016/04/14 13:54:38 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received
2016/04/14 13:54:38 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:38 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:38 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: read tcp 10.244.2.54:8300: i/o timeout
2016/04/14 13:54:38 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 13:54:40 [INFO] memberlist: Suspect ha-proxy-z1-0 has failed, no acks received
2016/04/14 13:54:41 [WARN] raft: Election timeout reached, restarting election 2016/04/14 13:54:41 [WARN] consul: failed to leave raft peer set gracefully, timeout
2016/04/14 13:54:41 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:41 [INFO] agent: requesting shutdown
2016/04/14 13:54:41 [INFO] consul: shutting down server
2016/04/14 13:54:41 [ERR] memberlist: Failed to send indirect ping: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send indirect ping: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send indirect ping: use of closed network connection
{"timestamp":"1460642081.770420313","source":"confab","message":"confab.agent-client.leave.leave.response","log_level":1,"data":{}}
{"timestamp":"1460642081.787928581","source":"confab","message":"confab.controller.stop-agent.wait","log_level":1,"data":{}}
{"timestamp":"1460642081.788016319","source":"confab","message":"confab.agent-runner.wait.get-process","log_level":1,"data":{}}
{"timestamp":"1460642081.788021088","source":"confab","message":"confab.agent-runner.wait.get-process.result","log_level":1,"data":{"pid":4539}}
{"timestamp":"1460642081.788771868","source":"confab","message":"confab.agent-runner.wait.signal","log_level":1,"data":{"pid":4539}}
2016/04/14 13:54:41 [ERR] memberlist: Failed to send gossip to 10.244.16.18:8301: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send gossip to 10.244.2.138:8301: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send gossip to 10.244.20.2:8301: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send gossip to 10.244.2.22:8301: use of closed network connection
2016/04/14 13:54:41 [ERR] memberlist: Failed to send gossip to 10.244.0.154:8301: use of closed network connection
2016/04/14 13:54:41 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
2016/04/14 13:54:42 [INFO] memberlist: Suspect cell-z2-0 has failed, no acks received
2016/04/14 13:54:42 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: dial tcp 10.244.0.58:8300: connection refused
2016/04/14 13:54:45 [INFO] memberlist: Marking brain-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:45 [INFO] serf: EventMemberFailed: brain-z2-0 10.244.18.6
2016/04/14 13:54:45 [INFO] memberlist: Marking access-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:45 [INFO] serf: EventMemberFailed: access-z2-0 10.244.18.22
2016/04/14 13:54:45 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:46 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:46 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 13:54:46 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: read tcp 10.244.2.54:8300: i/o timeout
2016/04/14 13:54:48 [INFO] memberlist: Marking brain-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:48 [INFO] serf: EventMemberFailed: brain-z1-0 10.244.16.6
2016/04/14 13:54:49 [INFO] memberlist: Marking consul-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:49 [INFO] serf: EventMemberFailed: consul-z2-0 10.244.2.54
2016/04/14 13:54:49 [INFO] memberlist: Marking cc-bridge-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:49 [INFO] serf: EventMemberFailed: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:49 [INFO] memberlist: Marking cell-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:49 [INFO] serf: EventMemberFailed: cell-z1-0 10.244.16.10
2016/04/14 13:54:50 [INFO] memberlist: Marking uaa-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:50 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 13:54:51 [ERR] raft-net: Failed to decode incoming command: transport shutdown
2016/04/14 13:54:51 [INFO] memberlist: Marking route-emitter-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:51 [INFO] serf: EventMemberFailed: route-emitter-z1-0 10.244.16.18
2016/04/14 13:54:53 [INFO] agent: shutdown complete
{"timestamp":"1460642093.327833652","source":"confab","message":"confab.agent-runner.wait.success","log_level":1,"data":{}}
{"timestamp":"1460642093.331074953","source":"confab","message":"confab.controller.stop-agent.cleanup","log_level":1,"data":{}}
{"timestamp":"1460642093.331202269","source":"confab","message":"confab.agent-runner.cleanup.remove","log_level":1,"data":{"pidfile":"/var/vcap/sys/run/consul_agent/consul_agent.pid"}}
{"timestamp":"1460642093.331332207","source":"confab","message":"confab.agent-runner.cleanup.remove.failed","log_level":2,"data":{"error":"remove /var/vcap/sys/run/consul_agent/consul_agent.pid: no such file or directory","pidfile":"/var/vcap/sys/run/consul_agent/consul_agent.pid"}}
{"timestamp":"1460642093.331429482","source":"confab","message":"confab.controller.stop-agent.cleanup.failed","log_level":2,"data":{"error":"remove /var/vcap/sys/run/consul_agent/consul_agent.pid: no such file or directory"}}
{"timestamp":"1460642093.331509352","source":"confab","message":"confab.controller.stop-agent.success","log_level":1,"data":{}}
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:54:57] error : 'consul_agent' failed to start
[UTC Apr 14 13:54:57] info : 'consul_agent' start action done
Raw
consul_z1-1
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:49:26] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:50:01] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:50:01] info : monit daemon at 102 awakened
[UTC Apr 14 13:50:26] error : 'consul_agent' failed to start
[UTC Apr 14 13:50:26] info : Awakened by User defined signal 1
[UTC Apr 14 13:50:26] info : 'consul_agent' stop action done
[UTC Apr 14 13:53:58] info : start service 'consul_agent' on user request
[UTC Apr 14 13:53:58] info : monit daemon at 102 awakened
[UTC Apr 14 13:53:58] info : Awakened by User defined signal 1
[UTC Apr 14 13:53:58] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460642038.406530857","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460642038.407234669","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z1-1","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.0.58","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460642038.407860518","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460642038.407881260","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460642038.407893658","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460642038.407903910","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460642038.407914162","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460642038.407977581","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460642038.412675619","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460642038.412804127","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460642038.412853241","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642038.413581371","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
{"timestamp":"1460642039.433968782","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642039.434643745","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z1-1'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.0.58 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 13:53:59 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:53:59 [INFO] serf: EventMemberJoin: consul-z1-1.dc1 10.244.0.58
2016/04/14 13:53:59 [INFO] serf: Attempting re-join to previously known node: uaa-z1-0: 10.244.0.134:8301
2016/04/14 13:53:59 [WARN] serf: Failed to re-join any previously known node
2016/04/14 13:53:59 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:00 [INFO] consul: adding server consul-z1-1.dc1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:00 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 13:54:00 [INFO] agent: Joining cluster...
2016/04/14 13:54:00 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 13:54:00 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 13:54:00 [INFO] serf: Re-joined to previously known node: uaa-z1-0: 10.244.0.134:8301
2016/04/14 13:54:00 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 13:54:00 [INFO] agent: (LAN) joined: 2 Err: <nil>
2016/04/14 13:54:00 [INFO] agent: Join completed. Synced with 2 initial agents
{"timestamp":"1460642040.490830183","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642040.628770351","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.2.130","10.244.0.134","10.244.2.138","10.244.0.22","10.244.0.54","10.244.16.22","10.244.16.2","10.244.18.18","10.244.2.22","10.244.2.134","10.244.20.6","10.244.16.10","10.244.16.14","10.244.18.22","10.244.18.6","10.244.18.14","10.244.18.10","10.244.0.138","10.244.16.18","10.244.0.34","10.244.0.154","10.244.20.2","10.244.0.58","10.244.16.6","10.244.18.2","10.244.0.130"],"wan":false}}
{"timestamp":"1460642040.629026651","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460642040.629045010","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 13:54:00 [INFO] agent.rpc: Accepted client: 127.0.0.1:39000
{"timestamp":"1460642040.950230360","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}}
{"timestamp":"1460642040.962010145","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642041.020547867","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.0.58","10.244.16.6","10.244.18.2","10.244.0.130","10.244.2.130","10.244.0.134","10.244.2.138","10.244.0.22","10.244.0.54","10.244.16.22","10.244.16.2","10.244.2.22","10.244.2.134","10.244.20.6","10.244.16.10","10.244.16.14","10.244.18.22","10.244.18.6","10.244.18.18","10.244.18.14","10.244.18.10","10.244.0.138","10.244.16.18","10.244.0.34","10.244.0.154","10.244.20.2"],"wan":false}}
{"timestamp":"1460642041.171573877","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":2,"expected_members_count":3,"is_last_node":false}}
{"timestamp":"1460642041.171644449","source":"confab","message":"confab.controller.configure-server.set-keys","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460642041.171672821","source":"confab","message":"confab.agent-client.set-keys.list-keys.request","log_level":1,"data":{}}
2016/04/14 13:54:01 [INFO] serf: Received list-keys query
2016/04/14 13:54:01 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:01 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:03 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: dial tcp 10.244.2.54:8300: connection refused
2016/04/14 13:54:04 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:04 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:04 [INFO] serf: Received list-keys query
2016/04/14 13:54:05 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: dial tcp 10.244.2.54:8300: connection refused
2016/04/14 13:54:06 [INFO] serf: Received list-keys query
2016/04/14 13:54:06 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:06 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 13:54:07 [WARN] memberlist: Refuting a suspect message (from: consul-z1-0)
2016/04/14 13:54:08 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:09 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since our last index is greater (2277, 2276)
2016/04/14 13:54:09 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 13:54:11 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:11 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
{"timestamp":"1460642052.388057232","source":"confab","message":"confab.agent-client.set-keys.list-keys.request.failed","log_level":2,"data":{"error":"25/26 nodes reported success"}}
{"timestamp":"1460642052.466287851","source":"confab","message":"confab.controller.configure-server.set-keys.failed","log_level":2,"data":{"error":"25/26 nodes reported success","keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
2016/04/14 13:54:13 [WARN] serf: reply for non-running query (LTime: 229, ID: 1100235400) From: consul-z1-0
2016/04/14 13:54:15 [INFO] agent.rpc: Accepted client: 127.0.0.1:39427
2016/04/14 13:54:15 [INFO] memberlist: Suspect api-worker-z1-0 has failed, no acks received
{"timestamp":"1460642055.390897989","source":"confab","message":"confab.controller.stop-agent.leave","log_level":1,"data":{}}
{"timestamp":"1460642055.392159700","source":"confab","message":"confab.agent-client.leave.leave.request","log_level":1,"data":{}}
2016/04/14 13:54:15 [INFO] agent.rpc: Graceful leave triggered
2016/04/14 13:54:15 [INFO] consul: server starting leave
2016/04/14 13:54:15 [INFO] serf: EventMemberLeave: consul-z1-1.dc1 10.244.0.58
2016/04/14 13:54:17 [INFO] consul: removing server consul-z1-1.dc1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:18 [INFO] raft: Duplicate RequestVote for same term: 282
2016/04/14 13:54:19 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:19 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:20 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since our last index is greater (2277, 2276) 2016/04/14 13:54:20 [INFO] serf: EventMemberLeave: consul-z1-1 10.244.0.58
2016/04/14 13:54:20 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:20 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received 2016/04/14 13:54:20 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 13:54:22 [INFO] memberlist: Suspect uaa-z2-0 has failed, no acks received
2016/04/14 13:54:22 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since our last index is greater (2277, 2276)
2016/04/14 13:54:22 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:22 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:23 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:23 [INFO] serf: Received install-key query
2016/04/14 13:54:23 [INFO] memberlist: Suspect database-z3-0 has failed, no acks received
2016/04/14 13:54:25 [INFO] memberlist: Marking api-worker-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:25 [INFO] serf: EventMemberFailed: api-worker-z1-0 10.244.0.154
2016/04/14 13:54:25 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:25 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 13:54:25 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received
2016/04/14 13:54:26 [WARN] consul: failed to leave raft peer set gracefully, timeout
2016/04/14 13:54:26 [INFO] agent: requesting shutdown
2016/04/14 13:54:26 [INFO] consul: shutting down server
2016/04/14 13:54:26 [ERR] memberlist: Failed to send gossip to 10.244.18.22:8301: use of closed network connection
2016/04/14 13:54:26 [ERR] memberlist: Failed to send gossip to 10.244.18.6:8301: use of closed network connection
{"timestamp":"1460642066.733146667","source":"confab","message":"confab.agent-client.leave.leave.response","log_level":1,"data":{}}
{"timestamp":"1460642066.733985186","source":"confab","message":"confab.controller.stop-agent.wait","log_level":1,"data":{}}
{"timestamp":"1460642066.733999729","source":"confab","message":"confab.agent-runner.wait.get-process","log_level":1,"data":{}}
{"timestamp":"1460642066.734011889","source":"confab","message":"confab.agent-runner.wait.get-process.result","log_level":1,"data":{"pid":4368}}
{"timestamp":"1460642066.734026670","source":"confab","message":"confab.agent-runner.wait.signal","log_level":1,"data":{"pid":4368}}
2016/04/14 13:54:26 [ERR] memberlist: Failed to send ping: use of closed network connection
2016/04/14 13:54:26 [ERR] memberlist: Failed to send ack: use of closed network connection
2016/04/14 13:54:29 [ERR] raft-net: Failed to decode incoming command: transport shutdown
2016/04/14 13:54:29 [ERR] raft-net: Failed to decode incoming command: transport shutdown
2016/04/14 13:54:29 [INFO] memberlist: Marking consul-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:29 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 13:54:30 [ERR] raft: Failed to make RequestVote RPC to 10.244.2.54:8300: read tcp 10.244.2.54:8300: i/o timeout
2016/04/14 13:54:31 [INFO] memberlist: Marking brain-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:31 [INFO] serf: EventMemberFailed: brain-z1-0 10.244.16.6
2016/04/14 13:54:32 [INFO] memberlist: Marking uaa-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:32 [INFO] serf: EventMemberFailed: uaa-z2-0 10.244.2.130
2016/04/14 13:54:33 [INFO] memberlist: Marking consul-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:33 [INFO] serf: EventMemberFailed: consul-z2-0 10.244.2.54
2016/04/14 13:54:33 [INFO] memberlist: Marking cc-bridge-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:33 [INFO] serf: EventMemberFailed: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:34 [INFO] memberlist: Marking database-z3-0 as failed, suspect timeout reached
2016/04/14 13:54:34 [INFO] serf: EventMemberFailed: database-z3-0 10.244.20.2
2016/04/14 13:54:34 [ERR] raft-net: Failed to decode incoming command: transport shutdown
2016/04/14 13:54:36 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:36 [ERR] raft-net: Failed to decode incoming command: transport shutdown
2016/04/14 13:54:36 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:36 [INFO] memberlist: Marking database-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:36 [INFO] serf: EventMemberFailed: database-z1-0 10.244.16.2
2016/04/14 13:54:40 [INFO] agent: shutdown complete
2016/04/14 13:54:40 [ERR] agent: failed to sync remote state: No cluster leader
{"timestamp":"1460642081.254880667","source":"confab","message":"confab.agent-runner.wait.success","log_level":1,"data":{}}
{"timestamp":"1460642081.272295475","source":"confab","message":"confab.controller.stop-agent.cleanup","log_level":1,"data":{}}
{"timestamp":"1460642081.272436142","source":"confab","message":"confab.agent-runner.cleanup.remove","log_level":1,"data":{"pidfile":"/var/vcap/sys/run/consul_agent/consul_agent.pid"}}
{"timestamp":"1460642081.272578955","source":"confab","message":"confab.agent-runner.cleanup.remove.failed","log_level":2,"data":{"error":"remove /var/vcap/sys/run/consul_agent/consul_agent.pid: no such file or directory","pidfile":"/var/vcap/sys/run/consul_agent/consul_agent.pid"}}
{"timestamp":"1460642081.272986889","source":"confab","message":"confab.controller.stop-agent.cleanup.failed","log_level":2,"data":{"error":"remove /var/vcap/sys/run/consul_agent/consul_agent.pid: no such file or directory"}}
{"timestamp":"1460642081.273078680","source":"confab","message":"confab.controller.stop-agent.success","log_level":1,"data":{}}
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:54:58] error : 'consul_agent' failed to start
[UTC Apr 14 13:54:58] info : 'consul_agent' start action done
Raw
consul_z2-0
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:49:29] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:50:02] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:50:02] info : monit daemon at 103 awakened
[UTC Apr 14 13:50:29] error : 'consul_agent' failed to start
[UTC Apr 14 13:50:29] info : Awakened by User defined signal 1
[UTC Apr 14 13:50:29] info : 'consul_agent' stop action done
[UTC Apr 14 13:53:59] info : start service 'consul_agent' on user request
[UTC Apr 14 13:53:59] info : Awakened by User defined signal 1
[UTC Apr 14 13:53:59] info : monit daemon at 103 awakened
[UTC Apr 14 13:53:59] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460642040.953299522","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460642040.956568241","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z2-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.2.54","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460642040.960468054","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460642040.961302519","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460642040.961354256","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460642040.961749315","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460642040.965524673","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460642040.965682507","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460642040.969331026","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460642040.969434261","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460642040.969457150","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642040.969911337","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
{"timestamp":"1460642041.974047899","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642041.998129606","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
{"timestamp":"1460642043.006001472","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642043.043555498","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
{"timestamp":"1460642044.100560427","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642044.124750376","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
{"timestamp":"1460642045.139105082","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642045.141265631","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
{"timestamp":"1460642046.194423914","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642046.238136053","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z2-0'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.2.54 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 13:54:06 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: Ignoring previous leave in snapshot
2016/04/14 13:54:06 [INFO] serf: EventMemberJoin: consul-z2-0.dc1 10.244.2.54
2016/04/14 13:54:06 [INFO] serf: Attempting re-join to previously known node: cc-bridge-z1-0: 10.244.16.14:8301
2016/04/14 13:54:06 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:06 [WARN] serf: Failed to re-join any previously known node
2016/04/14 13:54:06 [INFO] consul: adding server consul-z2-0.dc1 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:06 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:06 [INFO] agent: Joining cluster...
2016/04/14 13:54:06 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 13:54:06 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 13:54:07 [INFO] serf: Re-joined to previously known node: cc-bridge-z1-0: 10.244.16.14:8301
2016/04/14 13:54:07 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 13:54:07 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 13:54:07 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
{"timestamp":"1460642047.607166290","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460642048.561716557","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.2.54","10.244.20.2","10.244.16.2","10.244.18.6","10.244.0.58","10.244.0.54","10.244.0.22","10.244.0.34","10.244.18.18","10.244.18.14","10.244.20.6","10.244.16.6","10.244.16.14","10.244.16.18","10.244.2.134","10.244.18.22","10.244.16.10","10.244.18.10","10.244.0.138","10.244.2.130","10.244.2.22","10.244.2.138","10.244.0.154","10.244.16.22","10.244.0.134","10.244.18.2","10.244.0.130"],"wan":false}}
{"timestamp":"1460642048.643011570","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460642048.661832809","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 13:54:09 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:09 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:10 [INFO] agent.rpc: Accepted client: 127.0.0.1:39317
{"timestamp":"1460642051.020869732","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}} 2016/04/14 13:54:11 [INFO] agent: (LAN) joined: 3 Err: <nil>
2016/04/14 13:54:11 [INFO] agent: Join completed. Synced with 3 initial agents
{"timestamp":"1460642051.247671604","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
2016/04/14 13:54:12 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:12 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
{"timestamp":"1460642052.985519409","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.16.18","10.244.2.134","10.244.18.22","10.244.16.10","10.244.18.10","10.244.0.138","10.244.2.130","10.244.2.22","10.244.2.138","10.244.0.154","10.244.16.22","10.244.0.130","10.244.0.134","10.244.18.2","10.244.2.54","10.244.20.2","10.244.16.2","10.244.18.6","10.244.0.58","10.244.0.54","10.244.18.18","10.244.18.14","10.244.0.22","10.244.0.34","10.244.20.6","10.244.16.6","10.244.16.14"],"wan":false}}
{"timestamp":"1460642053.124922037","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":3,"expected_members_count":3,"is_last_node":true}}
{"timestamp":"1460642053.220628500","source":"confab","message":"confab.controller.configure-server.verify-synced","log_level":1,"data":{}}
{"timestamp":"1460642053.220708847","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642053.291633844","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642053.291810036","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:13 [INFO] memberlist: Suspect cell-z2-0 has failed, no acks received{"timestamp":"1460642054.344840050","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642054.449050426","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642054.449122190","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642055.450047493","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642055.801300287","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642055.802039146","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:16 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:16 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
{"timestamp":"1460642057.374919415","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:18 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 13:54:19 [INFO] memberlist: Marking consul-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:19 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 13:54:19 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 13:54:20 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 13:54:20 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 13:54:21 [INFO] memberlist: Suspect api-worker-z2-0 has failed, no acks received
{"timestamp":"1460642061.921586990","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642061.922854185","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642062.936272860","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642064.905606985","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642064.905680656","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:25 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:25 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
{"timestamp":"1460642065.917679548","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642066.724720716","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642066.724793673","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642067.726404190","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:28 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 13:54:29 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state{"timestamp":"1460642069.319727898","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642069.320193529","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:30 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
{"timestamp":"1460642070.328188181","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642070.747623205","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642070.747671604","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:30 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
2016/04/14 13:54:31 [INFO] memberlist: Marking blobstore-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:31 [INFO] serf: EventMemberFailed: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:31 [WARN] memberlist: Refuting a suspect message (from: consul-z2-0)
2016/04/14 13:54:31 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 13:54:31 [INFO] memberlist: Marking consul-z1-1 as failed, suspect timeout reached
2016/04/14 13:54:31 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 13:54:31 [INFO] memberlist: Marking api-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:31 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 13:54:31 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:31 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:31 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 13:54:31 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 13:54:31 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
{"timestamp":"1460642072.203332424","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:32 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: read tcp 10.244.0.58:8300: connection reset by peer
2016/04/14 13:54:32 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
2016/04/14 13:54:34 [WARN] raft: Heartbeat timeout reached, starting election
{"timestamp":"1460642074.356869698","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642074.357550859","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:34 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state 2016/04/14 13:54:34 [INFO] memberlist: Suspect cc-bridge-z2-0 has failed, no acks received
{"timestamp":"1460642075.368760109","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642075.945089102","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642075.945174694","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642077.041368484","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:37 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received
2016/04/14 13:54:39 [INFO] raft: Duplicate RequestVote for same term: 289
{"timestamp":"1460642079.511439800","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642079.553503513","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642080.629165173","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:42 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 13:54:43 [INFO] memberlist: Marking uaa-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:43 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 13:54:43 [INFO] memberlist: Marking cell-z3-0 as failed, suspect timeout reached
2016/04/14 13:54:43 [INFO] serf: EventMemberFailed: cell-z3-0 10.244.20.6
2016/04/14 13:54:43 [INFO] memberlist: Marking consul-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:43 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 13:54:43 [INFO] memberlist: Marking uaa-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:43 [INFO] serf: EventMemberFailed: uaa-z2-0 10.244.2.130
2016/04/14 13:54:43 [INFO] memberlist: Marking brain-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:43 [INFO] serf: EventMemberFailed: brain-z1-0 10.244.16.6
2016/04/14 13:54:44 [INFO] memberlist: Suspect uaa-z2-0 has failed, no acks received
2016/04/14 13:54:44 [INFO] memberlist: Marking router-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:44 [INFO] serf: EventMemberFailed: router-z1-0 10.244.0.22
2016/04/14 13:54:44 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 13:54:44 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 13:54:44 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state 2016/04/14 13:54:43 [INFO] memberlist: Marking route-emitter-z1-0 as failed, suspect timeout reached
{"timestamp":"1460642084.707511663","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642084.707628012","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:45 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
{"timestamp":"1460642085.829211473","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:43 [INFO] memberlist: Marking database-z2-0 as failed, suspect timeout reached
{"timestamp":"1460642086.553528786","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642086.553610086","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:46 [INFO] serf: EventMemberFailed: route-emitter-z1-0 10.244.16.18
2016/04/14 13:54:47 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 13:54:47 [INFO] memberlist: Suspect access-z2-0 has failed, no acks received
{"timestamp":"1460642087.647263765","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:47 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: EOF
{"timestamp":"1460642087.977870703","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642087.990929127","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:48 [ERR] agent: failed to sync remote state: No cluster leader
{"timestamp":"1460642089.253715038","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642089.473037004","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642089.482428789","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
{"timestamp":"1460642090.488684654","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642090.623487711","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642090.624460220","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:51 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 13:54:51 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 13:54:51 [WARN] memberlist: Refuting a suspect message (from: consul-z2-0)
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 13:54:51 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 13:54:51 [INFO] memberlist: Marking api-z1-0 as failed, suspect timeout reached
2016/04/14 13:54:51 [INFO] serf: EventMemberFailed: api-z1-0 10.244.0.138
2016/04/14 13:54:51 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
{"timestamp":"1460642091.627570391","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642091.739167690","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642091.739213228","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:51 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: dial tcp 10.244.0.58:8300: connection refused 2016/04/14 13:54:51 [INFO] serf: Received install-key query
2016/04/14 13:54:51 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.54:8300: EOF
{"timestamp":"1460642092.741711617","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
2016/04/14 13:54:53 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received{"timestamp":"1460642093.103164196","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642093.103218079","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:53 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 13:54:53 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
{"timestamp":"1460642094.145977020","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642094.870754480","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642094.870808601","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:54 [WARN] raft: Election timeout reached, restarting election
2016/04/14 13:54:54 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 13:54:55 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.58:8300: dial tcp 10.244.0.58:8300: connection refused
2016/04/14 13:54:55 [ERR] raft: Failed to make RequestVote RPC to 10.244.0.54:8300: EOF
2016/04/14 13:54:55 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 13:54:55 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 13:54:55 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 13:54:55 [INFO] memberlist: Suspect route-emitter-z2-0 has failed, no acks received
{"timestamp":"1460642095.876189947","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460642096.126706839","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"2275"}}
{"timestamp":"1460642096.126778841","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 13:54:56 [INFO] memberlist: Suspect access-z1-0 has failed, no acks received
{"timestamp":"1460642097.204947948","source":"confab","message":"confab.controller.configure-server.verify-synced.failed","log_level":2,"data":{"error":"timeout exceeded"}}
2016/04/14 13:54:57 [INFO] agent.rpc: Accepted client: 127.0.0.1:40780
{"timestamp":"1460642097.798163176","source":"confab","message":"confab.controller.stop-agent.leave","log_level":1,"data":{}}
2016/04/14 13:54:57 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
{"timestamp":"1460642097.844992161","source":"confab","message":"confab.agent-client.leave.leave.request","log_level":1,"data":{}}
2016/04/14 13:54:57 [INFO] agent.rpc: Graceful leave triggered
2016/04/14 13:54:57 [INFO] consul: server starting leave
2016/04/14 13:54:57 [INFO] serf: EventMemberLeave: consul-z2-0.dc1 10.244.2.54
2016/04/14 13:54:58 [INFO] consul: removing server consul-z2-0.dc1 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 13:54:59 [INFO] serf: EventMemberLeave: consul-z2-0 10.244.2.54 2016/04/14 13:54:59 [INFO] memberlist: Suspect router-z2-0 has failed, no acks received
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:54:59] error : 'consul_agent' failed to start
[UTC Apr 14 13:54:59] info : 'consul_agent' start action done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment