Skip to content

Instantly share code, notes, and snippets.

@bgandon

bgandon/consul_z1-0

Created April 14, 2016 14:48
Show Gist options
  • Save bgandon/cab53c22da66b24beff46389ba7f0bdc to your computer and use it in GitHub Desktop.
Save bgandon/cab53c22da66b24beff46389ba7f0bdc to your computer and use it in GitHub Desktop.
Download ZIP
Flapping membership in consul cluster of cf v231
Raw
consul_z1-0
# monit start consul_agent; tail -F /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log /var/vcap/monit/monit.log
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:55:08] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:55:42] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:55:42] info : monit daemon at 101 awakened
[UTC Apr 14 13:56:08] error : 'consul_agent' failed to start
[UTC Apr 14 13:56:08] info : Awakened by User defined signal 1
[UTC Apr 14 13:56:08] info : 'consul_agent' stop action done
[UTC Apr 14 14:35:25] info : start service 'consul_agent' on user request
[UTC Apr 14 14:35:25] info : monit daemon at 101 awakened
[UTC Apr 14 14:35:25] info : Awakened by User defined signal 1
[UTC Apr 14 14:35:25] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460644526.329886675","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460644526.330974817","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z1-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.0.54","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460644526.331701279","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460644526.331747293","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460644526.331767559","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460644526.331787109","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460644526.331820965","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460644526.331898212","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460644526.335610628","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460644526.335705757","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460644526.335749865","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644526.337535858","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
{"timestamp":"1460644527.345112801","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644527.346751928","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z1-0'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.0.54 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 14:35:27 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 14:35:27 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:35:27 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:35:27 [INFO] serf: EventMemberJoin: consul-z1-0.dc1 10.244.0.54
2016/04/14 14:35:27 [INFO] consul: adding server consul-z1-0.dc1 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:35:27 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:35:27 [INFO] agent: Joining cluster...
2016/04/14 14:35:27 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 14:35:27 [INFO] agent: (LAN) joined: 1 Err: <nil>
2016/04/14 14:35:27 [INFO] agent: Join completed. Synced with 1 initial agents
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
{"timestamp":"1460644528.348351002","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644528.354441166","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.0.54","10.244.0.58"],"wan":false}}
{"timestamp":"1460644528.355277538","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460644528.356315851","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 14:35:28 [INFO] agent.rpc: Accepted client: 127.0.0.1:34736
{"timestamp":"1460644528.483709335","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}}
{"timestamp":"1460644528.484447479","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644528.570201635","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.0.58","10.244.0.54"],"wan":false}}
{"timestamp":"1460644528.570808649","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":2,"expected_members_count":3,"is_last_node":false}}
{"timestamp":"1460644528.580743313","source":"confab","message":"confab.controller.configure-server.set-keys","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644528.613347530","source":"confab","message":"confab.agent-client.set-keys.list-keys.request","log_level":1,"data":{}}
2016/04/14 14:35:28 [INFO] serf: Received list-keys query
2016/04/14 14:35:28 [INFO] serf: Received list-keys query
{"timestamp":"1460644528.839042664","source":"confab","message":"confab.agent-client.set-keys.list-keys.response","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644528.839392185","source":"confab","message":"confab.agent-client.set-keys.install-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:28 [INFO] serf: Received install-key query
2016/04/14 14:35:28 [INFO] serf: Received install-key query
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] consul: Attempting bootstrap with nodes: [10.244.2.54:8300 10.244.0.54:8300 10.244.0.58:8300]
{"timestamp":"1460644529.059833527","source":"confab","message":"confab.agent-client.set-keys.install-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644529.060732365","source":"confab","message":"confab.agent-client.set-keys.use-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:29 [INFO] serf: Received use-key query
2016/04/14 14:35:29 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:35:29 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:35:29 [INFO] serf: Received use-key query
{"timestamp":"1460644529.394692421","source":"confab","message":"confab.agent-client.set-keys.use-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644529.395226479","source":"confab","message":"confab.agent-client.set-keys.success","log_level":1,"data":{}}
{"timestamp":"1460644529.395379305","source":"confab","message":"confab.agent-runner.run.write-pidfile","log_level":1,"data":{"path":"/var/vcap/sys/run/consul_agent/consul_agent.pid","pid":4802}}
{"timestamp":"1460644529.395574808","source":"confab","message":"confab.controller.configure-server.success","log_level":1,"data":{}}
2016/04/14 14:35:29 [INFO] raft: Election won. Tally: 2
2016/04/14 14:35:29 [INFO] raft: Node at 10.244.0.54:8300 [Leader] entering Leader state
2016/04/14 14:35:29 [INFO] consul: cluster leadership acquired
2016/04/14 14:35:29 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:35:29 [INFO] raft: pipelining replication to peer 10.244.2.54:8300
2016/04/14 14:35:29 [INFO] consul: member 'consul-z1-0' joined, marking health alive
2016/04/14 14:35:29 [INFO] consul: member 'consul-z1-1' joined, marking health alive
2016/04/14 14:35:29 [INFO] raft: pipelining replication to peer 10.244.0.58:8300
2016/04/14 14:35:29 [INFO] consul: member 'consul-z2-0' joined, marking health alive
2016/04/14 14:35:30 [INFO] serf: Received list-keys query
2016/04/14 14:35:30 [INFO] serf: Received install-key query
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 14:35:29] info : 'consul_agent' started
[UTC Apr 14 14:35:30] info : 'consul_agent' start action done
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
2016/04/14 14:35:31 [INFO] serf: Received use-key query
2016/04/14 14:35:31 [INFO] serf: Received list-keys query
2016/04/14 14:35:31 [INFO] serf: Received install-key query
2016/04/14 14:35:31 [INFO] serf: Received use-key query
2016/04/14 14:35:32 [INFO] agent: Synced service 'consul'
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 14:35:40] info : 'consul_agent' process is running with pid 4802
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 14:36:05 [INFO] consul: member 'cell-z3-0' joined, marking health alive
2016/04/14 14:36:05 [INFO] consul: member 'api-worker-z1-0' joined, marking health alive
2016/04/14 14:36:06 [INFO] consul: member 'cell-z2-0' joined, marking health alive
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:36:06 [INFO] consul: member 'router-z1-0' joined, marking health alive
2016/04/14 14:36:06 [INFO] consul: member 'brain-z1-0' joined, marking health alive
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:36:06 [INFO] consul: member 'cell-z1-0' joined, marking health alive
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:36:07 [INFO] consul: member 'api-z1-0' joined, marking health alive
2016/04/14 14:36:07 [WARN] raft: Failed to contact 10.244.0.58:8300 in 542.94454ms
2016/04/14 14:36:08 [INFO] consul: member 'api-z2-0' joined, marking health alive
2016/04/14 14:36:08 [INFO] consul: member 'blobstore-z1-0' joined, marking health alive
2016/04/14 14:36:09 [INFO] consul: member 'access-z2-0' joined, marking health alive
2016/04/14 14:36:11 [INFO] consul: member 'route-emitter-z1-0' joined, marking health alive
2016/04/14 14:36:12 [INFO] consul: member 'brain-z2-0' joined, marking health alive
2016/04/14 14:36:13 [INFO] consul: member 'uaa-z1-0' joined, marking health alive
2016/04/14 14:36:13 [INFO] memberlist: Suspect brain-z2-0 has failed, no acks received
2016/04/14 14:36:14 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
2016/04/14 14:36:14 [WARN] memberlist: Refuting a suspect message (from: cell-z3-0)
2016/04/14 14:36:15 [WARN] raft: Failed to contact 10.244.2.54:8300 in 1.17915167s
2016/04/14 14:36:17 [INFO] consul: member 'uaa-z2-0' joined, marking health alive
2016/04/14 14:36:17 [WARN] raft: Failed to contact 10.244.0.58:8300 in 868.832176ms
2016/04/14 14:36:19 [INFO] consul: member 'router-z2-0' joined, marking health alive
2016/04/14 14:36:19 [INFO] memberlist: Suspect api-worker-z1-0 has failed, no acks received
2016/04/14 14:36:19 [WARN] memberlist: Refuting a suspect message (from: cell-z2-0) 2016/04/14 14:36:20 [WARN] raft: Failed to contact 10.244.2.54:8300 in 890.669187ms
2016/04/14 14:36:20 [INFO] memberlist: Suspect cell-z3-0 has failed, no acks received
2016/04/14 14:36:21 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 14:36:22 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:22 [INFO] memberlist: Suspect consul-z2-0 has failed, no acks received
2016/04/14 14:36:22 [WARN] raft: Failed to contact 10.244.0.58:8300 in 1.144892398s
2016/04/14 14:36:22 [INFO] consul: member 'database-z3-0' joined, marking health alive
2016/04/14 14:36:22 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 14:36:22 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:23 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:24 [INFO] memberlist: Suspect route-emitter-z1-0 has failed, no acks received
2016/04/14 14:36:24 [WARN] raft: Failed to contact 10.244.2.54:8300 in 517.46228ms
2016/04/14 14:36:26 [INFO] memberlist: Suspect cc-bridge-z1-0 has failed, no acks received
2016/04/14 14:36:27 [WARN] raft: Failed to contact 10.244.2.54:8300 in 961.313359ms
2016/04/14 14:36:27 [WARN] raft: Failed to contact 10.244.0.58:8300 in 1.228776228s
2016/04/14 14:36:27 [WARN] raft: Failed to contact quorum of nodes, stepping down
2016/04/14 14:36:27 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 14:36:28 [INFO] consul: cluster leadership lost 2016/04/14 14:36:28 [INFO] raft: aborting pipeline replication to peer 10.244.0.58:8300
2016/04/14 14:36:28 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 14:36:28 [WARN] memberlist: Refuting a suspect message (from: cc-bridge-z1-0)
2016/04/14 14:36:28 [INFO] raft: aborting pipeline replication to peer 10.244.2.54:8300
2016/04/14 14:36:28 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:29 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:29 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:32 [INFO] raft: Duplicate RequestVote for same term: 2
2016/04/14 14:36:32 [ERR] consul: failed to wait for barrier: node is not the leader
2016/04/14 14:36:32 [ERR] consul.session: Invalidation failed: node is not the leader
2016/04/14 14:36:32 [ERR] consul.session: Invalidation failed: node is not the leader
2016/04/14 14:36:33 [INFO] memberlist: Marking database-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:33 [INFO] serf: EventMemberFailed: database-z1-0 10.244.16.2
2016/04/14 14:36:33 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:33 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:33 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:36:34 [INFO] memberlist: Suspect cell-z2-0 has failed, no acks received
2016/04/14 14:36:33 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:34 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 14:36:33 [INFO] memberlist: Marking api-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:34 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 14:36:34 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:34 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:36:36 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:39 [INFO] memberlist: Marking uaa-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: uaa-z2-0 10.244.2.130
2016/04/14 14:36:39 [INFO] memberlist: Marking cell-z3-0 as failed, suspect timeout reached
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: cell-z3-0 10.244.20.6
2016/04/14 14:36:40 [INFO] memberlist: Marking database-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:40 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 14:36:40 [INFO] memberlist: Marking router-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:40 [INFO] serf: EventMemberFailed: router-z1-0 10.244.0.22
2016/04/14 14:36:40 [INFO] memberlist: Marking api-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:40 [INFO] serf: EventMemberFailed: api-z1-0 10.244.0.138
2016/04/14 14:36:40 [INFO] memberlist: Marking cc-bridge-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:40 [INFO] serf: EventMemberFailed: cc-bridge-z1-0 10.244.16.14 2016/04/14 14:36:40 [INFO] memberlist: Suspect uaa-z1-0 has failed, no acks received
2016/04/14 14:36:42 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 14:36:43 [INFO] memberlist: Suspect api-z2-0 has failed, no acks received
2016/04/14 14:36:44 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:44 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:44 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:44 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:44 [INFO] memberlist: Suspect consul-z1-1 has failed, no acks received
2016/04/14 14:36:48 [WARN] memberlist: Refuting a suspect message (from: ha-proxy-z1-0)
2016/04/14 14:36:48 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:48 [INFO] memberlist: Suspect access-z1-0 has failed, no acks received
2016/04/14 14:36:48 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:36:49 [INFO] memberlist: Suspect router-z2-0 has failed, no acks received
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:49 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state 2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:36:50 [INFO] memberlist: Suspect database-z3-0 has failed, no acks received
2016/04/14 14:36:50 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:36:51 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:52 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 14:36:53 [INFO] memberlist: Marking uaa-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:53 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 14:36:53 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:53 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:53 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 14:36:54 [WARN] memberlist: Refuting a suspect message (from: consul-z1-0)
2016/04/14 14:36:54 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:54 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received
2016/04/14 14:36:54 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:36:55 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 14:36:56 [INFO] memberlist: Suspect api-worker-z2-0 has failed, no acks received
2016/04/14 14:36:56 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:56 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 14:36:58 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 14:36:58 [INFO] serf: EventMemberFailed: brain-z2-0 10.244.18.6
2016/04/14 14:36:58 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:58 [INFO] memberlist: Suspect router-z2-0 has failed, no acks received
2016/04/14 14:36:58 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:58 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:37:00 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:00 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received
2016/04/14 14:37:00 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 14:37:00 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:37:00 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:00 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:37:01 [INFO] raft: Node at 10.244.0.54:8300 [Candidate] entering Candidate state
2016/04/14 14:37:01 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:37:01 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:37:01 [INFO] memberlist: Suspect consul-z2-0 has failed, no acks received 2016/04/14 14:37:01 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:37:02 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 14:37:03 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:37:03 [INFO] serf: EventMemberFailed: consul-z2-0 10.244.2.54
2016/04/14 14:37:03 [INFO] consul: removing server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:37:04 [INFO] memberlist: Suspect cc-bridge-z1-0 has failed, no acks received
2016/04/14 14:37:05 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 14:37:05 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:37:06 [WARN] memberlist: Refuting a suspect message (from: access-z2-0)
2016/04/14 14:37:07 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
2016/04/14 14:37:07 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:37:07 [INFO] memberlist: Marking api-worker-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:07 [INFO] serf: EventMemberFailed: api-worker-z2-0 10.244.2.138
2016/04/14 14:37:07 [INFO] serf: EventMemberFailed: database-z1-0 10.244.16.2
2016/04/14 14:37:08 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:37:08 [INFO] raft: Node at 10.244.0.54:8300 [Follower] entering Follower state
2016/04/14 14:37:08 [INFO] memberlist: Suspect database-z2-0 has failed, no acks received
^
Raw
consul_z1-1
# monit start consul_agent; tail -F /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log /var/vcap/monit/monit.log
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:55:08] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:55:45] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:55:45] info : monit daemon at 102 awakened
[UTC Apr 14 13:56:08] error : 'consul_agent' failed to start
[UTC Apr 14 13:56:08] info : Awakened by User defined signal 1
[UTC Apr 14 13:56:08] info : 'consul_agent' stop action done
[UTC Apr 14 14:35:26] info : start service 'consul_agent' on user request
[UTC Apr 14 14:35:26] info : monit daemon at 102 awakened
[UTC Apr 14 14:35:26] info : Awakened by User defined signal 1
[UTC Apr 14 14:35:26] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460644527.109992266","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460644527.110257864","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z1-1","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.0.58","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460644527.110971212","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460644527.111042261","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460644527.111062050","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460644527.111106634","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460644527.111130714","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460644527.111224890","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460644527.112461090","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460644527.112529516","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460644527.112552643","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644527.113547325","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
{"timestamp":"1460644528.117180824","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644528.119172812","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z1-1'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.0.58 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 14:35:28 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-1.dc1 10.244.0.58
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-1.dc1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:35:28 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:35:28 [INFO] agent: Joining cluster...
2016/04/14 14:35:28 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] agent: (LAN) joined: 2 Err: <nil>
2016/04/14 14:35:28 [INFO] agent: Join completed. Synced with 2 initial agents
2016/04/14 14:35:28 [INFO] serf: Received list-keys query
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] consul: Attempting bootstrap with nodes: [10.244.0.58:8300 10.244.0.54:8300 10.244.2.54:8300]
2016/04/14 14:35:29 [INFO] serf: Received install-key query
{"timestamp":"1460644529.123956442","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644529.159529686","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.0.58","10.244.0.54","10.244.2.54"],"wan":false}}
{"timestamp":"1460644529.160391331","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460644529.160418034","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 14:35:29 [INFO] agent.rpc: Accepted client: 127.0.0.1:34764
{"timestamp":"1460644529.179212809","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}}
{"timestamp":"1460644529.179495573","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644529.267592192","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.0.58","10.244.0.54","10.244.2.54"],"wan":false}}
{"timestamp":"1460644529.267711163","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":3,"expected_members_count":3,"is_last_node":true}}
{"timestamp":"1460644529.267731905","source":"confab","message":"confab.controller.configure-server.verify-synced","log_level":1,"data":{}}
{"timestamp":"1460644529.267744541","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460644529.278259993","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"0","last_log_index":"0"}}
{"timestamp":"1460644529.299183846","source":"confab","message":"confab.agent-client.verify-synced.zero-index","log_level":2,"data":{"error":"commit index must not be zero"}}
2016/04/14 14:35:29 [INFO] serf: Received use-key query
2016/04/14 14:35:29 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:35:29 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since we have a leader: 10.244.0.54:8300
{"timestamp":"1460644530.303346872","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460644530.307875395","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"5","last_log_index":"5"}}
{"timestamp":"1460644530.308147430","source":"confab","message":"confab.agent-client.verify-synced.synced","log_level":1,"data":{}}
{"timestamp":"1460644530.308518410","source":"confab","message":"confab.controller.configure-server.set-keys","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644530.308992863","source":"confab","message":"confab.agent-client.set-keys.list-keys.request","log_level":1,"data":{}}
2016/04/14 14:35:30 [INFO] serf: Received list-keys query
2016/04/14 14:35:30 [INFO] serf: Received list-keys query
{"timestamp":"1460644530.617703199","source":"confab","message":"confab.agent-client.set-keys.list-keys.response","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644530.618467569","source":"confab","message":"confab.agent-client.set-keys.install-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:30 [INFO] serf: Received install-key query
2016/04/14 14:35:30 [INFO] serf: Received install-key query
{"timestamp":"1460644530.892082930","source":"confab","message":"confab.agent-client.set-keys.install-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644530.893675089","source":"confab","message":"confab.agent-client.set-keys.use-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:30 [INFO] serf: Received use-key query
2016/04/14 14:35:31 [INFO] serf: Received use-key query
2016/04/14 14:35:31 [INFO] serf: Received list-keys query
{"timestamp":"1460644531.390354872","source":"confab","message":"confab.agent-client.set-keys.use-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644531.390964031","source":"confab","message":"confab.agent-client.set-keys.success","log_level":1,"data":{}}
{"timestamp":"1460644531.391013145","source":"confab","message":"confab.agent-runner.run.write-pidfile","log_level":1,"data":{"path":"/var/vcap/sys/run/consul_agent/consul_agent.pid","pid":4625}}
{"timestamp":"1460644531.391087055","source":"confab","message":"confab.controller.configure-server.success","log_level":1,"data":{}}
2016/04/14 14:35:31 [INFO] serf: Received install-key query
2016/04/14 14:35:31 [INFO] serf: Received use-key query
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 14:35:31] info : 'consul_agent' started
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
2016/04/14 14:35:31 [INFO] agent: Synced service 'consul'
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 14:35:31] info : 'consul_agent' start action done
[UTC Apr 14 14:35:42] info : 'consul_agent' process is running with pid 4625
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 14:36:06 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 14:36:07 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 14:36:14 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received
2016/04/14 14:36:16 [INFO] memberlist: Suspect cell-z2-0 has failed, no acks received
2016/04/14 14:36:17 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:17 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:17 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:18 [INFO] memberlist: Suspect route-emitter-z1-0 has failed, no acks received
2016/04/14 14:36:20 [INFO] memberlist: Suspect api-worker-z2-0 has failed, no acks received
2016/04/14 14:36:22 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received
2016/04/14 14:36:23 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
2016/04/14 14:36:24 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received
2016/04/14 14:36:26 [INFO] memberlist: Suspect cell-z3-0 has failed, no acks received
2016/04/14 14:36:28 [WARN] memberlist: Refuting a dead message (from: consul-z2-0)
2016/04/14 14:36:28 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:28 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:30 [INFO] memberlist: Suspect database-z2-0 has failed, no acks received
2016/04/14 14:36:30 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:30 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:30 [INFO] serf: EventMemberFailed: database-z1-0 10.244.16.2
2016/04/14 14:36:31 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 14:36:32 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:34 [INFO] raft: Duplicate RequestVote for same term: 3
2016/04/14 14:36:36 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:36 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:36 [INFO] memberlist: Suspect api-z2-0 has failed, no acks received
2016/04/14 14:36:37 [WARN] memberlist: Refuting a suspect message (from: ha-proxy-z1-0)
2016/04/14 14:36:37 [INFO] memberlist: Marking database-z3-0 as failed, suspect timeout reached
2016/04/14 14:36:37 [INFO] serf: EventMemberFailed: database-z3-0 10.244.20.2
2016/04/14 14:36:38 [INFO] memberlist: Marking access-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:38 [INFO] serf: EventMemberFailed: access-z2-0 10.244.18.22 2016/04/14 14:36:38 [INFO] memberlist: Marking router-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:38 [INFO] memberlist: Marking cell-z3-0 as failed, suspect timeout reached
2016/04/14 14:36:38 [WARN] raft: Election timeout reached, restarting election 2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: cell-z3-0 10.244.20.6
2016/04/14 14:36:39 [INFO] memberlist: Marking router-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: router-z2-0 10.244.2.22
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: router-z1-0 10.244.0.22
2016/04/14 14:36:39 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 14:36:39 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 14:36:39 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:36:39 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:39 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 14:36:40 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:42 [INFO] memberlist: Suspect cell-z1-0 has failed, no acks received
2016/04/14 14:36:42 [INFO] memberlist: Marking consul-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:42 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 14:36:42 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1) 2016/04/14 14:36:43 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received
2016/04/14 14:36:46 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:46 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:46 [INFO] memberlist: Suspect ha-proxy-z1-0 has failed, no acks received
2016/04/14 14:36:48 [INFO] memberlist: Suspect router-z2-0 has failed, no acks received
2016/04/14 14:36:50 [INFO] raft: Election won. Tally: 2
2016/04/14 14:36:50 [INFO] raft: Node at 10.244.0.58:8300 [Leader] entering Leader state
2016/04/14 14:36:50 [INFO] consul: cluster leadership acquired
2016/04/14 14:36:50 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:50 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:50 [INFO] memberlist: Marking uaa-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:50 [INFO] serf: EventMemberFailed: uaa-z2-0 10.244.2.130
2016/04/14 14:36:50 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:50 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 14:36:50 [INFO] memberlist: Marking database-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:50 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 14:36:50 [INFO] memberlist: Marking router-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:50 [INFO] serf: EventMemberFailed: router-z2-0 10.244.2.22
2016/04/14 14:36:51 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:51 [ERR] raft: peer 10.244.2.54:8300 has newer term, stopping replication
2016/04/14 14:36:51 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 14:36:54 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22 2016/04/14 14:36:54 [ERR] consul.catalog: Register failed: node is not the leader
2016/04/14 14:36:54 [INFO] memberlist: Suspect consul-z2-0 has failed, no acks received
2016/04/14 14:36:54 [INFO] consul: cluster leadership lost
2016/04/14 14:36:54 [ERR] raft: peer 10.244.0.54:8300 has newer term, stopping replication
2016/04/14 14:36:54 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:36:54 [ERR] consul: failed to wait for barrier: leadership lost while committing log
2016/04/14 14:36:55 [WARN] memberlist: Refuting a suspect message (from: consul-z1-1)
2016/04/14 14:36:55 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:56 [WARN] raft: Rejecting vote from 10.244.2.54:8300 since our last term is greater (6, 1)
2016/04/14 14:36:56 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:56 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:57 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 14:36:57 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:57 [INFO] memberlist: Suspect uaa-z1-0 has failed, no acks received
2016/04/14 14:36:58 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:59 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:36:59 [INFO] memberlist: Suspect api-worker-z1-0 has failed, no acks received
2016/04/14 14:37:00 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:01 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:01 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received
2016/04/14 14:37:01 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:37:01 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
2016/04/14 14:37:02 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:37:02 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:37:02 [INFO] memberlist: Suspect cell-z3-0 has failed, no acks received
2016/04/14 14:37:04 [INFO] memberlist: Suspect database-z3-0 has failed, no acks received
2016/04/14 14:37:04 [INFO] memberlist: Marking consul-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:04 [INFO] serf: EventMemberFailed: consul-z2-0 10.244.2.54
2016/04/14 14:37:04 [INFO] consul: removing server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:37:04 [INFO] raft: Election won. Tally: 2
2016/04/14 14:37:05 [INFO] raft: Node at 10.244.0.58:8300 [Leader] entering Leader state
2016/04/14 14:37:05 [INFO] memberlist: Marking blobstore-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:05 [INFO] serf: EventMemberFailed: blobstore-z1-0 10.244.0.130
2016/04/14 14:37:05 [INFO] memberlist: Marking brain-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:05 [INFO] serf: EventMemberFailed: brain-z2-0 10.244.18.6
2016/04/14 14:37:05 [INFO] memberlist: Marking api-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:05 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 14:37:05 [INFO] memberlist: Marking cell-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:05 [INFO] serf: EventMemberFailed: cell-z1-0 10.244.16.10
2016/04/14 14:37:05 [INFO] consul: cluster leadership acquired
2016/04/14 14:37:05 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since we have a leader: 10.244.0.58:8300
2016/04/14 14:37:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:37:05 [WARN] raft: AppendEntries to 10.244.2.54:8300 rejected, sending older logs (next: 101)
2016/04/14 14:37:05 [INFO] memberlist: Marking api-worker-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:05 [INFO] serf: EventMemberFailed: api-worker-z2-0 10.244.2.138
2016/04/14 14:37:05 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 14:37:05 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:37:06 [INFO] raft: pipelining replication to peer 10.244.2.54:8300
2016/04/14 14:37:06 [WARN] raft: Failed to contact 10.244.0.54:8300 in 933.171004ms
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:37:06 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:37:06 [INFO] serf: EventMemberFailed: api-z1-0 10.244.0.138
2016/04/14 14:37:07 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 14:37:07 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received
2016/04/14 14:37:07 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:37:07 [INFO] consul: member 'api-z1-0' failed, marking health critical
2016/04/14 14:37:08 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 14:37:11 [WARN] raft: AppendEntries to 10.244.0.54:8300 rejected, sending older logs (next: 101)
2016/04/14 14:37:11 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:37:11 [INFO] memberlist: Suspect consul-z1-0 has failed, no acks received
2016/04/14 14:37:11 [INFO] memberlist: Marking router-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:11 [INFO] serf: EventMemberFailed: router-z2-0 10.244.2.22
2016/04/14 14:37:12 [WARN] raft: Failed to contact 10.244.0.54:8300 in 780.587251ms
2016/04/14 14:37:12 [WARN] raft: Failed to contact 10.244.2.54:8300 in 719.387333ms
2016/04/14 14:37:12 [WARN] raft: Failed to contact quorum of nodes, stepping down
2016/04/14 14:37:12 [INFO] raft: Node at 10.244.0.58:8300 [Follower] entering Follower state
2016/04/14 14:37:12 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul.session: Apply failed: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul: failed to reconcile member: {api-z1-0 10.244.0.138 8301 map[vsn_max:2 build:0.5.2:9a9cc934 role:node dc:dc1 vsn:2 vsn_min:1] failed 1 2 2 2 4 4}: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul: failed to reconcile: leadership lost while committing log
2016/04/14 14:37:12 [ERR] consul.catalog: Register failed: leadership lost while committing log
2016/04/14 14:37:12 [INFO] consul: cluster leadership lost
2016/04/14 14:37:12 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:37:12 [WARN] memberlist: Refuting a suspect message (from: consul-z1-1)
2016/04/14 14:37:12 [INFO] raft: aborting pipeline replication to peer 10.244.2.54:8300
2016/04/14 14:37:12 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 14:37:13 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since our last term is greater (11, 1)
2016/04/14 14:37:14 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:37:14 [INFO] raft: Node at 10.244.0.58:8300 [Candidate] entering Candidate state
Raw
consul_z2-0
# monit start consul_agent; tail -F /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log /var/vcap/monit/monit.log
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 13:55:11] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Apr 14 13:56:07] info : stop service 'consul_agent' on user request
[UTC Apr 14 13:56:07] info : monit daemon at 103 awakened
[UTC Apr 14 13:56:11] error : 'consul_agent' failed to start
[UTC Apr 14 13:56:11] info : Awakened by User defined signal 1
[UTC Apr 14 13:56:11] info : 'consul_agent' stop action done
[UTC Apr 14 14:35:27] info : start service 'consul_agent' on user request
[UTC Apr 14 14:35:27] info : monit daemon at 103 awakened
[UTC Apr 14 14:35:27] info : Awakened by User defined signal 1
[UTC Apr 14 14:35:27] info : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
{"timestamp":"1460644527.842602015","source":"confab","message":"confab.controller.write-consul-config.generate-configuration","log_level":1,"data":{}}
{"timestamp":"1460644527.843306780","source":"confab","message":"confab.controller.write-consul-config.write-configuration","log_level":1,"data":{"config":{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"info","node_name":"consul-z2-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["10.244.0.54","10.244.0.58","10.244.2.54"],"bind_addr":"10.244.2.54","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Atzo3VBv+YVDzQAzlQRPRA==","bootstrap_expect":3}}}
{"timestamp":"1460644527.843888283","source":"confab","message":"confab.controller.write-consul-config.success","log_level":1,"data":{}}
{"timestamp":"1460644527.844021082","source":"confab","message":"confab.controller.write-service-definitions.generate-definitions","log_level":1,"data":{}}
{"timestamp":"1460644527.844064236","source":"confab","message":"confab.controller.write-service-definitions.write","log_level":1,"data":{}}
{"timestamp":"1460644527.844076395","source":"confab","message":"confab.controller.write-service-definitions.success","log_level":1,"data":{}}
{"timestamp":"1460644527.844086885","source":"confab","message":"confab.controller.boot-agent.run","log_level":1,"data":{}}
{"timestamp":"1460644527.844272852","source":"confab","message":"confab.agent-runner.run.start","log_level":1,"data":{"args":["agent","-config-dir=/var/vcap/jobs/consul_agent/config","-recursor=8.8.8.8","-recursor=10.0.2.3"],"cmd":"/var/vcap/packages/consul/bin/consul"}}
{"timestamp":"1460644527.845355272","source":"confab","message":"confab.agent-runner.run.success","log_level":1,"data":{}}
{"timestamp":"1460644527.846406460","source":"confab","message":"confab.controller.boot-agent.verify-joined","log_level":1,"data":{}}
{"timestamp":"1460644527.850152016","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644527.853431702","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting raft data migration...
==> Starting Consul agent...
{"timestamp":"1460644528.854806185","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644528.858771801","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul-z2-0'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
Cluster Addr: 10.244.2.54 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>
==> Log data will now stream in as it occurs:
2016/04/14 14:35:28 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z2-0 10.244.2.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z2-0 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z2-0.dc1 10.244.2.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z2-0.dc1 (Addr: 10.244.2.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] agent: Joining cluster...
2016/04/14 14:35:28 [INFO] agent: (LAN) joining: [10.244.0.54 10.244.0.58 10.244.2.54]
2016/04/14 14:35:28 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:35:28 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:35:28 [INFO] consul: Attempting bootstrap with nodes: [10.244.2.54:8300 10.244.0.58:8300 10.244.0.54:8300]
2016/04/14 14:35:28 [INFO] agent: (LAN) joined: 3 Err: <nil>
2016/04/14 14:35:28 [INFO] agent: Join completed. Synced with 3 initial agents
2016/04/14 14:35:29 [INFO] serf: Received install-key query
2016/04/14 14:35:29 [INFO] serf: Received list-keys query
2016/04/14 14:35:29 [INFO] serf: Received use-key query
2016/04/14 14:35:29 [INFO] consul: New leader elected: consul-z1-0
{"timestamp":"1460644529.908136129","source":"confab","message":"confab.agent-client.verify-joined.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644529.944438219","source":"confab","message":"confab.agent-client.verify-joined.members.response","log_level":1,"data":{"members":["10.244.2.54","10.244.0.58","10.244.0.54"],"wan":false}}
{"timestamp":"1460644529.944978476","source":"confab","message":"confab.agent-client.verify-joined.members.joined","log_level":1,"data":{}}
{"timestamp":"1460644529.945003986","source":"confab","message":"confab.controller.boot-agent.success","log_level":1,"data":{}}
2016/04/14 14:35:29 [INFO] agent.rpc: Accepted client: 127.0.0.1:34797
{"timestamp":"1460644529.962204695","source":"confab","message":"confab.controller.configure-server.is-last-node","log_level":1,"data":{}}
{"timestamp":"1460644529.962683916","source":"confab","message":"confab.agent-client.is-last-node.members.request","log_level":1,"data":{"wan":false}}
{"timestamp":"1460644529.978646278","source":"confab","message":"confab.agent-client.is-last-node.members.response","log_level":1,"data":{"members":["10.244.2.54","10.244.0.58","10.244.0.54"],"wan":false}}
{"timestamp":"1460644529.978749514","source":"confab","message":"confab.agent-client.is-last-node.result","log_level":1,"data":{"actual_members_count":3,"expected_members_count":3,"is_last_node":true}}
{"timestamp":"1460644529.978802443","source":"confab","message":"confab.controller.configure-server.verify-synced","log_level":1,"data":{}}
{"timestamp":"1460644529.978883028","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460644529.990943432","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"3","last_log_index":"4"}}
{"timestamp":"1460644529.991359949","source":"confab","message":"confab.agent-client.verify-synced.not-synced","log_level":2,"data":{"error":"log not in sync"}}
2016/04/14 14:35:30 [INFO] serf: Received list-keys query
2016/04/14 14:35:30 [INFO] serf: Received install-key query
{"timestamp":"1460644530.994335413","source":"confab","message":"confab.agent-client.verify-synced.stats.request","log_level":1,"data":{}}
{"timestamp":"1460644531.131287575","source":"confab","message":"confab.agent-client.verify-synced.stats.response","log_level":1,"data":{"commit_index":"5","last_log_index":"5"}}
{"timestamp":"1460644531.131987810","source":"confab","message":"confab.agent-client.verify-synced.synced","log_level":1,"data":{}}
{"timestamp":"1460644531.132004023","source":"confab","message":"confab.controller.configure-server.set-keys","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644531.132009506","source":"confab","message":"confab.agent-client.set-keys.list-keys.request","log_level":1,"data":{}}
2016/04/14 14:35:31 [INFO] serf: Received list-keys query
2016/04/14 14:35:31 [INFO] serf: Received list-keys query
2016/04/14 14:35:31 [INFO] serf: Received use-key query
{"timestamp":"1460644531.415310621","source":"confab","message":"confab.agent-client.set-keys.list-keys.response","log_level":1,"data":{"keys":["Atzo3VBv+YVDzQAzlQRPRA=="]}}
{"timestamp":"1460644531.452200174","source":"confab","message":"confab.agent-client.set-keys.install-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:31 [INFO] serf: Received install-key query
2016/04/14 14:35:31 [INFO] serf: Received install-key query
2016/04/14 14:35:31 [INFO] agent: Synced service 'consul'
{"timestamp":"1460644531.588396072","source":"confab","message":"confab.agent-client.set-keys.install-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644531.588497400","source":"confab","message":"confab.agent-client.set-keys.use-key.request","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
2016/04/14 14:35:31 [INFO] serf: Received use-key query
2016/04/14 14:35:31 [INFO] serf: Received use-key query
{"timestamp":"1460644531.851143122","source":"confab","message":"confab.agent-client.set-keys.use-key.response","log_level":1,"data":{"key":"Atzo3VBv+YVDzQAzlQRPRA=="}}
{"timestamp":"1460644531.869692326","source":"confab","message":"confab.agent-client.set-keys.success","log_level":1,"data":{}}
{"timestamp":"1460644531.869971991","source":"confab","message":"confab.agent-runner.run.write-pidfile","log_level":1,"data":{"path":"/var/vcap/sys/run/consul_agent/consul_agent.pid","pid":4476}}
{"timestamp":"1460644531.874884605","source":"confab","message":"confab.controller.configure-server.success","log_level":1,"data":{}}
==> /var/vcap/monit/monit.log <==
[UTC Apr 14 14:35:32] info : 'consul_agent' started
[UTC Apr 14 14:35:32] info : 'consul_agent' start action done
[UTC Apr 14 14:35:42] info : 'consul_agent' process is running with pid 4476
==> /var/vcap/data/sys/log/consul_agent/consul_agent.stdout.log <==
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: brain-z1-0 10.244.16.6
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: blobstore-z1-0 10.244.0.130
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: access-z2-0 10.244.18.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cell-z1-0 10.244.16.10
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: route-emitter-z1-0 10.244.16.18
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: ha-proxy-z1-0 10.244.0.34
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-z1-0 10.244.0.138
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: api-worker-z2-0 10.244.2.138
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: access-z1-0 10.244.16.22
2016/04/14 14:36:05 [INFO] serf: EventMemberJoin: route-emitter-z2-0 10.244.18.18
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z2-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:36:05 [INFO] consul: New leader elected: consul-z1-0
2016/04/14 14:36:15 [INFO] memberlist: Suspect consul-z1-1 has failed, no acks received
2016/04/14 14:36:19 [INFO] memberlist: Suspect cell-z2-0 has failed, no acks received
2016/04/14 14:36:20 [INFO] memberlist: Suspect cc-bridge-z2-0 has failed, no acks received
2016/04/14 14:36:21 [WARN] memberlist: Refuting a suspect message (from: uaa-z1-0)
2016/04/14 14:36:22 [INFO] memberlist: Suspect database-z1-0 has failed, no acks received
2016/04/14 14:36:23 [INFO] memberlist: Suspect database-z2-0 has failed, no acks received
2016/04/14 14:36:24 [INFO] memberlist: Suspect uaa-z1-0 has failed, no acks received
2016/04/14 14:36:26 [INFO] memberlist: Marking consul-z1-1 as failed, suspect timeout reached
2016/04/14 14:36:26 [INFO] serf: EventMemberLeave: consul-z1-1 10.244.0.58
2016/04/14 14:36:26 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:26 [INFO] memberlist: Suspect api-z2-0 has failed, no acks received
2016/04/14 14:36:26 [WARN] memberlist: Refuting a suspect message (from: cc-bridge-z1-0)
2016/04/14 14:36:27 [INFO] memberlist: Suspect api-worker-z1-0 has failed, no acks received
2016/04/14 14:36:29 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:36:29 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:29 [INFO] memberlist: Marking cell-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:29 [INFO] serf: EventMemberFailed: cell-z2-0 10.244.18.10
2016/04/14 14:36:29 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received
2016/04/14 14:36:30 [INFO] serf: EventMemberFailed: database-z1-0 10.244.16.2
2016/04/14 14:36:30 [INFO] serf: EventMemberJoin: database-z1-0 10.244.16.2
2016/04/14 14:36:30 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:30 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:33 [INFO] memberlist: Marking api-worker-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:33 [INFO] raft: Duplicate RequestVote for same term: 2
2016/04/14 14:36:34 [INFO] raft: Duplicate RequestVote for same term: 2
2016/04/14 14:36:34 [INFO] serf: EventMemberFailed: api-worker-z1-0 10.244.0.154
2016/04/14 14:36:36 [INFO] memberlist: Suspect access-z2-0 has failed, no acks received
2016/04/14 14:36:36 [WARN] memberlist: Refuting a suspect message (from: brain-z1-0) 2016/04/14 14:36:36 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:36 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:38 [INFO] memberlist: Marking router-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:38 [INFO] serf: EventMemberFailed: router-z1-0 10.244.0.22
2016/04/14 14:36:39 [INFO] memberlist: Marking cc-bridge-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:39 [INFO] memberlist: Marking uaa-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:39 [INFO] serf: EventMemberFailed: uaa-z2-0 10.244.2.130
2016/04/14 14:36:38 [INFO] memberlist: Marking cell-z3-0 as failed, suspect timeout reached
2016/04/14 14:36:40 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 14:36:42 [INFO] serf: EventMemberFailed: cell-z3-0 10.244.20.6
2016/04/14 14:36:42 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 14:36:43 [INFO] serf: EventMemberJoin: cc-bridge-z1-0 10.244.16.14
2016/04/14 14:36:43 [INFO] raft: Duplicate RequestVote for same term: 3
2016/04/14 14:36:44 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state
2016/04/14 14:36:44 [INFO] memberlist: Marking cc-bridge-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:44 [INFO] serf: EventMemberFailed: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:44 [INFO] memberlist: Marking database-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:44 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 14:36:46 [INFO] memberlist: Suspect consul-z1-0 has failed, no acks received
2016/04/14 14:36:46 [INFO] memberlist: Marking consul-z1-0 as failed, suspect timeout reached
2016/04/14 14:36:46 [INFO] serf: EventMemberLeave: consul-z1-0 10.244.0.54
2016/04/14 14:36:46 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:36:48 [INFO] memberlist: Suspect ha-proxy-z1-0 has failed, no acks received
2016/04/14 14:36:49 [INFO] memberlist: Suspect brain-z1-0 has failed, no acks received
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:36:49 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:36:49 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/14 14:36:49 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: cc-bridge-z2-0 10.244.18.14
2016/04/14 14:36:49 [INFO] serf: EventMemberJoin: cell-z2-0 10.244.18.10
2016/04/14 14:36:50 [INFO] memberlist: Suspect api-worker-z1-0 has failed, no acks received
2016/04/14 14:36:51 [INFO] serf: EventMemberJoin: router-z1-0 10.244.0.22
2016/04/14 14:36:51 [INFO] serf: EventMemberJoin: cell-z3-0 10.244.20.6
2016/04/14 14:36:51 [INFO] memberlist: Suspect access-z2-0 has failed, no acks received
2016/04/14 14:36:51 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:51 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:52 [INFO] memberlist: Marking router-z2-0 as failed, suspect timeout reached
2016/04/14 14:36:52 [INFO] serf: EventMemberFailed: router-z2-0 10.244.2.22
2016/04/14 14:36:53 [INFO] memberlist: Suspect api-worker-z2-0 has failed, no acks received
2016/04/14 14:36:53 [INFO] serf: EventMemberJoin: uaa-z2-0 10.244.2.130
2016/04/14 14:36:54 [WARN] raft: Election timeout reached, restarting election
2016/04/14 14:36:54 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:54 [INFO] memberlist: Marking database-z3-0 as failed, suspect timeout reached
2016/04/14 14:36:54 [INFO] serf: EventMemberFailed: database-z3-0 10.244.20.2
2016/04/14 14:36:54 [INFO] memberlist: Marking consul-z1-1 as failed, suspect timeout reached
2016/04/14 14:36:54 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 14:36:54 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:54 [INFO] memberlist: Suspect cell-z3-0 has failed, no acks received
2016/04/14 14:36:55 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:36:56 [INFO] serf: EventMemberJoin: router-z2-0 10.244.2.22
2016/04/14 14:36:56 [INFO] serf: EventMemberFailed: consul-z1-0 10.244.0.54
2016/04/14 14:36:56 [INFO] serf: EventMemberJoin: database-z3-0 10.244.20.2
2016/04/14 14:36:56 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:36:56 [INFO] consul: removing server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:36:56 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:36:56 [INFO] memberlist: Suspect route-emitter-z2-0 has failed, no acks received
2016/04/14 14:36:57 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:36:59 [WARN] raft: Election timeout reached, restarting election 2016/04/14 14:36:59 [INFO] memberlist: Suspect blobstore-z1-0 has failed, no acks received
2016/04/14 14:36:59 [INFO] raft: Node at 10.244.2.54:8300 [Candidate] entering Candidate state
2016/04/14 14:36:59 [INFO] serf: EventMemberJoin: consul-z1-0 10.244.0.54
2016/04/14 14:37:00 [INFO] consul: adding server consul-z1-0 (Addr: 10.244.0.54:8300) (DC: dc1)
2016/04/14 14:37:00 [WARN] memberlist: Refuting a suspect message (from: consul-z2-0)
2016/04/14 14:37:00 [INFO] consul: New leader elected: consul-z1-1
2016/04/14 14:37:01 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:01 [INFO] raft: Duplicate RequestVote for same term: 10
2016/04/14 14:37:01 [INFO] memberlist: Suspect cc-bridge-z2-0 has failed, no acks received
2016/04/14 14:37:01 [INFO] serf: EventMemberFailed: database-z2-0 10.244.18.2
2016/04/14 14:37:02 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/14 14:37:03 [INFO] memberlist: Suspect database-z2-0 has failed, no acks received
2016/04/14 14:37:04 [INFO] memberlist: Suspect api-z1-0 has failed, no acks received 2016/04/14 14:37:04 [INFO] raft: Node at 10.244.2.54:8300 [Follower] entering Follower state
2016/04/14 14:37:05 [INFO] raft: Duplicate RequestVote for same term: 11
2016/04/14 14:37:05 [WARN] raft: Failed to get previous log: 103 log not found (last: 100)
2016/04/14 14:37:06 [INFO] memberlist: Suspect cc-bridge-z1-0 has failed, no acks received
2016/04/14 14:37:06 [INFO] memberlist: Marking brain-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:06 [INFO] serf: EventMemberFailed: brain-z2-0 10.244.18.6
2016/04/14 14:37:06 [INFO] memberlist: Marking consul-z1-1 as failed, suspect timeout reached
2016/04/14 14:37:06 [INFO] serf: EventMemberFailed: consul-z1-1 10.244.0.58
2016/04/14 14:37:06 [INFO] memberlist: Marking api-z2-0 as failed, suspect timeout reached
2016/04/14 14:37:06 [INFO] serf: EventMemberFailed: api-z2-0 10.244.2.134
2016/04/14 14:37:06 [INFO] consul: removing server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:37:07 [INFO] memberlist: Suspect ha-proxy-z1-0 has failed, no acks received
2016/04/14 14:37:09 [INFO] serf: EventMemberJoin: brain-z2-0 10.244.18.6
2016/04/14 14:37:10 [INFO] memberlist: Suspect consul-z1-0 has failed, no acks received
2016/04/14 14:37:10 [INFO] memberlist: Marking uaa-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:10 [INFO] serf: EventMemberFailed: uaa-z1-0 10.244.0.134
2016/04/14 14:37:10 [INFO] memberlist: Marking brain-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:10 [INFO] serf: EventMemberFailed: brain-z1-0 10.244.16.6
2016/04/14 14:37:10 [INFO] memberlist: Marking api-worker-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:10 [INFO] serf: EventMemberFailed: api-worker-z1-0 10.244.0.154
2016/04/14 14:37:10 [INFO] serf: EventMemberJoin: consul-z1-1 10.244.0.58
2016/04/14 14:37:10 [INFO] consul: adding server consul-z1-1 (Addr: 10.244.0.58:8300) (DC: dc1)
2016/04/14 14:37:11 [INFO] memberlist: Suspect router-z1-0 has failed, no acks received
2016/04/14 14:37:12 [INFO] serf: EventMemberJoin: api-worker-z1-0 10.244.0.154
2016/04/14 14:37:12 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since we have a leader: 10.244.0.58:8300
2016/04/14 14:37:12 [INFO] serf: EventMemberJoin: database-z2-0 10.244.18.2
2016/04/14 14:37:13 [ERR] raft-net: Failed to flush response: write tcp 10.244.2.53:57828: broken pipe
2016/04/14 14:37:13 [INFO] serf: EventMemberJoin: uaa-z1-0 10.244.0.134
2016/04/14 14:37:13 [INFO] serf: EventMemberJoin: api-z2-0 10.244.2.134
2016/04/14 14:37:13 [INFO] memberlist: Suspect route-emitter-z1-0 has failed, no acks received
2016/04/14 14:37:15 [INFO] memberlist: Marking api-z1-0 as failed, suspect timeout reached
2016/04/14 14:37:15 [INFO] serf: EventMemberFailed: api-z1-0 10.244.0.138
2016/04/14 14:37:15 [WARN] raft: Clearing log suffix from 110 to 110
2016/04/14 14:37:15 [INFO] memberlist: Suspect access-z1-0 has failed, no acks received
2016/04/14 14:37:16 [WARN] memberlist: Refuting a suspect message (from: route-emitter-z2-0)
2016/04/14 14:37:16 [INFO] memberlist: Suspect consul-z1-1 has failed, no acks received
2016/04/14 14:37:17 [WARN] raft: Rejecting vote from 10.244.0.54:8300 since we have a leader: 10.244.0.58:8300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment