etcd2 cluster

gistfile1.txt

-- Logs begin at Wed 2015-06-10 14:03:02 , end at Wed 2015-06-10 14:12:52 . --
Jun 10 14:03:30 ip-123-123-123-105.eu-west-1.compute.internal systemd[1]: Started etcd2.
Jun 10 14:03:30 ip-123-123-123-105.eu-west-1.compute.internal systemd[1]: Starting etcd2...
Jun 10 14:03:36 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:36 etcd: peerTLS: cert = /etc/ssl/etcd/certs/cont-cros-uact-001.crt, key = /etc/ssl/etcd/private/cont-cros-uact-001.key, ca = /etc/ssl/etcd/certs/etcd-ca.crt
Jun 10 14:03:39 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:39 etcd: listening for peers on https://123.123.123.105:2380
Jun 10 14:03:39 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:39 etcd: clientTLS: cert = /etc/ssl/etcd/certs/cont-cros-uact-001.crt, key = /etc/ssl/etcd/private/cont-cros-uact-001.key, ca = /etc/ssl/etcd/certs/etcd-ca.crt
Jun 10 14:03:42 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:42 etcd: listening for client requests on https://0.0.0.0:2379
Jun 10 14:03:42 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:42 etcdserver: datadir is valid for the 2.0.1 format
Jun 10 14:03:44 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:44 discovery: found self 7a10916397e4a7b0 in the cluster
Jun 10 14:03:44 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:44 discovery: found 1 peer(s), waiting for 2 more
Jun 10 14:03:45 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:45 discovery: found peer 7efaa3009a19f237 in the cluster
Jun 10 14:03:45 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:45 discovery: found 2 peer(s), waiting for 1 more
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 discovery: found peer f7ec2a3ed5ac497d in the cluster
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 discovery: found 3 needed peer(s)
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: name = bf215f9ed156445180ceeec6b6490ac7
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: data dir = /var/lib/etcd2
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: member dir = /var/lib/etcd2/member
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: heartbeat = 100ms
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: election = 1200ms
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: snapshot count = 10000
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: discovery URL= https://discovery.etcd.io/773a17e27b8bf1dd8d38689648dab784
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: advertise client URLs = https://123.123.123.105:2379
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: initial advertise peer URLs = https://123.123.123.105:2380
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: initial cluster = 3bf61f3b014248e98b7cf70017fde76e=https://123.123.123.6:2380,bf215f9ed156445180ceeec6b6490ac7=https://123.123.123.105:2380,ef5922c37a4240e6a1834d6fe872631e=https://123.123.123.143:2380
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: start member 7a10916397e4a7b0 in cluster fd803cf973d011e9
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 raft: 7a10916397e4a7b0 became follower at term 0
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 raft: newRaft 7a10916397e4a7b0 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 raft: 7a10916397e4a7b0 became follower at term 1
Jun 10 14:03:54 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:54 etcdserver: added local member 7a10916397e4a7b0 [https://123.123.123.105:2380] to cluster fd803cf973d011e9
Jun 10 14:03:55 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:55 etcdserver: added member 7efaa3009a19f237 [https://123.123.123.143:2380] to cluster fd803cf973d011e9
Jun 10 14:03:55 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:55 etcdserver: added member f7ec2a3ed5ac497d [https://123.123.123.6:2380] to cluster fd803cf973d011e9
Jun 10 14:03:56 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:56 raft: 7a10916397e4a7b0 is starting a new election at term 1
Jun 10 14:03:56 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:56 raft: 7a10916397e4a7b0 became candidate at term 2
Jun 10 14:03:56 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:56 raft: 7a10916397e4a7b0 received vote from 7a10916397e4a7b0 at term 2
Jun 10 14:03:56 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:56 raft: 7a10916397e4a7b0 [logterm: 1, index: 3] sent vote request to 7efaa3009a19f237 at term 2
Jun 10 14:03:56 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:56 raft: 7a10916397e4a7b0 [logterm: 1, index: 3] sent vote request to f7ec2a3ed5ac497d at term 2
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 [logterm: 1, index: 3, vote: 7a10916397e4a7b0] rejected vote from 7efaa3009a19f237 [logterm: 1, index: 3] at term 2
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 received vote rejection from f7ec2a3ed5ac497d at term 2
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 [q:2] has received 1 votes and 1 vote rejections
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 received vote rejection from 7efaa3009a19f237 at term 2
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 [q:2] has received 1 votes and 2 vote rejections
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft: 7a10916397e4a7b0 became follower at term 2
Jun 10 14:03:57 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:57 raft.node: 7a10916397e4a7b0 elected leader 7efaa3009a19f237 at term 2
Jun 10 14:03:58 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:58 rafthttp: starting client stream to 7efaa3009a19f237 at term 2
Jun 10 14:03:59 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:03:59 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:04 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:04 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:05 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:05 etcdhttp: unexpected error: etcdserver: request timed out
Jun 10 14:04:06 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:06 etcdhttp: unexpected error: etcdserver: request timed out
Jun 10 14:04:09 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:09 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:14 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:14 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:19 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:19 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 is starting a new election at term 2
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 became candidate at term 3
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 received vote from 7a10916397e4a7b0 at term 3
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to 7efaa3009a19f237 at term 3
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to f7ec2a3ed5ac497d at term 3
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft.node: 7a10916397e4a7b0 lost leader 7efaa3009a19f237 at term 3
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 [term: 3] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 rafthttp: encountered error reading the client log stream: read tcp 123.123.123.143:2380: use of closed network connection
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 rafthttp: client streaming to 7efaa3009a19f237 at term 2 has been stopped
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 [term: 3] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:21 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:21 raft: 7a10916397e4a7b0 [term: 3] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:22 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:22 raft: 7a10916397e4a7b0 [term: 3] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 received vote rejection from f7ec2a3ed5ac497d at term 3
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 [q:2] has received 1 votes and 1 vote rejections
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 etcdhttp: unexpected error: etcdserver: request timed out
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 is starting a new election at term 3
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 became candidate at term 4
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 received vote from 7a10916397e4a7b0 at term 4
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to 7efaa3009a19f237 at term 4
Jun 10 14:04:23 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:23 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to f7ec2a3ed5ac497d at term 4
Jun 10 14:04:24 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:24 raft: 7a10916397e4a7b0 [term: 4] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:24 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:24 raft: 7a10916397e4a7b0 [term: 4] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:24 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:24 raft: 7a10916397e4a7b0 [term: 4] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:24 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:24 raft: 7a10916397e4a7b0 [logterm: 2, index: 66, vote: 7a10916397e4a7b0] rejected vote from f7ec2a3ed5ac497d [logterm: 2, index: 70] at term 4
Jun 10 14:04:24 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:24 raft: 7a10916397e4a7b0 [term: 4] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 etcdhttp: unexpected error: etcdserver: request timed out
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 etcdserver: publish error: etcdserver: request timed out
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 received vote rejection from f7ec2a3ed5ac497d at term 4
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 [q:2] has received 1 votes and 1 vote rejections
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 is starting a new election at term 4
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 became candidate at term 5
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 received vote from 7a10916397e4a7b0 at term 5
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to 7efaa3009a19f237 at term 5
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 [logterm: 2, index: 66] sent vote request to f7ec2a3ed5ac497d at term 5
Jun 10 14:04:25 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:25 raft: 7a10916397e4a7b0 [logterm: 2, index: 66, vote: 7a10916397e4a7b0] rejected vote from f7ec2a3ed5ac497d [logterm: 2, index: 70] at term 5
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 received vote rejection from f7ec2a3ed5ac497d at term 5
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 [q:2] has received 1 votes and 1 vote rejections
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 [term: 5] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 [term: 5] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 [term: 5] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:26 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:26 raft: 7a10916397e4a7b0 [term: 5] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:27 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:27 raft: 7a10916397e4a7b0 [term: 5] received a MsgVote message with higher term from f7ec2a3ed5ac497d [term: 6]
Jun 10 14:04:27 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:27 raft: 7a10916397e4a7b0 became follower at term 6
Jun 10 14:04:27 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:27 raft: 7a10916397e4a7b0 [logterm: 2, index: 66, vote: 0] voted for f7ec2a3ed5ac497d [logterm: 2, index: 70] at term 6
Jun 10 14:04:28 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:28 raft: 7a10916397e4a7b0 [logterm: 0, index: 70] rejected msgApp [logterm: 2, index: 70] from f7ec2a3ed5ac497d
Jun 10 14:04:28 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:28 raft.node: 7a10916397e4a7b0 elected leader f7ec2a3ed5ac497d at term 6
Jun 10 14:04:28 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:28 raft: 7a10916397e4a7b0 [term: 6] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:28 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:28 raft: 7a10916397e4a7b0 [term: 6] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]
Jun 10 14:04:28 ip-123-123-123-105.eu-west-1.compute.internal etcd2[598]: 2015/06/10 14:04:28 raft: 7a10916397e4a7b0 [term: 6] ignored a MsgHeartbeat message with lower term from 7efaa3009a19f237 [term: 2]