zk exception

 [DEBUG] - initializing zookeeper cluster
May 27 23:38:26 deis-04 sh[1368]: [DEBUG] - adding node %v to zookeeper cluster172.17.8.103
May 27 23:38:26 deis-04 sh[1368]: [DEBUG] - set /zookeeper/nodes/172.17.8.103/id -> 1
May 27 23:38:26 deis-04 sh[1368]: [DEBUG] - adding node %v to zookeeper cluster172.17.8.101
May 27 23:38:26 deis-04 sh[1368]: [DEBUG] - set /zookeeper/nodes/172.17.8.101/id -> 2
May 27 23:38:27 deis-04 sh[1368]: [DEBUG] - adding node %v to zookeeper cluster172.17.8.102
May 27 23:38:27 deis-04 sh[1368]: [DEBUG] - set /zookeeper/nodes/172.17.8.102/id -> 3
May 27 23:38:27 deis-04 sh[1368]: [DEBUG] - adding node %v to zookeeper cluster172.17.8.100
May 27 23:38:27 deis-04 sh[1368]: [DEBUG] - set /zookeeper/nodes/172.17.8.100/id -> 4
May 27 23:38:29 deis-04 sh[1368]: [INFO] - waiting for confd to write initial templates...
May 27 23:38:29 deis-04 sh[1368]: [DEBUG] - running confd -onetime -node 127.0.0.1:4001 -confdir /app
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Backend set to etcd
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Starting confd
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Backend nodes set to 127.0.0.1:4001
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Target config /opt/zookeeper/conf/fleet-zoo_cfg.dynamic out of sync
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Target config /opt/zookeeper/conf/fleet-zoo_cfg.dynamic has been updated
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Target config /opt/zookeeper/conf/server.list out of sync
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27T23:38:29Z deis-04 confd[13]: INFO Target config /opt/zookeeper/conf/server.list has been updated
May 27 23:38:29 deis-04 sh[1368]: + main
May 27 23:38:29 deis-04 sh[1368]: + export PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.:/jre/bin
May 27 23:38:29 deis-04 sh[1368]: + PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.:/jre/bin
May 27 23:38:29 deis-04 sh[1368]: + cp /app/zoo.cfg /opt/zookeeper-data/zoo.cfg
May 27 23:38:29 deis-04 sh[1368]: + ln -s /opt/zookeeper-data/zoo.cfg /opt/zookeeper/conf/zoo.cfg
May 27 23:38:29 deis-04 sh[1368]: + cp /opt/zookeeper/conf/fleet-zoo_cfg.dynamic /opt/zookeeper-data/zoo_cfg.dynamic
May 27 23:38:29 deis-04 sh[1368]: ++ cut -d , -f 1
May 27 23:38:29 deis-04 sh[1368]: ++ sed -e 's/172.17.8.103:3888//;s/^,//;s/,$//'
May 27 23:38:29 deis-04 sh[1368]: + ZKHOST=172.17.8.100:3888
May 27 23:38:29 deis-04 sh[1368]: + echo 'adding 172.17.8.103 as server to the zookeeper cluster'
May 27 23:38:29 deis-04 sh[1368]: adding 172.17.8.103 as server to the zookeeper cluster
May 27 23:38:29 deis-04 sh[1368]: + echo ''
May 27 23:38:29 deis-04 sh[1368]: ++ cat /opt/zookeeper-data/data/myid
May 27 23:38:29 deis-04 sh[1368]: + /opt/zookeeper/bin/zkCli.sh -server 172.17.8.100:3888 reconfig -add 'server.1=172.17.8.103:2181:2888:participant;172.17.8.103:3888'
May 27 23:38:29 deis-04 sh[1368]: Connecting to 172.17.8.100:3888
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,400 [myid:] - INFO  [main:Environment@109] - Client environment:zookeeper.version=3.5.0-alpha-1615249, built on 08/01/2014 22:13 GMT
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,404 [myid:] - INFO  [main:Environment@109] - Client environment:host.name=<NA>
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,404 [myid:] - INFO  [main:Environment@109] - Client environment:java.version=1.8.0_45
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,406 [myid:] - INFO  [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,407 [myid:] - INFO  [main:Environment@109] - Client environment:java.home=/jre
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,407 [myid:] - INFO  [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper/bin/../lib/servlet-api-2.5-20081211.jar:/opt/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-6.1.26.jar:/opt/zookeeper/bin/../lib/jetty-6.1.26.jar:/opt/zookeeper/bin/../lib/javacc.jar:/opt/zookeeper/bin/../lib/jackson-mapper-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../zookeeper-3.5.0-alpha.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,408 [myid:] - INFO  [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,411 [myid:] - INFO  [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,411 [myid:] - INFO  [main:Environment@109] - Client environment:java.compiler=<NA>
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,412 [myid:] - INFO  [main:Environment@109] - Client environment:os.name=Linux
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,412 [myid:] - INFO  [main:Environment@109] - Client environment:os.arch=amd64
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,413 [myid:] - INFO  [main:Environment@109] - Client environment:os.version=3.19.3
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,413 [myid:] - INFO  [main:Environment@109] - Client environment:user.name=root
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,413 [myid:] - INFO  [main:Environment@109] - Client environment:user.home=/root
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,414 [myid:] - INFO  [main:Environment@109] - Client environment:user.dir=/app
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,415 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.free=26MB
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,418 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.max=247MB
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,418 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.total=30MB
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,420 [myid:] - INFO  [main:ZooKeeper@709] - Initiating client connection, connectString=172.17.8.100:3888 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@deb6432
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,445 [myid:] - INFO  [main-SendThread(172.17.8.100:3888):ClientCnxn$SendThread@1093] - Opening socket connection to server 172.17.8.100/172.17.8.100:3888. Will not attempt to authenticate using SASL (unknown error)
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,546 [myid:] - WARN  [main-SendThread(172.17.8.100:3888):ClientCnxn$SendThread@1213] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
May 27 23:38:29 deis-04 sh[1368]: java.net.ConnectException: Connection refused
May 27 23:38:29 deis-04 sh[1368]: at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
May 27 23:38:29 deis-04 sh[1368]: at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
May 27 23:38:29 deis-04 sh[1368]: at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
May 27 23:38:29 deis-04 sh[1368]: at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
May 27 23:38:29 deis-04 sh[1368]: KeeperErrorCode = ConnectionLoss for
May 27 23:38:29 deis-04 sh[1368]: [INFO] - zookeeper: running...
May 27 23:38:29 deis-04 sh[1368]: [DEBUG] - starting pprof http server in port 6060
May 27 23:38:29 deis-04 sh[1368]: JMX enabled by default
May 27 23:38:29 deis-04 sh[1368]: Using config: /opt/zookeeper/bin/../conf/zoo.cfg
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,966 [myid:] - INFO  [main:QuorumPeerConfig@109] - Reading configuration from: /opt/zookeeper/bin/../conf/zoo.cfg
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,983 [myid:] - WARN  [main:QuorumPeerConfig@448] - Non-optimial configuration, consider an odd number of servers.
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,993 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,993 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
May 27 23:38:29 deis-04 sh[1368]: 2015-05-27 23:38:29,994 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,006 [myid:1] - INFO  [main:QuorumPeerMain@136] - Starting quorum peer
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,022 [myid:1] - INFO  [main:NIOServerCnxnFactory@670] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,026 [myid:1] - INFO  [main:NIOServerCnxnFactory@683] - binding to port /172.17.8.103:3888
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,059 [myid:1] - INFO  [main:Slf4jLog@67] - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,104 [myid:1] - INFO  [main:QuorumPeer@1205] - Local sessions disabled
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,105 [myid:1] - INFO  [main:QuorumPeer@1216] - Local session upgrading disabled
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,105 [myid:1] - INFO  [main:QuorumPeer@1185] - tickTime set to 2000
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,106 [myid:1] - INFO  [main:QuorumPeer@1227] - minSessionTimeout set to 4000
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,106 [myid:1] - INFO  [main:QuorumPeer@1238] - maxSessionTimeout set to 40000
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,106 [myid:1] - INFO  [main:QuorumPeer@1253] - initLimit set to 10
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,139 [myid:1] - INFO  [main:QuorumPeer@671] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,152 [myid:1] - INFO  [main:QuorumPeer@686] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,171 [myid:1] - INFO  [main:Slf4jLog@67] - jetty-6.1.26
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,213 [myid:1] - INFO  [main:Slf4jLog@67] - Started [email protected]:8080
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,218 [myid:1] - INFO  [main:JettyAdminServer@93] - Started AdminServer on port 8080, command URL /commands
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,222 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@550] - My election bind port: /172.17.8.103:2888
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,244 [myid:1] - INFO  [QuorumPeer[myid=1]/172.17.8.103:3888:QuorumPeer@913] - LOOKING
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,247 [myid:1] - INFO  [QuorumPeer[myid=1]/172.17.8.103:3888:FastLeaderElection@888] - New election. My id =  1, proposed zxid=0x0
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,251 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@682] - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,260 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@384] - Cannot open channel to 2 at election address /172.17.8.101:2888
May 27 23:38:30 deis-04 sh[1368]: java.net.ConnectException: Connection refused
May 27 23:38:30 deis-04 sh[1368]: at java.net.PlainSocketImpl.socketConnect(Native Method)
May 27 23:38:30 deis-04 sh[1368]: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
May 27 23:38:30 deis-04 sh[1368]: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
May 27 23:38:30 deis-04 sh[1368]: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
May 27 23:38:30 deis-04 sh[1368]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
May 27 23:38:30 deis-04 sh[1368]: at java.net.Socket.connect(Socket.java:589)
May 27 23:38:30 deis-04 sh[1368]: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:369)
May 27 23:38:30 deis-04 sh[1368]: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:407)
May 27 23:38:30 deis-04 sh[1368]: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:346)
May 27 23:38:30 deis-04 sh[1368]: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:480)
May 27 23:38:30 deis-04 sh[1368]: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:459)
May 27 23:38:30 deis-04 sh[1368]: at java.lang.Thread.run(Thread.java:745)
May 27 23:38:30 deis-04 sh[1368]: 2015-05-27 23:38:30,267 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@384] - Cannot open channel to 3 at election address /172.17.8.102:2888