Some Dude ericacm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| trait ClusterService { | |
| def initialize() | |
| def enabled: Boolean | |
| def clusterStatus: ClusterStatus | |
| def nodeId: String | |
| } | |
| case class ClusterStatus( | |
| current: String, | |
| isLeader: Boolean, |
ericacm
/ gist:4315253
Created
December 17, 2012 01:56
Test after building from source. I also removed the RetryLoop. This is what happens after I kill the ZK leader. The original leader localhost:14181 loses leadership and no member gets hasLeadership though the participant.isLeader is true for localhost:14189.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This is localhost:14181 - the original leader | |
| 20:45:43.175 [RMI TCP Connection(2)-127.0.0.1-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - nodeId=localhost:14181 hasLeadership=true, Cluster participants: localhost:14181, localhost:14189, leader=localhost:14181 | |
| 20:46:00.017 [ThreadPoolTaskScheduler-1] ERROR com.netflix.curator.ConnectionState - Connection timed out for connection string (localhost:14181,localhost:14185,localhost:14189) and timeout (15000) / elapsed (16724) | |
| org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss | |
| at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:94) ~[curator-client-1.2.3.jar:na] | |
| at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:105) [curator-client-1.2.3.jar:na] | |
| at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:410) [curator-framework-1.2.3.jar:na] | |
| at com.netflix.curator.framework.imps.GetChildrenBuilderImpl$3.call(Ge |
ericacm
/ gist:4247058
Created
December 9, 2012 21:18
ZK leader killed, app leadership moves to different cluster member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original leader (localhost:14181): | |
| 16:11:18.002 [ForkJoinPool-1-worker-9] DEBUG c.n.s.n.DeliveryService$DeliverySerializer - received SerializedDelivery: none | |
| 16:11:18.002 [ForkJoinPool-1-worker-9] DEBUG c.n.s.n.DeliveryServiceImpl - delivering noDigest events | |
| 16:11:18.128 [main-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - watchedEvent: eventType=NodeChildrenChanged state=SyncConnected | |
| 16:11:18.139 [main-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - nodeId=localhost:14181 isLeader=true, Cluster participants: localhost:14181, localhost:14185, leader=localhost:14181 | |
| 16:11:18.250 [main-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - watchedEvent: eventType=None state=Disconnected | |
| 16:11:24.498 [main-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - nodeId=localhost:14181 isLeader=true, Cluster participants: localhost:14181, localhost:14185, leader=localhost:14181 | |
| 16:11:24.500 [main-EventThread] INFO c.nyx.services.ZookeeperServiceImpl - watchedEvent: eventType=None state= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @Value("${deliveryService.distributeSchedule:0,30 * * * * *}") | |
| var distributeSchedule: String = _ | |
| schedulerService.addJob( | |
| SchedulerJob("distributeEvents", () => scheduledDistribute(), distributeSchedule)) | |
| def scheduledDistribute() = | |
| if (clusterService.clusterStatus.isLeader) | |
| distribute() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| trait ClusterService { | |
| def initialize() | |
| def enabled: Boolean | |
| def clusterStatus: ClusterStatus | |
| } | |
| case class ClusterStatus( | |
| current: String, | |
| isLeader: Boolean, leader: String, | |
| participants: Iterable[String]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def refreshWorkers() { | |
| val cs = clusterService.clusterStatus | |
| for (participant <- cs.participants) { | |
| val path = if (cs.current == participant) | |
| akkaConfig.localActorPath(workerPath) | |
| else | |
| remotePathForParticipant( | |
| akkaConfig, participant, workerPath) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def initialize() { | |
| // Embedded server startup here (see part 2) | |
| // ... | |
| curatorFramework = | |
| CuratorFrameworkFactory.newClient( | |
| connectString, sessionTimeoutSec*1000, | |
| connectionTimeoutSec*1000, new RetryOneTime(1)) | |
| curatorFramework.start() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // hosts is in the form host:port,host:port | |
| @Value("${zookeeperService.hosts:localhost}") | |
| @ManagedGetter @BeanProperty | |
| var hosts: String = _ | |
| val hostsArr = hosts.split(",") | |
| val serversKeyArr = new Array[String](hostsArr.length) | |
| val serversValArr = new Array[String](hostsArr.length) | |
| for ((server, idx) <- hostsArr.zipWithIndex) { | |
| val (host, port) = if (server.contains(":")) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| if (clientEnabled) { | |
| startThread("leaderSelection") { | |
| while (electingLeader) { | |
| selectLeader() | |
| electLoop.acquire() | |
| if (electingLeader) log.info("Lost connection to Zookeeper - reselecting leader") | |
| } | |
| } | |
| log.info("Waiting for leader to be selected") | |
| leaderSelected.acquire() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| startThread("leaderSelection") { | |
| while (electingLeader) { | |
| selectLeader() | |
| electLoop.acquire() | |
| if (electingLeader) log.info("Lost connection to Zookeeper - reselecting leader") | |
| } | |
| } | |
| log.info("Waiting for leader to be selected") | |
| leaderSelected.acquire() |