Zookeeper and Exhibitor

Zookeeper and Exhibitor.md

Overview

Applications use Apache Zookeeper to coordinate distributed systems. Soabase Exhibitor originally developed by Netflix is a management tool for Zookeeper. It provides APIs to interact with the Zookeeper cluster. This document will cover the usage of non-destructible Exhibitor APIs.

• Zookeeper recipes
• Exhibitor

Troubleshooting Zookeeper Processes

If zookeeper process(es) stops,

• Exhibitor will restart the zookeeper process. Data gets replicated. Application support team need not be notified
• There may be slowness in app processing till the cluster is up

If the whole zookeeper cluster goes down,

• Exhibitor will restart the cluster automatically. Application support team need not be notified
• There will be slowness/halt in app processing till the cluster is up

If zookeeper and exhibitor process stops,

• Ops team should get notified and they should take steps to restart the processes
• No changes to the app
• Application processes will fail and perform retries

Access Zookeeper through Rest endpoints

Setup

1. Install Postman Chrome extension
2. Create a postman json file with contents provided below in comments section
3. Import the collection to postman
4. Create appropriate environments in postman like dev,staging,prod etc.

Functionality

• Analyze lock
• Usage listing
• Cluster status (Code 3 ins the response means connected)
• Node status
• Get node log
• 4ltr usage (https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkCommands)

Copy the content below and create a json file. Import the file to postman and follow instructions as mentioned in the Gist

{ "id": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "name": "zookeeper", "description": "", "order": [ "06f152f9-4ed1-c3d8-3cd8-e925c06af4eb", "223ea82b-69b9-782a-5ad3-3346ff6ece2e", "27357adb-2bbd-3e88-f223-66fba582508e", "e5e3853e-5d66-6bdc-3da2-8bbbd82dfae9", "d728ab8b-00e0-feaf-cd8b-96fbff783bd1", "b0e8aff0-5574-d0f1-3df3-68a3041019d1" ], "folders": [], "timestamp": 1462457753694, "owner": 0, "public": false, "requests": [ { "id": "06f152f9-4ed1-c3d8-3cd8-e925c06af4eb", "headers": "Content-Type: application/json\n", "url": "http://zoo{{env}}01:8080/exhibitor/v1/explorer/analyze", "preRequestScript": null, "pathVariables": {}, "method": "POST", "data": [], "dataMode": "raw", "version": 2, "tests": null, "currentHelper": "normal", "helperAttributes": {}, "time": 1477412041047, "name": "Analyze Lock", "description": "Analyze a set of paths assuming they are Curator Lock paths and return details about lock ownership and possible deadlocks.", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "responses": [], "rawModeData": "{\n \"max\": 1,\n \"path\": \"/app/lock\"\n}" }, { "id": "223ea82b-69b9-782a-5ad3-3346ff6ece2e", "headers": "Content-Type: application/json\n", "url": "http://zoo{{env}}01:8080/exhibitor/v1/explorer/usage-listing", "preRequestScript": null, "pathVariables": {}, "method": "POST", "data": [], "dataMode": "raw", "version": 2, "tests": null, "currentHelper": "normal", "helperAttributes": {}, "time": 1477412644975, "name": "Usage listing", "description": "Produces a detailed listing from a starting path showing all the paths, creation time, immediate children count and deep children count.", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "responses": [], "rawModeData": "{\n \"maxChildrenForTraversal\": 1000,\n \"startPath\": \"/app/lock\"\n}" }, { "id": "27357adb-2bbd-3e88-f223-66fba582508e", "headers": "", "url": "http://zoo{{env}}01:8080/exhibitor/v1/cluster/status", "pathVariables": {}, "preRequestScript": null, "method": "GET", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "data": null, "dataMode": "params", "name": "Cluster status", "description": "Returns the status of the entire cluster", "descriptionFormat": "html", "time": 1477407518068, "version": 2, "responses": [], "tests": null, "currentHelper": "normal", "helperAttributes": {} }, { "id": "b0e8aff0-5574-d0f1-3df3-68a3041019d1", "headers": "", "url": "http://zoo{{env}}01:8080/exhibitor/v1/cluster/log", "pathVariables": {}, "preRequestScript": null, "method": "GET", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "data": null, "dataMode": "params", "name": "Get log", "description": "Return the Exhibitor log from the local instance.", "descriptionFormat": "html", "time": 1477408216120, "version": 2, "responses": [], "tests": null, "currentHelper": "normal", "helperAttributes": {} }, { "id": "d728ab8b-00e0-feaf-cd8b-96fbff783bd1", "headers": "", "url": "http://zoo{{env}}01:8080/exhibitor/v1/cluster/4ltr/stat", "pathVariables": {}, "preRequestScript": null, "method": "GET", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "data": null, "dataMode": "params", "name": "4ltr", "description": "Return the result of the ZooKeeper four letter word on the local instance.", "descriptionFormat": "html", "time": 1477407913723, "version": 2, "responses": [], "tests": null, "currentHelper": "normal", "helperAttributes": {} }, { "id": "e5e3853e-5d66-6bdc-3da2-8bbbd82dfae9", "headers": "", "url": "http://zoo{{env}}01:8080/exhibitor/v1/cluster/state/", "preRequestScript": null, "pathVariables": {}, "method": "GET", "data": null, "dataMode": "params", "version": 2, "tests": null, "currentHelper": "normal", "helperAttributes": {}, "time": 1477412019469, "name": "Node status", "description": "Returns the system state of the given instance in the ensemble.", "collectionId": "3a20e5eb-7c62-f479-49d6-edf90e29b225", "responses": [] } ] }