Setting up multinode Elasticsearch cluster with Docker

setting-up-multinode-elasticsearch-cluster-with-docker.md

Setting up multinode Elasticsearch cluster with Docker

This Wiki demonstrates how to start a multinode elasticsearch cluster with docker from multiple hosts (servers).

For a smooth setup, please ensure your servers are configured as per elasticsearch instructions.

Requirements

1. At least 2 (publicly) accessible servers.
2. Docker installation on all the servers.

Setup

In this setup, we'll assume we would like to set up a 2-node cluster.

Server/host 1

Create .env file

cat >/path/to/.env <<EOF
cluster.name=cluster-name
node.name=node1
http.host=0.0.0.0
transport.host=0.0.0.0
network.host=node1.elasticsearch.net
cluster.initial_master_nodes=node1
# comma-separate as many as needed...
discovery.seed_hosts=node2.elasticsearch.net
EOF

Start docker container

To start the container, we'll need a couple of things in place:

1. Path to the .env file created earlier.
2. --network option set to host. This is important because without it, we will not be able to enroll other nodes in the cluster.

docker run --name elastic \
  --network host \
  --detach \
  --memory 1GB \
  --restart unless-stopped \
  --ulimit memlock=-1:-1 \
  --log-driver local \
  --volume elastic-data:/usr/share/elasticsearch/data \
  --volume elastic-config:/usr/share/elasticsearch/config \
  --env-file /path/to/.env \
  docker.elastic.co/elasticsearch/elasticsearch:8.7.1

N/B: Feel free to add as many docker run options as necessary.

Reset elastic user's password

The value set here will be necessary in the following step - checking cluster is running.

$ docker exec -it elastic /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic

Check cluster is running

$ curl -ku elastic: <replace-with-new-password >https://node1.elasticsearch.net:9200/_cat/nodes

N/B: The curl command above disables certificate verification since we are using elasticsearch's self-signed certs.

You should see something along the following lines if everything worked out.

<IP-address> 65 93 0 0.00 0.00 0.00 cdfhilmrstw * node1

Create an enrollment token

This will be used by nodes from other hosts/servers to join the cluster.

$ docker exec -it elastic /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token --scope node

N/B: Ensure the token is used within 30 minutes, since it is likely to expire after that.

Server/host 2

Create .env file

cat >/path/to/.env <<EOF
ENROLLMENT_TOKEN=<TODO: Replace with value from enrollment token created in server/host 1>
cluster.name=cluster-name
node.name=node2
http.host=0.0.0.0
transport.host=0.0.0.0
network.host=node2.elasticsearch.net
# comma-separate as many as needed...
#discovery.seed_hosts=node1.elasticsearch.net
EOF

N/B: discovery.seed_hosts is commented out for now. We'll need it afterwards to allow the node to rejoin the cluster.

Start docker container

To start the container, we'll need a couple of things in place:

1. Path to the .env file created earlier.
2. --network option set to host. This is important because without it, we will not be able to enroll other nodes in the cluster.

docker run --name elastic \
  --network host \
  --detach \
  --memory 1GB \
  --restart unless-stopped \
  --ulimit memlock=-1:-1 \
  --log-driver local \
  --volume elastic-data:/usr/share/elasticsearch/data \
  --volume elastic-config:/usr/share/elasticsearch/config \
  --env-file /path/to/.env \
  docker.elastic.co/elasticsearch/elasticsearch:8.7.1

N/B: Feel free to add as many docker run options as necessary.

Check cluster is running

$ curl -ku elastic:<replace-with-new-password-from-server-1> https://node2.elasticsearch.net:9200/_cat/nodes

N/B: The curl command above disables certificate verification since we are using elasticsearch's self-signed certs.

You should see something along the following lines if everything worked out.

<IP-address> 65 93 0 0.00 0.00 0.00 cdfhilmrstw * node1
<IP-address> 65 93 0 0.00 0.00 0.00 cdfhilmrstw - node2

Rejoin cluster after restart

Do the following to allow the node to automatically rejoin the cluster after a restart.

Stop and remove docker container

$ docker stop elastic
$ docker rm elastic

Update .env file

1. Remove ENROLLMENT_TOKEN. Unless you've pruned your docker volume for some reason, it is no longer necessary since the node is already registered with the cluster.
2. Uncomment, discovery.seed_hosts, to allow the node to automatically rejoin the cluster after termination.

The updated .env file should look as follows

cat >/path/to/.env <<EOF
cluster.name=cluster-name
node.name=node2
http.host=0.0.0.0
transport.host=0.0.0.0
network.host=node2.elasticsearch.net
# comma-separate as many as needed...
discovery.seed_hosts=node1.elasticsearch.net
EOF

Start the docker container again

Refer to Start docker container for host/server 2.