Skip to content

Instantly share code, notes, and snippets.

@federico-garcia

federico-garcia/consul.md

Last active September 27, 2020 17:53
Show Gist options
  • Save federico-garcia/4a6ac6b19f7e648b1b61ea2538fdfb00 to your computer and use it in GitHub Desktop.
Save federico-garcia/4a6ac6b19f7e648b1b61ea2538fdfb00 to your computer and use it in GitHub Desktop.
Download ZIP
Getting started with Consul
Raw
consul.md

Consul

Consul provides a solution to the following challenges:

  • Service Discovery. It acts as an internal DNS server. No more internal load balancers?
  • Failure Detection. Health checking. It automatically removes failed nodes and services from the DNS response it gives to clients. Consul runs an agent in each server which are in charge of the health of all local services, by doing this, it distribute the health checking task. Agents talk to each other to share information about the services/nodes they are monitoring, adding or removing nodes or services as they become available or start failing.
  • Multi Datacenter. Consul is multi datacenter aware so you can have services available in multiple datacenters but using consul you can connect to locally deployed services in the same datacenter where the request is being originated, instead of going to the other datacenter for no reason. Datacenter is a logical concept. Talks to local services first and failover to ther datacenters if it needs to.
  • Service Configuration. You can store application or service configuration parameters in Consul's key-value store and all changes you make are sent to the application/service almost immediately causing the application/service to react right away to these changes.

You can play with demo deployment of Consul here

There is a consul glossary you can check anytime here

Monitoring nodes

Consul agents can run as clients or servers depending on their configuration. Agents monitor services in a node and report status back to the Consul server.

vagrant init --minimal wesmcclure/ubuntu1404-docker
vagrant up
vagrant ssh consul-server
ip a
docker info

Go and install consul on the consul-server. We can manually install it using an script file:

chmod + /vagrant/install.consul.sh
/vagrant/install.consul.sh

or we can execute the script file within the Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "wesmcclure/ubuntu1404-docker"
  config.vm.provision "shell", path: "install.consul.sh", privileged: false

  config.vm.define "consul-server-1" do |cs|
    cs.vm.hostname = "consul-server-1"
    cs.vm.network "private_network", ip: "172.20.20.31"
  end

  config.vm.define "consul-server-2" do |cs|
    cs.vm.hostname = "consul-server-2"
    cs.vm.network "private_network", ip: "172.20.20.32"
  end

end

then you can destroy and re-create the VM:

vagrant destroy
vagrant up
vagrant ssh consul-server-1
consul

Starting the consul agent as a server in the consul-server-1 VM:

consul agent -server -bootstrap-expect=1 \
    -data-dir=/tmp/consul -node=agent-one -bind=172.20.20.31 \
    -enable-script-checks=true -config-dir=/etc/consul.d -client 0.0.0.0 -ui

Starting the consul agent as a client in the consul-server-2 VM:

consul agent -data-dir=/tmp/consul -node=agent-two \
    -bind=172.20.20.32 -enable-script-checks=true -config-dir=/etc/consul.d -client 0.0.0.0 -ui

Connect to consul-server-1 from another terminal and join the consul agent running in consul-server-2

vagrant ssh consul-server-1
consul join 172.20.20.32

Checking the members of the consul cluster

consul members

Consul provides a Web UI you can use to manage all its activity: services, nodes, key-value store, datacenters and ACL.

http://172.20.20.31:8500/ui

Even if you're connecting from a client consul agent to view the UI interface, all the information is coming a from a Consul server.

Consul also provides a REST API so you get information out of the cluster programatically. You can get more information here.

Consul also provides a DNS interface on port 8600 you can use to make DNS queries for nodes and services registered in Consul.

dig @<consul-agent-address> -p <consul-dns-interface-port> <node-name>.node.consul
dig @172.20.20.31 -p 8600 consul-server-1.node.consul

dig @<consul-agent-address> -p <consul-dns-interface-port> <service-name>.service.consul SRV
dig @172.20.20.31 -p 8600 consul.service.consul SRV

Consul also provides an RPC interface on port 8400 which is used for communication between the consul CLI and the consul agents. So you can from your computer, get information about a consul agent in a remote host if you change the default RPC address to the address of the remote host. e.g. run each command with the following flag -rpc-addr=172.20.20.31:8400

Executing a command on mutiple remote consul servers. The response from all Consul agents is sent to your terminal. Take into account this is disabled by dafult, if you want to this you need to set the parameter disable_remote_exec to false.

consul exec -h
consul exex [options] [command]
consul exec uptime
consul exec -node=<name> uptime

Leaving a Consul cluster gracefully

consul leave

Service Discovery

Here is what we want to build

consul

A service can be registered either by providing a service definition or by making the appropriate calls to the HTTP API.

First, create a directory for Consul configuration. Consul loads all configuration files in the configuration directory, so a common convention on Unix systems is to name the directory something like /etc/consul.d (the .d suffix implies "this directory contains a set of configuration files").

Here is a pretty simple, service definition

{
  "service": {
    "name": "web", 
    "port": 8080
  }
}

Registering a service definition with a new config file web.service.consul.json. You can register srevices even if there are not real services running in the node, service regustration is independent of service management. If not health checks are added to the service, the service is reported as running if the node is running OK.

ip=$(ifconfig eth1 | grep 'inet addr' | awk '{ print substr($2,6) }')  
consul agent -advertise $ip -config-file /vagrant/common.consul.json -config-file /vagrant/web.service.consul.json &

Adding a simple health check to the service registration config file

{
  "service": {
    "name": "web", 
    "port": 8080,
    "check": {
      "http": "http://localhost:8080",
      "interval": "10s"
    }
  }
}

You can send Consul a SIGHUP so it reloads its configuration file without having to stop and restart the service

killall -s 1 consul

You'll see in the Consul UI teh service is now marked as failing due to the health check. When a service running in a node is failing, Consul won't include that service's lcoation when asked for that service. Once the service in that node gets back to healthy, Consul will include that location as part of its DNS response.

We need to add the nginx web server so the health check is reported as passing. To do that, we'll take advantage of the Docker instance running in all the servers we created with Vagrant. Here is the script we can execute to run nginx as a dokcer container

#!/usr/bin/env bash

# Create an html page with the ip of this node
ip=$(ifconfig eth1 | grep 'inet addr' | awk '{ print substr($2,6) }')
echo "<h1>$ip $(hostname)</h1>" > /home/vagrant/ip.html

# Run nginx via docker
# mount the ip.html page as a volume in the root of the default nginx site
# thus we can access this page via `curl localhost:8080/ip.html`
docker run -d \
           --name web \
           -p 8080:80 \
           --restart unless-stopped \
           -v /home/vagrant/ip.html:/usr/share/nginx/html/ip.html:ro \
           nginx:alpine

SSH into the web server VM where you registered the web service and execute the following commands to get nginx up and running

chmod +x /vagrant/setup-web-server.sh
/vagrant/setup-web-server.sh

Performing a basic sanity test on Consul configuration files

consul configtest -config-file <config-file-path>

In order to set-up the HAProxy, we need to register the LB service in Consul using the following config file

{
  "service": {
    "name": "lb", 
    "port": 80,
    "check": {
      "http": "http://localhost",
      "interval": "10s"
    }
  }
}

Register the LB service as follows

vagrant ssh lb
killall consul
ip=$(ifconfig eth1 | grep 'inet addr' | awk '{ print substr($2,6) }')  
consul agent -advertise $ip -config-file /vagrant/common.consul.json -config-file /vagrant/lb.service.consul.json &

The service will fail since there are not a real service running on port 80 in the LB node. That's OK. We'll fix that in a minute.

Putting Consul nodes in maintenance node. This puts all services running on this node in critical mode, which means, this node won't be part of the DNS response Consul provides during service discovery.

consul maint -enable -reason <enter the reason why this node is in maint node for others to see>
consul maint
consul maint -disable

There are basically 4 ways to register services in Consul: service definition/configuration files, using the Consul HTTP API , make use your app Consul aware and using a tool like Registrator, Docker swarm, Nomad and others.

Dynamic LB Config with consul-template

Setting up HAProxy, wecna use this sample config file

global
  maxconn 4096

defaults
  mode http
  timeout connect 5s
  timeout client 50s
  timeout server 50s

listen http-in
  bind *:80
  server web 1 172.20.20.21:8080
  server web 3 172.20.20.22:8080
  server web 3 172.20.20.23:8080

and use the following script to create a HAProxy container that uses the confifuration file we just created above

#!/usr/bin/env bash

cp /vagrant/provision/haproxy.cfg /home/vagrant/.

# Run haproxy in a docker container
# Mount the haproxy config file
docker run -d \
           --name haproxy \
           -p 80:80 \
           --restart unless-stopped \
           -v /home/vagrant/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \
           haproxy:1.7-alpine

Now lets run this from the LB machine

vagrant ssh lb
chmod +x /vagrant/provision/setup.lb.sh
/vagrant/provision/setup.lb.sh

Now the LB service is passing the health check. If you want to see the LB in action, just go to the following URL and you'll see how the LB cycles through all nginx servers (1, 2, 3).

http://172.20.20.21/ip.html

What happens when one of the nginx servers fails? we need to get that server out or circualtion, meaning, we need to change the configuration of the LB so it doesn't use the failing server. In ordet to dynamically change its configuration, we can use a tool called Consul Template to react to changes in Consul and generate a new config file for HAProxy.

Using the following Consul template for HAProxy we can re-generate the HAProxy config file based on changes on Consul.

global
  maxconn 4096

defaults
  mode http
  timeout connect 5s
  timeout client 50s
  timeout server 50s

listen http-in
  bind *:80{{ range service "web" }}
server {{ .Node }} {{ .Address }}:{{ .Port }}{{ end }}

Testing the consul template file in DRY mode

consul-template -template /vagrant/provision/haproxy.ctmpl -dry

<-- Output -->
global
  maxconn 4096

defaults
  mode http
  timeout connect 5s
  timeout client 50s
  timeout server 50s

listen http-in
  bind *:80
server web-server-1 172.20.20.21:8080
server web-server-2 172.20.20.22:8080
server web-server-3 172.20.20.23:8080

Let's start Consul-template using the following config file so evertyime there is a change in Consul, it automatically regenarates the HAProxy config file and execute the command to restart HAProxy so it takes the changes in the configuration file. lb.consul-template.hcl

template {
  // This is the source file on disk to use as the input template. This is often
  // called the "Consul Template template". This option is required.
  source = "/home/vagrant/haproxy.ctmpl"

  // This is the destination path on disk where the source template will render.
  // If the parent directories do not exist, Consul Template will attempt to
  // create them.
  destination = "/home/vagrant/haproxy.cfg"

  // This is the optional command to run when the template is rendered. The
  // command will only run if the resulting template changes. The command must
  // return within 30s (configurable), and it must have a successful exit code.
  // Consul Template is not a replacement for a process monitor or init system.
  command = "docker restart haproxy"
}

and start Consul-template with the following command:

consul-template -config /vagrant/provision/lb.consul-template.hcl

If you SSH into web-server-3 and kill nginx, Consul-template should generate a new HAProxy configuration so the web-server-3 instance is not used to send traffic to.

vagrant ssh web-server-3
docker stop web

In a browser, open the following URL and you'll see web-server-3 was taken out of service in HAProxy automatically, no requests are sent to that server and no requests are failing.

http://172.20.20.11/ip.html

If you start the nginx service again in web-server-3, you'll see HAProxy will start sending requests to that server almost immediately.

If you want make changes to the Consul-template template file, you can send a HUP signal to the Consul-template process to reload its template configuration

killall -s HUP consul-template

Rolling updates with Consul maintenance mode

consul maint -enable -reason <enter the reason why this node is in maint node for others to see>
<update the application to the next version or any other update to the server>
consul maint -disable

Reactive configuration via Key/value Store

Consul provides an easy to use KV store. Here is the official documentation and also provides HTTP API endpoints to manage the KV store.

You can use the KV store to manage settings and then make your apps react to changes to its settings. All in one place, Consul.

Regenarating HAPoxy configuration file. You need to edit the HAProxy template file so Consul-template replaces the keys with the values store in Consul. e.g "prod/portal/haproxy/maxconn"

global
    maxconn {{key "prod/portal/haproxy/maxconn"}}

defaults
    mode http
    timeout connect {{key "prod/portal/haproxy/timeout-connect"}}
    timeout client {{key "prod/portal/haproxy/timeout-client"}}
    timeout server {{key "prod/portal/haproxy/timeout-server"}}

You can create the KV pair using the Consul UI, Consul CLI or using the Consul KV API.

vagrant ssh consul-server-1
consul kv put prod/portal/haproxy/maxconn 4096
consul kv put prod/portal/haproxy/timeout-connect 5s
consul kv put prod/portal/haproxy/timeout-client 50s
consul kv put prod/portal/haproxy/timeout-server 50s
consul kv put prod/portal/haproxy/stats enable

Since we changed the HAProxy configiration template file we need to ask Consul-template to reload its configration

killall -s HUP consul-template

Once this is complete, any change we make to the keys will re-generate the HAProxy configuration file and restart the service. This is way better than changing settings manually.

Health Checking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment