Simple multi cloud Docker cluster using Docker Swarm

This guide explains how to create a simple multi cloud Docker cluster using Docker Swarm, Docker Machine and the three top cloud providers nowadays - Google Compute Engine, Microsoft Azure and AWS.

Prerequisites

This guide assumes you have a Linux host with Docker CE installed. If you are using Docker for Mac or Docker for Windows you can avoid the Docker Machine set up since it comes included.

Install Docker Machine

The nodes provisioning will be done using Docker Machine. Follow the official guide to install it.

Nodes provisioning

Create instance in GCE - Google Compute Engine

It assumes you have already an GCE account with at least one project.

Install Google Cloud SDK

Follow https://cloud.google.com/sdk/docs/quickstarts.

Configure credentials

Go to https://console.cloud.google.com/apis/credentials/serviceaccountkey
In Service Account choose Compute Engine...
In Key type keep JSON
Click in Create to download

After download the JSON file, put it in a location and export it as follow:

export GOOGLE_APPLICATION_CREDENTIALS=<CREDENTIALS_ABSOLUTE_PATH>

Create the instance

Assuming we want to create a Debian 8 based instance:

docker-machine create --driver google \
    --google-project <PROJECT_ID> \
    --google-machine-type f1-micro \
    --google-zone europe-west1-b \
    --google-machine-image https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-8-jessie-v20180510 \
    gce1

Docker Machine will create the remote instance in GCE and provision it with Docker Engine.

After the execution docker-machine ls will show the new remote host:

NAME   ACTIVE   DRIVER   STATE     URL                        SWARM   DOCKER        ERRORS
gce1   -        google   Running   tcp://35.189.XX.XXX:2376           v18.05.0-ce

Notes

GCE allows to sign up with prepaid credit card
$300 free credit for 12 first months
f1-micro single instance is free forever
Docker Machine driver reference for GCE https://docs.docker.com/machine/drivers/gce/
The Docker Machine driver does not work with the configuration and authentication created with the GCE SDK CLI tool
To check the list of images gcloud compute images list --uri
To check the list of zones gcloud compute zones list
The full list of zones with hardware details can be found here https://cloud.google.com/compute/docs/regions-zones/
The zone europe-west1-b has the most powerful CPU in Europe nowadays
Complex UI but intuitive CLI SDK

Create instance in Microsoft Azure

Install Azure CLI

Follow https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest.

Create the instance

Assuming we want to create a Debian 8 based instance:

docker-machine create --driver azure \
    --azure-subscription-id <SUBSCRIPTION_ID> \
    --azure-size Standard_B1s \
    --azure-location westeurope \
    --azure-image credativ:Debian:8:latest \
    ma1

After the execution docker-machine ls will show the new remote host:

NAME   ACTIVE   DRIVER   STATE     URL                        SWARM   DOCKER        ERRORS
gce1   -        google   Running   tcp://35.189.XXX.XX:2376           v18.05.0-ce   
ma1    -        azure    Running   tcp://23.97.XXX.XXX:2376           v18.05.0-ce

Notes

Multiple pages broken using Firefox forcing me to use Chrome to continue the sign up... Others broken with both...
$200 free credit for 30 first days
Standard_B1s instance is 12 months free (750 hours/month)
Docker Machine driver reference for Microsoft Azure https://docs.docker.com/machine/drivers/azure/
To check the list of images az vm image list --output table
To check the list of locations az account list-locations --output table
The full list of Linux VM instances https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-general
Not complex UI but not very intuitive CLI SDK

Create instance in AWS - Amazon Web Services

Install AWS CLI

Follow https://docs.aws.amazon.com/cli/latest/userguide/installing.html.

Configure credentials

Go to https://console.aws.amazon.com/iam/home?#/security_credential and create a new Access keys if you do not have one already
Execute aws configure and fill the steps

Create the instance

Assuming we want to create a Debian 8 based instance:

docker-machine create --driver amazonec2 \
    --amazonec2-instance-type t2.micro \
    --amazonec2-region eu-central-1 \
    --amazonec2-ami ami-5900cc36 \
    --amazonec2-device-name /dev/xvda \
    --amazonec2-ssh-user admin \
    aws1

After the execution docker-machine ls will show the new remote host:

NAME   ACTIVE   DRIVER      STATE     URL                        SWARM   DOCKER        ERRORS
aws1   -        amazonec2   Running   tcp://35.157.XX.XX:2376            v18.05.0-ce   
gce1   -        google      Running   tcp://35.189.XXX.XX:2376           v18.05.0-ce   
ma1    -        azure       Running   tcp://23.97.XXX.XXX:2376           v18.05.0-ce

Notes

t2.micro instance is 12 month free (750 hours/month)
Docker Machine driver reference for AWS https://docs.docker.com/machine/drivers/aws/
To check the list of regions aws ec2 describe-regions
To check the official list of Debian Jessie images aws ec2 describe-images --owners 379101102735 --filters "Name=name,Values=debian-jessie-*"
More info about official Debian Jessie AMIs https://wiki.debian.org/Cloud/AmazonEC2Image/Jessie
High number of AMIs and its custom configuration makes the process of choose and create instances complex

Creating the cluster

After the nodes provisioning we can proceed with the creation of the Docker Swarm.

Create the manager

Lets use the GCE instance as manager:

SSH to the GCE instance docker-machine ssh gce1
Make this node as manager in the Docker Swarm sudo docker swarm init --advertise-addr <THIS_HOST_IP>

The output of the last command should be something like this:

Swarm initialized: current node (jycadm63k7m5162bxvkqlgsem) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-2aon4rvr50khj9cb7w1t80610gxxpl9v8gkc7n7qf8yxj03prc-1ozrrrs676zgqb0vcvekkiyqk 35.189.xxx.xx:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

By default GCE only allows very specific incoming traffic so we have to proceed configuring the firewall as follow:

Allow cluster management communications

gcloud compute firewall-rules create docker-swarm-manager \
    --action allow \
    --source-ranges 0.0.0.0/0 \
    --rules tcp:2377

Allow communication among nodes

gcloud compute firewall-rules create docker-swarm-discovery \
    --action allow \
    --source-ranges 0.0.0.0/0 \
    --rules tcp:7946,udp:7946

Allow overlay network traffic

gcloud compute firewall-rules create docker-swarm-network \  
    --action allow \
    --source-ranges 0.0.0.0/0 \
    --rules udp:4789

More information related with the use of the firewall can be found in Using Firewall Rules.

Now we can exit from this host.

If you need to remain this output, execute docker swarm join-token worker from this manager host any time.

Add nodes to the manager

Azure

SSH to the Microsoft Azure instance docker-machine ssh ma1
Lets join this node to the Docker Swarm sudo docker swarm join --token SWMTKN-1-2aon4rvr50khj9cb7w1t80610gxxpl9v8gkc7n7qf8yxj03prc-1ozrrrs676zgqb0vcvekkiyqk 35.189.xxx.xx:2377

The output should be:

This node joined a swarm as a worker.

Exit from this host.

Now, from the manager host, if we execute docker node ls we should see something like this:

NAME   ACTIVE   DRIVER   STATE     URL                        SWARM   DOCKER        ERRORS
gce1   -        google   Running   tcp://35.189.xxx.xx:2376           v18.05.0-ce   
ma1    -        azure    Running   tcp://23.97.xxx.xxx:2376           v18.05.0-ce

Proceed allowing needed traffic in the firewall:

Allow communication among nodes

az network nsg rule create \
    --resource-group docker-machine \
    --nsg-name ma1-firewall \
    --name dockerSwarmDiscovery \
    --priority 301 \
    --access Allow \
    --protocol '*' \
    --destination-port-range 7946

Allow overlay network traffic

az network nsg rule create \
    --resource-group docker-machine \
    --nsg-name ma1-firewall \
    --name dockerSwarmNetwork \
    --priority 302 \
    --access Allow \
    --protocol udp \
    --destination-port-range 4789

More information about firewall management using az network nsg rule CLI command.

AWS

Same operation for the AWS instance:

docker-machine ssh aws1
sudo docker swarm join --token SWMTKN-1-2aon4rvr50khj9cb7w1t80610gxxpl9v8gkc7n7qf8yxj03prc-1ozrrrs676zgqb0vcvekkiyqk 35.189.XXX.XX:2377

And exit from this host.

Once again, from the manager host, docker node ls should show something like this:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
iir58ycxlfgxldfmyc4ztx8mf     aws1                Ready               Active                                  18.05.0-ce
jycadm63k7m5162bxvkqlfsem *   gce1                Ready               Active              Leader              18.05.0-ce
vur0qe657al0591uhyab4r6f4     ma1                 Ready               Active                                  18.05.0-ce

Proceed allowing needed traffic in the firewall:

Allow communication among nodes

aws ec2 authorize-security-group-ingress \
    --group-name docker-machine \
    --cidr 0.0.0.0/0 \
    --port 7946 \
    --protocol tcp

aws ec2 authorize-security-group-ingress \
    --group-name docker-machine \
    --cidr 0.0.0.0/0 \
    --port 7946 \
    --protocol udp

Allow overlay network traffic

aws ec2 authorize-security-group-ingress \
    --group-name docker-machine \
    --cidr 0.0.0.0/0 \
    --port 4789 \
    --protocol udp

More information related with the use of the firewall can be found in authorize-security-group-ingress.

Deploy services in the cluster

Kernel IP forwarding comes disabled by default in GCE, to enable it:

gcloud compute ssh gce1 \
    --command 'sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf; sudo sysctl -p'

SSH to the manager instance - GCE in our example docker-machine ssh gce1
Create the service:

sudo docker service create \
    --replicas 3 \
    --name webserver \
    --publish published=80,target=80 \
    nginx

Open port 80 in each host using its CLI:

GCE

gcloud compute firewall-rules create docker-swarm-webserver \
    --action allow \
    --source-ranges 0.0.0.0/0 \
    --rules tcp:80

Azure

az network nsg rule create \
    --resource-group docker-machine \
    --nsg-name ma1-firewall \
    --name dockerSwarmWebserver \
    --priority 1000 \
    --access Allow \
    --protocol tcp \
    --destination-port-range 80

AWS

aws ec2 authorize-security-group-ingress \
    --group-name docker-machine \
    --cidr 0.0.0.0/0 \
    --protocol tcp \
    --port 80

Now we have three Nginx instances running in our cluster where three can be increased or decreased on demand using docker service update.

Thanks to the routing mesh of Docker Swarm the service can be reach from any IP that belongs to a node in the cluster.

To get the all the powerful features of Docker Swarm is ideal to go with an immutable infrastructure.

Guide based on the Docker Swarm tutorial.

ssbrewster/multi-cloud-docker-swarm.md

Simple multi cloud Docker cluster using Docker Swarm

Prerequisites

Install Docker Machine

Nodes provisioning

Create instance in GCE - Google Compute Engine

Install Google Cloud SDK

Configure credentials

Create the instance

Create instance in Microsoft Azure

Install Azure CLI

Create the instance

Create instance in AWS - Amazon Web Services

Install AWS CLI

Configure credentials

Create the instance

Creating the cluster

Create the manager

Allow cluster management communications

Allow communication among nodes

Allow overlay network traffic

Add nodes to the manager

Azure

Allow communication among nodes

Allow overlay network traffic

AWS

Allow communication among nodes

Allow overlay network traffic

Deploy services in the cluster

GCE

Azure

AWS