CoreOS on Digital Ocean using Terraform

README.md

Terraform, CoreOS, and Digital Ocean

Let's use Terraform to easily get a CoreOS cluster up on Digital Ocean. In this example we will get a 5 node CoreOS cluster up and running on the Digital Ocean 8GB size.

Install Terraform

Grab a copy of Terraform for your platform from http://www.terraform.io/downloads.html. Follow the instructions at http://www.terraform.io/intro/getting-started/install.html by getting Terraform in your PATH and testing that it works.

Digital Ocean API Key

You will need your Digital Ocean API key handy. Head over to https://cloud.digitalocean.com/settings/applications if you do not yet have one. Generate a new Personal Access Token for both read and write. Export the token as an environment variable so we can use it later on:

$ export DO_TOKEN=a7b9f84e6e6b3b6835c48873a8b144c98ae5b7105c05caefd174dc578d21d389

SSH Key

Terraform provisioners currently only work with passwordless private SSH keys. We will need to generate an SSH key without a password:

$ ssh-keygen -q -t rsa -f ~/.ssh/coreos_digitalocean -N '' -C coreos_digitalocean
Generating public/private rsa key pair.
Your identification has been saved in /Users/andy/.ssh/coreos_digitalocean.
Your public key has been saved in /Users/andy/.ssh/coreos_digitalocean.pub.
The key fingerprint is:
82:3d:e4:a1:b7:a4:ac:bc:98:b7:7f:0c:f9:50:a0:c8 coreos_digitalocean
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|    .            |
|.. . .o          |
|.E.  *..         |
|    ooB S        |
|   .++ +         |
|    o=.          |
| +..  +          |
|   ¯\_(ツ)_/¯    |
+-----------------+

Add this private SSH key to your agent so we can later SSH to a CoreOS node using it:

$ ssh-add ~/.ssh/coreos_digitalocean
Identity added: /Users/andy/.ssh/coreos_digitalocean (/Users/andy/.ssh/coreos_digitalocean)

Add a new key at https://cloud.digitalocean.com/ssh_keys using the contents from ~/.ssh/coreos_digitalocean.pub. Then query the Digital Ocean API to get the ID for this new key:

$ curl -X GET "https://api.digitalocean.com/v2/account/keys" -H "Authorization: Bearer $DO_TOKEN"
{
  "ssh_keys":[
    {
      "id":234700,
      "fingerprint":"82:3d:e4:a1:b7:a4:ac:bc:98:b7:7f:0c:f9:50:a0:c8",
      "public_key":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwXXVkdOaMOf/6LgyR4BnaECs/ChgVbBFQT+wX2jPI57Z59fvWAV5X7XmDYMa6370yfAMMwslOlbuz5EI9xiRE1WxiCkqMQjMI3ottHiM7RYEr5GigKC6Lre9Vs2TIDB6LfGiSgR6PlbVq5zuPn50IK2w5emBXP2HljJLGNGgxE5PyGKPunmFJQfRLgevvgb0dVMrsVIx1b+jLMZtvZ5ziRj4imVdlNtr1WvOBMT7n34AA+B9p/kiOWNbRZ7AcsknliBKNFkJ4kMgF98TyHKayd9oDxZEPgtjHQ3hg4Z6YY3cFYDw63fuGwSpc+ou7XaEYlRqIu9680sBgLnTB+BJ coreos_digitalocean",
      "name":"coreos_digitalocean"
    }
  ],
  "meta":{
    "total":2
  }
}

Cool, our SSH key ID is 234700. We'll need that in a couple steps from now.

Cloud Config

Copy and save the cloud-config.yml from this gist to a new folder. Then grab a new discovery URL from https://discovery.etcd.io/new. Edit your cloud-config.yml to add the URL to the discovery: line (replacing <replace_with_your_discovery_url>). Save this file.

Terraform Plan

Copy the coreos_digitalocean.tf file from this gist to the same folder as your cloud-config.yml. If you want a smaller cluster (such as 3 nodes instead of 5) then feel free to remove output and resource sections accordingly.

Build

Time to apply the Terraform plan! Be sure and replace 234700 with the correct ID of your actual SSH key on Digital Ocean:

$ terraform apply -var do_token=$DO_TOKEN -var size=8gb -var ssh_keys=234700 -var key_file=$HOME/.ssh/coreos_digitalocean
digitalocean_droplet.coreos5: Creating...
  image:              "" => "ubuntu-14-04-x64"
  name:               "" => "coreos5"
  private_networking: "" => "true"
  region:             "" => "nyc2"
  size:               "" => "8gb"
  ssh_keys.#:         "" => "1"
  ssh_keys.0:         "" => "234700"
digitalocean_droplet.coreos4: Creating...
  image:              "" => "ubuntu-14-04-x64"
  name:               "" => "coreos4"
  private_networking: "" => "true"
  region:             "" => "nyc2"
  size:               "" => "8gb"
  ssh_keys.#:         "" => "1"
  ssh_keys.0:         "" => "234700"
digitalocean_droplet.coreos3: Creating...
  image:              "" => "ubuntu-14-04-x64"
  name:               "" => "coreos3"
  private_networking: "" => "true"
  region:             "" => "nyc2"
  size:               "" => "8gb"
  ssh_keys.#:         "" => "1"
  ssh_keys.0:         "" => "234700"
digitalocean_droplet.coreos2: Creating...
  image:              "" => "ubuntu-14-04-x64"
  name:               "" => "coreos2"
  private_networking: "" => "true"
  region:             "" => "nyc2"
  size:               "" => "8gb"
  ssh_keys.#:         "" => "1"
  ssh_keys.0:         "" => "234700"
digitalocean_droplet.coreos1: Creating...
  image:              "" => "ubuntu-14-04-x64"
  name:               "" => "coreos1"
  private_networking: "" => "true"
  region:             "" => "nyc2"
  size:               "" => "8gb"
  ssh_keys.#:         "" => "1"
  ssh_keys.0:         "" => "234700"
digitalocean_droplet.coreos1: Provisioning with 'file'...
digitalocean_droplet.coreos2: Provisioning with 'file'...
digitalocean_droplet.coreos4: Provisioning with 'file'...
digitalocean_droplet.coreos5: Provisioning with 'file'...
digitalocean_droplet.coreos3: Provisioning with 'file'...
digitalocean_droplet.coreos1: Provisioning with 'remote-exec'...
digitalocean_droplet.coreos2: Provisioning with 'remote-exec'...
digitalocean_droplet.coreos3: Provisioning with 'remote-exec'...
digitalocean_droplet.coreos4: Provisioning with 'remote-exec'...
digitalocean_droplet.coreos1: Creation complete
digitalocean_droplet.coreos5: Provisioning with 'remote-exec'...
digitalocean_droplet.coreos3: Creation complete
digitalocean_droplet.coreos4: Creation complete
digitalocean_droplet.coreos5: Creation complete
digitalocean_droplet.coreos2: Creation complete

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

  address_coreos1 = 104.131.194.111
  address_coreos2 = 104.131.194.114
  address_coreos3 = 104.131.194.32
  address_coreos4 = 104.131.194.115
  address_coreos5 = 104.131.194.112

Give the provisioning about 5 minutes to complete. The CoreOS bootstrap process will reboot the node twice. You may be prompted for a password when trying to SSH. This usually just means the process has not completed yet. Give it some time!

Test

After your patience has finally run dry, give fleet a try using one of the node IP addresses:

$ export FLEETCTL_TUNNEL=104.131.194.111
$ fleetctl list-machines
MACHINE		IP		METADATA
b9770473...	104.131.194.113	-
f3cf244b...	104.131.194.32	-
f4d2c4cf...	104.131.194.115	-
f5bcd2a1...	104.131.194.112	-
f61ee1f5...	104.131.194.114	-

Success! Enjoy your CoreOS cluster on Digital Ocean.

Issues

Some issues I ran into. You could too!

Known Hosts

If you SSH to a node before the bootstrap process is complete you will get the server identity added to your ~/.ssh/known_hosts file. When you try to SSH to the finished CoreOS node, the SSH host key will have changed and, you will be prompted with an error similar to:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
31:33:fd:0b:92:1e:d0:59:24:f7:c0:e7:34:a1:4b:35.
Please contact your system administrator.
Add correct host key in /Users/andy/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/andy/.ssh/known_hosts:278
RSA host key for 104.131.194.112 has changed and you have requested strict checking.
Host key verification failed.

You will need to edit your ~/.ssh/known_hosts file to remove the offending line (278 in this case) to fix the issue.

Error Applying Plan

You could get a Terraform error like:

Error applying plan:

1 error(s) occurred:

* Script exited with non-zero exit status: 127

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

This usually means one or more nodes failed to run the provisioning script (curl couldn't download file, cloud-config.yml didn't get copied properly, etc.). If this happens, you can usually just run the terraform apply command again to complete the failed nodes.

cloud-config.yml

#cloud-config

coreos:
  etcd:
    # Get a new discovery URL from https://discovery.etcd.io/new
    discovery: <replace_with_your_discovery_url>
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

coreos_digitalocean.tf

variable "do_token" {}
variable "ssh_keys" {}
variable "key_file" {}
variable "size" {
  default = "4gb"
}

provider "digitalocean" {
  token = "${var.do_token}"
}

resource "digitalocean_droplet" "coreos1" {
  name = "coreos1"
  image = "ubuntu-14-04-x64"
  private_networking = true
  region = "nyc2"
  size = "${var.size}"
  ssh_keys = ["${var.ssh_keys}"]

  provisioner "file" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    source = "cloud-config.yml"
    destination = "/root/cloud-config.yml"
  }

  provisioner "remote-exec" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    inline = [
      "curl -sL -O https://raw.githubusercontent.com/ibuildthecloud/coreos-on-do/master/coreos-on-do.sh",
      "bash -l -i coreos-on-do.sh -V current -C alpha -c /root/cloud-config.yml"
    ]
  }
}

resource "digitalocean_droplet" "coreos2" {
  name = "coreos2"
  image = "ubuntu-14-04-x64"
  private_networking = true
  region = "nyc2"
  size = "${var.size}"
  ssh_keys = ["${var.ssh_keys}"]

  provisioner "file" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    source = "cloud-config.yml"
    destination = "/root/cloud-config.yml"
  }

  provisioner "remote-exec" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    inline = [
      "curl -sL -O https://raw.githubusercontent.com/ibuildthecloud/coreos-on-do/master/coreos-on-do.sh",
      "bash -l -i coreos-on-do.sh -V current -C alpha -c /root/cloud-config.yml"
    ]
  }
}

resource "digitalocean_droplet" "coreos3" {
  name = "coreos3"
  image = "ubuntu-14-04-x64"
  private_networking = true
  region = "nyc2"
  size = "${var.size}"
  ssh_keys = ["${var.ssh_keys}"]

  provisioner "file" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    source = "cloud-config.yml"
    destination = "/root/cloud-config.yml"
  }

  provisioner "remote-exec" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    inline = [
      "curl -sL -O https://raw.githubusercontent.com/ibuildthecloud/coreos-on-do/master/coreos-on-do.sh",
      "bash -l -i coreos-on-do.sh -V current -C alpha -c /root/cloud-config.yml"
    ]
  }
}

resource "digitalocean_droplet" "coreos4" {
  name = "coreos4"
  image = "ubuntu-14-04-x64"
  private_networking = true
  region = "nyc2"
  size = "${var.size}"
  ssh_keys = ["${var.ssh_keys}"]

  provisioner "file" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    source = "cloud-config.yml"
    destination = "/root/cloud-config.yml"
  }

  provisioner "remote-exec" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    inline = [
      "curl -sL -O https://raw.githubusercontent.com/ibuildthecloud/coreos-on-do/master/coreos-on-do.sh",
      "bash -l -i coreos-on-do.sh -V current -C alpha -c /root/cloud-config.yml"
    ]
  }
}

resource "digitalocean_droplet" "coreos5" {
  name = "coreos5"
  image = "ubuntu-14-04-x64"
  private_networking = true
  region = "nyc2"
  size = "${var.size}"
  ssh_keys = ["${var.ssh_keys}"]

  provisioner "file" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    source = "cloud-config.yml"
    destination = "/root/cloud-config.yml"
  }

  provisioner "remote-exec" {
    connection {
      user = "root"
      key_file = "${var.key_file}"
    }

    inline = [
      "curl -sL -O https://raw.githubusercontent.com/ibuildthecloud/coreos-on-do/master/coreos-on-do.sh",
      "bash -l -i coreos-on-do.sh -V current -C alpha -c /root/cloud-config.yml"
    ]
  }
}

output "address_coreos1" {
  value = "${digitalocean_droplet.coreos1.ipv4_address}"
}

output "address_coreos2" {
  value = "${digitalocean_droplet.coreos2.ipv4_address}"
}

output "address_coreos3" {
  value = "${digitalocean_droplet.coreos3.ipv4_address}"
}

output "address_coreos4" {
  value = "${digitalocean_droplet.coreos4.ipv4_address}"
}

output "address_coreos5" {
  value = "${digitalocean_droplet.coreos5.ipv4_address}"
}