README.md

Running Kubernetes Example on CoreOS, Part 2

• https://coreos.com/blog/running-kubernetes-example-on-CoreOS-part-2/
• https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/guestbook/README.md

Step Zero: Build up a Kubernetes cluster

$ git clone https://gist.github.com/9d74dc702c49d41035f7.git
$ vagrant up

It will boot up one for etcd discovery, one master and two minion servers (minion-x).

Setup an SSH tunnel

Setup an SSH tunnel to the Kubernetes API Server in order to use kubecfg on your local machine.

$ curl -OL http://storage.googleapis.com/kubernetes/darwin/kubecfg
$ chmod +x kubecfg
$ mv kubecfg /usr/local/bin

$ vagrant ssh-config master > ssh.config
$ ssh -f -nNT -L 8080:127.0.0.1:8080 -F ssh.config master
$ kubecfg list pods
Name                Image(s)            Host                Labels
----------          ----------          ----------          ----------

Cf.) https://github.com/kelseyhightower/kubernetes-coreos#running-commands-remotely

Step One: Turn up the redis master

$ kubecfg -c redis-master.json create pods
I0731 16:18:06.205838 13060 request.go:220] Waiting for completion of /operations/1
I0731 16:18:26.208799 13060 request.go:220] Waiting for completion of /operations/1
I0731 16:18:46.212773 13060 request.go:220] Waiting for completion of /operations/1
.
.
.
Name                Image(s)            Host                Labels
----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    /                   name=redis-master

$ kubecfg list pods
Name                Image(s)            Host                Labels
----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    192.168.12.11/      name=redis-master

$ vagrant ssh minion-1 -c "docker ps"
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                    NAMES
2d40ca4bff5f        dockerfile/redis:latest     redis-server /etc/re   47 seconds ago      Up 47 seconds                                k8s--master--redis_-_master_-_2--ec662fc1
425efffe8238        busybox:buildroot-2014.02   sh -c 'rm -f nap &&    5 minutes ago       Up 5 minutes        0.0.0.0:6379->6379/tcp   k8s--net--redis_-_master_-_2--882f503e

Step Two: Turn up the master service

$ kubecfg -c redis-master-service.json create services
I0731 16:31:24.530943 13428 request.go:220] Waiting for completion of /operations/2
Name                Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

$ kubecfg list services
Name                Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

Step Three: Turn up the replicated slave pods

$ kubecfg -c redis-slave-controller.json create replicationControllers
I0731 16:41:07.076419 13563 request.go:220] Waiting for completion of /operations/3
I0731 16:41:27.080095 13563 request.go:220] Waiting for completion of /operations/3
I0731 16:41:47.083393 13563 request.go:220] Waiting for completion of /operations/3
.
.
.
Name                   Image(s)                   Selector            Replicas
----------             ----------                 ----------          ----------
redisSlaveController   brendanburns/redis-slave   name=redisslave     2

$ kubecfg list pods
Name                                   Image(s)                   Host                Labels
----------                             ----------                 ----------          ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master
9053847d-1914-11e4-9ad9-080027558144   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController
a82d9013-1914-11e4-9ad9-080027558144   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController

Step Four: Create the redis slave service

$ kubecfg -c redis-slave-service.json create services
I0731 16:49:09.808457 13880 request.go:220] Waiting for completion of /operations/6
Name                Labels              Selector            Port
----------          ----------          ----------          ----------
redisslave          name=redisslave     name=redisslave     10001

$ kubecfg list services
Name                Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000
redisslave          name=redisslave     name=redisslave     10001

Step Five: Create the frontend pod

$ kubecfg -c frontend-controller.json create replicationControllers
I0731 16:51:36.054889 13946 request.go:220] Waiting for completion of /operations/7
I0731 16:51:56.059760 13946 request.go:220] Waiting for completion of /operations/7
I0731 16:52:16.063003 13946 request.go:220] Waiting for completion of /operations/7
.
.
.
Name                 Image(s)                 Selector            Replicas
----------           ----------               ----------          ----------
frontendController   brendanburns/php-redis   name=frontend       2

$ kubecfg list pods
Name                                   Image(s)                   Host                Labels
----------                             ----------                 ----------          ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master
9053847d-1914-11e4-9ad9-080027558144   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController
4b1165ed-1916-11e4-9ad9-080027558144   brendanburns/php-redis     192.168.12.11/      name=frontend,replicationController=frontendController
a82d9013-1914-11e4-9ad9-080027558144   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController
b0129fd6-1915-11e4-9ad9-080027558144   brendanburns/php-redis     192.168.12.12/      name=frontend,replicationController=frontendController

$ open http://192.168.12.11:8000
$ open http://192.168.12.12:8000

discovery.yml

#cloud-config

coreos:
  etcd:
      addr: $public_ipv4:4001
      peer-addr: $public_ipv4:7001

  units:
    - name: etcd.service
      command: start

frontend-controller.json

{
  "id": "frontendController",
  "kind": "ReplicationController",
  "apiVersion": "v1beta1",
  "desiredState": {
    "replicas": 2,
    "replicaSelector": {"name": "frontend"},
    "podTemplate": {
      "desiredState": {
         "manifest": {
           "version": "v1beta1",
           "id": "frontendController",
           "containers": [{
             "name": "php-redis",
             "image": "brendanburns/php-redis",
             "ports": [{"containerPort": 80, "hostPort": 8000}]
           }]
         }
       },
       "labels": {"name": "frontend"}
      }},
  "labels": {"name": "frontend"}
}

master.yml

#cloud-config

coreos:
  etcd:
      discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
      addr: $public_ipv4:4001
      peer-addr: $public_ipv4:7001

  units:
    - name: cbr0.netdev
      command: start
      content: |
        [NetDev]
        Kind=bridge
        Name=cbr0

    - name: cbr0.network
      command: start
      content: |
        [Match]
        Name=cbr0

        [Network]
        Address=%BRIDGE_IP_ADDR%.1/24

        [Route]
        #Destination=10.0.0.0/8
        #Gateway=0.0.0.0

    - name: cbr0-interface.network
      command: start
      content: |
        [Match]
        Name=ens34

        [Network]
        Bridge=cbr0

    - name: nat.service
      command: start
      content: |
        [Unit]
        Description=NAT non container traffic

        [Service]
        ExecStartPre=/usr/sbin/iptables -t nat -F
        ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o enp0s8 -j MASQUERADE ! -d %BRIDGE_IP_ADDR%.0/24
        RemainAfterExit=yes
        Type=oneshot

    - name: docker.service
      command: restart
      content: |
        [Unit]
        After=network.target
        Description=Docker Application Container Engine
        Documentation=http://docs.docker.io

        [Service]
        ExecStartPre=-/usr/bin/ip link set dev docker0 down
        ExecStartPre=-/usr/sbin/brctl delbr docker0
        ExecStartPre=/bin/mount --make-rprivate /
        ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// -b cbr0 --iptables=false

        [Install]
        WantedBy=multi-user.target

    - name: etcd.service
      command: start

    - name: download-kubernetes.service
      command: start
      content: |
        [Unit]
        After=network-online.target
        Before=apiserver.service
        Before=controller-manager.service
        Description=Download Kubernetes Binaries
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=network-online.target

        [Service]
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/apiserver
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/controller-manager
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubecfg
        ExecStart=/usr/bin/chmod +x /opt/bin/apiserver
        ExecStart=/usr/bin/chmod +x /opt/bin/controller-manager
        ExecStart=/usr/bin/chmod +x /opt/bin/kubecfg
        RemainAfterExit=yes
        Type=oneshot

    - name: apiserver.service
      command: start
      content: |
        [Unit]
        ConditionFileIsExecutable=/opt/bin/apiserver
        Description=Kubernetes API Server
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes

        [Service]
        ExecStart=/opt/bin/apiserver \
        --address=127.0.0.1 \
        --port=8080 \
        --etcd_servers=http://127.0.0.1:4001 \
        --machines=%MINION_IP_ADDRS% \
        --logtostderr=true
        Restart=always
        RestartSec=2

        [Install]
        WantedBy=multi-user.target

    - name: controller-manager.service
      command: start
      content: |
        [Unit]
        ConditionFileIsExecutable=/opt/bin/controller-manager
        Description=Kubernetes Controller Manager
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes

        [Service]
        ExecStart=/opt/bin/controller-manager \
        -master=127.0.0.1:8080 \
        -logtostderr=true
        Restart=always
        RestartSec=2

        [Install]
        WantedBy=multi-user.target

minion.yml

#cloud-config

coreos:
  etcd:
      discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
      addr: $public_ipv4:4001
      peer-addr: $public_ipv4:7001

  units:
    - name: cbr0.netdev
      command: start
      content: |
        [NetDev]
        Kind=bridge
        Name=cbr0

    - name: cbr0.network
      command: start
      content: |
        [Match]
        Name=cbr0

        [Network]
        Address=%BRIDGE_IP_ADDR%.1/24

        [Route]
        #Destination=10.0.0.0/8
        #Gateway=0.0.0.0

    - name: cbr0-interface.network
      command: start
      content: |
        [Match]
        Name=ens34

        [Network]
        Bridge=cbr0

    - name: nat.service
      command: start
      content: |
        [Unit]
        Description=NAT non container traffic

        [Service]
        ExecStartPre=/usr/sbin/iptables -t nat -F
        ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o enp0s8 -j MASQUERADE ! -d %BRIDGE_IP_ADDR%.0/24
        RemainAfterExit=yes
        Type=oneshot

    - name: docker.service
      command: restart
      content: |
        [Unit]
        After=network.target
        Description=Docker Application Container Engine
        Documentation=http://docs.docker.io

        [Service]
        ExecStartPre=-/usr/bin/ip link set dev docker0 down
        ExecStartPre=-/usr/sbin/brctl delbr docker0
        ExecStartPre=/bin/mount --make-rprivate /
        ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// -b cbr0 --iptables=false

        [Install]
        WantedBy=multi-user.target

    - name: etcd.service
      command: start

    - name: fleet.service
      command: start
      runtime: no
      content: |
        [Unit]
        Description=fleet

        [Service]
        Environment=FLEET_PUBLIC_IP=$public_ipv4
        ExecStart=/usr/bin/fleet

    - name: download-kubernetes.service
      command: start
      content: |
        [Unit]
        After=network-online.target
        Before=kubelet.service
        Before=proxy.service
        Description=Download Kubernetes Binaries
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=network-online.target

        [Service]
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubelet
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/proxy
        ExecStart=/usr/bin/chmod +x /opt/bin/kubelet
        ExecStart=/usr/bin/chmod +x /opt/bin/proxy
        RemainAfterExit=yes
        Type=oneshot

    - name: kubelet.service
      command: start
      content: |
        [Unit]
        ConditionFileIsExecutable=/opt/bin/kubelet
        Description=Kubernetes Kubelet
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes

        [Service]
        ExecStart=/opt/bin/kubelet \
        --address=0.0.0.0 \
        --port=10250 \
        --hostname_override=$public_ipv4 \
        --etcd_servers=http://127.0.0.1:4001 \
        --logtostderr=true
        Restart=always
        RestartSec=2

        [Install]
        WantedBy=multi-user.target

    - name: proxy.service
      command: start
      content: |
        [Unit]
        ConditionFileIsExecutable=/opt/bin/proxy
        Description=Kubernetes Proxy
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes

        [Service]
        ExecStart=/opt/bin/proxy --etcd_servers=http://127.0.0.1:4001 --logtostderr=true
        Restart=always
        RestartSec=2

        [Install]
        WantedBy=multi-user.target

redis-master-service.json

{
  "id": "redismaster",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 10000,
  "selector": {
    "name": "redis-master"
  }
}

redis-master.json

{
  "id": "redis-master-2",
  "kind": "Pod",
  "apiVersion": "v1beta1",
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "redis-master-2",
      "containers": [{
        "name": "master",
        "image": "dockerfile/redis",
        "ports": [{
          "containerPort": 6379,
          "hostPort": 6379
        }]
      }]
    }
  },
  "labels": {
    "name": "redis-master"
  }
}

redis-slave-controller.json

{
  "id": "redisSlaveController",
  "kind": "ReplicationController",
  "apiVersion": "v1beta1",
  "desiredState": {
    "replicas": 2,
    "replicaSelector": {"name": "redisslave"},
    "podTemplate": {
      "desiredState": {
         "manifest": {
           "version": "v1beta1",
           "id": "redisSlaveController",
           "containers": [{
             "name": "slave",
             "image": "brendanburns/redis-slave",
             "ports": [{"containerPort": 6379, "hostPort": 6380}]
           }]
         }
       },
       "labels": {"name": "redisslave"}
      }},
  "labels": {"name": "redisslave"}
}

redis-slave-service.json

{
  "id": "redisslave",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 10001,
  "labels": {
    "name": "redisslave"
  },
  "selector": {
    "name": "redisslave"
  }
}

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

NUM_MINIONS = 2

BASE_IP_ADDR    = ENV['BASE_IP_ADDR'] || "192.168.12"
ETCD_DISCVERY   = "#{BASE_IP_ADDR}.101"
MASTER_IP_ADDR  = "#{BASE_IP_ADDR}.10"
MINION_IP_ADDRS = NUM_MINIONS.times.collect { |i| BASE_IP_ADDR + ".#{i+11}" }
BRIDGE_IP_ADDRS = (NUM_MINIONS + 1).times.collect { |i| "10.244.#{i}" }

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "yungsang/coreos"

  config.vm.define "discovery" do |discovery|
    discovery.vm.hostname = "discovery"

    discovery.vm.network :private_network, ip: ETCD_DISCVERY

    discovery.vm.provision :file, source: "./discovery.yml", destination: "/tmp/vagrantfile-user-data"

    discovery.vm.provision :shell do |sh|
      sh.privileged = true
      sh.inline = <<-EOT
        mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
      EOT
    end
  end

  config.vm.define "master" do |master|
    master.vm.hostname = "master"

    master.vm.network :forwarded_port, guest: 4001, host: 4001

    master.vm.network :private_network, ip: MASTER_IP_ADDR

    master.vm.provision :file, source: "./master.yml", destination: "/tmp/vagrantfile-user-data"

    master.vm.provision :shell do |sh|
      sh.privileged = true
      sh.inline = <<-EOT
        sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data
        sed -e "s/%BRIDGE_IP_ADDR%/#{BRIDGE_IP_ADDRS[0]}/g" -i /tmp/vagrantfile-user-data
        sed -e "s/%MINION_IP_ADDRS%/#{MINION_IP_ADDRS.join(',')}/g" -i /tmp/vagrantfile-user-data

        mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
      EOT
    end
  end

  NUM_MINIONS.times do |i|
    config.vm.define "minion-#{i+1}" do |minion|
      minion.vm.hostname = "minion-#{i+1}"

      minion.vm.network :private_network, ip: MINION_IP_ADDRS[i]

      minion.vm.provision :file, source: "./minion.yml", destination: "/tmp/vagrantfile-user-data"

      minion.vm.provision :shell do |sh|
        sh.privileged = true
        sh.inline = <<-EOT
          sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data
          sed -e "s/%BRIDGE_IP_ADDR%/#{BRIDGE_IP_ADDRS[i+1]}/g" -i /tmp/vagrantfile-user-data

          mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
        EOT
      end
    end
  end
end