apokalyptik/debian-jessie-kubernetes-1.3-manual-install-guide.md

The Goal

Set up Kubernetes on 3 Debian Jessie virtual machines: One master. Two nodes. Additionally do this without any "magic" so that what is required to be running to make everything work is plain and obvious.

We will be using flannel for the inter-machine networking layer. Mainly because it is useful and it seems to be pretty popular.

The Setup

The VMs will be net-installed debian 8.5.0 AMD64 They will be running inside virtualbox, and have the addresses:

kube-master.apokalyptik.lan 10.0.1.80
kube-node1.apokalyptik.lan 10.0.1.81
kube-node2.apokalyptik.lan 10.0.1.82

Virtualbox because I don't have a whole bunch of actual machines sitting around to do this on these days, and this emulates setting it up from scratch, by hand, on 3 machines anyhow.

The flannel network will be 192.168.1.0/16 (because the numbers are very different which makes it easy to see that it is, or isnt, working)

IMPORTANT THINGS YOU NEED TO READ

THIS IS IMPORTANT TOO

I am just figuring out how to get all of this up and running. There are security issues, and best practices that are not taken into account here. Some of these I know and skipped to make this document a good introduction for just how to get things working (example: you should have two etcd clusters running. one for flanneld and one for kubernetes and while all nodes and masters need to be able to read/write the flanneld etcd cluster only masters should be able to access the kubernetes etcd cluster at all). And probably a great many I don't, or don't yet know exactly how to address (example kubelet can probably run as the kube user instead of root with some permissions work? maybe?)

The point is that this document is a starting point for getting kubernetes up and running. It's a highly distilled crash course in just what has to be there for it to literally work at all. Kubernetes and the surrounding software ecosystem is big. REALLY BIG. and it's going to take a lot of time and work to understand it

Last note

I am not 100% sure of this. but if you wanted, for some reason, a single machine kubernetes setup I think you can probably simply replace the kubelet service file on the master with the one from the nodes.

Order of Operations

The Master:

Expectations:

The Master will need the following processes when all is said and done

Non-Kubernetes:

docker
etcd
flanneld

Kubernetes:

apiserver
controller-manager
kublet (in "master mode" (my words))
proxy
scheduler

Step 1: Get Ready

Updated grub (you did pay attention and read the things I told you to read, right?)
Reboot

echo '10.0.1.80 kube-master.apokalyptik.lan kube-master' >> /etc/hosts
echo '10.0.1.81 kube-node1.apokalyptik.lan kube-node1' >> /etc/hosts
echo '10.0.1.82 kube-node2.apokalyptik.lan kube-node2' >> /etc/hosts

apt-get -fqqy install apt-transport-https ca-certificates curl

Yes... we want a single 132M binary from inside a 352M tarball from inside another 1.4G tarball. Such is life. I expect you'll just get this file once and copy it around...

curl \
  --progress-bar \
  --location \
  'https://github.com/kubernetes/kubernetes/releases/download/v1.3.4/kubernetes.tar.gz' \
      | tar --to-stdout -zxf - kubernetes/server/kubernetes-server-linux-amd64.tar.gz \
      | tar --to-stdout -zxf - kubernetes/server/bin/hyperkube > /usr/bin/hyperkube 
chmod a+x /usr/bin/hyperkube
cd /usr/bin/
/usr/bin/hyperkube --make-symlinks

mkdir -p /var/lib/k8s groupadd kube useradd kube -g kube -d /var/lib/k8s/ -s /bin/false chown kube:kube /var/lib/k8s

Step 2: Install Docker

apt-key adv \
  --keyserver hkp://p80.pool.sks-keyservers.net:80 \
  --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo debian-jessie main' \
  > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get -fqqy install docker-engine

Step 3: Install ETCD

Give it a home

groupadd etcd
useradd etcd -d /var/lib/k8s/etcd -s /bin/false -g etcd
mkdir -p /var/lib/k8s/etcd
chown etcd:etcd /var/lib/k8s/etcd -R

Download it, and install it

cd /usr/local/src
curl \
	--silent \
	--location \
	'https://github.com/coreos/etcd/releases/download/v3.0.4/etcd-v3.0.4-linux-amd64.tar.gz' \
    | tar -zvxf-
cd etcd-v3.0.4-linux-amd64
cp etcd /usr/bin/
cp etcdctl /usr/bin/

Define it

cat << EOF > /lib/systemd/system/etcd.service
[Unit]
Description=etcd key-value store
Documentation=https://github.com/coreos/etcd

[Service]
User=etcd
Type=notify
ExecStart=/usr/bin/etcd \\
  --advertise-client-urls http://10.0.1.80:4001 \\
  --data-dir /var/lib/k8s/etcd \\
  --listen-client-urls http://10.0.1.80:4001,http://127.0.0.1:4001
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
EOF

# Start it

systemctl daemon-reload
systemctl enable etcd
service etcd start

Step 4: FlannelD

Pre-Configure FlannelD via ETCD

etcdctl set /coreos.com/network/config '{ "Network": "192.168.1.0/16" }'

Download it and install it

cd /usr/local/src
curl \
  --silent \
  --location \
  'https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz' \
    | tar -zvxf-
cd flannel-0.5.5
cp flanneld /usr/bin
mkdir -p /var/lib/k8s/flannel/networks

Define it

cat << EOF > /lib/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service

[Service]
Type=notify
Restart=always
RestartSec=5
ExecStart=/usr/bin/flanneld \\
  -etcd-endpoints=http://10.0.1.80:4001 \\
  -logtostderr=true \\
  -subnet-dir=/var/lib/k8s/flannel/networks \\
  -subnet-file=/var/lib/k8s/flannel/subnet.env
[Install]
WantedBy=multi-user.target
EOF

Start it

systemctl daemon-reload
systemctl enable flanneld
service flanneld start

Step 5: Reconfigure docker to use FlannelD

Redefine it

cat << EOF > /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket flanneld.service etcd.service
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=-/var/lib/k8s/flannel/subnet.env
ExecStart=/usr/bin/dockerd -H fd:// --bip=\${FLANNEL_SUBNET} --mtu=\${FLANNEL_MTU}
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
MountFlags=shared

[Install]
WantedBy=multi-user.target
EOF

Give kube access gpasswd -a kube docker

Reload it

systemctl daemon-reload
service docker restart

Step 6: Kubernetes API Server (finally, on to kubernetes!)

make a home

mkdir -p /var/lib/k8s/kubernetes/crt
chown kube:kube /var/lib/k8s/kubernetes/crt /var/lib/k8s/kubernetes

define a service account

if [[ ! -f /var/lib/k8s/kubernetes/kube-serviceaccount.key ]]; then
	openssl genrsa -out /var/lib/k8s/kubernetes/kube-serviceaccount.key 2048 2>/dev/null
fi
chown kube:kube /var/lib/k8s/kubernetes/kube-serviceaccount.key

Define it

cat << EOF > /lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target etcd.service flanneld.service

[Service]
EnvironmentFile=-/var/lib/k8s/flannel/subnet.env
User=kube
ExecStart=/usr/bin/apiserver \\
	--advertise_address=10.0.1.80 \\
	--cert-dir=/var/lib/k8s/kubernetes/crt \\
	--service-account-key-file=/var/lib/k8s/kubernetes/kube-serviceaccount.key \\
	--service-account-lookup=false \\
	--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota \\
	--bind-address=0.0.0.0 \\
	--insecure-bind-address=0.0.0.0 \\
	--insecure-port=8080 \\
	--etcd-servers=http://10.0.1.80:4001 \\
	--portal_net=\${FLANNEL_NETWORK} \\
	--logtostderr=true
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-apiserver
service kube-apiserver start

Step 7: Kubernetes controller-manager

Define it

cat << EOF > /lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
User=kube
ExecStart=/usr/bin/controller-manager \\
  --service-account-private-key-file=/var/lib/k8s/kubernetes/kube-serviceaccount.key \\
  --root-ca-file=/var/lib/k8s/kubernetes/crt/apiserver.crt \\
  --enable-hostpath-provisioner=false \\
  --pvclaimbinder-sync-period=15s \\
  --master=10.0.1.80:8080 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-controller-manager
service kube-controller-manager start

Step 8 Kubernetes Kubelet (Master (slightly different invocation than nodes))

Define it

cat << EOF > /lib/systemd/system/kube-kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
User=root
ExecStart=/usr/bin/kubelet \\
  --cert-dir=/var/lib/k8s/kubernetes/ \\
  --chaos-chance=0.0 \\
  --container-runtime=docker \\
  --register-schedulable=false \\
  --address=0.0.0.0 \\
  --cpu-cfs-quota=false \\
  --api-servers=10.0.1.80:8080 \\
  --cluster-dns=8.8.8.8 \\
  --port=10250 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-kubelet
service kube-kubelet start

Step 9: Kubernetes Proxy

Define it

cat << EOF > /lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
User=root
ExecStart=/usr/bin/proxy \\
  --master=http://10.0.1.80:8080 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-proxy
service kube-proxy start

Step 10: Kubernetes Scheduler

Define it

cat << EOF > /lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service

[Service]
User=kube
ExecStart=/usr/bin/scheduler \\
  --master=http://10.0.1.80:8080 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-scheduler
service kube-scheduler start

The Nodes:

Expectations:

The Nodes will need the following processes when all is said and done

Non-Kubernetes:

docker
flanneld

Kubernetes:

kublelet
proxy

Step 1: Get Ready

Updated grup (you did pay attention and read the things I told you to read, right?)
Reboot

echo '10.0.1.80 kube-master.apokalyptik.lan kube-master' >> /etc/hosts
echo '10.0.1.81 kube-node1.apokalyptik.lan kube-node1' >> /etc/hosts
echo '10.0.1.82 kube-node2.apokalyptik.lan kube-node2' >> /etc/hosts

apt-get -fqqy install apt-transport-https ca-certificates curl

Yes... we want a single 132M binary from inside a 352M tarball from inside another 1.4G tarball. Such is life. I expect you'll just get this file once and copy it around...

curl \
  --progress-bar \
  --location \
  'https://github.com/kubernetes/kubernetes/releases/download/v1.3.4/kubernetes.tar.gz' \
    | tar --to-stdout -zxf - kubernetes/server/kubernetes-server-linux-amd64.tar.gz \
      | tar --to-stdout -zxf - kubernetes/server/bin/hyperkube > /usr/bin/hyperkube 
chmod a+x /usr/bin/hyperkube
cd /usr/bin/
/usr/bin/hyperkube --make-symlinks

mkdir -p /var/lib/k8s
groupadd kube
useradd kube -g kube -d /var/lib/k8s/ -s /bin/false
chown kube:kube /var/lib/k8s

Step 2: Install Docker

apt-key adv \
  --keyserver hkp://p80.pool.sks-keyservers.net:80 \
  --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo debian-jessie main' \
  > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get -fqqy install docker-engine

Step 3: Install Flanneld

Download it and install it

cd /usr/local/src
curl \
  --silent \
  --location \
  'https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz' \
    | tar -zvxf-
cd flannel-0.5.5
cp flanneld /usr/bin
mkdir -p /var/lib/k8s/flannel/networks

Define it

cat << EOF > /lib/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service

[Service]
Type=notify
Restart=always
RestartSec=5
ExecStart=/usr/bin/flanneld \\
  -etcd-endpoints=http://10.0.1.80:4001 \\
  -logtostderr=true \\
  -subnet-dir=/var/lib/k8s/flannel/networks \\
  -subnet-file=/var/lib/k8s/flannel/subnet.env
[Install]
WantedBy=multi-user.target
EOF

Start it

systemctl daemon-reload
systemctl enable flanneld
service flanneld start

Step 4: Reconfigure docker to use FlannelD

Redefine it

cat << EOF > /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket flanneld.service
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=-/var/lib/k8s/flannel/subnet.env
ExecStart=/usr/bin/dockerd -H fd:// --bip=\${FLANNEL_SUBNET} --mtu=\${FLANNEL_MTU}
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
MountFlags=shared

[Install]
WantedBy=multi-user.target
EOF

Give kube access

gpasswd -a kube docker

Reload it

systemctl daemon-reload
service docker restart

Step 5: Kubernetes Proxy

Define it

cat << EOF > /lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=flanneld.service

[Service]
User=root
ExecStart=/usr/bin/proxy \\
  --master=http://10.0.1.80:8080 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-proxy
service kube-proxy start

Step 6 Kubernetes Kubelet

Define it

cat << EOF > /lib/systemd/system/kube-kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=flanneld.service

[Service]
User=root
ExecStart=/usr/bin/kubelet \\
  --cert-dir=/var/lib/k8s/kubernetes/ \\
  --chaos-chance=0.0 \\
  --container-runtime=docker \\
  --address=0.0.0.0 \\
  --cpu-cfs-quota=false \\
  --api-servers=10.0.1.80:8080 \\
  --cluster-dns=8.8.8.8 \\
  --port=10250 \\
  --logtostderr=true
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable kube-kubelet
service kube-kubelet start

Now we are going to use a public docker image (nginx) as a hello world.

Create a deployment/pod/thing

kubectl run my-nginx --image=nginx --replicas=2 --port=80

kubectl get deployments

NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
my-nginx   2         2         2            2           1m

kubectl get pods

NAME                        READY     STATUS    RESTARTS   AGE
my-nginx-2494149703-2xovs   1/1       Running   0          1m
my-nginx-2494149703-hbesv   1/1       Running   0          1m

once it's ready (see above) expose it

kubectl expose deployment my-nginx --type="LoadBalancer"

root@kube-master:/usr/local/bin# kubectl get services

NAME         CLUSTER-IP        EXTERNAL-IP   PORT(S)   AGE
kubernetes   192.168.0.1       <none>        443/TCP   53m
my-nginx     192.168.176.186   <pending>     80/TCP    12s

Now make sure we can get to it via flannel

curl -I 192.168.176.186

HTTP/1.1 200 OK
Server: nginx/1.11.1
Date: Sat, 06 Aug 2016 20:01:49 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 31 May 2016 14:40:22 GMT
Connection: keep-alive
ETag: "574da256-264"
Accept-Ranges: bytes

erikbgithub · 2017-03-20T16:20:29Z

At this moment it may be smarter to just paste the contents directly, won't it? The content is only available in other ppl's repos. I just paste it for linking below:

Kubernetes runs well in containers on most Linux distributions that support Docker v1.10 (or above), however there are a few small bugs in some of the default installations.

Official Docker Engine Packages and systemd

See docker/docker#19625 for details

If you install Docker through the official package repository on a systemd-enabled distribution, you will need ensure that docker.service unit has MountFlags=shared instead of default MountFlags=slave.

This is simple to fix with a drop-in unit:

sudo mkdir -p /etc/systemd/system/docker.service.d/
cat << EOF | sudo tee /etc/systemd/system/docker.service.d/clear_mount_propagtion_flags.conf
[Service]
MountFlags=shared
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker.service

Debian 8 and cgroups

See kubernetes/kubernetes#23816 for details

If you are using Debian 8 (Jessie), you will need to check if memory cgroup is enabled, fixing it requires a reboot.

Open /etc/default/grub in your favourite text editor
Add cgroup_enable=memory swapaccount=1 to GRUB_CMDLINE_LINUX
Run sudo update-grub2 followed by sudo reboot

syswombat · 2017-09-13T07:26:35Z

Set up Kubernetes on 3 Debian Jessie virtual machines: One master. Two nodes.
could you give some Information about CPU/RAM/Disksize needed? Thanks a lot

syswombat · 2017-10-13T11:03:17Z

Hello
both link below IMPORTANT THINGS YOU NEED TO READ are giving a 404 error

synergiator · 2017-10-31T09:27:36Z

as of October 2017, latest K8s version is 1.8.1. Does it yet make sense to dive into this tutorial?
See also for example: https://blog.alexellis.io/kubernetes-in-10-minutes/

	`kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml`

	* The following commands will help you inspect and see when it's done:

	`kubectl get --namespace="kube-system" pods`
	`kubectl get --namespace="kube-system" deployments`
	`kubectl get --namespace="kube-system" services`

	also `docker ps` on the nodes should show the running container somewhere

	once finished you should be able to get to http://10.0.1.80:8080/ui/

apokalyptik/debian-jessie-kubernetes-1.3-manual-install-guide.md

The Goal

The Setup

IMPORTANT THINGS YOU NEED TO READ

THIS IS IMPORTANT TOO

Last note

Order of Operations

The Master:

Expectations:

Step 1: Get Ready

Step 2: Install Docker

Step 3: Install ETCD

Step 4: FlannelD

Step 5: Reconfigure docker to use FlannelD

Step 6: Kubernetes API Server (finally, on to kubernetes!)

Step 7: Kubernetes controller-manager

Step 8 Kubernetes Kubelet (Master (slightly different invocation than nodes))

Step 9: Kubernetes Proxy

Step 10: Kubernetes Scheduler

The Nodes:

Expectations:

Step 1: Get Ready

Step 2: Install Docker

Step 3: Install Flanneld

Step 4: Reconfigure docker to use FlannelD

Step 5: Kubernetes Proxy

Enable it

Step 6 Kubernetes Kubelet

erikbgithub commented Mar 20, 2017 •

edited

Loading

syswombat commented Sep 13, 2017

syswombat commented Oct 13, 2017

synergiator commented Oct 31, 2017 •

edited

Loading

apokalyptik/debian-jessie-kubernetes-1.3-manual-install-guide.md

The Goal

The Setup

IMPORTANT THINGS YOU NEED TO READ

THIS IS IMPORTANT TOO

Last note

Order of Operations

The Master:

Expectations:

Step 1: Get Ready

Step 2: Install Docker

Step 3: Install ETCD

Step 4: FlannelD

Step 5: Reconfigure docker to use FlannelD

Step 6: Kubernetes API Server (finally, on to kubernetes!)

Step 7: Kubernetes controller-manager

Step 8 Kubernetes Kubelet (Master (slightly different invocation than nodes))

Step 9: Kubernetes Proxy

Step 10: Kubernetes Scheduler

The Nodes:

Expectations:

Step 1: Get Ready

Step 2: Install Docker

Step 3: Install Flanneld

Step 4: Reconfigure docker to use FlannelD

Step 5: Kubernetes Proxy

Enable it

Step 6 Kubernetes Kubelet

erikbgithub commented Mar 20, 2017 • edited Loading

Official Docker Engine Packages and systemd

Debian 8 and cgroups

syswombat commented Sep 13, 2017

syswombat commented Oct 13, 2017

synergiator commented Oct 31, 2017 • edited Loading

erikbgithub commented Mar 20, 2017 •

edited

Loading

synergiator commented Oct 31, 2017 •

edited

Loading