Deploy BOSH Lite for Kubernetes on Linux

Indeed, it seems worthwhile to write down the few steps necessary to install BOSH Lite, to deploy Kubernetes on BOSH (formerly named kubo, lately know as Cloud Foundry Container Runtime or just cfcr), and to login to the Kubernetes CLI (kubectl) and Dashbaord.

As a software developer you might be interested in a complete and working installation. As a computer geek as myself you might be more kin to know the ingredients of this recipe and the way they combine with each other. In this article I have tried to serve both clients.

In this mixture a few generally available components have found their usage, by the time of writing they were:

$ uname -mrs
Linux 4.15.0-43-generic x86_64
$ lsb_release -ds
Ubuntu 18.04.1 LTS
$ vboxmanage --version
5.2.18_Ubuntur123745
$ vboxmanage list extpacks | grep Version -B1
Pack no. 0: Oracle VM VirtualBox Extension Pack
Version: 5.2.18
$ bosh --version
version 5.4.0-891ff634-2018-11-14T00:22:02Z
$ credhub --version
CLI Version: 2.2.0
$ kubectl version --client --short
Client Version: v1.13.2

Not all of these components need to be versioned at these exact levels. However, as your mileage may vary, it is a good idea to get as close as possible to these versions. So let's start.

1. Linux Ubuntu OS

Be sure to be à jour. Update apt package management sysem.

sudo apt update && sudo apt upgrade

2. Dependency packages

It may be necessary to install curl, git and ruby.

sudo apt install curl git ruby

3. VirtualBox and its Extension Pack

VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at server, desktop and embedded use.

curl -Lo virtualbox.deb https://download.virtualbox.org/virtualbox/5.2.22/virtualbox-5.2_5.2.22-126460~Ubuntu~bionic_amd64.deb
sudo dpkg -i virtualbox.deb && rm virtualbox.deb

wget https://download.virtualbox.org/virtualbox/5.2.22/Oracle_VM_VirtualBox_Extension_Pack-5.2.22.vbox-extpack
sudo vboxmanage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.2.22.vbox-extpack
rm Oracle_VM_VirtualBox_Extension_Pack-5.2.22.vbox-extpack

4. BOSH CLI for Linux

The bosh CLI is the command line interface for interacting with what BOSH does, from deployment operations to software release management. BOSH is purposefully used to deploy Cloud Foundry at large, whereas, BOSH Lite is used for local deployments, like the one in this article.

curl -Lo bosh https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-5.4.0-linux-amd64
chmod +x bosh && sudo chown root:root bosh && sudo mv bosh /usr/local/bin/

5. The cli for credhub

CredHub provides a CLI and API to manage credentials (usernames, passwords, certificates, keys and other arbitrary values) in a secure fashion. The API exposed by the CredHub server for BOSH are also accessed by the credhub CLI.

curl -L https://github.com/cloudfoundry-incubator/credhub-cli/releases/download/2.2.0/credhub-linux-2.2.0.tgz | tar -zx
sudo chown root:root credhub && sudo mv credhub /usr/local/bin/

6. kubectl for Linux

Use the Kubernetes command-line tool, kubectl, to deploy and manage applications on Kubernetes. Using kubectl, you can inspect cluster resources; create, delete, and update components; look at your new cluster; and bring up example apps.

sudo snap install kubectl --classic

Preparation

BOSH Lite

Following docs and tutorials of BOSH, we begin by cloning (i.e. downloading) the bosh-deployment repository.

mkdir -p bosh
cd bosh
git clone https://github.com/cloudfoundry/bosh-deployment
cd bosh-deployment

If you feel like you may need some more power (cpus, memory or ephemeral_disk) for your Cloud Foundry deployment, then double these cloud_properties to get a little boost in performance. Below you can see the difference between the default values found within the virtualbox/cpi.yml file at the line # Configure sizes, and the supposedly better settings:

# Configure sizes
- type: replace
  path: /resource_pools/name=vms/cloud_properties?
  value:

-    cpus: 2
+    cpus: 4
-    memory: 4096
+    memory: 8192
-    ephemeral_disk: 16_384
+    ephemeral_disk: 32_768

Note: The property paravirtprovider is not explicitly assigned, which leads to its default minimal value. That opens the case to further experimenting. As the standard stemcell (see below) hosting the BOSH Director virtual machine is a Linux Ubuntu one, there is a good chance that appending the property and value paravirtprovider: kvm might bring some performance boost. By the way, feel free to fork this article and experiment further.

Furthermore, it is a sensible advice to always use the latest stemcell. That also requires us to make a change on the virtualbox/cpi.yml file:

- type: replace
  path: /resource_pools/name=vms/stemcell?
  value:

-    url: https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=170.9
-    sha1: dff5ee88f85c902389d8f9960e06fdf82b233c1f
+    url: https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=170.21
+    sha1: 49104015fb229ce1af5629bae79a163415e2d1c2

By now, let's just perform these three steps:

Create the BOSH Director virtual machine

bosh create-env bosh.yml  --state state.json  -o virtualbox/cpi.yml  -o virtualbox/outbound-network.yml  -o bosh-lite.yml  -o bosh-lite-runc.yml  -o uaa.yml  -o credhub.yml  -o jumpbox-user.yml  --vars-store creds.yml  -v director_name=bosh-lite  -v internal_ip=192.168.50.6  -v internal_gw=192.168.50.1  -v internal_cidr=192.168.50.0/24  -v outbound_network_name=NatNetwork

Note: Non-wrapping in the command is intended. This step will take some time.

Set a route and environment variables

sudo ip route add 10.244.0.0/16 via 192.168.50.6

export BOSH_CA_CERT=`bosh int creds.yml --path /director_ssl/ca`
export BOSH_ENVIRONMENT=vbox
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int creds.yml --path /admin_password`

export CREDHUB_SERVER=https://192.168.50.6:8844
export CREDHUB_CLIENT=credhub-admin
export CREDHUB_SECRET=`bosh int creds.yml --path=/credhub_admin_client_secret`
export CREDHUB_CA_CERT="$(bosh int creds.yml --path=/credhub_tls/ca)"$'\n'"$(bosh int creds.yml --path=/uaa_ssl/ca)"

Note: More details on the create-env.sh script.

Set alias and then login

bosh alias-env vbox -e 192.168.50.6
bosh env 	# Just to test your success #
bosh login

Note: As all necessary environment variables are pre-set on the previous step, bosh login may not be needed.

Cloud Foundry Container Runtime

Now, it is Kubernetes deployment time. Let's give it a run in few steps:

Set up DNS and Cloud Config

cd ..
git clone https://github.com/cloudfoundry/cf-deployment
bosh update-cloud-config cf-deployment/iaas-support/bosh-lite/cloud-config.yml
bosh update-runtime-config bosh-deployment/runtime-configs/dns.yml --name dns

Upload a stemcell

Before BOSH can deploy your Cloud Foundry Container Runtime, it needs a base VM image to start with. In the BOSH ecosystem, these images are called stemcells.

git clone https://github.com/cloudfoundry-incubator/kubo-deployment
cd kubo-deployment
STEMCELL_OS=$(bosh int manifests/cfcr.yml --path=/stemcells/0/os)
STEMCELL_NAME="bosh-warden-boshlite-${STEMCELL_OS}-go_agent"
STEMCELL_VERSION=$(bosh int manifests/cfcr.yml --path=/stemcells/alias=default/version)
bosh upload-stemcell "https://bosh.io/d/stemcells/${STEMCELL_NAME}?v=${STEMCELL_VERSION}"

Deploy CFCR on BOSH

export BOSH_DEPLOYMENT=cfcr
sed -i 's/  lifecycle: errand/# lifecycle: errand/' manifests/cfcr.yml   # Deploy apply-addons instance #
bosh deploy --no-redact manifests/cfcr.yml -o manifests/ops-files/misc/single-master.yml -o manifests/ops-files/iaas/virtualbox/bosh-lite.yml -o manifests/ops-files/worker_count.yml -v worker_count=1
bosh run-errand apply-specs
cd ..

Note: deploy may take a while. More info on deploy_cfcr_lite.

Set the cluster kubo-lite and use your context kubo

export KUBO_ADMIN_PASSWORD=`credhub get -n /bosh-lite/cfcr/kubo-admin-password -q`

kubectl config set-cluster kubo-lite --server="https://master.bosh-lite.com:8443" --insecure-skip-tls-verify=true
kubectl config set-credentials kubo-admin --token="$KUBO_ADMIN_PASSWORD"
kubectl config set-context kubo --cluster=kubo-lite --user=kubo-admin
kubectl config use-context kubo

Test your access

$ kubectl get all
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.200.1   <none>        443/TCP   5m

Get more info

kubectl cluster-info
kubectl get node -o wide
kubectl get all -n kube-system

Get the token

export KUBO_ADMIN_TOKEN=`kubectl get secrets "$(kubectl get secrets -n kube-system | grep clusterrole-aggregation-controller | awk '{print $1}')" -n kube-system -o json | jq -r .data.token | base64 -d`

Access the dashboard

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

Fire up your favourite browser and go to
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login
Choose token and fill in the field with the output of echo $KUBO_ADMIN_TOKEN.

Well done. Congratulations! You're done. 😄

Source:

Cloud Foundry BOSH, VirtualBox (2018) https://bosh.io/docs/bosh-lite
Cloud Foundry Container Runtime (2018) https://github.com/cloudfoundry-incubator/kubo-release
kubo-deployment (2019) https://github.com/cloudfoundry-incubator/kubo-deployment

References / Acknowledgments:

[1] Toshiaki Maki: Deploy CFCR on BOSH Lite (2018)
https://github.com/making/cfcr-lite

[2] Brendan Nolan: Kubo/CFCR on Bosh Lite (2018)
https://github.com/bstick12/kubo-bosh-lite

[3] Toshiaki Maki: Cloud Foundry Container Runtime (a.k.a Kubo / Kubernetes on BOSH) 0.11.0 BOSH-Lite (2018)
https://blog.ik.am/entries/440

About the author:

Della Porta is fascinated by computer sciences, distributed systems and cloud technologies.

dellaporta/bosh_k8s_linux.md