Play with Kops / Tectonic / channels

00-k8s-provisioning.md

Comparison between cluster provisioning tools on AWS

1. Kops

2. Tectonic

3. Getting kops channels tool

channels.md

Kops - Channels

Compiling

$ docker run -it  -w /go/src/k8s.io/ golang:1.8 bash.
root@64fe59077e2b:/go/src/k8s.io# git clone https://github.com/kubernetes/kops.git
Cloning into 'kops'...
remote: Counting objects: 143858, done.
remote: Compressing objects: 100% (83/83), done.
remote: Total 143858 (delta 42), reused 44 (delta 18), pack-reused 143752
Receiving objects: 100% (143858/143858), 154.71 MiB | 2.33 MiB/s, done.
Resolving deltas: 100% (81700/81700), done.
Checking connectivity... done.
Checking out files: 100% (30333/30333), done.
root@64fe59077e2b:/go/src/k8s.io# cd kops/
root@64fe59077e2b:/go/src/k8s.io/kops# git checkout 1.7.1
Note: checking out '1.7.1'
root@64fe59077e2b:/go/src/k8s.io/kops# make channels
go install  -ldflags "-X k8s.io/kops.Version=1.7.1 " k8s.io/kops/channels/cmd/channels
root@64fe59077e2b:/go/src/k8s.io/kops# which channels
/go/bin/channels
root@64fe59077e2b:/go/src/k8s.io/kops# cd /go/bin
root@64fe59077e2b:/go/bin# zip -9 channels-linux-amd64.zip channels
  adding: channels (deflated 77%)
root@64fe59077e2b:/go/bin# ls -lah
total 64M
drwxrwxrwx 1 root root 4.0K Nov 27 13:16 .
drwxrwxrwx 1 root root 4.0K Nov 27 13:11 ..
-rwxr-xr-x 1 root root  52M Nov 27 13:12 channels
-rw-r--r-- 1 root root  12M Nov 27 13:16 channels-linux-amd64.zip

Addon tracking

How does channels keep track of deployed addons?

Review kube-system namespace annotations

kubectl get ns kube-system -o json | jq .
{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "annotations": {
      "addons.k8s.io/core.addons.k8s.io": "{\"version\":\"1.4.0\",\"channel\":\"s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml\"}",
      "addons.k8s.io/dns-controller.addons.k8s.io": "{\"version\":\"1.7.1\",\"channel\":\"s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml\",\"id\":\"k8s-1.6\"}",
      "addons.k8s.io/kube-dns.addons.k8s.io": "{\"version\":\"1.14.5\",\"channel\":\"s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml\",\"id\":\"k8s-1.6\"}",
      "addons.k8s.io/kube-state-metrics.addons.k8s.io": "{\"version\":\"v1.1.0-rc.0\",\"channel\":\"beekeeper/addons.yaml\"}",
      "addons.k8s.io/kubernetes-dashboard": "{\"version\":\"1.7.1\",\"channel\":\"kubernetes-dashboard\"}",
      "addons.k8s.io/limit-range.addons.k8s.io": "{\"version\":\"1.5.0\",\"channel\":\"s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml\"}",
      "addons.k8s.io/monitoring-standalone": "{\"version\":\"1.6.0\",\"channel\":\"monitoring-standalone\"}",
      "addons.k8s.io/namespaces.honestbee.io": "{\"version\":\"1.1.2\",\"channel\":\"beekeeper/addons.yaml\"}",
      "addons.k8s.io/storage-aws.addons.k8s.io": "{\"version\":\"1.6.0\",\"channel\":\"s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml\"}",
      "addons.k8s.io/tiller.addons.k8s.io": "{\"version\":\"2.7.2\",\"channel\":\"beekeeper/addons.yaml\",\"id\":\"k8s-1.7\"}",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"kube-system\",\"namespace\":\"\"}}\n"
    },
   ...
}

Using jq We can filter annotations down to those that contain addons term and unwrap the json embedded.

kubectl get ns kube-system -o json | \
  jq '.metadata.annotations | with_entries(select(.value | contains("addons"))) | map_values(fromjson)'

Which returns

{
  "addons.k8s.io/core.addons.k8s.io": {
    "version": "1.4.0",
    "channel": "s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml"
  },
  "addons.k8s.io/dns-controller.addons.k8s.io": {
    "version": "1.7.1",
    "channel": "s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml",
    "id": "k8s-1.6"
  },
  "addons.k8s.io/kube-dns.addons.k8s.io": {
    "version": "1.14.5",
    "channel": "s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml",
    "id": "k8s-1.6"
  },
  "addons.k8s.io/kube-state-metrics.addons.k8s.io": {
    "version": "v1.1.0-rc.0",
    "channel": "beekeeper/addons.yaml"
  },
  "addons.k8s.io/limit-range.addons.k8s.io": {
    "version": "1.5.0",
    "channel": "s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml"
  },
  "addons.k8s.io/namespaces.honestbee.io": {
    "version": "1.1.2",
    "channel": "beekeeper/addons.yaml"
  },
  "addons.k8s.io/storage-aws.addons.k8s.io": {
    "version": "1.6.0",
    "channel": "s3://305788746247-bee02-kops-state-store/bee02-cluster.training.honestbee.com/addons/bootstrap-channel.yaml"
  },
  "addons.k8s.io/tiller.addons.k8s.io": {
    "version": "2.7.2",
    "channel": "beekeeper/addons.yaml",
    "id": "k8s-1.7"
  }
}

or use this

kubectl get ns kube-system -o json \
  | jq '.metadata.annotations | with_entries(select(.value | contains("addons"))) | map_values(fromjson | .version)'
{
  "addons.k8s.io/core.addons.k8s.io": "1.4.0",
  "addons.k8s.io/dns-controller.addons.k8s.io": "1.7.1",
  "addons.k8s.io/kube-dns.addons.k8s.io": "1.14.5",
  "addons.k8s.io/kube-state-metrics.addons.k8s.io": "v1.1.0-rc.0",
  "addons.k8s.io/limit-range.addons.k8s.io": "1.5.0",
  "addons.k8s.io/namespaces.honestbee.io": "1.1.2",
  "addons.k8s.io/storage-aws.addons.k8s.io": "1.6.0",
  "addons.k8s.io/tiller.addons.k8s.io": "2.7.2"
}

kops.md

KOPS

Kubernetes clusters on AWS

Quickstart

Create an AWS account and get a domain name follow the excelent kops on aws guides

(for Sydney use ap-southeast-2)

easy DNS validation with Gsuite dig

export CLUSTER=staging
export DOMAIN=spawnhub
export NAME=${CLUSTER}.${DOMAIN}.com
export KOPS_STATE_STORE=s3://kubernetes-${DOMAIN}-com-state-store
export AWS_REGION=ap-southeast-2
export KEY_NAME=k8s-${CLUSTER}-${DOMAIN}

# make bucket (with location constraint)
aws s3 mb ${KOPS_STATE_STORE} --region ${AWS_REGION}
aws s3api put-bucket-versioning --bucket ${KOPS_STATE_STORE##*/} --versioning-configuration Status=Enabled


# aws ec2 describe-availability-zones --region ap-southeast-2

mkdir -p ~/.ssh/
aws --region ${AWS_REGION} ec2 create-key-pair --key-name ${KEY_NAME} --query 'KeyMaterial' --output text > ~/.ssh/${KEY_NAME}.pem
chmod 400 ~/.ssh/${KEY_NAME}.pem
ssh-add ~/.ssh/${KEY_NAME}.pem
ssh-keygen -y -f ~/.ssh/${KEY_NAME}.pem > ~/.ssh/${KEY_NAME}.pub

#potentially need --dns-zone if multiple zones match the domain name

kops create cluster \
    --zones ap-southeast-2a \
    --ssh-public-key ~/.ssh/${KEY_NAME}.pub \
    --authorization RBAC
    ${NAME}


kops update cluster ${NAME} --yes # to actually create the cluster

# takes max 5 min..

kops validate cluster ${NAME}

# get dashboard - https://github.com/kubernetes/dashboard/releases
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.3/src/deploy/kubernetes-dashboard.yaml

# alternatively - add DNS mapper? (not applied for this talk)
https://github.com/kubernetes/kops/blob/75718857b69756ac8374a1001363eadfa6f09a4f/addons/route53-mapper/README.md

# Ideally use something like kube-lego to use https with cog...

Install Helm's Tiller to cluster

helm init

Boot sequence

nodeup is the component that installs packages and sets up the OS, sufficiently for Kubelet In addition, nodeup installs:

• Protokube, which is a kops-specific component (in charge of etcd cluster mostly at the moment)

nodeup creates manifests in /etc/kubernetes/manifests, a directory the kubelet is configured to monitor.

Traditionally, flannel was used to allocate and assign subnets for each node.

in Kops, the kubelet tries to contact the API server and register its node. only when the API Server is up and allocates a PodCIDR to the node, the actual Container Runtime will start.

in detail:

• when API server can be reached
• kubelet creates Node object representing itself
• kube-controller-manager assigns PodCIDR
• kubelet sees PodCIDR assignment and configures local docker bridge (cbr0)
• node will be marked as ready and scheduler will assign pods to the node
• kubelet handles assigned pods

the kubelet will set up the local bridge with the allocated CIDR, only then the container runtime can start.

Prior to this, all core containers run with hostNetwork=true and can start without the bridge being available.

On Masters, API server listens on localhost unsecured, controller manager and scheduler can thus connect through localhost without a token.

a dns-controller running in k8s will create DNS records with the cloud provider to allow discovery of the master nodes.

Protokube:

• Discovers EBS volumes by doing an AWS query using labels

resource "aws_ebs_volume" "a-etcd-events-c01-honestbee-jp" {
  availability_zone = "ap-northeast-1a"
  size              = 20
  type              = "gp2"
  encrypted         = false

  tags = {
    KubernetesCluster    = "c01.honestbee.jp"
    Name                 = "a.etcd-events.c01.honestbee.jp"
    "k8s.io/etcd/events" = "a/a"
    "k8s.io/role/master" = "1"
  }
}

• Tries to safe_format_and_mount EBS volumes
• when a volume is mounted, write a manifest for etcd into /etc/kubernetes/manifests
• configures DNS for the etcd nodes (can't use dns-controller as API requires etcd)

Technical details

ClusterSpec

docs

NodeUp

nodeup https://kubeupv2.s3.amazonaws.com/kops/1.6.0/linux/amd64/nodeup

./nodeup --install-systemd-unit --conf=${INSTALL_DIR}/kube_env.yaml --v=8

Assets

• kubelet
• kubectl
• cni
• kops utils
• protokube

Sample kube_env.yaml

Assets:
- 57afca200aa6cec74fcc3072cae12385014f59c0@https://storage.googleapis.com/kubernetes-release/release/v1.6.2/bin/linux/amd64/kubelet
- 984095cd0fe8a8172ab92e2ee0add49dfc46e0c2@https://storage.googleapis.com/kubernetes-release/release/v1.6.2/bin/linux/amd64/kubectl
- 1d9788b0f5420e1a219aad2cb8681823fc515e7c@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
- a821df4b965dbb119ed207c397b605a1fd99db3c@https://kubeupv2.s3.amazonaws.com/kops/1.6.0/linux/amd64/utils.tar.gz
ClusterName: c01.honestbee.jp
ConfigBase: s3://honestbee-kops-training-clusters-state/c01.honestbee.jp
InstanceGroupName: master-ap-northeast-1a
Tags:
- _automatic_upgrades
- _aws
- _kubernetes_master #nodes just drop this tag...
channels:
- s3://honestbee-kops-training-clusters-state/c01.honestbee.jp/addons/bootstrap-channel.yaml
protokubeImage:
  hash: ca9f3a641d4130616f09403afbf5eed749633a86
  name: protokube:1.6.0
  source: https://kubeupv2.s3.amazonaws.com/kops/1.6.0/images/protokube.tar.gz

tectonic-playground.md

Tectonic

Deep dive into Self Hosted

Ref

Provide simpler installation flow and sustainable cluster lifecycle upgrade/management.

Goals

1. Self hosted should reduce the number of components required on the host.
2. Reduces the number of files written to disk (less Config Mngmnt requirements)
3. Introspection (kube system logs provide insights to kube components)
4. Kubernetes itself is subject to upgrades via k8s API itself
5. Easier HA

Dogfooding k8s will likely improve installation experience for other projects as well

Layers

0. kubelet
1. etcd
2. apiserver
3. scheduler / controller manager / proxy
4. dns & addons

Bootkube State (Q2 2017)

• providing 2-4 self-hosting since July 2016
• providing 1-4 self-hosting since Q1 2017

Temporary k8s control plane to control kubelet and spin up full blown control plane

Step 1 relies on System hosted kubelet Step 2 relies on self-hosted kubelet with --exit-on-lock-contention Step 3 system hosted has to watch for posix lock to ensure self-hosted stays alive

Note

• control components pinned to control nodes
• LB for API server depends on off-cluster DNS/TLS/LB

Hand-off of control

• during hand-off, self-hosted components unsupervised and danger for crashes
• periodic snapshotting of pods manifests to local node manifest directory (Any pod which contains the checkpointer.alpha.coreos.com/checkpoint=true annotation will be considered a viable "parent pod" which should be checkpointed.)

Note on static pod manifests (--pod-manifest-path or --manifest-url): The major downside of using static Pods is that they are a special class of Pod. While visible in the API server (as “mirror” Pods), they are read-only and cannot be controlled by it.

1-4 Self hosting Updates

Kubelet

As kubelet itself uses a system hosted kublet (not self-hosted) through kubelet-wrapper, when an update is required - a node annotation is made which is read by a long running daemonset which updates the kubelet-wrapper config.

This allows kubelet version updates from the cluster API

API Server, Scheduler and Controller Manager

stateless and self-hosted - managed through cluster API (set image version, update service label selectors)

Note: May cause temp downtime and may require fix depending on Kubernetes versioning policy and support for version skew between the control plane components

etcd self-hosted

Progress is being made with integration of etcd-operator into bootkube...

link

PRs

Points against Bootkube

• cluster version tied to bootkube version
• maintenance burden to keep bootkube version updated
• more moving parts and complexity increase chance of failure

How do other proposal rate against the above points:

• kubeadm also requires a release per k8s version
• kubelet pod API also adds additional complexity and doesn't cover hand-off as bootkube does

Roadmap

Ideally kubelet pod API replaces bootkube in the long run

Tectonic - Terraform

If the tectonic_base_domain is set to kube.example.com a Route 53 zone must exist for this domain and the AWS nameservers must be configured for the domain.

Based on kops DNS subdomain config.

In this scenario you want to contain all kubernetes records under a subdomain of a domain you host in Route53. This requires creating a second hosted zone in route53, and then setting up route delegation to the new zone.

In this example you own example.com and your records for Kubernetes would look like c01.tectonic.example.com

# Assuming a public domain for Cluster API ($DOMAIN is public zone)
ID=$(uuidgen)
export DOMAIN=example.com
export CLUSTER_DOMAIN=tectonic.${DOMAIN}
export CHILD_ID=$(aws route53 create-hosted-zone --name ${CLUSTER_DOMAIN} --caller-reference $ID --query "HostedZone.Id" --output text)
export PARENT_ID=$(aws route53 list-hosted-zones --query="HostedZones[?Name=='${DOMAIN}.' && Config.PrivateZone==\`false\`].Id" --output=text)
# create NS records for child zone in parent zone
aws route53 get-hosted-zone --id ${CHILD_ID} --query="DelegationSet. {Comment:'Create ${CLUSTER_DOMAIN} NS record in ${DOMAIN} domain',Changes:[{Action:'CREATE',ResourceRecordSet:{Name:'${CLUSTER_DOMAIN}',Type:'NS',TTL:\`300\`,ResourceRecords:NameServers[*].{Value:@}}}]}" > subdomain.json
aws route53 change-resource-record-sets  --hosted-zone-id ${PARENT_ID} --change-batch file://subdomain.json

To provision Tectonic console Admin login:

curl -Lo bcrypt-tool.tar.gz https://github.com/coreos/bcrypt-tool/releases/download/v1.0.0/bcrypt-tool-v1.0.0-darwin-amd64.tar.gz
# sha512sum bcrypt-tool-v1.0.0-linux-amd64.tar.gz
tar -zxvf bcrypt-tool.tar.gz
./bcrypt-tool/bcrypt-tool

Read Tectonic with Terraform docs

Download tectonic release

curl -O https://releases.tectonic.com/tectonic-1.6.7-tectonic.1.tar.gz # download
tar -xzvf tectonic-1.6.7-tectonic.1.tar.gz # extract the tarball
cd tectonic

cluster c01.tectonic.example.com

export PATH=$PATH:$(pwd)/tectonic-installer/darwin
export TERRAFORM_CONFIG=$(pwd)/.terraformrc

# terraform init should be sufficient?
# config s3 backend and use terraform env?
terraform get platforms/aws

export CLUSTER=c01
mkdir -p build/${CLUSTER}
cp examples/terraform.tfvars.aws build/${CLUSTER}/terraform.tfvars

Review:

• tectonic_admin_email
• tectonic_admin_password_hash
• tectonic_aws_region
• tectonic_aws_ssh_key
• tectonic_base_domain
• tectonic_cluster_name
• tectonic_vanilla_k8s
• tectonic_worker_count
• tectonic_master_count

Plan and Apply

terraform plan -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws
terraform apply -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws

Note: Seems you can not provision 2 clusters from the same install dir, the generated/ dir is overwritten causing terraform plan no longer to work across the 2 clusters. Why? Known Issue

1. Should learn to use terraform env
2. Should update the location of assets directory

Assets directory issue:

1. the bootkube Terraform module is using the Terraform TLS Provider to generate self-signed certificates and stores everything in the generated directory:

resource "tls_private_key" "apiserver" {
  algorithm = "RSA"
  rsa_bits  = "2048"
}
resource "local_file" "apiserver-key" {
  content  = "${tls_private_key.apiserver.private_key_pem}"
  filename = "./generated/tls/apiserver.key"
}

1. the bootkube Terraform module also generates the kubeconfig

data "template_file" "kubeconfig" {
  template = "${file("${path.module}/resources/kubeconfig")}"

  vars {
    ca_cert      = "${base64encode(var.ca_cert == "" ? join(" ", tls_self_signed_cert.kube-ca.*.cert_pem) : var.ca_cert)}"
    kubelet_cert = "${base64encode(tls_locally_signed_cert.kubelet.cert_pem)}"
    kubelet_key  = "${base64encode(tls_private_key.kubelet.private_key_pem)}"
    server       = "${var.kube_apiserver_url}"
  }
}
resource "local_file" "kubeconfig" {
  content  = "${data.template_file.kubeconfig.rendered}"
  filename = "./generated/auth/kubeconfig"
}

...

1. the aws/s3.tf module gets the output from aws/tectonic.tf which uses the Archive.data_archive Terraform resource:

data "archive_file" "assets" {
  type       = "zip"
  source_dir = "./generated/"

  # Because the archive_file provider is a data source, depends_on can't be
  # used to guarantee that the tectonic/bootkube modules have generated
  # all the assets on disk before trying to archive them. Instead, we use their
  # ID outputs, that are only computed once the assets have actually been
  # written to disk. We re-hash the IDs (or dedicated module outputs, like module.bootkube.content_hash)
  # to make the filename shorter, since there is no security nor collision risk anyways.
  #
  # Additionally, data sources do not support managing any lifecycle whatsoever,
  # and therefore, the archive is never deleted. To avoid cluttering the module
  # folder, we write it in the TerraForm managed hidden folder `.terraform`.
  output_path = "./.terraform/generated_${sha1("${module.tectonic.id} ${module.bootkube.id}")}.zip"
}

# Bootkube / Tectonic assets
resource "aws_s3_bucket_object" "tectonic-assets" {
  bucket = "${aws_s3_bucket.tectonic.bucket}"
  key    = "assets.zip"
  source = "${data.archive_file.assets.output_path}"
  acl    = "private"

  # To be on par with the current Tectonic installer, we only do server-side
  # encryption, using AES256. Eventually, we should start using KMS-based
  # client-side encryption.
  server_side_encryption = "AES256"
}

You can either import these into the new cluster state (to share secrets... which is a bad idea), or you can add the cluster name to the output directory of all assets (TLS /

Watch cluster come to life

1. Ensure api endpoint resolves

dig +short c01-api.tectonic.honestbee.jp

1. Watch for external api ELB when instances come into service

   or... to troubleshoot master

   • get public ip and ssh -i ~/path/to/key core@
   • on the first master node: journalctl -u bootkube -f
   • journalctl -u kubelet -f

2. Monitor cluster services booting...

KUBECONFIG=generated/auth/kubeconfig kubectl get po --all-namespaces -w

If you are using Tectonic license (free for up to 10 nodes) Ingress will exist for tectonic-console and tectonic-identity

open https://${CLUSTER}.${CLUSTER_DOMAIN}

LDAP Set up with Tectonic Identity

See CoreOS Instructions

Example LDAP set up using jumpcloud.com (for free - up to 10 users - need 1 user for ldap.service)

1. Create a user for ldap.service and give the user permission to Bind and search Jumpcloud LDAP service, also make sure to enable Jumpcloud LDAP in the Directories tab of this user

2. Create the users in Jumpcloud and make sure to enable Jumpcloud LDAP in Directories tab of each user

3. Create Group in Jumpcloud and make sure to enable Jumpcloud LDAP in Directories tab

4. If you do not create Linux Group, Jumpcloud will expose the group as (objectClass=groupOfNames) and you need to use the Distinguished Name (dn) attribute of the user to match the member attribute of the group

         userSearch:
           baseDN: 'ou=Users,o=<organization_id>,dc=jumpcloud,dc=com'
           filter: (objectClass=person)
           username: mail
           idAttr: uid
           emailAttr: mail
           nameAttr: cn
         groupSearch:
           baseDN: 'ou=Users,o=<organization_id>,dc=jumpcloud,dc=com'
           filter: (objectClass=groupOfNames)
           userAttr: DN
           groupAttr: member
           nameAttr: cn
   

5. If you do enable create Linux Group in jumpCloud, the group will exist as (objectClass=posixGroup) as well and you can use the uid attribute of the user to match the memberUid attribute of the group

         userSearch:
           baseDN: 'ou=Users,o=<organization_id>,dc=jumpcloud,dc=com'
           filter: (objectClass=person)
           username: mail
           idAttr: uid
           emailAttr: mail
           nameAttr: cn
         groupSearch:
           baseDN: 'ou=Users,o=<organization_id>,dc=jumpcloud,dc=com'
           filter: (objectClass=posixGroup)
           userAttr: uid
           groupAttr: memberUid
           nameAttr: cn
   

6. In tectonic console, create a Cluster Role Binding for Cluster Role admin to Subject: group and the name matching the posixGroup name

Tip: To troubleshoot the set up, download the OSX ldapsearch script or search ldap manually as follows (using TLS and port 636):

export ORG_ID="555..."
export LDAP_SERVICE_DN="uid=ldap.service,ou=Users,o=${ORG_ID},dc=jumpcloud,dc=com"
export LDAP_SERVICE_PASSWORD='...'

ldapsearch -H ldaps://ldap.jumpcloud.com:636 -x -b "ou=Users,o=${ORG_ID},dc=jumpcloud,dc=com" -D ${LDAP_SERVICE_DN} -w $LDAP_SERVICE_PASSWORD "(objectClass=person)"

ldapsearch -H ldaps://ldap.jumpcloud.com:636 -x -b "ou=Users,o=${ORG_ID},dc=jumpcloud,dc=com" -D ${LDAP_SERVICE_DN} -w $LDAP_SERVICE_PASSWORD "(objectClass=posixGroup)"

GitHub Set up with Tectonic Identity

Enable GitHub authentication with Tectonic Identity ...

Sample deployment on k8s

Download your cluster config file from console or keep using the kubelet config file...

export KUBECONFIG=generated/auth/kubeconfig 
kubectl run --image=quay.io/coreos/example-app:v1.0 simple-deployment --port=80
po=$(kubectl get po -l run=simple-deployment -o name | head -n 1)
kubectl port-forward ${po##*/} 8080:80
open http://localhost:8080

expose service using ELB (old way)

KUBECONFIG=generated/auth/kubeconfig kubectl expose deploy simple-deployment --target-port=80 --type=LoadBalancer

Tectonic vs Vanilla

Note tectonic out of the box deployments:

• container-linux-update-operator (deployment)
• container-linux-update-agent-ds (daemonset)
• heapster
• node-exporter (daemonset)
• pod-checkpointer
• kube-state-metrics
• prometheus-operator
• tectonic-console
• tectonic-identity
• tectonic-ingress-controller
• tectonic-stats-emitter

Vanilla:

• kube-controller-manager
• kube-dns
• kube-scheduler
• pod-checkpointer (makes snapshots of manifest to node /etc/kubernetes/manifests for fast recovery)

Missing nice to have (will need to configure to bootstrap by ourselves):

• container-linux-update-operator (Deployment)
• container-linux-update-agent-ds (deamonset)
• heapster (to run auto scalers?)
• kube-state-metrics (we can get from DataDog chart)
• no identity manager set up (need to set up oidc with google auth or dex for GitHub)

Other stuff we need to add (in any case):

• ALB ingress controller
• cluster-autoscaler

Default Tectonic RBAC roles:

• admin
• cluster-admin
• edit
• user
• view
• kube-state-metrics
• prometheus-k8s
• prometheus-operator
• extension-apiserver-authentication-reader

Tectonic

kubectl get roles,rolebindings --all-namespaces
NAMESPACE     NAME                                              AGE
kube-public   roles/system:controller:bootstrap-signer          8h
kube-system   roles/extension-apiserver-authentication-reader   8h
kube-system   roles/system:controller:bootstrap-signer          8h
kube-system   roles/system:controller:token-cleaner             8h

NAMESPACE     NAME                                              AGE
kube-public   rolebindings/system:controller:bootstrap-signer   8h
kube-system   rolebindings/system:controller:bootstrap-signer   8h
kube-system   rolebindings/system:controller:token-cleaner      8h

kubectl get clusterroles,clusterrolebindings
NAME                                                        AGE
clusterroles/admin                                          8h
clusterroles/cluster-admin                                  8h
clusterroles/edit                                           8h
clusterroles/kube-state-metrics                             8h
clusterroles/prometheus-k8s                                 8h
clusterroles/prometheus-operator                            8h
clusterroles/system:auth-delegator                          8h
clusterroles/system:basic-user                              8h
clusterroles/system:controller:attachdetach-controller      8h
clusterroles/system:controller:certificate-controller       8h
clusterroles/system:controller:cronjob-controller           8h
clusterroles/system:controller:daemon-set-controller        8h
clusterroles/system:controller:deployment-controller        8h
clusterroles/system:controller:disruption-controller        8h
clusterroles/system:controller:endpoint-controller          8h
clusterroles/system:controller:generic-garbage-collector    8h
clusterroles/system:controller:horizontal-pod-autoscaler    8h
clusterroles/system:controller:job-controller               8h
clusterroles/system:controller:namespace-controller         8h
clusterroles/system:controller:node-controller              8h
clusterroles/system:controller:persistent-volume-binder     8h
clusterroles/system:controller:pod-garbage-collector        8h
clusterroles/system:controller:replicaset-controller        8h
clusterroles/system:controller:replication-controller       8h
clusterroles/system:controller:resourcequota-controller     8h
clusterroles/system:controller:route-controller             8h
clusterroles/system:controller:service-account-controller   8h
clusterroles/system:controller:service-controller           8h
clusterroles/system:controller:statefulset-controller       8h
clusterroles/system:controller:ttl-controller               8h
clusterroles/system:discovery                               8h
clusterroles/system:heapster                                8h
clusterroles/system:kube-aggregator                         8h
clusterroles/system:kube-controller-manager                 8h
clusterroles/system:kube-dns                                8h
clusterroles/system:kube-scheduler                          8h
clusterroles/system:node                                    8h
clusterroles/system:node-bootstrapper                       8h
clusterroles/system:node-problem-detector                   8h
clusterroles/system:node-proxier                            8h
clusterroles/system:persistent-volume-provisioner           8h
clusterroles/user                                           8h
clusterroles/view                                           8h
NAME                                                               AGE
clusterrolebindings/admin-user                                     8h
clusterrolebindings/cluster-admin                                  8h
clusterrolebindings/devops                                         6h
clusterrolebindings/discovery                                      8h
clusterrolebindings/kube-state-metrics                             8h
clusterrolebindings/prometheus-k8s                                 8h
clusterrolebindings/prometheus-operator                            8h
clusterrolebindings/system:basic-user                              8h
clusterrolebindings/system:controller:attachdetach-controller      8h
clusterrolebindings/system:controller:certificate-controller       8h
clusterrolebindings/system:controller:cronjob-controller           8h
clusterrolebindings/system:controller:daemon-set-controller        8h
clusterrolebindings/system:controller:deployment-controller        8h
clusterrolebindings/system:controller:disruption-controller        8h
clusterrolebindings/system:controller:endpoint-controller          8h
clusterrolebindings/system:controller:generic-garbage-collector    8h
clusterrolebindings/system:controller:horizontal-pod-autoscaler    8h
clusterrolebindings/system:controller:job-controller               8h
clusterrolebindings/system:controller:namespace-controller         8h
clusterrolebindings/system:controller:node-controller              8h
clusterrolebindings/system:controller:persistent-volume-binder     8h
clusterrolebindings/system:controller:pod-garbage-collector        8h
clusterrolebindings/system:controller:replicaset-controller        8h
clusterrolebindings/system:controller:replication-controller       8h
clusterrolebindings/system:controller:resourcequota-controller     8h
clusterrolebindings/system:controller:route-controller             8h
clusterrolebindings/system:controller:service-account-controller   8h
clusterrolebindings/system:controller:service-controller           8h
clusterrolebindings/system:controller:statefulset-controller       8h
clusterrolebindings/system:controller:ttl-controller               8h
clusterrolebindings/system:default-sa                              8h
clusterrolebindings/system:discovery                               8h
clusterrolebindings/system:kube-controller-manager                 8h
clusterrolebindings/system:kube-dns                                8h
clusterrolebindings/system:kube-scheduler                          8h
clusterrolebindings/system:node                                    8h
clusterrolebindings/system:node-proxier                            8h

Vanila k8s RBAC roles:

kubectl get roles,rolebindings --all-namespaces 
NAMESPACE     NAME                                              AGE
kube-public   roles/system:controller:bootstrap-signer          3h
kube-system   roles/extension-apiserver-authentication-reader   3h
kube-system   roles/system:controller:bootstrap-signer          3h
kube-system   roles/system:controller:token-cleaner             3h
NAMESPACE     NAME                                              AGE
kube-public   rolebindings/system:controller:bootstrap-signer   3h
kube-system   rolebindings/system:controller:bootstrap-signer   3h
kube-system   rolebindings/system:controller:token-cleaner      3h



kubectl get clusterroles,clusterrolebindings
NAME                                                        AGE
clusterroles/admin                                          3h
clusterroles/cluster-admin                                  3h
clusterroles/edit                                           3h
clusterroles/system:auth-delegator                          3h
clusterroles/system:basic-user                              3h
clusterroles/system:controller:attachdetach-controller      3h
clusterroles/system:controller:certificate-controller       3h
clusterroles/system:controller:cronjob-controller           3h
clusterroles/system:controller:daemon-set-controller        3h
clusterroles/system:controller:deployment-controller        3h
clusterroles/system:controller:disruption-controller        3h
clusterroles/system:controller:endpoint-controller          3h
clusterroles/system:controller:generic-garbage-collector    3h
clusterroles/system:controller:horizontal-pod-autoscaler    3h
clusterroles/system:controller:job-controller               3h
clusterroles/system:controller:namespace-controller         3h
clusterroles/system:controller:node-controller              3h
clusterroles/system:controller:persistent-volume-binder     3h
clusterroles/system:controller:pod-garbage-collector        3h
clusterroles/system:controller:replicaset-controller        3h
clusterroles/system:controller:replication-controller       3h
clusterroles/system:controller:resourcequota-controller     3h
clusterroles/system:controller:route-controller             3h
clusterroles/system:controller:service-account-controller   3h
clusterroles/system:controller:service-controller           3h
clusterroles/system:controller:statefulset-controller       3h
clusterroles/system:controller:ttl-controller               3h
clusterroles/system:discovery                               3h
clusterroles/system:heapster                                3h
clusterroles/system:kube-aggregator                         3h
clusterroles/system:kube-controller-manager                 3h
clusterroles/system:kube-dns                                3h
clusterroles/system:kube-scheduler                          3h
clusterroles/system:node                                    3h
clusterroles/system:node-bootstrapper                       3h
clusterroles/system:node-problem-detector                   3h
clusterroles/system:node-proxier                            3h
clusterroles/system:persistent-volume-provisioner           3h
clusterroles/view                                           3h
NAME                                                               AGE
clusterrolebindings/cluster-admin                                  3h
clusterrolebindings/kubernetes-dashboard                           7m
clusterrolebindings/system:basic-user                              3h
clusterrolebindings/system:controller:attachdetach-controller      3h
clusterrolebindings/system:controller:certificate-controller       3h
clusterrolebindings/system:controller:cronjob-controller           3h
clusterrolebindings/system:controller:daemon-set-controller        3h
clusterrolebindings/system:controller:deployment-controller        3h
clusterrolebindings/system:controller:disruption-controller        3h
clusterrolebindings/system:controller:endpoint-controller          3h
clusterrolebindings/system:controller:generic-garbage-collector    3h
clusterrolebindings/system:controller:horizontal-pod-autoscaler    3h
clusterrolebindings/system:controller:job-controller               3h
clusterrolebindings/system:controller:namespace-controller         3h
clusterrolebindings/system:controller:node-controller              3h
clusterrolebindings/system:controller:persistent-volume-binder     3h
clusterrolebindings/system:controller:pod-garbage-collector        3h
clusterrolebindings/system:controller:replicaset-controller        3h
clusterrolebindings/system:controller:replication-controller       3h
clusterrolebindings/system:controller:resourcequota-controller     3h
clusterrolebindings/system:controller:route-controller             3h
clusterrolebindings/system:controller:service-account-controller   3h
clusterrolebindings/system:controller:service-controller           3h
clusterrolebindings/system:controller:statefulset-controller       3h
clusterrolebindings/system:controller:ttl-controller               3h
clusterrolebindings/system:default-sa                              3h
clusterrolebindings/system:discovery                               3h
clusterrolebindings/system:kube-controller-manager                 3h
clusterrolebindings/system:kube-dns                                3h
clusterrolebindings/system:kube-scheduler                          3h
clusterrolebindings/system:node                                    3h
clusterrolebindings/system:node-proxier                            3h

in Vanilla:

kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.1/src/deploy/kubernetes-dashboard.yaml

Dashboard vs Tectonic console:

• dashboard does not provide an interface for Roles and RoleBindings (will just need to manage those as code and through terminal)

Kops default roles:

kubectl get roles --all-namespaces
NAMESPACE     NAME                                        AGE
kube-public   system:controller:bootstrap-signer          4m
kube-system   extension-apiserver-authentication-reader   4m
kube-system   system:controller:bootstrap-signer          4m
kube-system   system:controller:token-cleaner             4m

Summary of commands to provision cluster

export DOMAIN=honestbee.jp
export CLUSTER_DOMAIN=tectonic.${DOMAIN}
export INSTALLER_PATH=$(pwd)/tectonic-installer/darwin/installer
export PATH=$PATH:$(pwd)/tectonic-installer/darwin
sed "s|<PATH_TO_INSTALLER>|$INSTALLER_PATH|g" terraformrc.example > .terraformrc
export TERRAFORM_CONFIG=$(pwd)/.terraformrc
terraform get platforms/aws
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_REGION=ap-northeast-1
export CLUSTER=c01
mkdir -p build/${CLUSTER}
cp examples/terraform.tfvars.aws build/${CLUSTER}/terraform.tfvars
#edit terraform.tfvars
code .
export KEY_NAME=c01-tectonic-honestbee-jp
aws --region ${AWS_REGION} ec2 create-key-pair --key-name ${KEY_NAME} --query 'KeyMaterial' --output text > ~/.ssh/${KEY_NAME}.pem
chmod 400 ~/.ssh/${KEY_NAME}.pem
ssh-add ~/.ssh/${KEY_NAME}.pem
ssh-keygen -y -f ~/.ssh/${KEY_NAME}.pem > ~/.ssh/${KEY_NAME}.pub
echo "~/.ssh/${KEY_NAME}.pub"
echo "${KEY_NAME}" | pbcopy #paste in terraform.tfvars
terraform plan -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws
terraform apply -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws
dig +short c01-api.tectonic.honestbee.jp
KUBECONFIG=generated/auth/kubeconfig kubectl get no

Notes on Tectonic bootstrapping process:

• If you have multiple masters, only 1 of them will be the bootkube master, the others won't run bootkube!
• Vanilla k8s is creating Load balancer for tectonic console, which is never used in vanilla deploys
• Tectonic runs AWS scripts using quay.io/coreos/awscli (which has vulnerabilities at the time of writing)
• the AWS Scripts on the masters:

  • get instance id from metadata service:

    curl http://169.254.169.254/latest/meta-data/instance-id
    

  • get autoscaling group name:

    autoscaling describe-auto-scaling-instances --region="ap-northeast-1" --instance-ids=...
    

  • loop until ASG is at desired capacity:

    aws autoscaling describe-auto-scaling-groups --region="$REGION" --auto-scaling-group-names=...
    jq ".AutoScalingGroups[0] .DesiredCapacity" == jq ".AutoScalingGroups[0] .Instances | sort_by(.InstanceId) | .[].InstanceId" | wc -l
    

  • Test if current instanceid == jq ".AutoScalingGroups[0] .Instances | sort_by(.InstanceId) | .[].InstanceId" | head -n1.

    Only if current instanceid is the first in the ASG, it will run bootkube...

Kops

Etcd disaster recovery

Practice Disaster Recovery

source

Destroy the cluster

Before destroying, get rid of ELBs

export KUBECONFIG=generated/auth/kubeconfig 
kubectl get services --all-namespaces -o json | jq -r ".items[] | select(.spec.type==\"LoadBalancer\") | .metadata.name"

terraform plan --destroy -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws
terraform destroy -var-file=build/${CLUSTER}/terraform.tfvars -state=build/${CLUSTER}/terraform.tfstate platforms/aws

Notice a few hiccups on resource management, could be terraform issue.