Skip to content

Instantly share code, notes, and snippets.

@wittawasw

wittawasw/readme.adoc

Forked from arun-gupta/readme.adoc
Created November 16, 2021 14:43
Show Gist options
  • Save wittawasw/b7dc2cef5fff546eb6dfef34e816b20d to your computer and use it in GitHub Desktop.
Save wittawasw/b7dc2cef5fff546eb6dfef34e816b20d to your computer and use it in GitHub Desktop.
Download ZIP
Federated Kubernetes Cluster using Kops on AWS
Raw
readme.adoc

Let’s set up a federation between earth and mars Kubernetes cluster on AWS.

Earth cluster

  1. Create hosted zone:

    ID=$(uuidgen) && \
   aws route53 create-hosted-zone \
   --name earth.kubernetes-aws.io \
   --caller-reference $ID \
   | jq .DelegationSet.NameServers

  2. Setup NS records in DNS provider. Make sure dig earth.kubernetes-aws.io NS return NS records:

    dig earth.kubernetes-aws.io NS

; <<>> DiG 9.8.3-P1 <<>> earth.kubernetes-aws.io NS
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16919
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;earth.kubernetes-aws.io.	IN	NS

;; ANSWER SECTION:
earth.kubernetes-aws.io. 3271	IN	NS	ns-752.awsdns-30.net.
earth.kubernetes-aws.io. 3271	IN	NS	ns-1196.awsdns-21.org.
earth.kubernetes-aws.io. 3271	IN	NS	ns-1799.awsdns-32.co.uk.
earth.kubernetes-aws.io. 3271	IN	NS	ns-436.awsdns-54.com.

;; ADDITIONAL SECTION:
ns-436.awsdns-54.com.	162638	IN	A	205.251.193.180
ns-752.awsdns-30.net.	161105	IN	A	205.251.194.240
ns-1196.awsdns-21.org.	161034	IN	A	205.251.196.172
ns-1799.awsdns-32.co.uk. 148088	IN	A	205.251.199.7

;; Query time: 13 msec
;; SERVER: 10.4.4.10#53(10.4.4.10)
;; WHEN: Fri Aug 25 14:44:22 2017
;; MSG SIZE  rcvd: 245

  3. Create cluster:

    kops create cluster \
  --name=earth.kubernetes-aws.io \
  --master-count=3 \
  --master-zones us-east-1a,us-east-1b,us-east-1c \
  --node-count=3 \
  --zones=us-east-1a,us-east-1b,us-east-1c \
  --authorization=rbac \
  --state=s3://kubernetes-aws-io \
  --kubernetes-version=1.7.0 \
  --yes

  4. Validate cluster:

    kops validate cluster earth.kubernetes-aws.io --state=s3://kubernetes-aws-io
Validating cluster earth.kubernetes-aws.io

INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	m3.medium	1	1	us-east-1a
master-us-east-1b	Master	m3.medium	1	1	us-east-1b
master-us-east-1c	Master	c4.large	1	1	us-east-1c
nodes			Node	t2.medium	3	3	us-east-1a,us-east-1b,us-east-1c

NODE STATUS
NAME				ROLE	READY
ip-172-20-117-87.ec2.internal	node	True
ip-172-20-125-87.ec2.internal	master	True
ip-172-20-45-153.ec2.internal	master	True
ip-172-20-55-15.ec2.internal	node	True
ip-172-20-64-27.ec2.internal	master	True
ip-172-20-76-15.ec2.internal	node	True

Your cluster earth.kubernetes-aws.io is ready

  5. Set alias for context:

    kubectl config set-context earth --cluster=earth.kubernetes-aws.io --user=earth.kubernetes-aws.io
Context "earth" created.

  6. Get nodes:

    kubectl --context=earth get nodes
NAME                            STATUS    AGE       VERSION
ip-172-20-117-87.ec2.internal   Ready     6m        v1.7.0
ip-172-20-125-87.ec2.internal   Ready     7m        v1.7.0
ip-172-20-45-153.ec2.internal   Ready     8m        v1.7.0
ip-172-20-55-15.ec2.internal    Ready     6m        v1.7.0
ip-172-20-64-27.ec2.internal    Ready     7m        v1.7.0
ip-172-20-76-15.ec2.internal    Ready     6m        v1.7.0

Mars cluster

  1. Create hosted zone:

    ID=$(uuidgen) && \
   aws route53 create-hosted-zone \
   --name mars.kubernetes-aws.io \
   --caller-reference $ID \
   | jq .DelegationSet.NameServers

  2. Setup NS records in DNS provider. Make sure dig mars.kubernetes-aws.io NS return NS records:

    dig mars.kubernetes-aws.io NS

; <<>> DiG 9.8.3-P1 <<>> mars.kubernetes-aws.io NS
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64774
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;mars.kubernetes-aws.io.		IN	NS

;; ANSWER SECTION:
mars.kubernetes-aws.io.	2630	IN	NS	ns-26.awsdns-03.com.
mars.kubernetes-aws.io.	2630	IN	NS	ns-964.awsdns-56.net.
mars.kubernetes-aws.io.	2630	IN	NS	ns-1052.awsdns-03.org.
mars.kubernetes-aws.io.	2630	IN	NS	ns-1965.awsdns-53.co.uk.

;; ADDITIONAL SECTION:
ns-26.awsdns-03.com.	143078	IN	A	205.251.192.26
ns-964.awsdns-56.net.	157684	IN	A	205.251.195.196
ns-1052.awsdns-03.org.	23776	IN	A	205.251.196.28
ns-1965.awsdns-53.co.uk. 75558	IN	A	205.251.199.173

;; Query time: 15 msec
;; SERVER: 10.4.4.10#53(10.4.4.10)
;; WHEN: Fri Aug 25 14:44:43 2017
;; MSG SIZE  rcvd: 243

  3. Create cluster

    kops create cluster \
  --name=mars.kubernetes-aws.io \
  --master-count=3 \
  --master-zones us-east-2a,us-east-2b,us-east-2c \
  --node-count=3 \
  --zones=us-east-2a,us-east-2b,us-east-2c \
  --authorization=rbac \
  --state=s3://kubernetes-aws-io \
  --kubernetes-version=1.7.0 \
  --yes

  4. Validate

    kops validate cluster mars.kubernetes-aws.io --state=s3://kubernetes-aws-io
Validating cluster mars.kubernetes-aws.io

INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-2a	Master	c4.large	1	1	us-east-2a
master-us-east-2b	Master	c4.large	1	1	us-east-2b
master-us-east-2c	Master	c4.large	1	1	us-east-2c
nodes			Node	t2.medium	3	3	us-east-2a,us-east-2b,us-east-2c

NODE STATUS
NAME						ROLE	READY
ip-172-20-107-105.us-east-2.compute.internal	node	True
ip-172-20-126-49.us-east-2.compute.internal	master	True
ip-172-20-41-181.us-east-2.compute.internal	node	True
ip-172-20-62-64.us-east-2.compute.internal	master	True
ip-172-20-89-187.us-east-2.compute.internal	node	True
ip-172-20-89-96.us-east-2.compute.internal	master	True

Your cluster mars.kubernetes-aws.io is ready

  5. Set alias for context:

    kubectl config set-context mars --cluster=mars.kubernetes-aws.io --user=mars.kubernetes-aws.io
Context "mars" modified.

  6. Get nodes:

    kubectl --context=mars get nodes
NAME                                           STATUS    AGE       VERSION
ip-172-20-107-105.us-east-2.compute.internal   Ready     9m        v1.7.0
ip-172-20-126-49.us-east-2.compute.internal    Ready     10m       v1.7.0
ip-172-20-41-181.us-east-2.compute.internal    Ready     9m        v1.7.0
ip-172-20-62-64.us-east-2.compute.internal     Ready     10m       v1.7.0
ip-172-20-89-187.us-east-2.compute.internal    Ready     9m        v1.7.0
ip-172-20-89-96.us-east-2.compute.internal     Ready     10m       v1.7.0

Setup federation

  1. Download k8s client binary:

    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-darwin-amd64.tar.gz
tar xzvf kubernetes-client-darwin-amd64.tar.gz

  2. Check context:

    kubectl config get-contexts
CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
          earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
          mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
*         mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
          earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io

  3. Use earth as host cluster:

    kubectl config use-context earth
Switched to context "earth".

  4. Create RBAC role binding:

    kubectl create \
   clusterrolebinding \
   admin-to-cluster-admin-binding \
   --clusterrole=cluster-admin \
   --user=admin

  5. Deploy the federation control plane in host cluster:

    kubefed \
    init \
    interstellar \
    --host-cluster-context=earth \
    --dns-provider=aws-route53 \
    --dns-zone-name=kubernetes-aws.io

    Shows the output:

    Creating a namespace federation-system for federation system components... done
Creating federation control plane service..... done
Creating federation control plane objects (credentials, persistent volume claim)... done
Creating federation component deployments... done
Updating kubeconfig... done
Waiting for federation control plane to come up.......................................................... done
Federation API server is running at: aadbdf9358d1411e7a4ba0a49363c58c-672820986.us-east-1.elb.amazonaws.com

  6. Get contexts again to see the newly created namespace:

    kubectl config get-contexts
CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
          mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
          mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
*         earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io
          earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
          interstellar              interstellar              interstellar

  7. Change the context to federation context:

    kubectl config use-context interstellar

  8. Join earth and mars cluster to the federation:

    kubefed join \
   earth \
   --host-cluster-context=earth \
   --cluster-context=earth
kubefed join \
   mars \
   --host-cluster-context=earth \
   --cluster-context=mars

    --cluster-context defaults to cluster name

  9. Check status of the clusters in the federation:

    kubectl --context=interstellar get clusters
NAME      STATUS    AGE
earth     Ready     4h
mars      Ready     3h

Cleanup

  1. Unjoin earth and mars cluster from federation:

    kubefed unjoin earth --cluster-context=earth
kubefed unjoin mars --cluster-context=earth

  2. Delete clusters

    kops delete cluster --name=earth.kubernetes-aws.io --yes
kops delete cluster --name=mars.kubernetes-aws.io --yes

Misc

  1. Use aws ec2 describe-availability-zones --region=us-east-1 to find out a region with 3+ AZ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment