Skip to content

Instantly share code, notes, and snippets.

@detiber
Last active May 17, 2018 14:37
Show Gist options
  • Save detiber/e18d907c41901fbb5e12ffa1af5750f8 to your computer and use it in GitHub Desktop.
Save detiber/e18d907c41901fbb5e12ffa1af5750f8 to your computer and use it in GitHub Desktop.
Testing external etcd upgrade

Testing external etcd upgrade

Insecure etcd

Using Ubuntu with kubeadm, kubelet, and kubectl v1.9.5 installed (specifically the libvirt vagrant box heptio/quickstart-ubuntu)

  • install v1.9.7

    sudo apt-get install -y curl software-properties-common
    sudo apt-add-repository ppa:lihiwish/etcd3-xenial
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install -y etcd kubelet=1.9.7-00 kubeadm=1.9.7-00 kubectl=1.9.7-00
    cat <<EOF > kubeadm-conf.yaml
    apiVersion: kubeadm.k8s.io/v1alpha1
    kind: MasterConfiguration
    etcd:
      endpoints:
      - http://127.0.0.1:2379
    EOF
    sudo kubeadm init --config kubeadm-conf.yaml
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    export kubever=$(kubectl version | base64 | tr -d '\n')
    kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"
    kubectl taint nodes --all node-role.kubernetes.io/master-
  • upgrade to v1.10.2

    sudo apt install kubeadm=1.10.2-00
    sudo kubeadm upgrade plan
    sudo kubeadm upgrade apply v1.10.2
    sudo apt install kubectl=1.10.2-00 kubelet=1.10.2-00
    sudo systemctl restart kubelet
    • Current Result:

      vagrant@ubuntu:~$ sudo kubeadm upgrade plan    
      [preflight] Running pre-flight checks.
      [upgrade] Making sure the cluster is healthy:
      [upgrade/config] Making sure the configuration is correct:
      [upgrade/config] Reading configuration from the cluster...
      [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
      could not read manifests from: /etc/kubernetes/manifests, error: failed to check if etcd pod implements TLS: failed to read manifest for "/etc/kubernetes/manifests/etcd.yaml": open /etc/kubernetes/manifests/etcd.yaml: no such file or directory
    • Using kubeadm from https://github.com/detiber/kubernetes/tree/external_etcd_1_10

      • Plan

        vagrant@ubuntu:~$ sudo ./kubeadm upgrade plan                                                                                                                                                
        [preflight] Running pre-flight checks.         
        [upgrade] Making sure the cluster is healthy:  
        [upgrade/config] Making sure the configuration is correct:                                    
        [upgrade/config] Reading configuration from the cluster...                                    
        [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'                                                                            
        [upgrade/plan] computing upgrade possibilities 
        [upgrade] Fetching available versions to upgrade to                                           
        [upgrade/versions] Cluster version: v1.9.7     
        [upgrade/versions] kubeadm version: v1.10.3-beta.0.11+97c21a54b50f66                          
        [upgrade/versions] Latest stable version: v1.10.2                                             
        [upgrade/versions] Latest version in the v1.9 series: v1.9.7                                  
        
        External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':                                                                      
        COMPONENT   CURRENT   AVAILABLE                
        Etcd        3.1.0     3.1.12                   
        
        Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':                                                                            
        COMPONENT   CURRENT      AVAILABLE             
        Kubelet     1 x v1.9.7   v1.10.2               
        
        Upgrade to the latest stable version:          
        
        COMPONENT            CURRENT   AVAILABLE       
        API Server           v1.9.7    v1.10.2         
        Controller Manager   v1.9.7    v1.10.2         
        Scheduler            v1.9.7    v1.10.2         
        Kube Proxy           v1.9.7    v1.10.2         
        Kube DNS             1.14.8    1.14.8          
        
        You can now apply the upgrade by executing the following command:                             
        
              kubeadm upgrade apply v1.10.2          
        
        Note: Before you can perform this upgrade, you have to update kubeadm to v1.10.2.             
        
        _____________________________________________________________________                         
      • Apply

        vagrant@ubuntu:~$ sudo ./kubeadm upgrade apply v1.10.2
        [preflight] Running pre-flight checks.
        [upgrade] Making sure the cluster is healthy:
        [upgrade/config] Making sure the configuration is correct:
        [upgrade/config] Reading configuration from the cluster...
        [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
        [upgrade/version] You have chosen to change the cluster version to "v1.10.2"
        [upgrade/versions] Cluster version: v1.9.7
        [upgrade/versions] kubeadm version: v1.10.3-beta.0.11+97c21a54b50f66
        [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
        [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
        [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.10.2"...
        Static pod: kube-apiserver-ubuntu hash: 85853dacfc28ca73f8ecafc777840ce3
        Static pod: kube-controller-manager-ubuntu hash: 86d3edb6f8c3c83c3039223896aa3ce1
        Static pod: kube-scheduler-ubuntu hash: 14166f4128d60694f8e58670f97b1763
        [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests350538632"
        [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests350538632/kube-apiserver.yaml"
        [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests350538632/kube-controller-manager.yaml"
        [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests350538632/kube-scheduler.yaml"
        [certificates] Generated etcd/ca certificate and key.
        [certificates] Generated apiserver-etcd-client certificate and key.
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests807271751/kube-apiserver.yaml"
        [upgrade/staticpods] Waiting for the kubelet to restart the component
        Static pod: kube-apiserver-ubuntu hash: 85853dacfc28ca73f8ecafc777840ce3
        Static pod: kube-apiserver-ubuntu hash: 85853dacfc28ca73f8ecafc777840ce3
        Static pod: kube-apiserver-ubuntu hash: 85853dacfc28ca73f8ecafc777840ce3
        Static pod: kube-apiserver-ubuntu hash: 9456f3d1476d0fa3adc7a17b08cdea57
        [apiclient] Found 1 Pods for label selector component=kube-apiserver
        [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests807271751/kube-controller-manager.yaml"
        [upgrade/staticpods] Waiting for the kubelet to restart the component
        Static pod: kube-controller-manager-ubuntu hash: 86d3edb6f8c3c83c3039223896aa3ce1
        Static pod: kube-controller-manager-ubuntu hash: 9403ba0735cfafa48d7924f8165f263d
        [apiclient] Found 1 Pods for label selector component=kube-controller-manager
        [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests807271751/kube-scheduler.yaml"
        [upgrade/staticpods] Waiting for the kubelet to restart the component
        Static pod: kube-scheduler-ubuntu hash: 14166f4128d60694f8e58670f97b1763
        Static pod: kube-scheduler-ubuntu hash: 454b0ba64d68fe4ae13a2d1ca9625979
        [apiclient] Found 1 Pods for label selector component=kube-scheduler
        [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
        [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
        [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
        [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
        [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
        [addons] Applied essential addon: kube-dns
        [addons] Applied essential addon: kube-proxy
        
        [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.10.2". Enjoy!
        
        [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.

Secure etcd

Using Ubuntu with kubeadm, kubelet, and kubectl v1.9.5 installed (specifically the libvirt vagrant box heptio/quickstart-ubuntu)

  • install v1.9.7

    sudo apt install curl software-properties-common
    sudo add-apt-repository ppa:gophers/archive
    sudo apt-add-repository ppa:lihiwish/etcd3-xenial
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    sudo apt update
    sudo apt install golang-1.10-go etcd kubelet=1.9.7-00 kubeadm=1.9.7-00 kubectl=1.9.7-00
    export PATH=/usr/lib/go-1.10/bin:${PATH}:~/go/bin
    go get -u github.com/cloudflare/cfssl/cmd/...
    
    # Create the etcd CA
    mkdir etcd-ca
    pushd etcd-ca
    cfssl print-defaults csr > ca-csr.json
    cat << EOF > ca-config.json
    {
      "signing": {
        "default": {
          "expiry": "43800h"
        },
        "profiles": {
          "server": {
            "expiry": "43800h",
            "usages": [
              "signing",
              "key encipherment",
              "server auth"
            ]
          },
          "client": {
            "expiry": "43800h",
            "usages": [
              "signing",
              "key encipherment",
              "client auth"
            ]
          },
          "peer": {
            "expiry": "43800h",
            "usages": [
              "signing",
              "key encipherment",
              "server auth",
              "client auth"
            ]
          }
        }
      }
    }
    EOF
    cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    
    # Create the etcd server cert
    cat << EOF > server.json
    {
      "CN": "$(hostname -f)",
      "hosts": [
        "localhost",
        "127.0.0.1"
      ],
      "key": {
        "algo": "ecdsa",
        "size": 256
      }
    }
    EOF
    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server
    
    # Create the etcd peer cert
    cat << EOF > peer.json
    {
      "CN": "$(hostname -f)",
      "hosts": [
        "localhost",
        "127.0.0.1"
      ],
      "key": {
        "algo": "ecdsa",
        "size": 256
      }
    }
    EOF
    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer peer.json | cfssljson -bare peer
    
    # Create the etcd client cert
    cat << EOF > client.json
    {
      "CN": "$(hostname -f)",
      "key": {
        "algo": "ecdsa",
        "size": 256
      }
    }
    EOF
    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client
    popd
    
    sudo mkdir -p /etc/etcd
    sudo cp etcd-ca/ca.pem /etc/etcd/ca.crt
    for cert in server peer; do
      sudo cp etcd-ca/${cert}.pem /etc/etcd/${cert}.crt
      sudo cp etcd-ca/${cert}-key.pem /etc/etcd/${cert}.key
    done
    sudo chown -R etcd:etcd /etc/etcd
    
    cat <<EOF | sudo tee -a /etc/default/etcd
    ETCD_NAME=$(hostname)
    ETCD_CERT_FILE=/etc/etcd/server.crt
    ETCD_KEY_FILE=/etc/etcd/server.key
    ETCD_CLIENT_CERT_AUTH=true
    ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
    ETCD_PEER_CLIENT_CERT_AUTH=true
    ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
    ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
    ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
    ETCD_LISTEN_PEER_URLS=https://localhost:2380
    ETCD_LISTEN_CLIENT_URLS=https://localhost:2379
    ETCD_INITIAL_ADVERTISE_PEER_URLS=https://localhost:2380
    ETCD_INITIAL_CLUSTER="$(hostname)=https://localhost:2380"
    ETCD_INITIAL_CLUSTER_STATE=new
    ETCD_ADVERTISE_CLIENT_URLS=https://localhost:2379
    EOF
    
    sudo systemctl stop etcd
    sudo rm -rf /var/lib/etcd/default
    sudo systemctl start etcd
    
    sudo cp etcd-ca/ca.pem /etc/kubernetes/etcd-ca.crt
    sudo cp etcd-ca/client.pem /etc/kubernetes/etcd-client.crt
    sudo cp etcd-ca/client-key.pem /etc/kubernetes/etcd-client.key
    
    cat <<EOF > kubeadm-conf.yaml
    apiVersion: kubeadm.k8s.io/v1alpha1
    kind: MasterConfiguration
    etcd:
      endpoints:
        - https://127.0.0.1:2379
      caFile: /etc/kubernetes/etcd-ca.crt
      certFile: /etc/kubernetes/etcd-client.crt
      keyFile: /etc/kubernetes/etcd-client.key
    EOF
    sudo kubeadm init --config kubeadm-conf.yaml
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    export kubever=$(kubectl version | base64 | tr -d '\n')
    kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"
    kubectl taint nodes --all node-role.kubernetes.io/master-
    • upgrade to v1.10.2
    sudo apt install kubeadm=1.10.2-00
    sudo kubeadm upgrade plan
    sudo kubeadm upgrade apply v1.10.2
    sudo apt install kubectl=1.10.2-00 kubelet=1.10.2-00
    sudo systemctl restart kubelet
    • Current Result:

      vagrant@ubuntu:~$ sudo kubeadm upgrade plan    
      [preflight] Running pre-flight checks.
      [upgrade] Making sure the cluster is healthy:
      [upgrade/config] Making sure the configuration is correct:
      [upgrade/config] Reading configuration from the cluster...
      [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
      could not read manifests from: /etc/kubernetes/manifests, error: failed to check if etcd pod implements TLS: failed to read manifest for "/etc/kubernetes/manifests/etcd.yaml": open /etc/kubernetes/manifests/etcd.yaml: no such file or directory
      • Using kubeadm from https://github.com/detiber/kubernetes/tree/external_etcd_1_10

      • Plan

        vagrant@ubuntu:~$ sudo ./kubeadm upgrade plan                                                                                                                                                
        [preflight] Running pre-flight checks.         
        [upgrade] Making sure the cluster is healthy:  
        [upgrade/config] Making sure the configuration is correct:                                    
        [upgrade/config] Reading configuration from the cluster...                                    
        [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'                                                                            
        [upgrade/plan] computing upgrade possibilities 
        [upgrade] Fetching available versions to upgrade to                                           
        [upgrade/versions] Cluster version: v1.9.7     
        [upgrade/versions] kubeadm version: v1.10.3-beta.0.11+97c21a54b50f66                          
        [upgrade/versions] Latest stable version: v1.10.2                                             
        [upgrade/versions] Latest version in the v1.9 series: v1.9.7                                  
        
        External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':                                                                      
        COMPONENT   CURRENT   AVAILABLE                
        Etcd        3.1.0     3.1.12                   
        
        Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':                                                                            
        COMPONENT   CURRENT      AVAILABLE             
        Kubelet     1 x v1.9.7   v1.10.2               
        
        Upgrade to the latest stable version:          
        
        COMPONENT            CURRENT   AVAILABLE       
        API Server           v1.9.7    v1.10.2         
        Controller Manager   v1.9.7    v1.10.2         
        Scheduler            v1.9.7    v1.10.2         
        Kube Proxy           v1.9.7    v1.10.2         
        Kube DNS             1.14.8    1.14.8          
        
        You can now apply the upgrade by executing the following command:                             
        
              kubeadm upgrade apply v1.10.2          
        
        Note: Before you can perform this upgrade, you have to update kubeadm to v1.10.2.             
        
        _____________________________________________________________________                         
      • Apply

        vagrant@ubuntu:~$ sudo ./kubeadm upgrade apply v1.10.2                                        
        [preflight] Running pre-flight checks.         
        [upgrade] Making sure the cluster is healthy:  
        [upgrade/config] Making sure the configuration is correct:                                    
        [upgrade/config] Reading configuration from the cluster...                                    
        [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'                                                                            
        [upgrade/version] You have chosen to change the cluster version to "v1.10.2"                  
        [upgrade/versions] Cluster version: v1.9.7     
        [upgrade/versions] kubeadm version: v1.10.3-beta.0.11+97c21a54b50f66                          
        [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y                 
        [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]                                                                                 
        [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.10.2"...        
        Static pod: kube-apiserver-ubuntu hash: 761b9375c09ca68e1bfde114f3755875                      
        Static pod: kube-controller-manager-ubuntu hash: 24627a5bd0bd470320381e4d437c0bab             
        Static pod: kube-scheduler-ubuntu hash: 96944ce896a1ba4844bab386f40c0acc                      
        [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests949774848"                                                                           
        [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests949774848/kube-apiserver.yaml"                                       
        [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests949774848/kube-controller-manager.yaml"                     
        [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests949774848/kube-scheduler.yaml"                                       
        [certificates] Generated etcd/ca certificate and key.                                         
        [certificates] Generated apiserver-etcd-client certificate and key.                           
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests279255903/kube-apiserver.yaml"                                        
        [upgrade/staticpods] Waiting for the kubelet to restart the component                         
        Static pod: kube-apiserver-ubuntu hash: 761b9375c09ca68e1bfde114f3755875                      
        Static pod: kube-apiserver-ubuntu hash: b2cdff92d7243dfd2f56894b0c6bf531                      
        [apiclient] Found 1 Pods for label selector component=kube-apiserver                          
        [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!                        
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests279255903/kube-controller-manager.yaml"                      
        [upgrade/staticpods] Waiting for the kubelet to restart the component                         
        Static pod: kube-controller-manager-ubuntu hash: 24627a5bd0bd470320381e4d437c0bab             
        Static pod: kube-controller-manager-ubuntu hash: e5bd07bb06a751c539f2b83fd0391e19             
        [apiclient] Found 1 Pods for label selector component=kube-controller-manager                 
        [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!               
        [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests279255903/kube-scheduler.yaml"                                        
        [upgrade/staticpods] Waiting for the kubelet to restart the component                         
        Static pod: kube-scheduler-ubuntu hash: 96944ce896a1ba4844bab386f40c0acc                      
        Static pod: kube-scheduler-ubuntu hash: b4e42f52dd09a5166ee4c6c6c1306a4c                      
        [apiclient] Found 1 Pods for label selector component=kube-scheduler                          
        [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!                        
        [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace                                                                                   
        [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials                                               
        [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token                                                            
        [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster                                                                         
        [addons] Applied essential addon: kube-dns     
        [addons] Applied essential addon: kube-proxy   
        
        [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.10.2". Enjoy!                  
        
        [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.  
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment