etcd-v3migrate

Manual data migration of the etcd cluster

In case the migration fails (for some reason), we can finish the migration manually. Depending on a point in which the migration fails we will need the following commands (followed by all the remaining commands):

    Before the etcd migration started:
    In this case it is recommended to re-run the migration playbook again. The cluster can end up with master services stopped. They must be started and running before the migration is repeated.

    Before the first member got migrated:
    Before the command is run the etcd service must be stopped.
    Raw

    ETCDCTL_API=3 /usr/bin/etcdctl migrate --data-dir=/var/lib/etcd

    The command is expected to respond with finished transforming keys message.

    Before the second member got added to the cluster:
    Add the second member (run on a host of the second etcd member)
    Raw

    /usr/bin/etcdctl --cert-file /etc/etcd/peer.crt \
        --key-file /etc/etcd/peer.key \
        --ca-file /etc/etcd/ca.crt \
        -C https://$IP1:2379 member add $host2 https://$IP2:2380

    Before the third member got added to the cluster:
    Add the third member (run on a host of the third etcd member)
    Raw

    /usr/bin/etcdctl --cert-file /etc/etcd/peer.crt \
        --key-file /etc/etcd/peer.key \
        --ca-file /etc/etcd/ca.crt \
        -C https://$IP1:2379 member add $host3 https://$IP3:2380

The /usr/bin/etcdctl member add command will print a list of environment variables that need to be updated (or set) in the /etc/etcd/etcd.conf file.
If the etcd cluster has more than three members, the step for the next etcd member(s) is analogous to the step for the third member.
Each time a new member is added, the etcd service must be started ASAP so the new member can join the cluster.
It is expected the new etcd members are cleaned of the previous content of their data (e.g. by removing content of the /var/lib/etcd/member directory.
Re-attachment of leases

The concept of TTLs of keys in v2 data model has been re-designed in v3 data model. Given the data migration does not migrate TTLs, they need to be migrated separately.
The OpenShift has the following list of important keys whose TTLs need to be set separately (with default value of TTL in parenthesis):

    /kubernetes.io/events (1h)
    /kubernetes.io/masterleases (10s)
    /openshift.io/oauth/accesstokens (86400s)
    /openshift.io/oauth/authorizetokens (500s)
    /openshift.io/leases/controllers (30s)

The TTL of the /openshift.io prefixed keys in the list may vary depending on the current configuration in the /etc/origin/master/master-config.yml file.
Current migration playbook takes the current TTL values from the master configuration file and applies them during the migration (replacing the default TTL values in the list).
The TTL values and their location:

    /openshift.io/oauth/accesstokens under oauthConfig.tokenConfig.accessTokenMaxAgeSeconds
    /openshift.io/oauth/authorizetokens under oauthConfig.tokenConfig.authorizeTokenMaxAgeSeconds
    /openshift.io/leases/controllers under controllerLeaseTTL

All the TTLs are migrated by running the following command on the first master:
Raw

oadm migrate etcd-ttl \
    --cert /etc/origin/master/master.etcd-client.crt \
    --key /etc/origin/master/master.etcd-client.key \
    --cacert /etc/origin/master/master.etcd-ca.crt \
    --etcd-address 'https://$IP:2379' \
    --ttl-keys-prefix <KEY> \
    --lease-duration <TTL>

where , resp. is key from the list of important keys, resp. value of the TTL.
For example, if the TTL of the /openshift.io/leases/controllers will be set to 50s, you can run:
Raw

oadm migrate etcd-ttl \
    --cert /etc/origin/master/master.etcd-client.crt \
    --key /etc/origin/master/master.etcd-client.key \
    --cacert /etc/origin/master/master.etcd-ca.crt \
    --etcd-address 'https://$IP:2379' \
    --ttl-keys-prefix "/openshift.io/leases/controllers" \
    --lease-duration "50s"

Re-configuration of all masters

The following keys in the master configuration file need to be set to switch to etcd v3 backend:
Raw

kubernetesMasterConfig:
    apiServerArguments
        storage-backend:
            - etcd3
        storage-media-type:
            - application/vnd.kubernetes.protobuf

The configuration needs to be updated on each master host.
Start of all master services

To start master run the following command on all master hosts (after the master configuration got updated):
Raw

systemctl start atomic-openshift-master-api atomic-openshift-master-controllers