mixed normal and preemtible kubernetes cluster

gistfile1.txt

# Kubernetes clusters using preemtible instances

## Motivation

People experimenting with kubernetes clusters on the GKE not
necessarily have money to keep a full cluster on at all time. GKE
clusters can be easily resized, but this still incurs in the full
instance cost when the cluster is up.

Google has added preemptible instances that are ideal for many
applications seeing spikes in usages or for simply experimenting with
scaling. Preemptible instances last 24 hours top and can be destroyed
by google w/ 30 seconds advance notice.

## Howto

Here's how I've got a kubernetes cluster using a mix of normal and
preemtible interface. It should be possible to have preemptible only cluster,
but I haven't tried it.

### Creating a preemtible pool

#### Create an additional pool
The fisrt step is to create a preemptible pool and make so instances
belonging to it join an existing kubernetes cluster.

gcloud container node-pools create preemptible-reserve --machine-type=n1-standard-1 --num-nodes=1

Unfortunately we need to create at least one VM here, as 0 is not accepted.

kubectl get node -o wide
NAME                                          STATUS    AGE
gke-CLUSTER-default-pool-0d56be95-f0eo         Ready     7d
gke-CLUSTER-default-pool-0d56be95-xt05         Ready     7d
gke-CLUSTER-preemtible-reserve-d5a5e03b-3kjt   Ready     42s

At this point we have two pools.

#### Make the pool preemptible

##### Make the template preentible

In the dashboard, go to instance template. Select and copy the
template containing preemtible-reserve in it. We need to copy because
otherwise we're not allowed to edit. This maybe because the template
is in use, I haven't verified.

While in the copy page open the tab Management/Disk/Network. The default tab
(Management) will have a availability policy/preemptability
setting. Switch it to On.

Click on create. The template name will be OLD_NAME-1.

If you go to the template page, you'll see that the old template is in
use by some group, while the new one is unused.

##### Make the pool use the new template
Click on the group used by the old template. Click on edit at the top.
Select the template and make the '-1' version the one used.

IMPORTANT: the page will warn you that this change only apply to newly created VMs and will have a link to a gcloud command that restart existing VMs with the new setting. Remember we have that one lone VM from when we create the new pool. You have two options:

- destroy the machine (see later). New machines will be preemptible.
- execute the command given in this page to convert the existing VM to
preemptible. The command looks like: gcloud compute --project
"PROJECT-NAME" instance-groups managed recreate-instances "GROUP"
--zone "us-east1-b" --instance "INSTANCE-NAME" so even if you didn't
write it down, you should be able to reproduce it.

Save the edited group

### Manage the premptible pool

#### "destroy" reserve
gcloud container clusters resize CLUSTER --node-pool preemptible-reserve --size 0

# resize reserve pool
gcloud container clusters resize iodine --node-pool reserve --size 2

# "destroy" reserve
gcloud container clusters resize iodine --node-pool reserve --size 0

# restart reserve
gcloud container clusters resize iodine --node-pool reserve --size 1