Windows on Kube

NOTE: This is based on the gist https://gist.github.com/wsong/34beb37ebd5d22c1bf1266aaeea085d4. That gist was used to set up the Dockercon SF demo; this one is for Dockercon EU 2018.

The below instructions describe how to set up a cluster with one Linux manager and one Windows worker such that you can schedule Kubernetes pods with Windows images on the Windows worker.

These instructions are for a Windows Server 2019 node. You can create these nodes on Azure (as of November 27, 2018).

On Azure, I recommend VMs of size D4s v3 or larger (4 CPUs, 16GB memory)

Linux manager setup

Make sure that your Linux node has Docker version 18.09 or above.

First, set up your /etc/kubernetes/azure.json file. Write a file called /etc/kubernetes/azure.json on your Linux manager with the following contents:

{
    "tenantId": "<tenant ID>",
    "subscriptionId": "<subscription ID>",
    "aadClientId": "<client ID>",
    "aadClientSecret": "<client secret>",
    "resourceGroup": "<VM resource group>",
    "location": "<VM location>",
    "subnetName": "default",
    "securityGroupName": "<VM security group>",
    "vnetName": "<VM vnet>",
    "useManagedIdentityExtension": false,
    "useInstanceMetadata": true
}

<tenant ID>, <subscription ID>, <client ID>, and <client secret> correspond to an Azure service principal. If you don't have an Azure service principal already, ask Deep Debroy or Flavio Crisciani for these values (as of May 31, 2018).

Fill in all the <VM *> values with the appropriate values for your VM from the Azure portal.

Next, install UCP:

export NODE_IP=<Linux node private IP address>
export NODE_PUBLIC_IP=<Linux node public IP address>
# This version of UCP is a special build with Kube 1.13.0 and a tweak to
# the node inventory logic so that it can handle Windows Kube nodes
export UCP_VERSION=3.1.2-cc99ac1
# VM_SUBNET_CIDR should be the CIDR for your VM's subnet in Azure. It should be of the form `172.16.3.0/24`. 
# You can find it by clicking on your VM's "Virtual network/subnet" section in the Azure dashboard and 
# grabbing the "Address space" value.
export VM_SUBNET_CIDR='<VM subnet CIDR>'

docker login
docker run --rm dockereng/ucp:$UCP_VERSION images --list --image-version dev: | xargs -n 1 docker pull
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock \
    dockereng/ucp:$UCP_VERSION \
    install --swarm-port 3376 --force-insecure-tcp --admin-password orcaorcaorca --image-version dev: --host-address $NODE_IP --san $NODE_PUBLIC_IP --cloud-provider azure --pod-cidr $VM_SUBNET_CIDR

Windows setup

Installing Docker

Make sure that your Windows node has Docker version 18.09 or above. You can install Docker like this:

Enable-WindowsOptionalFeature -Online -FeatureName containers –All

Then restart the Windows machine and run this:

Invoke-WebRequest -uri https://download.docker.com/components/engine/windows-server/18.09/docker-18.09.0.zip -OutFile docker.zip -UseBasicParsing
Expand-Archive .\docker.zip $env:ProgramFiles -f
$env:Path += ";$env:programfiles\docker"
setx /M PATH "$env:Path"
dockerd --register-service
start-service docker
netsh advfirewall set allprofiles state off

Download Kube binaries

Now, set up Kube. These instructions are partially based on the ones in https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#cluster-subnet-def, although note that that link has a lot of extraneous instructions that won't be necessary for this demo.

Run the below commands in Powershell as an administrator:

$wc = New-Object net.webclient
$wc.Downloadfile("https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-windows-amd64.tar.gz", "C:\kubelet.tar.gz")

tar -xf C:\kubelet.tar.gz

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$wc2 = New-Object System.net.webclient
$wc2.Downloadfile("https://s3-us-west-2.amazonaws.com/ddebroywin1/Nov30-1/k.zip", "C:\k.zip")
Expand-Archive -Force C:\k.zip -DestinationPath C:\

cp C:\kubernetes\node\bin\kubelet.exe C:\k\kubelet.exe
cp C:\kubernetes\node\bin\kube-proxy.exe C:\k\kube-proxy.exe

Set-MpPreference -DisableRealtimeMonitoring $true
docker pull mcr.microsoft.com/windows/nanoserver:1809
# NOTE: This looks like a bug but is actually correct. The Dockerfile in
# C:/k/Dockerfile that we use below is hardcoded to use
# microsoft/nanoserver:1709, but we actually want to use this image.
docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:1709
cd C:/k/
docker build -t kubeletwin/pause .

Copy UCP certs

Now you need to manually copy over the UCP certs to the Windows node.

On the Linux manager, run:

docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/ca.pem

Write the output to C:\kubecerts\ca.pem on the Windows node. Do the same for cert.pem and key.pem, writing them to C:\kubecerts\cert.pem and C:\kubecerts\key.pem, respectively.

docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/cert.pem
docker run --rm -it -v ucp-node-certs:/kubecerts busybox cat /kubecerts/key.pem

Write out config files

Write the below file to C:\kubecerts\kubelet.conf on the Windows node. Note that you need to replace <Linux public IP address> with the public IP address of the Linux manager:

apiVersion: v1
kind: Config
clusters:
- name: ucp
  cluster:
    insecure-skip-tls-verify: true
    server: https://<Linux public IP address>:6443
users:
- name: kubelet
  user:
    client-certificate: C:\kubecerts\cert.pem
    client-key: C:\kubecerts\key.pem
contexts:
- context:
    cluster: ucp
    user: kubelet
  name: kubelet-context
current-context: kubelet-context

Write this file to C:\kubecerts\env.ps1:

$global:KubeDnsSearchPath = "default.svc.cluster.local"
$global:KubeDnsServiceIp = "10.96.0.10"
$global:KubeServiceCIDR = "10.96.0.0/16"
$global:MasterIP = "<Linux private IP address>"
$global:MasterSubnet = "<VM subnet CIDR>"
$global:KubeClusterCIDR = "<VM subnet CIDR>"

Replace <Linux private IP address> with the private IP of your Linux node and replace <VM subnet CIDR> with the same value that you used for VM_SUBNET_CIDR in the Linux section.

Write out azure.json file

Now, copy the azure.json file from your Linux node to C:\kubecerts\azure.json.

Start Kubelet and kube-proxy

Now start up the kubelet on the Windows node. Note that you need to replace <Windows private IP address> with the private IP of the Windows node:

$Env:NODE_IP = "<Windows private IP address>"
Import-Module c:\kubecerts\env.ps1
c:\k\kubeletstart.ps1

Now in a separate Powershell Window, start the Kube proxy:

$Env:NODE_IP = "<Windows private IP address>"
Import-Module c:\kubecerts\env.ps1
c:\k\kubeproxystart.ps1

Manually allocate IPs

Now go to the Azure console and go to your Windows machine. Click Networking > Network Interface > IP configurations > +Add. Create a new IP configuration (give it a name like ipconfig2) and choose Dynamic Allocation, Public IP address: Disabled. Repeat this process 3 or 4 times. This will allocate IPs to the Windows node that your pods can use.

Creating pods

Now go to the UCP web UI by visiting https://<Linux public IP>:443 and log into the UCP UI with username admin and password orcaorcaorca (or whatever you entered in the install command when you installed UCP on the Linux node). Click on "+ Create" in the Kubernetes menu in the left-hand nav.

Now deploy this pod by copying these contents into the text field and selecting "default" as the namespace:

apiVersion: v1
kind: Pod
metadata:
  name: testpod
spec:
  nodeSelector:
    beta.kubernetes.io/os: windows
  containers:
  - name: testpod
    image: mcr.microsoft.com/windows/nanoserver:1809
    command: ["cmd", "/c", "ping 127.0.0.1 -n 100"]

Now run go to Pods in the left-hand nav and observe that the Windows pod is running on the Windows node.

wsong/windows_on_kube_dceu.md