TCE 0.12.x and TKGm 1.5.x on vSphere - Static IP configuration

INSTRUCTIONS.md

Enabling Static IPs for VMs with TCE / TKGm on vSphere

This is experimental and your mileage may vary

Steps

1. Install TCE Management Cluster - not documented as nothing is different then usual
2. Install Metal3 IPAM Provider
3. Install CAPV IPAM Controller from spectrocloud
4. Add YTT Overlays
5. Create cluster config file
6. Deploy clusters

Install Metal3 IPAM Provider

kubectl create ns capm3-system
kubectl apply -f https://github.com/metal3-io/ip-address-manager/releases/download/v1.1.3/ipam-components.yaml

Install CAPV IPAM Controller from SpectroCloud

The IPAM controller from spectrocloud will be the bridge between CAPV and the Metal3 IPAM Controller. As SpectroCloud dont release images or manifests currently i have built the image and published it to a public GHCR registry and put a ready to use manifest in a seperate gist.

kubectl apply -f https://gist.githubusercontent.com/vrabbi/b20af526c091cced11495f578a5a3fc5/raw/128d922f9497272b952580d6e2e357020669a5db/capv-ipam-controller.yaml

Add YTT Overlays

In order to make use of static IPs we need to make some additions to the TCE config files.

1. Add an overlay to set dhcp configuration to false on vsphere machine templates which will make CAPV wait for the IPAM to allocate addresses

cat << EOF > ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-static-ip-overlay.yaml
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@ if data.values.USE_STATIC_IPS:
#@overlay/match by=overlay.subset({"kind": "VSphereMachineTemplate"}), expects="1+"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  #@overlay/match missing_ok=True
  labels:
    #@overlay/match missing_ok=True
    cluster.x-k8s.io/ip-pool-name: #@ data.values.CLUSTER_NAME
    #@overlay/match missing_ok=True
    cluster.x-k8s.io/network-name: #@ data.values.VSPHERE_NETWORK.split("/")[-1]
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.index(0)
        - dhcp4: false
#@ end
EOF

2. Create file adding in a few additional data values we will need in order to configure out static ip pools

cat << EOF > ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-static-ip-default-values.yaml
#@data/values

#@overlay/match-child-defaults missing_ok=True
---
USE_STATIC_IPS: false
FIRST_IP:
LAST_IP:
SUBNET_PREFIX: 24
DEFAULT_GATEWAY: 
DNS_SERVER: 8.8.8.8
EOF

3. Create File to create the needed IP Pool object per cluster

cat << EOF > ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-static-ip-ippool-addition.yaml
#@ load("@ytt:data", "data")
#@ if data.values.USE_STATIC_IPS:
---
apiVersion: ipam.metal3.io/v1alpha1
kind: IPPool
metadata:
  name: #@ data.values.CLUSTER_NAME
  namespace: #@ data.values.NAMESPACE
  labels:
    cluster.x-k8s.io/network-name: #@ data.values.VSPHERE_NETWORK.split("/")[-1]
spec:
  clusterName: #@ data.values.CLUSTER_NAME
  pools:
    - start: #@ data.values.FIRST_IP
      end: #@ data.values.LAST_IP
      prefix: #@ data.values.SUBNET_PREFIX
      gateway: #@ data.values.DEFAULT_GATEWAY
  prefix: #@ data.values.SUBNET_PREFIX
  gateway: #@ data.values.DEFAULT_GATEWAY
  namePrefix: #@ "ip-{}".format(data.values.CLUSTER_NAME)
  dnsServers: 
  - #@ data.values.DNS_SERVER
#@ end
EOF

Cretae a cluster config file

You create a cluster configuration file just like any other cluster with DHCP but then you add the following key/values to it:

USE_STATIC_IPS: true #! if not set the default is false and DHCP will be used
FIRST_IP: "" #! this needs to be the first ip from a range you want your machines to recieve for this cluster
LAST_IP: "" #! this needs to be the last ip from a range you want your machines to recieve for this cluster
SUBNET_PREFIX: "" #! Optional (Default is 24). This is the Subnet Mask Prefix of the network the VMs are being deployed on
DEFAULT_GATEWAY: "" #! The default gateway for the VM network
DNS_SERVER: "" #! Optional (Default is 8.8.8.8). This is the DNS server you want configured on your nodes.

Deploy the cluster

tanzu cluster create -f <CLUSTER CONFIG FILE>