Set up a full Airflow installation on Windows using Hyper-V, Kubernetes and Helm

High-level steps

Prerequisites
Create the core node
Create the worker nodes
Bring up Kubernetes cluster
Install Airflow with Helm

Prerequisites

You'll need a reasonably high-end windows PC for this. Say 16 core/32 vcpu(thread) with >=64GB ram.
Ideally a router that you can use to assign dhcp reservations to specific mac addresses.
Install Hyper-V
Add an external virtual switch (If you have a second ethernet card, I'd recommend assigning it to that one)
I use a secondary (D:) drive for my virtual machines, I'll call that path %vroot% (mine is D:\virtualmachine)
I'll be using ubuntu 24.04 server as the VM distribution. Download the image here
A way of SSHing into your VMs. I use WSL.
Anything below contained in %percentsigns% should be substituted for your own information.
You should probably also have read some of the documentation!
- Airflow
- Microk8s
- Helm

Creating the core node

The steps for creating the core and worker nodes are very similar. I'll go through creating the core node, then for the workers, just list things to do differently. I find it very useful to have this core node active at all times for things like testing postgres and docker, outside of when I'm using airflow.

Open Hyper V
Create new VM in %vroot% with:
- Your chosen %machinename%
- Your chosen %vroot% location
- 1024GB expandable hard drive
- Generation 2
- 4096MB memory
- External Switch
- Virtual hard disk in %vroot%\%machinename%
- Install operating system later
Open settings for your new machine:
- Under Processor, change #cpus to 4
- Under Security, change Secure Boot to 'Microsoft UEFI Certificate Authority'
- Under SCSI controller, create a dvd drive then mount the ubuntu iso
- Click Apply (bottom right)
- Under Firmware, change DVD to top of boot order
- Under Checkpoints, disable automatic checkpoints (I disable checkpoints entirely)
- Click Ok
Connect to and start the VM
Install ubuntu using defaults except for:
- uncheck LVM in hdd options (use spacebar to toggle the checkbox)
- your name, username, servername and password
- Install openssh server
- Install microk8s
- Reboot
Log in using your username/password
- sudo su
- apt install nano net-tools
- ifconfig
- Take a note of the %nodeip% address for eth0
From WSL copy your public ssh key into the machine:
- scp ~/.ssh/id_rsa.pub %username%@%nodeip%:~/.ssh/authorized_keys
- Now is also a good time to reserve an IP address for this machine on your router
Back in the VM window, do:
- chmod 0644 ~/.ssh/authorized_keys
In WSL, ssh into the VM with
- ssh %nodeip%
- sudo su
- You can close the other window now, we're now going to use ssh so we can copy/paste
Disable password for sudo (not recommended but meh)
- sudo nano /etc/sudoers
  - Change ~third-to-last line (starting %sudo) to
  - %sudo ALL=(ALL:ALL) NOPASSWD: ALL
  - Save and exit nano
Disable swap
- swapoff -a
- nano /etc/fstab
  - comment out the last line (starting /swap.img)
  - save and exit nano
- rm /swap.img
Install utils
- apt update
- apt upgrade -y
- apt install -y python3-pip python3-virtualenv gnupg2 wget
Install postgres
- sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
- curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg
- apt install -y postgresql-16 postgresql-contrib-16
- sudo -u postgres psql postgres
  - ```
  create role airflow with superuser login password '%airflowdbpassword%';
  create database airflow with owner airflow;
  \q
```
- nano /etc/postgresql/16/main/pg_hba.conf
  - Add this line near the bottom
  - host all all 0.0.0.0/0 scram-sha-256
  - THIS IS INSECURE. DO NOT DO THIS IN PRODUCTION
- nano /etc/postgresql/16/main/postgresql.conf
  - uncomment listen_addresses = '*'
- service postgresql restart
(Optional) Install samba
- This bit is also not secure but gives you an easy way of transferring files to the VM
- apt install samba
- mkdir /home/share/
- chown %username%:%username% /home/share/`
- chmod 0777 /home/share/
- nano /etc/samba/smb.conf
  - Add this near the bottom
  - ```
  [public]
     comment = Share Directories
     browseable = yes
     writable = yes
     public = yes
     create mask = 0775
     directory mask = 0775
     path = /home/share
     guest ok = yes
```
- systemctl restart smbd
- You can now mount this share by going to "This PC" in Windows Explorer and clicking "Map network drive"
Allow the current user to use microk8s without sudo
- sudo usermod -a -G microk8s $USER
- mkdir -p ~/.kube
- chmod 0700 ~/.kube
Tidy up
- apt autoclean
- apt autoremove
- fstrim /
- poweroff
Now that we've turned off the VM, we can reduce the size of the virtual hard drive by doing:
- Open settings for the VM
- Under SCSI Controller
  - Select the hard drive and click edit
  - Click through the wizard selecting the option to compact the drive
- Whilst we're here, also under SCSI Controller, remove the dvd drive

Creating the worker nodes

Creating worker nodes is much the same as the core node, but with a few steps missing. Don't worry if you want more than 1 worker node, we can just create a template with one of them, then copy it for as many workers as we need. I suggest using a %machinename% of worker1, with %username% and %password% = worker.

So, repeat the above steps again but change:

(2) Create VM
- Use 1024MB memory (the default)
(3) VM Settings
- Use 2 CPUs
(5) Install Ubuntu
- %name%, %username%, %password% = worker
- %machinename% = worker1
- Select the "minimise" option when offered
(12) Skip the last line (starting apt install)
(13) Skip this step (postgres)
(14) Skip this step (samba)

Additional step, install containerd:

wget https://github.com/containerd/containerd/releases/download/v1.7.16/containerd-1.7.16-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.16-linux-amd64.tar.gz
sudo mkdir /etc/containerd
containerd config default > config.toml
sudo cp config.toml /etc/containerd
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo cp containerd.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
systemctl status containerd
poweroff

To create additional workers:

Make sure you have shut down the worker1 VM
Go into %vpath% and make a copy the worker folder, renaming it to something like "worker_template"
In the Hyper-V console:
- Click "Import Virtual Machine" to start the wizard
- Navigate to the copy of the worker (%vpath%"worker_template)
- In "Choose Import Type" select "make a copy of the virtual machine (create a new unique ID)"
- In "Choose Destination" check "Store the VM in a different location"
- Change all paths to %vpath%/workerX (where X is the number of the worker you're creating)
- Do the same in "Choose Storage Folders"

Set up the Kerbernetes cluster

At this point, to make life easy, I would do the following:

Copy my private ssh key to the core node
- scp ~/.ssh/id_rsa %username%@%ip%:~/.ssh/
ssh in to the core node
- chmod 0600 .ssh/id_rsa
Bring up a tmux session
- sudo apt install tmux
- tmux
Create a console (ctrl-b, ctrl-c) for each of the workers
For each of those consoles, ssh into the relevant worker
I can then switch between workers with ctrl-b,ctrl-n

Also, on the core node, set up the following aliases, so we don't have to keep typing in microk8s
alias kubectl='microk8s kubectl'
alias helm='microk8s helm'
And add these to ~/.bash_aliases so they persist to next login

On the core node, check microk8s is running:
- microk8s status --wait-ready
- kubectl get nodes
Let's add a node to the kubernetes cluster!
- On the core node, run
  - microk8s add-node
- Copy the connection string ending in --worker
  - it should look like
  - microk8s join 192.168.0.11:25000/018fc2062deaa9409be53babbc29c9d5/26f06c9315a2 --worker
- Switch to the next node to add then paste and run that command
- It might take a minute or so, but hopefully it should say
  - "Successfully joined the cluster"
- repeat this step for all of your nodes
Check the cluster is up and running
- kubectl get nodes

Install Airflow

Get the Airflow helm chart
- mkdir /home/share/airflow
- cd /home/share/airflow
- helm repo add apache-airflow https://airflow.apache.org
- helm repo update
The helm chart for airflow should now be stored in
- /home/alex/.cache/helm/repository/

There are a few things we need to override in the default airflow helm chart config

create a new config file:

nano overrides.yaml
add the following:

 postgresql:
   enabled: false

 data:
   metadataConnection:
     user: airflow
     pass: %airflowdbpassword%
     protocol: postgresql
     host: %corenodeip%
     port: 5432
     db: airflow
     sslmode: disable

Install airflow on the cluster
- helm install airflow apache-airflow/airflow --namespace airflow --create-namespace --debug -f overrides.yaml
- If you get a problem with timeouts, check that your database is running and you can connect to it as the airflow user
- If you can connect to the db from the core node, but not from a worker node or from WSL, then you've missed the pg_hba step.
- Also check that you've given the correct ip address and db password in the overrides.yaml.
- If anything goes wrong, uninstall with
  - kubectl delete airflow
  - kubectl delete namespace airflow
- and try again.

alexcthomas/airflow_on_windows_hyperv_kubernetes_helm.md