Background

Falco

Falco is an open source product for container runtime security.

Installation for Kubernetes

There is a very simple way using Daemonset. But, it does not work in AKS (Azure Kubernetes Service) out of the box.

Reason

Falco's container (falcosecurity/falco) gets a suitable driver file from Amazon S3 but there is no file to fit with the AKS worker nodes.

Workaround

According to the Falco's instrallation instruction, we need to build our own driver which is suitable for AKS.

Build driver

There is an instruction to build Falco from source here: https://falco.org/docs/source/

Falco builder container

A container image for building Falco is published as falcosecurity/falco-builder: https://hub.docker.com/r/falcosecurity/falco-builder

YAML to create a pod with infinite loop

apiVersion: v1
kind: Pod
metadata:
  name: builder
  labels:
    app: falco-builder
spec:
  containers:
    - name: builder
      image: falcosecurity/falco-builder:latest
      command:
        - bash
        - -c
        - "tail -f /dev/null"
      volumeMounts:
        - mountPath: /host/usr
          name: host-usr
          readOnly: true
        - mountPath: /host/lib
          name: host-lib
          readOnly: true
        - mountPath: /host/boot
          name: host-boot
          readOnly: true
  volumes:
    - name: host-usr
      hostPath:
        path: /usr
    - name: host-lib
      hostPath:
        path: /lib
    - name: host-boot
      hostPath:
        path: /boot

Running a pod

Assume that the YAML written above is stored as builder-pod.yaml and kubectl is configured to connect to an AKS cluster.

$ kubectl apply -f builder-pod.yaml

Then, a "builder" pod will be created.

$ kubectl get po --watch
NAME                    READY   STATUS    RESTARTS   AGE
builder                 1/1     Running   0          14s

Entering into the builder container

$ kubectl exec -it builder -- bash

A shell prompt will be shown.

[root@builder /]#

Check kernel info

[root@builder /]# uname -r
4.15.0-1075-azure

Check mounted volumes

[root@builder /]# ls /host/usr/src
linux-azure-headers-4.15.0-1071  linux-azure-headers-4.15.0-1075  linux-headers-4.15.0-1071-azure  linux-headers-4.15.0-1075-azure
[root@builder /]# ls /host/lib/modules
4.15.0-1071-azure  4.15.0-1075-azure

Create symlinks

[root@builder /]# ln -s /host/usr/src/linux-headers-4.15.0-1075-azure /usr/src/linux-headers-4.15.0-1075-azure
[root@builder /]# ln -s /host/lib/modules/4.15.0-1075-azure /lib/modules/4.15.0-1075-azure

Create /source and /build directories

Builder tool (container's entrypoint) assumes that there are /source and /build directories.

[root@builder /]# mkdir /source
[root@builder /]# mkdir /build

Clone the Falco repository

[root@builder /]# cd /source
[root@builder /]# git clone https://github.com/falcosecurity/falco.git

Set an environment variable

To build a driver, we need to set the following variable:

[root@builder /]# export BUILD_DRIVER=ON

Environment variables information is shown by the following command:

[root@builder /]# /usr/bin/entrypoint usage

Building driver

[root@builder /]# /usr/bin/entrypoint cmake
[root@builder /]# /usr/bin/entrypoint driver

A "falco-probe.ko" file will be created in /build/release/driver.

[root@builder source]# ls -l /build/release/driver/
total 696
drwxr-xr-x 4 root root   4096 Mar 29 15:11 CMakeFiles
-rw-r--r-- 1 root root   7382 Mar 29 15:11 Makefile
drwxr-xr-x 3 root root   4096 Mar 29 15:11 bpf
-rw-r--r-- 1 root root   2966 Mar 29 15:11 cmake_install.cmake
-rw-r--r-- 1 root root 685416 Mar 29 15:14 falco-probe.ko
drwxr-xr-x 3 root root   4096 Mar 29 15:14 src

Exit from the container

[root@builder /]# exit

Copy the driver file from container

$ kubectl cp builder:/build/release/driver/falco-probe.ko falco-probe.ko

Distribute the driver file

The Falco container will originally get a driver file from Falco's Amazon S3 bucket. However, if you set an environment variable PROBE_URL, the container will download from your own location.

Create an Azure BLOB container

In this text, I use the following storage account and BLOB container:

Storage account: myfalcodriver
BLOB container: falco-driver

You should make the container (or uploaded files) as public because the Falco container download the file without authentication.

You may use the Shared Access Signature (SAS) feature of Azure BLOB but I have not tested yet.

Upload the driver file

File name

According to the "falco-probe-load" script in the Falco container, the container will download a driver file of the following name:

${PROBE_NAME}-${FALCO_VERSION}-${ARCH}-${KERNEL_RELEASE}-${HASH}.ko

PROBE_NAME is a constant value of "falco-probe".

If there is no preset value, FALCO_VERSION is set as $(falco --version). In this case, the value is "0.21.0". However, FALCO_VALUE is preset as "latest" in the container and this value is used to download.

ARCH is set as $(uname -m) and the value is "x86_64" for AKS nodes.

KERNEL_RELEASE is set as $(uname -r) and the value is "4.15.0-1075-azure" in this case as shown above.

HASH is an MD5 value of a file selected in the following order:

/proc/config.gz => No such file in this case.
/boot/config-$KERNEL_RELEASE => No such file in this case.
/${HOST_ROOT}/boot/config-$KERNEL_RELEASE (HOST_ROOT=host in the container) => Existing.
/${HOST_ROOT}/usr/lib/ostree-boot-config-${KERNEL_RELEASE}
/lib/modules/${KERNEL_RELEASE}/config

Hash value

The hash value can be obtained with the following command:

$ kubectl exec builder -- bash -c 'md5sum /host/boot/config-4.15.0-1075-azure'
8c08dfda78162ce1acd87575d5d37842  /host/boot/config-4.15.0-1075-azure

Upload by Azure CLI

Actually, only the "latest" file will be used by the Falco container, we upload both the "0.21.0" (actual version number) and "latest" files to easily distinguish which version is "latest".

The container will download the driver file from stable/sysdig-probe-binaries/ directory. The followings are the Azure CLI commands to upload.

$ az storage blob upload --account-name myfalcodriver --container-name falco-driver --file falco-probe.ko --name stable/sysdig-probe-binaries/falco-probe-0.21.0-x86_64-4.15.0-1075-azure-8c08dfda78162ce1acd87575d5d37842.ko
$ az storage blob upload --account-name myfalcodriver --container-name falco-driver --file falco-probe.ko --name stable/sysdig-probe-binaries/falco-probe-latest-x86_64-4.15.0-1075-azure-8c08dfda78162ce1acd87575d5d37842.ko

Start or reload Falco's daemonset

Set the download URL in the daemonset manifest

Add the following environment variable in the daemonset manifest stored in the Falco repository. (You should change "myfalcodriver" and "falco-driver" to the actual values for you.)

You can see a full YAML in the Falco GitHub repository (integrations/k8s-using-daemonset/k8s-with-rbac/falco-daemonset-configmap.yaml) and the installation instruction for the daemonset in the page written at the top of this article.

          env:                                                                                                                                                      
          - name: PROBE_URL                                                                                                                                         
            value: "https://myfalcodriver.blob.core.windows.net/falco-driver"

Start daemonset

Assume that the YAML is saved as falco-daemonset-configmap-azure.yaml.

$ kubectl apply -f falco-daemonset-configmap-azure.yaml

If Falco starts correctly, you can see the Falco logs by (change xxxxx to the actual value):

$ kubectl logs falco-daemonset-xxxxx

Reload for the existing daemonset

If there is already a daemonset in your cluster, you can delete the existing pod to reload the new driver (change xxxxx to the actual value).

$ kubectl delete falco-daemonset-xxxxx

kazusato/building-falco-probe-driver-for-aks.md