hackerzhut/pod-baseline.yaml

Overview

Proof of concept demo testing out my comments on the "Bare Minimum Pod Security" proposal:

Consider this a proof of concept of "Bare Medium" Pod Security

Three policy levels, mapping to Pod Security Standards levels:

privileged
baseline
restricted

Three actions that can be taken, expressed as a namespace label of podsecurity.kubernetes.io/<action>=<level>:

allow=<level> (reject pods that exceed <level>)
warn=<level> (warn the user when creating a pod that exceeds <level>)
audit=<level> (add info to the audit log when creating a pod that exceeds <level>)

Ability to indicate the version of the policy

as it existed at a particular Kubernetes version should be used:
- podsecurity.kubernetes.io/<action>.version=v1.7
or the latest incarnation (default):
- podsecurity.kubernetes.io/<action>.version=latest

Implications of the expressiveness of this structure:

a namespace can have at most one level associated with a given action
multiple actions can reference the same level (can warn and audit baseline)
multiple actions can reference different levels (can warn restricted and audit baseline)
multiple actions can reference different versions of the same level (can allow baseline v1.7 and warn+audit baseline latest)

Ability to query namespaces based on this structure:

can find namespaces explicitly selecting to enforce
- kubectl get ns --show-labels -l podsecurity.kubernetes.io/allow
can find namespaces explicitly selecting a particular enforcement level
- kubectl get ns --show-labels -l podsecurity.kubernetes.io/allow=baseline
can find namespaces explicitly selecting a particular enforcement level that are using a policy version that is not latest
- kubectl get ns --show-labels -l 'podsecurity.kubernetes.io/allow=restricted,podsecurity.kubernetes.io/allow.version,podsecurity.kubernetes.io/allow.version!=latest'
can find namespaces explicitly selecting to enforce that haven't indicated what policy version they want
- kubectl get ns --show-labels -l 'podsecurity.kubernetes.io/allow,!podsecurity.kubernetes.io/allow.version'

Demo

Branch at https://github.com/liggitt/kubernetes/tree/bare-medium-pod-security

# Create a bunch of namespaces
kubectl create ns demo-0-no-policy
kubectl label  ns demo-0-no-policy demo=true

kubectl create ns demo-1-privileged
kubectl label  ns demo-1-privileged demo=true podsecurity.kubernetes.io/allow=privileged

kubectl create ns demo-2-baseline
kubectl label  ns demo-2-baseline demo=true podsecurity.kubernetes.io/allow=baseline

kubectl create ns demo-2-baseline-warn
kubectl label  ns demo-2-baseline-warn demo=true podsecurity.kubernetes.io/warn=baseline

kubectl create ns demo-3-restricted
kubectl label  ns demo-3-restricted demo=true podsecurity.kubernetes.io/allow=restricted

kubectl create ns demo-3-restricted-warn
kubectl label  ns demo-3-restricted-warn demo=true podsecurity.kubernetes.io/warn=restricted

# namespace using the restricted policy as it existed in 1.7 (before the allowPrivilegeEscalation field existed)
kubectl create ns demo-3-restricted-legacy
kubectl label  ns demo-3-restricted-legacy demo=true podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=v1.7

clear

kubectl get ns -l demo --show-labels


# try to create a privileged pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -p -n 1 kubectl apply -f pod-privileged.yaml -n
kubectl get pods -A --field-selector metadata.name=privileged-pod

# try to create a baseline pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -p -n 1 kubectl apply -f pod-baseline.yaml -n
kubectl get pods -A --field-selector metadata.name=baseline-pod

# try to create a restricted pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-restricted.yaml -n
kubectl get pods -A --field-selector metadata.name=restricted-pod

# auto-create
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-privileged.yaml -n
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-baseline.yaml -n
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-restricted.yaml -n
clear

# Now I have a mix of namespaces, pods, and policy levels

kubectl get pods -A

# What about cluster upgrades?
# Before an upgrade to v1.21, I could find enforcing namespaces that hadn't explicitly opted into the latest version and pin them to v1.20:
kubectl label ns --all -l 'podsecurity.kubernetes.io/allow,!podsecurity.kubernetes.io/audit.version' podsecurity.kubernetes.io/audit.version=v1.20

# Then upgrade the cluster to v1.21 and unpin at my leisure.
# To see what that would look like, zoom in on demo-3-restricted-legacy...
# This is a good example of a difficult to manage namespace.
# This namespace is pinned to an old version of the restricted policy,
# and has a pod that would be disallowed by the latest version of that policy

kubectl get ns demo-3-restricted-legacy --show-labels
kubectl get pods -n demo-3-restricted-legacy

# First, it is easy to find namespaces like this, explicitly indicating a level and version that is not "latest"
kubectl get ns -l 'podsecurity.kubernetes.io/allow,podsecurity.kubernetes.io/allow.version,podsecurity.kubernetes.io/allow.version!=latest' --show-labels

# When I'm ready to update the namespace, I have the following options:

# 1. Audit the new restricted policy and monitor audit logs for "a while":
kubectl label ns demo-3-restricted-legacy podsecurity.kubernetes.io/audit=restricted podsecurity.kubernetes.io/audit.version=latest

# 2. Surface warnings to the user if their stuff violated the latest restricted policy for "a while":
kubectl label ns demo-3-restricted-legacy podsecurity.kubernetes.io/warn=restricted podsecurity.kubernetes.io/warn.version=latest

# 3. Dry-run switching to the latest restricted policy and let the server warn me if it knows of pods that would be disallowed:
kubectl label --dry-run=server --overwrite ns demo-3-restricted-legacy podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=latest
# (No labels were changed in the checking of this policy)
kubectl get ns demo-3-restricted-legacy --show-labels

# Other interesting dry-run incantations:
# 1. check if it's safe to lock down all namespaces to latest baseline
kubectl label --dry-run=server --overwrite ns --all podsecurity.kubernetes.io/allow=baseline podsecurity.kubernetes.io/allow.version=latest

# 2. check if it's safe to change the cluster default to latest baseline
kubectl label --dry-run=server --overwrite ns -l '!podsecurity.kubernetes.io/allow' podsecurity.kubernetes.io/allow=baseline podsecurity.kubernetes.io/allow.version=latest

# 3. check if it's safe to change the cluster default to latest restricted
kubectl label --dry-run=server --overwrite ns -l '!podsecurity.kubernetes.io/allow' podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=latest

	kind: Pod
	apiVersion: v1
	metadata:
	name: baseline-pod
	spec:
	containers:
	- name: default
	image: k8s.gcr.io/pause:3.2

	kind: Pod
	apiVersion: v1
	metadata:
	name: privileged-pod
	spec:
	containers:
	- name: privileged
	image: k8s.gcr.io/pause:3.2
	securityContext:
	privileged: true

	kind: Pod
	apiVersion: v1
	metadata:
	name: restricted-pod
	spec:
	containers:
	- name: default
	image: k8s.gcr.io/pause:3.2
	securityContext:
	allowPrivilegeEscalation: false