Proof of concept demo testing out my comments on the "Bare Minimum Pod Security" proposal:
- Exemptions for users, namespaces, and runtimeclasses
- Support for restricted profile
- Policy versioning
Consider this a proof of concept of "Bare Medium" Pod Security
Three policy levels, mapping to Pod Security Standards levels:
privileged
baseline
restricted
Three actions that can be taken, expressed as a namespace label of podsecurity.kubernetes.io/<action>=<level>
:
allow=<level>
(reject pods that exceed<level>
)warn=<level>
(warn the user when creating a pod that exceeds<level>
)audit=<level>
(add info to the audit log when creating a pod that exceeds<level>
)
Ability to indicate the version of the policy
- as it existed at a particular Kubernetes version should be used:
podsecurity.kubernetes.io/<action>.version=v1.7
- or the latest incarnation (default):
podsecurity.kubernetes.io/<action>.version=latest
Implications of the expressiveness of this structure:
- a namespace can have at most one level associated with a given action
- multiple actions can reference the same level (can warn and audit
baseline
) - multiple actions can reference different levels (can warn
restricted
and auditbaseline
) - multiple actions can reference different versions of the same level (can allow
baseline
v1.7
and warn+auditbaseline
latest
)
Ability to query namespaces based on this structure:
- can find namespaces explicitly selecting to enforce
kubectl get ns --show-labels -l podsecurity.kubernetes.io/allow
- can find namespaces explicitly selecting a particular enforcement level
kubectl get ns --show-labels -l podsecurity.kubernetes.io/allow=baseline
- can find namespaces explicitly selecting a particular enforcement level that are using a policy version that is not latest
kubectl get ns --show-labels -l 'podsecurity.kubernetes.io/allow=restricted,podsecurity.kubernetes.io/allow.version,podsecurity.kubernetes.io/allow.version!=latest'
- can find namespaces explicitly selecting to enforce that haven't indicated what policy version they want
kubectl get ns --show-labels -l 'podsecurity.kubernetes.io/allow,!podsecurity.kubernetes.io/allow.version'
Branch at https://github.com/liggitt/kubernetes/tree/bare-medium-pod-security
# Create a bunch of namespaces
kubectl create ns demo-0-no-policy
kubectl label ns demo-0-no-policy demo=true
kubectl create ns demo-1-privileged
kubectl label ns demo-1-privileged demo=true podsecurity.kubernetes.io/allow=privileged
kubectl create ns demo-2-baseline
kubectl label ns demo-2-baseline demo=true podsecurity.kubernetes.io/allow=baseline
kubectl create ns demo-2-baseline-warn
kubectl label ns demo-2-baseline-warn demo=true podsecurity.kubernetes.io/warn=baseline
kubectl create ns demo-3-restricted
kubectl label ns demo-3-restricted demo=true podsecurity.kubernetes.io/allow=restricted
kubectl create ns demo-3-restricted-warn
kubectl label ns demo-3-restricted-warn demo=true podsecurity.kubernetes.io/warn=restricted
# namespace using the restricted policy as it existed in 1.7 (before the allowPrivilegeEscalation field existed)
kubectl create ns demo-3-restricted-legacy
kubectl label ns demo-3-restricted-legacy demo=true podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=v1.7
clear
kubectl get ns -l demo --show-labels
# try to create a privileged pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -p -n 1 kubectl apply -f pod-privileged.yaml -n
kubectl get pods -A --field-selector metadata.name=privileged-pod
# try to create a baseline pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -p -n 1 kubectl apply -f pod-baseline.yaml -n
kubectl get pods -A --field-selector metadata.name=baseline-pod
# try to create a restricted pod in each demo namespace and show the results
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-restricted.yaml -n
kubectl get pods -A --field-selector metadata.name=restricted-pod
# auto-create
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-privileged.yaml -n
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-baseline.yaml -n
kubectl get ns -l demo -o jsonpath={.items[*].metadata.name} | xargs -t -n 1 kubectl apply -f pod-restricted.yaml -n
clear
# Now I have a mix of namespaces, pods, and policy levels
kubectl get pods -A
# What about cluster upgrades?
# Before an upgrade to v1.21, I could find enforcing namespaces that hadn't explicitly opted into the latest version and pin them to v1.20:
kubectl label ns --all -l 'podsecurity.kubernetes.io/allow,!podsecurity.kubernetes.io/audit.version' podsecurity.kubernetes.io/audit.version=v1.20
# Then upgrade the cluster to v1.21 and unpin at my leisure.
# To see what that would look like, zoom in on demo-3-restricted-legacy...
# This is a good example of a difficult to manage namespace.
# This namespace is pinned to an old version of the restricted policy,
# and has a pod that would be disallowed by the latest version of that policy
kubectl get ns demo-3-restricted-legacy --show-labels
kubectl get pods -n demo-3-restricted-legacy
# First, it is easy to find namespaces like this, explicitly indicating a level and version that is not "latest"
kubectl get ns -l 'podsecurity.kubernetes.io/allow,podsecurity.kubernetes.io/allow.version,podsecurity.kubernetes.io/allow.version!=latest' --show-labels
# When I'm ready to update the namespace, I have the following options:
# 1. Audit the new restricted policy and monitor audit logs for "a while":
kubectl label ns demo-3-restricted-legacy podsecurity.kubernetes.io/audit=restricted podsecurity.kubernetes.io/audit.version=latest
# 2. Surface warnings to the user if their stuff violated the latest restricted policy for "a while":
kubectl label ns demo-3-restricted-legacy podsecurity.kubernetes.io/warn=restricted podsecurity.kubernetes.io/warn.version=latest
# 3. Dry-run switching to the latest restricted policy and let the server warn me if it knows of pods that would be disallowed:
kubectl label --dry-run=server --overwrite ns demo-3-restricted-legacy podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=latest
# (No labels were changed in the checking of this policy)
kubectl get ns demo-3-restricted-legacy --show-labels
# Other interesting dry-run incantations:
# 1. check if it's safe to lock down all namespaces to latest baseline
kubectl label --dry-run=server --overwrite ns --all podsecurity.kubernetes.io/allow=baseline podsecurity.kubernetes.io/allow.version=latest
# 2. check if it's safe to change the cluster default to latest baseline
kubectl label --dry-run=server --overwrite ns -l '!podsecurity.kubernetes.io/allow' podsecurity.kubernetes.io/allow=baseline podsecurity.kubernetes.io/allow.version=latest
# 3. check if it's safe to change the cluster default to latest restricted
kubectl label --dry-run=server --overwrite ns -l '!podsecurity.kubernetes.io/allow' podsecurity.kubernetes.io/allow=restricted podsecurity.kubernetes.io/allow.version=latest