Kyvenro EKS

kyverno-notation-eks.md

Kyverno + AWS Notation: Enforcing Signed Container Images in EKS

Overview

This guide demonstrates how to enforce signed container image verification in an Amazon EKS cluster using:

• Kyverno – Kubernetes policy engine
• AWS Notation (kyverno-notation-aws) – signature verification
• AWS Signer – image signing
• IRSA (IAM Roles for Service Accounts) – secure AWS access

Goal:
Allow only trusted, signed images to be deployed in Kubernetes.

---

What is Notation?

Notation is an open-source CLI tool and library from the Notary Project, a CNCF (Cloud Native Computing Foundation) initiative. It provides a standard way to sign and verify OCI artifacts - container images, Helm charts, and other artifacts stored in OCI-compliant registries like AWS ECR.

Key concepts:

• OCI Signature Specification - Notation follows the OCI signature spec. Signatures are stored as OCI artifacts alongside the image in the same registry. There's no need for a separate signature store; ECR holds both the image and its signature.
• Plugin-based architecture - The Notation CLI handles the signing/verification workflow, but the actual cryptographic operations (key management, signing algorithms) are delegated to plugins. This means Notation itself is cloud/vendor-agnostic - the plugin determines where keys live and how signing happens.
• Trust Policies & Trust Stores - Notation uses trust policies to define which images need verification and which signing identities (certificates/profiles) are trusted. Trust stores hold the root certificates or public keys used during verification.

---

Notation Signature Verification Levels

Notation's trust policy supports four signature verification levels that control how strictly signature checks are enforced. Each level determines which verification steps are performed and whether failures are blocking or logged.

strict (recommended for production)

The most restrictive level. All verification checks must pass:

• Signature integrity check
• Signature is not expired
• Authenticity - signed by a trusted identity
• Expiry - signing certificate is still valid
• Revocation - certificate has not been revoked

If any check fails, the verification fails. Use this when you want to guarantee that only fully trusted, valid signatures are accepted.

signatureVerification:
  level: strict

permissive

Performs all the same checks as strict, but treats some failures as warnings instead of errors:

• Signature integrity and authenticity must still pass (hard failures)
• Expiry and revocation checks are performed but logged as warnings if they fail - verification still succeeds

Use this when you want signature authenticity enforced but can tolerate expired or revocation-unchecked signatures (e.g., during a migration period or when revocation infrastructure is not yet in place).

signatureVerification:
  level: permissive

audit

Performs all verification checks but never blocks - all failures are logged as warnings. Verification always succeeds regardless of the outcome.

Use this for:

• Initial rollout - observe which images would fail before enforcing
• Monitoring - track signing compliance without disrupting deployments
• Debugging - understand what verification failures look like in your environment

signatureVerification:
  level: audit

skip

Skips signature verification entirely. No checks are performed. Verification always succeeds.

Use this only to temporarily exempt specific registry scopes from verification (e.g., third-party base images you don't control). Avoid using skip broadly.

signatureVerification:
  level: skip

Choosing the Right Level

Level	Integrity	Authenticity	Expiry	Revocation	Failures
strict	required	required	required	required	block
permissive	required	required	warn	warn	partial block
audit	warn	warn	warn	warn	log only
skip	skipped	skipped	skipped	skipped	n/a

A common rollout strategy:

1. Start with audit to observe which images pass/fail without impacting workloads
2. Move to permissive once you're confident in your signing pipeline but still stabilizing revocation/expiry
3. Graduate to strict for full enforcement in production

Note: This is Notation's verification level, which controls how the signature itself is validated. It is separate from Kyverno's validationFailureAction (Enforce vs Audit), which controls whether Kyverno blocks the admission request or just logs a policy violation. In practice, you configure both: Notation's level in the trust policy, and Kyverno's action in the ClusterPolicy.

---

How AWS Signer Works with Notation

AWS Signer is a fully managed code-signing service. For container image signing, it acts as a Notation plugin (com.amazonaws.signer.notation.plugin).

Why AWS Signer?

• No private key management - AWS Signer manages the signing keys for you. You never handle, rotate, or store private keys yourself.
• Signing Profiles - You create a Signing Profile that defines the signing configuration (algorithm, validity period). This profile is the identity behind your signatures.
• Audit trail - All signing operations are logged in AWS CloudTrail.

The Signing Flow

1. You push a container image to ECR (by digest)
2. You run: notation sign <ecr-image-uri>@sha256:<digest>
3. Notation delegates to the AWS Signer plugin
4. AWS Signer signs the image digest using your Signing Profile's managed key
5. Notation pushes the resulting signature back to ECR as an OCI artifact
6. The signature is now stored alongside the image in the same ECR repository

The Verification Flow

1. notation verify (or kyverno-notation-aws) fetches the signature artifact from ECR
2. The AWS Signer plugin validates the signature against the Signing Profile
3. The trust policy is checked - is this signing identity trusted for this registry scope?
4. Result: verified or rejected

---

How It All Fits Together on EKS

Developer pushes image to ECR
    |
    v
notation sign (via AWS Signer plugin) --> signature stored in ECR
    |
    v
kubectl apply (deploy to EKS)
    |
    v
Kyverno admission controller intercepts the request
    |
    v
kyverno-notation-aws extension verifies the image signature
    |
    v
Signed + Trusted --> Admitted    |    Unsigned/Untrusted --> Denied

• Kyverno acts as the admission controller - it intercepts every Pod/Deployment creation.
• kyverno-notation-aws is a Kyverno extension that knows how to call Notation's verification logic with the AWS Signer plugin.
• IRSA gives the extension pod the AWS permissions it needs to pull signatures from ECR and check revocation status with AWS Signer.

---

Signing an Image with Notation + AWS Signer

Before you can verify signed images in your cluster, you need to sign them. Here's how:

Install the Notation CLI

# Download the latest release (check https://notaryproject.dev/docs/user-guides/install-notation/ for current version)
curl -Lo notation.tar.gz https://d2hvyiie56hcat.cloudfront.net/linux/amd64/notation_<VERSION>_linux_amd64.tar.gz
tar -xzf notation.tar.gz -C /usr/local/bin/ notation
notation version

Install the AWS Signer Plugin for Notation

# Install the plugin
aws signer get-signing-profile  # verify AWS CLI has signer support

# Download and install the plugin
# See: https://docs.aws.amazon.com/signer/latest/developerguide/image-signing-prerequisites.html
notation plugin install com.amazonaws.signer.notation.plugin --url <plugin-download-url>

# Verify
notation plugin list

Create an AWS Signer Signing Profile

aws signer put-signing-profile \
  --profile-name notation_test \
  --platform-id Notation-OCI-SHA384-ECDSA

# Note the profile ARN - you'll need it for trust policies
# arn:aws:signer:<REGION>:<ACCOUNT_ID>:/signing-profiles/notation_test

Sign an Image

# Authenticate to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com

# Push your image (if not already pushed)
docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign:v1

# Get the image digest (important: always sign by digest, not tag)
IMAGE_DIGEST=$(aws ecr describe-images \
  --repository-name notation-sign \
  --image-ids imageTag=v1 \
  --query 'imageDetails[0].imageDigest' \
  --output text)

# Sign the image
notation sign <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign@${IMAGE_DIGEST} \
  --plugin com.amazonaws.signer.notation.plugin \
  --id arn:aws:signer:us-east-1:<ACCOUNT_ID>:/signing-profiles/notation_test

After signing, you can verify locally:

# Set up a local trust policy and trust store first, then:
notation verify <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign@${IMAGE_DIGEST}

---

Prerequisites

• EKS Cluster with OIDC enabled
• kubectl, helm installed
• notation CLI and the AWS Signer plugin for Notation (see Signing an Image above)
• AWS ECR repository
• AWS Signer Signing Profile
• Kyverno v1.11+ (required for external image verification support)

---

1. Install Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

---

2. Clone Kyverno Notation Extension

git clone https://github.com/nirmata/kyverno-notation-aws
cd kyverno-notation-aws

---

3. Create IAM Role for Verification

Create an IAM role that the kyverno-notation-aws extension will assume (via IRSA) to pull signatures from ECR and validate them against AWS Signer.

Attach managed policy:

• AmazonEC2ContainerRegistryReadOnly

Custom policy for AWS Signer access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "signer:GetRevocationStatus",
        "signer:DescribeSigningJob"
      ],
      "Resource": "*"
    }
  ]
}

Note: Resource: "*" is acceptable here since signer:GetRevocationStatus and signer:DescribeSigningJob do not support resource-level permissions.

---

4. Configure Trust Policy (IRSA)

Add the following trust policy to the IAM role created in step 3. This allows the kyverno-notation-aws service account to assume the role via OIDC federation.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:kyverno-notation-aws:kyverno-notation-aws",
          "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Replace <ACCOUNT_ID>, <REGION>, and <OIDC_ID> with your EKS cluster's values. You can retrieve the OIDC ID with:

aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text | cut -d'/' -f5

---

5. Configure Kyverno Notation Extension

Edit configs/install.yaml to add the IRSA annotation and AWS region:

vim configs/install.yaml

Under the ServiceAccount metadata, add the IRSA role annotation:

metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>

Under the container env section, set the AWS region:

env:
  - name: AWS_REGION
    value: us-east-1

Then apply:

kubectl apply -f configs/install.yaml

---

6. Install CRDs & Trust Store

Note: CRDs and the trust store must be applied before the trust policy in step 7. The trust policy references CRD types that won't exist until this step is complete.

kubectl apply -f configs/crds/
kubectl apply -f configs/samples/truststore.yaml

---

7. Configure Trust Policy

vim configs/samples/trustpolicy.yaml

registryScopes:
  - "*"

trustedIdentities:
  - "arn:aws:signer:us-east-1:<ACCOUNT_ID>:signing-profiles/notation_test"

Security warning: registryScopes: ["*"] trusts the signing identity for all registries. This is acceptable for testing, but for production you should scope it to your specific ECR repositories:

registryScopes:
  - "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign"

kubectl apply -f configs/samples/trustpolicy.yaml
kubectl get trustpolicy

---

8. Configure Kyverno Policy

First, create the target namespace referenced in the policy:

kubectl create namespace test-notation

Then edit the Kyverno ClusterPolicy:

vim configs/samples/kyverno-policy.yaml

Below are the relevant excerpts from the ClusterPolicy - refer to the full file for the complete resource definition:

spec:
  validationFailureAction: Enforce   # Enforce = deny on failure; use "Audit" to log without blocking
  failurePolicy: Fail                # Fail-closed if the extension is unreachable

  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              namespaces:
                - test-notation
              kinds:
                - Pod
                - Deployment
      verifyImages:
        - key: imageReferences
          value:
            - "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign*"

kubectl apply -f configs/samples/kyverno-policy.yaml
kubectl get cpol

---

9. Fix RBAC Issue

Kyverno's admission controller may fail to verify images if it lacks permissions to read secrets (used by the notation extension for TLS/certificate data). If you see RBAC errors in the Kyverno admission controller logs, apply the following Helm values override:

Create a file called kyverno-values-override.yaml:

features:
  policyExceptions:
    enabled: true
    namespace: "*"

admissionController:
  rbac:
    clusterRole:
      extraResources:
        - apiGroups: [""]
          resources:
            - secrets
          verbs:
            - get
            - list
            - watch

Then upgrade the Kyverno Helm release:

helm upgrade kyverno kyverno/kyverno -n kyverno -f kyverno-values-override.yaml

---

10. Validate Policy Enforcement (Unsigned Image)

Push an unsigned image to ECR and verify that Kyverno blocks it:

docker pull tomcat
docker tag tomcat:latest <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign:tomcat
docker push <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign:tomcat

Get the image digest (always use digests for verification):

IMAGE_DIGEST=$(aws ecr describe-images \
  --repository-name notation-sign \
  --image-ids imageTag=tomcat \
  --query 'imageDetails[0].imageDigest' \
  --output text)

IMAGE_URI="<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign@${IMAGE_DIGEST}"

(Optional) Verify locally that the image is unsigned - this requires a local trust policy and trust store to be configured on your machine (see Notation documentation):

notation verify $IMAGE_URI

Expected:

Error: signature verification failed

Now deploy to the cluster - Kyverno should deny it:

kubectl create deployment tomcat --image=$IMAGE_URI -n test-notation

Expected:

Denied by Kyverno

---

11. Validate Signed Image

Sign the image first (see Signing an Image above), then verify in-cluster:

SIGNED_IMAGE_URI="<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/notation-sign@<SIGNED_IMAGE_DIGEST>"

(Optional) Verify locally:

notation verify $SIGNED_IMAGE_URI

Expected:

Successfully verified signature

Deploy to the cluster - Kyverno should allow it:

kubectl create deployment signed --image=$SIGNED_IMAGE_URI -n test-notation

Expected:

deployment.apps/signed created

---

Key Takeaways

• Enforces image trust at admission level
• Uses AWS-native signing
• IRSA + RBAC are critical
• Always prefer digest over tags