supply chain security

gistfile1.txt

What is supply chain security?

Can we provide a guarentee that software we ship is free from bugs and vulnerabilities?
That would be nice, but no.

Software is more than just the executable.  It is also
* the library dependencies
  * including an often vast tree of transitive dependencies
* the build system
  * e.g. go version, rust version, toolchain etc.
* the container it runs in
* the OS it runs on
* Kubernetes
* the hardware it runs on
* any utilities or APIs that it interacts with

This all makes up the supply chain.  And any piece of the supply chain
might have bugs, vulnerabilities, or even malicious code.  As software
providers, we're mostly focused on the dependencies, build system, and
container because that's what we control and ship to customers.

We could MAYBE try to guarentee that software we ship is free from
vulnerabilities in the supply chain at the time that we ship it but...
Vulnerabilities are discovered in real-time after software ships. We
can't see the future so it's impossible to know if anything in our
supply chain will have a vulnerability discovered in it in the future.

So if we can't guarentee that our supply chain is secure, the next
best thing is to DESCRIBE our supply chain in sufficient details such
that consumers of our software can know if discovered vulnerabilities
affect them (and to what degree).

As a software provider, supply chain security is about DESCRIBING
the supply chain accurately and in detail.

I work on Buoyant Enterprise for Linkerd, lovingly abbrebiated as BEL.
BEL is an enterprise distribution of the open-source CNCF service mesh
Linkerd.  As an enterprise distribution, one of the things that 
differentiates it from open-source Linkerd is that we provide this
supply chain security information.

---- 

What things can we describe about our supply chain?

1. We can cryptographically sign our images to ensure the software
that consumers run came from us unmodified.  Protects against
MITM or supply chain injection attacks in transit.

2. Comprehensive list of all dependencies and their exact verions
* called a Software Bill of Materals or SBOM
* several formats exist
  * spdx
  * cycloneDX
Can be generated by tools like syft and snyk
Consumers can cross-reference the SBOM against vulnerability databases
to know if newly discovered vulnerabilities affect them!
This is what grype does.

3. Build attestations.  We can provide a detailed and auditable
description of all steps in our build process.  This can be
provided as a Suppy-Chain Levels for Software Artifacts (SLSA)
documents.  Helps consumers audit our build process.

4. VEX documents (Vulnerability Exploitability eXchange) allow
us to express if certain vulnerabilities in our supply chain
do not affect this software and are not exploitable.

Each of these attestations (SBOMs, SLSA, and VEX) can be signed
as well to ensure they are from the software provider and haven't
been tampered with.

----

How are these attestations produced?
How are they shipped to consumers?
How do consumers act on them?

Thankfully, generating SBOM and SLSA attestations is easy with
docker buildx or the docker/build-push-action Github Action:

    - uses: docker/build-push-action
      with:
        ...
        provenance: true # <-- SLSA
        sbom: true # <-- SPDX

You could also generate SBOMs directly with Syft.

But how these attestations are stored is where things get a bit spicy
and it requires talking about the OCI (open container initiative) format.

An image registry tag like ghcr.io/buoyantio/proxy:enterprise-2.19.5 points
to an OCI Image Index, which is a bundle of images, typically one per platform.
For example, ghcr.io/buoyantio/proxy:enterprise-2.19.5 is an image index
containing two images for the BEL Proxy: one for amd64 and one for arm64.

When we build with attestations, those attestations (the SBOM and SLSA) are
added to the image index as well.

This seems fine but..... what if we want to add more attestations?

For example: imagine that we ship software including SBOM and SLSA 
attestations and the whole bundle is signed and delivered.  Then a
vulnerability in our supply chain is discovered but we know that it
is not exploitable in our software.  We want to publish a VEX document
attesting this.  But since attestations live inside the image index,
we would need to generate a new image index with the additional
attestation and sign it all over again, even though the software
itself has not changed.

* Adding attestations requires RE-SIGNING. The image index itself
needs to be mutated. This goes against the concept of IMMUTABLE RELEASES

Thankfully, the OCI 1.1 standard provides a solution to this problem:
called Referrers.

Referrers are objects that can live in a registry outside of an
image index but can point to (ie referrer to) other objects in
the registry.

    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 995,
      "digest": "sha256:f76e9041216aa33acda434144b2b24ce0471f24437767295652442b1201303b3",
      "annotations": {
        "org.opencontainers.image.created": "2026-03-07T01:39:12Z",
        "vnd.buoci.artifact-type": "application/slsa+json",
        "vnd.buoci.predicate-type": "https://slsa.dev/provenance/v0.2",
        "vnd.buoci.role": "image-attestation",
        "vnd.buoci.subject.digest": "sha256:698c60c828d8c396595738e5548c484d23b5be6fc958d55eac846eace73f95df",
        "vnd.buoci.subject.name": "proxy"
      },
      "artifactType": "application/slsa+json",
      "subject": { 
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "size": 4902,
        "digest": "sha256:698c60c828d8c396595738e5548c484d23b5be6fc958d55eac846eace73f95df",
        "platform": {
          "architecture": "amd64",
          "os": "linux"
        }
      },
      "layers": [
        {
          "mediaType": "application/slsa+json",
          "size": 83226,
          "digest": "sha256:a9dedf28268327f0e1d3aece5cb84f79ebe3aa9766dcc431f5ab2c03abe157ae"
        }
      ]
    },

Moving attestations out of the image index and making them referrers instead
means that the image index itself is immutable and we can continue to
publish whatever attestations we want without needing to mutate or
resign the image index. Of course, all of the attestations can be individually
signed as well.

This is what we do at Buoyant for BEL.  We actually have custom build tooling
which extracts the attestations from inside our image index and turns them
into OCI 1.1 Referrers instead.

✗ oras discover ghcr.io/buoyantio/controller:enterprise-2.19.5 --platform linux/arm64
ghcr.io/buoyantio/controller@sha256:c72bcd7abf50faba085b73600aaf89afb7638c6216e0ff5e929cfd6ee3c2b053
├── application/spdx+json
│   └── sha256:c33a6e39b8cdf124f3f977b6ad0874c299f4d19b6e10a97ad7bed623fcaa0637
│       └── [annotations]
│           ├── vnd.buoci.subject.digest: sha256:c72bcd7abf50faba085b73600aaf89afb7638c6216e0ff5e929cfd6ee3c2b053
│           ├── vnd.buoci.subject.name: controller
│           ├── org.opencontainers.image.created: "2026-03-07T01:37:40Z"
│           ├── vnd.buoci.artifact-type: application/spdx+json
│           ├── vnd.buoci.predicate-type: https://spdx.dev/Document
│           └── vnd.buoci.role: image-attestation
└── application/slsa+json
    └── sha256:5057b574111d0f83c27124f37c779278993f5ea1da9810b0c7170c1649a1ace4
        └── [annotations]
            ├── vnd.buoci.artifact-type: application/slsa+json
            ├── vnd.buoci.predicate-type: https://slsa.dev/provenance/v0.2
            ├── vnd.buoci.role: image-attestation
            ├── vnd.buoci.subject.digest: sha256:c72bcd7abf50faba085b73600aaf89afb7638c6216e0ff5e929cfd6ee3c2b053
            ├── vnd.buoci.subject.name: controller
            └── org.opencontainers.image.created: "2026-03-07T01:37:40Z"

Speaking of signing, there are a number of different ways to do it
* sigstore / cosign
* notary
* manual private key signing

Finally, how do consumers interact with this stuff? Remember that supply
chain security is all about communicating information to the consumer,
which means it's all for nothing if they don't do anything with that information.

Consumers can discover referrer attestations using a tool called oras (see above).

They can verify signatures using cosign.

The can feed the discovered SBOMs into Grype to be alerted of vulnerabilities
that affect them.

Kyverno can be used to write policies and enforce them at runtime.  For example,
you can write a kyverno policy which states that all BEL images must
be signed by buoyant and must have an SPDX attestation (also signed by buoyant).
Attempting to run images which don't pass this validation will be rejected.

apiVersion: policies.kyverno.io/v1alpha1
kind: ImageValidatingPolicy
metadata:
  name: bel-2-19-index-attestations
spec:
  failurePolicy: Fail
  webhookConfiguration:
    timeoutSeconds: 20
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE","UPDATE"]

  matchImageReferences:
    - glob: "ghcr.io/buoyantio/proxy:enterprise-2.19.*"
    - glob: "ghcr.io/buoyantio/proxy-init:enterprise-2.19.*"
    - glob: "ghcr.io/buoyantio/controller:enterprise-2.19.*"
    - glob: "ghcr.io/buoyantio/extension-init:enterprise-2.19.*"

  validationActions: [Deny]
  validationConfigurations:
    mutateDigest: true
    required: true
    verifyDigest: true

  # Configure your signing attestor
  attestors:
    - name: bel-keyless
      cosign:
        ctlog:
          url: https://rekor.sigstore.dev
        keyless:
          identities:
            - issuer: https://token.actions.githubusercontent.com
              subject: https://github.com/BuoyantIO/enterprise-linkerd/.github/workflows/push-oci.yml@refs/heads/enterprise-2.19

  attestations:
    - name: spdx
      intoto:
        type: https://spdx.dev/Document

  validations:
    - expression: >-
        images.containers.map(image, verifyImageSignatures(image,  [attestors["bel-keyless"]])).all(e, e > 0)
      message: "Failed image signature verification"

    - expression: >-
        images.containers.map(image, verifyAttestationSignatures(image, attestations.spdx, [attestors["bel-keyless"]])).all(e, e > 0)
      message: "Failed to verify SBOM attestation with Cosign keyless"