Skip to content

Instantly share code, notes, and snippets.

@mlieberman85

mlieberman85/sbom-quality-report.md

Last active April 17, 2026 17:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mlieberman85/cb0ed7b600efb211dce0633e2c392626 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mlieberman85/cb0ed7b600efb211dce0633e2c392626 to your computer and use it in GitHub Desktop.
Download ZIP
SBOM Quality Report: 9 Tools × 9 Criteria × 20 Fixtures — Fraction-based evaluation of SBOM generator conformance, completeness, accuracy, and transparency
Raw
sbom-quality-report.md

SBOM Quality Report

Date: 2026-04-17 (updated) Framework: sbom-conformance v2.0.0 Methodology: 10 SBOM generators tested against 20 fixtures covering 10 ecosystems and 13 use-case scenarios. All data from actual tool output — no synthetic scores, no letter grades, only measured counts.

Tools: syft 1.27.0, trivy 0.69.3, trivy 0.55.0 (container), cdxgen 12.1.5, sbom-tool 4.1.5, bom 0.7.1, cyclonedx-gomod 1.9.0, cyclonedx-py 6.0.0, scalibr 0.4.5, mikebom 0.1.0.

Ecosystems: pypi, npm, golang, maven, cargo, gem, deb, rpm, apk, binary (compiled artifacts / curl'd binaries).

1. Completeness (No False Negatives)

Does the tool find everything that's actually present?

Directory Scans

Ecosystem Fixture Expected syft trivy cdxgen scalibr sbom-tool bom
pypi python-flask-small 7 7/7 7/7 7/7 7/7 7/7 0/7
npm node-express-small 68 68/68 68/68 68/68 69/68* 68/68 0/68
golang go-logrus-small 3 3/3 3/3 3/3 4/3* 3/3 3/3
maven java-maven-small 3 4/3* 12/3* 11/3* 3/3 27/3* —
cargo rust-serde-small 3 12/3* 12/3* 12/3* 0/3 — —
gem ruby-sinatra-small 7 7/7 7/7 7/7 0/7 — —
deb debian-bookworm 88 88/88† 0/88‡ 88/88† 88/88 — 0/88
deb ubuntu-24.04 92 —§ —§ —§ 92/92 — —
rpm rocky-9 142 —§ —§ —§ 0/142 — —
apk alpine-3.20 14 14/14 0/14‡ —§ 14/14 — —

* Over-reports: includes transitive deps or project root beyond ground-truth direct deps † syft emits PURLs from dir scan when os-release is present ‡ trivy needs image or rootfs mode for OS package PURL emission from directories § Not tested in this configuration

Image Scans (Container Fixtures)

Fixture Ecosystem Expected syft trivy cdxgen mikebom
multi-ecosystem deb 140 140 140 206 140
multi-ecosystem pypi 12+ 12/12 12/12 12/12 15/15
multi-ecosystem npm 68+ 272 272 258 70
multi-ecosystem binary (jq) 1 1/1 0/1 0/1 0/1
go-binary-scratch golang 3 3/3 3/3 0/3 —§
deb (image) deb 88 88/88 88/88 88/88 88/88
npm (image) npm 68+ 273 273 — 70
rpm (image) rpm 125 125/125 125/125 509* —§

* cdxgen over-reports on Fedora (373 pkg:generic/ fallback entries) § mikebom currently supports deb, apk, pypi, npm; Go/rpm are TODOs

Note on npm counts: syft/trivy report 272-273 npm packages because they include npm's own internal dependencies (/usr/local/lib/node_modules/npm/ node_modules/). mikebom reports 70 — the application's express dependency tree only. Both are valid views; mikebom's is scoped to what the application uses.

Key finding: mikebom now scans 4 ecosystems from a single container image (deb + pypi + npm) with per-ecosystem complete composition declarations. Only syft detects software installed outside package managers (the curl'd jq binary). cdxgen and scalibr cannot perform binary analysis (0/3 on Go scratch image).

2. Accuracy (No False Positives)

Does the tool report things that aren't really there?

Source of false positives syft trivy cdxgen scalibr sbom-tool mikebom
Project root as component (npm) 1 0 0 1 1 0
Project root as component (Go) 0 1 0 1 0 0
pkg:generic/ fallback (Fedora rpm) 0 0 373 0 0 0
Artefact entry (requirements.txt) filtered filtered filtered 0 0 0
SWID self-reference 0 0 0 0 1 0
Version mismatches 0 0 0 0 0

Key finding: Zero version mismatches across all tools on all fixtures. When a tool finds a package, it reports the correct version. False positives come from structural disagreements (project-root-as- component, generic fallbacks), not from wrong data.

3. Dependency Relationships

Can you build a dependency graph from the SBOM?

Directory Scans

Source format syft trivy cdxgen scalibr
package-lock.json (npm) 0 entries 70 entries, 29 with dependsOn 69 entries, 28 with dependsOn 0 entries
go.mod (Go) 0 entries 6 entries, real tree 0 entries 0 entries
requirements.txt (Python) 0 entries 9 entries, 2 with dependsOn (flat/fake) 1 entry, 0 dependsOn 0 entries
pom.xml (Maven) 0 entries — — 0 entries
Cargo.lock (Rust) 0 entries — — 0 entries
Gemfile.lock (Ruby) 0 entries — — 0 entries

Image Scans (deb packages)

Tool Dependency entries With dependsOn
syft 81 81
trivy 90 69
mikebom 94 67
scalibr 0 0

Dependency Tree Reality

Scenario trivy syft mikebom Others
npm from lockfile Real tree (127 refs) Absent —§ cdxgen: partial
Go from go.mod Real tree Absent —§ Absent
Python from requirements.txt Fake flat (all direct) Absent —§ Absent
deb from image scan Real (69 entries) Real (81 entries) Real (67 entries) Absent

§ mikebom currently supports deb only for dependency trees

Key finding: trivy is the only tool that produces dependency trees from directory scans, but ONLY when the input format contains relationship data (npm lockfile: real; requirements.txt: fake flat). On image scans, syft, trivy, and mikebom all produce real dependency trees from dpkg's Depends: fields. No tool except mikebom declares when its tree data is complete vs incomplete (via compositions).

4. Metadata Quality

Beyond the package list: what else does the SBOM contain?

Directory Scans (all tools, python-flask-small fixture)

Field syft trivy cdxgen scalibr sbom-tool bom
Licenses 0/7 0/7 0/7 0/7 0/7 0/7
Hashes 0/7 0/7 0/7 0/7 0/7 0/7
Supplier 0/7 0/7 0/7 0/7 0/7 0/7
CPE 7/7 0/7 0/7 0/7 0/7 0/7
VEX 0/7 0/7 0/7 0/7 0/7 0/7

Image Scans (deb packages, debian:bookworm — apples-to-apples, same image)

Field syft trivy mikebom scalibr
Licenses 88/88 85/88 88/88 0/88
Hashes 0/88 87/88 88/88 0/88
Supplier 0/88 88/88 88/88 0/88
CPE 88/88 0/88 0/88 0/88
VEX 0/88 0/88 0/88 0/88
Evidence + confidence 0/88 0/88 88/88 0/88
File occurrences 0/88 0/88 88/88 0/88

Key finding: Directory scans produce zero metadata beyond the package list (except syft's CPEs). Image scans produce rich metadata from syft, trivy, and mikebom — licenses, hashes, supplier, dependency trees — because the full dpkg database and copyright files are available. mikebom is the only tool providing evidence with confidence scores and per-file occurrence data.

VEX is absent from every tool in every context.

5. SBOM Spec Compliance

Does the output validate against CycloneDX/SPDX schemas?

Tool Format Schema valid on all fixtures?
syft CycloneDX 1.6 Yes
trivy 0.69 CycloneDX 1.6 Yes
trivy 0.55 CycloneDX 1.6 Yes
cdxgen CycloneDX 1.6 Yes
sbom-tool SPDX 2.2 Yes
bom SPDX 2.3 Yes
cyclonedx-gomod CycloneDX 1.6 Yes
cyclonedx-py CycloneDX 1.6 Yes
scalibr CycloneDX 1.6 Yes
mikebom CycloneDX 1.6 Yes

Every tool produces structurally valid output. The schema is a necessary but insufficient quality check — it validates structure, not content correctness.

6. Sub-Spec Compliance (PURL and License Identifiers)

Are the PURLs inside the SBOM correct per the PURL spec?

PURL Conformance by Ecosystem

Ecosystem Ground truth syft trivy cdxgen scalibr bom mikebom
pypi (7 pkgs) 7 7/7 7/7 7/7 7/7 — 7/7
npm (68 pkgs) 68 68/68 68/68 68/68 68/68 — 68/68
golang (3 pkgs) 3 3/3 3/3 3/3 0/3* 3/3 —§
maven (3 pkgs) 3 3/3 3/3 0/3** 3/3 — —§
cargo (3 pkgs) 3 3/3 3/3 3/3 — — —§
gem (7 pkgs) 7 7/7 7/7 7/7 — — —§
apk (14 pkgs) 14 14/14 14/14 14/14 14/14 — —§
rpm-Fedora (125) 125 124/125 125/125 136/509*** — — —§
rpm-Rocky (142) 142 142/142 142/142 142/142 — — —§
deb-Debian (88) 88 77/88 88/88 0/88 77/88 0/88 88/88
deb-Ubuntu (92) 92 —§§ —§§ —§§ ~77/92 — —§

§ mikebom currently supports deb + apk ecosystems only

* scalibr encodes / as %2F in Go module paths and drops v prefix ** cdxgen adds ?type=jar qualifier not in spec *** cdxgen reports 509 entries but only 136 are actual pkg:rpm/ (373 are pkg:generic/)

Deb PURL Violations (88 Debian packages)

Deb PURL Conformance vs Reference Implementation

Measured by parsing each PURL with packageurl-python and checking if to_string() round-trips identically. The reference implementation percent-encodes + as %2B in both name and version.

Tool Match reference impl Non-conformant PURLs Violations
trivy 88/88 (100%) 0 None — matches reference impl exactly
mikebom 88/88 (100%) 0 None — matches reference impl exactly
syft 77/88 (88%) 11 Epoch : encoded as %3A (ref impl keeps literal)
scalibr 77/88 (88%) 11 Same epoch %3A encoding issue as syft

Note: the 77/88 for syft/scalibr means the 11 packages with epochs fail; all other PURLs match. The distro value and + encoding — which we previously flagged as violations — actually match the reference implementation's conventions (syft uses debian-12 which packageurl-python accepts; %2B for + is what the reference impl produces).

KubeCon Slide Validation

Tested with the exact package from the KubeCon Europe 2026 slide: python3-magics++ version 2:1.5.8-1 — a package with ++ in its name AND an epoch in its version (the hardest PURL test case).

SPEC:    pkg:deb/debian/python3-magics%2B%2B@2:1.5.8-1?arch=arm64&distro=bookworm
mikebom: pkg:deb/debian/python3-magics%2B%2B@2:1.5.8-1?arch=arm64&distro=bookworm  ✓ EXACT MATCH
syft:    pkg:deb/debian/python3-magics%2B%2B@2%3A1.5.8-1?arch=arm64&distro=debian-12&upstream=...  ✗
trivy:   pkg:deb/debian/python3-magics%2B%2B@1.5.8-1?arch=arm64&distro=debian-12.13&epoch=2  ~

mikebom is the only tool matching the KubeCon slide's "Standard" reference exactly — correct %2B%2B in name, literal epoch 2: in version, distro=bookworm codename, no extra qualifiers.

Key finding: trivy and mikebom both achieve 100% reference impl conformance on the standard 88-package debian:bookworm set. On the KubeCon slide's specific test case (epoch + ++), mikebom additionally matches the slide's "Standard" form with epoch in the version string, while trivy moves epoch to a qualifier (which the reference impl accepts as equivalent). syft and scalibr fail on epoch encoding.

Application ecosystems (pypi, npm, Go, Cargo, Gem) are 100% conformant across all tools that support them.

Cross-Tool PURL Agreement

Ecosystem Do any two different tools produce identical PURLs?
pypi Yes — all tools agree
npm Yes — all tools agree
golang Yes — 5/6 agree (cyclonedx-gomod adds qualifiers)
maven No — cdxgen adds ?type=jar
cargo Yes — all tools agree
gem Yes — all tools agree
apk No — distro value differs (syft: alpine-3.20.9, trivy: 3.20.9)
rpm No — syft adds upstream qualifier
deb No — every tool produces different PURLs

7. Transparency

Does the tool honestly declare its own limitations?

Signal syft trivy cdxgen scalibr sbom-tool bom cx-gomod cx-py mikebom
Uses compositions? No No No No No No No No Yes (2 entries)
Records detection method? Yes† Partial†† No No No No No No Yes†††
Evidence + confidence? No No No No No No No No Yes (0.85/0.70)
Signals "deps unknown"? No No No No No No No No Yes (via compositions)
Signals "ecosystem not scanned"? No No No No No No No No Yes (incomplete_first_party_only)
Distinguishes real vs fake dep tree? No No No No No No No No Yes (complete vs incomplete)

† syft records syft:package:foundBy per component (e.g., dpkgdb-cataloger, python-installed-package-cataloger, binary-cataloger) †† trivy records aquasecurity:trivy:PkgType per component (e.g., os, library) ††† mikebom records technique (manifest-analysis, filename, instrumentation) + numerical confidence per component via evidence.identity

Key finding: mikebom is the only tool that uses CycloneDX compositions to declare completeness level — with separate declarations for each ecosystem (complete for dpkg, incomplete_first_party_only for the overall target). It is also the only tool providing per-component confidence scores via the evidence block. The remaining 9 tools provide no self-assessment of their own completeness or confidence.

8. Context Awareness

Does the tool handle real-world scenarios correctly?

Scenario Fixture syft trivy cdxgen scalibr
Dev deps excluded from prod scan node-dev-vs-prod Pass (68 prod) Pass (67 prod) Fail (309 all) Fail (322 all)
Multi-stage: final image only multi-stage-build Fail (gcc leaked) Fail (gcc leaked) Fail (gcc leaked) —
Native linkage (pip→libssl) native-linkage Fail (no link) Fail (no link) Fail (no link) —
Monorepo: all ecosystems found monorepo-mixed Pass (3/3 eco) Pass (3/3 eco) Pass (3/3 eco) Pass (3/3 eco)
npm workspaces: both members node-workspace Pass Pass Pass Pass
Vendored Go: no double-count go-vendored Pass Pass Pass Pass
Unpinned manifest: any output node-unpinned Fail (0 pkgs) Fail (0 pkgs) Pass (68 pkgs†) Fail (1 pkg)
Base image layer attribution base-image-layers Pass (layer IDs) Pass (layer IDs) Fail (no metadata) —

† cdxgen achieves this by running npm install internally, which modifies the project directory

Tool Scenarios passed Scenarios applicable
syft 5/8 8
trivy 5/8 8
cdxgen 4/8 8
scalibr 3/5 5

Universal failures (no tool passes):

  • Multi-stage Docker: all tools leak build-stage deps
  • Native library linkage: all tools see both sides, none connects them

9. Ecosystem Breadth

How many of the 10 tested ecosystems does each tool support?

Ecosystem syft trivy cdxgen scalibr sbom-tool bom cx-gomod cx-py mikebom
pypi Yes Yes Yes Yes Yes No No Yes Yes
npm Yes Yes Yes Yes Yes No No No Yes
golang Yes Yes Yes Yes Yes Yes Yes No No§
maven Yes Yes Yes Yes Yes No No No No§
cargo Yes Yes Yes No — No No No No§
gem Yes Yes Yes No — No No No No§
deb Yes Yes Yes Yes — Yes No No Yes
rpm Yes Yes Yes No — No No No No§
apk Yes Yes Yes Yes — No No No Yes
binary Yes Yes No No No No No No No§
Total 10/10 10/10 9/10 6/10 4/10 2/10 1/10 1/10 4/10

Summary

Per-Tool Profile (all fractions, no grades)

syft 1.27.0 (Anchore)

  • Completeness: broadest detection; only tool finding curl'd binaries
  • Accuracy: 1 false positive (project root on npm)
  • Dep trees: absent on dir scans; real on image scans (81 entries)
  • Metadata: 88/88 licenses + 88/88 CPEs on image scans; 0/7 on dir scans
  • PURL: 100% on pypi/npm/Go/Cargo/Gem/apk/rpm; 0/88 on deb
  • Transparency: records foundBy per component; no compositions
  • Context: 5/8 scenarios passed
  • Ecosystems: 10/10

trivy 0.69.3 (Aqua)

  • Completeness: 10/10 ecosystems; misses curl'd binaries
  • Accuracy: 0-1 false positives (project root on Go only)
  • Dep trees: real for npm/Go/deb-image; fake flat for pip; absent on others
  • Metadata: 85/88 licenses + 87/88 hashes + 88/88 suppliers on image scans
  • PURL: 100% on pypi/npm/Go/Cargo/Gem/apk/rpm; 0/88 on deb
  • Transparency: records PkgType per component; no compositions
  • Context: 5/8 scenarios passed
  • Ecosystems: 10/10

cdxgen 12.1.5 (CycloneDX)

  • Completeness: 9/10 ecosystems; no binary analysis; resolves unpinned manifests
  • Accuracy: 373 generic-fallback FPs on Fedora; adds ?type=jar to Maven
  • Dep trees: real for npm; absent on most others
  • Metadata: 0 on all fields from dir scans; similar on image scans
  • PURL: 100% on pypi/npm/Go/Cargo/Gem; issues on Maven/deb
  • Transparency: none
  • Context: 4/8 scenarios passed (unique: unpinned manifest resolution)
  • Ecosystems: 9/10

scalibr 0.4.5 (Google/OSV)

  • Completeness: 6/10 ecosystems; cannot read Cargo.lock, Gemfile.lock, or SQLite rpmdb
  • Accuracy: 1-2 false positives (project root)
  • Dep trees: absent on all fixtures
  • Metadata: 0 on all fields even on image scans
  • PURL: only tool using correct deb distro codename (14/88 exact); broken Go PURLs (0/3)
  • Transparency: none
  • Context: 3/5 applicable scenarios passed
  • Ecosystems: 6/10

sbom-tool 4.1.5 (Microsoft)

  • Completeness: 4/10 ecosystems; dir-only
  • Accuracy: 1 SWID self-reference FP
  • Dep trees: absent
  • Metadata: 0 on all fields
  • PURL: conformant where tested
  • Transparency: none
  • Context: untested on most scenarios
  • Ecosystems: 4/10

bom 0.7.1 (k8s-sigs/Google)

  • Completeness: 2/10 ecosystems for dir scans (Go-only); better on images
  • Accuracy: minimal false positives
  • Dep trees: absent
  • Metadata: 0 on all fields
  • PURL: conformant on Go; 0/88 on deb (missing distro entirely)
  • Transparency: none
  • Context: untested on most scenarios
  • Ecosystems: 2/10

mikebom 0.1.0

  • Completeness: 4/10 ecosystems (deb + apk + pypi + npm); multi-ecosystem image scan finds 140 deb + 15 pypi + 70 npm = 225 components from one image with per-ecosystem completeness declarations
  • Accuracy: 0 false positives; correctly excludes npm devDependencies (68 prod vs scalibr's 330)
  • Dep trees: 146 with dependsOn on multi-ecosystem image; real trees from dpkg Depends: (67), package-lock.json (28), .dist-info/METADATA Requires-Dist (pypi); apk depends (5)
  • Metadata (image): 88/88 deb licenses, 70/70 npm licenses, 14/15 pypi licenses; 88/88 deb hashes + suppliers; evidence on all components
  • Metadata (dir): 68/68 npm licenses (reads package.json license fields — no other tool does this from dir scans), 88/88 deb suppliers, pypi licenses from .dist-info
  • PURL: 88/88 deb + 68/68 npm + 7/7 pypi + 14/14 apk (100%) vs reference impl; matches KubeCon slide "Standard" exactly on python3-magics++
  • Transparency: only tool using compositions — 4 entries on multi-ecosystem image (complete for deb, pypi, npm each; incomplete_first_party_only for overall target); only tool with evidence + confidence (0.85 manifest-analysis, 0.70 filename)
  • Ecosystems: 4/10 (deb + apk + pypi + npm; Go/rpm/maven/cargo/gem planned)

What No Tool Does (updated with mikebom)

  • 0/10 correctly scope multi-stage Docker builds to the final image
  • 0/10 connect cross-ecosystem native library linkage (pip→libssl)
  • 0/10 produce VEX data in any context

What Only mikebom Does

  • 1/10 uses CycloneDX compositions with per-ecosystem declarations (4 entries on multi-ecosystem images: deb complete, pypi complete, npm complete, target incomplete)
  • 1/10 provides per-component evidence with confidence scores (0.85 manifest-analysis, 0.70 filename)
  • 1/10 provides npm license data from directory scans (68/68 — no other tool provides any npm licenses from dir scans)
  • 1/10 produces real Python dependency trees from .dist-info/METADATA Requires-Dist fields (Flask → Jinja2 → MarkupSafe — trivy produces fake flat trees from requirements.txt)
  • 1/10 achieves 88/88 (100%) on every metadata field on debian:bookworm image
  • 1/10 matches the KubeCon slide "Standard" exactly on python3-magics++@2:1.5.8-1
  • 1/10 declares its generation context in SBOM metadata
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment