SOUP: dependency discovery, SBOM per build, and daily monitoring of the deployed version

[grafele]

Third-party components: the inventory of every build, daily monitoring of the deployed version, and
the classification and timeframes that govern the response. Covers DEV-190, DEV-191, DEV-192,
DEV-195, DEV-196, DEV-197.

Counterpart PR: QuickBirdEng/workflows#54 adds the three reusable workflows that call these
actions. That one should merge first. A product caller referencing
QuickBirdEng/workflows/.github/workflows/soup-sbom.yml@main fails until the file is on main.

What this does

Four points in time, each with its own record:

Trigger	Question	Record
Pull request	Is every vulnerability of this version fixed or dispositioned?	The SOUP record. Blocks the change if not
Tagged build	What is in this build, and is it assessed?	The inventory, attached to the release
Daily	Is the version in production affected today?	A dated record, written even when nothing is found
Quarterly or annually	What did not happen?	Reconciliation report

The daily run assesses what is actually deployed, resolved from the GitHub deployment record. Not
the newest tag and not the newest release, because neither states what is running.

Each tagged build produces three artefacts: the CycloneDX document with the assessment written onto
the findings, an SBOM Report stating composition, and a dated Dependency & Vulnerability
Report stating the assessment. Both PDFs are rendered from the CycloneDX document alone, so neither
can drift from it.

What the assessment covers

• Vulnerabilities, classified by a rule table whose inputs are applicability (VEX), KEV
  membership, EPSS and CVSS, in that order of precedence. kev-epss-enrichment supplies the two
  exploitation feeds and records the KEV catalog version and the EPSS model version with every score,
  because EPSS scores are not comparable between model versions.
• Dependency scope: whether a component is a direct choice of the product, development tooling
  that does not ship, or transitive. Read from the manifests, which is what makes coverage measurable
  against the actual choices rather than against the record list.
• The dependency graph, derived from the lockfile where the ecosystem carries one and from the
  resolving registry otherwise, so a finding can be traced to the direct dependency that pulled it in.
• Currency and obsolescence, including the build date of a container image, with supplier, licence
  and deprecation taken from the same registry data.
• The match against the SOUP records, which surfaces a version that has moved on or a component
  present in the build but absent from the records.

Three things it deliberately refuses to do

• Scan an artefact nobody classified. Discovery finds candidates. Every one must appear in
  .soup-scope.yml as included or excluded, each with a reason. An unclassified candidate stops the
  run, because "not listed" is otherwise indistinguishable from "deliberately excluded".
• Report success on an empty result. A run that could not complete says so. This was a real defect
  twice: a select() inside a jq object constructor made the whole construction empty, and the
  action wrote a 0-byte record while reporting all-clear.
• Guess a required parameter. tier, cra_scope and maintenance_interval come from the customer
  SLA and have no safe default. cra_scope defaults to unknown rather than false, because an
  unnecessary report costs less than an omitted one.

Where the rules live

Not in this repository. The work instruction is a controlled QMS document, which is where it has a
review workflow and a single version: WI-006-09 Dependency and Vulnerability Management.
policy-defaults.yml here is its machine-readable counterpart. The document is what a person reads
and what an auditor is shown, this is what the tooling applies. A change to one is a change to the
other and belongs in the same review. Comments in the code cite the document by section name, as
WI: Classification of a finding, so a reference survives the document being reordered.

soup-discovery/IMPLEMENTATION.md is the other half: the files, the scripts, the configuration keys
and the object store layout. It lives here because it describes this code and changes with it.

Two places where BSI TR-03161 is stricter than the process default, enforced rather than documented.
O.TrdP_2 requires the newest version or the one preceding it, so a product in TR-03161 scope must
state a patch limit. O.TrdP_8 is a prohibition, so accepting obsolescence with a recorded reason is
not available to it. A configuration that violates either is refused by validate-policy.sh.

Nothing here is product-specific

No ticket number and no product name in any comment, README or example. Tests excepted, where a
regression test names what it reproduces. Measured facts stay, because they are the reason the values
are what they are. Only the attribution goes.

Per-product files are not in this repository either. The scope and policy drafts and the deploy
workflow patch go to the product repositories, which is where a product's configuration belongs.
examples/ holds three generic templates instead: a policy, a policy showing a justified relaxation,
and a scope declaration whose entries are the cases worth showing, being an image in scope in one
artefact and out of scope in another, our own release-versioned image, a builder stage, and
infrastructure.

Measured against real repositories

Everything below is from actual runs, not from examples. On one backend product:

• 522 findings resolve through 3 actions. 492 of them are RPM packages inside a single image, and
  none is in code written here. Attaching the deadline to the finding produced 522 escalations for 3
  decisions, which is why timeframes attach to the remediation unit instead.
• 0 of 23 Critical findings were listed in KEV. A shared 72-hour deadline was applying the urgency
  of observed exploitation to findings that carried none. KEV is now its own track, and the other
  bands got timeframes that can be met.
• Excluding OS packages changed "currency unknown" from 343 of 386 to 43. The image is the SOUP,
  and its packages move when it does.
• Maintenance windows: two products hold, one has missed 1, one has missed 3, and one cannot be
  determined because its deployment workflow declares no environment.

Seven defects in my own implementation were found by running it rather than by reading it. Each has a
regression test.

Tests

soup-discovery/tests/run-tests.sh: 174 cases passing, plus 7 that hit live feeds and are skipped
without TEST_NETWORK=1. Most are regressions for defects found while building this.

Suggested reading order

1. WI-006-09 in the QMS, up to and including Parameters defined per project. What the process is
   and what each project decides.
2. soup-discovery/policy-defaults.yml. The same decisions in machine-readable form, with the reason
   for each value in comments.
3. soup-discovery/IMPLEMENTATION.md. Which script does what, and which configuration key reaches it.
4. patches/README.md. Which file goes into which repository, and the order to wire a product.
5. soup-discovery/scripts/classify-findings.py and group-remediation.py. The two places where a
   wrong rule would be hardest to notice.

Not in this PR

• No product is wired. The callers are staged under patches/product-repo/ as templates. Nothing
  runs on a schedule yet, which also means no timeframe is running: both clocks start on the date a
  run first reports a finding.
• The scope declarations are drafts derived from reading the repositories, and they belong in the
  product repositories. Each needs confirmation by its project before the product is released under
  this document.
• Object store credentials are not distributed. Without them a run keeps its records as 90-day
  workflow artefacts and the reconciliation cannot cover a longer period.
• Tool validation for this tooling is a separate piece of work and is not part of this PR.

One thing worth a second opinion

The maintenance window model replaced a version that derived the deadline from each product's
observed release rhythm. That did not work: three of four products had a rhythm that had already
lapsed, so a finding was born overdue. One product's would have been 109 days late on the day it was
found. A product now declares a commitment instead, and the deadline is the next window on that grid.
The commitment is capped by the SLA tier and the cap cannot be waived. The rules are under
WI: The maintenance window.

[grafele]

End-to-end verified on a real runner — six iterations on a temporary branch (mindnet@soup/e2e-test, workflow runs only there, touches no releases or stores; delete after merge). Both composite actions pinned to this PR branch. Final state: both jobs green, and the monitor record on the runner is byte-for-byte what the design says it should be:

"verdict": "incomplete",
"not_scanned": [
  {"name": "Production", "version": "v1.0.15", "why": "release exists but carries no sbom-v1.0.15.cdx.json — released before the SBOM pipeline, …"},
  {"name": "mobile",     "version": "v1.0.15", "why": "production release carries no SBOM asset"}
]

That is the honest pre-adoption answer; it becomes an all-clear the moment a release carries its SBOM asset. The SBOM job assessed the full repository on the runner: 34 candidates, scope gate 20/14/0, 5442 components, 1725 findings → 118 remediation units, 0 in KEV, three grq-4 contradictions surfaced against real SOUP records.

The iterations found five defects no local test could reach, all fixed on this branch with regression tests (159 total now):

1. 35bb128 — the action never passed a policy to the pipeline, so no CI bundle had ever been assessed. Local runs always passed it by hand.
2. The runner lacked gh; the prerequisite check caught it loudly (runner provisioning is the durable fix; the e2e bootstraps a pinned binary meanwhile).
3. 3fa73bd — with contents: read only, the per-environment deployments fetch 403s; the old || continue silently dropped Production. Environments the token cannot read are now named gaps, and the callers grant deployments: read.
4. 670bb47 — gh api prints the HTTP error body to stdout; captured unchecked, {"message":"Resource not accessible…"} became a phantom environment name and every real environment vanished errorlessly. Error bodies are never accepted as data now, and the env names fall back to the deployment records themselves.
5. 8e44649 — Linux caps a single process argument at 128KB (MAX_ARG_STRLEN); folding deployment pages via --argjson blew it, jq died, and the empty substitution emptied every downstream list. macOS has a larger limit, which is why 150+ local tests never saw it. Pages accumulate in files now, and a ~400KB fixture reproduces the failure on any Linux machine the suite runs on.

Pattern across all five: the wrong answer looked exactly like a right one. Every fix converts silence into a named reason.