Add cache validation

[inteon]

Add verify-cache command that validates the cache, preventing cache poisoning.

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [inteon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[maelvls]

Does it mean we need to run make verify-cache in every single Prow job? I imagine that's not a big downside, but I wish it was a bit more automatic

[inteon]

Yes, I'll update the prowjob definitions to first call make verify-cache.

[maelvls]

at first, I didn't believe it as I thought that the inode modification timestamp mismatch would prevent the attack. And I thought that the report was wrong as it mentioned poisonning _bin/downloaded/kind which doesn't actually exists...

I realized that I was wrong.

$ echo $'#!/bin/bash\necho I GOT YOU>&2'> ~/.cache/makefile-modules/downloaded/tools/kind@v0.31.0_darwin_arm64 && chmod +x ~/.cache/makefile-modules/downloaded/tools/kind@v0.31.0_darwin_arm64
$ make _bin/tools/kind
$ _bin/tools/kind
I GOT YOU

I'm surprised that we aren't verifying the hash when creating the symlink. Right now, we verify after downloading and then never verify again.

And since some artifacts are large tarballs, we would need check their tarballs' hash and untar them just before creating the symlink.

My recommendation would be exactly this: instead of uncompressing + checking hash on download, I'd uncompress + check hash upon creating the symlink. This would 100% mitigate the issue IMO.

Quick proof of concept for verifying hashes at runtime instead of at the time of download: cert-manager/cert-manager#8833

[wallrj]

Thanks @inteon — this has grown into the right shape: hash-check every restored file against the reviewed SHA-256 in 00_mod.mk, purge anything unknown or mismatched, and go mod verify the module cache. The reviewed hash is the trust anchor, so a poisoned file simply fails to match and is removed — no signing or fork-scoping needed.

Two thoughts:

1. It's a step you have to remember. verify-cache only protects a job that calls it, after restore and before any tool runs. Nothing enforces that ordering, so a job that forgets is silently unprotected. @maelvls' make: check the hash at runtime instead of after downloading cert-manager#8833 verifies at symlink-creation time, which has the opposite trade-off: automatic, but only fires for $(NEEDS_*) targets and is bypassed by anything that runs _bin/tools/<x> off PATH directly. The strongest version is content-addressing: key the cache directory by the reviewed hash, so a poisoned file isn't found under its expected key and verification collapses into a lookup — no separate step, no PATH bypass. Short of that, at least wire verify-cache in as a documented, mandatory first step after every cache restore.

2. go mod verify is belt-and-braces. GOMODCACHE is already verified against go.sum/GOSUMDB on every use, so a poisoned module cache is caught at build time regardless. No harm keeping it, but it's not where the exposure is — the tool cache, verified only at download until this PR, is.

Net: I'd like the tool-cache half of this to land here in makefile-modules, folding in cert-manager/cert-manager#8833's "verify at point of use, not just at download" insight rather than running it as a separate command. Happy to help push it over the line.

with claude fable-5