Key go-installed tool binaries on the Go toolchain that builds them

[wallrj-cyberark]

Motivation

The Go 1.27 upgrade (#704) is breaking downstream repositories in a way which does not reproduce on a clean checkout — and which comes and goes between CI runs.

In cert-manager/cert-manager#9174 (the Renovate PR which vendors this module at 455529c), make-verify failed because a cached openapi-gen binary, built by Go 1.26, can no longer parse the Go 1.27 standard library (build log):

+++ Generating ACME openapi...
OpenAPI code generation error: failed making a parser: error(s) in "math/rand/v2":
.../goroot/src/math/rand/v2/rand.go:213:17: method must have no type parameters

A tool built from source is cached at:

$(DOWNLOAD_DIR)/tools/<tool>@<version>_<os>_<arch>

That path says nothing about the Go toolchain which produced the binary, so bumping VENDORED_GO_VERSION does not invalidate it. The download directory is deliberately persisted between CI runs — for cert-manager, preset-local-cache copies _bin/downloaded forward from a hostPath cache shared per Prow node; for other repositories it is a GitLab CI or GitHub Actions cache.

The failure is intermittent because it depends on which node a job lands on. Failed jobs do not write back their cache ("Local cache [update]: Job failed, not updating cache"), and cert-manager's master branch — still on Go 1.26.6, with openapi-gen at the same tool version — keeps re-priming node caches with a Go 1.26-built binary at the same path. A run scheduled on a master-primed node fails; a run on a node with a cold cache rebuilds the tool with Go 1.27 and passes (as the later runs of cert-manager/cert-manager#9174 did, where the job now fails only on the golangci-lint findings addressed by cert-manager/cert-manager#9175). The stale binaries cannot be cleared without purging the node caches by hand.

Changes

Include the Go toolchain version in the download path of tools built with go install:

$(DOWNLOAD_DIR)/tools/<tool>@<version>_<goversion>_<os>_<arch>

where <goversion> is the version of the toolchain which actually builds the tool: go$(VENDORED_GO_VERSION) when Go is vendored, otherwise the system Go's go env GOVERSION. Keying on the actual toolchain means a system Go upgrade also invalidates the cache, a repository which does not vendor Go cannot mislabel binaries in the shared $HOME/.cache/makefile-modules cache, and such a repository does not rebuild every tool when a module bump changes a VENDORED_GO_VERSION it never uses.

To achieve this:

• tool_defs now exports a $(XXX_DOWNLOAD_PATH) variable instead of declaring the symlink rule directly; go_dependency overrides that variable for the tools it builds; and the symlink rule is generated afterwards, for every tool, from $(XXX_DOWNLOAD_PATH).
• The versioned binary is a normal (not order-only) prerequisite of the unversioned $(bin_dir)/tools/xxx symlink, so rebuilding the binary always re-points the symlink. In the steady state the symlink resolves to that same file, so nothing is remade. Go-built tools additionally depend on a $(bin_dir)/scratch/GO_TOOLCHAIN_VERSION stamp file, which catches reverting to an older, already-cached toolchain, where file modification times cannot be trusted.

Downloaded (non-Go) tools are unaffected: their paths and their learned checksums are unchanged, so make learn-tools-shas needs no rerun.

Known limitations:

• The first build after this lands will rebuild every Go tool once, and the previous, unsuffixed binaries will linger in the caches until they are evicted. No eviction is added here: pruning superseded _go*_ binaries would thrash the $HOME cache shared between repositories on different toolchains.
• If a tool's own go.mod carries a toolchain directive which upgrades beyond the invoking Go, the label understates the truth. That does not affect the problem being fixed here, which is that a Go upgrade must invalidate previously built binaries.

Testing

Verified locally against the two repositories where the breakage was observed, by copying the patched module over their vendored copy:

• cert-manager/cert-manager at #9174 — removed _bin/tools/openapi-gen and rebuilt: the binary was rebuilt at openapi-gen@v0.0.0-20260721132016-d427ff9ee9ad_go1.27.0_linux_amd64 and make generate-codegen then produced no diff, i.e. the make-verify failure on that pull request is entirely the stale-binary problem.
• venctl (vendors this module) — a fresh build produced klone@v0.3.0_go1.27.0_linux_amd64; a repeat run was a no-op; setting VENDORED_GO_VERSION := 1.26.5 rebuilt and relinked to klone@v0.3.0_go1.26.5_linux_amd64, and switching back relinked again without rebuilding.
• Simulated a non-vendored downstream repository with a stub go on PATH: a fresh build cached klone@v0.3.0_go1.24.5_linux_amd64 — the system toolchain, not VENDORED_GO_VERSION; a repeat run was a no-op; a toolchain bump rebuilt and re-pointed the symlink; a downgrade re-pointed to the already-cached binary without rebuilding; and a pre-existing symlink whose modification time was newer than every stamp file was still re-pointed after the rebuild.
• Downloaded tools were checked to be unchanged: helm@v4.2.4_linux_amd64, no suffix, no re-download.
• make test-e2e passes in this repository.
• scripts/learn_tools_shas.sh was exercised in dry-run mode for one Go tool and one downloaded tool, to confirm the helper makefile still evaluates cleanly under --warn-undefined-variables.

with claude fable-5

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign erikgb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Keys Go-installed tool binaries by vendored Go version, preventing stale cached binaries after toolchain upgrades.

Changes:

• Adds per-tool download path variables.
• Adds Go versions to Go-built binary paths.
• Relinks tools when the vendored Go version changes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.