Skip to content

Harden PyPI install verification against stale cache - #102

Open
lelia wants to merge 3 commits into
mainfrom
lelia/release-pypi-verification
Open

Harden PyPI install verification against stale cache#102
lelia wants to merge 3 commits into
mainfrom
lelia/release-pypi-verification

Conversation

@lelia

@lelia lelia commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What changed

  • force every post-publish install attempt to bypass pip's HTTP cache
  • explicitly verify against the canonical https://pypi.org/simple/ index
  • use python -m pip consistently for install/uninstall
  • extend verification from 10 minutes (30 × 20s) to 30 minutes (60 × 30s) and avoid sleeping after the final failure
  • on failed attempts, log when the release already exists on the JSON API — making "index propagation delay" distinguishable from "publish actually failed" in the logs
  • quote workflow output files and the version-check URL so the workflow passes actionlint

The verify step is now line-for-line consistent with the same hardening in SocketDev/socket-python-cli#290.

Why

The 3.4.2 release uploaded both distributions successfully, and PyPI's JSON API/project page showed the release, but the post-publish pip install socketdev==3.4.2 loop repeatedly saw a stale Simple-index response whose version list ended at 3.3.0. It ultimately exhausted all 30 attempts and failed after 10m52s.

pip caches HTTP responses by default. If the first install attempt races PyPI index propagation, the existing loop can reuse that initial stale response on every retry. It also performs its 30 checks at roughly 0–580 seconds and then sleeps once more before failing, so it never makes a post-expiry check beyond the Simple index's observed 10-minute cache lifetime.

Root cause findings (cross-repo)

This was not specific to this repo or release. socket-python-cli hit the identical failure the same day: socketsecurity v2.5.9 (uploaded 16:40 UTC) failed its release verify loop with the Simple index stuck at 2.5.8 for the full 10-minute budget, which also skipped its Docker publish. Two independent packages, seven hours apart, same signature: JSON API fresh immediately, CDN-cached Simple index stale for 10+ minutes.

Notably, while this repo's verify loop was failing (23:30–23:39 UTC), other CI runners were successfully installing socketdev==3.4.2 from PyPI — the staleness was specific to individual CDN cache nodes, i.e. a delayed/partial purge propagation on PyPI's side. PyPI's status page reported no incident, and pip was ruled out as a cause (the previous day's successful releases used the same pip 26.2.1). Both indexes recovered on their own.

This is a latent race rather than a package compatibility failure. The same verification loop was present in the successful 3.3.0 release, where attempt 1 missed the new version and attempt 2 happened to find it 20 seconds later — the propagation lag has always existed; on 2026-08-05 it grew past the loop's budget.

With this change, every retry performs a fresh lookup and the retry budget extends well beyond the observed staleness window.

Validation

  • actionlint .github/workflows/release.yml
  • parsed .github/workflows/release.yml with Ruby's YAML parser
  • git diff --check

References

Prior reports of the same PyPI Simple-index staleness class:

@lelia
lelia marked this pull request as ready for review August 6, 2026 00:31
@lelia
lelia requested a review from a team as a code owner August 6, 2026 00:31
@lelia

lelia commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 07ae95f. Configure here.

Match the socket-python-cli release workflow hardening (PR #290 there):
the 2026-08-05 propagation delay exceeded 10 minutes from the release
runner's vantage point, so extend the retry budget to 30 minutes, and
log when the JSON API already has the version but the Simple index does
not, making CDN propagation delay distinguishable from a failed publish
in the logs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: lelia <2418071+lelia@users.noreply.github.com>
@lelia lelia changed the title fix(ci): harden PyPI install verification against stale cache Harden PyPI install verification against stale cache Aug 6, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: lelia <2418071+lelia@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant