fix(strix): serialize scans per repository to stop shared-key rate-limit storms - #1297
fix(strix): serialize scans per repository to stop shared-key rate-limit storms#1297seonghobae wants to merge 9 commits into
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📝 WalkthroughWalkthroughStrix 워크플로의 동시성 그룹을 저장소·이벤트 유형 기준으로 변경했다. 진행 중인 실행과 대기 실행을 유지한다. 종료된 PR의 기존 실행은 별도 작업에서 GitHub API로 취소한다. 관련 계약 테스트와 변경 로그를 갱신했다. ChangesStrix 실행 제어
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🟠 High · up to The repository-level scan serialization change is not ready to merge because its contract test currently conflicts with the workflow, queued scans can still be dropped beyond the concurrency limit, and cleanup may cancel another pull request’s scan when commits are shared. These issues can cause required evidence to be skipped or the wrong validation run to be canceled. Sequence Diagram(s)sequenceDiagram
participant PR as pull_request_target
participant WF as Strix 워크플로
participant API as GitHub Actions API
PR->>WF: closed 이벤트 전달
WF->>API: Strix 실행 목록 조회
API-->>WF: queued 및 in_progress 실행 반환
WF->>API: 현재 실행을 제외한 일치 실행 취소
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…mit storms Root cause: the per-PR concurrency group let sibling PRs in one repository scan concurrently; each run retried the shared NVIDIA NIM key up to three times, producing litellm.RateLimitError storms and fail-closed gate failures on every open PR (observed across ContextualWisdomLab/contextual-orchestrator 2026-08-23/24). Change: scope the concurrency group per repository (event class still separated so required pull_request_target evidence never interleaves with default-branch repository_dispatch retries), set cancel-in-progress: false with queue: max so queued evidence runs are preserved, and update the queue contract test to encode the new serialization contract. Accuracy is prioritized over scan latency; queued runs already fetch the expected head SHA directly, so late-started runs stay head-exact.
60390f3 to
d7133e2
Compare
…evel concurrency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Current-head review loop
Please run the current-head review and required Checks again. No merge is requested until those exact-head gates and an eligible approval are present. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Exact-head validation and review disposition for ab17b10 (base 613a33e):
The remaining informational event-class thread is dispositioned by the documented contract above. Current required hosted Checks are still QUEUED and no current independent approval is present; predecessor evidence is not reused, so the PR remains on the normal protected path and is not merged yet. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Additional Inkspan-bound acceptance evidence for this existing Strix capacity owner; no duplicate source writer or gate bypass is being created.
This broadens the reproduced rate-limit failure from contextual-orchestrator to the Inkspan review-control lane and supports the repository/event-class serialization RCA here. After this PR is protected-main integrated, use a fresh legitimate generation rather than predecessor-run evidence: rerun the then-current |
Current-head review disposition
This is an accepted latency tradeoff with a bounded recovery path, not a source defect or deadlock. Required hosted checks and independent approval remain pending; keep the PR on the protected normal path. |
Exact-head reviewRevalidated current head Local evidence: |
|
Additional exact-head validation for |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
309bbf5965c30ae2918b0ffabf66670b99ed7d13. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- Strix Security Scan/strix: FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/32719353720/job/97407260950)
- Strix Security Scan/strix: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/32719353720/job/97407260950)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["CI script: test_strix_quick_gate.sh"]
S3 --> I3["review and security gate shell path"]
I3 --> R3["Review risk: CI script: test_strix_quick_gate.sh"]
R3 --> V3["bash -n plus Strix self-test"]
Evidence --> S4["Test: test_required_workflow_queue_contract.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_required_workflow_queue_contract.py"]
R4 --> V4["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed. Findings1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
Failed checks:
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["CI script: test_strix_quick_gate.sh"]
S3 --> I3["review and security gate shell path"]
I3 --> R3["Review risk: CI script: test_strix_quick_gate.sh"]
R3 --> V3["bash -n plus Strix self-test"]
Evidence --> S4["Test: test_required_workflow_queue_contract.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_required_workflow_queue_contract.py"]
R4 --> V4["targeted test run"]
|
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
309bbf5965c30ae2918b0ffabf66670b99ed7d13. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- Strix Security Scan/strix: FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/32719353720/job/97407260950)
- Strix Security Scan/strix: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/32719353720/job/97407260950)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["CI script: test_strix_quick_gate.sh"]
S3 --> I3["review and security gate shell path"]
I3 --> R3["Review risk: CI script: test_strix_quick_gate.sh"]
R3 --> V3["bash -n plus Strix self-test"]
Evidence --> S4["Test: test_required_workflow_queue_contract.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_required_workflow_queue_contract.py"]
R4 --> V4["targeted test run"]
|
Live queue evidence 2026-08-25T02:37Z, not OpenCode approval and not merge evidence. This exact head While this job holds a runner, #1316@
|
|
Same-head required Strix Overlay fallbacks were still luna (
That is the third hub reproduction this hour of #1311 (with #1311 |
| cancel_runs() { | ||
| local status="$1" | ||
| local runs_url="repos/${TARGET_REPOSITORY}/actions/runs?event=pull_request_target&status=${status}&per_page=100" | ||
| gh api --paginate "$runs_url" | | ||
| jq -r --arg pr "$CLOSED_PR_NUMBER" --arg head_sha "$CLOSED_PR_HEAD_SHA" --arg current "$CURRENT_RUN_ID" ' | ||
| .workflow_runs[] | ||
| | select((.id | tostring) != $current) | ||
| | select(.name == "Strix Security Scan") | ||
| | select(.head_sha == $head_sha or any(.pull_requests[]?; ((.number | tostring) == $pr))) | ||
| | .id | ||
| ' | | ||
| while IFS= read -r run_id; do | ||
| [ -n "$run_id" ] || continue | ||
| gh api --method POST "repos/${TARGET_REPOSITORY}/actions/runs/${run_id}/cancel" >/dev/null | ||
| echo "Cancelled Strix run ${run_id} for closed PR #${CLOSED_PR_NUMBER}." | ||
| done | ||
| } | ||
|
|
||
| cancel_runs queued | ||
| cancel_runs in_progress |
There was a problem hiding this comment.
🔍 Closed-PR cleanup misses repository_dispatch and fork-PR runs
cancel_runs filters only event=pull_request_target, so a repository_dispatch scan for the closed PR is not cancelled. The pull_requests[] matcher is empty for fork runs; if head_sha also does not match, a closed fork PR's in-progress scan is left running and holds the per-repo queue. Same-repo PRs match by head SHA and PR number.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Root cause (5 Whys)
litellm.RateLimitError: Nvidia_nimException(429) and fail-closed evidence rules.pr-{N}), so sibling PRs never shared a queue.Fix
pull_request_targetevidence never interleaves with default-branchrepository_dispatchretries).cancel-in-progress: false+queue: max: queued evidence runs are preserved — nothing is dropped; accuracy prioritized over latency.refs/pull/<n>/head has already advancedhandling), so late-started runs stay head-exact.Evidence
Nvidia_nimException - RateLimitError, all fallback models exhausted, zero-finding-before-infrastructure-failure fail-closed.pytest tests/ -q→ 1342 passed (+ updated queue-contract tests), YAML validated.Summary by CodeRabbit
새로운 기능
문서
버그 수정