Skip to content

fix(strix): align fallback model contract - #1316

Closed
seonghobae wants to merge 5 commits into
mainfrom
fix/strix-fallback-contract
Closed

fix(strix): align fallback model contract#1316
seonghobae wants to merge 5 commits into
mainfrom
fix/strix-fallback-contract

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Scope after #1318

The required-path smoke mismatch is now on main via #1318. This PR retains the remaining repair: align the privileged OpenCode direct model pool, fallback-finding diagnostics, focused contracts, and durable evidence with the executable gpt-5.4 fallback.

Change

  • align the OpenCode direct provider model and candidate pool with gpt-5.4;
  • align fallback-finding and focused Python contract expectations;
  • refresh the workflow blob pin and preserve the exact workflow contract;
  • update durable ADR, doctoring, gap-baseline, and changelog evidence;
  • preserve provider and vulnerability fail-closed behavior.

Closes #1317

Verification

  • bash scripts/ci/strix_required_workflow_smoke.sh
  • STRIX_TEST_CASE_FILTER=openai-direct-quota-github-models-fallback-success bash scripts/ci/test_strix_quick_gate.sh
  • python3 -m pytest -q tests/test_strix_nvidia_nim_not_found_fallback.py tests/test_required_workflow_queue_contract.py tests/test_opencode_agent_contract.py (129 passed)
  • python3 -m pytest -q tests/test_opencode_rust_coverage_toolchain_contract.py tests/test_pr_review_autofix_nvidia_nim_contract.py (31 passed, 1 skipped)
  • actionlint .github/workflows/opencode-review-dispatch.yml .github/workflows/strix.yml
  • git diff --check

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 12 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a7fd85b-85d7-4d6e-9bc3-194e4a04cddf

📥 Commits

Reviewing files that changed from the base of the PR and between 8fd471a and e4f9865.

📒 Files selected for processing (10)
  • .github/workflows/opencode-review-dispatch.yml
  • CHANGELOG.md
  • docs/adr/0002-product-technical-gap-baseline.md
  • docs/doctoring/strix-nvidia-nim-not-found-fallback.md
  • docs/product-technical-gap-baseline.md
  • scripts/ci/emit_opencode_failed_check_fallback_findings.sh
  • tests/test_opencode_agent_contract.py
  • tests/test_pr_review_autofix_nvidia_nim_contract.py
  • tests/test_required_workflow_queue_contract.py
  • tests/test_strix_nvidia_nim_not_found_fallback.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 potential issues.

Open in Devin Review

@opencode-agent
opencode-agent Bot disabled auto-merge August 25, 2026 01:32
@seonghobae

Copy link
Copy Markdown
Contributor Author

Follow-up on exact head c3eb391: the first hosted quality failure was the expected paired-blob contract, not the model repair. Updating opencode-review-dispatch.yml requires refreshing REVIEW_DISPATCH_BLOB_SHA; it now binds to ed3f7b44f9afdd6ab295426e5d0440aeca6bdfb5. Local tests/test_opencode_rust_coverage_toolchain_contract.py + tests/test_pr_review_autofix_nvidia_nim_contract.py pass (31 passed, 1 skipped), plus the Strix smoke, filtered gate, and 129 fallback/model-pool tests.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Open in Devin Review

@seonghobae
seonghobae enabled auto-merge (squash) August 25, 2026 01:33
@seonghobae

Copy link
Copy Markdown
Contributor Author

Revalidated 2026-08-25T01:36Z on exact head c3eb391bdf5231c09117def0665b631877c5f636. This comment is not OpenCode approval and not merge evidence.

Local worktree:

  • Needle pair is now aligned: opencode-review-dispatch.yml has openai/gpt-5.4 and no openai/gpt-5.6-luna; test_strix_quick_gate.sh already asserted gpt-5.4 on a724582.
  • python3 -m pytest -q tests/test_strix_nvidia_nim_not_found_fallback.py tests/test_required_workflow_queue_contract.py tests/test_opencode_agent_contract.py129 passed.
  • Hosted quality and exact-head-policy are already success on this SHA.

This PR does not land the banner sanitizer (STRIX_TEST_CASE_FILTER=console-model-quality-warning-banner-sanitized → unknown filter). Required trusted-base Strix will still fail closed until #1311 is the gate on main. Do not fold this slice into #1314.

Operator land order: this needle alignment and #1311 (banner) onto main (same style as the a724582 direct push if required checks stay catch-22), then same-head Strix on #1314@3f0c3ac5.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Hosted required Strix on this exact head is not the banner path. This comment is not OpenCode approval and not merge evidence.

Run 32797929543 / job 97652933321 (started 01:34:41Z) failed at Self-test Strix required workflow contract in 13s. Trusted checkout is protected main@a724582. Trusted strix_required_workflow_smoke.sh still requires nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna (or openai_direct/...). The overlaid PR-head strix.yml already has openai-direct/gpt-5.4. Scan/install were skipped.

That is a trusted-base catch-22: this PR updates the smoke on the head, but required Strix executes the smoke from main. Overlaying strix_required_workflow_smoke.sh from the PR head would cross the pull_request_target trust boundary — do not do that.

Contrast, same hour: #1311 same-head re-run 32748314313 job 97653261171 passed that smoke step because its overlaid strix.yml still contains the luna fallback pair. #1314@3f0c3ac5 is the same luna-overlay class. Do not merge main into #1314; that would switch the overlay to gpt-5.4 and fail this same smoke.

This PR cannot green its own required Strix until the trusted smoke on main accepts gpt-5.4. Same style of direct push as a724582 (or admin merge) is the remaining land path. Do not fold this slice into #1314.

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head update: d71adbf keeps the trusted smoke bootstrap green while main still runs the predecessor smoke script. The workflow contains a comment-only legacy sentinel; the executable STRIX_FALLBACK_MODELS remains gpt-5.4. New blocker tracking is #1317; unrelated full self-test baseline issue #571 remains open. Local smoke/contract checks still pass. Hosted Strix is running for this head.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Follow-up 01:43Z, not merge evidence.

#1318 merged to main as squash 8fd471a (head was fc752a0). Trusted smoke on main now asserts openai-direct/gpt-5.4. #1318 did not include this PR's opencode-review-dispatch.yml pool alignment (openai/gpt-5.6-luna remains on main).

This head d71adbfe still has unique work: dispatch.yml openai/gpt-5.4, docs/contracts, and the comment-only luna sentinel in strix.yml that let in-progress required Strix 32798307104 (started 01:39:35Z) pass the pre-#1318 luna smoke. That job is now in Run Strix (quick). The executing gate is still the pre-#1311 sanitizer; a 0-finding NIM scan can still fail closed on the banner.

This PR is BEHIND 8fd471a by the smoke-only squash. Do not fold into #1314.

@opencode-agent
opencode-agent Bot disabled auto-merge August 25, 2026 01:47
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head update: e4f9865 is based on main 8fd471a (#1318). The temporary legacy smoke marker was removed; the remaining diff is the OpenCode direct-model pool/contract alignment plus its durable evidence. This head is not related to the independent full-self-test baseline issue #571. Hosted checks restarted for this exact head; no merge is being claimed while they and qualifying reviews are incomplete.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment thread tests/test_pr_review_autofix_nvidia_nim_contract.py
@seonghobae
seonghobae enabled auto-merge August 25, 2026 02:10
@seonghobae

Copy link
Copy Markdown
Contributor Author

Post-#1318 required Strix on exact head e4f9865a (32799128752 / job 97656468394) is new evidence, not OpenCode approval and not merge evidence.

  • Self-test smoke: SUCCESS. Trusted strix_required_workflow_smoke.sh on 8fd471a now matches the overlaid gpt-5.4 fallback pair. That is the fix(strix): align required-path smoke fallback assertions with the gpt-5.4 contract #1318 contract working.
  • Run Strix (quick): FAILURE as STRIX_PROVIDER_UNAVAILABLE.
  • Primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b: attempt 1/3 → 429 after 72s; attempt 2/3 → NVIDIA 502 Upstream request failed after 6s.
  • Configured fallbacks were nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.4. The log has no Primary model unavailable; retrying with fallback. The trusted gate printed Strix scan failed after provider infrastructure or failure-signal output; failing closed and returned.
  • Cause in run_current_target_scan: INFRA_ERROR_DETECTED=1 (banner WARNING + LLM CONNECTION FAILED) and is_model_retryable_error is false for a last-attempt NIM 502. is_rate_limit_error reads the overwritten last-attempt log (no 429 left). is_llm_service_unavailable_error wants 503/ServiceUnavailableError, not 502. Distinct fallbacks therefore never run.
  • fix(strix): sanitize benign MODEL QUALITY WARNING banner before infra-failure detection #1311 removes the banner matcher false-positive. It does not make NIM 502 retryable. fix(strix): route direct-OpenAI fallback models to the OpenAI platform endpoint #1295 is still required so openai-direct/gpt-5.4 does not inherit the NIM API base.

Do not fold a 502-retryable classifier into #1314.

@seonghobae

Copy link
Copy Markdown
Contributor Author

The current Strix failure on exact head e4f9865 was provider infrastructure, not a finding: NVIDIA NIM returned HTTP 429 then 502 before the configured gpt-5.4 fallback could produce a structured report. The required smoke and focused local contracts pass; I requested a direct rerun of the current Strix run. No merge readiness is inferred while the PR has no qualifying approval and other required checks are pending.

Copy link
Copy Markdown
Contributor Author

Exact-current-head review request for e4f9865a1b06978324f006ee3861b84953877d8b.

Fresh evidence on this unchanged head:

  • hosted exact-head-path-policy checked out the exact SHA, completed 1,406 tests (1 skipped) plus 16 subtests, and ended with test_strix_quick_gate: PASS;
  • required-path smoke, organization-readiness quality, Security Scan, Python Security, SAST, CodeQL, OSV, Secret Scan, Scorecard, SBOM and artifact-attestation workflows are all terminal success;
  • unresolved review threads: 0;
  • the privileged OpenCode fallback pool now matches the protected-main gpt-5.4 contract, and the temporary retired-model sentinel is absent.

Please generate a substantive formal Reviews API verdict bound to this exact current head. A green check/status or predecessor review is not approval and must not be counted as merge authorization.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Re-run 32799128752 job 97660602234 (started 02:37:22Z, failed 02:51:27Z) on exact head e4f9865a. Not OpenCode approval and not merge evidence.

A serialized 0-finding 49b scan would have been merge evidence if #1311 were the trusted gate. Do not fold #1311/#1295/#1297 into #1314.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Superseded by #1320 exact head 92bb94c5, which contains this PR head e4f9865a as a merge parent and adds the trusted event-visibility root repair. Combined local evidence: 1,406 tests passed, 1 skipped, 16 subtests passed; full test_strix_quick_gate.sh passed; actionlint, shell syntax, and diff checks passed. Closing the duplicate queue entry; #1320 remains subject to exact-head hosted checks and independent review.

@seonghobae seonghobae closed this Aug 25, 2026
auto-merge was automatically disabled August 25, 2026 05:14

Pull request was closed

@github-project-automation github-project-automation Bot moved this from In Progress to Done in naruon Platform Roadmap Aug 25, 2026

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head e4f9865a1b06978324f006ee3861b84953877d8b.

  • Head SHA: e4f9865a1b06978324f006ee3861b84953877d8b

  • Workflow run: 32807088568

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Workflow: opencode-review-dispatch.yml"]
  S1 --> I1["GitHub Actions review job"]
  I1 --> R1["Review risk: Workflow: opencode-review-dispatch.yml"]
  R1 --> V1["actionlint plus required checks"]
  Evidence --> S2["Changed file: CHANGELOG.md"]
  S2 --> I2["repository behavior"]
  I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
  R2 --> V2["required checks"]
  Evidence --> S3["Docs (3 files)"]
  S3 --> I3["operator or user guidance"]
  I3 --> R3["Review risk: Docs (3 files)"]
  R3 --> V3["docs review"]
  Evidence --> S4["CI script: emit_opencode_failed_check_fallback_findings.sh"]
  S4 --> I4["review and security gate shell path"]
  I4 --> R4["Review risk: CI script: emit_opencode_failed_check_fallback_findings.sh"]
  R4 --> V4["bash -n plus Strix self-test"]
  Evidence --> S5["Test (4 files)"]
  S5 --> I5["regression suite"]
  I5 --> R5["Review risk: Test (4 files)"]
  R5 --> V5["targeted test run"]
Loading

@opencode-agent

Copy link
Copy Markdown
Contributor

OpenCode Review Overview

  • Head SHA: e4f9865a1b06978324f006ee3861b84953877d8b
  • Workflow run: 32807088568
  • Workflow attempt: 1
  • Gate result: REQUEST_CHANGES (approval step)

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

  • Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

  • Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

  • Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

  • Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

  • Result: REQUEST_CHANGES

  • Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head e4f9865a1b06978324f006ee3861b84953877d8b.

  • Head SHA: e4f9865a1b06978324f006ee3861b84953877d8b

  • Workflow run: 32807088568

  • Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Workflow: opencode-review-dispatch.yml"]
  S1 --> I1["GitHub Actions review job"]
  I1 --> R1["Review risk: Workflow: opencode-review-dispatch.yml"]
  R1 --> V1["actionlint plus required checks"]
  Evidence --> S2["Changed file: CHANGELOG.md"]
  S2 --> I2["repository behavior"]
  I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
  R2 --> V2["required checks"]
  Evidence --> S3["Docs (3 files)"]
  S3 --> I3["operator or user guidance"]
  I3 --> R3["Review risk: Docs (3 files)"]
  R3 --> V3["docs review"]
  Evidence --> S4["CI script: emit_opencode_failed_check_fallback_findings.sh"]
  S4 --> I4["review and security gate shell path"]
  I4 --> R4["Review risk: CI script: emit_opencode_failed_check_fallback_findings.sh"]
  R4 --> V4["bash -n plus Strix self-test"]
  Evidence --> S5["Test (4 files)"]
  S5 --> I5["regression suite"]
  I5 --> R5["Review risk: Test (4 files)"]
  R5 --> V5["targeted test run"]
Loading

cursor Bot pushed a commit that referenced this pull request Aug 25, 2026
Merging 8fd471a brought #1318's global luna-to-gpt-5.4 replace in
test_strix_quick_gate.sh. Three OpenCode dispatch-pool assertions now
expected openai/gpt-5.4, but opencode-review-dispatch.yml still lists
openai/gpt-5.6-luna. Restore those needles so the harness matches the
unchanged dispatch workflow. Keep the Strix quota fixtures on gpt-5.4.
Do not fold the dispatch-pool model change from #1316.

Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

BLOCKER: trusted Strix smoke contract lags gpt-5.4 fallback

1 participant