Skip to content

fix(strix): replace dead GitHub Models fallback with direct OpenAI #1226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seonghobae merged 3 commits into from
Aug 22, 2026
Merged

fix(strix): replace dead GitHub Models fallback with direct OpenAI #1226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
48c57df
fix(strix): replace dead GitHub Models fallback with direct OpenAI
seonghobae Aug 22, 2026
60af164
fix(strix): update NVIDIA-fallback regression test for the new fallba…
seonghobae Aug 22, 2026
67441eb
Merge remote-tracking branch 'origin/main' into fix/strix-dead-github…
seonghobae Aug 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/strix.yml
View file
Open in desktop

@devin-ai-integration devin-ai-integration Bot Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: GitHub Models fallback credential step is now unused

The Prepare GitHub Models fallback credentials step (.github/workflows/strix.yml:666-687) provisions STRIX_GITHUB_MODELS_KEY_FILE/STRIX_GITHUB_MODELS_API_BASE_FILE, which the gate consumes only for github_models/* fallbacks. This PR removed every github_models/* entry from the fallback list, so the step now provisions credentials nothing reads. Its presence is still pinned by strix_required_workflow_smoke.sh:150 and test_strix_quick_gate.sh:352.

(Refers to this code)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -812,7 +812,7 @@ jobs:
STRIX_LLM_MAX_RETRIES: 1
STRIX_TRANSIENT_RETRY_PER_MODEL: 2
STRIX_TRANSIENT_RETRY_BACKOFF_SECONDS: 60
STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openai_direct' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'openrouter' && 'github_models/openai/o3 github_models/openai/gpt-5-chat' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat' || '' }}
STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openrouter' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna' || '' }}

@devin-ai-integration devin-ai-integration Bot Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Direct-OpenAI fallback never routes to a valid provider

Every provider mode's fallback is set to openai-direct/gpt-5.6-luna, but the gate only rewrites the underscore form openai_direct/ (child_model_for_api_base at scripts/ci/strix_quick_gate.sh:2310-2317), so this name reaches the model backend as unknown provider openai-direct and the fallback errors. When the primary model is exhausted the required check fails instead of degrading to direct OpenAI — the outcome this change aims to prevent. The primary-model step converts the hyphen form to underscore at strix.yml, but this fallback value skips that step.

Prompt for agents 
The STRIX_FALLBACK_MODELS value at .github/workflows/strix.yml:815 uses the model name openai-direct/gpt-5.6-luna (hyphen) on all provider-mode branches. The Strix gate consumes STRIX_FALLBACK_MODELS directly without the hyphen-to-underscore conversion that the primary-model 'Prepare Strix model input file' step applies (strix.yml:762-763). The gate's child_model_for_api_base (scripts/ci/strix_quick_gate.sh:2310-2317) only strips the openai_direct/ (underscore) prefix down to openai/; it does not recognize openai-direct/ (hyphen), so the fallback name is passed to strix/LiteLLM as an unknown provider and the fallback fails. Change the fallback value to the gate-internal form openai_direct/gpt-5.6-luna on all four branches, and update the assertions that lock in the hyphen form: scripts/ci/strix_required_workflow_smoke.sh:159 and scripts/ci/test_strix_quick_gate.sh:349-351. Confirm consistency with the primary openai_direct model, which the gate already receives as openai_direct/gpt-5.6-luna.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@devin-ai-integration devin-ai-integration Bot Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 openai-direct fallback uses the wrong key/base outside openai_direct mode

In openrouter, nvidia_nim, and github_models modes the new openai-direct fallback would authenticate with the primary provider's key and base, not OpenAI's: child_model_for_api_base reuses LLM_API_KEY/LLM_API_BASE_FILE for non-github, non-vertex models (scripts/ci/strix_quick_gate.sh:2244-2260, 2360-2366), and only STRIX_GITHUB_MODELS_KEY_FILE is provisioned for cross-provider fallback (strix.yml). secrets.STRIX_OPENAI_API_KEY/OPENAI_API_KEY is loaded as LLM_API_KEY only in openai_direct mode (strix.yml). So the graceful-degradation goal holds only for openai_direct primaries; the nvidia_nim branch (the motivating LineageWeave case) would still fail to reach direct OpenAI even after the prefix is corrected.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

STRIX_GITHUB_MODELS_API_BASE_FILE: ${{ env.STRIX_GITHUB_MODELS_API_BASE_FILE }}
STRIX_GITHUB_MODELS_KEY_FILE: ${{ env.STRIX_GITHUB_MODELS_KEY_FILE }}
STRIX_FAIL_ON_PROVIDER_SIGNAL: "1"
Expand Down
28 changes: 28 additions & 0 deletions PR_GOVERNANCE_AUDIT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,34 @@ token and inference endpoint supplied per fallback model
provider quota outage degrades to a slower scan instead of skipping evidence.
The pinned ban on GPT-4.1-or-weaker Strix evidence stays in force.

2026-08-22 KST GitHub Models retirement: GitHub Models entered a platform-wide
"scheduled retirement brownout" returning HTTP 410 for every request
(confirmed directly against `models.github.ai`), so the
`github_models/openai/o3 github_models/openai/gpt-5-chat` universal fallback
wired on 2026-07-13 across all four provider modes (`github_models`,
`openai_direct`, `openrouter`, `nvidia_nim`) is now dead on every path, not
just an outage of one model. Observed impact on LineageWeave PR #392: the
NVIDIA NIM primary and its NVIDIA fallback both failed, cascaded through the
dead GitHub Models fallback, and the run failed the required check entirely
(`STRIX_FALLBACK_MODELS` exhausted) instead of degrading. Separately, the
`nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5` fallback that did run
before the GitHub Models cascade produced a fabricated CRITICAL finding
(a hardcoded-secret report against a `frontend/src/config.ts` line that does
not exist in that form on the target branch) — Strix's own in-run quality
warning already flags this model as "not a recommended frontier model...
weaker models may miss vulnerabilities or produce lower-quality findings",
and this run is a concrete instance of that risk materializing as a false
required-check failure, not a missed finding. Fix: `STRIX_FALLBACK_MODELS`
now falls back to `openai-direct/gpt-5.6-luna` (Strix's own top-recommended
model, already wired via `STRIX_OPENAI_API_KEY`/`OPENAI_API_KEY`) instead of
the dead GitHub Models pair, on all four provider-mode branches. The
`nvidia_nim` branch keeps its NVIDIA-hosted fallback as an interim retry
before this openai-direct fallback; that retains the existing free/low-cost
NVIDIA-first policy and is a separate cost/quality tradeoff this fix does not
revisit. GitHub Models remains a selectable `github_models` primary mode for
now (unchanged scope) but is no longer relied on as a silent universal
fallback.

## Live Repository Inventory

Live generated: 2026-06-26 KST via GitHub REST/GraphQL APIs. PR #28 post-merge refresh: 2026-06-23 16:05 KST. PR #37 post-merge refresh: 2026-06-23 21:50 KST. clearfolio PR #13 post-merge refresh: 2026-06-24 04:48 KST. Non-actionable Findings refresh: 2026-06-25 KST. PR #58, #65, #66, #68, #71, #79, and #80 post-merge refreshes: 2026-06-25 to 2026-06-26 KST. The 2026-07-02 18:15 KST refresh found 17 public non-fork repositories, adding `kaefa` and `waf-ids-ai-soc` to the prior public non-fork inventory. The public fork inventory still contains 6 repositories. `VibeSec` was not in that target set, and `appguardrail` was.
Expand Down
3 changes: 2 additions & 1 deletion scripts/ci/strix_required_workflow_smoke.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,8 @@ assert_file_contains "$gate_script" "NPM_CONFIG_IGNORE_SCRIPTS" "Strix gate disa
assert_file_contains "$full_gate_test" "assert_strix_workflow_pr_trigger_hardened" "Full Strix harness remains available outside the required path"

assert_file_contains "$workflow_file" "nvidia_nim/nvidia/nemotron-3-super-120b-a12b" "Strix defaults public scans to the current hosted NVIDIA NIM model"
assert_file_contains "$workflow_file" "nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat" "Strix tries another NVIDIA hosted model before GitHub Models"
assert_file_contains "$workflow_file" "nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna" "Strix tries another NVIDIA hosted model before falling back to direct OpenAI"
assert_file_not_contains "$workflow_file" "github_models/openai/o3" "Strix fallback list must not depend on GitHub Models, which is in platform-wide retirement"
assert_file_contains "$workflow_file" "Nvidia_nimException" "Strix workflow recognizes provider-scoped NVIDIA NIM failures"
assert_file_contains "$gate_script" "is_nvidia_nim_not_found_error" "Strix gate classifies NVIDIA NIM model-catalog 404s"

Expand Down
7 changes: 4 additions & 3 deletions scripts/ci/test_strix_quick_gate.sh
View file
Open in desktop

@devin-ai-integration devin-ai-integration Bot Aug 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 New fallback path has no functional test coverage

Only string-presence assertions were updated. The single executable fallback scenario (openai-direct-quota-github-models-fallback-success at test_strix_quick_gate.sh) still exercises a github_models/openai/o3 fallback routed to models.github.ai. Nothing runs an openai-direct fallback candidate, so the routing defects go undetected by the harness.

(Refers to this code)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -346,9 +346,10 @@ assert_strix_workflow_pr_trigger_hardened() {
assert_file_contains "$workflow_file" "https://integrate.api.nvidia.com/v1" "strix workflow routes NVIDIA NIM scans to the hosted endpoint"
assert_file_contains "$workflow_file" "LLM_API_BASE_FILE" "strix workflow passes the GitHub Models API base through a trusted input file"
assert_file_not_contains "$workflow_file" '${{ secrets.STRIX_OPENAI_API_KEY || github.token }}' "strix workflow must not use fallback-secret syntax for LLM API keys"
assert_file_contains "$workflow_file" "github_models/openai/o3 github_models/openai/gpt-5-chat" "strix workflow keeps GitHub Models fallback on tool-capable OpenAI models without GPT-4.1 downgrade"
assert_file_contains "$workflow_file" "steps.gate.outputs.provider_mode == 'openai_direct' && 'github_models/openai/o3 github_models/openai/gpt-5-chat'" "strix workflow gives direct-OpenAI scans GitHub Models fallbacks so provider quota outages degrade instead of skipping"
assert_file_contains "$workflow_file" "steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 github_models/openai/o3 github_models/openai/gpt-5-chat'" "strix workflow gives NVIDIA NIM scans contracted fallbacks"
assert_file_contains "$workflow_file" "openai-direct/gpt-5.6-luna" "strix workflow keeps a direct-OpenAI fallback on a tool-capable, Strix-recommended model without GPT-4.1 downgrade"
assert_file_contains "$workflow_file" "steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna'" "strix workflow gives direct-OpenAI scans a same-provider fallback so transient errors degrade instead of skipping"
assert_file_contains "$workflow_file" "steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna'" "strix workflow gives NVIDIA NIM scans contracted fallbacks"
assert_file_not_contains "$workflow_file" "STRIX_FALLBACK_MODELS: \${{ steps.gate.outputs.provider_mode == 'github_models' && 'github_models/openai/o3" "strix workflow fallback list must not depend on GitHub Models, which is in platform-wide retirement"
assert_file_contains "$workflow_file" "Prepare GitHub Models fallback credentials" "strix workflow provisions GitHub Models fallback credentials for direct-OpenAI scans"
assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_KEY_FILE" "strix gate reads the optional GitHub Models fallback key file"
assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_API_BASE_FILE" "strix gate routes github_models fallback models through the GitHub Models endpoint"
Expand Down
3 changes: 1 addition & 2 deletions tests/test_strix_nvidia_nim_not_found_fallback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,7 @@ def test_workflow_uses_available_free_first_nvidia_plan(self) -> None:
)
self.assertIn(
"steps.gate.outputs.provider_mode == 'nvidia_nim' && "
f"'{FREE_NVIDIA_FALLBACK} github_models/openai/o3 "
"github_models/openai/gpt-5-chat'",
f"'{FREE_NVIDIA_FALLBACK} openai-direct/gpt-5.6-luna'",
workflow,
)

Expand Down
Loading