Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/strix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -708,6 +708,29 @@ jobs:
echo "STRIX_OPENAI_FALLBACK_KEY_FILE=$openai_fallback_key_file" >> "$GITHUB_ENV"
fi

- name: Prepare direct-OpenAI fallback credentials
# The nvidia_nim, github_models, and openrouter fallback matrices all
# end with openai-direct/gpt-5.6-luna. Without OpenAI credentials those
# candidates previously ran against the primary provider endpoint with
# the wrong key and failed closed on every provider outage.
if: steps.gate.outputs.provider_mode == 'nvidia_nim' || steps.gate.outputs.provider_mode == 'openrouter' || steps.gate.outputs.provider_mode == 'github_models'
env:
OPENAI_FALLBACK_KEY: ${{ secrets.STRIX_OPENAI_API_KEY || secrets.OPENAI_API_KEY }}
run: |
umask 077
sanitized="$(printf '%s' "$OPENAI_FALLBACK_KEY" | tr -d '\r\n')"
trimmed="$(printf '%s' "$sanitized" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
if [ -z "$trimmed" ]; then
echo '::notice::No direct-OpenAI key available; Strix scans run without the openai-direct terminal fallback.'
exit 0
fi
openai_fallback_key_file="$RUNNER_TEMP/openai_fallback_key.txt"
printf '%s' "$trimmed" > "$openai_fallback_key_file"
echo "STRIX_OPENAI_FALLBACK_KEY_FILE=$openai_fallback_key_file" >> "$GITHUB_ENV"
openai_fallback_api_base_file="$RUNNER_TEMP/openai_fallback_api_base.txt"
printf '%s' 'https://api.openai.com/v1' > "$openai_fallback_api_base_file"
echo "STRIX_OPENAI_FALLBACK_API_BASE_FILE=$openai_fallback_api_base_file" >> "$GITHUB_ENV"
Comment on lines +711 to +732

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Two steps write the same OpenAI fallback key

For nvidia_nim and openrouter modes both the existing GitHub Models fallback step and the new direct-OpenAI step write the same openai_fallback_key.txt and export STRIX_OPENAI_FALLBACK_KEY_FILE. Contents are identical, so this is harmless redundancy; only the new step adds the API-base file that constitutes the actual fix.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


- name: Prepare Vertex AI credentials
if: steps.gate.outputs.provider_mode == 'vertex_ai'
env:
Expand Down Expand Up @@ -837,6 +860,7 @@ jobs:
STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openrouter' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna' || '' }}
STRIX_GITHUB_MODELS_API_BASE_FILE: ${{ env.STRIX_GITHUB_MODELS_API_BASE_FILE }}
STRIX_GITHUB_MODELS_KEY_FILE: ${{ env.STRIX_GITHUB_MODELS_KEY_FILE }}
STRIX_OPENAI_FALLBACK_API_BASE_FILE: ${{ env.STRIX_OPENAI_FALLBACK_API_BASE_FILE }}
STRIX_OPENAI_FALLBACK_KEY_FILE: ${{ env.STRIX_OPENAI_FALLBACK_KEY_FILE }}
STRIX_FAIL_ON_PROVIDER_SIGNAL: "1"
STRIX_VERTEX_FALLBACK_MODELS: ""
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,17 @@ Semantic Versioning where the repository publishes a release.

### Fixed

- Recognize the hyphenated `openai-direct/` fallback alias (pinned verbatim by
protected main's own trusted `strix_required_workflow_smoke.sh`, so the
`STRIX_FALLBACK_MODELS` string itself cannot change) in
`child_model_for_api_base`, alongside the existing underscored
`openai_direct/` form. Previously the hyphenated alias passed through
unrecognized and unrewritten, so a NIM-exhaustion fallback to
`openai-direct/gpt-5.6-luna` reached LiteLLM as a literal, unrecognized
provider string (`litellm.BadRequestError: LLM Provider NOT provided`)
instead of the intended `openai/gpt-5.6-luna`, observed after NVIDIA NIM
rate-limited both the primary and first fallback model in three
consecutive Strix runs.
- Publish only the sanitized cumulative Strix report tree, avoiding a later
copy of relative scanner output that could reintroduce known internal warning
text into uploaded security evidence.
Expand Down
63 changes: 43 additions & 20 deletions scripts/ci/strix_quick_gate.sh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Call to deleted function prints error each scan

is_explicit_openai_model is still invoked at line 2571, but this PR deleted its definition, so every model attempt runs an undefined command and bash writes is_explicit_openai_model: command not found into the gate log. The branch it guarded never runs; the new is_openai_direct_model block above already sets the same key, so credentials are unchanged.

(Refers to this code)

Prompt for agents
The function is_explicit_openai_model() was removed in this PR (its former definition around old lines 406-415 was replaced by the STRIX_OPENAI_FALLBACK_API_BASE_FILE validation block), but a call to it remains at line 2571 in run_strix_once. This dangling reference makes bash emit 'is_explicit_openai_model: command not found' to stderr (captured into the gate console log) on every model attempt. The block is now redundant because the new is_openai_direct_model check at line 2561 already sets child_llm_api_key to STRIX_OPENAI_FALLBACK_KEY for the exact same set of models (openai-direct/* and openai_direct/*). Remove the entire stale if block (lines 2571-2576) so no removed function is referenced.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Dead openai-direct branch left behind

The merged case openai_direct/* | openai-direct/* at line 2503 now handles both spellings, making the later standalone openai-direct/* case unreachable dead code. Harmless but worth removing with its stale comment.

(Refers to this code)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -380,12 +380,12 @@ if [ -n "$STRIX_GITHUB_MODELS_KEY_FILE" ]; then
fi
fi

# Optional cross-provider fallback credentials for direct-OpenAI fallback
# models (openai-direct/... or openai_direct/...). When the primary model runs
# against NVIDIA NIM, OpenRouter, or GitHub Models, its LLM_API_KEY cannot
# authenticate a direct-OpenAI fallback; this file carries the OpenAI key.
# Optional: without it, explicit direct-OpenAI models keep using LLM_API_KEY,
# which is correct whenever the primary already runs against direct OpenAI.
# Optional cross-provider fallback credentials for openai-direct/ (and the
# underscore alias openai_direct/) fallback models. When the primary provider
# is NVIDIA NIM, GitHub Models, or OpenRouter, those fallbacks must
# authenticate against the direct OpenAI API with an OpenAI key instead of
# reusing LLM_API_KEY_FILE / LLM_API_BASE_FILE. Both files are optional;
# without them openai-direct fallbacks keep requiring LLM_API_BASE_FILE.
STRIX_OPENAI_FALLBACK_KEY_FILE="${STRIX_OPENAI_FALLBACK_KEY_FILE:-}"
if [ -n "$STRIX_OPENAI_FALLBACK_KEY_FILE" ] && { [ ! -f "$STRIX_OPENAI_FALLBACK_KEY_FILE" ] || [ -L "$STRIX_OPENAI_FALLBACK_KEY_FILE" ]; }; then
echo "ERROR: STRIX_OPENAI_FALLBACK_KEY_FILE must reference a regular file containing the API key." >&2
Expand All @@ -403,16 +403,14 @@ if [ -n "$STRIX_OPENAI_FALLBACK_KEY_FILE" ]; then
fi
fi

is_explicit_openai_model() {
case "$1" in
openai_direct/* | openai-direct/*)
return 0
;;
*)
return 1
;;
esac
}
STRIX_OPENAI_FALLBACK_API_BASE_FILE="${STRIX_OPENAI_FALLBACK_API_BASE_FILE:-}"
if [ -n "$STRIX_OPENAI_FALLBACK_API_BASE_FILE" ] && { [ ! -f "$STRIX_OPENAI_FALLBACK_API_BASE_FILE" ] || [ -L "$STRIX_OPENAI_FALLBACK_API_BASE_FILE" ]; }; then
echo "ERROR: STRIX_OPENAI_FALLBACK_API_BASE_FILE must reference a regular file containing the API base URL." >&2
exit 2
fi
if [ -n "$STRIX_OPENAI_FALLBACK_API_BASE_FILE" ] && ! STRIX_OPENAI_FALLBACK_API_BASE_FILE="$(resolve_trusted_input_file "STRIX_OPENAI_FALLBACK_API_BASE_FILE" "$STRIX_OPENAI_FALLBACK_API_BASE_FILE")"; then
exit 2
fi

require_non_negative_integer() {
local value="$1"
Expand Down Expand Up @@ -789,6 +787,20 @@ is_github_models_model() {
esac
}

## True when the model routes to the direct OpenAI API through one of the two
## accepted spellings. The workflow's fallback matrices use the hyphen alias
## while some self-tests and callers pass the underscore form.
is_openai_direct_model() {
case "$1" in
openai-direct/* | openai_direct/*)
return 0
;;
*)
return 1
;;
esac
}

is_github_models_api_compatible_model() {
case "$1" in
openai/openai/* | github_models/* | \
Expand Down Expand Up @@ -2417,7 +2429,13 @@ resolved_llm_api_base_for_model() {

local api_base_file="$LLM_API_BASE_FILE"
local api_base_file_name="LLM_API_BASE_FILE"
if is_github_models_model "$model" && [ -n "${STRIX_GITHUB_MODELS_API_BASE_FILE:-}" ]; then
if is_openai_direct_model "$model" && [ -n "${STRIX_OPENAI_FALLBACK_API_BASE_FILE:-}" ]; then
# Cross-provider fallback: openai-direct/* (and openai_direct/*)
# candidates must reach the direct OpenAI API even when the primary
# provider selected a different LLM_API_BASE_FILE endpoint.
api_base_file="$STRIX_OPENAI_FALLBACK_API_BASE_FILE"
api_base_file_name="STRIX_OPENAI_FALLBACK_API_BASE_FILE"
elif is_github_models_model "$model" && [ -n "${STRIX_GITHUB_MODELS_API_BASE_FILE:-}" ]; then
# Cross-provider fallback: when the active primary provider uses a
# different API base (for example OpenRouter), github_models/* fallback
# attempts must still route through the GitHub Models inference endpoint.
Expand Down Expand Up @@ -2482,8 +2500,8 @@ child_model_for_api_base() {
fi

case "$model" in
openai_direct/*)
printf 'openai/%s\n' "${model#openai_direct/}"
openai_direct/* | openai-direct/*)
printf 'openai/%s\n' "${model#*/}"
Comment thread
seonghobae marked this conversation as resolved.
return 0
;;
# The workflow contract spells the direct-OpenAI fallback with a hyphen
Expand Down Expand Up @@ -2540,7 +2558,12 @@ run_strix_once() {
local child_llm_api_key=""
if ! is_vertex_model "$(normalize_model "$model")"; then
child_llm_api_key="$LLM_API_KEY"
if is_github_models_model "$(normalize_model "$model")" && [ -n "$STRIX_GITHUB_MODELS_KEY" ]; then
if is_openai_direct_model "$model" && [ -n "$STRIX_OPENAI_FALLBACK_KEY" ]; then
# Cross-provider fallback: openai-direct/* (and openai_direct/*)
# models authenticate with the direct-OpenAI key, not the primary
# provider key in LLM_API_KEY_FILE.
child_llm_api_key="$STRIX_OPENAI_FALLBACK_KEY"
elif is_github_models_model "$(normalize_model "$model")" && [ -n "$STRIX_GITHUB_MODELS_KEY" ]; then
# Cross-provider fallback: github_models/* models authenticate
# with the GitHub Models token, not the direct-OpenAI key.
child_llm_api_key="$STRIX_GITHUB_MODELS_KEY"
Expand Down
1 change: 1 addition & 0 deletions scripts/ci/test_strix_quick_gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,7 @@ assert_strix_workflow_pr_trigger_hardened() {
assert_file_contains "$workflow_file" "steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna'" "strix workflow gives NVIDIA NIM scans contracted fallbacks"
assert_file_not_contains "$workflow_file" "STRIX_FALLBACK_MODELS: \${{ steps.gate.outputs.provider_mode == 'github_models' && 'github_models/openai/o3" "strix workflow fallback list must not depend on GitHub Models, which is in platform-wide retirement"
assert_file_contains "$workflow_file" "Prepare GitHub Models fallback credentials" "strix workflow provisions GitHub Models fallback credentials for direct-OpenAI scans"
assert_file_contains "$workflow_file" "if: steps.gate.outputs.provider_mode == 'nvidia_nim' || steps.gate.outputs.provider_mode == 'openrouter' || steps.gate.outputs.provider_mode == 'github_models'" "strix workflow provisions direct-OpenAI fallback credentials for GitHub Models scans"
assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_KEY_FILE" "strix gate reads the optional GitHub Models fallback key file"
assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_API_BASE_FILE" "strix gate routes github_models fallback models through the GitHub Models endpoint"
assert_file_not_contains "$workflow_file" 'github_models/deepseek/deepseek-r1-0528 | github_models/deepseek/deepseek-v3-0324)' "strix workflow keeps DeepSeek GitHub Models restricted to fallback-only routing"
Expand Down
57 changes: 57 additions & 0 deletions tests/test_strix_nvidia_nim_not_found_fallback.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,34 @@ def _classifies_as_nvidia_not_found(log_text: str) -> bool:
return completed.returncode == 0


def _child_model_for_api_base(model: str, llm_api_base_value: str) -> str:
"""Execute the production model-alias normalizer against one input pair."""

gate_source = STRIX_GATE.read_text(encoding="utf-8")
function_source = "\n".join(
_function_block(gate_source, name)
for name in (
"is_github_models_api_base",
"is_github_models_model",
"child_model_for_api_base",
)
)
script = "\n".join(
(
"set -euo pipefail",
function_source,
'child_model_for_api_base "$1" "$2"',
)
)
completed = subprocess.run(
["bash", "-c", script, "strix-normalizer", model, llm_api_base_value],
check=True,
capture_output=True,
text=True,
)
return completed.stdout.strip()


def _workflow_signal_pattern(workflow: str, variable_name: str) -> str:
"""Extract one single-quoted POSIX ERE assigned in the Strix workflow."""

Expand Down Expand Up @@ -205,6 +233,12 @@ def test_workflow_uses_available_free_first_nvidia_plan(self) -> None:
f"'{FREE_NVIDIA_FALLBACK} openai-direct/gpt-5.6-luna'",
workflow,
)
self.assertIn(
"steps.gate.outputs.provider_mode == 'nvidia_nim' || "
"steps.gate.outputs.provider_mode == 'openrouter' || "
"steps.gate.outputs.provider_mode == 'github_models'",
workflow,
)

default_gate = workflow.split("- name: Gate Strix secrets", maxsplit=1)[1]
default_gate = default_gate.split(
Expand All @@ -213,6 +247,29 @@ def test_workflow_uses_available_free_first_nvidia_plan(self) -> None:
)[0]
self.assertNotIn(RETIRED_PRIMARY_MODEL, default_gate)

def test_gate_normalizes_hyphenated_openai_direct_fallback_alias(self) -> None:
"""Route the NIM-exhaustion fallback alias to a real LiteLLM provider.

`STRIX_FALLBACK_MODELS`' NVIDIA NIM entry ends in the hyphenated
`openai-direct/gpt-5.6-luna` alias (the workflow's user-facing input
spelling, also pinned verbatim by protected main's own trusted
`strix_required_workflow_smoke.sh`, so this exact string cannot
change). The gate must still resolve it to LiteLLM's `openai/`
provider -- the same target the underscored `openai_direct/` alias
already reaches -- or NVIDIA NIM rate-limiting the primary and first
fallback model leaves the run one hop from
`litellm.BadRequestError: LLM Provider NOT provided`.
"""

self.assertEqual(
_child_model_for_api_base("openai-direct/gpt-5.6-luna", ""),
"openai/gpt-5.6-luna",
)
self.assertEqual(
_child_model_for_api_base("openai_direct/gpt-5.6-luna", ""),
"openai/gpt-5.6-luna",
)

def test_outer_workflow_requires_litellm_context_for_nvidia_404(self) -> None:
"""Reject provider-like target text in the outer neutralization gate."""

Expand Down
Loading