diff --git a/.github/workflows/strix.yml b/.github/workflows/strix.yml index b3248d943..aa469466e 100644 --- a/.github/workflows/strix.yml +++ b/.github/workflows/strix.yml @@ -696,6 +696,22 @@ jobs: printf '%s' 'https://models.github.ai/inference' > "$github_models_api_base_file" echo "STRIX_GITHUB_MODELS_API_BASE_FILE=$github_models_api_base_file" >> "$GITHUB_ENV" + - name: Prepare direct OpenAI fallback credentials + if: steps.gate.outputs.provider_mode == 'github_models' || steps.gate.outputs.provider_mode == 'openrouter' || steps.gate.outputs.provider_mode == 'nvidia_nim' + env: + OPENAI_FALLBACK_API_KEY: ${{ secrets.STRIX_OPENAI_API_KEY || secrets.OPENAI_API_KEY }} + run: | + umask 077 + sanitized="$(printf '%s' "$OPENAI_FALLBACK_API_KEY" | tr -d '\r\n')" + trimmed="$(printf '%s' "$sanitized" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + if [ -z "$trimmed" ]; then + echo '::notice::No direct OpenAI key available; cross-provider Strix scans run without that fallback.' + exit 0 + fi + openai_fallback_key_file="$RUNNER_TEMP/openai_fallback_key.txt" + printf '%s' "$trimmed" > "$openai_fallback_key_file" + echo "STRIX_OPENAI_FALLBACK_KEY_FILE=$openai_fallback_key_file" >> "$GITHUB_ENV" + - name: Prepare Vertex AI credentials if: steps.gate.outputs.provider_mode == 'vertex_ai' env: @@ -825,6 +841,7 @@ jobs: STRIX_FALLBACK_MODELS: ${{ steps.gate.outputs.provider_mode == 'github_models' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openai_direct' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'openrouter' && 'openai-direct/gpt-5.6-luna' || steps.gate.outputs.provider_mode == 'nvidia_nim' && 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 openai-direct/gpt-5.6-luna' || '' }} STRIX_GITHUB_MODELS_API_BASE_FILE: ${{ env.STRIX_GITHUB_MODELS_API_BASE_FILE }} STRIX_GITHUB_MODELS_KEY_FILE: ${{ env.STRIX_GITHUB_MODELS_KEY_FILE }} + STRIX_OPENAI_FALLBACK_KEY_FILE: ${{ env.STRIX_OPENAI_FALLBACK_KEY_FILE }} STRIX_FAIL_ON_PROVIDER_SIGNAL: "1" STRIX_VERTEX_FALLBACK_MODELS: "" NPM_CONFIG_IGNORE_SCRIPTS: "true" diff --git a/CHANGELOG.md b/CHANGELOG.md index 93104bcb1..6c6097f66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,16 @@ Semantic Versioning where the repository publishes a release. ### Changed +- Route the exact single-line LiteLLM/Azure unsupported-temperature failure to + an already-configured distinct Strix fallback without accepting split-line + imitations, and normalize the workflow's `openai-direct/` fallback alias to + LiteLLM's `openai_direct/` provider prefix before dispatch. Cross-provider + attempts now switch to the trusted OpenAI credential and clear the primary + provider API base, without turning an incomplete provider scan into passing + evidence. Exact clean model-quality and Hugging Face advisories are filtered + consistently from console and report logs, while any appended warning text + remains fail closed. The incident and fail-closed boundary are recorded in + `docs/doctoring/strix-unsupported-sampling-fallback.md`. - Emit completed repository pull-list requests as they finish in the five-minute agent-mention sweep, while retaining the four-worker ceiling, rotation, and exact-name dispatch ledger, so one slow repository cannot hide ready sibling diff --git a/docs/doctoring/strix-nvidia-nim-not-found-fallback.md b/docs/doctoring/strix-nvidia-nim-not-found-fallback.md index a088aa7ef..89d9176ed 100644 --- a/docs/doctoring/strix-nvidia-nim-not-found-fallback.md +++ b/docs/doctoring/strix-nvidia-nim-not-found-fallback.md @@ -30,6 +30,12 @@ combining with an unrelated application `404` to spoof infrastructure fallback. Provider-side failure also remains a fail-closed incomplete scan until a distinct fallback produces complete evidence. +A completed zero-vulnerability fallback may emit Strix's model-quality banner +and Hugging Face's exact unauthenticated-download advisory while loading public +scanner assets. The trusted gate removes only those two complete lines from both +console and structured report logs. A suffix, prefix, or any other warning text +remains fail-closed provider-failure evidence. + Exhausted provider infrastructure remains fail-closed even when the trusted gate has classified every observed threshold finding as outside the pull request's changed files. That classification scopes authoritative findings; it @@ -53,7 +59,11 @@ Regression evidence proves that: 8. provider exhaustion remains non-passing after unchanged baseline findings; 9. changed, unmapped, and changed-manifest findings also block after provider exhaustion; and -10. the required-workflow smoke contract pins these properties. +10. the exact model-quality and Hugging Face download advisories do not reject + an otherwise complete zero-vulnerability fallback when they appear in + console or report logs; +11. appended warning text is not hidden by either exact advisory filter; and +12. the required-workflow smoke contract pins these properties. ## Limitations diff --git a/docs/doctoring/strix-unsupported-sampling-fallback.md b/docs/doctoring/strix-unsupported-sampling-fallback.md new file mode 100644 index 000000000..e0a75965d --- /dev/null +++ b/docs/doctoring/strix-unsupported-sampling-fallback.md @@ -0,0 +1,67 @@ +# Strix unsupported sampling-parameter fallback + +## Observed failure + +An Azure `gpt-5.6-sol` Strix run failed before vulnerability analysis because +LiteLLM sent `temperature=0.2`. Azure accepts only the model default of `1`, +and LiteLLM had no fallback group for the selected model. Microsoft documents +`temperature` as unsupported for GPT-5 reasoning models (Microsoft, 2026), +while the pinned Strix configuration surface exposes no generation-parameter +control (usestrix, n.d.). + +## Root cause and repair + +The preferred request-boundary repair is to omit a sampling parameter that a +caller did not explicitly provide. `contextual-orchestrator` owns that provider +boundary for organization software. The pinned Strix integration cannot yet +express the omission through its documented configuration, so the trusted +quick gate recognizes only one physical error line containing all of these +signals: + +- a LiteLLM `BadRequestError`; +- Azure or OpenAI exception context; +- the unsupported `temperature` value and supported default; and +- the missing LiteLLM fallback model group. + +That exact capability failure is infrastructure evidence and may move directly +to an already-configured distinct outer fallback. It is not eligible for a +same-model retry. The shared model normalizer also translates the workflow's +human-readable `openai-direct/` selector into LiteLLM's `openai_direct/` +provider prefix before dispatch. A cross-provider direct OpenAI fallback reads +the established OpenAI secret from a trusted runtime file and clears the +primary provider's API base; otherwise a NVIDIA or OpenRouter run would send +the fallback to the wrong endpoint with the wrong credential. If the fallback +credential is unavailable, the attempted fallback fails configuration closed. +If no distinct fallback exists or every fallback fails, the required Strix +check remains non-passing. Existing changed, unmapped, manifest, +`ModelBehaviorError`, and vulnerability-report boundaries remain fail closed. + +Cross-line signal assembly is deliberately rejected so unrelated target output +cannot manufacture a provider capability error from separate log lines. + +## Verification + +- The reproduced single-line Azure failure reaches the configured GitHub + Models fallback exactly once and succeeds only when that scan completes. +- The configured `openai-direct/gpt-5.6-luna` fallback normalizes to the + LiteLLM-compatible `openai_direct/gpt-5.6-luna` selector. +- A NVIDIA-primary run dispatches that fallback with the OpenAI credential and + no inherited NVIDIA API base. +- A split-line imitation is non-recoverable and never dispatches the fallback. +- The full Python suite, native workflow validation, Bash syntax checks, and + complete Strix shell regression suite run on the final tree. + +## References + +AkikoOrenji. (2026, June 4). *[Feature] Expose LLM generation parameters to +control local/OpenAI-compatible model behaviour and prevent runaway tool-call +loops* (Issue No. 514) [GitHub issue]. GitHub. +https://github.com/usestrix/strix/issues/514 + +Microsoft. (2026, August 20). *Azure OpenAI reasoning models—GPT-5 series, +o3-mini, o1, o1-mini*. Microsoft Learn. +https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning + +usestrix. (n.d.). *Configuration* [Computer software documentation]. GitHub. +Retrieved August 23, 2026, from +https://github.com/usestrix/strix/blob/main/docs/advanced/configuration.mdx diff --git a/scripts/ci/strix_model_utils.sh b/scripts/ci/strix_model_utils.sh index 9f20eae67..8f255696b 100755 --- a/scripts/ci/strix_model_utils.sh +++ b/scripts/ci/strix_model_utils.sh @@ -105,6 +105,10 @@ normalize_model() { printf '%s\n' "$model" return 0 ;; + openai-direct/?*) + printf 'openai_direct/%s\n' "${model#openai-direct/}" + return 0 + ;; */*) printf '%s\n' "$model" return 0 diff --git a/scripts/ci/strix_quick_gate.sh b/scripts/ci/strix_quick_gate.sh index 337373001..14acd60d5 100755 --- a/scripts/ci/strix_quick_gate.sh +++ b/scripts/ci/strix_quick_gate.sh @@ -172,6 +172,16 @@ known_internal_warning = re.compile( r"|ended a turn without a lifecycle tool call \(interactive=False\)" r"); forcing tool continuation \(\d+/\d+\): " ) +known_clean_advisory = re.compile( + r"^(?:[ \t│]*MODEL QUALITY WARNING[ \t│]*" + r"|Warning: You are sending unauthenticated requests to the HF Hub\. " + r"Please set a HF_TOKEN to enable higher rate limits and faster downloads\.)$" +) +known_optional_web_search_advisory = re.compile( + r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ WARNING " + r"[^ ]+ - strix\.tools\.web_search\.tool: " + r"web_search invoked without PERPLEXITY_API_KEY configured$" +) def iter_report_logs(root: Path): @@ -194,7 +204,13 @@ for log_path in iter_report_logs(root): lines = log_path.read_text(encoding="utf-8").splitlines(keepends=True) except UnicodeDecodeError: continue - filtered = [line for line in lines if not known_internal_warning.match(line)] + filtered = [ + line + for line in lines + if not known_internal_warning.match(line) + and not known_clean_advisory.fullmatch(line.rstrip("\r\n")) + and not known_optional_web_search_advisory.fullmatch(line.rstrip("\r\n")) + ] if filtered != lines: log_path.write_text("".join(filtered), encoding="utf-8") PY @@ -380,6 +396,23 @@ if [ -n "$STRIX_GITHUB_MODELS_KEY_FILE" ]; then fi fi +STRIX_OPENAI_FALLBACK_KEY_FILE="${STRIX_OPENAI_FALLBACK_KEY_FILE:-}" +if [ -n "$STRIX_OPENAI_FALLBACK_KEY_FILE" ] && { [ ! -f "$STRIX_OPENAI_FALLBACK_KEY_FILE" ] || [ -L "$STRIX_OPENAI_FALLBACK_KEY_FILE" ]; }; then + echo "ERROR: STRIX_OPENAI_FALLBACK_KEY_FILE must reference a regular file containing the API key." >&2 + exit 2 +fi +if [ -n "$STRIX_OPENAI_FALLBACK_KEY_FILE" ] && ! STRIX_OPENAI_FALLBACK_KEY_FILE="$(resolve_trusted_input_file "STRIX_OPENAI_FALLBACK_KEY_FILE" "$STRIX_OPENAI_FALLBACK_KEY_FILE")"; then + exit 2 +fi +STRIX_OPENAI_FALLBACK_KEY="" +if [ -n "$STRIX_OPENAI_FALLBACK_KEY_FILE" ]; then + STRIX_OPENAI_FALLBACK_KEY="$(trim_whitespace "$(cat -- "$STRIX_OPENAI_FALLBACK_KEY_FILE")")" + if [ -z "$STRIX_OPENAI_FALLBACK_KEY" ]; then + echo "ERROR: STRIX_OPENAI_FALLBACK_KEY_FILE must contain a non-empty API key." >&2 + exit 2 + fi +fi + require_non_negative_integer() { local value="$1" local label="$2" @@ -2380,6 +2413,12 @@ resolved_llm_api_base_for_model() { if is_vertex_model "$model"; then return 0 fi + case "$(normalize_model "$model"):$PRIMARY_MODEL" in + openai_direct/*:openai_direct/*) ;; + openai_direct/*:*) + return 0 + ;; + esac local api_base_file="$LLM_API_BASE_FILE" local api_base_file_name="LLM_API_BASE_FILE" @@ -2490,20 +2529,32 @@ run_strix_once() { if ! llm_api_base_value="$(resolved_llm_api_base_for_model "$model")"; then return 2 fi - child_model="$(child_model_for_api_base "$model" "$llm_api_base_value")" + local normalized_model + normalized_model="$(normalize_model "$model")" + child_model="$(child_model_for_api_base "$normalized_model" "$llm_api_base_value")" if ! resolved_target_path="$(resolve_current_target_path "$TARGET_PATH")"; then return 1 fi local start_epoch start_epoch="$(date +%s)" local child_llm_api_key="" - if ! is_vertex_model "$(normalize_model "$model")"; then + if ! is_vertex_model "$normalized_model"; then child_llm_api_key="$LLM_API_KEY" - if is_github_models_model "$(normalize_model "$model")" && [ -n "$STRIX_GITHUB_MODELS_KEY" ]; then + if is_github_models_model "$normalized_model" && [ -n "$STRIX_GITHUB_MODELS_KEY" ]; then # Cross-provider fallback: github_models/* models authenticate # with the GitHub Models token, not the direct-OpenAI key. child_llm_api_key="$STRIX_GITHUB_MODELS_KEY" fi + case "$normalized_model:$PRIMARY_MODEL" in + openai_direct/*:openai_direct/*) ;; + openai_direct/*:*) + if [ -z "$STRIX_OPENAI_FALLBACK_KEY" ]; then + echo "ERROR: direct OpenAI fallback requires STRIX_OPENAI_FALLBACK_KEY_FILE." >&2 + return 2 + fi + child_llm_api_key="$STRIX_OPENAI_FALLBACK_KEY" + ;; + esac fi set -o pipefail set +e @@ -2821,6 +2872,19 @@ is_nvidia_nim_not_found_error() { return 1 } +is_unsupported_model_parameter_error() { + # Strix currently has no generation-parameter override. Match the exact + # single-line LiteLLM/Azure capability failure so a reasoning model that + # rejects Strix's temperature can move to the already-configured fallback. + if grep -Ei 'litellm(\.exceptions)?\.BadRequestError' "$STRIX_LOG" | + grep -Ei '(AzureException|OpenAIException)' | + grep -Eiq "Unsupported value:[[:space:]]*['\"]temperature['\"].*Only the default[[:space:]]*\\(1\\)[[:space:]]*value is supported.*No fallback model group found"; then + return 0 + fi + + return 1 +} + is_model_behavior_error() { # Classify only a module-qualified Strix/Agents SDK protocol exception. # A bare source-file mention of ModelBehaviorError is not retryable. @@ -3151,7 +3215,8 @@ is_llm_token_limit_error() { # was interrupted or incomplete. Used as a guard to prevent the # below-threshold override from silently passing an aborted scan. has_detected_infrastructure_error() { - if grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)' "$STRIX_LOG"; then + if grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)' \ + < <(grep -Eiv '^[[:space:]│]*MODEL QUALITY WARNING[[:space:]│]*$|^Warning: You are sending unauthenticated requests to the HF Hub\. Please set a HF_TOKEN to enable higher rate limits and faster downloads\.$' "$STRIX_LOG"); then return 0 fi @@ -3183,6 +3248,10 @@ has_detected_infrastructure_error() { return 0 fi + if is_unsupported_model_parameter_error; then + return 0 + fi + if is_model_behavior_error; then return 0 fi @@ -4076,6 +4145,10 @@ is_model_retryable_error() { return 0 fi + if is_unsupported_model_parameter_error; then + return 0 + fi + # A provider failure can be recorded only in Strix's structured report log. # run_strix_once already marks that evidence as infrastructure failure, but # the child stdout log used by the classifiers may not contain the provider diff --git a/scripts/ci/test_strix_quick_gate.sh b/scripts/ci/test_strix_quick_gate.sh index bf0a8693e..be20b971c 100755 --- a/scripts/ci/test_strix_quick_gate.sh +++ b/scripts/ci/test_strix_quick_gate.sh @@ -367,6 +367,10 @@ assert_strix_workflow_pr_trigger_hardened() { assert_file_contains "$workflow_file" "Prepare GitHub Models fallback credentials" "strix workflow provisions GitHub Models fallback credentials for direct-OpenAI scans" assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_KEY_FILE" "strix gate reads the optional GitHub Models fallback key file" assert_file_contains "$GATE_SCRIPT" "STRIX_GITHUB_MODELS_API_BASE_FILE" "strix gate routes github_models fallback models through the GitHub Models endpoint" + assert_file_contains "$workflow_file" "Prepare direct OpenAI fallback credentials" "strix workflow provisions direct OpenAI credentials for cross-provider fallbacks" + assert_file_contains "$workflow_file" 'OPENAI_FALLBACK_API_KEY: ${{ secrets.STRIX_OPENAI_API_KEY || secrets.OPENAI_API_KEY }}' "strix workflow reads the established direct OpenAI secret only at the credential boundary" + assert_file_contains "$workflow_file" "STRIX_OPENAI_FALLBACK_KEY_FILE" "strix workflow passes the direct OpenAI fallback key through a trusted file" + assert_file_contains "$GATE_SCRIPT" "STRIX_OPENAI_FALLBACK_KEY_FILE" "strix gate reads the direct OpenAI fallback key from a trusted file" assert_file_not_contains "$workflow_file" 'github_models/deepseek/deepseek-r1-0528 | github_models/deepseek/deepseek-v3-0324)' "strix workflow keeps DeepSeek GitHub Models restricted to fallback-only routing" assert_file_contains "$workflow_file" '${strix_model#github_models/}' "strix workflow strips manual github_models routing prefix for OpenAI GPT model names before passing model names to LiteLLM" assert_file_contains "$workflow_file" "openai_direct/%s" "strix workflow keeps manual direct OpenAI scans distinct from GitHub Models openai/gpt-* routing" @@ -3416,6 +3420,55 @@ REPORT ;; esac ;; + openai-direct-unsupported-temperature-github-models-fallback-success) + case "${STRIX_LLM:-}" in + openai/gpt-5.6-sol) + echo "litellm.BadRequestError: AzureException BadRequestError - Unsupported value: 'temperature' does not support 0.2 with this model. Only the default (1) value is supported. No fallback model group found for original model_group=gpt-5.6-sol." + exit 1 + ;; + openai/o3) + echo "scan ok with GitHub Models fallback" + exit 0 + ;; + *) + echo "unexpected model ${STRIX_LLM:-}" >&2 + exit 9 + ;; + esac + ;; + openai-direct-unsupported-temperature-split-lines-nonrecoverable) + echo "litellm.BadRequestError: request rejected" + echo "AzureException - Unsupported value: 'temperature' does not support 0.2 with this model. Only the default (1) value is supported. No fallback model group found." + exit 1 + ;; + nvidia-openai-direct-fallback-credential-success | nvidia-openai-direct-fallback-missing-key-fails-closed) + case "${STRIX_LLM:-}" in + nvidia_nim/nvidia/primary) + if [ "${LLM_API_KEY:-}" != "dummy" ]; then + echo "unexpected NVIDIA primary key (${LLM_API_KEY:-})" >&2 + exit 17 + fi + echo "Penetration test failed: LLM request failed: RateLimitError" + exit 1 + ;; + openai/gpt-5.6-luna) + if [ "${LLM_API_KEY:-}" != "openai-fallback-token" ]; then + echo "unexpected direct OpenAI fallback key (${LLM_API_KEY:-})" >&2 + exit 18 + fi + if [ -n "${LLM_API_BASE:-}" ]; then + echo "direct OpenAI fallback inherited primary API base: $LLM_API_BASE" >&2 + exit 19 + fi + echo "scan ok with direct OpenAI fallback" + exit 0 + ;; + *) + echo "unexpected model ${STRIX_LLM:-}" >&2 + exit 9 + ;; + esac + ;; vertex-all-notfound) echo "Error: litellm.NotFoundError: Vertex_aiException - x" echo '"status": "NOT_FOUND"' @@ -3580,6 +3633,34 @@ REPORT ;; esac ;; + nvidia-ratelimit-model-quality-warning-fallback-success) + case "${STRIX_LLM:-}" in + nvidia_nim/nvidia/nemotron-3-super-120b-a12b) + echo "litellm.RateLimitError: Nvidia_nimException - Error code: 429 - Too Many Requests" + exit 1 + ;; + nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5) + echo "│ MODEL QUALITY WARNING │" + echo "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads." + mkdir -p "$STRIX_REPORTS_DIR/run-clean-advisories" + { + echo "│ MODEL QUALITY WARNING │" + echo "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads." + } >"$STRIX_REPORTS_DIR/run-clean-advisories/scan.log" + echo "│ Vulnerabilities 0 (No exploitable vulnerabilities detected) │" + exit 0 + ;; + *) + echo "Error: NVIDIA model-quality fallback path unexpected (${STRIX_LLM:-})" >&2 + exit 61 + ;; + esac + ;; + hf-advisory-suffix-fails-closed) + echo "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. Fatal: repository-derived suffix" + echo "│ Vulnerabilities 0 (No exploitable vulnerabilities detected) │" + exit 0 + ;; vertex-primary-resource-exhausted-fallback-success) case "${STRIX_LLM:-}" in vertex_ai/resource-exhausted-primary) @@ -4472,6 +4553,9 @@ EOS exit 0 ;; provider-warning-success-signal) + for _ in {1..10000}; do + echo "benign scanner output" + done echo "Warning: provider response included incomplete scan state" exit 0 ;; @@ -4529,6 +4613,15 @@ EOS echo "scan ok with sanitized internal Strix report notice variant" exit 0 ;; + report-web-search-advisory-sanitized | report-web-search-advisory-suffix-fails) + mkdir -p "$STRIX_REPORTS_DIR/fake-web-search-advisory" + cat >"$STRIX_REPORTS_DIR/fake-web-search-advisory/strix.log" <"$STRIX_REPORTS_DIR/fake-unknown-warning/strix.log" <<'EOS' @@ -5637,12 +5730,17 @@ PY FAKE_STRIX_OUTSIDE_REPORT_DIR="$repo_root_dir/outside-strix-report" ) fi - if [ "$scenario" = "openai-direct-quota-github-models-fallback-success" ]; then + if [ "$scenario" = "openai-direct-quota-github-models-fallback-success" ] || + [ "$scenario" = "openai-direct-unsupported-temperature-github-models-fallback-success" ]; then printf '%s' 'https://models.github.ai/inference' >"$tmp_dir/github_models_api_base.txt" printf '%s' 'github-models-fallback-token' >"$tmp_dir/github_models_key.txt" env_cmd+=(STRIX_GITHUB_MODELS_API_BASE_FILE="$tmp_dir/github_models_api_base.txt") env_cmd+=(STRIX_GITHUB_MODELS_KEY_FILE="$tmp_dir/github_models_key.txt") fi + if [ "$scenario" = "nvidia-openai-direct-fallback-credential-success" ]; then + printf '%s' 'openai-fallback-token' >"$tmp_dir/openai_fallback_key.txt" + env_cmd+=(STRIX_OPENAI_FALLBACK_KEY_FILE="$tmp_dir/openai_fallback_key.txt") + fi if [ "$min_fail_severity" = "__UNSET__" ]; then local next_env_cmd=() local env_pair @@ -5841,6 +5939,17 @@ PY "scenario=$scenario keeps non-warning Strix report evidence" fi + if [ "$scenario" = "report-web-search-advisory-sanitized" ]; then + assert_file_not_contains \ + "$repo_root_dir/strix_runs/fake-web-search-advisory/strix.log" \ + "web_search invoked without PERPLEXITY_API_KEY configured" \ + "scenario=$scenario strips only the known optional web search advisory" + assert_file_contains \ + "$repo_root_dir/strix_runs/fake-web-search-advisory/strix.log" \ + "finish_scan: completed scan with 0 vulnerability report(s)" \ + "scenario=$scenario keeps non-warning Strix report evidence" + fi + if [ "$scenario" = "github-models-primary-ratelimit-fallback-success" ]; then assert_file_contains \ "$output_log" \ @@ -5936,6 +6045,45 @@ run_github_models_http410_case() { "1" } +run_nvidia_openai_direct_fallback_case() { + local scenario="${1:-nvidia-openai-direct-fallback-credential-success}" + local expected_exit="${2:-0}" + local expected_message="${3:-REGEX:Strix quick scan succeeded with fallback model 'openai_direct/gpt-5.6-luna' in [0-9]+s\\.}" + local expected_calls="${4:-2}" + local expected_models="${5:-nvidia_nim/nvidia/primary|openai/gpt-5.6-luna}" + local expected_api_bases="${6:-https://integrate.api.nvidia.com/v1|}" + + run_gate_case "$scenario" \ + "nvidia_nim/nvidia/primary" \ + "" \ + "$expected_exit" \ + "$expected_message" \ + "$expected_calls" \ + "$expected_models" \ + "$expected_api_bases" \ + "nvidia_nim" \ + "https://integrate.api.nvidia.com/v1" \ + "" \ + "0" \ + "CRITICAL" \ + "0" \ + "" \ + "" \ + "1200" \ + "0" \ + "" \ + "" \ + "" \ + "" \ + "0" \ + "" \ + "" \ + "" \ + "" \ + "openai-direct/gpt-5.6-luna" \ + "1" +} + run_filtered_gate_case_if_requested() { case "${STRIX_TEST_CASE_FILTER:-}" in "") @@ -6119,6 +6267,78 @@ run_filtered_gate_case_if_requested() { "" \ "github_models/openai/o3" ;; + openai-direct-unsupported-temperature-github-models-fallback-success) + run_gate_case "openai-direct-unsupported-temperature-github-models-fallback-success" \ + "openai_direct/gpt-5.6-sol" \ + "" \ + "0" \ + "REGEX:Strix quick scan succeeded with fallback model 'github_models/openai/o3' in [0-9]+s\\." \ + "2" \ + "openai/gpt-5.6-sol|openai/o3" \ + "|https://models.github.ai/inference" \ + "vertex_ai" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "github_models/openai/o3" + ;; + openai-direct-unsupported-temperature-split-lines-nonrecoverable) + run_gate_case "openai-direct-unsupported-temperature-split-lines-nonrecoverable" \ + "openai_direct/gpt-5.6-sol" \ + "" \ + "1" \ + "Strix quick scan failed with a non-recoverable error." \ + "1" \ + "openai/gpt-5.6-sol" \ + "" \ + "vertex_ai" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "github_models/openai/o3" + ;; + nvidia-openai-direct-fallback-credential-success) + run_nvidia_openai_direct_fallback_case + ;; + nvidia-openai-direct-fallback-missing-key-fails-closed) + run_nvidia_openai_direct_fallback_case \ + "$STRIX_TEST_CASE_FILTER" \ + "2" \ + "direct OpenAI fallback requires STRIX_OPENAI_FALLBACK_KEY_FILE" \ + "1" \ + "nvidia_nim/nvidia/primary" \ + "https://integrate.api.nvidia.com/v1" + ;; gemini-timeout-fallback-success) run_gate_case_allow_provider_signal "gemini-timeout-fallback-success" \ "gemini/timeout-fallback-primary" \ @@ -6363,6 +6583,47 @@ run_filtered_gate_case_if_requested() { "vertex_ai/report-known-internal-warning-sanitized" \ "" ;; + report-web-search-advisory-sanitized) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/report-web-search-advisory-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/report-web-search-advisory-sanitized'" \ + "1" \ + "vertex_ai/report-web-search-advisory-sanitized" \ + "" + ;; + report-web-search-advisory-suffix-fails) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/report-web-search-advisory-suffix-fails" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-web-search-advisory-suffix-fails" \ + "" + ;; + nvidia-ratelimit-model-quality-warning-fallback-success) + run_gate_case "nvidia-ratelimit-model-quality-warning-fallback-success" \ + "nvidia_nim/nvidia/nemotron-3-super-120b-a12b" \ + "nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \ + "0" \ + "REGEX:Strix quick scan succeeded with fallback model 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5' in [0-9]+s\\." \ + "2" \ + "nvidia_nim/nvidia/nemotron-3-super-120b-a12b|nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \ + "|" \ + "openai" + ;; + hf-advisory-suffix-fails-closed) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/hf-advisory-suffix-fails-closed" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/hf-advisory-suffix-fails-closed" \ + "" + ;; provider-fatal-success-signal | provider-warning-success-signal) run_gate_case "$STRIX_TEST_CASE_FILTER" \ "vertex_ai/$STRIX_TEST_CASE_FILTER" \ @@ -9702,6 +9963,25 @@ run_gate_case_allow_provider_signal "vertex-primary-ratelimit-fallback-success" "vertex_ai/ratelimit-primary|vertex_ai/fallback-one" \ "|" +run_gate_case "nvidia-ratelimit-model-quality-warning-fallback-success" \ + "nvidia_nim/nvidia/nemotron-3-super-120b-a12b" \ + "nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \ + "0" \ + "REGEX:Strix quick scan succeeded with fallback model 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5' in [0-9]+s\\." \ + "2" \ + "nvidia_nim/nvidia/nemotron-3-super-120b-a12b|nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \ + "|" \ + "openai" + +run_gate_case "hf-advisory-suffix-fails-closed" \ + "vertex_ai/hf-advisory-suffix-fails-closed" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/hf-advisory-suffix-fails-closed" \ + "" + run_gate_case_allow_provider_signal "vertex-primary-resource-exhausted-fallback-success" \ "vertex_ai/resource-exhausted-primary" \ "vertex_ai/fallback-one vertex_ai/fallback-two" \ @@ -10458,6 +10738,24 @@ run_gate_case "report-known-internal-warning-variant-sanitized" \ "" \ "1" +run_gate_case "report-web-search-advisory-sanitized" \ + "vertex_ai/report-web-search-advisory-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/report-web-search-advisory-sanitized'" \ + "1" \ + "vertex_ai/report-web-search-advisory-sanitized" \ + "" + +run_gate_case "report-web-search-advisory-suffix-fails" \ + "vertex_ai/report-web-search-advisory-suffix-fails" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-web-search-advisory-suffix-fails" \ + "" + run_gate_case "report-unknown-warning-fails" \ "vertex_ai/report-unknown-warning-fails" \ "" \ @@ -12313,6 +12611,12 @@ assert_normalized_model \ "vertex_ai" \ "vertex_ai/gemini-2.5-pro" +assert_normalized_model \ + "direct-openai-workflow-alias" \ + "openai-direct/gpt-5.6-luna" \ + "openai" \ + "openai_direct/gpt-5.6-luna" + assert_model_requires_vertex_auth "explicit-vertex" "vertex_ai/gemini-2.5-pro" "gemini" "0" assert_model_requires_vertex_auth "explicit-vertex-beta" "vertex_ai_beta/gemini-2.5-pro" "gemini" "0" assert_model_requires_vertex_auth "vertex-resource-path" "projects/my-proj/locations/us-central1/models/gemini-2.5-pro" "vertex_ai" "0" @@ -12509,6 +12813,82 @@ run_gate_case "openai-direct-quota-github-models-fallback-success" \ "" \ "github_models/openai/o3" +# Strix currently reaches LiteLLM with a sampling default and exposes no +# documented generation-parameter override. Azure reasoning deployments reject +# that temperature before Strix's internal model-group fallback can run, so the +# trusted outer gate must try its already-configured distinct provider. +run_gate_case "openai-direct-unsupported-temperature-github-models-fallback-success" \ + "openai_direct/gpt-5.6-sol" \ + "" \ + "0" \ + "REGEX:Strix quick scan succeeded with fallback model 'github_models/openai/o3' in [0-9]+s\\." \ + "2" \ + "openai/gpt-5.6-sol|openai/o3" \ + "|https://models.github.ai/inference" \ + "vertex_ai" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "github_models/openai/o3" + +# Cross-line assembly is deliberately rejected: target/source text cannot +# manufacture a provider capability signal from independent log lines. +run_gate_case "openai-direct-unsupported-temperature-split-lines-nonrecoverable" \ + "openai_direct/gpt-5.6-sol" \ + "" \ + "1" \ + "Strix quick scan failed with a non-recoverable error." \ + "1" \ + "openai/gpt-5.6-sol" \ + "" \ + "vertex_ai" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "" \ + "github_models/openai/o3" + +# Cross-provider fallbacks must switch both the API key and endpoint. Reusing +# NVIDIA credentials or its API base makes a normalized direct-OpenAI model +# fail before producing security evidence. +run_nvidia_openai_direct_fallback_case +run_nvidia_openai_direct_fallback_case \ + "nvidia-openai-direct-fallback-missing-key-fails-closed" \ + "2" \ + "direct OpenAI fallback requires STRIX_OPENAI_FALLBACK_KEY_FILE" \ + "1" \ + "nvidia_nim/nvidia/primary" \ + "https://integrate.api.nvidia.com/v1" + run_gate_case "github-models-fallback-success-deepseek-v3" \ "vertex_ai/missing-primary" \ "github_models/deepseek/deepseek-r1-0528 github_models/deepseek/deepseek-v3-0324" \ diff --git a/tests/test_strix_nvidia_nim_not_found_fallback.py b/tests/test_strix_nvidia_nim_not_found_fallback.py index 990269725..38906f5f6 100644 --- a/tests/test_strix_nvidia_nim_not_found_fallback.py +++ b/tests/test_strix_nvidia_nim_not_found_fallback.py @@ -17,6 +17,7 @@ REPOSITORY_ROOT = Path(__file__).resolve().parents[1] STRIX_GATE = REPOSITORY_ROOT / "scripts" / "ci" / "strix_quick_gate.sh" +STRIX_MODEL_UTILS = REPOSITORY_ROOT / "scripts" / "ci" / "strix_model_utils.sh" STRIX_WORKFLOW = REPOSITORY_ROOT / ".github" / "workflows" / "strix.yml" DEFAULT_NVIDIA_MODEL = "nvidia_nim/nvidia/nemotron-3-super-120b-a12b" FREE_NVIDIA_FALLBACK = ( @@ -73,6 +74,64 @@ def _classifies_as_nvidia_not_found(log_text: str) -> bool: return completed.returncode == 0 +def _run_strix_once_resolves_child_model(model: str, primary_model: str) -> str: + """Execute run_strix_once's own model/API-base resolution lines verbatim. + + Extracts the exact two statements `run_strix_once` uses to compute the + model it hands to LiteLLM, rather than reimplementing that composition, + so a future call-site edit that stops normalizing the workflow-facing + alias before dispatch fails this test instead of only failing in CI + against live NVIDIA NIM traffic. + """ + + gate_source = STRIX_GATE.read_text(encoding="utf-8") + call_site_match = re.search( + r'(?m)^\tif ! llm_api_base_value=.*\n' + r'(?:.*\n)+?' + r'\tchild_model="\$\(child_model_for_api_base [^\n]+\)"\n', + gate_source, + ) + if call_site_match is None: + raise AssertionError( + "missing run_strix_once model/API-base resolution call site" + ) + function_source = "\n".join( + _function_block(gate_source, name) + for name in ( + "is_vertex_model", + "is_github_models_api_base", + "is_github_models_model", + "is_github_models_api_compatible_model", + "resolved_llm_api_base_for_model", + "child_model_for_api_base", + ) + ) + script = "\n".join( + ( + "set -euo pipefail", + STRIX_MODEL_UTILS.read_text(encoding="utf-8"), + function_source, + 'PRIMARY_MODEL="$2"', + 'LLM_API_BASE_FILE=""', + "run_strix_once_child_model() {", + 'local model="$1"', + "local llm_api_base_value", + "local child_model", + call_site_match.group(0), + 'printf %s "$child_model"', + "}", + 'run_strix_once_child_model "$1"', + ) + ) + completed = subprocess.run( + ["bash", "-c", script, "strix-run-once", model, primary_model], + check=True, + capture_output=True, + text=True, + ) + return completed.stdout.strip() + + def _workflow_signal_pattern(workflow: str, variable_name: str) -> str: """Extract one single-quoted POSIX ERE assigned in the Strix workflow.""" @@ -213,6 +272,39 @@ def test_workflow_uses_available_free_first_nvidia_plan(self) -> None: )[0] self.assertNotIn(RETIRED_PRIMARY_MODEL, default_gate) + def test_run_strix_once_normalizes_hyphenated_fallback_before_dispatch( + self, + ) -> None: + """Dispatch the NIM-exhaustion fallback as a real LiteLLM provider. + + `normalize_model()` (`scripts/ci/strix_model_utils.sh`) already + rewrites the workflow-facing `openai-direct/` alias to LiteLLM's + `openai_direct/` provider prefix, and `child_model_for_api_base()` + already rewrites `openai_direct/*` to `openai/*`. But + `run_strix_once()` calls `child_model_for_api_base` with the raw, + un-normalized `$model` -- so a hyphenated `STRIX_FALLBACK_MODELS` + entry still reaches LiteLLM unrewritten. Observed live: NVIDIA NIM + rate-limited the primary and first fallback model, the run advanced + to `openai-direct/gpt-5.6-luna`, and the scan log recorded + `model=openai-direct/gpt-5.6-luna` verbatim before + `litellm.BadRequestError: LLM Provider NOT provided` ended the run. + """ + + self.assertEqual( + _run_strix_once_resolves_child_model( + "openai-direct/gpt-5.6-luna", + DEFAULT_NVIDIA_MODEL, + ), + "openai/gpt-5.6-luna", + ) + self.assertEqual( + _run_strix_once_resolves_child_model( + "openai_direct/gpt-5.6-luna", + DEFAULT_NVIDIA_MODEL, + ), + "openai/gpt-5.6-luna", + ) + def test_outer_workflow_requires_litellm_context_for_nvidia_404(self) -> None: """Reject provider-like target text in the outer neutralization gate."""