Test: pin the CFC model gate's rejection of unversioned Gemini Early Access ids - #189
Open
AmaadMartin wants to merge 1 commit into
Open
Test: pin the CFC model gate's rejection of unversioned Gemini Early Access ids#189AmaadMartin wants to merge 1 commit into
AmaadMartin wants to merge 1 commit into
Conversation
…ni Early Access ids The gate in Runner._new_invocation_context had no test, so deleting its raise or widening its predicate broke nothing. Add TestRunnerCfcModelGate, which covers all four decision points both ways, and a comment recording why the gate is narrower than BuiltInCodeExecutor's Gemini-id check: the accept path reroutes the invocation to the Live API. The gate's logic is unchanged.
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Link to an existing issue (if applicable):
N/A
Or, if no issue exists, describe the change:
Problem: The CFC model gate in
Runner._new_invocation_contexthas no test. Deleting itsraise, or widening its predicate to admit unversioned Gemini Early Access ids, breaks nothing in CI. Nothing at the call site says why the gate is narrower thanBuiltInCodeExecutor's "is this a Gemini id" check, so readers keep re-deriving the reasoning and reading the difference as an accidental inconsistency.Solution: Unversioned Gemini Early Access ids stay rejected at the CFC gate, in both adk-python and adk-js. The accept path does not just install
BuiltInCodeExecutor: it reroutes the invocation to the Live API (flows/llm_flows/base_llm_flow.py), so the gate asks whether the model can serve CFC there. In adk-js the same accept path throwsCFC is not yet supported in callLlmAsync, so no reading of "admit Early Access" produces working behaviour there. This change addsTestRunnerCfcModelGateand a four-line comment; the gate's logic is byte-identical, and the matching adk-js test is queued as a separate task.Collision check: I listed all 187 open pull requests on the fork and matched their changed files. No open pull request touches
src/google/adk/runners.pyortests/unittests/test_runners.py.Deviation from the plan: the new async test carries an explicit
@pytest.mark.asyncio.asyncio_modeisautotoday, so the marker is not required, but every other async test in this file declares it and separate open work switches pytest-asyncio to strict mode.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
TestRunnerCfcModelGateadds 8 test cases intests/unittests/test_runners.py. Cases 1-6 drive_new_invocation_contextdirectly, matching the existingTestRunnerCacheConfigprecedent in the same file. Case 7 drivesrunner.run_async, which proves the rejection reaches a caller of the public entry point. The accept path reroutes to the Live API, so a unit test does not drive it end to end.The tests use real objects: a real
LlmAgent, a realRunner, a realInMemorySessionService, and oneBaseLlmfake (testing_utils.MockModel) for the non-Gemini case, which a plain string cannot reach becauseLLMRegistrydoes not resolve it.Coverage of the gate block, measured with
coverage run --branch: every line and every branch is executed, with no missing branch pairs.Mutation evidence. I ran the new tests against seven mutations of the gate. Every mutation is caught, and every one of the 8 cases is killed by at least one mutation.
raise ValueError(...)run_asyncearly-expidsrun_asyncagent.code_executor is Nonekeeps_an_already_installed_code_executorrun_config.support_cfcconditionis_inert_when_support_cfc_is_falsehasattr(self.agent, 'canonical_model')guardis_inert_for_an_agent_without_a_canonical_modelaccepts_gemini_2_and_installs_the_code_executorSample failure message from the first mutation:
The tests assert nothing about
gemini-3.0-pro,gemini-1.5-pro, or Vertex publisher paths. The gate's handling of those ids is a separate, real divergence from adk-js and is tracked as its own task; pinning today's answers would cement it.CI: Unit Tests (3.10-3.14), Mypy Check (3.10-3.13) and A2A v0.3 Tests (3.10-3.14) all pass. The Pre-commit Linter fails for a reason unrelated to this change: its
update-constraintshook regeneratesconstraints-*.txtwith today's snapshot date, so the hook fails on every open pull request in this fork. Open pull request #163 covers that hook.Manual End-to-End (E2E) Tests:
No runtime behaviour changes, so there is nothing a user can observe differently. Reviewer-facing verification is the comment at the gate plus the mutation table above.
Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.