Fix: make the process-group teardown check in test_unsafe_local_code_executor deterministic - #181
Open
AmaadMartin wants to merge 4 commits into
Open
Fix: make the process-group teardown check in test_unsafe_local_code_executor deterministic#181AmaadMartin wants to merge 4 commits into
AmaadMartin wants to merge 4 commits into
Conversation
added 4 commits
August 8, 2026 02:01
…stic
The liveness predicate in test_unsafe_local_code_executor.py treated only "Z"
as a post-mortem /proc state. Linux also reports "X" ("x" on older kernels) for
a process that has been reaped but whose /proc entry is still being torn down,
so the predicate called an exited process alive. The teardown wait then made
that reachable: the loop stopped on the reading that showed the fork was dead,
and the assertion threw that reading away and sampled /proc again.
The predicate now knows every post-mortem state and treats a vanished entry as
not alive instead of raising IndexError, and the assertion consumes the reading
the wait settled on. Both deadlines use a monotonic clock. A new table test
covers the predicate for each state.
The wait that the assertion consumes is now a named helper, so the rule it enforces -- the verdict is the reading the wait settled on -- has a deterministic test instead of depending on a real process losing a race. A second test pins the not-alive answer for a pid with no /proc entry.
pytest resolves a dotted target itself, so the test no longer reaches into sys.modules for its own module object, and the sys import goes with it.
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
N/A - no public issue.
Problem:
tests/unittests/code_executors/test_unsafe_local_code_executor.py::TestUnsafeLocalCodeExecutor::test_kill_execution_kills_what_the_code_spawnedfails intermittently. Its_is_alive()helper treated onlyZas a post-mortem/procstate, but Linux also reportsX(xon older kernels) for a process that has been reaped and is being torn down. The wait loop stopped on the reading that showed the fork was dead, and then the assertion read/proca second time; if the fork movedZ->Xin that gap, the assertion called a dead process alive and failed. The same helper raisedIndexErrorinstead of answering, if the entry vanished betweenopen()andread().Solution:
_is_alive()now knows every post-mortem state and treats a vanished entry as not alive. The assertion consumes the reading the wait settled on, through a_has_exited()helper, so no second sample can overturn it. This is sound because a process that has exited does not come back. Both deadlines usetime.monotonic(), matchingtest_container_code_executor.py.The test keeps its full strength. It still spawns a real execution process, still makes the executed code
os.fork()a real grandchild, still calls the real_kill_execution(), and still fails if the grandchild survives. It also now asserts the fork was running before the kill, so the check cannot pass vacuously.src/google/adk/code_executors/unsafe_local_code_executor.pyis unchanged, and no other test in the file is touched.Deviation from the plan I worked from: the plan wrote the corrected wait inline in the test body. Inline, the loop body is only reached when the fork takes more than one poll to die, so it cannot be covered deterministically and the settled-reading rule has no test of its own. I extracted
_has_exited()instead and covered both.Collision check:
gh pr list --repo AmaadMartin/adk-python --state open --limit 100returned 100 open PRs. Eight touch this file (#164, #165, #130, #106, #101, #100, #83, #81). I diffed each for_is_alive,"Z",rsplit,rpartitionand the state set: none changes the predicate, so none lands this fix. #101 is the closest - it moves the same wait into an_await_death()helper, but keeps both defects (Z-only, and a second read after the loop). I did not stack on it: it also rewritesunsafe_local_code_executor.py, and this change must stay test-only. Whichever merges second resolves a small conflict in one function.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
Four new tests, all deterministic and all at module level next to the helpers they cover:
test_only_a_process_that_has_not_exited_counts_as_alive- a table overR,S,D,Z,X,x, a name containing parentheses, and an empty read.test_a_pid_with_no_proc_entry_is_not_alive- theOSErrorpath.test_the_wait_keeps_the_reading_it_settled_on- a scripted probe returns alive, dead, alive; the wait must report the second reading.test_the_wait_gives_up_when_the_process_outlives_the_deadline- a live process and a zero budget.Proof the tests can fail. Each mutation was applied to the source, run, then reverted.
_DEAD_STATES->frozenset("Z")(the bug)Xandxrowsassert True is False where True = _shows_a_live_process('4321 (python3) x 0 -1 -1 0 -1')""rowIndexError: list index out of rangerpartition(")")->partition(")")(py (3)) Zrowassert True is False where True = _shows_a_live_process('4321 (py (3)) Z 1 4321 4321 0 -1')test_the_wait_keeps_the_reading_it_settled_onassert False where False = _has_exited(4321, 10)return exited->return Truetest_the_wait_gives_up_when_the_process_outlives_the_deadlineassert not True where True = _has_exited(1159762, 0)except OSError: return False->return Truetest_a_pid_with_no_proc_entry_is_not_aliveassert not True where True = _is_alive(-1)Coverage of the changed file, measured with
pytest --cov-branch: 96%, up from 95% onmain. Every line and branch this PR adds is covered. The four remaining misses are pre-existing and untouched: thepytest.skip()escape hatch, and three lines of the integration test'sfinallycleanup that only run when the test has already failed.Manual End-to-End (E2E) Tests
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
Everything below ran on the pushed commit, Linux, Python 3.14.4.
Honest limit on the reproduction. I could not make the flake fire on this machine:
mainalso passed 20/20 file runs and 8 rounds ofpytest tests/unittests/code_executors -n auto, and a standalone harness running the pre-fix wait-and-re-read against a real killed grandchild passed 3600 rounds. So the local runs above confirm no regression; they do not by themselves prove the race is gone.What I did measure directly, with a harness that kills a
setsidgroup the same way the executor does and then records every/procstate the grandchild passes through:and the state windows, over another 300 kills:
So
Xis reached routinely, and_is_alive()called it alive (M1 proves that). The rate depends on how far apart the machine puts the two reads: here they are microseconds apart against a ~317 usZwindow, which is why 3600 rounds were not enough; underpytest -n autoon a saturated runner the second read can slip a whole scheduling quantum, which is when it lands inX. The fix removes the second read, so the rate no longer matters.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.
CI
Unit Tests(Python 3.10-3.14),Mypy Check(3.10-3.13) andA2A v0.3 Tests(3.10-3.14) all pass.Pre-commit Linteris red, and it is red on the base commit too. Run31226527302on352d11d3— the commit this branch is based on, with no changes of mine — fails the same job while every other job passes. The hook isupdate-constraints: it recompilesconstraints-*.txtagainst today's date and reports the header dates as a diff. This PR touches no dependency file. Open PRs #163 and #149 address that hook.