fix(eval): match JUnit names ignoring optional eval. prefix - #61
Draft
rootkiller6788 wants to merge 1 commit into
Draft
fix(eval): match JUnit names ignoring optional eval. prefix#61rootkiller6788 wants to merge 1 commit into
rootkiller6788 wants to merge 1 commit into
Conversation
tests.json stores expected test names inconsistently: some tasks use the `eval.tests.*` package root (pytest's module path under /workspace/eval) while others use `tests.*`. JUnit classnames are always rooted at `eval.tests.*`, so _process_branch_xml's literal comparison counts every test as both missing (injected not_run) and unexpected for tasks without the prefix, halving the score of a fully-passing branch. The same literal comparison in get_branches_to_eval made re-evaluation non-idempotent. Compare both sides after stripping an optional leading `eval.` prefix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The evaluator compares the expected test list in
tests.jsonagainst JUnit classnames literally, but the two disagree on the package root.tests.jsonis inconsistent: some tasks storeeval.tests.foo(csview) while others storetests.foo(cmatrix, the bundled calculator fixture). JUnit classnames are always rooted ateval.tests.foobecause pytest collects the suite from/workspace/eval.For tasks whose metadata omits the prefix,
_process_branch_xmltherefore treats every test as both:not_runrecord, andA branch that fully passed is counted twice and scores half. The bundled fixture
data/test_runs/correct/testorg__calculator.abc1234reproduces this exactly: three passingeval.tests.*records plus threenot_runtests.*records. The same literal comparison inget_branches_to_evalmakes a completed branch look incomplete, soprogrambench evalre-runs it on every non---forcepass.Fix
Compare test names in a canonical form that strips an optional leading
eval.prefix from both sides. Added_canonical_test_nameand applied it to the expected/observed matching in_process_branch_xmlandget_branches_to_eval. Parsed result names are left unchanged, so existingeval.jsonfiles and ignored-test filtering are unaffected.Verification
tests/test_eval.py(namespace mismatch matches, missing test stillnot_run, re-eval idempotent) fail onmainand pass with the fix.ruff checkandruff format --checkclean at 0.15.7.Refs #56