Skip to content

fix(eval): match JUnit names ignoring optional eval. prefix - #61

Draft
rootkiller6788 wants to merge 1 commit into
facebookresearch:mainfrom
rootkiller6788:fix/eval-namespace-matching
Draft

fix(eval): match JUnit names ignoring optional eval. prefix#61
rootkiller6788 wants to merge 1 commit into
facebookresearch:mainfrom
rootkiller6788:fix/eval-namespace-matching

Conversation

@rootkiller6788

Copy link
Copy Markdown

Problem

The evaluator compares the expected test list in tests.json against JUnit classnames literally, but the two disagree on the package root. tests.json is inconsistent: some tasks store eval.tests.foo (csview) while others store tests.foo (cmatrix, the bundled calculator fixture). JUnit classnames are always rooted at eval.tests.foo because pytest collects the suite from /workspace/eval.

For tasks whose metadata omits the prefix, _process_branch_xml therefore treats every test as both:

  1. missing from the XML, injecting a synthetic not_run record, and
  2. unexpected in the XML, emitting a "not in tests.json" warning.

A branch that fully passed is counted twice and scores half. The bundled fixture data/test_runs/correct/testorg__calculator.abc1234 reproduces this exactly: three passing eval.tests.* records plus three not_run tests.* records. The same literal comparison in get_branches_to_eval makes a completed branch look incomplete, so programbench eval re-runs it on every non---force pass.

Fix

Compare test names in a canonical form that strips an optional leading eval. prefix from both sides. Added _canonical_test_name and applied it to the expected/observed matching in _process_branch_xml and get_branches_to_eval. Parsed result names are left unchanged, so existing eval.json files and ignored-test filtering are unaffected.

Verification

  • New regression tests in tests/test_eval.py (namespace mismatch matches, missing test still not_run, re-eval idempotent) fail on main and pass with the fix.
  • Full suite: 68 passed.
  • ruff check and ruff format --check clean at 0.15.7.

Refs #56

tests.json stores expected test names inconsistently: some tasks use the
`eval.tests.*` package root (pytest's module path under /workspace/eval)
while others use `tests.*`. JUnit classnames are always rooted at
`eval.tests.*`, so _process_branch_xml's literal comparison counts every
test as both missing (injected not_run) and unexpected for tasks without
the prefix, halving the score of a fully-passing branch. The same literal
comparison in get_branches_to_eval made re-evaluation non-idempotent.

Compare both sides after stripping an optional leading `eval.` prefix.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant