TB empty-gold fairness: pure_refusal-only gate under full catalog

[Dominic Nguyen (datduyng)]

Summary

Empty-gold TB negatives (expectedActions: []) are scored as zero actions across the full loaded tool catalog. Label-fair Q&A/definition negatives were not zero-action-scorer-fair: models correctly fire chat/help/history, tanking negative pass.

This PR ships a pipeline fairness contract so empty gold is only accepted for hard abstains:

• Synthesizer + quality-verifier prompts: empty-gold kind is pure_refusal only (leave-alone / don't / never / cancel of the target; no questions, no alternate task).
• negativeFairness module: path-keyed 1:1 negativeAssessments (no regex path parse); fail-closed on missing/duplicate paths; fairEmptyGold only when kind is in the allowlist (TRANSLATION_BENCH_FAIR_EMPTY_GOLD_KINDS).
• Partial generation: when complete coverage is optional, emitted caseCount / coverage metadata match the cases actually produced (so validation can succeed).
• No gold-row hand edits and no empty-gold scorer loosening.

Measured 3-model eval (honest)

Regenerated draft under the new policy (998/1000 rows; 2 slots exhausted data-quality, unrelated to fairness), then evaluated azure/gpt-5.6-{sol,terra,luna}.

Metric	Result
Rows / empty-gold negs	998 / 998
Kind mix	100% pure_refusal
Unfair rate (LLM audit)	0%
Overall pass	36.8%
Negative pass (zero fire on empty gold)	13.9%
Neg fire-on-empty	47.9%
Positive pass	59.8%
Pos abstention FNR	3.2%
Tool / param	83.5% / ~71.6%
Eval cost	~$190

Finding: fairness regen fixed the dataset (kind mix + unfair rate), but did not produce a large negative-pass lift. Checkpoint inspection shows models often emit the refused action itself on clear hard-abstains (negation-following failure). Translator/prompt work is out of scope here.

Test plan

• pnpm run build in ts/packages/benchmarks
• pnpm run jest-esm --testPathPattern='(negativeFairness|datasetGenerator)\\.spec\\.js' (43/43)
• Smoke + full regen fairness audit (0% unfair, 100% pure_refusal)
• 3-model headless eval on regenerated draft; scale-metrics recorded

[Copilot]

Pull request overview

Adds a fairness contract ensuring empty-gold translation-bench negatives represent only pure refusals.

Changes:

• Adds path-keyed LLM fairness assessments and enforcement.
• Updates generation/verifier prompts and gold-action validation.
• Supports concurrent generation and accurate partial-result metadata.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.AUTOGEN.md	Refreshes generated documentation metadata.
actionValidation.ts	Normalizes gold actions for schema validation.
benchmark.ts	Uses normalized gold-action validation.
dataQualityVerifier.ts	Enforces semantic negative-fairness assessments.
datasetGenerator.ts	Adds concurrency and partial-generation handling.
generationCandidate.ts	Preserves required parameter objects during cleanup.
index.ts	Exports the fairness module.
negativeFairness.ts	Implements fairness parsing and enforcement.
quality-verifier.prompt.yaml	Defines full-catalog fairness review requirements.
synthesizer.prompt.yaml	Restricts empty-gold negatives to pure refusals.
utteranceDisambiguation.ts	Documents separate negative handling.
translationBench.datasetGenerator.spec.ts	Updates generation fixtures for fairness assessments.
translationBench.negativeFairness.spec.ts	Tests fairness validation and semantic enforcement.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.