Skip to content

test(ci): add command validator regression coverage - #443

Open
ayobamiseun wants to merge 1 commit into
addyosmani:mainfrom
ayobamiseun:test/validate-commands
Open

test(ci): add command validator regression coverage#443
ayobamiseun wants to merge 1 commit into
addyosmani:mainfrom
ayobamiseun:test/validate-commands

Conversation

@ayobamiseun

Copy link
Copy Markdown
Contributor

Summary

validate-commands.js currently runs against the repository's eight valid command triples in CI. That proves the catalog passes, but not that the validator rejects the invalid states it exists to catch.

This PR adds a fixture-driven node:test suite that runs the real CLI inside temporary repositories and asserts its public exit codes and diagnostics. It then runs that suite in the existing command-validation CI job.

Coverage added

Six regression cases cover the validator's main contracts:

Case Expected behavior
Matching command twins Pass, including the plan to planning stem mapping
Claude command missing a TOML twin Fail with the missing directory named
TOML command missing a Claude twin Fail and count the canonical command correctly
Description drift across tools Fail and print all three descriptions
Present file with no description Fail with an actionable malformed-description diagnostic
Escaped quotes in TOML Parse and compare the normalized description correctly

The passing stem-mapping fixture also uses single-quoted TOML, preserving coverage for that supported syntax.

Implementation

  • Copies validate-commands.js into a fresh temporary repository for each test scenario.
  • Writes only the command files needed for that scenario, then spawns the validator as a CLI.
  • Removes every temporary repository after its test.
  • Adds no dependencies and makes no production changes to the validator.

CI

The existing validate-commands job now runs:

node scripts/validate-commands.js
node --test scripts/validate-commands-test.js

This keeps the real-catalog gate and the validator self-tests together.

Verification

  • node --test scripts/validate-commands-test.js - 6/6 pass
  • node --test scripts/run-evals-test.js scripts/validate-commands-test.js - 18/18 pass
  • node scripts/validate-commands.js - 8 commands, 0 errors
  • node scripts/validate-skills.js - 24 skills, 0 errors, 0 warnings
  • node scripts/run-evals.js - 124 checks passed, 0 errors, 0 warnings; rank-1 86%
  • git diff --check upstream/main...HEAD - clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant