Skip to content

FEAT: Convert pyrit_scan mode-flags to subcommands + stale docs corrections - #2401

Open
Justin Song (jsong468) wants to merge 3 commits into
microsoft:mainfrom
jsong468:cli_subcommands
Open

FEAT: Convert pyrit_scan mode-flags to subcommands + stale docs corrections#2401
Justin Song (jsong468) wants to merge 3 commits into
microsoft:mainfrom
jsong468:cli_subcommands

Conversation

@jsong468

Copy link
Copy Markdown
Contributor

Description

  • Convert pyrit_scan's mode-flags into argparse subcommands (verbs): run,
    list-scenarios/list-initializers/list-targets/list-converters/list-datasets,
    add-initializer, scenario-results, scenario-history, start-server, stop-server.
    This mirrors pyrit_shell 1:1 and lets argparse enforce which flags belong to which
    command, deleting hand-written validation.
  • run owns the scenario positional, run flags, and the dynamically-injected
    scenario-declared flags
    (two-pass parse: generic parse, fetch scenario metadata,
    re-parse with the scenario's flags attached to the run sub-parser).
  • Backward compatibility via a single pre-parse translation shim: every legacy flag
    form is rewritten to its verb with a deprecation warning (removed_in="1.3.0"), keeping
    the old surface working for one release. Consolidating it in one function makes the
    eventual removal a clean single-commit deletion. scenario-results is the one exception
    — it shipped only days ago with no released flag form, so it's verb-only.
  • Keep implicit-run working during the window (pyrit_scan <scenario>
    run <scenario>, with a deprecation warning), since running a scenario is the most
    common invocation and breaking it hard would defeat a gentle migration. Trade-off: a
    mistyped verb is read as a scenario name and surfaces as a server-side "scenario not
    found" — acceptable and close to today's behavior.
  • Shim removal is a separate follow-up PR (when 1.3.0 lands): this PR is the
    structural change; the legacy shim then deletes on its own.
  • Delete now-redundant flat-parser helpers (_is_command_specified,
    _validate_results_flags) — subparsers make them unnecessary.

Key design decisions

  • Globals on a shared parent parser, not the top parser. Global options
    (--server-url, --config-file, --log-level, --request-timeout, --start-server,
    --startup-timeout) are inherited by every subparser via parents=[global_parser],
    which avoids argparse's default-clobbering. Natively they work after the verb; the
    shim reorders a global placed before a verb so both orderings work (and a leading
    global isn't misread as an implicit scenario name).
  • --start-server stays a global modifier and a verb. It also means "auto-start the
    server before running this other command," which is cross-cutting and can't live in one
    verb; the standalone start-server verb covers the explicit "just start it" case.
  • Separate list-* verbs (not a single list <what>) to match the shell's
    do_list_* commands exactly.

Tests

  • Reworked test_pyrit_scan.py around verbs: TestParseArgs and TestMain use the verb
    forms; added TestLegacyArgvShim covering legacy flags, implicit-run, standalone
    --start-server, and global-before-verb reordering.
  • Full CLI unit suite passes.

Documentation

  • Converted the scanner and scenario docs from flag forms to verbs
    (doc/scanner/1_pyrit_scan, doc/scanner/2_pyrit_shell,
    doc/code/scenarios/2_custom_scenario_parameters, and related pages).
  • Corrected inaccurate flag usage: removed the non-existent --initialization-scripts
    flag and --initializers on list-* commands; documented that custom scenarios and
    converters are loaded via the config file's initialization_scripts section (run by the
    backend at startup), and that list-scenarios (not run --help) is the path to discover
    a scenario's declared parameters.
  • Kept the paired .py / .ipynb doc sources in sync.

@jsong468 Justin Song (jsong468) changed the title FEAT: Convert pyrit_scan mode-flags to subcommands FEAT: Convert pyrit_scan mode-flags to subcommands + stale docs corrections Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant