Add azd ai eval extension for Foundry evaluations - #9500
Open
Mohamed Hessien (m7md7sien) wants to merge 209 commits into
Open
Add azd ai eval extension for Foundry evaluations#9500Mohamed Hessien (m7md7sien) wants to merge 209 commits into
Mohamed Hessien (m7md7sien) wants to merge 209 commits into
Conversation
New azd extension exposing azd ai eval, registering the azure.ai.eval service-target provider. Scaffold only: manifest, entrypoint, root command.
Lifted eval_api (models, operations, poller, generation, portal_urls) and dataset_api from azure.ai.agents, de-agent-scoped. Added evalcore with IsTransientError and an EvaluatorList that accepts either a bare string or a mapping with a threshold. Skipped artifacts.go and eval_config.go since the config model differs.
… commands - EvalConfig and GenerateConfig types with validation covering cross-references, duplicate names, unsupported target types, and evaluation levels - ResolveGroup picks the only group or errors with the available names - ArtifactPath accepts a directory or an explicit file path for local_dir - evalContext resolves the project endpoint (flag, azd env, host env) and builds both clients against the azd developer CLI credential - dataset create/update/list/show/delete with -o json - Tier-0 tests for parsing, validation, group resolution, and path handling
- buildEvalGroupRequest maps evaluators to testing criteria, keeping the builtin prefix on evaluator_name while stripping it from name, and carries per-evaluator thresholds in initialization_parameters - run resolves the group from --eval-id, a pinned id, or the azd environment, creating it when absent, then binds the dataset to the run since the group has no dataset binding today - Local datasets are sent inline with optional truncation; registered datasets are referenced by id
- evaluator upload/update/list/show/builtins/delete; rubric evaluators only in M1, code evaluators deferred to M2 with the folder walk and RBAC they require - normalizeRubricBody accepts a bare definition or a full document - results show/export with per-criteria pass and fail counts, --failed-only, and JSON or CSV output, replacing the counts-only view - Added ListEvaluators, ListEvaluatorVersions, DeleteEvaluatorVersion, and CancelOpenAIEvalRun to the eval client
init scaffolds both YAML files and the artifact directories without any service call, so it works offline and unauthenticated. Built-ins are referenced from the group but never declared as custom evaluators. A dataset flag containing a path becomes a local source; a bare name references a registered dataset. Tests assert the scaffold loads and validates, and that paths are used verbatim rather than re-rooted.
…tion Registers the azure.ai.eval service target so azd up and azd deploy reach this extension; the extension ships no deploy command of its own. - Reads the eval config from the service entry's inline properties, the same AdditionalProperties channel the agents extension uses - Deploy reconciles datasets, then evaluators, then eval groups, since a group references the versions the first two resolve to; it fails fast and the next deploy resumes - Datasets are change-detected with a local SHA-256 digest kept in the azd environment, because the dataset API returns no content hash and comparing against the service would mean downloading the blob every deploy - Evaluator definitions come back inline, so those are compared directly - Package and Publish are no-ops; eval artifacts are plain files already on disk
- generate submits the rubric and dataset generation jobs, downloads the artifacts locally, and writes source references into the deployment spec - MergeArtifactRefs edits through the yaml Node API so comments, key order, and hand-edited sibling keys survive; matching is by name and merging is idempotent - Raised the client poll budget from 2s x 300 to 5s x 720. The old 10 minute limit gave up while the service was still working, which is the timeout that forced a second command - A supplied --evaluator or a local --dataset is honored and its generation is skipped - Tests cover comment preservation, section creation, idempotence, and fingerprinting
Adds the templated release pipeline for the new extension and lists it as a dependency of the microsoft.foundry meta-package. The registry.json artifact entries are generated by the release, so they are not hand-authored here.
… tests Live testing against a real project found two issues. 1. The dataset model only bound snake_case URIs (data_uri, blob_uri), but the project endpoint returns camelCase (dataUri). ResolvedBlobURI therefore returned empty, which would have failed the generate download much later with no useful error. Both spellings are now accepted. 2. Built-in evaluators do not share one input contract. builtin.ifeval requires instruction_id_list and is rejected under the agent-target data mapping with MissingRequiredDataMapping. The live tests now select an evaluator whose inputs match, and the helper documents why. Live tests are gated behind the live build tag and AZURE_AI_EVAL_E2E_LIVE, and clean up every resource they create. Verified: builtin listing, the full dataset pending-upload lifecycle with version auto-increment, and eval group creation returning 201.
…published contract The builder sent one fixed data mapping and one fixed set of initialization parameters to every evaluator. That only suited agent-target quality evaluators, and the service rejected the rest. The evaluator listing publishes a full contract per evaluator: definition.data_schema (accepted and required inputs), definition.init_parameters, and supported_evaluation_levels. The builder now reads it and shapes each testing criterion accordingly. This fixes four concrete defects. - Required inputs were never honoured, so builtin.ifeval (instruction_id_list), builtin.similarity (ground_truth) and builtin.retrieval (context) all failed with MissingRequiredDataMapping. Fields not supplied by the agent target are now bound to dataset columns, and the item schema declares them. - Inputs an evaluator does not accept were sent anyway. - initialization_parameters always carried model, deployment_name and threshold. No evaluator accepts 'model', and builtin.ifeval accepts nothing at all. Parameters are now filtered to the declared properties, and a required one that is missing is reported locally. - evaluation_level was sent as run metadata, where it has no effect. It is an initialization parameter on the evaluators that declare it. It also encodes an exclusivity rule the service enforces: 'messages' and 'query'/'response' cannot both appear in a mapping, so the evaluation level selects between the conversation and turn shapes. A missing dataset column is now caught before the request is sent and names the column, rather than surfacing as a 400 pointing at testing_criteria[0].data_mapping. An evaluator with no published contract keeps the previous agent-target shape, so custom evaluators are unaffected. Verified against a live project: all ten built-ins are accepted, where two previously failed. The live test exercises the shipping builder rather than a hand-rolled request, so a regression in this logic fails the suite. Adds DeleteOpenAIEval so those tests clean up after themselves.
…invokes Without build.ps1 and build.sh the dev kit reported a successful build in under a second and produced no binary, so azd x pack had nothing to package and azd x publish failed with 'Artifacts not found'. Copied from azure.ai.agents with the version package path retargeted, plus its golangci config. Verified end to end: build, pack, publish, install from the local registry, and 'azd ai eval --help' listing every command.
…ly returns The listing spells it evaluator_type, so the TYPE column in 'evaluator list' and 'evaluator builtins' was always blank. Both spellings are now accepted.
Running a real 'azd deploy' against the service-target provider surfaced two failures that no unit test covered. Evaluator references only decoded from YAML. azd hands the service entry to the extension as JSON, so a group written as '- builtin.task_adherence' -- the form the CLI's own init command writes -- failed with 'cannot unmarshal string into EvaluatorRef'. EvaluatorList now decodes and encodes the mixed string-or-mapping form through JSON as well, and a test asserts the two decoders agree. The dataset reconciler passed the declared version straight to UploadNewVersion, which derives the next version from it. A declaration without an explicit version passed empty, so every deploy retried 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset once that version existed. It now looks up the latest registered version first. Verified against a live project: first deploy publishes the dataset at 1.0 and creates the group; an unchanged redeploy reports 'unchanged at version 1.0' and uploads nothing; and editing the dataset publishes 2.0 and recreates the group, since groups are immutable.
azd core does not resolve $ref for extensions. It strips the ServiceConfig fields it owns and leaves $ref at the top of the map for the owning extension to resolve, so a service authored the way the spec documents it -- host: azure.ai.eval plus $ref: ./evals/azure.yaml -- parsed to an empty config. azd deploy then reported success in three seconds having created nothing, which is worse than failing. The provider now calls foundry.ResolveFileRefs with the project root from the azd project client. Relative source paths inside an included file are written against that file, but ResolveFileRefs inlines content without rebasing them, so the include's own directory is now the base for source resolution. Verified against a live project: the $ref form deploys, and the dataset fingerprint matches the one from the equivalent inline config, confirming both forms resolve to the same file.
…ation changes Change detection only covered upstream artifacts, so retargeting a group at a different agent, swapping an evaluator, or changing the judge model left the old group in place. Groups are immutable, so the edit silently had no effect and later runs kept evaluating the previous definition. The group's declaration is now fingerprinted alongside the dataset and evaluator artifacts. The id and description are excluded: one is server assigned and the other is cosmetic, so neither should force a recreate. The digest is recorded when an existing group is reused as well as when one is created. Recording it only on create meant a group deployed before this change never established a baseline, and the first edit after it would still go undetected. Verified against a live project: changing the target produced a new group id, and two further deploys with no change reused it.
The data-plane clients trace every request and response through log.Printf, which Go writes to stderr by default, so a plain command interleaved raw URLs and status lines with its own output. A long generate run was mostly HTTP traces. Ports the debug setup from the agents extension: the standard logger is discarded unless --debug or AZD_EXT_DEBUG is set, and debug output goes to a dated file rather than the terminal. The hook chains the SDK PersistentPreRunE instead of replacing it. Assigning PersistentPreRun has no effect once the E variant is set, and overwriting the E variant would drop the SDK own setup. Also reports jobs as submitted when generate is given --no-wait, which is a successful submission rather than an empty result.
…fails Data generation with an agent source is accepted and then fails within seconds with DataGenerationJobSystemError, whose message says only that something went wrong and to try again. It is not transient: it reproduces for every agent tried, while the identical request without the agent source runs normally. The CLI now names the agent, says a retry will not help, and points at the two workarounds, instead of relaying advice that cannot succeed.
The spec lists run start, list, show and cancel, and M1 requires every operation to be reachable atomically, but run was a single composite command with no subcommands. Listing runs, inspecting one, and cancelling an in-flight run were unreachable, even though the client already had the calls. Adds run list, run show and run cancel. Each takes the eval group id as an optional argument and otherwise falls back to the id recorded in the azd environment, matching results show. Cancelling a run that already reached a terminal state is refused locally, because the service reports success either way and the CLI would otherwise claim to have cancelled a finished run. Two related fixes. Passing --project-endpoint disabled the azd environment cache entirely: the environment name was only resolved when the endpoint came from azd, so every cached eval group and run id lookup returned empty. The name is now resolved independently of where the endpoint came from. The spec documents --wait and --no-wait, but cobra does not derive the negative form from a bool, so --no-wait was rejected as an unknown flag. Verified live: start with --no-wait, list, show, cancel, and the terminal-state guard on a second cancel. JSON output checked on the new subcommands.
The spec lists run start alongside list, show and cancel. The behaviour existed only as the composite `azd ai eval run`, so the atomic name in the spec did not resolve. Both forms are now built by one constructor, so their flags cannot drift apart, and a test asserts that.
… work Exercising the atomic write commands against a live project found three failures. None were covered by tests, because none of these paths had been run end to end. dataset update always collided. It passed the --version flag straight to UploadNewVersion, which derives the next version from what it is given, so an omitted flag restarted at 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset. The flag help promised the opposite, that omitting it would take the next version. This is the same defect that was fixed in the deploy reconciler earlier, so the discovery is now centralised in DatasetClient.UploadNextVersion and both callers use it, rather than being fixed twice and available to be missed a third time. evaluator upload rejected every hand-authored rubric. The service needs a type discriminator on the definition, and without it fails the whole request with "The request field is required", which names a field that is present. Generated rubrics carry the type, so only the hand-authored path documented in the spec was affected. The type is now filled in when absent and left alone when set. evaluator show returned 404. It omitted the version segment from the path, but the service has no route for an unversioned evaluator, despite the doc comment claiming the latest would be fetched. The latest version is now resolved first, comparing numerically because versions are integers as strings and a lexical compare ranks "9" above "15" -- the service already publishes evaluators at version 15 and 17. Verified live: dataset create, show, update to 2.0, list and delete; evaluator upload, show resolving the latest, update to version 2, list and delete. Both suites leave nothing behind.
…nfig Deploying a config that declares a custom evaluator, rather than only built-in ones, failed in two ways. Every earlier test used built-ins, so neither showed up. The evaluator was republished on every deploy. The service enriches a definition when it stores it, so a rubric consisting of nothing but type and dimensions comes back carrying data_schema, init_parameters and metrics it was never given. Comparing whole documents therefore never matched. Only the keys the author actually wrote are compared now, structurally, so key order and formatting are not changes either. This is what the spec means by repeated azd up creating no redundant versions. The eval group was then rejected with a request for a model that had been set. Evaluators disagree on what the judge model is called: built-ins declare deployment_name, and a custom rubric declares model. The builder sent only deployment_name, so the custom evaluator saw its required parameter missing. The judge model is now bound under whichever name the evaluator declares. Verified live: first deploy publishes the evaluator and creates the group, two redeploys report it unchanged and publish nothing, editing the rubric publishes the next version and recreates the group, and a further redeploy is a no-op again.
…d dataset The flag is documented as taking a path or the name of a registered dataset, and means use this one instead of generating. It only suppressed generation when the value looked like a local path, so passing the name of an existing dataset still submitted a generation job and, since agent-seeded generation is currently broken server-side, failed the whole command. --evaluator already skipped unconditionally, so the two flags disagreed. Both the skip and the default-spec synthesis now key off whether the flag was supplied at all. This was the last thing standing between a generated config and the documented end-to-end flow. Verified live: init scaffolds a group referencing its own rubric, generate writes that rubric and merges the reference into the same file while preserving comments and ordering, azd up registers the dataset and evaluator and creates the group, and the run completes and scores against the generated rubric.
…listing
GET /datasets/{name}/versions returns nothing for a second or two after a
version is created, even though the version itself reads back immediately.
Measured: empty at 0s, populated at 2s.
That undermines the version discovery added for dataset update, which reads the
listing to decide what to increment from. An empty listing is ambiguous -- it
means either a new dataset or a stale read -- so back-to-back create and update
could still restart at 1.0 and take a 409.
Rather than delaying every first upload to wait for the index, a conflict is
now treated as the stale read it is: re-read the listing, which by then
reflects reality, and retry once. The common path is unchanged.
The live test asserted on the first listing response and was failing for the
same reason. It now polls, and says why.
Verified: create immediately followed by update produces 2.0 rather than a
conflict, and the full live suite passes.
…requires The shared extension build template invokes ci-build.ps1 and ci-test.ps1 from the extension directory. Neither existed, so the release pipeline added alongside this extension would have failed on its first run. Both are modelled on the agents extension with two deliberate differences. ci-build.ps1 reads version.txt from the extension directory rather than its parent, where no such file exists, so the default works when the pipeline is not supplying -Version. It accepts -BuildRecordMode, which the template always passes, but builds nothing extra: this extension has no record/playback mode and no pipeline step consumes a record binary. ci-test.ps1 passes --junitfile explicitly. The pipeline publishes **/junitTestReport.xml from the extension directory, and the extension template does not set GOTESTSUM_JUNITFILE the way the CLI build does, so without this no test results would surface in the build. Verified locally with gotestsum installed: 97 tests across 7 suites reported. Also adds the README and CHANGELOG that 17 of the 21 extensions ship. The README documents the deployed shape, the command surface, the rubric weight constraint, and how to run the live tests.
The manifest declared two capabilities the extension did not back. metadata was declared but the command was never registered, so azd could not discover the command tree: azd ai eval metadata failed with unknown command while the same call against a peer extension returned its full tree. azd uses this for discovery, so the declaration was actively misleading. The command is now registered and reports nine commands. lifecycle-events was declared but no event handlers exist. The SDK only starts its event manager when handlers are registered, so the capability was an unused permission rather than a broken promise. It is removed; the listen command is still invoked because the service-target-provider capability triggers it, which a deploy after the change confirms. Adds tests over the manifest so neither can drift again: every declared capability must be backed by the command that implements it, the declared provider name must match the host the code registers, and version.txt must agree with the manifest version, which until now was only a comment asking for it.
…ction from a file Auditing every flag and API sequence the spec documents against the running extension turned up two gaps. The spec describes a drift check that was never implemented. It matters because of how change detection works: when local content is unchanged, the version recorded at the last deploy is reused, so a version published outside the repo would be silently ignored and the eval group pinned to older data. A deploy now fails when the service holds a newer version than the recorded one, naming both versions. An explicit version: on the declaration skips the check, because that is the author stating which version they want. This was added after testing the remedy the error message suggests and finding it did not work -- the message now describes something that does. --gen-instruction-file was documented but absent. A useful generation instruction is usually longer than fits on a command line, and putting it in a file makes it reviewable with the rest of the config. Verified live: publishing a version out-of-band fails the next deploy, and pinning that version lets it through.
… edits M1 exits on all the spec examples running end to end, so I ran them verbatim. Two did not. --eval-id could never work. It is meant to run an existing group ignoring the config, and appears in both the CI/CD example and the recovery advice, but a run needs a target and a dataset and an eval group carries neither: the group holds only its testing criteria, and the dataset travels on the run. Every --eval-id invocation failed asking for a target. The pairing survives in the group's previous run, so re-running a group now repeats what it last ran, and a group that has never run says so and points at the config-based path. The failure-and-recovery example promised an error that did not exist. A run sends a local dataset inline, so unregistered local edits were evaluated silently and the results could not be traced to any dataset version. That now fails with the message the spec documents, once a deploy has recorded a fingerprint to compare against. Before that there is nothing to have drifted from, and running is how a group first comes into existence. Verified live: the CI/CD example returns JSON with a run id, and with unregistered edits the config-based run fails while --eval-id succeeds, which is exactly the recovery the spec describes.
…ploy spec init --dataset ./tests/golden.jsonl wrote that path into evals/azure.yaml unchanged, but source: is resolved relative to the file it appears in, so the deploy looked for evals/tests/golden.jsonl and failed on a file the user had just pointed at. This is the spec's bring-your-own-data example exactly as written, so that example could never have worked. The path is now rebased onto the output directory, with forward slashes so the config reads the same on every platform, and absolute paths left alone. With this the documented examples all run end to end, which is what M1 exits on: bring-your-own-data through init, azd up and run; results show --failed-only -O writing its file; and the CI/CD sequence of dataset create, run start -o json and results export --format csv.
agent.context.traces accepts source, window and sample, but the generation API takes a day count and nothing else, so source and sample were parsed and dropped without a word. An author who set sample: 500 believed they had narrowed the trace selection when nothing had changed. Both fields are documented in the spec, so this was reachable by following it. They are now reported as having no effect, naming each one, with the verb agreeing so one field reads "has" and two read "have". The warning goes to stdout rather than stderr because azd does not surface an extension's stderr -- written to stderr it was invisible in a real run even though the unit test passed -- and is suppressed under -o json so the output stays parseable.
The feed carries a single version per extension, so this replaces beta.16 rather than adding to it. Dropping the beta subversion means the number has to move somewhere else to stay ahead: azd compares with Masterminds/semver and `1.0.0-beta` sorts below `1.0.0-beta.16`, which its isDowngrade guard then skips. `1.0.1-beta` is the same shape and still reads as an upgrade from beta.16.
Mohamed Hessien (m7md7sien)
added
area/evals
`init` learned to tell a project that provisions from one that only deploys. The three errors that also name a deploy command did not, so a reader who hit one was still sent to `azd up` in a project where it fails compiling a missing infra/main.bicep: DatasetHasUnregisteredEdits, EvalNotDeployed and EvalNotDeployedYet. They now take the resolved command. evalContext answers it once per command from the project azd reports, since it cannot change while one runs, and nextSteps takes the same string so the phrasing has a single source.
1.0.1-beta went out before the three errors learned to name the command that works, so the feed's single version no longer matched the branch.
The heading still said 1.0.0-beta.1 while version.txt and extension.yaml say 1.0.2-beta, and the release pipeline reads the entry for the version it is building. Verified by running eng/scripts/New-ReleaseNotesFile.ps1 the way the pipeline does: the notes came out as the literal string "[Missing change log entry]", and exit code 0 -- so a release would have published that quietly rather than failing. The fixes shipped since the initial entry had never been recorded, so they are listed too.
Four checks were red. None of them was a flake. `unused` found seven dead symbols: reportGenerated, writeYAML, the three progress-mark constants, resolveEvalIDFromConfig and the datasetColumns wrapper it was the only caller of, plus an envName field nothing ever read. Deleting resolveEvalIDFromConfig also orphaned the yaml import in init.go. gosec: test directories were created 0o755 and a fixture chmodded 0o444, both looser than the rule allows; two tmp.Close() calls dropped their error; the source-walking tests and the blob listing carry a nolint with the reason, since walking this package's own source and fetching the SAS the dataset service issued are what those calls are for. The download test's `credentialPath` field was reported as a hardcoded credential -- renamed to `uriPath`, which removes the finding rather than suppressing it. cspell: nine British spellings normalized, and the identifiers it could not know (projectctx, exterrors, ARMID, unbuildable, preselection) added to the dictionary. `evaulators` is in there deliberately -- it is the misspelling the unknown-key tests are built on, and the suite caught me "correcting" it. `go fix` inlined the one-line verdict helpers into new(true) and rewrote a hand-rolled contains as slices.Contains. The helpers were dead after that, so they are gone; what they documented already lives on OutputResult.Passed.
`isCredentialFailure` matched the phrase "failed to acquire a token" anywhere in an error's text. Any error is free to contain it: a pool that could not acquire a token bucket lease was reported as an expired login, with `azd auth login` as the fix. It now decides on the SDK's own types -- AuthenticationFailedError and AuthenticationRequiredError -- so a reworded message still classifies and a lookalike string does not. The two credential names stay as a fallback because credentialUnavailableError is unexported, so a credential that never ran can only be recognised by the name it puts in its own message. Also corrects two changelog bullets that would have shipped in the release notes: `init` writes `evals/azure.eval.yaml`, not `evals/eval_generate.yaml` plus `evals/azure.yaml`; and reconciliation is described as happening on deploy rather than specifically under `azd up`, which does not work for a project that ships no infrastructure.
From a pre-release review pass: - RequestFailed's doc still described the substring matching it no longer does, thirty lines above isCredentialFailure's doc saying the opposite. It is the comment a reader meets first, on the exported function. - "executable not found on path" was reported as an expired login, advising `azd auth login` -- which cannot be run when azd is what will not run. Split out as its own case with an answer that applies. - deployCommand cached whatever a failed Project().Get implied, so one gRPC hiccup downgraded the hint for the rest of the process. context.go already refuses this pattern for environments, fifty lines above; now azdProject returns an error and only a real answer is cached. - init read the project a second time, through a second azd client, for a value already bound at the top of the same closure -- and discarded the error there while treating it as fatal here, so the two could disagree. - projectCanProvision's comment claimed to mirror azd's detection. It mirrors the provider sniff; azd is also satisfied by infra layers, an Aspire AppHost, and a resources: block. Now says which, and what being wrong costs.
Carries the credential-classifier split, the review's five fixes, and the two corrected changelog bullets. Heading, version.txt and extension.yaml all moved together this time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9549
Internal work item: https://msdata.visualstudio.com/Vienna/_workitems/edit/5363979/
Adds
azd ai eval— define and run Foundry evaluations from the terminal.Replaces #9339, which pointed at a branch missing ~3,400 lines of eval work that had only ever landed downstream, including a fix for a bug that 404'd any dataset whose name contained a space.
Surface
Notable
One
generate, with a selector.generatesubmits both generation jobs in parallel;--dataset/--evaluatornarrows to one. Each reports its own outcome, the command exits non-zero if either failed, and whatever succeeded is still recorded.--no-waitresumes throughazd ai eval job, where the selector is required because a job id alone does not say which collection to poll.Runs know what they are scoring. The run data source is decided by configuration: a
source:block hands gathering to the service (traces, stored responses), otherwise rows come from the dataset andtarget:says what to invoke — an agent, a model, or nothing at all. Previously only agent targets existed, soinit --source traceswrote a configuration its ownrunrefused, and a model target was validated as legal then sent as an agent.initdetects rather than demands.--targetdefaults to the project's only agent, prompts when there are several, and names the flag under--no-prompt.Every user-facing string lives in one file (
internal/messages/messages.go) — so the whole voice of the CLI can be reviewed in one sitting. The move surfaced real duplicates, including three copies of one error and two contradictory--formatmessages where one was unreachable.The configuration is
azure.eval.yaml, prefixed for azd the wayazure.yamlis. An existingeval.yamlis still read and written back to, so nothing breaks and no project grows a second file. A directory holding both is refused:azure.yamlreferences one by name, so silently preferring one would mean editing one configuration whileazd updeployed the other.Waiting is bounded. A run that never reaches a terminal state used to hold the terminal open indefinitely. The wait now stops after two hours and is treated the way
--no-waitis — the run is still going server-side, so the caller gets the reattach line and exit 0 rather than an error about a failure that did not happen.Every
showrenders a detail view, with the machine-readable document behind-o json.run output showwas the last one emitting raw JSON whatever was asked for, which made the command a person reaches for after a failing listing the hardest one in the CLI to read.The end-to-end suites were not running
go test ./...reported a clean run for this extension while compiling none oftests/cli,tests/liveortests/hero— all three carry build tags, andgo list ./tests/...answers "matched no packages". CI type-checks them but never executes them, so 1,822 lines of end-to-end tests had gone unrun and were holding seven failures.Running them found
--format jsonlbeing refused by a guard that named only json and csv while the flag's own help offered jsonl;evaluator showmissing the--output-filethat the reconciliation error tells the reader to adopt a remote change with; and a missing PASS THRESHOLD column. The other four were stale assertions, re-pinned to the spec rather than to the implementation.Verified live
Against a real Foundry project, end to end:
azd ai eval init --source traceswith no other flags, then a run scoring 20 real conversations from Application Insights.--no-wait -o json, gate pass exit 0, gate breach exit 1).azd, which is where the remaining not-found and--output-filedefects came from.Green: 524 unit tests,
tests/cli66,tests/live5,tests/hero8,go vet -tags live,hero,gofmt, andci-test.ps1exit 0.Reviewed
Nine model-assisted review passes (Sonnet 4.6, GPT-5.3-Codex, Gemini 3.1 Pro, Grok 4.5, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna) found real defects that are fixed here — among them dataset job commands calling their collection on the wrong API version,
--no-wait -o jsonlosing the job ids it exists to report, a declared target with no name being silently scored as no target, two nil-deref panics in the endpoint cascade, andevaluator show --output-filetruncating the local definition it was meant to update if the write failed.Not included, deliberately
No
registry.jsonentry and nomicrosoft.foundrybundle dependency — both need published artifacts, and listing an unpublished extension in the bundle breaksazd extension install microsoft.foundryfor everyone.Known gaps
initdoes not yet prompt for evaluators or judge model; it detects sensible values instead. The spec calls for prompts with an error under--no-prompt. Not changed here because the--evaluatordefault also schedules rubric generation, which is what makes theinit→generateflow work; reshaping that is broader than this PR.dataset versions liston an unknown name lists nothing and exits 0 here, while the dataset extension (Add azd ai dataset extension for Foundry datasets #9499) answers the same question with an error. A list is a filter rather than a lookup, so empty-and-successful is defensible and two tests depend on it — but the two extensions should agree. Raised for reviewers rather than settled unilaterally.main.goreports 1 unless the error chain carries an*internal.ExitCodeError, which is constructed only on the hooks path. Spec Scenario 5 wants 2 for "regressed" versus 1 for "could not run". The fix is in azd core, not here.Draft — not requesting reviewers yet.