Skip to content

Adding fix workflow to failing pipelines - #48554

Open
ReilleyMilne (ReilleyMilne) wants to merge 15 commits into
mainfrom
reilleymilne/pipeline-analysis-workflow
Open

Adding fix workflow to failing pipelines#48554
ReilleyMilne (ReilleyMilne) wants to merge 15 commits into
mainfrom
reilleymilne/pipeline-analysis-workflow

Conversation

@ReilleyMilne

Copy link
Copy Markdown
Member
  • Simplifies the analysis workflow.
  • Adds an additional agentic workflow to automatically suggest fixes to failing pipelines. This extends the already existing analysis workflow.
  • Currently the fix is done in an agent authored branch + PR, so this PR also includes a workflow to clean up those opened branches. The PR's opened by GitHub agentic workflows are automatically closed by the agentics-maintenance.yml run.

Example Demos:
Analysis: ReilleyMilne#58
Fix: ReilleyMilne#59

Fixes: Azure/azure-sdk-tools#16205

Copilot AI balanced review requested due to automatic review settings August 12, 2026 20:32
@ReilleyMilne
ReilleyMilne (ReilleyMilne) requested a review from a team as a code owner August 12, 2026 20:32
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automated diagnosis and draft fix proposals for failing Azure SDK pipelines, addressing Azure/azure-sdk-tools#16205.

Changes:

  • Simplifies pipeline analysis with direct check_suite handling.
  • Adds an agentic auto-fix workflow that creates draft fix PRs.
  • Adds cleanup for completed auto-fix branches.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
.github/workflows/pipeline-analysis-next-steps.md Updates analysis and auto-fix dispatch logic.
.github/workflows/pipeline-analysis-next-steps.lock.yml Compiles the updated analysis workflow.
.github/workflows/pipeline-analysis-next-steps-trigger.yml Removes the superseded dispatcher.
.github/workflows/pipeline-analysis-branch-cleanup.yml Cleans obsolete auto-fix branches.
.github/workflows/pipeline-analysis-auto-fix.md Defines draft fix generation and retargeting.
.github/workflows/pipeline-analysis-auto-fix.lock.yml Compiles the auto-fix workflow.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/pipeline-analysis-next-steps.md
Comment thread .github/workflows/pipeline-analysis-next-steps.md
Copilot AI review requested due to automatic review settings August 12, 2026 21:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/pipeline-analysis-next-steps.md:180

  • failed_pipeline_tests is a platform-to-test-title map, not a collection containing artifact_file_path (the checked-in pipeline-analysis skill likewise describes title lists). Therefore this condition can never obtain a path, and because azsdk_get_pipeline_llm_artifacts is absent from the MCP allowlist, the workflow cannot fetch the detailed test data that the following sentence requires before diagnosis/fixability classification. Allow and call azsdk_get_pipeline_llm_artifacts first, then pass its returned artifact paths to the test-run tools.
4. Inspect every `failed_pipeline_tests` entry returned by the analysis. If an
  `artifact_file_path` is present, call `azsdk_get_failed_test_run_data` exactly once per unique
  artifact with `failedTestRunsPath` set to that path. Call `azsdk_get_failed_test_case_data` only

.github/workflows/pipeline-analysis-next-steps.md:43

  • The replaced trigger explicitly skipped draft PRs, but this new gate treats drafts as eligible. As a result, expected failures while an author is still drafting can consume Copilot runs and automatically open fix PRs. Preserve the previous draft guard unless enabling automation on drafts is intentional.
          const matchingPulls = pulls.filter(
            pull => pull.state === "open" && pull.head.sha === suite.head_sha
          );

@benbp

Copy link
Copy Markdown
Member

It would be nice if we can work in some logging in a pre/post step around the main work to show the impact on github API rate limits. We have an azure pipelines example here: https://github.com/Azure/azure-rest-api-specs/blob/5b13925393aba9cb9248171d1bbab9287336f569/eng/common/pipelines/templates/steps/emit-rate-limit-metrics.yml#L14 but Mike Harder (@mikeharder) may be able to show you a better example of where we are doing it in github workflow contexts.

@benbp

Copy link
Copy Markdown
Member

Otherwise, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build an agent workflow that turns analysis into targeted fix proposals (azure-sdk-for-python)

3 participants