Add read-only workflow failure classifier - #9543
Open
malmstein wants to merge 5 commits into
Open
Conversation
The nightly release-blocking workflows already auto-create an Asana task when they fail, but that task carries only the run URL and, for e2e, a one-line assertion message. Everything that makes a failure actionable still has to be reconstructed by hand on maintenance day: which step failed, whether it is our change or infrastructure, how often it has happened recently, and which commits could have caused it. This agentic workflow does that reconstruction from the run itself. It reads the failed job and step, the job logs, the Maestro per-flow results, the recent run history for the same workflow, and the commits between the last green run and the failing SHA. It also checks whether a fix has already landed after the failing SHA, since a human often fixes a nightly hours before anyone reads the task. This first version is deliberately read-only: the diagnosis goes to the run summary and nowhere else. No safe-outputs are configured, so it cannot comment, open issues or PRs, or write to Asana. The intent is to review a week of real diagnoses before granting it any write access. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second dry run, against a Privacy Tests failure, exposed three gaps. The Flank instrumentation logs carry a jq'd block of Test / Failure / URL triples and a matrix line with the pass/fail count. Without those the agent has exception traces it cannot attribute to a test, so name them the way the Maestro flow results are already named. The already-fixed search needed a direction check. A candidate fix only explains a failure away when it is not already an ancestor of the failing SHA; when it is, the failure is a recurrence or a new symptom, which is a more serious finding rather than a lesser one. When a previous PR fixed one occurrence of an error signature, sibling tests often still carry the unfixed pattern. That is the cheapest finding to act on and it is invisible from the failure alone, so look for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback from the maintenance AoR: helpful, but too much text. The run URL, the workflow name and the release-blocking classification are all already on the task the reader is looking at, so repeating them was pure length. What the DRI actually wants is the likely cause, in one line, from a bounded set: flaky test, feature-flag rollout, recent change, infrastructure, or unclear. The report now leads with that cause and carries only the failing test, one error excerpt, the single fact supporting the cause, and the two context lines worth having. Every line is droppable when it says nothing, under a hard 12-line budget. The per-suite E2E failure tasks are being retired to cut noise, so the report must also stand alone as a comment on the one workflow failure task rather than assume a suite-specific task is open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example the team signed off on separates clauses with commas, so match it in the template and say so explicitly, since the report is posted where the team reads it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
malmstein
marked this pull request as ready for review
August 20, 2026 15:22
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0178566. Configure here.
Two review findings on the trigger. The membership gate required the actor to hold admin, maintainer or write. This workflow is started by a scheduled nightly rather than by a person, so relying on the inherited actor to carry a role risks silently skipping every automatic diagnosis. It reads only same-repo, non-fork runs on develop and writes nothing, so the gate can only cost us legitimate runs. Drop it with roles: all, keeping the fork and same-repository guards the compiler adds. The trigger also had no conclusion filter, so a green nightly still started the whole agent just to have it report that nothing failed. Gate the activation on the run having failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task/Issue URL: https://app.asana.com/1/137249556945/project/1214901934989258/task/1217636465596685
Tech Design URL (if applicable):
API Proposals URL(s) (if applicable):
Description
Adds an agentic workflow that diagnoses a failed release-blocking nightly run and writes up the likely cause, so the engineer on maintenance rotation gets a diagnosis instead of a bare "job failed" task. It reads the failing step, the job logs, the Maestro per-flow results or the Flank test/failure pairs, the recent run history, and the commits between the last green run and the failing SHA; it does not create tasks (the existing steps in
nightly.yml,privacy.ymland the Maestro reporter already do that) and it does not judge release-blocking severity, which is a lookup from which workflow failed rather than a judgement. This first version is read-only: nosafe-outputsare configured, so it can only write to the run summary, and the report itself leads with a likely cause from a fixed set (flaky test, feature-flag rollout, recent change, infrastructure, unclear) under a 12-line cap, per review feedback from the maintenance AoR.The one non-obvious thing:
roles: alldeliberately drops the membership gate. Aworkflow_runstarted by a scheduled nightly inherits an actor that cannot be relied on to hold a repo role, and if it does not, every automatic diagnosis is skipped silently. The workflow reads only same-repo, non-fork runs ondevelopand writes nothing, and the fork and same-repository guards stay in place.Steps to test this PR
Compilation
gh aw compile workflow-failure-classifierreports 0 errors, 0 warningsRead-only guarantees, in the lock file
GITHUB_READ_ONLY: "1"is set on the GitHub MCP serverGH_AW_SAFE_OUTPUTS_HANDLER_CONFIGappears anywhereactions: readand no write permissionsTrigger
github.event.workflow_run.conclusion == 'failure', so a green nightly starts nothingLive run, only possible once this is on
develop, sinceworkflow_dispatchis only registered for workflows on the default branchgh workflow run workflow-failure-classifier.lock.yml --ref develop -f run_id=32324415014GpcTest#whenProtectionsAreEnableGpcSetCorrectly, and references Pin HttpsUpgradesTest interactions to the tab's WebView #9534UI changes