Skip to content

Ignore Dependabot security updates in the failure watcher - #757

Merged
oschwald merged 1 commit into
mainfrom
wstorey/stf-1245-dependabot-failure-watcher-emails-go-to-the-correct-person
Jul 30, 2026
Merged

Ignore Dependabot security updates in the failure watcher#757
oschwald merged 1 commit into
mainfrom
wstorey/stf-1245-dependabot-failure-watcher-emails-go-to-the-correct-person

Conversation

@horgh

@horgh horgh commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

GitHub runs Dependabot version updates and Dependabot security updates under one
workflow name, Dependabot Updates, and the watcher counted both. Security
updates routinely fail for reasons no pull request can fix -- the advisory is
against a dependency the project does not declare directly, or no patched
version is reachable -- so the watcher stays red every week on those and trains
everyone to ignore it.

This filters security-update runs out by title, documents why both halves of
" in /. for " are load-bearing, and bounds the gh run list query server-side
with --created instead of fetching all of history and filtering locally.

See the commit message for the full reasoning, including why reading the
directory list out of dependabot.yml was tried and rejected.

This workflow is shared verbatim across MaxMind repos. The change was developed
in maxmind/device-android (maxmind/device-android#71)
and is applied here unmodified; the resulting file is byte-identical in every
repo.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Dependabot failure monitoring to focus on failed version update runs.
    • Excluded unrelated Dependabot update types and added clearer handling for end-to-end related runs.
    • Refined the monitoring window and run selection to make failure detection more reliable.
  • Documentation
    • Updated workflow inline guidance to clarify how different Dependabot run categories are evaluated and which runs are included or excluded.

Copilot AI review requested due to automatic review settings July 30, 2026 16:59
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The Dependabot failure watcher documents run classification rules and checks recent Dependabot version update failures, excluding selected directory-specific runs before reporting failure states.

Changes

Dependabot version failure filtering

Layer / File(s) Summary
Document and filter version update failures
.github/workflows/dependabot-failure-watcher.yml
The workflow documents Dependabot run categories and title patterns, applies server-side eight-day filtering with a higher run limit, excludes matching titles, and reports version update failures using the existing failure conclusions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit watching builds in flight,
Sorting update runs just right.
Eight days close, the filters hop,
Failed version scans rise to the top.
“No failures!” thumps my tiny feet.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: the failure watcher now ignores Dependabot security-update runs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch wstorey/stf-1245-dependabot-failure-watcher-emails-go-to-the-correct-person

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

The workflow change is narrowly scoped, well-documented, and aligns with the stated goal of suppressing non-actionable security-update failures without impacting version-update failure detection.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Updates the shared “Dependabot Failure Watcher” GitHub Actions workflow to reduce alert fatigue by ignoring Dependabot security update workflow runs (which frequently fail for non-actionable reasons), while continuing to fail on Dependabot version update failures within the last 8 days.

Changes:

  • Excludes security-update runs by filtering workflow runs based on displayTitle patterns (notably " in /. for " and specific e2e/{js,ts} paths).
  • Improves documentation in the workflow file to explain the run-title taxonomy and why the matching is intentionally specific.
  • Switches from client-side time filtering to server-side bounding via gh run list --created, with an increased limit to avoid truncating in-window results.
File summaries
File Description
.github/workflows/dependabot-failure-watcher.yml Filters out Dependabot security-update runs and bounds gh run list by date server-side, with expanded rationale docs.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

GitHub runs both Dependabot version updates and Dependabot security
updates under one workflow name, "Dependabot Updates", and the watcher
counted both. Security updates routinely fail for reasons no pull request
can fix -- the advisory is against a dependency this project does not
declare directly, or no patched version is reachable. Left alone the
watcher stays red every week on those and trains everyone to ignore it.

Filter those runs out by title. Version updates are unaffected.

The title check is subtler than it looks, so document it properly. A
security job is marked by "/." AND a " for " suffix together; version
updates are either "/." with no " for " (the scheduled scan) or "/" with
one (the pull request). Both halves of " in /. for " are therefore
load-bearing -- matching on " in /." alone would discard every scan run,
which is most of the version-update runs and the shape the failures this
watcher was written for actually took.

That "/." spelling only separates the two at the repo root. In a
subdirectory a security update and a version update's pull request render
identically, so drop " in /e2e/{js,ts} for " by name as well. The Node
repos this workflow is shared with carry committed lockfiles under e2e/js
and e2e/ts, whose transitive dev dependencies attract advisories no pull
request can fix, and nothing in either is shipped code. That is 16
unactionable failures in each of GeoIP2-node and minfraud-api-node over
retained history; repos without those directories are unaffected.

Unlike the root filter, this one is not free. Both Node repos configure
npm with directories: ["/", "**/*"], and that glob does match e2e/js and
e2e/ts, so those directories do get version updates -- there is an open
version-update pull request under e2e/ts in both repos as this is
written. Dropping the pattern discards their pull-request refresh
failures along with the security jobs, and the ecosystem label is no help
because Dependabot writes "npm_and_yarn" for both. Taken anyway: the
scheduled scan is what this watcher primarily exists to catch and is
still reported for those directories, so what is given up is the narrower
"one open pull request has gone stale" signal for two directories of test
scaffolding. After filtering, 4 genuine failures remain reported in
GeoIP2-node and 3 in minfraud-api-node.

Reading the directories out of dependabot.yml would look more general and
was the earlier plan here, but it fails green. Entries may use globs, and
minfraud-api-dotnet's directories: ["**/*"] yields titles like "nuget in
/**/*" for the scan and "nuget in /MaxMind.MinFraud for
System.Net.Http.Json" for the pull request, neither of which any literal
comparison against the configured value matches -- so its two real nuget
failures would have been dropped without a word. A stale denylist
re-introduces noise, which is loud; a stale allowlist hides failures.

Name the three kinds of run in the comment while here, because the
scheduled scan and the per-pull-request refresh are easy to conflate: the
refresh runs are one per open pull request and are triggered by pushes to
the base branch or by rebases, not by the schedule, so they arrive in
bursts after merges. The scan is the kind this watcher primarily exists
to catch, which is what makes hiding a hypothetical refresh failure under
e2e an acceptable cost rather than a hole.

Bound the query server-side with --created instead of fetching all of
history and filtering by date locally, so --limit now caps an
already-narrowed window rather than standing in for one, and the run list
drops from several API pages to one. --limit rises 100 -> 500 as a
backstop: it still applies before the title filter, and reaching it would
silently drop the oldest in-window runs.

This workflow is shared verbatim across MaxMind repos. The change was
developed in maxmind/device-android and is applied here unmodified; see
that repo's commit for the measurements it was derived from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@horgh
horgh force-pushed the wstorey/stf-1245-dependabot-failure-watcher-emails-go-to-the-correct-person branch from 7c8fafa to 0c04a80 Compare July 30, 2026 17:53

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/dependabot-failure-watcher.yml:
- Around line 104-113: Update the run collection around gh run list so it
detects when the 500-result limit is saturated and does not report an all-clear
from an incomplete newest-first subset. Fail closed when the cap is reached, or
paginate through all date-filtered runs before applying the existing jq failure
filters; preserve the current exclusions and failure reporting for the complete
result set.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b544adf7-48c8-4573-a7a7-5a083e2a5cea

📥 Commits

Reviewing files that changed from the base of the PR and between 7c8fafa and 0c04a80.

📒 Files selected for processing (1)
  • .github/workflows/dependabot-failure-watcher.yml

Comment on lines +104 to +113
runs=$(gh run list \
--repo "$REPO" \
--workflow "Dependabot Updates" \
--limit 100 \
--json conclusion,createdAt,displayTitle,url \
--jq "[.[] | select((.conclusion == \"failure\" or .conclusion == \"startup_failure\" or .conclusion == \"timed_out\") and .createdAt >= \"$since\")]")
--created ">=$since" \
--limit 500 \
--json conclusion,createdAt,displayTitle,url)
failures=$(echo "$runs" | jq '
[.[]
| select((.displayTitle | contains(" in /. for ")) | not)
| select((.displayTitle | test(" in /e2e/(js|ts) for ")) | not)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate workflow =="
fd -a 'dependabot-failure-watcher.yml' . || true

echo "== relevant section =="
if [ -f ".github/workflows/dependabot-failure-watcher.yml" ]; then
  nl -ba .github/workflows/dependabot-failure-watcher.yml | sed -n '90,145p'
fi

echo "== repo action versions / gh availability clues =="
git ls-files .github/workflows | xargs -r sed -n '/runs-as: ubuntu|actions\/setup-node|setup-node|apt-get update|gh run list/p' 2>/dev/null | head -120

Repository: maxmind/GeoIP2-java

Length of output: 242


🌐 Web query:

GitHub CLI gh run list default limit --json behavior truncated latest run count

💡 Result:

In the GitHub CLI (gh), the gh run list command retrieves workflow runs with a default limit of 20 [1][2][3]. You can adjust this count using the -L or --limit flag [1][4]. Regarding the behavior of --json and truncation: 1. Data Truncation: There is no inherent, documented behavior in the GitHub CLI that forces truncation of data when using the --json flag [5]. If data appears truncated in output, it is generally not a standard feature of the --json output mechanism itself [5]. Users have occasionally reported issues regarding long string truncation, but these are typically treated as bugs or specific formatting issues rather than intended CLI behavior [5]. 2. JSON Fields: The --json flag requires a comma-separated list of fields [1][6]. If you do not know which fields are available, you can omit the arguments to the --json flag (e.g., gh run list --json) to see a list of valid fields for that command [6]. Valid fields commonly supported by the GitHub API for workflow runs include databaseId, number, url, status, conclusion, workflowName, createdAt, startedAt, updatedAt, event, headBranch, headSha, and displayTitle [7]. The gh run list command fetches data from the GitHub Actions REST API; therefore, the consistency and completeness of the returned data are primarily governed by that API rather than by client-side truncation in the GitHub CLI [5][7][8].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'pwd=%s\n' "$(pwd)"
printf 'files:\n'
find .github/workflows -maxdepth 1 -type f 2>/dev/null | sort | head -100 || true

printf 'dependabot failure watcher path candidates:\n'
find . -name 'dependabot-failure-watcher.yml' -o -name 'dependabot-failure-watcher.yaml' 2>/dev/null | sort

printf 'workflow size:\n'
wc -l .github/workflows/dependabot-failure-watcher.yml 2>/dev/null || true

printf 'relevant section:\n'
sed -n '90,145p' .github/workflows/dependabot-failure-watcher.yml 2>/dev/null || true

printf 'gh run list related searches:\n'
rg -n "gh run list|limit 500|500|no runs failed|all-clear|fails" .github/workflows/dependabot-failure-watcher.yml .github/workflows 2>/dev/null || true

Repository: maxmind/GeoIP2-java

Length of output: 3951


🌐 Web query:

site:cli.github.com/manual/gh_run_list --limit gh run list returns runs created since

💡 Result:

The --limit flag for the gh run list command does not filter runs by date; instead, it specifies the maximum number of workflow runs to fetch [1]. The default value for this flag is 20 [1]. To filter runs by the date they were created, you must use the --created flag [1]. The --created flag accepts date ranges and specific dates in formats such as YYYY-MM-DD [1].

Citations:


🌐 Web query:

GitHub Actions run list API default 30 max 100 gh run list pagination

💡 Result:

When interacting with GitHub's REST API and the GitHub CLI (gh), pagination behavior and limits differ as follows: GitHub REST API (Workflow Runs) For endpoints such as List workflow runs for a repository or List workflow runs for a workflow, the API supports pagination using the per_page and page parameters [1][2][3]. - per_page default: 30 [1][2][3] - per_page maximum: 100 [1][2][3] - The API may return up to 1,000 results when specific filtering parameters (such as actor, branch, event, or status) are used [1][2][3]. GitHub CLI (gh run list) The GitHub CLI command gh run list abstracts these API calls. - --limit (-L) default: 20 [4][5] - The --limit flag allows you to specify the maximum number of runs to fetch [4][5]. - When using the lower-level gh api command, you can use the --paginate flag to automatically fetch all pages of results [6]. Note that when using --paginate with certain API endpoints or filters, there may be a hard limit of 1,000 results returned [7].

Citations:


🌐 Web query:

CLI gh run list 500 limit GitHub Actions API pagination

💡 Result:

For the gh run list command, the --limit flag allows you to specify the maximum number of workflow runs to fetch [1]. The current maximum value supported by this flag is 1,000 [2]. There is no native support for an "unlimited" option or a value higher than 1,000 for this specific command [3][2]. If you need to process or retrieve more than 1,000 workflow runs, you have two primary options: 1. Iterative batching: You can run gh run list in a loop, filtering by creation date or processing runs in batches of 1,000 until no more results are returned [4][2]. 2. Using gh api with pagination: The gh api command serves as a direct interface to the GitHub REST API and supports an --paginate flag [5][6]. Unlike gh run list, this flag allows the CLI to automatically follow pagination links until all available results are retrieved [5][7]. When using gh api for this purpose, you can target the workflow runs endpoint directly: gh api --paginate "/repos/{owner}/{repo}/actions/runs?per_page=100" [6] Note that when using gh api --paginate, you should be aware that it makes sequential HTTP requests, which will count against your GitHub API rate limits [7]. Additionally, some specific API endpoints—particularly those involving certain search parameters—may have their own server-side result caps (e.g., 1,000 results) regardless of client-side pagination [8]. If you encounter this, refining your query with more specific filters (such as status or date ranges) is recommended to retrieve the complete data set [4][8].

Citations:


Detect when gh run list hits its result cap.

The workflow already notes that gh run list returns runs newest-first and can drop the oldest in-window runs when --limit is reached. If the 500-run cap is reached, filter only applies to the fetched subset, so older failed version updates can be omitted and the all-clear can still be reported. Fail closed on query saturation or paginate until the full date-filtered result set is collected.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/dependabot-failure-watcher.yml around lines 104 - 113,
Update the run collection around gh run list so it detects when the 500-result
limit is saturated and does not report an all-clear from an incomplete
newest-first subset. Fail closed when the cap is reached, or paginate through
all date-filtered runs before applying the existing jq failure filters; preserve
the current exclusions and failure reporting for the complete result set.

@oschwald
oschwald merged commit e8134bc into main Jul 30, 2026
31 checks passed
@oschwald
oschwald deleted the wstorey/stf-1245-dependabot-failure-watcher-emails-go-to-the-correct-person branch July 30, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants