From 43124474c9d3f62e3cae6d8c08d775bb90630a49 Mon Sep 17 00:00:00 2001 From: William Storey Date: Thu, 30 Jul 2026 16:58:35 +0000 Subject: [PATCH] Ignore Dependabot security updates in the failure watcher GitHub runs both Dependabot version updates and Dependabot security updates under one workflow name, "Dependabot Updates", and the watcher counted both. Security updates routinely fail for reasons no pull request can fix -- the advisory is against a dependency this project does not declare directly, or no patched version is reachable. Left alone the watcher stays red every week on those and trains everyone to ignore it. Filter those runs out by title. Version updates are unaffected. The title check is subtler than it looks, so document it properly. A security job is marked by "/." AND a " for " suffix together; version updates are either "/." with no " for " (the scheduled scan) or "/" with one (the pull request). Both halves of " in /. for " are therefore load-bearing -- matching on " in /." alone would discard every scan run, which is most of the version-update runs and the shape the failures this watcher was written for actually took. That "/." spelling only separates the two at the repo root. In a subdirectory a security update and a version update's pull request render identically, so drop " in /e2e/{js,ts} for " by name as well. The Node repos this workflow is shared with carry committed lockfiles under e2e/js and e2e/ts, whose transitive dev dependencies attract advisories no pull request can fix, and nothing in either is shipped code. That is 16 unactionable failures in each of GeoIP2-node and minfraud-api-node over retained history; repos without those directories are unaffected. Unlike the root filter, this one is not free. Both Node repos configure npm with directories: ["/", "**/*"], and that glob does match e2e/js and e2e/ts, so those directories do get version updates -- there is an open version-update pull request under e2e/ts in both repos as this is written. Dropping the pattern discards their pull-request refresh failures along with the security jobs, and the ecosystem label is no help because Dependabot writes "npm_and_yarn" for both. Taken anyway: the scheduled scan is what this watcher primarily exists to catch and is still reported for those directories, so what is given up is the narrower "one open pull request has gone stale" signal for two directories of test scaffolding. After filtering, 4 genuine failures remain reported in GeoIP2-node and 3 in minfraud-api-node. Reading the directories out of dependabot.yml would look more general and was the earlier plan here, but it fails green. Entries may use globs, and minfraud-api-dotnet's directories: ["**/*"] yields titles like "nuget in /**/*" for the scan and "nuget in /MaxMind.MinFraud for System.Net.Http.Json" for the pull request, neither of which any literal comparison against the configured value matches -- so its two real nuget failures would have been dropped without a word. A stale denylist re-introduces noise, which is loud; a stale allowlist hides failures. Name the three kinds of run in the comment while here, because the scheduled scan and the per-pull-request refresh are easy to conflate: the refresh runs are one per open pull request and are triggered by pushes to the base branch or by rebases, not by the schedule, so they arrive in bursts after merges. The scan is the kind this watcher primarily exists to catch, which is what makes hiding a hypothetical refresh failure under e2e an acceptable cost rather than a hole. Bound the query server-side with --created instead of fetching all of history and filtering by date locally, so --limit now caps an already-narrowed window rather than standing in for one, and the run list drops from several API pages to one. --limit rises 100 -> 500 as a backstop: it still applies before the title filter, and reaching it would silently drop the oldest in-window runs. This workflow is shared verbatim across MaxMind repos. The change was developed in maxmind/device-android and is applied here unmodified; see that repo's commit for the measurements it was derived from. Co-Authored-By: Claude Opus 5 --- .../workflows/dependabot-failure-watcher.yml | 100 ++++++++++++++++-- 1 file changed, 90 insertions(+), 10 deletions(-) diff --git a/.github/workflows/dependabot-failure-watcher.yml b/.github/workflows/dependabot-failure-watcher.yml index 83a5036..a260266 100644 --- a/.github/workflows/dependabot-failure-watcher.yml +++ b/.github/workflows/dependabot-failure-watcher.yml @@ -2,9 +2,77 @@ name: Dependabot Failure Watcher # Dependabot version updates run as GitHub Actions workflow runs named # "Dependabot Updates". This scheduled job looks back over the past week for any -# of those runs that failed and fails itself if it finds one, so a silently-broken -# ecosystem surfaces as a red scheduled run instead of only a red triangle in the -# Dependabot tab that nobody checks. +# version-update run that failed and fails itself if it finds one, so a +# silently-broken ecosystem surfaces as a red scheduled run instead of only a red +# triangle in the Dependabot tab that nobody checks. Security-update runs share +# that workflow name and are deliberately excluded -- see below. +# +# GitHub reuses the "Dependabot Updates" name for three different kinds of run: +# +# 1. A version update's scheduled scan: one run per .github/dependabot.yml +# entry, on the schedule set there. It works out what is out of date and +# opens or updates pull requests. This is the kind this watcher primarily +# exists to catch -- when a scan breaks, the whole ecosystem quietly stops +# being updated and nothing else tells anyone. +# 2. A version update's per-pull-request refresh: one run per already-open +# Dependabot pull request, rebasing or re-checking it. These are not driven +# by the schedule at all -- a push to the base branch, a rebase, or an +# "@dependabot recreate" comment triggers them, so they arrive in bursts +# after merges rather than at the scheduled time. A failure here means one +# open pull request has gone stale, which is worth knowing but is much +# narrower than a broken scan. +# 3. A security update: one ad-hoc job per vulnerable package, triggered by a +# Dependabot alert rather than by dependabot.yml at all. +# +# Kind 3 routinely fails for reasons no pull request can fix: the advisory is +# against a dependency this project does not declare directly, or no patched +# version is reachable. Counting those would keep this workflow permanently red +# and train everyone to ignore it, so they are filtered out below. +# +# Of the fields "gh run list --json" exposes, only the title separates the three +# -- event, headBranch and actor are identical. Titles come in these shapes: +# +# - " in /." -- kind 1 at the repo root, which +# has no " for " suffix +# - " in " -- kind 1 elsewhere, path verbatim +# - " in / for " -- kind 2 at the repo root +# - " in for " -- kind 2 elsewhere +# - " in /. for " -- kind 3 at the repo root +# - " in for " -- kind 3 elsewhere, where +# is wherever the vulnerable manifest was discovered +# +# At the root, then, kind 3 is marked by "/." AND a " for " suffix together, and +# BOTH HALVES of " in /. for " are load-bearing -- do not shorten it. Matching on +# " in /." alone would also discard every kind 1 run, which is most of the runs +# here and the shape both failures this watcher was written for actually took. +# +# Outside the root, kinds 2 and 3 cannot be told apart by title, so the filter +# has to name directories instead. e2e/js and e2e/ts (in the Node repos this +# workflow is shared with) are consumer smoke tests carrying committed +# lockfiles, so their transitive dev dependencies attract advisories that no +# pull request can fix, and nothing in them is shipped code. +# +# Be clear about the cost, because it is not zero: both Node repos configure npm +# with directories: ["/", "**/*"], and that glob does match e2e/js and e2e/ts, +# so those directories DO get version updates. Dropping the pattern therefore +# discards their kind 2 failures as well as their kind 3 ones -- there is an +# open version-update pull request under e2e/ts in both repos as this is +# written. The npm ecosystem label does not rescue the distinction either: +# Dependabot writes "npm_and_yarn" for both kinds, so "npm_and_yarn in /e2e/ts +# for js-yaml" could be either a security job or the refresh of a +# version-update pull request. +# +# Accepted deliberately anyway. Kind 1 is what this watcher primarily exists to +# catch and is still reported for those directories, so what is given up is the +# narrower "one open pull request has gone stale" signal, for two directories of +# test scaffolding, in exchange for dropping 16 unactionable failures in each of +# the two Node repos over retained history. +# +# Reading the directories out of dependabot.yml instead looks more general but is +# worse: entries may use globs (directories: ["**/*"]), which never match a title +# literally, so genuine failures would be dropped without a word. Prefer a +# denylist: when it goes stale it re-introduces noise, which is loud, whereas a +# stale allowlist hides failures, which is silent. # # Runs entirely within this repo (no external service). A failed scheduled run # emails the person who last edited the cron below. Note: GitHub auto-disables @@ -22,22 +90,34 @@ jobs: check-dependabot-runs: runs-on: ubuntu-latest steps: - - name: Fail if any Dependabot update failed in the last 8 days + - name: Fail if any Dependabot version update failed in the last 8 days env: GH_TOKEN: ${{ github.token }} REPO: ${{ github.repository }} run: | since=$(date -u -d '8 days ago' +%Y-%m-%dT%H:%M:%SZ) - failures=$(gh run list \ + # --created filters server-side, so --limit applies to runs already + # narrowed to the window rather than to all of history. Runs come back + # newest-first, so reaching the limit would drop the oldest in-window + # runs and this step would report all-clear without them -- hence a + # limit far above any plausible week's worth of runs. + runs=$(gh run list \ --repo "$REPO" \ --workflow "Dependabot Updates" \ - --limit 100 \ - --json conclusion,createdAt,displayTitle,url \ - --jq "[.[] | select((.conclusion == \"failure\" or .conclusion == \"startup_failure\" or .conclusion == \"timed_out\") and .createdAt >= \"$since\")]") + --created ">=$since" \ + --limit 500 \ + --json conclusion,createdAt,displayTitle,url) + failures=$(echo "$runs" | jq ' + [.[] + | select((.displayTitle | contains(" in /. for ")) | not) + | select((.displayTitle | test(" in /e2e/(js|ts) for ")) | not) + | select(.conclusion == "failure" + or .conclusion == "startup_failure" + or .conclusion == "timed_out")]') count=$(echo "$failures" | jq 'length') if [ "$count" -gt 0 ]; then - echo "::error::$count failed Dependabot update run(s) in the last 8 days:" + echo "::error::$count failed Dependabot version update run(s) in the last 8 days:" echo "$failures" | jq -r '.[] | "- \(.displayTitle) (\(.createdAt))\n \(.url)"' exit 1 fi - echo "No failed Dependabot update runs in the last 8 days." + echo "No failed Dependabot version update runs in the last 8 days."