Ignore Dependabot security updates in the failure watcher - #29
Conversation
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Dependabot failure watcher now documents monitored run types and excludes specific security-update titles. Its enforcement step uses server-side date filtering, title exclusions, and expanded failure conclusions before reporting results. ChangesDependabot failure watcher
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR updates the “Dependabot Failure Watcher” GitHub Actions workflow to ignore Dependabot security update runs (which share the same workflow name as version updates) so the watcher only flags failures that indicate a broken version-update ecosystem.
Changes:
- Filters out security-update workflow runs by matching
displayTitlepatterns (notably excluding titles containingin /. forand runs for/e2e/jsand/e2e/ts). - Uses
gh run list --created ">=$since"to bound results server-side instead of fetching and filtering full history locally. - Expands inline documentation explaining the three “Dependabot Updates” run types and why the title substrings are intentionally specific.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/dependabot-failure-watcher.yml:
- Around line 88-98: The run scan in the workflow currently treats the fixed
--limit 500 result as complete even when older in-window runs are omitted.
Update the run-listing logic around the runs command to paginate through all
matching action runs, or detect that the configured limit was reached and mark
the scan as truncated so the workflow fails closed instead of reporting
all-clear.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6e105caf-ccbe-43ec-8063-aac9204f2db5
📒 Files selected for processing (1)
.github/workflows/dependabot-failure-watcher.yml
| # --created filters server-side, so --limit applies to runs already | ||
| # narrowed to the window rather than to all of history. Runs come back | ||
| # newest-first, so reaching the limit would drop the oldest in-window | ||
| # runs and this step would report all-clear without them -- hence a | ||
| # limit far above any plausible week's worth of runs. | ||
| runs=$(gh run list \ | ||
| --repo "$REPO" \ | ||
| --workflow "Dependabot Updates" \ | ||
| --limit 100 \ | ||
| --json conclusion,createdAt,displayTitle,url \ | ||
| --jq "[.[] | select((.conclusion == \"failure\" or .conclusion == \"startup_failure\" or .conclusion == \"timed_out\") and .createdAt >= \"$since\")]") | ||
| --created ">=$since" \ | ||
| --limit 500 \ | ||
| --json conclusion,createdAt,displayTitle,url) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
repo="${1:?usage: $0 OWNER/REPO}"
since="$(date -u -d '8 days ago' +%Y-%m-%dT%H:%M:%SZ)"
workflow_id="$(
gh api --paginate --slurp "repos/$repo/actions/workflows?per_page=100" |
jq -r '[.[] | .workflows[] | select(.name == "Dependabot Updates") | .id] | first // empty'
)"
count="$(
gh api --paginate --slurp \
"repos/$repo/actions/workflows/$workflow_id/runs?created=>=$since&exclude_pull_requests=true&per_page=100" |
jq '[.[] | .workflow_runs[]] | length'
)"
echo "Runs in window: $count"
test "$count" -le 500Repository: maxmind/ipfeed-draft
Length of output: 198
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "tracked workflow:"
git ls-files .github/workflows | grep -F 'dependabot-failure-watcher.yml' || true
echo
echo "relevant workflow excerpt:"
wc -l .github/workflows/dependabot-failure-watcher.yml
sed -n '1,180p' .github/workflows/dependabot-failure-watcher.yml | cat -n
echo
echo "github cli availability/version:"
(gh --version 2>&1 || true)
echo
echo "github CLI help snippets for run list workflow/created/limit:"
(gh run list --help | sed -n '/Workflow run/,/2>/p' || true)Repository: maxmind/ipfeed-draft
Length of output: 7270
🌐 Web query:
GitHub REST API list workflow runs created parameter per_page default 25 limit 1000
💡 Result:
For the GitHub REST API "List workflow runs" endpoints, the per_page parameter has a default value of 30, with a maximum allowed value of 100 per page [1][2][3]. While you can retrieve up to 1,000 total results when using specific filtering parameters (such as created, actor, or status), this limit applies to the total number of results accessible via pagination, not the per_page parameter itself [1][2][3]. To retrieve more than the per_page limit in a single request, you must iterate through subsequent pages using the Link header provided in the API response [4].
Citations:
- 1: https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2026-03-10
- 2: https://docs.github.com/en/enterprise-cloud@latest/rest/actions/workflow-runs?apiVersion=2022-11-28
- 3: https://docs.github.com/en/enterprise-server@3.19/rest/actions/workflow-runs?apiVersion=2022-11-28
- 4: https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api?apiVersion=2026-03-10
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "gh run list help:"
(gh run list --help | sed -n '/usage:/,/Options:/p' || true)
echo
echo "search for gh CLI invocation semantics around --limit/per-page:"
rg -n "run list|--limit|per page|workflow_runs|Workflow runs|created" --glob '*.md' . || true
echo
echo "repository workflow run counts in the default 8-day window:"
repo="$(gh repo view --json nameWithOwner -q .nameWithOwner 2>/dev/null || true)"
if [ -z "$repo" ]; then
echo "could not identify repository"
exit 0
fi
echo "repo=$repo"
workflow_id="$(
gh api --paginate --slurp "repos/$repo/actions/workflows?per_page=100" |
jq -r '[.[] | select(.name == "Dependabot Updates")] | .[0].id // empty'
)"
if [ -z "$workflow_id" ]; then
echo "Dependabot Updates workflow id not found"
fi
since="$(date -u -d '8 days ago' +%Y-%m-%dT%H:%M:%SZ)"
echo "since=$since"
echo "workflow_id=$workflow_id"
# Use pagination explicitly; per_page max is 100 for this endpoint.
total=0
first_count=0
latest=$(gh api "repos/$repo/actions/workflows/$workflow_id/runs?per_page=1&created=$since" 2>/dev/null | jq '.workflow_runs[0].id // empty' || true)
latest_name=$(gh api "repos/$repo/actions/workflows/$workflow_id/runs?per_page=1&created=$since" 2>/dev/null | jq '.workflow_runs[0].display_title // empty' || true)
for page in $(seq 1 12); do
page_runs="$(
gh api \
"repos/$repo/actions/workflows/$workflow_id/runs?perPage=100&created=$since&page=$page" \
2>/dev/null | jq '.workflow_runs | length'
)" || break
[ -n "$page_runs" ] || break
total=$((total + page_runs))
[ $page -eq 1 ] && first_count=$page_runs
[ "$page_runs" -lt 100 ] && break
done
echo "first_page_runs=$first_count"
echo "paginated_total_window_runs=$total"
echo "latest_id=$latest"
echo "latest_display_title=$latest_name"
# Simulate current gh limit behavior under a 100-per-request assumption and
# identify which failed in-window runs would be dropped with current/recommended limits.
python3 - <<'PY' "$workflow_id" "$since" "$total" "$first_count" "$latest"
import json, sys
# This probe only uses the counts fetched from the public API; it does not execute repo code.
workflow_id, since, total, first_count, latest = sys.argv[1:5]
print("probe_total", total)
print("dropped_above_current_500", max(0, total - 500))
print("dropped_above_suggested_1000", max(0, total - 1000))
PYRepository: maxmind/ipfeed-draft
Length of output: 661
Fail closed when the run scan is truncated.
Reaching the --limit discards older in-window runs because results are newest-first, but the script still exits successfully. Paginate through the action runs API and mark this window as truncated/failed, or keep the limit at the maximum supported window size and fail closed when truncated.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/dependabot-failure-watcher.yml around lines 88 - 98, The
run scan in the workflow currently treats the fixed --limit 500 result as
complete even when older in-window runs are omitted. Update the run-listing
logic around the runs command to paginate through all matching action runs, or
detect that the configured limit was reached and mark the scan as truncated so
the workflow fails closed instead of reporting all-clear.
GitHub runs both Dependabot version updates and Dependabot security
updates under one workflow name, "Dependabot Updates", and the watcher
counted both. Security updates routinely fail for reasons no pull request
can fix -- the advisory is against a dependency this project does not
declare directly, or no patched version is reachable. Left alone the
watcher stays red every week on those and trains everyone to ignore it.
Filter those runs out by title. Version updates are unaffected.
The title check is subtler than it looks, so document it properly. A
security job is marked by "/." AND a " for " suffix together; version
updates are either "/." with no " for " (the scheduled scan) or "/" with
one (the pull request). Both halves of " in /. for " are therefore
load-bearing -- matching on " in /." alone would discard every scan run,
which is most of the version-update runs and the shape the failures this
watcher was written for actually took.
That "/." spelling only separates the two at the repo root. In a
subdirectory a security update and a version update's pull request render
identically, so drop " in /e2e/{js,ts} for " by name as well. The Node
repos this workflow is shared with carry committed lockfiles under e2e/js
and e2e/ts, whose transitive dev dependencies attract advisories no pull
request can fix, and nothing in either is shipped code. That is 16
unactionable failures in each of GeoIP2-node and minfraud-api-node over
retained history; repos without those directories are unaffected.
Unlike the root filter, this one is not free. Both Node repos configure
npm with directories: ["/", "**/*"], and that glob does match e2e/js and
e2e/ts, so those directories do get version updates -- there is an open
version-update pull request under e2e/ts in both repos as this is
written. Dropping the pattern discards their pull-request refresh
failures along with the security jobs, and the ecosystem label is no help
because Dependabot writes "npm_and_yarn" for both. Taken anyway: the
scheduled scan is what this watcher primarily exists to catch and is
still reported for those directories, so what is given up is the narrower
"one open pull request has gone stale" signal for two directories of test
scaffolding. After filtering, 4 genuine failures remain reported in
GeoIP2-node and 3 in minfraud-api-node.
Reading the directories out of dependabot.yml would look more general and
was the earlier plan here, but it fails green. Entries may use globs, and
minfraud-api-dotnet's directories: ["**/*"] yields titles like "nuget in
/**/*" for the scan and "nuget in /MaxMind.MinFraud for
System.Net.Http.Json" for the pull request, neither of which any literal
comparison against the configured value matches -- so its two real nuget
failures would have been dropped without a word. A stale denylist
re-introduces noise, which is loud; a stale allowlist hides failures.
Name the three kinds of run in the comment while here, because the
scheduled scan and the per-pull-request refresh are easy to conflate: the
refresh runs are one per open pull request and are triggered by pushes to
the base branch or by rebases, not by the schedule, so they arrive in
bursts after merges. The scan is the kind this watcher primarily exists
to catch, which is what makes hiding a hypothetical refresh failure under
e2e an acceptable cost rather than a hole.
Bound the query server-side with --created instead of fetching all of
history and filtering by date locally, so --limit now caps an
already-narrowed window rather than standing in for one, and the run list
drops from several API pages to one. --limit rises 100 -> 500 as a
backstop: it still applies before the title filter, and reaching it would
silently drop the oldest in-window runs.
This workflow is shared verbatim across MaxMind repos. The change was
developed in maxmind/device-android and is applied here unmodified; see
that repo's commit for the measurements it was derived from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
31e0057 to
4312447
Compare
GitHub runs Dependabot version updates and Dependabot security updates under one
workflow name,
Dependabot Updates, and the watcher counted both. Securityupdates routinely fail for reasons no pull request can fix -- the advisory is
against a dependency the project does not declare directly, or no patched
version is reachable -- so the watcher stays red every week on those and trains
everyone to ignore it.
This filters security-update runs out by title, documents why both halves of
" in /. for "are load-bearing, and bounds thegh run listquery server-sidewith
--createdinstead of fetching all of history and filtering locally.See the commit message for the full reasoning, including why reading the
directory list out of
dependabot.ymlwas tried and rejected.This workflow is shared verbatim across MaxMind repos. The change was developed
in maxmind/device-android (maxmind/device-android#71)
and is applied here unmodified; the resulting file is byte-identical in every
repo.
Summary by CodeRabbit