Problem
The e2e-test.yml workflow runs on a weekly schedule (cron: "0 6 * * 1", Mondays 06:00 UTC) in addition to PR/merge triggers. When the scheduled run fails, no failure notification is delivered.
This was discovered while working on #95: the DigitalOcean e2e job had been failing for some time (an expired E2E_DO_TOKEN), but no failure emails were received, so the breakage went unnoticed until it surfaced on an unrelated PR's checks.
Why it happens
GitHub only emails scheduled-workflow failures to the account that last modified the workflow file, and it stops sending after repeated failures on the same schedule. So the missing emails are expected GitHub behavior, not a separate bug — but it means a broken weekly run can silently rot.
Suggested fix
Add an explicit failure-notification step to the scheduled path of e2e-test.yml so breakage is surfaced reliably. Options:
- A notification step gated on
if: failure() && github.event_name == 'schedule' (Slack webhook, email, or gh issue create to auto-open a tracking issue).
- Alternatively/additionally, a lightweight monitor that alerts if the weekly run hasn't succeeded recently.
Related
Notes
Credentials rotate/expire periodically (DO token, Vultr API key, GCP creds), so this will recur without proactive alerting.
Problem
The
e2e-test.ymlworkflow runs on a weekly schedule (cron: "0 6 * * 1", Mondays 06:00 UTC) in addition to PR/merge triggers. When the scheduled run fails, no failure notification is delivered.This was discovered while working on #95: the DigitalOcean e2e job had been failing for some time (an expired
E2E_DO_TOKEN), but no failure emails were received, so the breakage went unnoticed until it surfaced on an unrelated PR's checks.Why it happens
GitHub only emails scheduled-workflow failures to the account that last modified the workflow file, and it stops sending after repeated failures on the same schedule. So the missing emails are expected GitHub behavior, not a separate bug — but it means a broken weekly run can silently rot.
Suggested fix
Add an explicit failure-notification step to the scheduled path of
e2e-test.ymlso breakage is surfaced reliably. Options:if: failure() && github.event_name == 'schedule'(Slack webhook, email, orgh issue createto auto-open a tracking issue).Related
E2E_DO_TOKEN) has been resolved by rotating the token; this issue tracks the missing-notification gap so the next expiry/failure doesn't go unnoticed.Notes
Credentials rotate/expire periodically (DO token, Vultr API key, GCP creds), so this will recur without proactive alerting.