Scheduled drift detection: weekly gates + monthly notebooks re-run against fresh resolves

[AlexanderFengler]

Part of the ecosystem self-healing rollout (spine PR lnccbrown/HSSMSpine#35 is the aggregation layer).

• new drift.yml: weekly (lint + fast + slow) and monthly (notebooks) scheduled re-runs of the existing gates via workflow_call, against a fresh PyPI resolve of unchanged main — with no committed lockfile this is the detector for upstream/toolchain releases breaking us (ruff 0.16 and pyrefly both did, silently, last month)
• failures create/update ONE deduped drift-labeled issue; green runs close it; force_fail dispatch input rehearses the path
• run_slow_tests.yml / check_notebooks.yml gain workflow_call:
• fixes riding along: SKIP_NOTEBOOKS bare-filename entries never matched (workshop notebooks ran while "skipped"); notebook run sharded 4-way (single job could exceed the 6 h limit); drift runs clear the --exitfirst/--reruns addopts; dead cache-hit guards removed from the setup-env composite actions

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added scheduled weekly quality checks and monthly notebook validation.
  • Added automated reporting for failed checks, including issue tracking and resolution updates.
  • Enabled notebook and slow-test workflows to be reused by other automation workflows.
  • Added manual controls for running notebooks and handling check failures.

• Improvements

  • Notebook checks now run across four parallel jobs for faster feedback.
  • Improved notebook selection and workshop exclusions for more reliable validation.
  • Installation steps now run consistently across environment setup workflows.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 94c8de21-f56f-4214-a8ae-83ae3370e7dc

📥 Commits

Reviewing files that changed from the base of the PR and between 1996f76 and b1bc75e.

📒 Files selected for processing (3)

• .github/setup-env-notebooks/action.yml
• .github/setup-env/action.yml
• .github/workflows/check_notebooks.yml

🚧 Files skipped from review as they are similar to previous changes (3)

• .github/setup-env-notebooks/action.yml
• .github/workflows/check_notebooks.yml
• .github/setup-env/action.yml

---

📝 Walkthrough

Walkthrough

The CI setup actions now install dependencies unconditionally. Notebook checks support four-way sharding. Slow tests and notebook checks support reusable workflow calls. A scheduled Drift workflow runs checks and reports failures through GitHub issues.

Changes

CI workflow orchestration

Layer / File(s)	Summary
Reusable checks and notebook sharding .github/setup-env-notebooks/action.yml, .github/setup-env/action.yml, .github/workflows/check_notebooks.yml, .github/workflows/run_slow_tests.yml	Dependency installation no longer checks a nonexistent cache step. Notebook checks run across four shards. Notebook and slow-test workflows support workflow_call.
Scheduled Drift checks and issue reporting .github/workflows/drift.yml	The Drift workflow runs scheduled or manually selected checks, aggregates results, and creates, comments on, or closes a deduplicated drift issue.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DriftWorkflow
  participant ReusableChecks
  participant ResultReporter
  participant GitHubIssues
  DriftWorkflow->>ReusableChecks: run configured CI checks
  ReusableChecks-->>ResultReporter: return job results
  ResultReporter->>GitHubIssues: close, comment on, or create drift issue

Loading

Possibly related PRs

• lnccbrown/HSSM#1135: Both changes update CI setup cache conditions and notebook-check configuration.

Suggested reviewers: krishnbera

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the scheduled drift detection workflows and their weekly and monthly test coverage.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch drift-detection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/setup-env-notebooks/action.yml:
- Around line 29-30: Update the stale-guard explanation comments at
.github/setup-env-notebooks/action.yml lines 29-30 and
.github/setup-env/action.yml lines 29-30 to state that the nonexistent cache
step/output reference evaluated false and skipped dependency installation,
rather than claiming it evaluated true. No workflow logic changes are needed.

In @.github/workflows/drift.yml:
- Around line 53-56: Add a job-level concurrency configuration to the report job
that uses a stable repository-specific group and sets cancel-in-progress to
false, ensuring concurrent drift issue lookup-and-create runs queue and execute
serially.
- Around line 75-93: Update the gate-result evaluation loop in the drift
workflow so the issue-closing branch runs only when every gate result is
success, not merely when FAILED_JOBS is empty. Treat cancelled results as
non-success and preserve any existing drift issue; retain the current failure
reporting and force_fail behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5139c8e9-df07-4f74-aa50-fbb9b7a827dc

📥 Commits

Reviewing files that changed from the base of the PR and between 8114267 and 064d1cf.

📒 Files selected for processing (5)

• .github/setup-env-notebooks/action.yml
• .github/setup-env/action.yml
• .github/workflows/check_notebooks.yml
• .github/workflows/drift.yml
• .github/workflows/run_slow_tests.yml

[AlexanderFengler]

Status and merge-order note.

The three red run_tests legs here are not caused by this PR — they're the numpy/numba resolution break tracked in #1144, which lands on any fresh resolve of unchanged main. #1137 is the fix; once that merges, a rerun here should go green. (My original attribution in #1144 was wrong, corrected there: it's numba, not pytensor, reached via a pre-release whose metadata omitted numpy's cap.)

Pushed since opening, from the review of the sibling drift workflows in ssm-simulators#318 and LANfactory#107 — the same report-job logic is shared across all five repos, so a bug found in one applied here too:

• A cancelled run no longer counts as green. report runs under if: always(), which includes cancellation, and the old logic only treated an exact failure as bad — so a cancelled weekly run would have closed an open drift issue with "green again" on no evidence. Closing now requires at least one genuine success and nothing cancelled. skipped stays neutral, which matters here specifically: the monthly notebooks job is skipped in every weekly run and must not block the close. Verified against all seven realistic result combinations.
• issues: write scoped to the report job, so the token that runs freshly-resolved third-party test code stays read-only.
• Serialized issue reconciliation via a job-level concurrency group — the gh issue list read and the create/close write aren't atomic.

This PR must not merge before #1142 without an update: that PR removes the test_args input this workflow passes, and gates run_slow_tests.yml behind a commit-message opt-in a scheduled run can't satisfy. Details and options are in my comment there. Plan: let #1137 and #1142 land first, then rebase this and adapt the fast and slow jobs in one commit.

[AlexanderFengler]

Correcting my earlier comment: this does not need to wait for #1142. I overstated it there, so to be precise about what the interaction actually is.

Today this PR is correct and self-consistent. Rebased onto main at 11195b39, all three run_tests legs are green, and every input it passes exists on main (run_tests.yml still declares both run_slow_tests and test_args). Nothing about merging this is blocked.

The #1142 interaction is a one-way, time-shifted obligation, not a merge dependency:

• Merging this first does not harm Drastically reduced number and run time of integration tests #1142 in any way — that PR merges just as cleanly either way.
• Merging Drastically reduced number and run time of integration tests #1142 later invalidates drift.yml, because it removes the test_args input the fast job passes and adds a commit-message gate to run_slow_tests.yml that a schedule event can never satisfy.
• So the requirement is simply: whoever merges Drastically reduced number and run time of integration tests #1142 should expect a one-commit follow-up here. I'll push it as soon as Drastically reduced number and run time of integration tests #1142 lands.

I looked at whether I could make this workflow immune to the ordering so no follow-up is needed at all. The clean version would be to drop test_args and call run_tests.yml with run_slow_tests: true instead of calling run_slow_tests.yml — both inputs exist before and after #1142. I decided against it for now, for two concrete reasons:

1. It would lose the batching. run_slow_tests.yml splits the suite into three parallel batches (Missing Data / Core MCMC / Remaining Slow) and the slowest takes ~47 min. Unbatched, against run_tests.yml's 90-minute timeout, that's a real timeout risk — and on main today run_slow_tests: true means all tests, not just slow ones, so it would be even longer.
2. --exitfirst. Dropping test_args gives up the -o addopts= override, so a drift run would stop at the first failure. For PRs that's right; for drift triage the useful signal is the full blast radius — which suites broke and how many tests — since that's what distinguishes "one upstream API changed" from "the whole stack moved".

Both of those objections disappear once #1142 lands, since it removes the batching and the test_args mechanism itself. So the follow-up commit is genuinely better done after that PR, not worked around now.

Worth knowing about the failure mode if the follow-up is ever missed: the weekly run would fail as a workflow configuration error, which files no issue — silent. The freshness budgets are the backstop: hssm-toolchain-lint and hssm-fast-tests go red after 10 days without a successful job, hssm-slow-suite after 17. So it surfaces, just late. That's the layered escalation working as designed, but the follow-up commit is obviously the better path.