ci: add actionlint and a test-environment guard - #155
Merged
Conversation
Two guardrails for two silent failures that both happened here.
actionlint. A duplicate env: key in cmcp's release.yml made GitHub
refuse to load the workflow. It reported that as a "workflow file issue"
on unrelated pushes, the release event ran nothing, and a version sat
unpublished for hours. yaml.safe_load keeps the last value for a
duplicate key without complaining, so local validation passed.
actionlint reports it directly:
key "env" is duplicated in element of "steps" section
Fetched by pinned version and checksum-verified rather than run as a
third-party action, so a check that exists to guard the workflow supply
chain does not add to it. Runs only when .github/workflows changes.
Every repo is clean against it today.
Test-environment guard. pytest puts the source tree on the path, so an
in-process import always finds it. A test that shells out does not get
that: a plain subprocess resolves the distribution normally and, with a
released wheel also installed, finds site-packages. The subprocess then
exercises a published version while the suite reports a pass. That is
how a tutorial test graded against an old schema and looked green.
conftest probes a subprocess once per session and fails loudly if the
package resolves outside the repository. CI installs editable, so it is
a no-op there. A package not importable from a subprocess at all is
fine: that is a path-only setup, not a shadowing install.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XbDBXDWWvMFa7c2jGgyq9t
actionlint runs shellcheck over run: blocks on the CI runners, which my local Windows run could not, so these only appeared once the check was in CI. All six across the org are genuine; none was a false positive. SC2086, unquoted expansion in the cosign step: a digest is word-split before cosign sees it. Quoted. SC2162, read without -r in the file-validation loops: a backslash in a path is mangled rather than passed through. SC2155, export with command substitution in the keygen step: export masks the command's exit status, so an openssl failure produced an empty key and carried on. Declared and exported separately, so set -e can see the failure. SC2129, four separate appends to $GITHUB_STEP_SUMMARY: one redirection block instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XbDBXDWWvMFa7c2jGgyq9t
The conftest tripped this repo family's own gates: ruff format on the tuple layout, PLW1510 on subprocess.run without an explicit check, and E402 where the guard was appended below the module body. check=False is the honest value here, since a non-zero exit only means the package is not importable from a subprocess, which the caller already handles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XbDBXDWWvMFa7c2jGgyq9t
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two guardrails, each for a silent failure that actually happened during this security sweep.
1. actionlint
I introduced a duplicate
env:key in cmcp'srelease.ymlwhile hardening tag interpolation. GitHub Actions refuses to load a workflow with a duplicate key, reported it as a "workflow file issue" on unrelated pushes, ran nothing for the release event, and left a version unpublished for hours.It got through because
yaml.safe_loadkeeps the last value for a duplicate key without complaining. actionlint says it plainly:Verified against a reproduction of the exact broken file before adding it.
On how it's installed: fetched by pinned version with a checksum verification, not run as a third-party action. actionlint publishes no official action, and a check whose whole purpose is guarding the workflow supply chain should not add a new dependency to it.
Triggered only on changes under
.github/workflows, so it costs nothing on ordinary PRs. Every repo in the org is clean against it today, so this adds no backlog.2. Test-environment guard
pytest puts the source tree on the path, so an in-process import always finds this tree and looks right. A test that shells out gets no such help:
subprocess.run([sys.executable, ...])resolves the distribution normally, and with a released wheel also installed it finds site-packages.The subprocess then exercises a published version while the suite reports a pass. That is exactly how
trace-spec's tutorial test graded against the previous schema and still looked green, and whycmcp's distribution smoke test reported a version that had nothing to do with the tree.The guard probes a subprocess once per session and fails with an actionable message:
It is a no-op in CI, which installs editable, and verified as such against a correctly configured local venv. A package that is not importable from a subprocess at all is deliberately allowed: that is a path-only setup, not a shadowing install, so nobody gets locked out of running the suite.
Applied to the seven repos that install their own package editable in CI.
examples,integrationsanddemostest against third-party distributions on purpose and get actionlint only.🤖 Generated with Claude Code
https://claude.ai/code/session_01XbDBXDWWvMFa7c2jGgyq9t