Skip to content

Fix: derive the reported status from job-results - #1

Merged
jglanz merged 1 commit into
masterfrom
fix/report-real-job-status
Sep 8, 2026
Merged

Fix: derive the reported status from job-results#1
jglanz merged 1 commit into
masterfrom
fix/report-real-job-status

Conversation

@heifner

@heifner heifner commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

The status in the notification was never derived from job-results. The parser word-split the input and matched each word against ^([^:]+):([^:]+)$, so any comma-separated list arrived as a single multi-colon word, matched nothing, and produced {}. Every probe against {} is false, so the code fell through to its else branch and reported SUCCESS.

Both C++ build repos passed commas, so wire-sysio and wire-cdt have been posting ✅ SUCCESS over failing builds. wire-sysio run 34144933071 failed its gcc build leg and reported green; its log shows the parse landing on Result: {} and Overall: SUCCESS. Merge gates were never affected — the runs showed red on GitHub, only the chat message was wrong.

What changed

Commas are accepted alongside spaces and newlines, and the status derivation is inverted to fail closed: SUCCESS only when at least one entry parsed and every one is exactly success, anything unrecognised is UNKNOWN, and input that parses to nothing is a hard error. Under the old ordering any parsing defect degraded silently to green, which is why this went unnoticed for months.

Three further defects, all of which predate this PR:

  • JOB_DETAILS was accumulated in a while read on the right of a pipe, so every append landed in a subshell and was discarded. The per-job breakdown never rendered for anyone, including the callers whose single-pair input parsed correctly.
  • The documented JSON form never worked. Inputs were interpolated unquoted into a double-quoted shell assignment, which strips the inner quotes and leaves invalid JSON that fails exactly like the comma case. The same interpolation let a workflow name or branch name containing a quote corrupt the payload. Inputs now cross into the script through the environment, and the payload is assembled with jq --arg rather than string concatenation.
  • Errors were raised from inside a command substitution, where exit ends only the subshell and the ::error:: line is captured into the caller's variable instead of being logged. Failures surfaced as a step that died without stating a reason.

Structure

The implementation moves to scripts/notify.sh so it can be exercised directly; action.yaml now only maps inputs to the environment. NOTIFY_DRY_RUN=1 prints the payload and skips the webhook.

tests/run-tests.sh covers each separator form, the status precedence, the fail-closed paths, the rendered payload, and the webhook call itself against a local sink that rejects anything which is not well-formed JSON. Each of the four defects above was reintroduced individually to confirm the suite catches it rather than merely passing alongside it.

Compatibility

Every current caller's job-results value — wire-docs, wire-ethereum (x2), eth-validator-deposit-ui, wire-cdt (x2) and wire-sysio, comma-separated forms included — was replayed against the new script and reports what a correct implementation would. The comma callers are fixed by this PR alone, independently of the workflow changes in Wire-Network/wire-sysio#602 and Wire-Network/wire-cdt#116.

Callers pinned to @v1 pick this up only once the tag is moved.

README

Three uses: paths named a repository and workflow path that do not exist. notification-type: mattermost appeared in two examples where the code compares against 1 and 2. Both corrected, and the separators and fail-closed contract are now documented.

The status in the notification was never derived from `job-results`. The parser word-split the input and matched each word against `^([^:]+):([^:]+)$`, so any comma-separated list arrived as a single multi-colon word, matched nothing, and produced `{}`. Every probe against `{}` is false, so the code fell through to its `else` branch and reported SUCCESS. Both C++ build repos passed commas, so wire-sysio and wire-cdt posted `✅ SUCCESS` over failing builds — wire-sysio run 34144933071 failed its gcc leg and still reported green.

Accept commas alongside spaces and newlines, and invert the status derivation to fail closed: SUCCESS only when at least one entry parsed and every one is exactly `success`, anything unrecognised is UNKNOWN, and input that parses to nothing is a hard error. Under the old ordering any parsing defect degraded silently to green, which is why this went unnoticed.

Three further defects fixed:

`JOB_DETAILS` was accumulated in a `while read` on the right of a pipe, so every append landed in a subshell and was discarded. The per-job breakdown never rendered for anyone, including the callers whose single-pair input parsed correctly.

The documented JSON form never worked. Inputs were interpolated unquoted into a double-quoted shell assignment, which strips the inner quotes and leaves invalid JSON that failed the same way — and left workflow names and branch names able to break the payload or the script. Inputs now cross into the script through the environment, and the payload is assembled with `jq --arg` instead of string concatenation.

Errors were raised from inside a command substitution, where `exit` ends only the subshell and the `::error::` line is captured into the caller's variable rather than logged. Failures surfaced as a step that died with no stated reason.

The implementation moves to `scripts/notify.sh` so it can be tested; `action.yaml` now only maps inputs to the environment. `tests/run-tests.sh` covers every separator form, the status precedence, the fail-closed paths, the rendered payload, and the webhook call itself against a local sink; each of the four defects above fails the suite when reintroduced.

Existing callers are unaffected: every current `job-results` value, comma-separated included, was replayed against the new script and reports the same status a correct implementation would.

README: correct three `uses:` paths that named a repository and workflow path that do not exist, replace `notification-type: mattermost` with the `1` the code actually compares against, and document the separators and the fail-closed contract.
@jglanz
jglanz merged commit 3f39e7c into master Sep 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants