fix(roles/system_update, roles/schedule_reboot): report only what a run actually did - #326
Draft
NavidSassan wants to merge 6 commits into
Draft
fix(roles/system_update, roles/schedule_reboot): report only what a run actually did#326NavidSassan wants to merge 6 commits into
NavidSassan wants to merge 6 commits into
Conversation
Member
Author
|
Waiting for the run in the test-vDC on 2026-08-28, merging after that |
NavidSassan
marked this pull request as draft
August 26, 2026 11:43
…pened update-and-reboot used to be queued by at(1) from the "updates are available" branch of notify-and-schedule, so it never ran on a week with nothing pending. Moving it onto its own timer in v8.0.0 made it run every update day, but its closing send_msg was left unconditional, so every host mailed "System updated without Reboot" weekly. The body made it worse: a bare `yum history info` prints the last transaction, so the mail described an earlier window, kernel installs included. Capture the dnf transaction id before and after the transaction, report only the one this run created, and stay quiet when there is none. The reboot checks still run either way, so a reboot carried over from an earlier window is picked up and its spool entry says that no packages changed instead of repeating an old transaction. Debian gates on the apt-get summary line. The security lane gets the same treatment. Its check-update gate already kept it from mailing on a quiet day, but it wrote a bare `yum history info` into the spool, so an upgrade that installed nothing after all handed do-reboot an unrelated transaction. On a day where both lanes run, that is the weekly lane's transaction from minutes earlier. Verified against dnf 4.7 (Rocky 8), 4.14 (Rocky 9) and 4.20 (Rocky 10), and apt 2.6 (Debian 12), 3.0 (Debian 13) and 2.7 (Ubuntu 24.04).
… when it was clean before Verified against aide 0.19.2 on Rocky 9.
The RedHat lane mails and exits when the transaction fails; the Debian lane checked neither apt-get update nor apt-get upgrade. A failing maintainer script returns 100, but the run continued into the reboot decision and the closing notification, so a broken transaction produced a "System updated without Reboot" mail or rebooted a half-configured host. Check both. The upgrade failure mail carries stdout and stderr, since apt puts the package names on one and `E: Sub-process ... returned an error code` on the other. apt-get update only reports a non-zero status for hard errors such as an unreadable sources list; a mirror that fails to answer produces a warning and exit 0, so its stderr keeps going to the journal, where an outage stays visible. Closing that gap needs a Debian counterpart to the RedHat retry loop and is not part of this change. Verified against apt 2.6 on Debian 12.
…ay name SENDER was built as "$SUBJECT_PREFIX <address>", so a subject prefix such as "[system_update] " put brackets into an unquoted display name. RFC 5322 permits only atoms there and brackets are specials, which makes the header invalid: a strict parser returns an empty address for it, not merely a mangled display name. Any consumer built on one sees mail with no sender at all. Hosts without a prefix were unaffected, which is why this went unnoticed. Quote the display name and escape an embedded backslash or double quote. While here, pass the bare address to `sendmail -f`, which is what an envelope sender is; Postfix extracted it from the full mailbox string anyway, but that relied on lenient parsing. Verified against Postfix 3.5.25 on Rocky 9
…e reboot reason
The AIDE re-baseline used the RHEL unit name, config default and gzipped
database throughout, so on Debian and Ubuntu it never ran: aide-common ships
dailyaidecheck.{service,timer}, keeps the database uncompressed, and its aide
binary has no default config file, which makes --config mandatory. All four are
now selected per family. Verified against aide 0.19.2 on Rocky 9 and aide 0.18.3
on Debian 12.
A reboot request carried the transaction report alone, so a reboot left over
from an earlier window arrived as nothing but "No packages were changed in this
run.". Both lanes now keep the needs-restarting output that asked for it.
Documents the AIDE gating and the abort-on-failure behaviour in the README, and
drops the Requirements bullet telling the admin to install needrestart via the
apps playbook: tasks/Debian.yml has always installed it. The pre_update_code
example called send_msg with globals it no longer takes, so it sent an empty
mail; it now passes subject and body. Both lanes log the same sentence for a
no-op run and use long grep options.
… fixes system_update runs the regular update lane twice with a reboot in between, so the second run lands on a host that is genuinely up to date: it has to report no change, request no reboot and send no "System updated without Reboot" mail. The journal is scoped with --boot rather than InvocationID, which systemd keeps in a cgroup xattr needing CAP_SYS_ADMIN and is therefore empty in some environments. schedule_reboot gets a bracketed mail subject prefix in its inventory and asserts the delivered notification carries a quoted From display name, which is what an unquoted RFC 5322 display name breaks.
NavidSassan
force-pushed
the
fix-system-update
branch
from
August 26, 2026 16:26
c192278 to
86f3303
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The weekly update mails were misreporting on every host. This started as a review of two weeks of
system_update/schedule_rebootmail from a ~24 host fleet, where 20 of 21 "System updated without Reboot" mails on one update day were false.What was wrong
1. A mail on every update day, describing an earlier run.
update-and-rebootused to be queued byat(1)from the "updates are available" branch ofnotify-and-schedule, so it never ran on a week with nothing pending. Moving it onto its own timer in v8.0.0 made it run every update day, but its closingsend_msgstayed unconditional. The body made it worse: a bareyum history infoprints the last transaction, so a quiet week produced one mail per host listing the previous week's packages, kernel installs included. A mail titled "System updated without Reboot" whose body was a kernel install that had required one.The security lane already had the right gate, but wrote a bare
yum history infointo the reboot spool, so an upgrade that installed nothing after all handeddo-rebootan unrelated transaction. On a day where both lanes run, that is the weekly lane's transaction from minutes earlier.2. AIDE was re-baselined at the wrong time, and unconditionally. The
aide --updateblock sat directly below the RedHat transaction while the Debian transaction runs further down, so on Debian the database was refreshed against the pre-upgrade filesystem and the nextaide-checkflagged every file the upgrade had touched. It has been that way since the block was introduced.It also ran unconditionally. The script already detects that
aide-check.servicewas failing beforehand, mails about it and saves the log, then re-baselined three lines later and restarted the unit, clearing the failed state and burying the very finding it had just reported.3. The Debian lane checked no return codes. A failing maintainer script returns 100, but the run continued into the reboot decision and the closing notification, so a broken transaction produced a success mail or rebooted a half-configured host.
4. The
Fromheader was invalid whenever a subject prefix is set.SENDERwas built as"$SUBJECT_PREFIX <address>", putting brackets into an unquoted display name. RFC 5322 permits only atoms there, so a strict parser returns an empty address, not merely a mangled display name. Any consumer built on one sees mail with no sender at all. Hosts without a prefix were unaffected, which is why it went unnoticed.What changed
Each commit is self-contained:
b09a243a6be246a4e977bb95apt-get updateandapt-get upgrade.c192278bFromdisplay name; pass the bare address tosendmail -f.The reboot checks still run when nothing was updated, so a reboot carried over from an earlier window is picked up either way; its spool entry then says that no packages changed instead of repeating an old transaction.
Behaviour changes worth noting
Not covered
apt-get updateexits 0 when a mirror merely fails to answer, printing aW: Failed to fetchwarning and carrying on with the old lists. It only reports a non-zero status for hard errors such as an unreadable sources list, so the new check catches a broken configuration, not a mirror outage. Its stderr keeps going to the journal, where an outage stays visible. Closing that gap needs a Debian counterpart tomakecache_with_retryand is deliberately not part of this PR.Verification
Templates rendered for RedHat and Debian, with and without
cache_only, pre/post hooks and Rocket.Chat, all passingbash -nandshellcheck;pre-commit run --filesclean.Behaviour was measured in containers rather than reasoned about:
yum -y updateagain exits 0 with "Nothing to do" and leaves the transaction id unchanged, while a bareyum history infostill prints the previous transaction in full. Rendered script run twice: one mail, not two. Carried-over reboot case: reboot still requested, spool entry readsNo packages were changed in this run.aide --checkexits 1 on a mismatch (aide 0.19.2, Rocky 9), which is what makesis-faileda valid signal. Full gating matrix exercised: clean+updates re-baselines, clean+no-updates does not, already-failing does not and leaves the unit failed with the pre-update log preserved, AIDE-not-managed is a no-op.postinstexits 1 makesapt-get upgradereturn 100 and the script exit 1 with a failure mail carrying both streams, and no reboot requested. An unreachable mirror returns 0 with onlyW:warnings; a malformed sources list returns 100.addr=''withInvalidHeaderDefect; the new form parses intact, including prefixes containing a double quote or a backslash. End to end through Postfix 3.5.25 on Rocky 9, reading the queued message directly.