Skip to content

fix(roles/system_update, roles/schedule_reboot): report only what a run actually did - #326

Draft
NavidSassan wants to merge 6 commits into
mainfrom
fix-system-update
Draft

fix(roles/system_update, roles/schedule_reboot): report only what a run actually did#326
NavidSassan wants to merge 6 commits into
mainfrom
fix-system-update

Conversation

@NavidSassan

Copy link
Copy Markdown
Member

The weekly update mails were misreporting on every host. This started as a review of two weeks of system_update / schedule_reboot mail from a ~24 host fleet, where 20 of 21 "System updated without Reboot" mails on one update day were false.

What was wrong

1. A mail on every update day, describing an earlier run. update-and-reboot used to be queued by at(1) from the "updates are available" branch of notify-and-schedule, so it never ran on a week with nothing pending. Moving it onto its own timer in v8.0.0 made it run every update day, but its closing send_msg stayed unconditional. The body made it worse: a bare yum history info prints the last transaction, so a quiet week produced one mail per host listing the previous week's packages, kernel installs included. A mail titled "System updated without Reboot" whose body was a kernel install that had required one.

The security lane already had the right gate, but wrote a bare yum history info into the reboot spool, so an upgrade that installed nothing after all handed do-reboot an unrelated transaction. On a day where both lanes run, that is the weekly lane's transaction from minutes earlier.

2. AIDE was re-baselined at the wrong time, and unconditionally. The aide --update block sat directly below the RedHat transaction while the Debian transaction runs further down, so on Debian the database was refreshed against the pre-upgrade filesystem and the next aide-check flagged every file the upgrade had touched. It has been that way since the block was introduced.

It also ran unconditionally. The script already detects that aide-check.service was failing beforehand, mails about it and saves the log, then re-baselined three lines later and restarted the unit, clearing the failed state and burying the very finding it had just reported.

3. The Debian lane checked no return codes. A failing maintainer script returns 100, but the run continued into the reboot decision and the closing notification, so a broken transaction produced a success mail or rebooted a half-configured host.

4. The From header was invalid whenever a subject prefix is set. SENDER was built as "$SUBJECT_PREFIX <address>", putting brackets into an unquoted display name. RFC 5322 permits only atoms there, so a strict parser returns an empty address, not merely a mangled display name. Any consumer built on one sees mail with no sender at all. Hosts without a prefix were unaffected, which is why it went unnoticed.

What changed

Each commit is self-contained:

Commit Change
b09a243a Capture the dnf transaction id before and after the transaction, report only the one this run created, stay quiet when there is none. Same for the security lane.
6be246a4 Re-baseline AIDE after the transaction on both distributions, and only when the run changed packages and the check was passing beforehand.
e977bb95 Check both apt-get update and apt-get upgrade.
c192278b Quote the From display name; pass the bare address to sendmail -f.

The reboot checks still run when nothing was updated, so a reboot carried over from an earlier window is picked up either way; its spool entry then says that no packages changed instead of repeating an old transaction.

Behaviour changes worth noting

  • A host whose AIDE check was already failing keeps failing, instead of having the weekly update quietly clear it. That is the point of the change, but it is visible in monitoring.
  • Hosts with nothing to update go silent on their update day.

Not covered

apt-get update exits 0 when a mirror merely fails to answer, printing a W: Failed to fetch warning and carrying on with the old lists. It only reports a non-zero status for hard errors such as an unreadable sources list, so the new check catches a broken configuration, not a mirror outage. Its stderr keeps going to the journal, where an outage stays visible. Closing that gap needs a Debian counterpart to makecache_with_retry and is deliberately not part of this PR.

Verification

Templates rendered for RedHat and Debian, with and without cache_only, pre/post hooks and Rocket.Chat, all passing bash -n and shellcheck; pre-commit run --files clean.

Behaviour was measured in containers rather than reasoned about:

  • dnf, Rocky 9: an up-to-date host running yum -y update again exits 0 with "Nothing to do" and leaves the transaction id unchanged, while a bare yum history info still prints the previous transaction in full. Rendered script run twice: one mail, not two. Carried-over reboot case: reboot still requested, spool entry reads No packages were changed in this run.
  • AIDE: aide --check exits 1 on a mismatch (aide 0.19.2, Rocky 9), which is what makes is-failed a valid signal. Full gating matrix exercised: clean+updates re-baselines, clean+no-updates does not, already-failing does not and leaves the unit failed with the pre-update log preserved, AIDE-not-managed is a no-op.
  • apt, Debian 12: a package whose postinst exits 1 makes apt-get upgrade return 100 and the script exit 1 with a failure mail carrying both streams, and no reboot requested. An unreachable mirror returns 0 with only W: warnings; a malformed sources list returns 100.
  • Mail headers: parsed with a strict RFC 5322 parser. The old form yields addr='' with InvalidHeaderDefect; the new form parses intact, including prefixes containing a double quote or a backslash. End to end through Postfix 3.5.25 on Rocky 9, reading the queued message directly.

@NavidSassan

Copy link
Copy Markdown
Member Author

Waiting for the run in the test-vDC on 2026-08-28, merging after that

@NavidSassan
NavidSassan marked this pull request as draft August 26, 2026 11:43
…pened

update-and-reboot used to be queued by at(1) from the "updates are available"
branch of notify-and-schedule, so it never ran on a week with nothing pending.
Moving it onto its own timer in v8.0.0 made it run every update day, but its
closing send_msg was left unconditional, so every host mailed "System updated
without Reboot" weekly. The body made it worse: a bare `yum history info`
prints the last transaction, so the mail described an earlier window, kernel
installs included.

Capture the dnf transaction id before and after the transaction, report only
the one this run created, and stay quiet when there is none. The reboot checks
still run either way, so a reboot carried over from an earlier window is picked
up and its spool entry says that no packages changed instead of repeating an
old transaction. Debian gates on the apt-get summary line.

The security lane gets the same treatment. Its check-update gate already kept
it from mailing on a quiet day, but it wrote a bare `yum history info` into the
spool, so an upgrade that installed nothing after all handed do-reboot an
unrelated transaction. On a day where both lanes run, that is the weekly lane's
transaction from minutes earlier.

Verified against dnf 4.7 (Rocky 8), 4.14 (Rocky 9) and 4.20 (Rocky 10), and
apt 2.6 (Debian 12), 3.0 (Debian 13) and 2.7 (Ubuntu 24.04).
… when it was clean before

Verified against aide 0.19.2 on Rocky 9.
The RedHat lane mails and exits when the transaction fails; the Debian lane
checked neither apt-get update nor apt-get upgrade. A failing maintainer script
returns 100, but the run continued into the reboot decision and the closing
notification, so a broken transaction produced a "System updated without
Reboot" mail or rebooted a half-configured host.

Check both. The upgrade failure mail carries stdout and stderr, since apt puts
the package names on one and `E: Sub-process ... returned an error code` on the
other. apt-get update only reports a non-zero status for hard errors such as an
unreadable sources list; a mirror that fails to answer produces a warning and
exit 0, so its stderr keeps going to the journal, where an outage stays
visible. Closing that gap needs a Debian counterpart to the RedHat retry loop
and is not part of this change.

Verified against apt 2.6 on Debian 12.
…ay name

SENDER was built as "$SUBJECT_PREFIX <address>", so a subject prefix such as
"[system_update] " put brackets into an unquoted display name. RFC 5322 permits
only atoms there and brackets are specials, which makes the header invalid: a
strict parser returns an empty address for it, not merely a mangled display
name. Any consumer built on one sees mail with no sender at all. Hosts without
a prefix were unaffected, which is why this went unnoticed.

Quote the display name and escape an embedded backslash or double quote. While
here, pass the bare address to `sendmail -f`, which is what an envelope sender
is; Postfix extracted it from the full mailbox string anyway, but that relied
on lenient parsing.

Verified against Postfix 3.5.25 on Rocky 9
…e reboot reason

The AIDE re-baseline used the RHEL unit name, config default and gzipped
database throughout, so on Debian and Ubuntu it never ran: aide-common ships
dailyaidecheck.{service,timer}, keeps the database uncompressed, and its aide
binary has no default config file, which makes --config mandatory. All four are
now selected per family. Verified against aide 0.19.2 on Rocky 9 and aide 0.18.3
on Debian 12.

A reboot request carried the transaction report alone, so a reboot left over
from an earlier window arrived as nothing but "No packages were changed in this
run.". Both lanes now keep the needs-restarting output that asked for it.

Documents the AIDE gating and the abort-on-failure behaviour in the README, and
drops the Requirements bullet telling the admin to install needrestart via the
apps playbook: tasks/Debian.yml has always installed it. The pre_update_code
example called send_msg with globals it no longer takes, so it sent an empty
mail; it now passes subject and body. Both lanes log the same sentence for a
no-op run and use long grep options.
… fixes

system_update runs the regular update lane twice with a reboot in between, so
the second run lands on a host that is genuinely up to date: it has to report
no change, request no reboot and send no "System updated without Reboot" mail.
The journal is scoped with --boot rather than InvocationID, which systemd keeps
in a cgroup xattr needing CAP_SYS_ADMIN and is therefore empty in some
environments.

schedule_reboot gets a bracketed mail subject prefix in its inventory and
asserts the delivered notification carries a quoted From display name, which is
what an unquoted RFC 5322 display name breaks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant