The SSH alert rules are blind to saruman and morpheus — the two hosts where root takes a password

[Gerrrt]

Every rule in the authentication group of
stacks/observability/loki/rules/security.rules.yaml begins with the same
selector:

{log_type="authlog"}

Only two hosts ever produce that label. saruman and morpheus produce none,
so all five rules are structurally blind to them — and those are the two hosts
where root can be reached with a password.

Measured

2026-09-04, against Loki from oracle, running the rules' own selector:

sum by (host) (count_over_time({log_type="authlog"} |~ `Accepted (password|publickey|keyboard-interactive)` [7d]))
  oracle       110
  prometheus   258

sum by (host) (count_over_time({host=~".+"}          |~ `Accepted (password|publickey|keyboard-interactive)` [7d]))
  Saruman        1
  morpheus      43
  oracle       361
  prometheus   774

And directly:

{log_type="authlog", host=~"Saruman|morpheus"}   →  no results, 7 days

44 accepted SSH logins on those two hosts in a week, none of them visible to
SshBruteForce, SshBruteForceSevere, SshLoginFromUnexpectedSubnet,
SudoFailure or NewUserOrGroupCreated.

(The oracle/prometheus counts on the second query are inflated — those hosts
ship the same events by more than one route. The comparison that matters is
present-vs-absent, not the totals.)

Why each host misses

saruman — local.file_match "authlogs" in stacks/observability/alloy/config.alloy
hard-codes __path__ = "/var/log/auth.log", and loki.process "authlogs" is
the only thing that applies log_type = "authlog". saruman is Proxmox on
Debian 13 (node_uname_info → 7.0.14-12-pve), which is journald-only — there
is no /var/log/auth.log for that block to match. Its sshd lines do arrive,
via loki.source.journal, as log_type="journal", unit="ssh.service". So the
agent is working and the data is in Loki; only the label the rules key on is
absent.

morpheus — pfSense, not an Alloy host. It arrives through
loki.source.syslog "network" in syslog.alloy, which sets
log_type = "syslog" for everything it receives. Its sshd lines are in Loki
too, under that label.

So this is a labelling/selector mismatch, not a collection gap. Nothing is
missing from the store — the rules just cannot see it.

Why it is worth fixing rather than noting

The two invisible hosts are the worst two to lose:

• morpheus is the firewall, and it takes root by password
  (permitrootlogin yes, read on the host 2026-09-01). Its log shows
  Accepted keyboard-interactive/pam for root from two Hicks addresses on
  2026-09-02, so this is how the machine is actually used, not a latent setting.
• saruman accepted root by password from a Hicks address on 2026-09-02.

A password-guessing run against root@morpheus produces no SshBruteForce at
any volume. This is the same shape as #223 — a rule that reads correctly and
cannot fire — and part of the same family as #170, where half the rules had
nowhere to go.

Two possible fixes

1. Widen the rules. Replace {log_type="authlog"} with a selector that also
   admits saruman's journal and morpheus's syslog — e.g.
   {log_type=~"authlog|syslog"} plus the ssh.service journal stream. Cheapest
   change, but it widens what the line filters run against, so the
   SshLoginFromUnexpectedSubnet exclusions want re-checking against pfSense's
   line format.
2. Fix the labels at the shipper. Add a journal-side path that stamps
   log_type="authlog" on unit="ssh.service" (and the sudo/useradd lines
   the other two rules want), and match the relevant pfSense lines in
   loki.process "network_syslog", which already does per-line matching. More
   work, but it keeps the rules simple and makes the label mean one thing.

Either way scripts/check_loki_rules.sh should grow a case that fails when a
monitored host produces zero lines for a rule's selector, because that is the
failure mode here: the rules are valid, they load, and they match nothing.

Related

• Alloy is delivering ~1.5% of the host journal #194 — Alloy delivering ~1.5% of the host journal. Relevant if fix 1 or 2
  routes these through the journal: on saruman the delivered count matches
  what the agent reports reading (160/160 lines in 24h), so nothing is lost
  after the read, but hourly cron shows 24 entries for 2026-09-03 and one each
  for the two days before, so capture there is complete only from 2026-09-03.
  Worth settling before relying on journal-sourced auth alerting.
• TerminalSegmentReachedInternalNetwork cannot fire — the firewall logs blocks only #223 — TerminalSegmentReachedInternalNetwork cannot fire. Same class of
  defect.
• Decide whether Hicks should reach all of Winterfell and ImaginationLAN #228 — whether Hicks should reach all of Winterfell and ImaginationLAN.
  Both of the accepted root logins above came from Hicks addresses.

Found while working Lemmiwinks
#114; recorded in that
vault's systems/ssh_access §6, which is documentation of the gap and not a fix
for it.

🤖 Generated with Claude Code