Document measured update-status behavior from live cluster updates (single-node and 4-node)

[ddemlow]

Why

Follow-up to @wvancollenburg's review on #32. That PR corrected the update-status check to fail closed, but the guidance was still partly inferred. The review asked for two things this PR now delivers:

• "This needs verifying on a live cluster during an update."
• "Needs to be a brand new cluster without updates" — the no-update-history case.

Both were measured end to end across two real updates — a single-node cluster going 9.8.3 → 9.8.4 (30 min) and a 4-node cluster with running VMs going 9.6.30 → 9.6.32 (3.5 h) — sampled at a 5 s median interval (stretching to 20 s while correlation reads were timing out), from before each update started until it settled. Plus three never-updated clusters for the 404 case.

What was observed

1. "Effectively read-only" understated it — the API stops answering entirely, by hanging.

During apply, every /rest/v1/ endpoint tried (Cluster, Node, Drive, Condition, VirDomain, Update) returned a read timeout for minutes at a stretch. Reads fail too, and the failure mode is a hang: the connection is accepted and the backend never answers — not a refusal, not a 503. A client with no read timeout blocks indefinitely instead of getting an error it can handle.

update_status.json kept returning 200 the entire time, because the web server serves it as a static file independently of the REST backend. That is why it, and no REST endpoint, is the progress channel.

2. Both fields are mandatory — and now there's a reason, not just a convention.

phase	prepareStatus.state	updateStatus.masterState
never updated	(HTTP 404, HTML body)	(404)
prepare	DOWNLOAD BUNDLE → DOWNLOAD RPMS → UPDATE RPM	key absent entirely
apply	COMPLETE	EXECUTING ⇄ IN PROGRESS
settled	COMPLETE	COMPLETE

Each field alone reports "idle" through a whole phase. During prepare, updateStatus holds only percent and status — masterState doesn't exist, so .get() returns None, which is falsy. During apply, prepareStatus.state is already back to COMPLETE. masterState also has two in-progress values, so test != "COMPLETE" rather than matching a name.

And the original updateStage claim is confirmed dead: no top-level updateStage in any phase, in any sample.

2b. "Unreachable" is at least four different failure shapes, and none of them is a refused connection.

This is the finding most likely to change real client code. Across one update, the same client hitting the same cluster saw:

when	update_status.json	/rest/v1/*
never updated	404 + HTML	404
apply, backend busy	200 throughout	read timeout (accepted, never answered)
node tearing down to reboot	TLS error, ~100 s	TLS error
node fully down	read timeout, ~35 s	read timeout
back up, backend still starting	200	502 + HTML

The reboot alone moved through two shapes in sequence and took ~2m20s before the file answered again. Consequences:

• except Timeout is a bug here. requests.exceptions.SSLError subclasses ConnectionError, not Timeout — so a timeout-only handler lets the reboot through and the caller crashes. In curl that moment is exit 35 (SSL connect error), not 7; fully down becomes exit 28.
• Check the status code before parsing. The 404 and the 502 both return HTML, so r.json() raises — r.json().get("prepareStatus", {}) throws instead of failing closed.
• A 502 means "ask again later", never "idle" — it's the window where the front end is up but the REST backend hasn't finished starting.

3. A never-updated cluster returns 404 with an HTML body — so .json() raises; there is no empty-JSON case, and parsing before checking the status code throws. Seen on three never-updated clusters at three different builds (9.6.32, 9.7.8, 9.8.3), so it tracks never having updated, not the version.

4. Drive progress off percent/currentComponent — monotonic, no decreases in any sample. statusdetails is display text and at least one placeholder value recurs at several percentages.

5. /rest/v1/Condition is a trap for this. There's a first-class condition.updateInProgress flag, true during prepare — but it's a REST endpoint and dies during apply, exactly when needed. (Also documented: Condition returns the full catalogue of every possible condition on every call, each with a boolean value, so presence carries no information.)

6. POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag — no task to poll, so Rule 1's task-tag wait doesn't apply.

Multi-node changes the headline (added after the 4-node run)

The single-node run made the outage look total. It is per node. While the updating node was unreachable, its peers served both channels normally:

• 314 sample rounds had a healthy peer while another node was down; 2 had none.
• All four nodes followed the identical pattern, ~8–10 min of unreachability each.
• Nodes on the new version served alongside nodes still on the old one.

So the guidance is two-part, and neither half suffices: multi-endpoint failover for the per-node reboots that dominate the wall clock, and retry with backoff for the brief all-node window plus ordinary flakiness — healthy peers measured ~97–98% available, not 100%.

A hostname is one node, not the cluster. The cluster's DNS name failed and recovered in exactly the same sample rounds as one specific node IP, identical durations, across all three of that node's outages. A client configured with one hostname lost the cluster entirely while 3 of 4 nodes were healthy and serving; a client holding all four addresses never lost access. Rules 5 and 6 are the same problem, and both now say so.

Never build an endpoint list from networkStatus == "ONLINE". It is the obvious implementation and it selects the dead node: every peer reported the updating node as ONLINE / currentDisposition: IN while its API was completely unreachable, and it answered ICMP. Those fields describe cluster membership, not API reachability — correctly. Only a request tests an endpoint. Rule 6 now also cites Node.vips (empty, deprecated in the spec) rather than just asserting no VIP exists.

Cluster.icosVersion is the answering node's version. Mid-upgrade the same request returns different versions depending on which node serves it — a 2/2 split was observed directly. So version-gated feature detection is unreliable during an update; pin the answer for an operation rather than re-reading per call. It flips at that node's upgrade reboot, which also makes it a dependable per-node "done" signal.

Timing and progress: 3.5 h for 4 nodes with VMs vs 30 min for an empty single node, mostly VM migration (~15 min per node on top of a ~10 min reboot). totalComponents scales with cluster size (688 vs 180). percent is cluster-wide but not proportional to nodes completed — 51% at 1 of 4 — so don't scale it into an ETA. Multi-node prepare adds BEGIN and SYNC NODES.

Also

Qualifies the HyperCoreDynamicBalancer citation. It reads the correct two fields, does node failover, and is correct across the sequence above — but its guards are written if state and state != "COMPLETE", so an absent field reads as idle, and it can't distinguish a never-updated cluster from every node being unreachable mid-update. Both are now called out so the pattern isn't lifted unexamined.

Files

• CLAUDE.md — Rule 5 rewritten (tight version + phase table)
• docs/hypercore-api-field-notes.md — full detail
• docs/hypercore-api-reference.html — matching section, reusing the existing table.traps styling

Not observed — deliberately not documented

Both questions this section originally listed have since been answered by the 4-node run and are documented above:

• update_status.json does go away while its own node reboots — as a TLS error, then a read timeout.
• Nodes do not disagree about update progress; update_status.json is cluster-consistent. The divergence is in Cluster.icosVersion, which is per-node. That is the opposite of what I predicted.

Still genuinely unmeasured, and deliberately not documented: the cause of the single ~19 s all-node stall. One occurrence in one run — no mechanism is asserted for it.

🤖 Generated with Claude Code

[wvancollenburg]

Worth one sentence: a manually prepared, not-yet-applied update.

Normally prepare runs straight into update, but a prepare can be triggered on its own from the CLI. It's rare and deliberate, but if the apply is delayed or forgotten, the cluster can sit in that state for a while. I'd expect the file then to show prepareStatus.state: "COMPLETE" with masterState unknown, which the rule reads as busy even though REST is fully responsive. Busy is the right answer for a cluster mid-maintenance, so the rule doesn't need to change. But it's worth documenting, so that someone debugging automation that refuses to write to a healthy-looking cluster knows to check for a staged update. It would be worth it to check this state on one of the test clusters and see how the rules would apply