From 5a87cefcb7bed929aa37778e5b976af6742aa23f Mon Sep 17 00:00:00 2001 From: David Demlow Date: Thu, 10 Sep 2026 13:25:21 -0400 Subject: [PATCH 1/4] Document measured update-status behavior from a live cluster update Follow-up to the review on #32. That PR corrected the update-status check to fail closed, but the guidance was still partly inferred: the review asked for verification on a live cluster during an update, and noted that a cluster with no update history is a separate case needing a brand-new cluster to observe. Both have now been measured end to end across a real 9.8.3 -> 9.8.4 update on a single-node cluster, sampled every 5 seconds from before the update started until it settled. The docs are updated to match what was observed: 1. "The API is effectively read-only" understated it. During apply, every /rest/v1/ endpoint tried (Cluster, Node, Drive, Condition, VirDomain, Update) returned a read timeout for minutes at a stretch -- reads fail too, and the failure is a HANG, not a refusal and not a 503: the connection is accepted and the backend never answers. So clients must set an explicit read timeout or they block indefinitely. update_status.json kept returning 200 throughout, because it is served as a static file independently of the REST backend -- which is why it, and no REST endpoint, is the progress channel. 2. Adds the observed state machine, and the reason both fields are mandatory. During prepare, updateStatus contains only percent and status -- masterState does not exist, so a masterState-only check reads None, which is falsy, and reports idle while packages are downloading. During apply, prepareStatus.state is already back to COMPLETE, so a prepareStatus-only check reports idle mid-update. masterState also has two in-progress values (EXECUTING and IN PROGRESS), so test != "COMPLETE" rather than matching a name. 3. A cluster that has never updated returns HTTP 404 with an HTML error body, so .json() raises -- there is no empty-JSON case, and parsing before checking the status code throws. Observed on three never-updated clusters running three different builds (9.6.32, 9.7.8, 9.8.3), so it tracks never having updated rather than the software version. 4. Progress should be driven off percent/currentComponent, which advanced monotonically with no decreases in any sample. statusdetails is display text and at least one placeholder value recurs at several percentages. 5. /rest/v1/Condition looks like a better answer -- there is a first-class condition.updateInProgress flag, true during prepare -- but it is a REST endpoint and stops answering during apply, exactly when it is needed. Also notes that Condition returns the full catalogue of every possible condition on every call, each with a boolean value, so presence carries no information. 6. POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to poll and Rule 1's task-tag wait does not apply. Also qualifies the HyperCoreDynamicBalancer citation. It reads the correct two fields and does node failover, and it is correct across the sequence above -- but its guards are written as `if state and state != "COMPLETE"`, so an absent field reads as idle, and it cannot distinguish a never-updated cluster from every node being unreachable mid-update. Both are now called out so the pattern is not lifted unexamined. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 46 ++++++++-- docs/hypercore-api-field-notes.md | 138 ++++++++++++++++++++++++++---- docs/hypercore-api-reference.html | 14 ++- 3 files changed, 170 insertions(+), 28 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7eeb3fe..59e5196 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,15 +70,43 @@ instead of disabling verification. ### 5. Check for an in-progress cluster update before writing -While SC//HyperCore software is self-updating, the REST API is effectively read-only and -mutating calls fail. Before a batch of writes, check -`GET https:///update/update_status.json` (note: **not** under -`/rest/v1/`; no auth required). The cluster is idle only when **both** -`prepareStatus.state` and `updateStatus.masterState` are `"COMPLETE"`. Treat any -other value, a missing field, an unparseable body, or an unreachable node as -**busy** — this check must fail closed, since nodes reboot during an update and -the file may not exist yet on a cluster that has never updated. Working -implementation: `specific_task/HyperCoreDynamicBalancer/HyperCore_balancer.py`. +While SC//HyperCore software is self-updating the REST API does not just go +read-only — it stops answering entirely, **by hanging**. Measured on a real +9.8.3 → 9.8.4 update: every `/rest/v1/` endpoint returned a read timeout for +minutes (connection accepted, no response — not a refusal, not a `503`), while +`GET https:///update/update_status.json` kept returning 200 the whole +time. So **always set an explicit read timeout**, and use that file — not any +REST endpoint — to decide whether it is safe to write. It is not under +`/rest/v1/` and needs no auth. + +The cluster is idle only when **both** `prepareStatus.state` and +`updateStatus.masterState` are `"COMPLETE"`. There is no top-level +`updateStage`. You must check both, because each one alone reports "idle" +through a whole phase of a real update: + +| phase | `prepareStatus.state` | `updateStatus.masterState` | +|---|---|---| +| never updated | *(HTTP 404, HTML body)* | *(404)* | +| prepare | `DOWNLOAD BUNDLE` → `DOWNLOAD RPMS` → `UPDATE RPM` | **absent** | +| apply | `COMPLETE` | `EXECUTING` ⇄ `IN PROGRESS` | +| settled | `COMPLETE` | `COMPLETE` | + +**Fail closed.** Treat any other value, a missing field, a 404, an unparseable +body, a timeout, or an unreachable node as **busy**. Two traps in particular: +a cluster that has never updated returns **404 with an HTML body**, so +`.json()` raises rather than giving you an empty object; and `masterState` has +*two* in-progress values, so test `!= "COMPLETE"` rather than matching a name. + +`/rest/v1/Condition` has a tempting `condition.updateInProgress` flag, but it is +a REST endpoint and dies with the rest of the API mid-update — don't rely on it. +`POST /rest/v1/Update/{uuid}/apply` returns 200 with an **empty** `taskTag`, so +there is no task to wait on (see Rule 1). + +Reference implementation: `specific_task/HyperCoreDynamicBalancer/HyperCore_balancer.py` +— correct across the sequence above and does node failover, but its +`if state and state != "COMPLETE"` guards read an absent field as idle, so +don't lift that pattern on its own. Full detail: +`docs/hypercore-api-field-notes.md`. ### 6. There is no cluster VIP diff --git a/docs/hypercore-api-field-notes.md b/docs/hypercore-api-field-notes.md index 8a59642..de3241a 100644 --- a/docs/hypercore-api-field-notes.md +++ b/docs/hypercore-api-field-notes.md @@ -118,15 +118,62 @@ bundle, which gets you real verification without a CA-signed cert. ### 5. Cluster update awareness -While SC//HyperCore software is self-updating, the REST API is effectively read-only — -mutating calls fail. Before any batch of writes, check: +While SC//HyperCore software is self-updating, **the REST API is not merely +read-only — it becomes entirely unavailable, and it fails by hanging.** + +The findings below were measured end to end across a real update on a +single-node cluster going from 9.8.3 to 9.8.4, sampling every 5 seconds from +before the update started until it settled. + +#### Use `update_status.json`, not `/rest/v1/` ``` GET https:///update/update_status.json ``` -Note this is **not** under `/rest/v1/` and requires no auth. The response is -shaped like this (verified on 9.8.3 and 9.8.4): +Not under `/rest/v1/`, and needs no auth. During the apply phase this file kept +returning 200 continuously while **every** `/rest/v1/` endpoint tried +(`Cluster`, `Node`, `Drive`, `Condition`, `VirDomain`, `Update`) returned a +**read timeout** for minutes at a stretch. The file is served as a static file +by the front-end web server, independently of the REST backend — which is why it +is the progress channel and no REST endpoint is. + +⚠ **Set an explicit read timeout on every call.** The observed failure was a +*hang*, not a refusal and not a `503`: the TCP connection is accepted and the +backend never answers. A client with no read timeout blocks indefinitely instead +of getting an error it can handle. In `requests` that means +`timeout=(connect, read)` — a bare number sets only the connect timeout in some +client libraries, and no timeout at all is never correct here. + +#### The state machine, and why you must check *both* fields + +| phase | `prepareStatus.state` | `updateStatus.masterState` | +|---|---|---| +| never updated | *(HTTP 404 — an HTML error page, not JSON)* | *(404)* | +| prepare | `DOWNLOAD BUNDLE` → `DOWNLOAD RPMS` → `UPDATE RPM` | **key absent entirely** | +| apply | `COMPLETE` | `EXECUTING` ⇄ `IN PROGRESS`, `percent` climbing | +| settled | `COMPLETE` | `COMPLETE` | + +The cluster is idle only when **both** `prepareStatus.state` and +`updateStatus.masterState` are `"COMPLETE"`. There is no top-level +`updateStage` field in any phase. + +**Checking only one of the two fields fails open for an entire phase of the +update:** + +- During **prepare**, `updateStatus` contains only `percent` and `status`. + `masterState` does not exist, so `.get("masterState")` returns `None` — which + is falsy, and a `masterState`-only check reports the cluster *idle* while it + is downloading and installing packages. +- During **apply**, `prepareStatus.state` has already returned to `"COMPLETE"`. + A `prepareStatus`-only check reports the cluster *idle* while the update is + halfway through. +- `masterState` alternates between **two** in-progress values, `"EXECUTING"` and + `"IN PROGRESS"`. Test `!= "COMPLETE"`; never match against a specific + in-progress name. + +A settled response looks like this — note that `masterState`, `toVersion` and +`fromBuild` appear only once the apply phase has begun: ```json { @@ -142,23 +189,78 @@ shaped like this (verified on 9.8.3 and 9.8.4): } ``` -The cluster is idle only when **both** `prepareStatus.state` and -`updateStatus.masterState` are `"COMPLETE"`. There is no top-level -`updateStage` field. +#### A cluster that has never updated returns 404, not empty JSON + +The file does not exist until the cluster's first update, and the web server +answers with **HTTP 404 and an HTML error body**. So `response.json()` raises a +JSON decode error — there is no "empty JSON" case to handle, and code that +parses before checking the status code will throw. This was observed on three +never-updated clusters running three different builds (9.6.32, 9.7.8 and +9.8.3), so it tracks *never having updated*, not the software version. + +Once an update is triggered the file appears within a few seconds, so the +window in which a genuinely updating cluster still returns 404 is short — but a +correct check must still treat 404 as *unknown*, not as *idle*. + +#### Fail closed -**Fail closed.** Treat every one of these as *busy*, not idle: +Treat every one of these as **busy**, not idle: - either state present and not `"COMPLETE"` -- either field absent (a cluster that has never updated may have no - `update_status.json` at all, and `.get()` returning `None` must not read as - "idle") -- a non-JSON body or an HTTP error -- the node unreachable — nodes reboot during an update, so a connection - failure is a likely *symptom* of one - -Because a mid-update node can be down, check the file on more than one node -before concluding the cluster is idle. `specific_task/HyperCoreDynamicBalancer/HyperCore_balancer.py` -implements this pattern, including node failover. +- either field absent — `.get()` returning `None` must never read as "idle" +- a 404, a non-JSON body, or any HTTP error +- the request timing out, or the node otherwise unreachable — nodes reboot + during an update, so a hang or connection failure is a likely *symptom* of the + very thing you are checking for + +Because a mid-update node can be down or hanging, check the file on more than +one node before concluding the cluster is idle. + +#### Progress: use `percent`, not the status text + +`percent` and `currentComponent` advanced monotonically across the whole run +(no decreases in any sample). The human-readable `status.statusdetails` is for +display only — it is not a progress indicator, and at least one placeholder +value (`"No-op"`) recurs at several different percentages. Don't drive logic +off it. + +#### Don't use `/rest/v1/Condition` to detect an update + +It looks like the right answer — there is a first-class +`condition.updateInProgress` flag, and it was `true` throughout the prepare +phase. But `Condition` is a REST endpoint, so it stops answering during apply +along with the rest of the API, exactly when you need it. `update_status.json` +is the only channel that survives the whole update. + +If you read `Condition` for other reasons, note that it returns the **full +catalogue of every possible condition on every call** — over 250 entries — each +with a boolean `value`. Presence in the list carries no information; filter on +`value` being true. Only a handful are typically active. + +#### Applying an update returns no task tag + +`POST /rest/v1/Update/{uuid}/apply` returns **HTTP 200 with an empty task +tag**: `{"taskTag": "", "createdUUID": ""}`. There is no task to poll, so the +usual "wait for the task tag" rule does not apply — `update_status.json` is the +only way to follow progress. The standard `if task_tag:` guard handles this +correctly because `""` is falsy, but don't block waiting for a tag that will +never arrive. `{uuid}` is the version string exactly as `GET /rest/v1/Update` +returns it, e.g. `9.8.4.227597`. + +#### Reference implementation, with one caveat + +`specific_task/HyperCoreDynamicBalancer/HyperCore_balancer.py` implements this +check with node failover, and reads the correct two fields. Two things in it are +worth understanding before you copy it: + +- Its guards are written as `if state and state != "COMPLETE"`, so an **absent** + field is falsy and reads as idle. It is nonetheless correct across the + sequence above, because at every point at least one of the two fields is + present and not `"COMPLETE"` — but the pattern is not safe on its own. +- When *every* node fails the check it concludes no update is running, which is + right for a never-updated cluster but cannot distinguish that from every node + being unreachable mid-update. If you need that distinction, treat + all-nodes-unreachable as busy. ### 6. No cluster VIP — plan for node failover diff --git a/docs/hypercore-api-reference.html b/docs/hypercore-api-reference.html index ad89f7f..b23d65f 100644 --- a/docs/hypercore-api-reference.html +++ b/docs/hypercore-api-reference.html @@ -412,7 +412,19 @@

The six rules that prevent 90% of bugs

5

Check for an in-progress update first

-

While SC//HyperCore software self-updates, the API is effectively read-only. Before a batch of writes, check GET /update/update_status.json (no auth, not under /rest/v1/). The cluster is idle only when both prepareStatus.state and updateStatus.masterState are COMPLETE — there is no top-level updateStage field. Fail closed: treat any other value, a missing field, or an unreachable node as busy, since nodes reboot mid-update and a cluster that has never updated may not serve the file at all.

+

While SC//HyperCore software self-updates the API does not merely go read-only — it stops answering entirely, by hanging. Measured across a real 9.8.3 → 9.8.4 update: every /rest/v1/ endpoint returned a read timeout for minutes (connection accepted, no response — not a refusal, not a 503), while GET /update/update_status.json kept returning 200 throughout. Always set an explicit read timeout, and use that file — no auth, not under /rest/v1/ — to decide whether it is safe to write.

+

The cluster is idle only when both prepareStatus.state and updateStatus.masterState are COMPLETE; there is no top-level updateStage. Both are required, because each one alone reports “idle” through a whole phase of a real update:

+
+ + + + + + + +
PhaseprepareStatus.stateupdateStatus.masterState
never updatedHTTP 404, HTML body404
prepareDOWNLOAD BUNDLE → DOWNLOAD RPMS → UPDATE RPMkey absent entirely
applyCOMPLETEEXECUTING ⇄ IN PROGRESS
settledCOMPLETECOMPLETE
+

Fail closed: treat any other value, a missing field, a 404, an unparseable body, a timeout, or an unreachable node as busy. Two specific traps: a cluster that has never updated returns 404 with an HTML body, so .json() raises rather than handing you an empty object; and masterState has two in-progress values, so test != "COMPLETE" instead of matching a name. Drive progress off percent/currentComponent, which advance monotonically — not off statusdetails, which is display text and repeats.

+

/rest/v1/Condition offers a tempting condition.updateInProgress flag, but it is a REST endpoint and dies with the rest of the API mid-update. And POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to wait on (Rule 1). Full detail in docs/hypercore-api-field-notes.md.

6

There is no cluster VIP

From f7386eaf93c116ac1ea4693bd695d590d22ced50 Mon Sep 17 00:00:00 2001 From: David Demlow Date: Thu, 10 Sep 2026 13:29:44 -0400 Subject: [PATCH 2/4] "Unreachable" is four different failure shapes, none of them a refused connection Watched the node reboot phase of the same update. The single most practically important finding, and it changes what client code has to catch. Across one update the same client hitting the same cluster saw four distinct failure shapes: never updated update_status.json -> HTTP 404 + HTML body apply, backend busy 200 on update_status.json, /rest/v1/* read timeout (connection accepted, backend never answers) node tearing down TLS error, ~100 s node fully down read timeout, ~35 s back up, backend starting 200 on update_status.json, /rest/v1/* HTTP 502 + HTML The reboot alone moved through TWO shapes in sequence -- TLS error then read timeout -- and took ~2m20s before the file answered again. At no point was the error a refused connection, which is the one most people code for. Three consequences now documented: 1. Catching only timeouts is a bug. requests.exceptions.SSLError subclasses ConnectionError, NOT Timeout, so `except Timeout` lets the reboot through and the caller crashes. Verified against the requests exception hierarchy. With curl the same moment is exit 35 (SSL connect error), not 7; once the host is fully down it becomes exit 28. 2. Check the status code before parsing. Both the 404 and the 502 return HTML, so r.json() raises a decode error instead of yielding something inspectable -- code shaped like r.json().get("prepareStatus", {}) throws rather than failing closed. 3. A 502 means "ask again later", never "idle". It occupies the window where the front-end web server is up but the REST backend has not finished starting -- a genuinely mid-update state. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 19 ++++++++++---- docs/hypercore-api-field-notes.md | 42 ++++++++++++++++++++++++++++--- docs/hypercore-api-reference.html | 3 ++- 3 files changed, 55 insertions(+), 9 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 59e5196..0ed2c82 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -91,11 +91,20 @@ through a whole phase of a real update: | apply | `COMPLETE` | `EXECUTING` ⇄ `IN PROGRESS` | | settled | `COMPLETE` | `COMPLETE` | -**Fail closed.** Treat any other value, a missing field, a 404, an unparseable -body, a timeout, or an unreachable node as **busy**. Two traps in particular: -a cluster that has never updated returns **404 with an HTML body**, so -`.json()` raises rather than giving you an empty object; and `masterState` has -*two* in-progress values, so test `!= "COMPLETE"` rather than matching a name. +**Fail closed.** Treat any other value, a missing field, a 404, a 502, an +unparseable body, a timeout, a TLS failure, or an unreachable node as **busy**. + +⚠ **"Unreachable" is at least four different things.** One update produced all +of these, and none was a refused connection: **404 + HTML** (never updated), +**read timeout** (apply phase — connection accepted, backend silent), **TLS +error** then **read timeout** (node tearing down, then fully down — ~2m20s +total), and **502 + HTML** (back up, backend still starting). So: catching only +`Timeout` is a bug — `requests.exceptions.SSLError` subclasses +`ConnectionError`, not `Timeout`, so the reboot escapes it (in `curl` that +moment is exit 35, not 7). And check the status code *before* parsing, because +the 404 and the 502 both return HTML — `r.json()` raises rather than failing +closed. `masterState` also has *two* in-progress values, so test +`!= "COMPLETE"` rather than matching a name. `/rest/v1/Condition` has a tempting `condition.updateInProgress` flag, but it is a REST endpoint and dies with the rest of the API mid-update — don't rely on it. diff --git a/docs/hypercore-api-field-notes.md b/docs/hypercore-api-field-notes.md index de3241a..80f1b60 100644 --- a/docs/hypercore-api-field-notes.md +++ b/docs/hypercore-api-field-notes.md @@ -202,15 +202,51 @@ Once an update is triggered the file appears within a few seconds, so the window in which a genuinely updating cluster still returns 404 is short — but a correct check must still treat 404 as *unknown*, not as *idle*. +#### "Unreachable" is at least four different things + +This is the single most important practical finding, and the reason the rule +below is written the way it is. Across one update the same client hitting the +same cluster saw **four distinct failure shapes**, and *none of them was a +refused connection*: + +| when | `/update/update_status.json` | `/rest/v1/*` | +|---|---|---| +| never updated | **HTTP 404**, HTML body | 404 | +| apply, backend busy | 200 throughout | **read timeout** (accepted, never answered) | +| node tearing down to reboot | **TLS error** (~100 s) | TLS error | +| node fully down | **read timeout** (~35 s) | read timeout | +| just back, backend still starting | 200 | **HTTP 502**, HTML body | + +The reboot alone moved through *two* shapes in sequence — TLS error, then read +timeout — and took roughly 2m20s end to end before the file answered again. + +Three consequences for real client code: + +- **Catching only timeouts is not enough.** In `requests`, the reboot raises + `SSLError`, which subclasses `ConnectionError` and **not** `Timeout` — so + `except requests.exceptions.Timeout` lets it through and the caller crashes + during the reboot. Catch `RequestException`, or at minimum both + `ConnectionError` and `Timeout`. With `curl`, the same moment is exit code + **35** (SSL connect error), not 7 (couldn't connect); once the host is fully + down it becomes exit **28** (timeout). +- **Check the status code before parsing.** Both the 404 and the 502 return + **HTML**, so `response.json()` raises a decode error rather than giving you + something to inspect. Code shaped like `r.json().get("prepareStatus", {})` + throws instead of failing closed. +- **A 502 means "ask again later", never "idle".** It appears in the window + where the front-end web server is up but the REST backend has not finished + starting — a genuinely mid-update state that a naive check reads as an error + it can ignore. + #### Fail closed Treat every one of these as **busy**, not idle: - either state present and not `"COMPLETE"` - either field absent — `.get()` returning `None` must never read as "idle" -- a 404, a non-JSON body, or any HTTP error -- the request timing out, or the node otherwise unreachable — nodes reboot - during an update, so a hang or connection failure is a likely *symptom* of the +- a 404, a 502, a non-JSON body, or any other HTTP error +- the request timing out, failing TLS, or the node otherwise unreachable — + nodes reboot during an update, so any of these is a likely *symptom* of the very thing you are checking for Because a mid-update node can be down or hanging, check the file on more than diff --git a/docs/hypercore-api-reference.html b/docs/hypercore-api-reference.html index b23d65f..a9bfdc7 100644 --- a/docs/hypercore-api-reference.html +++ b/docs/hypercore-api-reference.html @@ -423,7 +423,8 @@

The six rules that prevent 90% of bugs

settledCOMPLETECOMPLETE
-

Fail closed: treat any other value, a missing field, a 404, an unparseable body, a timeout, or an unreachable node as busy. Two specific traps: a cluster that has never updated returns 404 with an HTML body, so .json() raises rather than handing you an empty object; and masterState has two in-progress values, so test != "COMPLETE" instead of matching a name. Drive progress off percent/currentComponent, which advance monotonically — not off statusdetails, which is display text and repeats.

+

“Unreachable” is at least four different things, and one update produced all of them — none a refused connection: 404 + HTML (never updated), read timeout (apply phase: connection accepted, backend silent), TLS error then read timeout (node tearing down, then fully down — ~2m20s total), and 502 + HTML (back up, backend still starting). Two consequences: catching only Timeout is a bug, because requests.exceptions.SSLError subclasses ConnectionError and not Timeout, so the reboot escapes it (in curl that moment is exit 35, not 7); and check the status code before parsing, since the 404 and the 502 both return HTML and r.json() raises rather than failing closed.

+

Fail closed: treat any other value, a missing field, a 404, a 502, an unparseable body, a timeout, a TLS failure, or an unreachable node as busy. Two specific traps: a cluster that has never updated returns 404 with an HTML body, so .json() raises rather than handing you an empty object; and masterState has two in-progress values, so test != "COMPLETE" instead of matching a name. Drive progress off percent/currentComponent, which advance monotonically — not off statusdetails, which is display text and repeats.

/rest/v1/Condition offers a tempting condition.updateInProgress flag, but it is a REST endpoint and dies with the rest of the API mid-update. And POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to wait on (Rule 1). Full detail in docs/hypercore-api-field-notes.md.

From 0b4018f5d084d2051f87baf13323acd9a60cb725 Mon Sep 17 00:00:00 2001 From: David Demlow Date: Thu, 10 Sep 2026 13:38:22 -0400 Subject: [PATCH 3/4] Condition also lags after an update, not just dies during one The update-detection guidance said /rest/v1/Condition dies with the API during apply. It fails at the other end too: condition.updateInProgress was still true about 17 s after masterState had already gone COMPLETE, clearing roughly half a minute later. A client gating writes on it would keep refusing to write after the update had finished. Measured on the same run as the rest of this PR, which completed 9.8.3 -> 9.8.4 in just over 30 minutes. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 6 ++++-- docs/hypercore-api-field-notes.md | 14 +++++++++++--- docs/hypercore-api-reference.html | 2 +- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0ed2c82..d8b4d31 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,8 +106,10 @@ the 404 and the 502 both return HTML — `r.json()` raises rather than failing closed. `masterState` also has *two* in-progress values, so test `!= "COMPLETE"` rather than matching a name. -`/rest/v1/Condition` has a tempting `condition.updateInProgress` flag, but it is -a REST endpoint and dies with the rest of the API mid-update — don't rely on it. +`/rest/v1/Condition` has a tempting `condition.updateInProgress` flag, but it +fails at both ends: the endpoint dies with the rest of the API mid-update, and +the flag also *lags* on the way out (measured still `true` ~17 s after +`masterState` went `COMPLETE`). Don't rely on it. `POST /rest/v1/Update/{uuid}/apply` returns 200 with an **empty** `taskTag`, so there is no task to wait on (see Rule 1). diff --git a/docs/hypercore-api-field-notes.md b/docs/hypercore-api-field-notes.md index 80f1b60..c91484c 100644 --- a/docs/hypercore-api-field-notes.md +++ b/docs/hypercore-api-field-notes.md @@ -264,9 +264,17 @@ off it. It looks like the right answer — there is a first-class `condition.updateInProgress` flag, and it was `true` throughout the prepare -phase. But `Condition` is a REST endpoint, so it stops answering during apply -along with the rest of the API, exactly when you need it. `update_status.json` -is the only channel that survives the whole update. +phase. It fails you at both ends: + +- **It stops answering during apply**, along with the rest of the API, exactly + when you need it. +- **It lags on the way out.** The flag was still `true` about 17 seconds after + `masterState` had already gone `COMPLETE`, and cleared roughly half a minute + after that. So a client gating writes on it would keep refusing to write after + the update was already finished. + +`update_status.json` is the only channel that survives the whole update and +tracks it accurately at both edges. If you read `Condition` for other reasons, note that it returns the **full catalogue of every possible condition on every call** — over 250 entries — each diff --git a/docs/hypercore-api-reference.html b/docs/hypercore-api-reference.html index a9bfdc7..097b7eb 100644 --- a/docs/hypercore-api-reference.html +++ b/docs/hypercore-api-reference.html @@ -425,7 +425,7 @@

The six rules that prevent 90% of bugs

“Unreachable” is at least four different things, and one update produced all of them — none a refused connection: 404 + HTML (never updated), read timeout (apply phase: connection accepted, backend silent), TLS error then read timeout (node tearing down, then fully down — ~2m20s total), and 502 + HTML (back up, backend still starting). Two consequences: catching only Timeout is a bug, because requests.exceptions.SSLError subclasses ConnectionError and not Timeout, so the reboot escapes it (in curl that moment is exit 35, not 7); and check the status code before parsing, since the 404 and the 502 both return HTML and r.json() raises rather than failing closed.

Fail closed: treat any other value, a missing field, a 404, a 502, an unparseable body, a timeout, a TLS failure, or an unreachable node as busy. Two specific traps: a cluster that has never updated returns 404 with an HTML body, so .json() raises rather than handing you an empty object; and masterState has two in-progress values, so test != "COMPLETE" instead of matching a name. Drive progress off percent/currentComponent, which advance monotonically — not off statusdetails, which is display text and repeats.

-

/rest/v1/Condition offers a tempting condition.updateInProgress flag, but it is a REST endpoint and dies with the rest of the API mid-update. And POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to wait on (Rule 1). Full detail in docs/hypercore-api-field-notes.md.

+

/rest/v1/Condition offers a tempting condition.updateInProgress flag, but it fails at both ends: the endpoint dies with the rest of the API mid-update, and the flag also lags on the way out — measured still true ~17 s after masterState had gone COMPLETE. And POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to wait on (Rule 1). Full detail in docs/hypercore-api-field-notes.md.

6

There is no cluster VIP

From 861682004f11793bf4dec09366f57c495de2ac87 Mon Sep 17 00:00:00 2001 From: David Demlow Date: Thu, 10 Sep 2026 17:33:16 -0400 Subject: [PATCH 4/4] Multi-node: the update outage is per-node, and a hostname is one node Watched a 4-node cluster with running VMs upgrade 9.6.30 -> 9.6.32 (3.5 h), alongside the single-node 9.8.3 -> 9.8.4 run already in this PR (30 min). The 4-node run was done specifically to test whether the single-node total blackout was an artifact of having no peer to answer. It largely was. Scope correction to what this PR previously said: - "the REST API becomes entirely unavailable" -> "the updating NODE's REST API". On multi-node it is a rolling outage: one node at a time for ~8-10 minutes, peers serving both channels normally. 314 sample rounds had a healthy peer while another node was down; 2 had none. All four nodes followed the same pattern, and nodes on the new version served alongside nodes still on the old. - update_status.json is no longer described as simply surviving. It is more available than REST, not guaranteed: it goes away when its own node reboots, and one ~19 s window had it timing out on every node at once. So the guidance is two-part and neither half suffices alone: multi-endpoint failover for the per-node reboots that dominate an upgrade's wall clock, and retry with backoff for the brief all-node window plus ordinary flakiness (healthy peers measured ~97-98% available, not 100%). New, and none of it observable on one node: - A HOSTNAME IS ONE NODE. The cluster's DNS name failed and recovered in exactly the same sample rounds as one specific node IP, identical durations, across all three of that node's outages. A client configured with one hostname lost the cluster entirely while 3 of 4 nodes were healthy and serving; a client holding all four addresses never lost access. Rules 5 and 6 are therefore the same problem, and each now says so. - NEVER build an endpoint list from networkStatus == "ONLINE". It is the obvious implementation and it selects the dead node: during an update every peer reported the updating node as ONLINE / currentDisposition IN while its API was completely unreachable, and it answered ICMP. Those fields describe cluster membership, not API reachability. Only a request tests an endpoint. Rule 6 also now cites Node.vips (empty, deprecated in the spec) rather than just asserting no VIP exists. - Cluster.icosVersion is the ANSWERING node's version, so mid-upgrade the same request returns different versions depending on which node serves it (a 2/2 split was observed). Version-gated feature detection is therefore unreliable during an update -- pin the answer for an operation. It flips at that node's upgrade reboot, which also makes it a reliable per-node "done" signal. update_status.json by contrast is cluster-consistent, and updateStatus.status.node names the node under work in backplane addressing. - Timing and progress: 3.5 h for 4 nodes with VMs vs 30 min for an empty single node, mostly VM migration (~15 min per node on top of a ~10 min reboot). totalComponents scales with cluster size (688 vs 180), and percent is cluster-wide but NOT proportional to nodes completed -- 51% at 1 of 4 -- so don't scale it into an ETA. Multi-node prepare adds BEGIN and SYNC NODES. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 47 +++++++++--- docs/hypercore-api-field-notes.md | 116 +++++++++++++++++++++++++++--- docs/hypercore-api-reference.html | 8 ++- 3 files changed, 151 insertions(+), 20 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d8b4d31..1aa6f07 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,14 +70,22 @@ instead of disabling verification. ### 5. Check for an in-progress cluster update before writing -While SC//HyperCore software is self-updating the REST API does not just go -read-only — it stops answering entirely, **by hanging**. Measured on a real -9.8.3 → 9.8.4 update: every `/rest/v1/` endpoint returned a read timeout for -minutes (connection accepted, no response — not a refusal, not a `503`), while -`GET https:///update/update_status.json` kept returning 200 the whole -time. So **always set an explicit read timeout**, and use that file — not any -REST endpoint — to decide whether it is safe to write. It is not under -`/rest/v1/` and needs no auth. +While SC//HyperCore software is self-updating, **the updating node's** REST API +does not just go read-only — it stops answering entirely, **by hanging**. +Measured on real updates (single-node 9.8.3 → 9.8.4; 4-node 9.6.30 → 9.6.32): +every `/rest/v1/` endpoint on that node returned a read timeout for minutes +(connection accepted, no response — not a refusal, not a `503`). So **always +set an explicit read timeout**, and use +`GET https:///update/update_status.json` — not any REST endpoint — to +decide whether it is safe to write. It is not under `/rest/v1/` and needs no +auth. (It is *more* available than REST, not guaranteed: it goes away when its +own node reboots.) + +**On multi-node it is a ROLLING outage — one node at a time, ~8–10 min each, +peers serving normally** (measured 314 sample rounds with a healthy peer vs 2 +with none). So **multi-endpoint failover plus retry is what makes an update +survivable** — see Rule 6, which is really the same problem. A 4-node upgrade +took 3.5 h; the single-node one 30 min. The cluster is idle only when **both** `prepareStatus.state` and `updateStatus.masterState` are `"COMPLETE"`. There is no top-level @@ -110,6 +118,16 @@ closed. `masterState` also has *two* in-progress values, so test fails at both ends: the endpoint dies with the rest of the API mid-update, and the flag also *lags* on the way out (measured still `true` ~17 s after `masterState` went `COMPLETE`). Don't rely on it. + +⚠ **`Cluster.icosVersion` is the ANSWERING node's version.** Mid-upgrade the +same request returns different versions depending on which node serves it, so +**version-gated feature detection is unreliable during an update** — pin the +answer for an operation rather than re-reading per call. It flips at that +node's reboot, so it is a good per-node "done" signal. `updateStatus.status.node` +names the node currently being worked on (in *backplane* addressing), and +`update_status.json` itself is cluster-consistent. `percent` is cluster-wide but +**not** proportional to nodes completed (51% at 1 of 4) — don't scale it for an +ETA. `POST /rest/v1/Update/{uuid}/apply` returns 200 with an **empty** `taskTag`, so there is no task to wait on (see Rule 1). @@ -124,7 +142,18 @@ don't lift that pattern on its own. Full detail: Every API endpoint is a specific node's IP. If that node goes down, that endpoint is dead even though the cluster is fine. Discover all node IPs via `GET /rest/v1/Node` (`lanIP` field) and implement client-side failover across -them for anything long-running. +them, **with retry**, for anything long-running. (The only VIP-shaped field is +`Node.vips` — empty, and deprecated in the spec.) + +**This is the same problem as Rule 5**, and an update is when it bites: a +client configured with the cluster **hostname** lost access completely during a +rolling upgrade while 3 of 4 nodes were serving — the hostname resolves to one +node and dies with it. A client holding all four addresses never lost access. + +⚠ **Never build the endpoint list from `networkStatus == "ONLINE"`.** During an +update every peer reported the updating node as `ONLINE` / `currentDisposition: +IN` while its API was unreachable (it answered ICMP too). Those fields describe +cluster membership, not API reachability. **Only a request tests an endpoint.** ## Most common naming traps (full table in docs/hypercore-api-field-notes.md) diff --git a/docs/hypercore-api-field-notes.md b/docs/hypercore-api-field-notes.md index c91484c..57a8fd1 100644 --- a/docs/hypercore-api-field-notes.md +++ b/docs/hypercore-api-field-notes.md @@ -118,15 +118,25 @@ bundle, which gets you real verification without a CA-signed cert. ### 5. Cluster update awareness -While SC//HyperCore software is self-updating, **the REST API is not merely -read-only — it becomes entirely unavailable, and it fails by hanging.** +While SC//HyperCore software is self-updating, **the REST API on the node +being updated is not merely read-only — it becomes entirely unavailable, and it +fails by hanging.** -The findings below were measured end to end across a real update on a -single-node cluster going from 9.8.3 to 9.8.4, sampling every 5 seconds from -before the update started until it settled. +Measured end to end across two real updates, sampling every ~5 seconds from +before each update started until it settled: a **single-node** cluster going +9.8.3 → 9.8.4 (30 minutes), and a **4-node** cluster with running VMs going +9.6.30 → 9.6.32 (3.5 hours). + +⚠ **On a multi-node cluster the outage is per node, and peers keep serving** — +see "Multi-node: it's a rolling outage" below. That distinction is the +difference between an unusable API and a usable one, so read both parts before +designing a client. #### Use `update_status.json`, not `/rest/v1/` +(Everything in this subsection was measured on the single-node cluster, where +there is no peer to answer. The multi-node scoping follows further down.) + ``` GET https:///update/update_status.json ``` @@ -238,6 +248,69 @@ Three consequences for real client code: starting — a genuinely mid-update state that a naive check reads as an error it can ignore. +#### Multi-node: it's a rolling outage, and failover is most of the answer + +On the 4-node cluster, while the node being updated was unreachable, **its +peers served both channels normally**. Measured across the whole upgrade: + +- **314 sample rounds** had a healthy peer while another node was down. +- **2 sample rounds** had every node unreachable at once (one ~19-second + window, during the first VM evacuation, before any node had rebooted; it did + not recur). +- All four nodes followed the identical pattern, one at a time, with reboot + outages of **7.5–10 minutes each**. +- Nodes on the new version served happily alongside nodes still on the old one. + +So the guidance is two-part, and neither half substitutes for the other: + +1. **Multi-endpoint failover** handles the per-node reboots, which dominate an + upgrade's wall clock. A client holding every node's address never lost + access across the entire 3.5-hour upgrade. +2. **Retry with backoff** handles the rest — the brief all-node window, where + no other endpoint would have helped, and ordinary flakiness: peers not being + updated still returned scattered timeouts, roughly **97–98% availability** + rather than 100%. A single failed call to a healthy peer is expected. + +⚠ **A single hostname is not a cluster address — it is one node.** The +cluster's DNS name failed and recovered in *exactly* the same sample rounds as +one specific node IP, with identical outage durations, in all three of that +node's outages. **A client configured with one hostname lost the cluster +entirely while three of four nodes were healthy and serving.** Configure the +node addresses, not a name. See Rule 6. + +#### Timing, and what `percent` does and doesn't mean + +- The 4-node upgrade took **3.5 hours**; the empty single-node one took **30 + minutes**. Most of the difference is VM migration — evacuating and restoring + a node's VMs took ~15 minutes *per node*, on top of a ~10-minute reboot. +- `totalComponents` scales with cluster size (688 for four nodes, 180 for one). +- ⚠ **`percent` is cluster-wide but NOT proportional to nodes completed** — it + read 51% with only one of four nodes upgraded, because the shared prepare and + pre-flight phases account for a large share. Don't scale it into an ETA. +- Multi-node prepare has two extra states: `BEGIN` → `DOWNLOAD BUNDLE` → + `DOWNLOAD RPMS` → **`SYNC NODES`** → `UPDATE RPM` → `COMPLETE`. + +#### Which node is being updated, and which are done + +- **`updateStatus.status.node` names the node currently being worked on** — but + in *backplane* addressing, not the LAN address you connect to. It's the clean + signal for following the rollout. +- **`update_status.json` is cluster-consistent**: every node reports identical + `prepareStatus`, `masterState` and `percent`. Nodes do not disagree about + update progress. +- **But `Cluster.icosVersion` is the ANSWERING node's version, not the + cluster's.** Mid-upgrade, the same request returns different versions + depending on which node serves it — a 2/2 split was observed directly. It + flips at that node's upgrade reboot, so it *is* a reliable per-node "this one + is done" signal — and simultaneously a trap: + +⚠ **Version-gated feature detection is unreliable during an upgrade.** For the +hours an upgrade runs, a client asking "what version is this cluster?" gets an +answer that depends on which node answered, and that can change on retry. If +you gate behaviour on version, pin the answer for the operation rather than +re-reading it per call. (`Node.activeVersion` is `0` and is not a version +source.) + #### Fail closed Treat every one of these as **busy**, not idle: @@ -273,8 +346,11 @@ phase. It fails you at both ends: after that. So a client gating writes on it would keep refusing to write after the update was already finished. -`update_status.json` is the only channel that survives the whole update and -tracks it accurately at both edges. +`update_status.json` is the channel that tracks the update accurately at both +edges. ⚠ It is **not** guaranteed, though: it disappears when its own node +reboots, and on the 4-node run one ~19-second window had it timing out on +every node at once. Treat it as *more available* than any `/rest/v1/` +endpoint, never as always-up. If you read `Condition` for other reasons, note that it returns the **full catalogue of every possible condition on every call** — over 250 entries — each @@ -311,13 +387,35 @@ worth understanding before you copy it: SC//HyperCore clusters do not provide a floating/virtual IP for the REST API. Every endpoint is a specific node's address; the same API is served from every node, but if the node you configured goes down, that endpoint is dead -even though the cluster is healthy. For anything long-running: +even though the cluster is healthy. The only VIP-shaped field in the API is +`Node.vips`, which is empty and marked deprecated in the spec. + +**This and Rule 5 are the same problem.** A rolling software update takes each +node down in turn for several minutes, so an update is the most likely time a +single-endpoint client will fail — measured directly: a client configured with +the cluster's hostname lost access completely during an upgrade while three of +four nodes were serving, and a client holding all four node addresses never +lost access at all. + +For anything long-running: - Discover all node addresses via `GET /rest/v1/Node` (the `lanIP` field) -- Implement client-side failover across the node list +- Implement client-side failover across the node list, **with retry** — healthy + peers were ~97–98% available during an upgrade, not 100% +- **A hostname is one node, not the cluster.** Don't treat a DNS name as an + address for "the cluster"; it resolves to a single node and dies with it - Don't rely on DNS round-robin alone — most HTTP stacks pick one resolved IP per connection and won't automatically retry siblings +⚠ **Do not use `networkStatus` to decide whether a node's API is usable.** +`GET /rest/v1/Node` filtered on `networkStatus == "ONLINE"` is the obvious way +to build an endpoint list, and it is wrong: during an update, all peers +reported the node being updated as `ONLINE` with `currentDisposition: IN` +while that node's API was completely unreachable. It answered ICMP too. Those +fields describe **cluster membership** — backplane and storage health — not API +reachability, and they are correct to do so. **The only test of an endpoint is +a request to it.** + --- ## Auth diff --git a/docs/hypercore-api-reference.html b/docs/hypercore-api-reference.html index 097b7eb..5356ae0 100644 --- a/docs/hypercore-api-reference.html +++ b/docs/hypercore-api-reference.html @@ -412,7 +412,8 @@

The six rules that prevent 90% of bugs

5

Check for an in-progress update first

-

While SC//HyperCore software self-updates the API does not merely go read-only — it stops answering entirely, by hanging. Measured across a real 9.8.3 → 9.8.4 update: every /rest/v1/ endpoint returned a read timeout for minutes (connection accepted, no response — not a refusal, not a 503), while GET /update/update_status.json kept returning 200 throughout. Always set an explicit read timeout, and use that file — no auth, not under /rest/v1/ — to decide whether it is safe to write.

+

While SC//HyperCore software self-updates, the updating node's API does not merely go read-only — it stops answering entirely, by hanging. Measured across real updates (single-node 9.8.3 → 9.8.4; 4-node 9.6.30 → 9.6.32): every /rest/v1/ endpoint on that node returned a read timeout for minutes (connection accepted, no response — not a refusal, not a 503). Always set an explicit read timeout, and use GET /update/update_status.json — no auth, not under /rest/v1/ — to decide whether it is safe to write. It is more available than REST, not guaranteed: it goes away when its own node reboots.

+

On multi-node it is a ROLLING outage — one node at a time, ~8–10 min each, peers serving normally (314 sample rounds with a healthy peer vs 2 with none; a 4-node upgrade took 3.5 h, the single-node one 30 min). So multi-endpoint failover plus retry is what makes an update survivable — which makes this and Rule 6 the same problem. Measured: a client using the cluster hostname lost access entirely while 3 of 4 nodes served, because a hostname resolves to one node; a client holding all four addresses never lost access.

The cluster is idle only when both prepareStatus.state and updateStatus.masterState are COMPLETE; there is no top-level updateStage. Both are required, because each one alone reports “idle” through a whole phase of a real update:

@@ -425,11 +426,14 @@

The six rules that prevent 90% of bugs

PhaseprepareStatus.stateupdateStatus.masterState

“Unreachable” is at least four different things, and one update produced all of them — none a refused connection: 404 + HTML (never updated), read timeout (apply phase: connection accepted, backend silent), TLS error then read timeout (node tearing down, then fully down — ~2m20s total), and 502 + HTML (back up, backend still starting). Two consequences: catching only Timeout is a bug, because requests.exceptions.SSLError subclasses ConnectionError and not Timeout, so the reboot escapes it (in curl that moment is exit 35, not 7); and check the status code before parsing, since the 404 and the 502 both return HTML and r.json() raises rather than failing closed.

Fail closed: treat any other value, a missing field, a 404, a 502, an unparseable body, a timeout, a TLS failure, or an unreachable node as busy. Two specific traps: a cluster that has never updated returns 404 with an HTML body, so .json() raises rather than handing you an empty object; and masterState has two in-progress values, so test != "COMPLETE" instead of matching a name. Drive progress off percent/currentComponent, which advance monotonically — not off statusdetails, which is display text and repeats.

+

Cluster.icosVersion is the ANSWERING node's version, so mid-upgrade the same request returns different versions depending on which node serves it — version-gated feature detection is unreliable during an update; pin the answer for an operation rather than re-reading per call. It flips at that node's reboot, making it a good per-node “done” signal. updateStatus.status.node names the node under work (in backplane addressing), update_status.json is cluster-consistent, and percent is cluster-wide but not proportional to nodes completed (51% at 1 of 4) — don't scale it for an ETA.

/rest/v1/Condition offers a tempting condition.updateInProgress flag, but it fails at both ends: the endpoint dies with the rest of the API mid-update, and the flag also lags on the way out — measured still true ~17 s after masterState had gone COMPLETE. And POST /rest/v1/Update/{uuid}/apply returns 200 with an empty taskTag, so there is no task to wait on (Rule 1). Full detail in docs/hypercore-api-field-notes.md.

6

There is no cluster VIP

-

Every endpoint is a specific node's IP. If that node dies, the endpoint is dead even though the cluster is fine. Discover all nodes via GET /Node (lanIP) and fail over client-side.

+

Every endpoint is a specific node's IP. If that node dies, the endpoint is dead even though the cluster is fine. Discover all nodes via GET /Node (lanIP) and fail over client-side, with retry — healthy peers were ~97–98% available during an upgrade, not 100%. The only VIP-shaped field is Node.vips: empty, and deprecated in the spec.

+

This is the same problem as Rule 5, and an update is when it bites — see above. A hostname is one node, not the cluster.

+

⚠ Never build the endpoint list from networkStatus == "ONLINE". During an update every peer reported the updating node as ONLINE / currentDisposition: IN while its API was completely unreachable — it answered ICMP too. Those fields describe cluster membership, not API reachability, and are correct to do so. Only a request tests an endpoint.