feat(os): the migration hold — chain services start only after the slot commits

docs/dev/appliance-release.md

@@ -120,14 +120,20 @@ PITHEAD_DATA_MIGRATION=true PITHEAD_MIN_OS_VERSION=X.Y.Z os/rauc/mkbundle.sh ...
 migration — normally this release's own version. `mkbundle.sh` refuses to build a
 migrating bundle without it.
 
-**Scoped follow-up — the migration runner.** Recording and enforcing the floor is done;
-*executing* the forward-only migration is not. The remaining half of the deadlock rule
-(`dual-distribution-plan.md` risk #6) is `pithead-boot` withholding the chain services
-(monerod/tari) until the slot commits on a `data_migration` update, so automatic A/B
-fallback stays pre-migration. Until that lands, the floor guard above prevents the *manual*
-rollback path from stranding data; the automatic-fallback ordering is bench (tier-4) work,
-asserted by `tests/os/run.sh --phase update`. The `db_schema` field the plan also lists
-lands with that runner — nothing reads it yet.
+**The migration hold — how a flagged update boots.** Installing a `data_migration` bundle
+also leaves a marker on `/data` (`.os-migration-pending`, stamped with the bundle's
+version). On the next boot, `pithead-boot` sees a marker matching its own version and
+brings the stack up **without** the chain services (monerod, tari, and their wallets — the
+holders of forward-only lmdb migrations): the A/B commit decision is made on everything
+else first. `doctor` reads the same marker and judges the deliberately-held chain
+containers by the sync-hold rule, so the commit gate gates on what is running instead of
+deadlocking on the hold. Only after `mark-good` does the boot path remove the marker and
+run a plain `up` — the chain services start, and the migration runs on a slot a fallback
+can no longer leave. If health fails instead, the slot stays uncommitted, the machine
+falls back, and the old OS boots normally — its data was never touched (the old boot path
+ignores a marker for a version it isn't). A non-migrating install clears any stale marker.
+The `db_schema` field the plan also lists stays out until something reads it — an
+unread manifest field is a claim, not a contract.
 
 ## Development loop
 

docs/dev/appliance-wizard.md

@@ -242,14 +242,13 @@ Five steps, each answering a hardware-validated failure:
    `hugepages_reserve_extra_mb` — RigForge's grow-only sysctl then sizes the shared pool as
    the single writer, and pithead's own HugePages write never shrinks a grown pool back.
 
-**Known gap at step 4 — forward-only data migrations.** Today `up` (step 3) starts the whole
-stack, monerod and tari included, *before* the commit at step 4. A release that runs a
-forward-only lmdb migration would therefore migrate `/data` before the slot commits, so a
-failed health check could leave the box unable to commit *or* fall back cleanly. The
-migration-deadlock rule (`dual-distribution-plan.md` risk #6) closes this by withholding the
-chain services until the slot commits on a `data_migration`-flagged update; that boot-path
-change is scoped follow-up. Until it lands, `pithead os-update` already refuses a *manual*
-rollback below the `/data` migration floor — see
+**Step 4 and forward-only data migrations.** On the first boot of a `data_migration`-flagged
+update, `up` (step 3) deliberately withholds the chain services — monerod, tari and their
+wallets, the holders of forward-only lmdb migrations — so the commit at step 4 is decided
+before any migration touches `/data`, and a failed health check still falls back onto data
+the old OS can read. The chain services start, and the migration runs, only after the slot
+commits. `pithead os-update` separately refuses a *manual* rollback below the `/data`
+migration floor — both halves are described in
 [`appliance-release.md`](appliance-release.md#compatibility-metadata-and-the-data-migration-floor).
 
 **Rule for changes:** anything generated from `config.json` or the program is derived and must

os/overlay/pithead-boot

@@ -60,7 +60,27 @@ fi
 ./pithead load-images
 
 ./pithead render || exit 1
-./pithead up || exit 1
+
+# The migration hold (#851): os-update leaves a marker when the installed bundle declares a
+# forward-only /data migration, stamped with that bundle's version. A marker matching THIS
+# slot's version means this is the migrating bundle's pre-commit boot — the chain services
+# (the holders of irreversible lmdb migrations) must not start until the commit decision is
+# made, so automatic A/B fallback always lands on data the old OS can still read. A marker
+# that does not match is a fallback boot (the migrating slot failed health): the migration
+# never ran, the data is untouched, and this boot proceeds normally. doctor reads the same
+# marker and judges the deliberately-held chain containers by the sync-hold rule, so the
+# commit gate below gates on everything that IS running instead of deadlocking on the hold.
+hold_chain=0
+if [ -f .os-migration-pending ] &&
+    [ "$(tr -d '[:space:]' <.os-migration-pending)" = "$(tr -d '[:space:]' <VERSION 2>/dev/null)" ]; then
+    hold_chain=1
+fi
+
+if [ "$hold_chain" = 1 ]; then
+    PITHEAD_HOLD_CHAIN=1 ./pithead up || exit 1
+else
+    ./pithead up || exit 1
+fi
 
 for _ in $(seq 90); do
     # localhost, not 127.0.0.1: the bare IP is not in the Caddyfile's site list, so it hits
@@ -74,6 +94,19 @@ for _ in $(seq 90); do
     if [ "${code:-000}" != "000" ] && ./pithead doctor --json >/dev/null 2>&1; then
         command -v rauc >/dev/null 2>&1 && rauc status mark-good
         echo "pithead-boot: stack is serving (HTTP $code) and doctor reports healthy — booted slot committed"
+        # Post-commit, the held chain services start and the forward migration runs NOW — the
+        # slot is committed, so a fallback can no longer strand migrated data. Marker removal
+        # comes first: if the chain start fails, the slot stays committed, the next boot takes
+        # the normal path and retries a plain `up` — re-holding would gate on a commit that has
+        # already happened. `|| true` for the same reason: a chain service that cannot start is
+        # a fault to alert on, never grounds to fail a boot the commit already accepted.
+        if [ "$hold_chain" = 1 ]; then
+            rm -f .os-migration-pending
+            # Logged BEFORE the up: the release line marks the commit boundary for the journal
+            # (and the tier-4 battery), and the up that starts the chain can take minutes.
+            echo "pithead-boot: slot committed — chain services released, the data migration runs now"
+            ./pithead up || true
+        fi
         # The on-box miner comes LAST, after the slot commit: it needs the stack's stratum
         # listening, and a miner that cannot start must never delay or block the commit — the
         # stack serving is the product's health, the miner is a passenger. Idempotent every

os/rootfs/Dockerfile

@@ -23,7 +23,10 @@ ENV DEBIAN_FRONTEND=noninteractive
 # program container networking, and pithead-install calls sgdisk (gdisk), partprobe (parted),
 # jq, lsblk, mkfs and the grub tools ON THE APPLIANCE — a tool the build host happens to have is
 # not a tool the image has. Each absence surfaces at runtime, so the set is audited against the
-# built image rather than added one failure at a time.
+# built image rather than added one failure at a time. squashfs-tools is in that set too:
+# `rauc info` extracts a bundle's manifest with unsquashfs, and without it `pithead os-update`
+# cannot read any bundle's compatibility metadata — every guard it carries was dead on the
+# appliance (found live on the bench, not by CI, whose stack tests stub rauc).
 RUN apt-get update && apt-get install -y --no-install-recommends \
         systemd systemd-sysv systemd-timesyncd systemd-resolved dbus udev \
         linux-image-amd64 initramfs-tools zstd \
@@ -33,6 +36,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
         fdisk gdisk parted e2fsprogs dosfstools \
         nftables iptables systemd-repart \
         avahi-daemon libnss-mdns \
+        squashfs-tools \
         openssh-server sudo less nano htpdate
 # NB: apt lists are kept on purpose — the Rugix core recipes run `apt-get install` in the build
 # chroot (fdisk/parted etc.) and need resolvable metadata; deleting them fails the bake.

pithead

@@ -230,7 +230,25 @@ stack_up() {
     # proceeds (signing is opt-in, #461), a present key that fails aborts before anything starts.
     verify_release_images
     # Docker Compose automatically picks up COMPOSE_PROFILES from .env
-    if ! compose_up_checked -d; then
+    #
+    # PITHEAD_HOLD_CHAIN=1 is the migration hold (#851), set only by the appliance boot path on
+    # the first boot of a data_migration bundle: bring up everything EXCEPT the chain services
+    # (monerod/tari and their wallets — the holders of forward-only lmdb migrations, and the
+    # only containers anything depends on together), so the A/B commit decision is made before
+    # any migration touches /data. The chain services start with a plain `up` after the commit.
+    if [ "${PITHEAD_HOLD_CHAIN:-0}" = 1 ]; then
+        local services c
+        local filter=()
+        for c in $REVENUE_CHAIN_CONTAINERS; do filter+=(-e "$c"); done
+        services=$(docker compose config --services 2>/dev/null)
+        [ -n "$services" ] || error "Could not list compose services for the chain hold."
+        log "Data migration pending — holding chain services ($REVENUE_CHAIN_CONTAINERS) until the slot commits."
+        services=$(printf '%s\n' "$services" | grep -vxF "${filter[@]}")
+        # shellcheck disable=SC2086 # word-splitting the service list is the point
+        if ! compose_up_checked -d $services; then
+            error "Stack failed to start — see the error above."
+        fi
+    elif ! compose_up_checked -d; then
         error "Stack failed to start — see the error above."
     fi
     log "Stack started successfully!"
@@ -2422,11 +2440,15 @@ os_running_variant() { # echoes debug|release|unknown
 }
 
 os_bundle_meta() { # $1: bundle, $2: [meta.pithead] key — echoes its value or empty, never fails
-    # `rauc info --output-format=json` exposes the manifest meta. Parsed defensively: find any
-    # object carrying a `pithead` child (so a RAUC that nests the manifest differently still
-    # resolves) and read the key out of it. Anything unparseable degrades to empty.
-    rauc info --output-format=json "$1" 2>/dev/null |
-        jq -r --arg k "$2" '[.. | objects | select(has("pithead")) | .pithead | objects | .[$k]? // empty] | map(select(. != "")) | first // empty' 2>/dev/null
+    # `rauc info --output-format=shell` is the one machine-readable format that carries the
+    # manifest's [meta.*] sections on the RAUC the appliance actually ships (1.11's JSON output
+    # OMITS them entirely — the json parser here read real bundles as unstamped, found by the
+    # tier-4 migration leg). Verification still runs: rauc info checks the bundle signature
+    # against the system keyring before printing anything. Anything unparseable degrades to
+    # empty, which every consumer treats as fail-closed "unstamped".
+    rauc info --output-format=shell "$1" 2>/dev/null |
+        sed -n "s/^RAUC_META_PITHEAD_$(printf '%s' "$2" | tr '[:lower:]' '[:upper:]')='\(.*\)'\$/\1/p" |
+        head -1
 }
 
 os_bundle_variant() { # $1: bundle path — echoes debug|release|unknown, never fails
@@ -2464,6 +2486,23 @@ os_raise_data_floor() { # $1: new floor — raise it, never lower (a migration o
     fi
 }
 
+# The migration-pending marker (#851): written by os-update when a data_migration bundle installs,
+# holding the version that bundle carries. On the next boot, pithead-boot and doctor read it back:
+# a marker matching the RUNNING version means "this boot must not start the chain services until
+# the slot commits" — the lmdb migration only runs once A/B fallback can no longer need the
+# pre-migration data. A marker that does NOT match the running version is a fallback boot (the
+# migrating slot failed health and the old OS is back); the old data was never touched, so it is
+# ignored. pithead-boot removes the marker once the migrating slot commits.
+os_migration_marker_file() { printf '%s' "${PITHEAD_MIGRATION_MARKER_FILE:-/data/pithead/.os-migration-pending}"; }
+
+os_migration_hold_active() { # rc 0 when this boot is the held, pre-commit boot of a migrating bundle
+    local f v
+    f=$(os_migration_marker_file)
+    [ -f "$f" ] || return 1
+    v=$(tr -d ' \t\r\n' <"$f" 2>/dev/null)
+    [ -n "$v" ] && [ "$v" = "$(os_running_version)" ]
+}
+
 os_update_needs_confirmation() { # $1: running variant, $2: bundle variant — rc 0 = confirm first
     # Consent is needed whenever an install flips the box's shell/SSH posture, in EITHER direction,
     # or when the bundle's posture can't be verified. A debug image bakes a standing root
@@ -2573,11 +2612,20 @@ os_update() {
     # A data_migration bundle raises the /data floor so a later rollback below it is refused. Armed
     # at install (conservative: before the migration actually runs on the new slot's next boot) —
     # the safe direction, and the only floor pithead can guarantee without executing the migration
-    # itself. Executing the forward-only migration + withholding chain services until the slot
-    # commits is the remaining half of the deadlock rule, tracked as follow-up (see appliance-release.md).
+    # itself. The marker beside it is the other half of the deadlock rule (#851): the next boot
+    # reads it and withholds the chain services until the slot commits, so automatic A/B fallback
+    # always lands on pre-migration data. A NON-migrating install clears any stale marker — it
+    # supersedes a migrating install that never booted, and its own boot must not hold anything.
+    local marker
+    marker=$(os_migration_marker_file)
     if [ "$bundle_migrates" = "true" ] && os_semver_ok "$bundle_min"; then
         os_raise_data_floor "$bundle_min"
         log "Recorded /data migration floor: OS >= $bundle_min required to read chain data after this update commits."
+        mkdir -p "$(dirname "$marker")" 2>/dev/null || true
+        printf '%s\n' "$bundle_version" >"$marker"
+        log "Marked a data migration pending: the next boot starts the chain services only after the new slot commits."
+    else
+        rm -f "$marker"
     fi
 }
 
@@ -3637,8 +3685,13 @@ REVENUE_MINER_CONTAINERS="p2pool xmrig-proxy"
 # Prints "ok" or "fail:<reason>". Kept pure — no engine calls — so the commit gate's honesty is
 # unit-testable without a running stack (tests/stack/run.sh). A non-revenue name is always "ok":
 # the rest of doctor covers those, and this must not judge containers outside its remit.
-revenue_container_verdict() { # <name> <state> <status>
-    local name="$1" state="$2" status="$3" running=0
+#
+# chain_hold=1 is the migration hold (#851): pithead-boot deliberately withholds the chain
+# containers until the slot commits, so — exactly like the miners' sync hold — a DOWN chain node
+# is then the expected state, not a crash, and only a running-but-unhealthy one is a fault.
+# Without this arm the commit gate would fail on the very hold it is gating, a deadlock.
+revenue_container_verdict() { # <name> <state> <status> [chain_hold]
+    local name="$1" state="$2" status="$3" chain_hold="${4:-0}" running=0
     # "running" from EITHER signal: podman and docker both print an "Up …" status for a live
     # container, and `.State` is "running". Reading both is belt-and-suspenders — some docker CLI
     # versions leave the `.State` ps field empty, and a chain node judged down on that alone would
@@ -3650,6 +3703,18 @@ revenue_container_verdict() { # <name> <state> <status>
     fi
     case " $REVENUE_CHAIN_CONTAINERS " in
     *" $name "*)
+        # Under the migration hold, judge a chain node by the miners' rule: down is deliberate.
+        if [ "$chain_hold" = 1 ]; then
+            case "$status" in
+            *'(unhealthy)'*)
+                if [ "$running" = 1 ]; then
+                    printf 'fail:%s is running but unhealthy (%s)\n' "$name" "$status"
+                else printf 'ok\n'; fi
+                ;;
+            *) printf 'ok\n' ;;
+            esac
+            return
+        fi
         # Chain node: it must be running and PAST its healthcheck. "starting"/"unhealthy" and any
         # stopped/exited/created state all mean not-ready-or-crashed. The boot gate loops, so a node
         # that is only starting simply gets retried rather than committed early.
@@ -3692,15 +3757,21 @@ revenue_container_verdict() { # <name> <state> <status>
 # honest gate: it rejects the healthy-looking-but-dead slot while staying quiet on the sync hold and
 # the long initial sync. Engine-aware (docker on DIY, podman on the appliance) and read-only.
 check_revenue_containers() {
-    local engine rows name state status verdict failed=0
+    local engine rows name state status verdict failed=0 chain_hold=0
     engine=$(container_engine)
     command -v "$engine" >/dev/null 2>&1 || return 0
     rows=$("$engine" ps -a --format '{{.Names}}\t{{.State}}\t{{.Status}}' 2>/dev/null) || return 0
     [ -n "$rows" ] || return 0
+    # The migration hold (#851): while pithead-boot withholds the chain services pre-commit,
+    # their absence is deliberate — the gate must not deadlock on the hold it is gating.
+    if os_migration_hold_active; then
+        chain_hold=1
+        dr_info "A data migration is pending — chain services are deliberately held until this slot commits."
+    fi
     # A here-string (not a pipe) keeps the loop in this shell, so `failed` survives it.
     while IFS=$'\t' read -r name state status; do
         [ -n "$name" ] || continue
-        verdict=$(revenue_container_verdict "$name" "$state" "$status")
+        verdict=$(revenue_container_verdict "$name" "$state" "$status" "$chain_hold")
         case "$verdict" in
         fail:*)
             dr_fail "${verdict#fail:}"

tests/os/README.md

@@ -37,7 +37,9 @@ runbook in [`docs/dev/release-server.md`](../../docs/dev/release-server.md).
   served, Tor-only egress actually enforced, built-in miner up. This is the phase that catches an
   appliance whose engine cannot run the product. Then the stack must return from a reboot with no
   hands on it, and the real commit gate — `pithead doctor --json` — must pass on that healthy
-  stack yet refuse once a revenue service is down.
+  stack yet refuse once a revenue service is down. The closing leg installs a `data_migration`
+  bundle through `pithead os-update` and proves the migration hold: the chain services stay down
+  until the slot commits, then start, with the pending marker consumed.
 - **rig** — answer `RigForge` on the same page and prove the other machine this image installs:
   it mines from the baked binary with no compile and no clearnet, starts no containers at all,
   and takes an A/B update — install, uncommitted rollback, self-commit, persistence — exactly