Skip to content

feat(os): the migration hold — chain services start only after the slot commits - #883

Merged
VijitSingh97 merged 4 commits into
develop-v2from
fix/851-migration-runner
Aug 4, 2026
Merged

feat(os): the migration hold — chain services start only after the slot commits#883
VijitSingh97 merged 4 commits into
develop-v2from
fix/851-migration-runner

Conversation

@VijitSingh97

Copy link
Copy Markdown
Collaborator

Closes #851 (the automatic-fallback half; the floor guard shipped in #863)

What

A data_migration bundle's forward-only lmdb migration must not run before the A/B commit decision — otherwise a failed health gate leaves a box that can neither commit nor cleanly fall back (dual-distribution-plan.md risk #6). This lands the runner:

  • os-update leaves a version-stamped marker on /data (.os-migration-pending) when a flagged bundle installs; a non-migrating install clears a stale one (it supersedes a migrating install that never booted).
  • pithead-boot, on a boot whose version matches the marker, brings the stack up without the chain services — monerod, tari and their wallets, the lmdb holders and the only containers anything depends on together (PITHEAD_HOLD_CHAIN=1 subsets the compose service list). A mismatched marker is a fallback boot onto untouched data and holds nothing.
  • doctor reads the same marker and judges the deliberately-held chain containers by the miners' sync-hold rule — the commit gate gates on everything that IS running instead of deadlocking on the very hold it gates. A running-but-unhealthy chain container still fails.
  • Post-commit: marker consumed, release line logged (the journal's commit boundary), then a plain up starts the chain — the migration runs on a slot a fallback can no longer leave. If the chain start fails, the slot stays committed and the next (normal-path) boot retries a plain up.

Found live on the bench, not by CI

Deploying today's develop-v2 tip through pithead os-update failed outright: the appliance image ships no unsquashfs, so rauc info could not extract any bundle's manifest — every guard #863 added was dead on the actual appliance (CI's stack tests stub rauc, so they can't see it). squashfs-tools joins the rootfs shell-out set, with the lesson recorded in the Dockerfile comment.

Tests

  • Stack suite (owning tier for the logic): marker lifecycle through the real os_update (written with the bundle version; cleared by a plain install), the os_migration_hold_active query (match / mismatch / absent / empty), the verdict's chain-hold arm (down→ok, running-unhealthy→still fail, miners unaffected), and a black-box pithead up under the hold asserting the compose invocation starts tor/p2pool/dashboard and none of the four chain services. 2136 passed, 0 failed; make lint green.
  • Tier-4: the KVM battery's provision phase grows a migration leg — build a flagged bundle, install through the real os-update on the guest (exercising unsquashfs for real), and prove from the boot journal plus a podman poll that monerod never runs pre-commit, the release line lands, monerod starts after it, and the marker is consumed. This leg has not executed yet — it runs on gouda only; I'm running the battery next and will report before merging.

The db_schema manifest field the plan lists stays out until something reads it — an unread field is a claim, not a contract (doc updated accordingly).

Adversarially verified in a fresh context: fallback-path traces (gate-never-passes, crash between mark-good and marker removal — self-healing re-hold on an already-good slot), subset-grep exactness both directions, no PITHEAD_HOLD_CHAIN leak into operator commands, doctor deadlock check.

🤖 Generated with Claude Code

VijitSingh97 and others added 2 commits August 3, 2026 23:52
…ot commits

The deadlock rule's automatic-fallback half (the floor guard shipped
earlier): a data_migration bundle's forward-only lmdb migration must not
run before the A/B commit decision, or a failed health check leaves a
box that can neither commit nor cleanly fall back.

os-update now leaves a version-stamped marker on /data when a flagged
bundle installs (and clears a stale one on a plain install). On the next
boot, pithead-boot sees a marker matching its own version and brings the
stack up without the chain services — monerod, tari and their wallets,
the lmdb holders and the only containers anything depends on together —
via PITHEAD_HOLD_CHAIN=1, which subsets the compose service list. doctor
reads the same marker and judges the deliberately-held chain containers
by the miners' sync-hold rule, so the commit gate gates on everything
that IS running instead of deadlocking on the hold it gates. Post-commit
the marker is consumed and a plain up starts the chain: the migration
runs on a slot a fallback can no longer leave. A mismatched marker is a
fallback boot onto untouched data and holds nothing.

Found live while deploying: the appliance image ships no unsquashfs, so
rauc info could not read any bundle's manifest and every os-update guard
was dead on the appliance — CI never saw it because the stack tests stub
rauc. squashfs-tools joins the rootfs shell-out set.

The KVM battery's provision phase grows a migration leg: build a flagged
bundle, install through the real os-update, and prove from the boot
journal and a podman poll that monerod never runs pre-commit, starts
post-release, and the marker is consumed. The db_schema manifest field
stays out until something reads it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first run of the leg failed exactly where the output was discarded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s it

The tier-4 migration leg's first live run refuted os_bundle_meta: RAUC
1.11's --output-format=json omits the manifest's [meta.*] sections
entirely, so every real bundle read as unstamped and os-update refused
it as a possible downgrade. The stack tests passed because their
fixtures were synthetic JSON in a shape real rauc never emits — the
second dead layer under the missing-unsquashfs one, and exactly why the
leg installs through the real os-update.

--output-format=shell carries the meta (RAUC_META_PITHEAD_*) and still
verifies the bundle signature before printing. Fixtures now mirror the
real format, and both os-update harnesses sandbox the migration-marker
path so a test run can never touch a real /data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@VijitSingh97

Copy link
Copy Markdown
Collaborator Author

Tier-4 evidence is in — the migration leg ran on the KVM bench (gouda) against the real image:

✓ os-update left the migration-pending marker (1.17.0)
✓ the migrating slot committed and released the chain services
✓ monerod never ran while the slot was uncommitted
✓ boot journal shows the chain hold
✓ monerod is running again post-commit (the migration window is over)
✓ the migration-pending marker was consumed

The leg's first two runs each refuted a layer of os-update on real hardware — the missing unsquashfs, then RAUC 1.11's JSON output omitting [meta.*] entirely (both fixed in this PR; os_bundle_meta now parses the shell format, which still signature-verifies). The phase's one remaining ✗ is the pre-existing egress-leg timing gap, filed as #887 — unrelated to this change.

@VijitSingh97
VijitSingh97 merged commit c29eb10 into develop-v2 Aug 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant