Parity microvm - #703
Merged
Merged
Conversation
…x gRPC tunnel, its pool/manager/workspace layer and the backend built on it, keeping only the worker-id encoding and client helpers lite still needs
…workspace export/import, PTY and exec-session streaming, and sealed-secret application, so the control plane reaches the guest over plain JSON instead of a brokered tunnel
…TTP calls to the guest's /oc surface plus tar-based checkpoint/restore through the blob store, which is the only checkpoint tier this runtime can offer
…essions, PTY and secret transport — the manager-side half of the surfaces the QEMU runtime already offered
…nd exec-session splices that translate the guest's framed protocol to the binary contract the SDK expects, plus an audit test pinning the recurring bug where a handler reaches for the worker registry without asking who holds the sandbox
…e so a sandbox row expires with its host instead of leaking, and classify a genuinely-absent host as terminal so reconcile can close it
…op the zombie-tick guard dropping it, so a short sandbox is charged for the time it actually ran
…secret rotation through the backend seam, which previously answered 200 with refreshed=0 while the box kept serving the old value
…host rather than suspending it, because suspended time still counts against the provider's lifetime cap and a credit halt outlives it
…global HTTP/2 agent stops breaking its own upgrades, and wait for the socket to open before returning a PTY session
breardon2011
force-pushed
the
parity-microvm
branch
from
September 2, 2026 01:40
f1b851f to
41b7905
Compare
Merged
…this runtime has no CPU knob (RunMicrovmInput carries no vCPU field), so carrying the customer's number into the usage record was the CPU form of the silent-wrong-size billing that delivered() already prevents for memory
breardon2011
marked this pull request as ready for review
September 2, 2026 17:01
…YERS marker and embed it, so custom templates compile from the SAME definition the pool image is built from — the marker sits after useradd (steps can chown into the home dir), before ENV PIP_USER=1 (a build-time pip install lands system-wide instead of /root/.local, invisible to the sandbox user), and before the agent COPY and ENTRYPOINT (no customer layer can shadow the agent); Split() fails loudly if the marker is ever removed
… — sibling of translateStepToCommand, not a replacement: a QEMU checkpoint captures the whole rootfs so shell steps persist there, while /oc/workspace/export archives only /home/sandbox, so system packages have to be baked into an image; apt_install maps to dnf with a Debian->AL2023 name table, package names with shell metacharacters are rejected rather than quoted, env renders sorted so an unchanged manifest hashes identically, and an unsupported step errors instead of silently vanishing
…ng templates table (no migration) — template_type='microvm_image' with the ARN in image_ref; the row is created at status=processing BEFORE the build so a build that dies partway is a visible stuck template rather than nothing, and SetMicrovmTemplateImage writes the ARN and flips to ready in ONE guarded statement because a ready row with no ARN cannot be launched from and would force create-time code to guess
…m/publish.sh — the artifact is DERIVED from the pool image's own ZIP (swap only the Dockerfile) so the agent binaries are byte-identical to every pooled box, and base-image/build-role ARNs are read off the pool image's ACTIVE version rather than configured, so they cannot go stale when it is rebuilt; a failed build reports the reason from the build record plus its CloudWatch log group, since a bare CREATE_FAILED cannot distinguish a typo'd package from an x86 binary on an ARM-only platform
…N rides on PLACEMENT (Claim is what picks the box; setting it only on the activation silently takes a pooled default box, which is how this was caught on dev), forces a cold launch since warm stock is default-image by construction, and never falls back to the default image; a template that is not ready or has no ARN is a 400, and Activate cross-checks the bound box against the expected ARN so a mismatch is loud instead of a healthy-looking box running the wrong image
… at 100 MicroVM images with 35 already used and 16 belonging to another workload, so image-per-template runs out after ~65 and would also block rebuilding the pool image; names are content-addressed from the manifest hash so an unchanged template reuses its slot, the cap counts processing as well as ready (or N concurrent builds outrun it), and every destructive path is gated on an osb-tpl- prefix because managed-agents-runtime-* images sit in the same account
…time — snapshot creates on this runtime go to the image pipeline instead of the checkpoint machinery (resolveImageManifest returns a checkpoint ID, which has no meaning here), ordered dedupe -> quota -> row -> build -> ready so a build that dies is a visible stuck template and a ready row always has an ARN; the build is always async because it takes minutes, marks failure on a detached context since the request is long gone, and an inline manifest on create is refused with the supported path named rather than silently ignored
…contract, and test Claim for real — GET /snapshots/:name only read image_cache, so a microvm template 404'd forever and the SDK's waitUntilReady treats 404 as 'still building', hanging a customer on a healthy template until their timeout (or forever without one); create/get now return the SnapshotInfo shape the SDK types, with an empty checkpointId because inventing one would resolve to nothing; replaces two struct-fixture tests with a recording fake that asserts Claim actually calls RunMicrovm with the template ARN and reports warm=false
…ere type-only exports, so the import the Image reference documents (import { Image } from "@opencomputer/sdk") failed at runtime with 'does not provide an export named Image', and there was no supported way to define a template through the SDK at all; caught by the first end-to-end template run, which could not import them
…ugs the first end-to-end runs found, none visible to unit tests: (1) status lived only in cell PG while GET /snapshots/:name is served by the EDGE from D1, so waitUntilReady polled a template that was ready in 3 minutes until its 20-minute timeout; (2) create-time resolution reads a SECOND D1 index, so a template reported ready and then 404'd; (3) dedupe returned the existing row instead of creating one under the requested name, reproducing that same ready-then-404 for a different reason — dedupe now reuses the IMAGE and still creates a row for the name asked for
…omputer/sdk 1.x lands on MicroVM, every published 0.x stays on QEMU, and orgs.runtime becomes a pin that overrides both, so a customer migrates by upgrading the package and rolls back by pinning the old major without us touching anything; also carries the same decision through the snapshot path (a template built on the wrong runtime is unusable) and routes exec by the sandbox's own worker_id rather than its org's runtime, because SDK routing makes a mixed org the steady state
… so Python callers migrate to MicroVM by upgrading exactly like TypeScript ones
… stays reversible — the cell PERSISTS the token's runtime into its own orgs table, so sending the SDK-derived answer there made an unpinned org's first create from a new SDK write itself in as a permanent pin, after which deliberately calling from an older SDK could not move it back; found on dev, where two orgs pinned themselves this way
motatoes
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the agent-tunnel MicroVM path with a data plane that reaches the guest over plain JSON, and brings the runtime to parity with the QEMU fleet.
−7,953 / +9,225 across 96 files. The old path — a per-box gRPC tunnel brokered by the control plane, plus its pool/manager/workspace layer — is deleted. Reaching the guest through the platform's own proxy is both faster and the only thing that works across it, and running two fillers against one regional quota meant each overran and starved the other.
What's new: a /oc/* front door in the guest (envs, exec, workspace export/import, PTY and exec-session streaming, sealed secrets); workspace checkpoints; hibernate-as-suspend; customer idle timeouts; and dispatch through the backend seam so managed runtimes stop falling through to the worker registry.
Bugs fixed along the way, each one silent before:
Verified on dev: 68 SDK assertions green twice over — lifeTY, preview URLs, reboot, hibernate/wake, checkpoints, idletimeout, secrets, and the refusals.
Known gaps, refused loudly rather than silently: fork-from checkpoint patches, mounts, resize.
QEMU shutdown — operator runbook
Hibernated QEMU sandboxes are not being preserved. 13,638 of them across 178 orgs become unwakeable; 98 were woken in the last 30d.
Before merge
LITE_WARM=0→ real depth. Every create is cold-launching today (1.9–7.9s).MAX_TOTAL_BOXES=230→ above 249 (7d peak). Confirm the AWS regional memory quota covers all traffic, not just the 2 orgs on lite now.-Confirm
opensandbox-agent-lite-prodwas published at 4096 MiB — prod has noDEFAULT_MEMORY_MBset and falls back to 4096, so an image built at publish.sh's 2048 default means every sandbox is metered at double what it gets. Set the variable explicitly.deploy/microvm/SIZE-TIERS.md). (4 gb already exists)Cutover
UPDATE orgs SET runtime='microvm'— new creates go to lite.