Add LCOW live migration source and destination APIs#2803
Conversation
Introduce an end-to-end live-migration workflow for LCOW sandboxes, letting a running sandbox be handed off from a source shim to a destination shim. - Add a migration Controller that sequences a single session through a source or destination lifecycle: the source prepares and exports an opaque sandbox snapshot (VM + per-pod state), while the destination imports it, patches each container onto its new IDs, builds the destination VM, transfers memory over a shared socket, and finalizes with a resume or stop. - Expose the migration gRPC/ttrpc surface on the LCOW shim service (PrepareAndExportSandbox, ImportSandbox, PrepareSandbox, TransferSandbox, FinalizeSandbox, Cancel, Cleanup, CreateDuplicateSocket, Notifications) and register it alongside the task and sandbox services. - Wire migration into the task lifecycle: route destination-side CreateTask for rehydrated containers into a patch path, and reject task mutations (state, delete, kill) while a session is in progress. - Add duplicated transport-socket adoption, a subscriber-based progress notification stream, and proto/HCS conversion helpers for migration options and events. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
marma-dev
left a comment
marma-dev
left a comment
There was a problem hiding this comment.
Overall looks good, doc.go / state.go diagrams are excellent and match the code — nice 👍 .
I would like run.Transfer and service-owned maps comments addressed (or explained). Rest are nice to have.
| c.state = StateTransferring | ||
|
|
||
| // Run the transfer; a failure marks the session failed for subscribers. | ||
| if err := c.runTransfer(ctx); err != nil { |
There was a problem hiding this comment.
It seems the background transfer goroutine does c.mu.Lock(); defer c.mu.Unlock() and then calls c.runTransfer(ctx) inside that critical section. runTransfer issues several HCS calls (StartLiveMigrationOnSource / StartWithMigrationOptions / StartLiveMigrationTransfer).
While those calls are in flight, State() (RLock) and especially Cancel() (Lock) will block. If the goal is that the caller returns immediately and cancellation happens out-of-band, being unable to acquire the lock to Cancel during the transfer kickoff partly defeats that goal.
If any of those HCS calls can block for a meaningful duration, this serializes all controller access behind them.
Should we consider transitioning to StateTransferring under the lock, releasing it, then running the HCS calls, and re-acquiring only to set the terminal state?
I am trying to confirm whether those Start* calls are fire-and-forget (completion arriving via notifications) — if so, the impact is smaller, but the lock-held-across-HCS pattern still seems fragile.
| opts.PodControllers[importedPod.PodID()] = importedPod | ||
|
|
||
| // Source container IDs must be unique across pods so the later patch | ||
| // lookup is unambiguous. | ||
| for containerID := range importedPod.ListContainers() { | ||
| if _, dup := pending[containerID]; dup { | ||
| return fmt.Errorf("duplicate source container id %q across imported pods: %w", containerID, errdefs.ErrInvalidArgument) | ||
| } | ||
|
|
||
| // Track the container as awaiting a patch and map it to its pod. | ||
| pending[containerID] = struct{}{} | ||
| opts.ContainerPodMapping[containerID] = importedPod.PodID() | ||
| } |
There was a problem hiding this comment.
If a later pod.Import (or a duplicate-container check) fails, the function returns an error but leaves earlier pods/containers already inserted into the service maps, while c.state stays StateIdle.
I think that even if the controller is never mutated on the basis of a half-specified request, it only holds for the controller struct, not for the borrowed maps. Building into local maps first and committing them only on success would keep failures atomic
| wantSize := int(unsafe.Sizeof(windows.WSAProtocolInfo{})) | ||
| if len(protocolInfo) < wantSize { | ||
| return fmt.Errorf("protocol info is %d bytes, want at least %d: %w", len(protocolInfo), wantSize, errdefs.ErrInvalidArgument) | ||
| } | ||
|
|
||
| // Decode the opaque caller-supplied bytes into the socket descriptor | ||
| // used to recreate the duplicated socket. | ||
| var info windows.WSAProtocolInfo | ||
| if err := binary.Read(bytes.NewReader(protocolInfo), binary.LittleEndian, &info); err != nil { | ||
| return fmt.Errorf("decode WSAProtocolInfo: %w", err) | ||
| } | ||
|
|
There was a problem hiding this comment.
Looks like we validate the buffer with int(unsafe.Sizeof(windows.WSAProtocolInfo{})) but decode with binary.Read(..., &info). binary.Read consumes binary.Size(info) bytes (its own reflection-based, unpadded layout), which is not guaranteed to equal unsafe.Sizeof if the struct has any alignment padding.
I believe for WSAProtocolInfo the fields are 4-byte-aligned so they likely match today, but relying on that coincidence is brittle.
Could we consider validating against binary.Size(windows.WSAProtocolInfo{}) (and/or decoding via the same mechanism the caller serialized with) so the size check and the decode agree?
| // Reattach each migrated pod's containers and processes on this host. | ||
| for podID, podCtrl := range c.podControllers { | ||
| if err := podCtrl.Resume(ctx, c.vmController, events, true /* isDestination */); err != nil { | ||
| return fmt.Errorf("resume pod %q from migration: %w", podID, err) |
There was a problem hiding this comment.
On the RESUME path, if one podCtrl.Resume fails partway through the loop, Finalize returns before setting StateFinalized, so the session stays in StateTransferCompleted with the VM already resumed and some pods live. Since finalizeSandboxInternal just propagates the error to the caller, what's the intended recovery here, is the caller expected to retry Finalize, or fall back to a STOP?
It seems like a retried RESUME would call vm.Resume again on an already-running VM, which vm.Resume rejects, so I wasn't sure how the partial-failure case is meant to unwind. I could be missing where that's handled upstream.
3 participants
Summary
Introduce an end-to-end live-migration workflow for LCOW sandboxes, letting a running sandbox be handed off from a source shim to a destination shim.
Add a migration Controller that sequences a single session through a source or destination lifecycle: the source prepares and exports an opaque sandbox snapshot (VM + per-pod state), while the destination imports it, patches each container onto its new IDs, builds the destination VM, transfers memory over a shared socket, and finalizes with a resume or stop.
Expose the migration gRPC/ttrpc surface on the LCOW shim service (PrepareAndExportSandbox, ImportSandbox, PrepareSandbox, TransferSandbox, FinalizeSandbox, Cancel, Cleanup, CreateDuplicateSocket, Notifications) and register it alongside the task and sandbox services.
Wire migration into the task lifecycle: route destination-side CreateTask for rehydrated containers into a patch path, and reject task mutations (state, delete, kill) while a session is in progress.
Add duplicated transport-socket adoption, a subscriber-based progress notification stream, and proto/HCS conversion helpers for migration options and events.
This PR needs #2790 to be merged.