From 70ce74fdc034b3f7e28f664298ba51b2e64492e3 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 15:19:21 -0400 Subject: [PATCH 01/26] docs: add JEP-0017 multi-exporter leases and inter-exporter port forwarding Extend the existing Lease to bind more than one exporter, and add automatic port forwarding between the exporters in a lease so devices can talk to each other. Motivated by Android Auto phone projection, which is a two-device protocol (Bluetooth pairing, then handover to peer-to-peer 5 GHz Wi-Fi) that Jumpstarter cannot express today. The lease gains optional spec.members[] -- each a role name plus the same selector fields a single-exporter lease already uses -- and optional spec.forwards[] joining a named port on one member to a named port on another. Ports are declared by drivers and carried in the exporter's existing report as an optional field, so a lease references headunit.rootcanal -> phone.controller and never learns an address; the exporter resolves that locally. Extending Lease rather than adding a LeaseGroup CR is the load-bearing decision (DD-1): an exporter claim is lease.Status.ExporterRef, persisted by a single Status().Update, so N members in one object bind in one atomic write and partial acquisition is structurally impossible. N child leases would bind independently and need an admission gate, acquisition timeout, release-all, and jittered backoff purely to recover from a state this design never enters. Ports carry a direction (provides/requires) rather than a medium or wire-format taxonomy (DD-6, DD-8). Direction is what actually needs checking: HCI is asymmetric, so forwarding one rootcanal into another wires controller to controller and cannot work -- rejected structurally, with no Bluetooth knowledge in the controller. The data plane is existing code: TemporaryTcpListener + forward_stream with a router peer stream substituted for the client stream, so router.proto is untouched and bt-peer participates with no Python changes at all. Prior art is stated precisely: OmniLab ATS does gang-schedule multi-device jobs and does have a multi-host mode, but SimpleScheduler.allocate requires every device in an allocation to share one LabLocator, so a multi-device job cannot span lab hosts. That single check is the gap this JEP targets. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 1727 +++++++++++++++++ docs/source/contributing/jeps/index.md | 2 + 2 files changed, 1729 insertions(+) create mode 100644 docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md new file mode 100644 index 000000000..035c863ac --- /dev/null +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -0,0 +1,1727 @@ +# JEP-0017: Multi-Exporter Leases and Inter-Exporter Port Forwarding + +| Field | Value | +| ----------------- | -------------------------------------------------------- | +| **JEP** | 0017 | +| **Title** | Multi-Exporter Leases and Inter-Exporter Port Forwarding | +| **Author(s)** | @kirkbrauer (Kirk Brauer) | +| **Status** | Draft | +| **Type** | Standards Track | +| **Created** | 2026-09-01 | +| **Updated** | 2026-09-01 | +| **Discussion** | *TBD (PR link)* | +| **Requires** | JEP-0014 | +| **Supersedes** | | +| **Superseded-By** | | + +--- + +## Abstract + +This JEP extends the existing `Lease` to bind more than one exporter, and adds +automatic port forwarding between those exporters so devices in one lease can +talk to each other. A lease gains an optional `spec.members[]` — each member a +role name (`phone`, `headunit`) plus the same selector fields a single-exporter +lease already uses — and an optional `spec.forwards[]` joining a **named port** +on one member to a named port on another. Ports are declared by drivers and +carried in the exporter's existing report, so a lease references +`headunit.rootcanal → phone.controller` and never learns an address; the +exporter resolves that locally. Because every member's exporter claim lives in +one object's status, binding a bench is a single atomic write: a lease holds +all of its devices or none. The data plane is the existing +`TcpPortforwardAdapter` with one substitution — a router peer stream in place +of a client stream — so `RouterService` and every driver work unchanged. This +realizes JEP-0014's deferred "composite leases — multiple exporters linked into +one logical lease" literally, and takes option 1 of JEP-0016's DD-8. + +## Motivation + +Jumpstarter's lease is the unit of exclusive access to *one* exporter. +`LeaseSpec` carries a single `selector` or a single `exporterRef`, and +`LeaseStatus` records a single `status.exporterRef`. Every layer above +inherits that shape: `RequestLease`/`Dial` are keyed by one lease, +`jmp shell` exports one `JUMPSTARTER_HOST`, and `JumpstarterTest` acquires +"a lease for a single exporter using the selector annotation". + +That is the right primitive for the majority of HiL work — one board, one +harness — but an entire class of tests is about *interaction between +devices*, and today Jumpstarter cannot express it. + +### The concrete problem: Android Auto phone projection + +Android Auto phone projection is a two-device protocol by construction. The +phone and the head unit first pair over **Bluetooth**; the head unit then +hands the session off to a peer-to-peer **5 GHz Wi-Fi** link, because +Bluetooth lacks the bandwidth for continuous video projection. Validating it +means holding a phone and a head unit *simultaneously*, keeping them +connected, driving both, and asserting on both — a phone-only or +head-unit-only lease cannot observe the handover at all. + +This is not hypothetical. Teams running phone-projection validation operate +fleets in the ~1000-device range across multiple labs, and the two things +they cannot get from existing tooling are (a) non-Android devices in the same +bench as Android ones — Linux and QNX IVI head units — and (b) pairing two +*virtual* devices to each other. + +Running that on Jumpstarter today requires the test author to hand-roll +everything the lease layer should provide: + +- **No atomic acquisition.** The client requests two independent leases. If + the second selector is unsatisfiable the first is already held, so the + test either blocks holding scarce hardware or unwinds by hand. Under + contention, two concurrent benches can each take one half of the pair and + deadlock until expiry — the classic gang-scheduling failure. +- **No shared lifetime.** Two leases expire independently. Half a bench + disappearing mid-run produces a failure that looks like a device fault. +- **No correlation for policy or observability.** `ExporterAccessPolicy` + sees two unrelated requests; JEP-0013 traces see two unrelated leases. + Nothing records that these devices were *a bench*. +- **No path between the devices.** Even with both leases in hand, the two + exporters have no way to exchange device-level traffic. Every Jumpstarter + stream is client↔exporter; there is no exporter↔exporter path. + +The last point is the hard one, and it is what makes this more than an +ergonomics change. + +### The virtual case: what nobody has shipped yet + +For virtual devices the connectivity problem is sharper. Cuttlefish and the +Android emulator can already pair virtual devices to each other — but only +within one host: + +- Cuttlefish shares its virtual radio media between instances launched from + a single `launch_cvd --num_instances=n`, or from separate `launch_cvd` + invocations that are pointed at the *same* `--vhost_user_mac80211_hwsim` + socket path and the same rootcanal/netsim daemon. Both mechanisms are + host-local Unix sockets and loopback TCP ports. +- The Android emulator's new networking backplane (36.5) is explicitly + described as bridging "all running instances on the same host machine". +- `podcvd`, Google's container wrapper, publishes each container's ports on + its own IP but keeps radio simulation inside the container group. + +So the state of the art for virtual multi-device Android testing is: *both +devices must live on one machine*. That constraint is exactly what a +cluster-scheduled, autoscaled pool of one-CVD-per-Pod exporters (JEP-0016) +breaks — and it is why JEP-0016's DD-8 deferred multi-CVD groups pending +"a cross-Pod virtual-radio story… real upstream-facing work." + +The encouraging part is that these simulators are reached over ordinary +sockets. `rootcanal`, the virtual Bluetooth controller, accepts HCI on a TCP +port — which is why `jumpstarter-driver-bt-peer` can already attach a +`bumble` peer to a CVD with `transport: "tcp-client:127.0.0.1:7300"`. +`netsim`, the Rust simulator that orchestrates Bluetooth, BLE, Wi-Fi and UWB +for both Cuttlefish and the emulator, accepts virtual chips over a +bidirectional gRPC stream (`PacketStreamer.StreamPackets`, whose first +`PacketRequest` carries a `ChipInfo` naming the device and chip kind). +`wmediumd` speaks over a frame socket. Every one of them is something a +process connects to. Nothing about "same host" is fundamental; it is an +artifact of the fact that nobody has forwarded those sockets between +machines under a common lease. + +### Prior art's ceiling, precisely + +Google's OmniLab Android Test Station is the closest existing system, and +being exact about what it can and cannot do matters, because the naive +version of this comparison is wrong. + +ATS 2.0 runs on Mobile Harness (open-sourced as `google/device-infra`), and +it **does** do multi-device. Ad-hoc testbeds gang-schedule N devices for one +job: `AdhocTestbedSchedulingUtil.findSubDevicesSupportingJob` performs +maximum-cardinality bipartite matching of `SubDeviceSpec`s against idle +devices, and `TestbedDevice`, `CompositeDevice`, and +`SubDeviceSynchronizationDriver` all exist. It also has a multi-host mode. +So "ATS cannot do multi-device" and "ATS is single-host" are both too strong. + +The actual constraint is one check in +`infra/controller/scheduler/simple/SimpleScheduler.java`. In `allocate`, +every device locator in an allocation must share a single `LabLocator`; +otherwise the allocation is refused with *"Lab locators do not match. Can not +create allocation"*. **A multi-device job is therefore confined to one lab +host.** Multi-host mode is one controller with N workers for fleet-level +pooling — it does not make a single job span hosts. And in either case there +is no bridged medium between roles: Tradefed states its own version of the +gap verbatim — *"No APIs yet exist to conduct operations from one device to +another, such as `device1.sync(device2)`."* + +Two further limits shape the opportunity. Mobile Harness's `Device` +abstraction is ADB-shaped and runs inside the lab-server JVM next to the +device, and its open-source platform coverage is `platform/android`, +`platform/androiddesktop`, and `platform/testbed` — there is no Linux, QNX, +serial, power, or CAN support, and extending it means in-tree Java and Bazel +rather than an out-of-tree package. + +This JEP targets exactly that seam: + +- **Physical ↔ physical**: a phone and a head unit on two exporters, on + **different lab hosts**, in one lease — the allocation `SimpleScheduler` + refuses. +- **Virtual ↔ virtual**: two CVDs in two Pods, on two nodes, pairing over + Bluetooth and Wi-Fi — the thing that has not been done anywhere. +- **Physical ↔ virtual (hybrid)**: a real phone in a lab rack paired to a + virtual head unit running in the cluster, via a gateway exporter that owns + a real radio. + +The last one is the interesting business case: labs have scarce physical +head units and abundant phones (or the reverse), and virtualizing the +abundant half while keeping the scarce half real is a direct cost and +throughput win no host-local scheduler can offer. + +### User Stories + +- **As an** Android Auto QA engineer, **I want to** lease a phone and a head + unit as one bench with roles, **so that** my projection test either gets + both devices or waits — never half a bench, never a deadlock against a + concurrent run. +- **As an** AAOS platform developer, **I want to** pair two Cuttlefish + devices running in different cluster Pods over Bluetooth and then hand off + to Wi-Fi, **so that** I can validate wireless projection in CI without a + physical lab or a single fat host. +- **As a** lab operator whose phones and head units are in different racks + or different buildings, **I want** one lease to span them, **so that** my + bench is not limited to devices plugged into the same machine. +- **As a** test author with a mixed bench, **I want** a QNX or Linux head + unit alongside an Android phone, **so that** device type is a driver + choice rather than a platform limit. +- **As a** Jumpstarter user who already knows leases, **I want** a bench to + be *a lease*, **so that** everything I know about `jmp create lease`, + expiry, release, and access policy carries over without learning a second + resource. + +## Proposal + +Three additions, none of them a new resource kind: + +- **Members.** A lease gains optional `spec.members[]`, each a role name plus + the same `selector` / `exporterRef` fields a single-exporter lease already + uses. A lease with members binds every one of them or none. +- **Ports.** A driver may declare named ports it **provides** (a service + listening locally) or **requires** (a socket it will dial). Ports travel in + the exporter's existing report; addresses never leave the exporter. +- **Forwards.** A lease gains optional `spec.forwards[]`, each joining a + provided port on one member to a required port on another. The exporters + establish the forward themselves over the existing router. + +A lease with members is called a **bench** informally, but it is not a new +kind of object: it is a `Lease`, it appears in `jmp get leases`, it expires +the way leases expire, and `spec.release` ends it. + +### Ports + +A port is a named connection point on a driver. There are exactly two +directions, and the direction is what makes a forward well-formed. + +| Direction | Meaning | Example | +| --- | --- | --- | +| `provides` | A service is listening; something may be forwarded *from* it | `rootcanal` on a Cuttlefish exporter — HCI on `127.0.0.1:7300` | +| `requires` | The driver will dial a local address; a forward may be delivered *to* it | `controller` on a `bt-peer` exporter — where its bumble stack expects an HCI controller | + +`provides` needs almost nothing new, because a `TcpNetwork` child already +*is* a named provided port — the child's name is the port name: + +```yaml +export: + cuttlefish: + type: jumpstarter_driver_cuttlefish.driver.Cuttlefish + children: + rootcanal: + type: jumpstarter_driver_network.driver.TcpNetwork + config: { host: 127.0.0.1, port: 7300 } + ports: + - name: rootcanal + direction: provides + protocol: hci-h4 # optional +``` + +`requires` is the genuinely new concept: a declaration that the driver will +dial a local address, and that the exporter should bind an inbound forward +there. + +```yaml +export: + bt_peer: + type: jumpstarter_driver_bt_peer.driver.BtPeer + config: + transport: "tcp-client:127.0.0.1:7300" # unchanged + ports: + - name: controller + direction: requires + listen: 127.0.0.1:7300 # local; never reported + protocol: hci-h4 # optional +``` + +Note what did *not* change: `bt-peer`'s Python is untouched, and its +`transport` string still points at `127.0.0.1:7300`. The lease simply makes +that address resolve to a rootcanal in another Pod on another node. A driver +that can talk to a local service can now talk to a remote one without +knowing the difference. + +### Declaring a bench + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: Lease +metadata: + name: android-auto-bench + namespace: jumpstarter-lab +spec: + clientRef: + name: ci-runner + duration: 45m + members: + - name: phone + selector: + matchLabels: + device-type: android-phone + android-version: "15" + - name: headunit + selector: + matchLabels: + device-type: aaos-headunit + forwards: + - name: bt + from: { member: headunit, port: rootcanal } # provides + to: { member: phone, port: controller } # requires +``` + +No addresses, no ports numbers, no medium taxonomy. The lease names roles and +port names; both sides' internals stay inside their exporters (DD-6). + +`members[].selector` and `members[].exporterRef` are the *same* fields as +the top-level `spec.selector` and `spec.exporterRef` — a member is the +existing lease request with a role name attached. The single-exporter form is +unchanged and exactly equivalent to a one-member lease, so today's leases are +already the degenerate case rather than a legacy shape to migrate (DD-2). + +A member may be marked `optional: true`, in which case the lease binds +without it and the role resolves to `None` on the client. + +### Acquiring and using a bench + +The existing commands take members and forwards; there is no parallel +command set. + +```console +$ jmp create lease \ + --member phone=device-type=android-phone \ + --member headunit=device-type=aaos-headunit \ + --forward bt=headunit.rootcanal:phone.controller \ + --duration 45m +android-auto-bench + +$ jmp get lease android-auto-bench +NAME ENDED CLIENT EXPORTER AGE +android-auto-bench false ci-runner phone=rack3-pixel8, 12s + headunit=cf-auto-7b2c +``` + +Port names are discoverable rather than tribal knowledge, which is the point +of putting them in the report (DD-7): + +```console +$ jmp get exporter cf-auto-7b2c -o json | jq '.status.devices[].ports' +[{"name":"rootcanal","direction":"provides","protocol":"hci-h4"}] +``` + +In a shell, roles become top-level names alongside the usual driver clients: + +```console +$ jmp shell --lease android-auto-bench +jumpstarter ⚡ android-auto-bench ➤ j phone adb shell getprop ro.product.model +Pixel 8 +jumpstarter ⚡ android-auto-bench ➤ j headunit power on +jumpstarter ⚡ android-auto-bench ➤ j forward status +NAME FROM TO MODE STATE A→B B→A +bt headunit.rootcanal phone.controller router connected 1.2 MiB 0.9 MiB +``` + +In Python, the existing `lease()` context manager grows `members` and +`forwards`, and a multi-member lease yields a mapping of roles to the same +client objects a single-exporter lease yields directly: + +```python +from jumpstarter.config.client import ClientConfigV1Alpha1 + +config = ClientConfigV1Alpha1.load("default") + +with config.lease( + members={"phone": "device-type=android-phone", "headunit": "device-type=aaos-headunit"}, + forwards=[Forward("bt", frm=("headunit", "rootcanal"), to=("phone", "controller"))], + duration=timedelta(minutes=45), +) as lease: + with lease.connect() as bench: + phone = bench.phone + headunit = bench.headunit + + headunit.power.on() + phone.adb.wait_for_device() + + # The forward is already up; drive the protocol, not the plumbing. + bench.forwards["bt"].wait_connected(timeout=30) + phone.bt_peer.start({"name": "Bumble-Phone"}) + phone.bt_peer.wait_connection(timeout=60) + + assert phone.adb.shell("dumpsys activity | grep -c CarProjection") == "1" +``` + +`config.lease(selector=...)` without `members` behaves exactly as it does +today and yields a single client — the one-member case is not special-cased +in the API, only in what `connect()` returns (DD-2). + +`JumpstarterTest` grows `members` and `forwards` class variables next to +`selector`, so existing pytest suites extend without a second base class. + +### Running existing multi-device suites + +Because a bench is a set of roles with ADB-capable members, it can be +projected into the config formats existing Android multi-device tests already +consume: + +```console +$ jmp get lease android-auto-bench -o mobly > testbed.yml +$ mobly_test.py -c testbed.yml --test_bed android-auto-bench +``` + +which emits a Mobly testbed whose `AndroidDevice` controllers point at the +per-member ADB endpoints Jumpstarter has forwarded locally, with the role +name carried through as the Mobly device label. A Mobile Harness +`SubDeviceSpec` mapping follows the same shape. This is the integration seam +with ATS/OmniLab: the tests and the results pipeline do not change, the +*bench* changes from "devices sharing one `LabLocator`" to "any mix of +physical and virtual devices anywhere the controller can reach." + +### How a forward comes up + +The data plane already exists. `adapters/portforward.py` implements +`TcpPortforwardAdapter` as `TemporaryTcpListener` + `client.stream_async()` + +`forward_stream()`. The inter-exporter version is the same three primitives +with one substitution — a router peer stream in place of the client stream: + +1. Once the lease is bound, the controller validates both ports against the + exporters' reports and instructs each exporter over its existing `Listen` + stream. +2. Each exporter calls the new `DialPeer` RPC and receives a router endpoint + plus a token whose `stream` claim is the **same** for both ends. +3. Both call `RouterService.Stream`. The router already "connects caller to + another caller of the same stream" — a symmetric rendezvous that needs no + protocol change to pair two exporters instead of a client and an exporter + (DD-5). +4. The `provides` side dials its local service (`127.0.0.1:7300`) and splices + it to the stream with `forward_stream`. The `requires` side opens a + `TemporaryTcpListener` on its declared `listen` address and splices each + accepted connection to the stream. +5. **Fast path**: if `DialPeer` returned a peer hint, each exporter races a + direct dial against the router path and keeps whichever completes first + (DD-4). + +```text + ┌──────── Lease: android-auto-bench (one object) ─────────┐ + │ status.members: │ + │ phone → Exporter/rack3-pixel8 │ + │ headunit → Exporter/cf-auto-7b2c │ + └───────┬─────────────────────────────────┬───────────────┘ + │ │ + ┌────────────▼────────────┐ ┌────────────▼────────────┐ + │ Exporter: rack3-pixel8 │ │ Exporter: cf-auto-7b2c │ + │ bt_peer │ │ cuttlefish │ + │ requires: controller │ │ provides: rootcanal │ + │ listener 127.0.0.1:7300│ │ dials 127.0.0.1:7300 │ + └────────────┬────────────┘ └────────────┬────────────┘ + │ │ + │ ┌───────────────────┐ │ + └─────►│ RouterService │◄─────┘ (default) + │ same stream id │ + └───────────────────┘ + └───────── direct peer ─────────┘ (fast path) +``` + +The client is not in the data path. + +### Attaching virtual radios + +With forwards as the mechanism, "bridging a radio" stops being a special +subsystem and becomes a question of which port each stack exposes: + +| Stack | Port | Direction | Notes | +| --- | --- | --- | --- | +| rootcanal (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP; hosts attach to it | +| `bt-peer` (bumble) | `controller` | requires | Dials an HCI controller | +| netsim | `netsim` | provides | gRPC `PacketStreamer`; attaching side must originate `ChipInfo` | +| `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | +| Gateway exporter (real adapter) | `hci` | provides | A real radio, presented as HCI | + +Two consequences worth stating plainly. First, HCI is **asymmetric**: a host +attaches to a controller. Forwarding one rootcanal's port into another +rootcanal wires controller to controller and nothing happens — which is +precisely why forwards are directional and why a `provides → provides` +forward is rejected (DD-6). Second, netsim's port is not a transparent +splice: the attaching side must speak `StreamPackets` and send `ChipInfo` +first, so the consumer is a protocol-terminating driver rather than a raw +socket. Whether two CVDs can be joined at rootcanal directly, or whether +netsim must sit in the middle because it is the component that presents a +controller to multiple hosts, is an open question the Phase 1 prototype +answers (see Unresolved Questions). + +For **physical ↔ virtual** there is no way to relay a real phone's internal +HCI: the radio is inside the device. The bridge happens in RF, at a +**gateway exporter** owning a real adapter and physically near the physical +device (DD-11). This is a lab-hardware requirement, not a software trick. + +### API / Protocol Changes + +All changes are **additive fields on existing types**, plus one new RPC. No +new CRD, no new API group, no existing field changes meaning. + +**Driver report** — ports are optional, and an exporter that reports none +simply cannot participate in forwards (DD-7): + +```protobuf +message DriverInstanceReport { + // ... fields 1-5 unchanged ... + repeated PortReport ports = 6; // NEW, optional +} + +message PortReport { + string name = 1; // "rootcanal", "controller" + PortDirection direction = 2; // PROVIDES | REQUIRES + optional string protocol = 3; // free-form; compared only if both ends set it +} + +enum PortDirection { + PORT_DIRECTION_UNSPECIFIED = 0; + PORT_DIRECTION_PROVIDES = 1; + PORT_DIRECTION_REQUIRES = 2; +} +``` + +The `listen` address of a `requires` port is deliberately **absent**: it is +local to the exporter and no other component needs it (DD-6). + +**`LeaseSpec`** gains two optional lists: + +```go +type LeaseSpec struct { + // ... all existing fields unchanged ... + + // Members of a multi-exporter lease. When empty, the lease binds a + // single exporter using Selector/ExporterRef exactly as before. + // +kubebuilder:validation:MaxItems=8 + Members []LeaseMember `json:"members,omitempty"` + + // Port forwards between members. Requires Members. + Forwards []LeaseForward `json:"forwards,omitempty"` +} + +type LeaseMember struct { + Name string `json:"name"` + Selector metav1.LabelSelector `json:"selector,omitempty"` + ExporterRef *corev1.LocalObjectReference `json:"exporterRef,omitempty"` + Optional bool `json:"optional,omitempty"` + AllowDisabled bool `json:"allowDisabled,omitempty"` +} + +type LeaseForward struct { + Name string `json:"name"` + From ForwardEndpoint `json:"from"` // must resolve to a `provides` port + To ForwardEndpoint `json:"to"` // must resolve to a `requires` port + // Auto (default) | Router | Direct | ClientRelay + Mode string `json:"mode,omitempty"` +} + +type ForwardEndpoint struct { + Member string `json:"member"` + Port string `json:"port"` +} +``` + +**`LeaseStatus`** gains parallel lists and keeps its scalar: + +```go +type LeaseStatus struct { + // ... all existing fields unchanged ... + // ExporterRef stays authoritative for single-exporter leases and is + // left nil for multi-member leases (DD-2). + + Members []LeaseMemberStatus `json:"members,omitempty"` + Forwards []LeaseForwardStatus `json:"forwards,omitempty"` +} + +type LeaseMemberStatus struct { + Name string `json:"name"` + ExporterRef *corev1.LocalObjectReference `json:"exporterRef,omitempty"` + Priority int `json:"priority,omitempty"` + SpotAccess bool `json:"spotAccess,omitempty"` +} + +type LeaseForwardStatus struct { + Name string `json:"name"` + State string `json:"state"` // Pending|Connecting|Connected|Reconnecting|Failed + Mode string `json:"mode,omitempty"` // transport actually in use + Message string `json:"message,omitempty"` +} +``` + +`ExporterStatus.Devices[]` gains the reported ports so the controller can +validate forwards against bound exporters. + +The existing CEL rules are extended, not replaced — the current +"one of selector or exporterRef is required" rule gains a `members` arm, plus +new rules for mutual exclusion, unique role names, forwards referencing +declared members, and member immutability (mirroring `tags` and `context`). + +**Protocol** — three additive fields and one new RPC: + +```protobuf +message RequestLeaseRequest { + google.protobuf.Duration duration = 1; // unchanged + LabelSelector selector = 2; // unchanged + repeated LeaseMember members = 3; // NEW + repeated LeaseForward forwards = 4; // NEW +} + +message GetLeaseResponse { + // ... fields 1-6 unchanged; exporter_uuid set only when single-member ... + repeated LeaseMemberStatus members = 7; // NEW + repeated LeaseForwardStatus forwards = 8; // NEW +} + +message DialRequest { + string lease_name = 1; // unchanged + optional string member_name = 2; // NEW: required for multi-member leases +} + +service ControllerService { + // ... existing RPCs unchanged ... + rpc DialPeer(DialPeerRequest) returns (DialPeerResponse); +} + +message DialPeerRequest { + string lease_name = 1; + string forward_name = 2; + string member_name = 3; +} + +message DialPeerResponse { + string router_endpoint = 1; + string router_token = 2; // `stream` claim shared by both ends + optional string peer_endpoint = 3; // fast-path hint + optional string peer_token = 4; // authenticates a direct dial +} +``` + +`ReleaseLeaseRequest`, `ListLeasesRequest`, `RouterService`, and +`router.proto` are untouched. + +**CLI surface** — existing commands, new flags: + +- `jmp create lease --member role=selector --forward name=m.port:m.port` +- `jmp get lease[s]` prints per-role exporters; `-o json|yaml|name` unchanged +- `jmp get lease -o mobly` +- `jmp get exporter ` shows declared ports +- `jmp shell --lease `, `j ...`, `j forward status` +- `jmp delete lease` / `jmp update lease` need no changes + +### Hardware Considerations + +- **Gateway exporters** for hybrid benches need a real Bluetooth adapter (a + USB HCI dongle is sufficient — `bumble` already supports `usb:0`) and, for + Wi-Fi, a 5 GHz-capable radio in the same RF space as the physical device. + This is new lab hardware, and the JEP does not pretend otherwise. +- **RF isolation.** Multiple physical benches in one room share the air. Labs + running more than one need shielded enclosures or channel planning; the + controller cannot schedule around collisions it cannot observe. Express RF + domains as exporter labels and let selectors keep benches apart. +- **Physical proximity is a scheduling constraint** for physical ↔ physical + radio benches, modeled with ordinary labels (`rf-domain: rack-3`), not new + machinery. +- **Latency budgets.** Bluetooth HCI is timing-sensitive: supervision + timeouts are seconds, but L2CAP/HCI flow control and A2DP jitter buffers + are far tighter. A router-relayed forward adds two gRPC hops; measuring + that budget is an acceptance criterion, and it is the main reason the + direct fast path exists. +- **Wi-Fi frame forwarding is the most latency-sensitive path.** `wmediumd` + models RSSI-based delivery and expects medium-like timing; a TCP substrate + introduces head-of-line blocking a real air interface does not have (DD-10). +- **Listener collisions.** A `requires` port binds a fixed local address. In + a Pod that is free; on a physical lab host already running its own + rootcanal it is not. This is a real deployment constraint, called out in + Unresolved Questions. +- **Degraded hardware.** If a member's exporter goes offline mid-lease, the + lease reports `Ready=False` naming the role and does *not* silently + continue with a partial bench (DD-3). + +## Design Decisions + +### DD-1: Multi-exporter representation — extend `Lease` vs. a new CR + +**Alternatives considered:** + +1. **Extend `Lease`** with `spec.members[]` / `status.members[]`; a + single-exporter lease is the one-member degenerate case. +2. **A new `LeaseGroup` CR owning N child `Lease` CRs.** +3. **Client-side coordination only** — the client acquires N leases and + correlates them by tag. + +**Decision:** Option 1 — extend `Lease`. + +**Rationale:** The deciding factor is how exporter exclusivity is actually +implemented. In `lease_controller.go`, an exporter is claimed by writing +`lease.Status.ExporterRef`, and exclusivity is enforced by scanning other +active leases for a claim on the same exporter +(`ListActiveLeases` → `attachExistingLeases` → `filterOutLeasedExporters`). +The claim is persisted by a **single** `r.Status().Update(ctx, &lease)`. + +All of a lease's exporter claims therefore live in one object's status, and +binding N of them is one write. Either every member's claim lands or none +does. **Partial acquisition is structurally impossible**, so there is nothing +to time out, nothing to release-and-retry, and no gang scheduler. + +Option 2 gets the opposite property. With N child leases, the existing +reconciler binds each independently with its own `Status().Update`, so "three +of four members bound" is a real, durable state in etcd. Everything needed to +recover from it — admission gating, an acquisition timeout, +release-all-on-timeout, jittered backoff, and a contention test to +demonstrate no deadlock — exists purely to clean up after a state option 1 +never enters. That is a large amount of new machinery in the most +correctness-critical controller in the project, in exchange for a separation +that buys little: JEP-0014 already described the goal as "multiple exporters +linked into **one logical lease**." + +Option 2's genuine advantage is that it does not touch `Lease`. That matters +less than it appears, because the compatibility risk concentrates in exactly +one field, `status.exporterRef`, which DD-2 handles directly. Option 2 also +does not avoid protocol work: `Dial(lease_name)` has no way to say *which* +device, so a member selector must be added either way. + +Option 3 provides no atomicity, no co-scheduling, no deadlock avoidance, and +leaves policy unable to reason about a bench — most of the motivation. + +**One thing this decision does not fix.** Atomic means no partial *holds*; it +does not mean two leases can never race for the same exporter. The +exclusivity scan reads a possibly-stale cache, and two leases racing for one +exporter write to *different* objects, so optimistic concurrency does not +catch it. That race exists today for single-exporter leases and is already +acknowledged in the code (*"we could have multiple clients trying to lease +the same exporters… we will need to construct a lease scheduler with the view +of all leases and exporters"*). This JEP neither worsens nor fixes it. A +multi-member lease does fit that future scheduler better than N correlated +objects would: one lease is one scheduling unit, the way one Pod is. + +### DD-2: Keep `status.exporterRef` scalar; add `status.members[]` alongside + +**Alternatives considered:** + +1. **Keep the scalar, add a parallel list.** `status.exporterRef` stays + authoritative for single-exporter leases and is left **nil** for + multi-member leases, which populate `status.members[]` instead. +2. **Promote the scalar to a list** and migrate every reader. +3. **Always populate both**, setting `status.exporterRef` to the first member. + +**Decision:** Option 1. + +**Rationale:** This is the whole compatibility story of DD-1. Every existing +reader of `status.exporterRef` — the `Dial` path, the `Exporter` print +column, JEP-0013 telemetry, and the JEP-0016 Host Orchestrator façade +(explicitly *a view over ordinary Lease CRs*) — keeps working +**byte-identically** for single-exporter leases, which is every lease that +exists today. + +For multi-member leases, option 1 makes old readers see a nil +`status.exporterRef`, which they already interpret as "not bound yet". That +is a **fail-safe** degradation: an unaware consumer shows an unbound lease +rather than confidently operating on one arbitrary device out of several. +Option 3 is the fail-dangerous version — the façade would present a +two-device bench as a single host and route Host Orchestrator calls to +whichever member sorted first. Option 2 is honest but forces a migration on +every consumer for a feature most will never see. + +The same rule applies on the wire: `GetLeaseResponse.exporter_uuid` stays set +only for single-member leases. In the client library, `lease.connect()` +returns a driver client directly for a single-member lease and a role mapping +for a multi-member one — a difference the caller opted into by passing +`members`. + +`Dial` is the one place where silence is unacceptable, because a multi-member +lease has no defensible default device. `DialRequest` gains an optional +`member_name`; calling `Dial` on a multi-member lease without it returns +`INVALID_ARGUMENT` naming the available roles rather than guessing. + +### DD-3: Partial-bench behavior when a member is lost mid-lease + +**Alternatives considered:** + +1. **Fail the lease** — set `Ready=False`, name the failed role, keep the + surviving members held until release or expiry. +2. **End the whole lease immediately** on any member loss. +3. **Continue silently** with the surviving members. + +**Decision:** Option 1. + +**Rationale:** Option 3 is how a two-device test turns into a confusing +one-device pass; it is never right. Between 1 and 2, keeping the survivors +held respects the test: a client that has just lost its head unit usually +wants to collect logs and artifacts from the phone before releasing, and an +abrupt teardown destroys the evidence needed to diagnose the failure. The +condition is loud, and the client library raises on the next call into the +failed role rather than returning stale data. + +This concerns a member lost *after* binding — distinct from acquisition, +which DD-1 makes all-or-nothing. + +### DD-4: Forward transport — router peer streams vs. client relay vs. pure P2P + +**Alternatives considered:** + +1. **Router peer streams with an optional direct P2P fast path.** +2. **Client-relayed** — the client pumps bytes between two streams. +3. **Direct peer-to-peer only.** + +**Decision:** Option 1. + +**Rationale:** Option 2 needs *zero* protocol change and could be prototyped +immediately — but it doubles RTT on a latency-sensitive path, routes lab +traffic through whatever machine ran the test, and breaks for headless CI and +sustained throughput. It is retained as an explicit `mode: client-relay` +debug transport, not as the architecture. + +Option 3 has the best data plane and fails at exactly the case this JEP is +for: the hybrid bench, where a lab exporter behind NAT and a cluster Pod are +not mutually routable. Jumpstarter's value in that topology is that the +controller and router are the only things both sides must reach. + +Option 1 keeps one authentication model — a forward is authorized by the +lease that names it, exactly as a stream is authorized by the lease that +names it — while permitting the fast data plane where it is available. The +fast path is an *optimization*, not a requirement: a bench that works over +the router works everywhere, and `mode: router` makes the slow path explicit +for tests that need reproducible timing. + +### DD-5: Router changes — none + +**Alternatives considered:** + +1. **Reuse `RouterService.Stream` unchanged**, issuing both endpoints a token + bearing the same `stream` claim. +2. **Add a peer-specific RPC** to `router.proto` with explicit A/B roles. + +**Decision:** Option 1 — no `router.proto` change. + +**Rationale:** `RouterService.Stream`'s documented contract is already +"Stream connects caller to another caller of the same stream", and its token +claims (`sub: jumpstarter client/exporter`, `stream: stream id`) already +accommodate an exporter as either party. The router is a symmetric rendezvous +that never needed to know which side was the client. A peer RPC would encode +an asymmetry that does not exist and fork the data path — two implementations +to keep correct, two places to fix flow-control bugs, and a second surface +for the datagram work in Future Possibilities. Everything peer-specific +belongs in token issuance, which is the controller's job. + +### DD-6: Named ports with direction, not addresses + +**Alternatives considered:** + +1. **Named ports on both ends**, each declaring `provides` or `requires`; + the lease references names only. +2. **Raw addresses in the lease** — `from: headunit.rootcanal`, + `to: {member: phone, listen: 127.0.0.1:7300}`. +3. **Untyped named endpoints** — names on both ends but no direction. + +**Decision:** Option 1. + +**Rationale:** Option 2 forces the *client* to know the internal architecture +of an exporter it did not configure: that `bt-peer` expects its controller on +7300, that the address is on loopback, that nothing else is bound there. That +is exporter-private detail leaking into a lease manifest, and it breaks the +moment a lab reconfigures a port. With named ports the lease says +`headunit.rootcanal → phone.controller` and each exporter resolves its own +address locally. + +Direction is what makes option 1 strictly better than option 3, and it took +three attempts to see why. A forward is inherently asymmetric — one side has +a listening service, the other gets a local listener — and that asymmetry is +exactly the relationship the underlying protocols have. HCI has a host side +and a controller side: `bt-peer` attaches to rootcanal *as a host*. Forward +one rootcanal's port into another rootcanal and you have wired controller to +controller, and nothing happens. A `provides → provides` forward is therefore +rejected structurally, with no knowledge of Bluetooth anywhere in the +controller. + +Direction also gives the validation something real to check (DD-7), which +name-matching alone cannot. + +### DD-7: Ports are an optional part of the exporter report + +**Alternatives considered:** + +1. **A new optional repeated `ports` field** on `DriverInstanceReport`, + mirrored into `ExporterStatus.Devices[]`. +2. **Encode ports as driver-instance labels**, which already flow through to + `ExporterStatus.Devices[].Labels` — zero proto and CRD change. +3. **No reporting** — ports live only in exporter config, and forwards fail + at connect time if misconfigured. + +**Decision:** Option 1, and optional in the strong sense: an exporter that +reports no ports simply cannot participate in forwards. + +**Rationale:** Option 3 loses the two things that make named ports usable. +Without reporting, the controller cannot validate a forward before +establishing it, and — more importantly — a client has no way to *discover* +port names, which just relocates the tribal-knowledge problem DD-6 set out to +solve. Ports in the report make `jmp get exporter` the answer to "what can I +wire up here", the same way it already answers "what drivers are here". That +also sits naturally alongside JEP-0011's introspection direction rather than +inventing a parallel discovery channel. + +Option 2 is genuinely tempting: `DriverInstanceReport.labels` already exists +and already reaches `ExporterStatus.Devices`, so ports could ship with no +schema change at all. It fails on the `provides`/`requires` asymmetry. A +`provides` port is already a driver instance — a `TcpNetwork` child with a +uuid in the report — so labelling it annotates something real. A `requires` +port **has no backing driver instance**; it is a declared need, so option 2 +would require synthesizing a phantom report entry with no methods purely to +carry two labels, plus two parallel key namespaces (`port.jumpstarter.dev/*` +and `port-protocol.jumpstarter.dev/*`) to keep in sync. + +Being optional is what makes this free: `repeated` defaults to empty, so +every existing exporter reports nothing new, no version negotiation is +needed, and an old exporter against a new controller is simply not a forward +endpoint — the correct answer anyway. + +**What is deliberately not reported:** the `listen` address of a `requires` +port. It is local to the exporter and no other component needs it (DD-6). +Keeping it out means the report carries only label-safe scalars, and means a +lab can re-address a port without touching anything outside that exporter. + +### DD-8: No protocol taxonomy; direction plus an optional tag + +**Alternatives considered:** + +1. **A `medium` enum** (`bluetooth | wifi | uwb | serial | can`) matched + between endpoints. +2. **A `format`/wire-protocol token** matched for equality. +3. **Direction only**, with an optional free-form `protocol` compared solely + when both ends declare it. + +**Decision:** Option 3. + +**Rationale:** Option 1 validates the wrong axis. `medium: bluetooth` would +approve a forward between a raw-HCI rootcanal port and a netsim +`PacketStreamer` port — both are "bluetooth", neither can talk to the other, +because one is raw bytes and the other is length-delimited protobuf behind a +`ChipInfo` handshake. It describes what traffic *represents*, not what the +bytes *are*, so it approves the one pairing that most needs catching. + +Option 2 fixes the axis but is still insufficient on its own: `hci-h4` on +both ends is exactly the controller-to-controller wiring DD-6 rejects. The +thing that had to be checked was never the payload format but the +*relationship*, and once direction is modelled, most of the value is already +captured without any taxonomy to define, version, or argue about. + +An optional `protocol` string remains useful for the residue — it catches +wiring a `vnc` provides-port into an HCI requires-port — but comparing it +only when both ends declare it keeps it opt-in. Driver authors who want the +check get it; nobody is forced to classify anything, and there is no closed +enum to maintain. + +The honest cost: a forward whose two ends speak different protocols and +declare no `protocol` tag will connect and then fail at the first byte. This +is the same guarantee `kubectl port-forward` gives, which nobody treats as a +defect, but it is a deliberate retreat from admission-time protocol +validation and is recorded as such. + +### DD-9: Where virtual radios attach + +**Alternatives considered:** + +1. **At the simulators' existing sockets** — rootcanal's HCI TCP port, + netsim's `PacketStreamer`, `wmediumd`'s frame socket — each exposed as a + port and joined by an ordinary forward. +2. **A dedicated bridge driver tier** — purpose-built `LinkEndpoint` drivers + that know about radios. +3. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the + HAL or socket layer. + +**Decision:** Option 1. + +**Rationale:** Every simulator involved is already something a process +connects to over a socket, and `jumpstarter-driver-bt-peer` already proves +the pattern by attaching `bumble` to rootcanal at +`tcp-client:127.0.0.1:7300`. Once forwards exist, "bridge a radio" reduces to +"forward the socket that was always there", and a `TcpNetwork` child pointed +at `127.0.0.1:7300` is configuration rather than code. + +Option 2 was in an earlier draft of this JEP and is now rejected as invented +machinery: it added a driver interface, a driver tier, and a medium taxonomy +to express something the existing network drivers plus a direction already +express. Option 3 is rejected because it changes the device under test — a +guest-side shim means the Bluetooth stack being exercised is not the one that +ships, invalidating precisely the pairing and handover behavior these tests +exist to verify. + +One consequence to be explicit about: not every port is a transparent splice. +netsim's `PacketStreamer` requires the attaching side to originate a +`StreamPackets` call with `ChipInfo` before traffic flows, so its consumer is +a protocol-terminating driver, not a raw socket. Forwards carry both kinds; +the difference lives in the driver at the `requires` end. + +### DD-10: Wi-Fi medium forwarding + +**Alternatives considered:** + +1. **Forward the `mac80211_hwsim` frame socket** between members so both + share one simulated medium. +2. **Attach at netsim's 802.11 MAC chip** — Wi-Fi as another chip kind on the + `PacketStreamer` port. +3. **Forward the projection socket at L4** — skip radio simulation and carry + the Android Auto TCP session directly. + +**Decision:** Option 1 as the target, with option 2 taken first where +netsim's Wi-Fi support covers the configuration; option 3 rejected as a +fidelity failure but retained as a diagnostic. + +**Rationale:** Wireless Android Auto's defining behavior is the *handover*: +pair over Bluetooth, then move the session to a peer-to-peer 5 GHz link. A +test that forwards the projection socket at L4 never exercises the handover, +the Wi-Fi Direct negotiation, or the failure modes that matter — it verifies +that a TCP proxy works. So the medium has to be simulated. + +Between 1 and 2, option 2 is far cheaper when it applies, because it reuses +the same forward machinery as Bluetooth. Option 1 is the general answer, +because `--vhost_user_mac80211_hwsim` is the mechanism Cuttlefish documents +for sharing a Wi-Fi medium between separately-launched instances, and +`wmediumd` is where RSSI and delivery modeling live. Option 1 is also the +hardest thing in this JEP and the most likely to need upstream work — it is +scheduled last (Phase 4) and its risk is called out explicitly. + +### DD-11: Physical ↔ virtual — gateway exporter + +**Alternatives considered:** + +1. **Gateway exporter with a real radio**, presenting the adapter as a + `provides` port; the virtual side attaches as if to any other controller. +2. **Emulate the physical device's peer in software** and never involve RF. +3. **Require both halves to be the same kind** — no hybrid benches. + +**Decision:** Option 1. + +**Rationale:** A physical device's radio is inside the device; its HCI is not +reachable from outside, so no software path puts a real phone and a simulated +head unit on the same medium without a real radio somewhere. Option 1 puts +that radio in exactly one place and keeps it a normal exporter with a normal +driver, so it schedules, leases, and reports like everything else. + +Option 2 is a useful *test double* but is not a hybrid bench: it is a virtual +bench with a hand-written model of the physical device, and it cannot find +bugs in the physical device's stack. Option 3 gives up the differentiator in +the Motivation. + +The consequence is stated plainly: hybrid benches require lab hardware and +physical proximity between the gateway and the device, and the gateway is a +shared RF resource that must be modelled as such. + +### DD-12: Access policy and port validation timing + +**Alternatives considered:** + +1. **Per-member policy evaluation, bind-time port validation.** Each member + is evaluated against `ExporterAccessPolicy` exactly as a standalone lease + would be; forwards are validated against the bound exporters' reports. +2. **Selection-time port validation** — ports surfaced as exporter CR labels + so member selectors only match exporters that have the required ports. +3. **A bench-shaped policy CRD** with rules over lease shape, size, and count. + +**Decision:** Option 1 for v1; option 2 recorded as a follow-on that depends +on JEP-0015; option 3 deferred. + +**Rationale:** Per-member evaluation is the least surprising and most secure +default: a multi-member lease can never reach an exporter its client could +not have leased directly, which makes the feature non-escalating by +construction. The lease's effective priority is the minimum across members +and its maximum duration the minimum of the per-member `maximumDuration` +values, so a bench never outlives or outranks its most restricted member. +`status.members[].priority` records each member's own value so the +aggregation is auditable. + +Port validation is bind-time in v1 because of a concrete constraint: lease +selection matches `labels.Set(exporter.Labels)` — **CR metadata labels** — +while reported ports live in `ExporterStatus.Devices[]`. Ports are therefore +visible to the controller *after* binding but not selectable *before* it. The +bind-time check (both named ports exist, directions are complementary, +declared protocols agree) needs no new machinery and catches every +misconfiguration, at the cost of holding the devices while it reports +`Invalid` — which is arguably better for debugging anyway. + +Option 2 is strictly better and worth doing: it would make a bench whose +ports cannot be satisfied report `Unsatisfiable` **without holding +anything**, consistent with DD-1's all-or-nothing property. It requires +reported device information to be reconciled into selectable exporter labels, +which is precisely the mechanism proposed in JEP-0015. Hand-maintained CR +labels are a stopgap but drift from reality, so this JEP does not depend on +them. + +Option 3 is a real requirement but separable, and should be designed once +there is operational experience with what benches people actually build. + +## Design Details + +### Binding: one pass, one write + +The existing `reconcileStatusExporterRef` generalizes to +`reconcileStatusMembers`, keeping its selection pipeline intact per member: + +```text +for each member in spec.members: + approved = policy-approved exporters matching member.selector + online = filter offline + unleased = filter out exporters claimed by other active leases + AND exporters already picked earlier in this same pass + ready = filter out exporters still cleaning up a prior lease + candidate[member] = best(ready) # in memory only, nothing written yet + +if any required member has no candidate: + set Pending/Unsatisfiable with the failing role named + write NO member claims + requeue + +else: + status.members = candidates # all of them + status.priority = min(member priorities) + single Status().Update() # atomic +``` + +Two properties follow, and they are the reason for DD-1: + +- **No partial holds.** Candidates live in memory until every required member + resolves. A lease that cannot complete writes no claims at all, so it holds + nothing while it waits and there is nothing to release on timeout. +- **No self-collision.** Because the pass excludes exporters already chosen + for an earlier member, two roles can share a selector without both + resolving to the same exporter — which is what a `phone` + `phone` + two-handset bench needs. + +The single-exporter path is the same code with one member: `spec.selector` is +normalized into a synthetic member at admission, and on success the result is +written to `status.exporterRef` rather than `status.members` (DD-2). There is +one selection implementation, not two. + +**What is unchanged, and still imperfect.** Exclusivity is still a read-scan +of other leases' claims against a possibly-stale cache, so two leases can +race for one exporter and both write, because they write different objects. +This is pre-existing and neither improved nor worsened here; the loser is +detected on a later reconcile and re-bound. + +### Lease state + +```text + ┌─────────┐ + │ Pending │◄────────────┐ requeue: some member unavailable + └────┬────┘ │ (nothing held) + │ │ + ┌──────┴───────┐ │ + ▼ ▼ │ +┌──────────────┐ ┌────────────────────┐ +│Unsatisfiable │ │ all members bound │──┘ +└──────────────┘ │ (one status write) │ + └─────────┬──────────┘ + │ forwards validated + requested + ▼ + ┌────────────┐ forward failure ┌──────────┐ + │ ForwardsUp │──────────────────►│ Degraded │ + └─────┬──────┘ └──────────┘ + │ all forwards connected + ▼ + ┌────────────┐ member lost ┌──────────┐ + │ Ready │───────────────►│ Degraded │ + └─────┬──────┘ └──────────┘ + │ release / expiry / spec.release + ▼ + ┌────────────┐ + │ Ended │ + └────────────┘ +``` + +Conditions reuse the existing `LeaseConditionType` values — `Pending`, +`Ready`, `Unsatisfiable`, `Invalid` — with `ForwardsReady` and `Degraded` +added. `Ready` for a multi-member lease requires all required members bound +*and* all declared forwards connected. Expiry, `status.ended`, the +`jumpstarter.dev/lease-ended` label, and `spec.release` behave exactly as for +a single-exporter lease, because it is the same object and the same code. + +### Forward validation and establishment + +Validation happens after binding, because the report that proves a port +exists belongs to a bound exporter, not to a selector (DD-12). For each +`spec.forwards[]` entry the controller checks, against +`ExporterStatus.Devices[].Ports`: + +1. `from.member` and `to.member` name declared members — enforced by CEL at + admission, before this point. +2. Both named ports exist on the respective bound exporters. +3. `from` resolves to a `PROVIDES` port and `to` to a `REQUIRES` port. +4. If both ports declare `protocol`, the values are equal (DD-8). + +A failure sets `Invalid` with the offending forward and reason named, and +leaves the members bound so the user can inspect the bench rather than +staring at a lease that silently refuses to bind. + +Establishment then reuses the existing port-forward primitives: + +1. The controller instructs both exporters over their existing `Listen` + streams. +2. Each calls `DialPeer(lease, forward, member)`. +3. The controller mints two tokens with an identical `stream` claim derived + from `(lease UID, forward name)` — a UUIDv5 in a fixed namespace, so the + value is stable across reconciles and both ends compute the same + rendezvous without coordination. `aud: jumpstarter router`, + `sub: jumpstarter exporter`, expiry clamped to the lease's end time. +4. Both call `RouterService.Stream`. +5. The `provides` side dials its local service and splices with + `forward_stream`. The `requires` side opens a `TemporaryTcpListener` on + its declared `listen` address and splices each accepted connection. +6. **Fast path**: with a `peer_endpoint`/`peer_token`, each exporter races a + direct dial against the router path; first handshake wins, loser is reset. + The direct listener authenticates `peer_token`, so a direct forward is not + a weaker trust boundary. `mode: direct` fails rather than falling back; + `mode: router` never attempts it. + +**Failure modes and handling:** + +| Failure | Behavior | +| --- | --- | +| Named port absent on a bound exporter | Lease `Invalid`; members stay bound for inspection | +| `provides → provides` or `requires → requires` | Lease `Invalid` (DD-6) | +| Declared protocols disagree | Lease `Invalid` (DD-8) | +| Protocols differ but neither declared | Connects, fails at first byte — accepted (DD-8) | +| Forward references an undeclared member | Rejected by CEL at admission; lease never created | +| `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | +| Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | +| Direct dial fails (fast path) | Silent fallback to the router path; recorded as a metric | +| A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | +| Client releases the lease | Forwards torn down first, then the lease ends normally | + +### Concurrency and ordering + +Reconciliation is single-writer per lease (standard controller-runtime work +queue), so no intra-lease locking is needed, and member selection is pure +computation followed by one write. Forward splicing on the exporter side runs +in the existing per-driver task group, so a stalled forward cannot block +driver calls on other children. + +Ordering between forwards and drivers is safe by construction: forwards come +up before `Ready`, and a `requires`-side driver only dials when the client +invokes it — `bt-peer` connects its transport on `start()`, well after the +lease is Ready. + +Because binding is a single status write, the read-your-writes hazard a +two-object design would face does not arise. + +### Security + +- **No privilege escalation by construction** (DD-12): every member is + evaluated against `ExporterAccessPolicy` exactly as a direct request would + be, so a multi-member lease reaches only exporters its client already could. +- **Forward authorization is lease-scoped**: `DialPeer` verifies the calling + exporter is currently bound to the named member of the named lease and that + the named forward lists it. Tokens expire with the lease. +- **A forward is a data path between two devices under one client's control**, + not a general exporter-to-exporter tunnel: an exporter can reach only a peer + a lease it is bound to explicitly names, and only through the named port. +- **Ports are opt-in surface.** A driver that declares no ports cannot be + forwarded to or from. A `requires` port is the only new inbound socket, it + binds an address the exporter chose, and it exists only while the lease + holds it. +- **The fast path is authenticated** — a direct peer connection presents + `peer_token`; it is not trusted for being on the same network. Exporters + supporting it open a listener, disabled by default and enabled per exporter + configuration. +- **`members` is immutable after creation**, so a bound lease cannot be + widened beyond what it was authorized for. +- **Physical RF is not access-controlled.** A gateway exporter's radio is + audible to anything in range; labs must treat RF proximity as a trust + boundary. + +### Observability + +JEP-0013 telemetry gains `lease.member` as a span attribute wherever +`lease.name` already appears, so per-role activity is separable within one +lease, plus per-forward counters (bytes each direction, reconnects, mode +actually used, direct-dial fallback rate). Time-to-bench (lease create → +`Ready`) is the headline metric; it is the existing lease-acquisition metric +extended to record the member count. + +## Test Plan + +### Unit Tests + +- `LeaseSpec` CEL validation: extended one-of rule, members/scalar mutual + exclusion, role-name uniqueness, forwards referencing declared members, + member immutability. +- Member selection: all-or-nothing binding, no self-collision, correct + `Unsatisfiable` role naming, and — the property DD-1 rests on — that a + reconcile which cannot satisfy every member writes **no** member claims. +- Single-exporter regression: existing lease controller tests pass + unmodified, and a one-member lease produces the same `status.exporterRef` + as the equivalent scalar lease. +- Aggregation: priority = min(member priorities), duration clamped to + min(member `maximumDuration`). +- Port report round-trip: driver-declared ports reach + `ExporterStatus.Devices[].Ports`; an exporter reporting none is treated as + non-forwardable; a report with no `ports` field is accepted unchanged. +- Forward validation: missing port, `provides→provides`, + `requires→requires`, protocol disagreement, and `listen` collision each + produce `Invalid` with the offending forward named. +- `Dial` without `member_name` on a multi-member lease returns + `INVALID_ARGUMENT` listing roles; with a valid role, routes correctly. +- `DialPeer` token issuance: identical `stream` claim for both ends, + stability across reconciles, expiry clamped to lease end, rejection when + the caller is not bound to the named member. +- Python client: role attribute access, `connect()` returning a bare client + for single-member and a role mapping for multi-member, + `bench.forwards[...]` state, raising on a `Degraded` role. + +### Integration Tests + +Against a kind cluster with the controller and mock exporters (`e2e/`): + +- Two mock exporters, one two-member lease, one forward between an + `EchoNetwork` provides-port and a requires-port — end-to-end establishment + through the real router with no device-specific code. +- Contention: more concurrent multi-member leases than capacity, asserting + every lease is either fully bound or holding nothing. +- Lease expiry, explicit release, and client disconnect; assert no leaked + router streams, no exporters left claimed, and no listeners left bound. +- Router-mode vs. direct-mode selection, including forced fallback. +- `jmp get lease -o mobly` output validated against Mobly's testbed schema. +- **Compatibility**: an N-1 client against an N controller for the full + single-exporter workflow; an N client issuing a single-exporter lease + against an N-1 controller; an N-1 *exporter* (reporting no ports) + registering against an N controller. + +### Hardware-in-the-Loop Tests + +- **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in + separate Pods (JEP-0016 `ExporterSet`), forwarded at rootcanal/netsim. + Assert BT discovery and pairing, and — Phase 4 — Wi-Fi association. + Runnable in CI on KVM-capable nodes with no lab hardware. This is the + headline result and should run on every merge once it exists. +- **Physical ↔ physical**: a physical phone and head unit on two exporters + **on different lab hosts** — the allocation ATS's `SimpleScheduler` + refuses. Requires lab hardware; runs on a labeled runner. +- **Hybrid**: physical phone + gateway exporter (USB HCI dongle) + virtual + head unit in-cluster. +- **Latency characterization**: HCI round-trip through a router forward vs. a + direct forward vs. host-local rootcanal, reported as a distribution. Its + result determines whether A2DP-class workloads are in scope for router mode. + +### Manual Verification + +- `jmp shell --lease` ergonomics: role-prefixed driver calls, + `j forward status`, Ctrl+C teardown leaving no held leases. +- `jmp get exporter` port discovery: a user who has never seen an exporter's + config can construct a working forward from its output alone. +- An existing Mobly multi-device suite run unmodified against an exported + testbed. +- `jmp get leases` and `kubectl get leases` on a mixed set of single- and + multi-member leases. + +## Acceptance Criteria + +**Lease plane** + +- [ ] `spec.members[]` / `status.members[]` on `Lease`, with extended CEL + validation and member immutability +- [ ] Binding is one `Status().Update`: a reconcile that cannot satisfy every + required member writes no member claims (verified by test) +- [ ] Two members with identical selectors bind two distinct exporters +- [ ] Contention test: concurrent multi-member leases over insufficient + capacity leave no lease partially holding devices +- [ ] `ExporterAccessPolicy` evaluated per member; a multi-member lease + cannot reach an exporter its client could not lease directly +- [ ] Lease priority = min(member priorities); duration clamped to + min(member `maximumDuration`) +- [ ] `status.exporterRef` unchanged for single-exporter leases and nil for + multi-member ones; existing lease controller tests pass unmodified +- [ ] `Dial` without `member_name` on a multi-member lease returns + `INVALID_ARGUMENT` naming the roles + +**Ports and forwards** + +- [ ] `PortReport` is optional on `DriverInstanceReport`; exporters reporting + no ports register and operate unchanged +- [ ] Declared ports appear in `ExporterStatus.Devices[].Ports` and in + `jmp get exporter` output +- [ ] `listen` addresses never appear in any report, CR status, or lease spec +- [ ] Forward validation rejects missing ports, `provides→provides`, + `requires→requires`, and declared-protocol mismatch, naming the forward +- [ ] Forwards establish over `RouterService` with no `router.proto` change +- [ ] Direct fast path is authenticated, falls back automatically, and is + observable (mode + fallback-rate metrics) +- [ ] `bt-peer` participates as a `requires` endpoint with **no Python + changes** — exporter configuration only +- [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` + integration test + +**Topologies** (each a phase gate, in order) + +- [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods on + separate nodes complete BR/EDR discovery and pairing over a forwarded + rootcanal/netsim port, in CI +- [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit + on exporters on different lab hosts complete Android Auto projection +- [ ] **Phase 3 — Hybrid**: a physical phone pairs with a virtual head unit + through a gateway exporter +- [ ] **Phase 4 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a + forwarded `mac80211_hwsim`/`wmediumd` medium, and an Android Auto + session completes the Bluetooth → Wi-Fi handover end to end +- [ ] Measured HCI round-trip latency through a router forward is published, + with a documented statement of which workloads it does and does not + support + +## Graduation Criteria + +### Experimental + +`members`, `forwards`, and port reporting ship behind a controller feature +gate, with Phases 1–2 complete. Signals sought: do real benches stay at two +members or grow; how often does the direct fast path apply; does anyone hit +`MaxItems=8`; how often do `listen` collisions occur on physical hosts; does +the nil-`status.exporterRef` convention (DD-2) surprise any consumer; is +bind-time port validation (DD-12) painful enough to justify accelerating the +JEP-0015 dependency. + +### Stable + +- Phases 1–4 complete, with Phase 4 green in CI for 30 consecutive days +- At least two `requires`-side drivers outside this JEP's reference set + (evidence the port model generalizes) +- No API changes to `members` / `forwards` / `PortReport` for one release + cycle +- Selection-time port validation either shipped on JEP-0015 or explicitly + deferred with a rationale + +## Backward Compatibility + +This proposal is **additive**, and deliberately concentrates compatibility +risk in one field, handled by DD-2. + +- **CRD**: `Lease` gains two optional spec lists and two optional status + lists; `ExporterStatus.Devices[]` gains an optional `ports` list. No + existing field changes type, meaning, or default. `Exporter`, + `ExporterAccessPolicy`, `ExporterSet`, and `VirtualTargetClass` are + otherwise untouched. Every lease that exists today validates unchanged. +- **`status.exporterRef`**: unchanged for single-exporter leases — every + lease created before this feature and every one created after that does not + pass `members`. Multi-member leases leave it nil, which existing consumers + already read as "not bound yet" (DD-2). The JEP-0016 façade, + `jmp get leases`, the `Dial` path, and JEP-0013 telemetry need no changes + to keep working correctly; they need changes only to *support* benches. +- **Driver report**: `ports` is a new optional repeated field. An exporter + built before this JEP reports nothing and is treated as non-forwardable — + the correct answer. No version negotiation, no migration. +- **Drivers**: unchanged. Ports are declared in exporter configuration; the + reference `requires` endpoint (`bt-peer`) needs no Python changes at all. + A driver with no ports is simply not a forward endpoint. +- **Protocol**: three new fields on existing messages and one new RPC. + Unknown fields are ignored by proto3, so an N-1 client talks to an N + controller unchanged. An N client requesting `members` from an N-1 + controller has them silently dropped — so the client probes for `DialPeer` + (or a controller version) and fails with a clear message rather than + acquiring a one-device lease it will misuse. +- **Operator upgrade**: a CRD schema addition, a standard bundle bump with no + conversion webhook. Rolling back removes multi-member leases; single-exporter + leases are unaffected, because they are ordinary leases in every respect. +- **Coexistence**: single- and multi-member leases share one exporter pool, + one scheduler, and one selection implementation. + +## Consequences + +### Positive + +- Multi-device testing becomes a first-class Jumpstarter concept, with atomic + acquisition, shared lifetime, and no deadlock. +- **Acquisition atomicity is structural, not engineered.** All of a lease's + claims are in one object written once, so there is no partial-hold state, + no acquisition timeout, no release-and-retry, and no gang scheduler. +- **A bench can span hosts** — the specific allocation ATS's `SimpleScheduler` + refuses, and the reason multi-device testing is capped at one lab machine + everywhere today. +- **The data plane is existing code.** Forwards are `TemporaryTcpListener` + + `forward_stream` with a router peer stream substituted for a client stream; + `router.proto` is untouched and no driver changes. +- A bench is a lease, so `jmp create lease`, expiry, `spec.release`, access + policy, telemetry, and `kubectl get leases` all apply unchanged. No second + resource, no second RBAC surface, no second lifecycle. +- **Ports are discoverable**, so a client can construct a forward from + `jmp get exporter` output instead of tribal knowledge or someone's YAML. +- Android Auto projection becomes expressible, and in the virtual-to-virtual + form expressible *in CI without a lab*. +- Cross-host virtual device pairing, which no shipping tool does, reduces to + forwarding a socket. +- Hybrid benches let labs virtualize the abundant half of a bench and keep the + scarce half real. +- Nothing here is Android-specific: CAN cross-connects between two ECUs, + serial cross-overs, and SOME/IP peer benches are all ports and forwards. +- The exporter = DUT invariant survives; JEP-0016 DD-8's option 1 becomes + available. + +### Negative + +- **`LeaseSpec` now has two shapes.** Scalar and members forms are mutually + exclusive, enforced by CEL, on the most heavily validated object in the API. + Every future change to lease semantics must reason about both. +- **`status.exporterRef` becomes conditional** — set only for single-exporter + leases. A documented, fail-safe convention (DD-2), but a subtlety every new + consumer must learn. +- **Ports are a third place to configure things**, alongside driver config and + labels, and a `requires` port hard-codes a local address that can collide. +- **Protocol mismatch can fail late** when neither port declares `protocol` + (DD-8) — a deliberate retreat from admission-time validation. +- **Port validation happens after binding** (DD-12), so an unsatisfiable + forward holds devices while reporting `Invalid`, until JEP-0015 makes ports + selectable. +- **No per-member early release.** A client finished with one device cannot + hand it back without ending the lease. +- **A new inbound socket on exporters** — the `requires` listener, plus the + optional direct fast-path listener. +- **Latency is a first-class risk**, not a footnote, and some timing-sensitive + protocols may not work in router mode. +- **Hybrid benches need new lab hardware** and physical proximity. +- **Phase 4 depends on upstream behavior** we do not control. + +### Risks + +- **Wi-Fi frame forwarding may not be viable over the router.** `wmediumd` + assumes medium-like timing; head-of-line blocking on a TCP substrate may + make association flaky or impossible except on the direct fast path. + Mitigation: Phase 4 is last, may conclude "direct mode only", and the + datagram work in Future Possibilities is the escalation path. +- **Bluetooth timing may be tighter than measured** — pairing may work while + A2DP streaming does not. Mitigation: latency characterization is an + acceptance criterion whose answer is published, not assumed. +- **`listen` collisions on physical hosts.** A lab host already running the + service a `requires` port wants to shadow cannot host that endpoint. + Mitigation: validation names the conflict explicitly; ephemeral-port + allocation is a Future Possibility. +- **Multi-member leases amplify the pre-existing binding race** — N chances + to lose a race, so re-bind churn grows with bench size. Mitigation: + `MaxItems=8` bounds it; the real fix is the global scheduler the code + already has a `TODO` for. +- **The scalar/list convention may leak** — some consumer may assume a bound + lease always has `status.exporterRef`. Mitigation: fail-safe by design, and + the compatibility matrix covers known consumers. +- **netsim and `vhost_user_mac80211_hwsim` are internal-ish interfaces** that + upstream may change. Mitigation: attach at documented seams, pin the + runtime image, treat divergence as a contribution opportunity. +- **Bench sprawl.** Mitigation: per-member policy limits reach, `MaxItems=8` + bounds size, bench-level quota is an explicit Future Possibility. + +## Rejected Alternatives + +- **Not doing this.** Multi-device testing stays a client-side workaround with + no atomicity and no device-to-device path, JEP-0016 DD-8 stays blocked, and + Jumpstarter inherits the same one-host ceiling as every existing tool. +- **A separate `LeaseGroup` CR owning N child leases** — DD-1. An earlier + draft proposed exactly this; it was rejected once the binding path was + examined, because N members in one object bind in one write whereas N child + leases make partial acquisition a durable state requiring machinery that + the single-object design never needs. +- **Naming it `LeaseSet`.** `Set` already means ReplicaSet/HPA semantics + here: `ExporterSet` (JEP-0014) is N *identical* instances from a template + with `minReplicas`/`maxReplicas`. A bench is heterogeneous members addressed + by role, so `LeaseSet` beside `ExporterSet` would actively mislead. (The + Kubernetes name for gang-scheduled heterogeneous members is `PodGroup`, + hence the earlier `LeaseGroup` — but DD-1 removes the need for any new kind.) +- **Promoting `status.exporterRef` to a list** — DD-2. +- **Raw addresses in the lease spec** — DD-6. Leaks exporter-private + architecture into a client-authored manifest and breaks when a lab + re-addresses a service. +- **Untyped endpoints without direction** — DD-6. Cannot reject the + controller-to-controller wiring that is the most likely mistake. +- **A `medium` taxonomy** (`bluetooth | wifi | uwb | …`) — DD-8. Validates + what traffic represents rather than what it is, and would approve a + raw-HCI-to-netsim forward, the one pairing most needing rejection. +- **A mandatory wire-format token** — DD-8. Right axis, but still approves + controller-to-controller, and imposes a taxonomy to define and version for + a check that direction already makes. +- **A dedicated `LinkEndpoint` driver interface and bridge-driver tier** — + DD-9. Invented machinery for something existing network drivers plus a + direction already express; a `TcpNetwork` child at `127.0.0.1:7300` is + configuration, not code. +- **Encoding ports as driver labels** — DD-7. Zero schema change, but a + `requires` port has no backing driver instance to label, forcing phantom + report entries and two parallel key namespaces. +- **Client-relayed forwards** — DD-4. Retained as an explicit + `mode: client-relay` debug transport, not the architecture. +- **Direct peer-to-peer only** — DD-4. Fails on the hybrid topology that + motivates the design. +- **A peer-specific RPC in `router.proto`** — DD-5. +- **Guest-side Bluetooth/Wi-Fi shims** — DD-9. Changes the device under test. +- **L4 forwarding of the Android Auto projection socket** — DD-10. Skips the + handover, which is the thing under test. Kept as a diagnostic. +- **N independently-leased devices behind one exporter.** Ruled out by + JEP-0016's exporter = DUT invariant. A group as a single composite DUT + (JEP-0016 DD-8 option 2) remains legitimate and orthogonal. +- **Building a Jumpstarter multi-device *test runner*.** This JEP stops at the + bench. Tradefed, Mobly, and pytest already run multi-device tests well; + Jumpstarter's contribution is the bench they run against. +- **Adopting ATS/OmniLab as the fleet layer.** Mobile Harness's `Device` + abstraction is ADB-shaped and runs in the lab-server JVM beside the device, + its OSS platform coverage is Android-only, and extending it means in-tree + Java and Bazel. Adopting it would import the one-`LabLocator` constraint + this JEP removes. The complementary direction — Jumpstarter *under* ATS via + a Mobile Harness `Device` or Mobly-controller shim, so existing results + pipelines keep working — is a Future Possibility, not a rejection. + +## Prior Art + +- **OmniLab Android Test Station / Mobile Harness** (`google/device-infra`) + is the closest system and the one to be precise about. It **does** + gang-schedule multi-device jobs: `AdhocTestbedSchedulingUtil` performs + maximum-cardinality bipartite matching of `SubDeviceSpec`s onto idle + devices, with `TestbedDevice`, `CompositeDevice`, and + `SubDeviceSynchronizationDriver` all present, and it has a multi-host mode + (one controller, N workers) for fleet pooling. The structural limit is in + `SimpleScheduler.allocate`: every device locator in an allocation must + share one `LabLocator`, so a multi-device job cannot span lab hosts. That + single check is the gap this JEP targets. *Terminology note for reviewers:* + a Mobile Harness `Driver` is a **test runner** (TradefedTest, MoblyTest), + not a device abstraction — its `Device`/`BaseDevice` is the analog of a + Jumpstarter driver. +- **Tradefed multi-device** contributes the role/requirement declaration + pattern (``) and `multi_target_preparer` for coordinated + setup such as Bluetooth pairing, and documents its own gap verbatim: *"No + APIs yet exist to conduct operations from one device to another, such as + `device1.sync(device2)`."* +- **LAVA MultiNode** is the closest HiL prior art: one job spans multiple + devices with named roles, and — notably — makes the multi-device unit *the + job itself* rather than a wrapper around N jobs, which independently + supports DD-1. Its synchronization primitives (`lava-sync`, `lava-send`, + `lava-wait`) coordinate *test scripts*, not devices; there is no bridged + medium between roles. +- **Mobly** contributes the testbed format this JEP exports to. +- **`kubectl port-forward`** is the mental model for a forward, including its + guarantee — it delivers bytes to a socket and does not verify the consumer + speaks the protocol (DD-8). +- **Cuttlefish multi-instance connectivity** (`--num_instances`, shared + `--vhost_user_mac80211_hwsim`, netsim, rootcanal, `wmediumd_control`) + defines what "correct" looks like for the virtual media this JEP forwards. +- **The Android emulator networking backplane (36.5)** independently + validates the demand, and its same-host scope defines the boundary this JEP + moves. +- **`bumble`** contributes the virtual-controller model and is already a + Jumpstarter dependency via `jumpstarter-driver-bt-peer`, whose existing + `tcp-client:127.0.0.1:7300` transport is the proof that a driver needs no + changes to become a `requires` endpoint. +- **Kubernetes gang scheduling** (Volcano `PodGroup`, coscheduling, Kueue + `Workload`) is the reference for what DD-1's option 2 would have required. + Worth noting *why* Kubernetes needs it and this JEP does not: a Pod's + placement is recorded on the Pod, so N Pods are N objects and gang + scheduling must be layered on top. A Jumpstarter lease records its claims on + itself, so N devices can be one object. + +## Unresolved Questions + +To resolve during review: + +- **Can two CVDs be joined at rootcanal directly, or is netsim required?** + HCI is host-to-controller, and two rootcanals are both controllers — so the + Phase 1 topology may need netsim in the middle as the component that + presents a controller to multiple hosts. This determines whether Phase 1's + forward is `rootcanal → controller` or `netsim → chip`, and it is a + prototype question, not a design one. +- **Should the scalar and members forms really be mutually exclusive?** The + alternative is `spec.selector` as a default for members that omit their + own — convenient for homogeneous benches, but two ways to express one thing. +- **How should `listen` collisions be handled?** A fixed address keeps drivers + unchanged but can collide on a physical host; an ephemeral port avoids + collisions but requires telling the driver its address, reintroducing + coupling. Currently fixed-and-validated. +- **Should a lease-level "sync" primitive exist?** LAVA provides `lava-sync`. + Cross-role barriers are implementable in the client library, but a + controller-mediated barrier would work across independently-driven roles. +- **Where does the exported Mobly testbed get its ADB endpoints?** Local + forwards from `jmp shell --lease` couple the export to a live session; a + long-lived per-member forward managed by the client library is the + alternative. +- **How are gateway exporters modeled?** As a member with its own role + (schedulable and auditable, but benches become three members where users + think in two), or as an attribute of the physical member's exporter? + +To resolve during implementation: + +- Exact `ListenResponse` variant shape for forward setup instructions. +- Whether the fast path should race the router dial or attempt direct first + with a short timeout — a measurable question. +- Whether forward reconnect should preserve the medium's logical state or + force a fresh pairing; likely protocol-specific. +- How `jmp get leases` renders N exporters in a table column readably. + +## Future Possibilities + +Not part of this proposal: + +- **Selection-time port validation** via JEP-0015 dynamic exporter labels, so + a bench whose ports cannot be satisfied reports `Unsatisfiable` without + holding anything (DD-12). +- **A global lease scheduler** with a view of all leases and exporters, which + the controller already carries a `TODO` for; it would close the binding race + for single- and multi-member leases alike. +- **Ephemeral `listen` allocation** for `requires` ports, removing the + collision constraint at the cost of driver coupling. +- **Fan-out forwards** — one `provides` port serving several `requires` ports, + for shared media with more than two participants. Today's forward is + strictly point-to-point. +- **Bench-level access policy and quota** (DD-12). +- **Per-member early release**, if the all-or-nothing lifetime proves coarse. +- **Datagram forwards.** `wmediumd` and other frame-oriented media want + datagram semantics with real boundaries and no head-of-line blocking; the + additive `FRAME_TYPE_DATAGRAM` extension to `RouterService.Stream` (and, + further out, QUIC unreliable datagrams) is a separate protocol JEP, for + which Phase 4 is the most compelling justification. +- **Jumpstarter under ATS** — a Mobile Harness `Device` or Mobly-controller + shim backed by a Jumpstarter lease, so Google's results pipeline keeps + working while gaining non-Android and cross-host devices. Complements + JEP-0016's Cloud Orchestrator seam. +- **A `Bench` template CRD** — a reusable named topology instantiated by + reference, the way `VirtualTargetClass` is to `ExporterSet`. A *template*, + not a second lease-like object, so it does not reopen DD-1. +- **Spawned-on-lease members** — a member satisfied by provisioning a new + JEP-0014 pool instance on demand, making a bench elastic in its virtual half. +- **Agent-facing bench skills** — an agent leasing a two-device bench to + reproduce an interaction bug, following JEP-0016's agent-native framing. + +## Implementation History + +- 2026-09-01: JEP drafted + +## References + +- [JEP-0014: Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) + — "Composite leases — multiple exporters linked into one logical lease" + (Future Possibilities), which this JEP realizes +- JEP-0015: Dynamic Exporter Labels — the mechanism selection-time port + validation depends on (DD-12) +- JEP-0016: Cuttlefish Kubernetes-Native Orchestration — DD-8, whose option 1 + this JEP implements +- [JEP-0013: Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) +- [JEP-0011: Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) + — the introspection direction port reporting extends +- [google/device-infra](https://github.com/google/device-infra) — Mobile + Harness, the ATS 2.0 engine (`SimpleScheduler`, `AdhocTestbedSchedulingUtil`) +- [OmniLab Android Test Station user guide](https://source.android.com/docs/core/tests/development/android-test-station/ats-user-guide) +- [Virtual devices in OmniLab ATS](https://source.android.com/docs/core/tests/development/android-test-station/ats-virtual-devices) +- [Tradefed: run tests with multiple devices](https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/multi-device) +- [Cuttlefish: test connectivity of multiple devices](https://source.android.com/docs/devices/cuttlefish/connectivity) +- [Mobly](https://github.com/google/mobly) — [testbed tutorial](https://github.com/google/mobly/blob/master/docs/tutorial.md) +- [Bumble, a Python Bluetooth stack](https://google.github.io/bumble/) — + [transports](https://google.github.io/bumble/transports/index.html) +- [netsim (`platform/tools/netsim`)](https://android.googlesource.com/platform/tools/netsim/) — + `proto/netsim/packet_streamer.proto` +- [google/android-cuttlefish](https://github.com/google/android-cuttlefish) +- [Test Multi-Device Interactions with the Android Emulator](https://android-developers.googleblog.com/2026/04/Test-Multi-Device-Interactions-with-the-Android-Emulator.html) +- [LAVA MultiNode](https://docs.lavasoftware.org/lava/multinode.html) + +--- + +*This JEP is licensed under the +[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0), +consistent with the Jumpstarter project.* diff --git a/docs/source/contributing/jeps/index.md b/docs/source/contributing/jeps/index.md index 03499e797..567afe4e8 100644 --- a/docs/source/contributing/jeps/index.md +++ b/docs/source/contributing/jeps/index.md @@ -38,6 +38,7 @@ For the full process definition, see [JEP-0000](JEP-0000-jep-process.md). | 0011 | [Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) | Accepted | @kirkbrauer (Kirk Brauer) | | 0013 | [Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) | Accepted | @mangelajo (Miguel Angel Ajo Pelayo) | | 0014 | [Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) | Approved | @mangelajo (Miguel Angel Ajo Pelayo) | +| 0017 | [Multi-Exporter Leases and Inter-Exporter Port Forwarding](JEP-0017-multi-exporter-leases-port-forwarding.md) | Draft | @kirkbrauer (Kirk Brauer) | ### Informational JEPs @@ -72,4 +73,5 @@ JEP-0010-renode-integration.md JEP-0011-protobuf-introspection-interface-generation.md JEP-0013-observability-telemetry-logs.md JEP-0014-virtual-scalable-exporters.md +JEP-0017-multi-exporter-leases-port-forwarding.md ``` From f80a45068d600d7570cb32d0f54fd3fd248e0bb3 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:16:34 -0400 Subject: [PATCH 02/26] docs: JEP-0017 generalize beyond Android Auto to phone projection and other media The mechanism is medium- and platform-agnostic, but the draft read as an Android feature and would have been reviewed as a niche one. Reframe the problem around the recurring shape -- two devices held at once plus a path between them -- with phone projection as the worked example rather than the motivation itself. Motivation now opens with a table of domains sharing that shape (phone projection, wireless peripherals, automotive networks, device-to-device apps, serial harnesses) and states that the pairing-then-handover pattern is a property of projection generally, covering Android Auto and CarPlay and head units running Android, Linux, or QNX. Add a second worked example with no radios in it: two ECUs sharing a CAN segment via a socketcand-style bridge, expressed with the same members and forwards fields and validated by the same provides/requires rules. Broaden the media table with CAN and serial cross-over rows so radios are visibly one case rather than the privileged one. Diversify user stories to include a CAN gateway bench and a BLE peripheral bench alongside the projection ones. Rename "Where virtual radios attach" to "Where simulated media attach" and "Attaching virtual radios" to "Attaching media, simulated and physical". Android Auto survives in four places, all as a named instance or the reference implementation for a phase gate, rather than as the premise. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 122 +++++++++++++----- 1 file changed, 92 insertions(+), 30 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 035c863ac..9c2902af1 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -32,7 +32,12 @@ all of its devices or none. The data plane is the existing `TcpPortforwardAdapter` with one substitution — a router peer stream in place of a client stream — so `RouterService` and every driver work unchanged. This realizes JEP-0014's deferred "composite leases — multiple exporters linked into -one logical lease" literally, and takes option 1 of JEP-0016's DD-8. +one logical lease" literally, and takes option 1 of JEP-0016's DD-8. The +worked example throughout is phone projection — a phone and a head unit +pairing over Bluetooth, then handing off to Wi-Fi — but nothing in the +mechanism is radio- or platform-specific: two ECUs joined over CAN, a BLE +peripheral and its gateway, or a plain serial cross-over are the same +declaration with different port names. ## Motivation @@ -47,21 +52,39 @@ That is the right primitive for the majority of HiL work — one board, one harness — but an entire class of tests is about *interaction between devices*, and today Jumpstarter cannot express it. -### The concrete problem: Android Auto phone projection +### The concrete problem: devices that must talk to each other -Android Auto phone projection is a two-device protocol by construction. The -phone and the head unit first pair over **Bluetooth**; the head unit then -hands the session off to a peer-to-peer **5 GHz Wi-Fi** link, because -Bluetooth lacks the bandwidth for continuous video projection. Validating it -means holding a phone and a head unit *simultaneously*, keeping them +A large class of tests is not about a device but about an *interaction* +between two of them. The shape recurs across domains: + +| Domain | Bench | What is exercised | +| --- | --- | --- | +| Phone projection | Phone + head unit | Pairing, then session handover to a high-bandwidth link | +| Wireless peripherals | Peripheral + host, or peripheral + gateway | Advertising, pairing, reconnection, roaming | +| Automotive networks | Two ECUs, or ECU + gateway | Bus arbitration, routing, diagnostics across a segment | +| Device-to-device apps | Two handsets | Discovery, transfer, sync over Bluetooth or Wi-Fi Direct | +| Serial / console harnesses | DUT + companion | Protocol conformance over a cross-over link | + +Every one of these needs the same two things Jumpstarter cannot give: **two +devices held at once**, and **a path between them**. + +**Phone projection is the worked example** used throughout this JEP, because +it exercises the hardest version of both requirements. It is a two-device +protocol by construction: the phone and the head unit first pair over +**Bluetooth**, then the head unit hands the session off to a peer-to-peer +**Wi-Fi** link, because Bluetooth lacks the bandwidth for continuous video. +Validating it means holding both devices *simultaneously*, keeping them connected, driving both, and asserting on both — a phone-only or -head-unit-only lease cannot observe the handover at all. +head-unit-only lease cannot observe the handover at all. The same structure +holds for Android Auto and for Apple CarPlay, and for a head unit running +Android, Linux, or QNX; the pairing-then-handover pattern is a property of +projection, not of one vendor's stack. This is not hypothetical. Teams running phone-projection validation operate fleets in the ~1000-device range across multiple labs, and the two things -they cannot get from existing tooling are (a) non-Android devices in the same -bench as Android ones — Linux and QNX IVI head units — and (b) pairing two -*virtual* devices to each other. +they cannot get from existing tooling are (a) heterogeneous benches — a Linux +or QNX head unit alongside an Android phone — and (b) pairing two *virtual* +devices to each other. Running that on Jumpstarter today requires the test author to hand-roll everything the lease layer should provide: @@ -168,20 +191,26 @@ throughput win no host-local scheduler can offer. ### User Stories -- **As an** Android Auto QA engineer, **I want to** lease a phone and a head - unit as one bench with roles, **so that** my projection test either gets - both devices or waits — never half a bench, never a deadlock against a +- **As a** phone-projection QA engineer, **I want to** lease a phone and a + head unit as one bench with roles, **so that** my test either gets both + devices or waits — never half a bench, never a deadlock against a concurrent run. -- **As an** AAOS platform developer, **I want to** pair two Cuttlefish +- **As an** automotive platform developer, **I want to** pair two *virtual* devices running in different cluster Pods over Bluetooth and then hand off to Wi-Fi, **so that** I can validate wireless projection in CI without a physical lab or a single fat host. -- **As a** lab operator whose phones and head units are in different racks - or different buildings, **I want** one lease to span them, **so that** my - bench is not limited to devices plugged into the same machine. +- **As a** lab operator whose devices are in different racks or different + buildings, **I want** one lease to span them, **so that** my bench is not + limited to devices plugged into the same machine. - **As a** test author with a mixed bench, **I want** a QNX or Linux head unit alongside an Android phone, **so that** device type is a driver choice rather than a platform limit. +- **As an** embedded engineer testing a CAN gateway, **I want** two ECU + exporters joined over a forwarded bus, **so that** I can exercise routing + between segments without physically cabling them to one machine. +- **As a** BLE peripheral developer, **I want** my DUT leased alongside a + central acting as its phone, **so that** pairing and reconnection are + covered in CI rather than by hand at a desk. - **As a** Jumpstarter user who already knows leases, **I want** a bench to be *a lease*, **so that** everything I know about `jmp create lease`, expiry, release, and access policy carries over without learning a second @@ -295,6 +324,30 @@ already the degenerate case rather than a legacy shape to migrate (DD-2). A member may be marked `optional: true`, in which case the lease binds without it and the role resolves to `None` on the client. +#### A bench with no radios in it + +Nothing above is projection-specific. The same two fields express two ECUs +sharing a CAN segment, where one exporter fronts the bus over TCP (via +`socketcand` or an equivalent bridge) and the other dials it: + +```yaml +spec: + members: + - name: gateway + selector: { matchLabels: { ecu-role: gateway } } + - name: node + selector: { matchLabels: { ecu-role: body-controller } } + forwards: + - name: powertrain-bus + from: { member: gateway, port: can0 } # provides + to: { member: node, port: can } # requires +``` + +Only the port names differ. The controller applies the same validation — +both ports exist, `provides` meets `requires`, declared protocols agree — and +the same forward machinery carries the bytes. A serial cross-over between a +DUT and a companion board is the same shape again. + ### Acquiring and using a bench The existing commands take members and forwards; there is no parallel @@ -436,10 +489,12 @@ with one substitution — a router peer stream in place of the client stream: The client is not in the data path. -### Attaching virtual radios +### Attaching media, simulated and physical -With forwards as the mechanism, "bridging a radio" stops being a special -subsystem and becomes a question of which port each stack exposes: +With forwards as the mechanism, "bridging a medium" stops being a special +subsystem and becomes a question of which port each stack exposes. Radios are +the demanding case, but nothing in the table below is privileged — a CAN bus +or a serial cross-over is the same declaration: | Stack | Port | Direction | Notes | | --- | --- | --- | --- | @@ -448,8 +503,11 @@ subsystem and becomes a question of which port each stack exposes: | netsim | `netsim` | provides | gRPC `PacketStreamer`; attaching side must originate `ChipInfo` | | `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | | Gateway exporter (real adapter) | `hci` | provides | A real radio, presented as HCI | +| `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | +| Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | -Two consequences worth stating plainly. First, HCI is **asymmetric**: a host +Two consequences worth stating plainly, both of which generalize beyond +radios. First, HCI is **asymmetric**: a host attaches to a controller. Forwarding one rootcanal's port into another rootcanal wires controller to controller and nothing happens — which is precisely why forwards are directional and why a `provides → provides` @@ -928,7 +986,7 @@ is the same guarantee `kubectl port-forward` gives, which nobody treats as a defect, but it is a deliberate retreat from admission-time protocol validation and is recorded as such. -### DD-9: Where virtual radios attach +### DD-9: Where simulated media attach **Alternatives considered:** @@ -972,14 +1030,16 @@ the difference lives in the driver at the `requires` end. 2. **Attach at netsim's 802.11 MAC chip** — Wi-Fi as another chip kind on the `PacketStreamer` port. 3. **Forward the projection socket at L4** — skip radio simulation and carry - the Android Auto TCP session directly. + the projection session's TCP connection directly. **Decision:** Option 1 as the target, with option 2 taken first where netsim's Wi-Fi support covers the configuration; option 3 rejected as a fidelity failure but retained as a diagnostic. -**Rationale:** Wireless Android Auto's defining behavior is the *handover*: -pair over Bluetooth, then move the session to a peer-to-peer 5 GHz link. A +**Rationale:** Wireless phone projection's defining behavior is the +*handover*: pair over Bluetooth, then move the session to a peer-to-peer +Wi-Fi link. This holds for Android Auto and CarPlay alike, and it is the +reason the medium cannot be skipped. A test that forwards the projection socket at L4 never exercises the handover, the Wi-Fi Direct negotiation, or the failure modes that matter — it verifies that a TCP proxy works. So the medium has to be simulated. @@ -1366,11 +1426,12 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): separate nodes complete BR/EDR discovery and pairing over a forwarded rootcanal/netsim port, in CI - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit - on exporters on different lab hosts complete Android Auto projection + on exporters on different lab hosts complete a phone projection + session (Android Auto in the reference implementation) - [ ] **Phase 3 — Hybrid**: a physical phone pairs with a virtual head unit through a gateway exporter - [ ] **Phase 4 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a - forwarded `mac80211_hwsim`/`wmediumd` medium, and an Android Auto + forwarded `mac80211_hwsim`/`wmediumd` medium, and a projection session completes the Bluetooth → Wi-Fi handover end to end - [ ] Measured HCI round-trip latency through a router forward is published, with a documented statement of which workloads it does and does not @@ -1452,7 +1513,8 @@ risk in one field, handled by DD-2. resource, no second RBAC surface, no second lifecycle. - **Ports are discoverable**, so a client can construct a forward from `jmp get exporter` output instead of tribal knowledge or someone's YAML. -- Android Auto projection becomes expressible, and in the virtual-to-virtual +- Phone projection becomes expressible — Android Auto, CarPlay, and + head units on Android, Linux, or QNX alike — and in the virtual-to-virtual form expressible *in CI without a lab*. - Cross-host virtual device pairing, which no shipping tool does, reduces to forwarding a socket. @@ -1555,7 +1617,7 @@ risk in one field, handled by DD-2. motivates the design. - **A peer-specific RPC in `router.proto`** — DD-5. - **Guest-side Bluetooth/Wi-Fi shims** — DD-9. Changes the device under test. -- **L4 forwarding of the Android Auto projection socket** — DD-10. Skips the +- **L4 forwarding of the projection session's socket** — DD-10. Skips the handover, which is the thing under test. Kept as a diagnostic. - **N independently-leased devices behind one exporter.** Ruled out by JEP-0016's exporter = DUT invariant. A group as a single composite DUT From beb8960303cff2f844c239ed898012000abd5f12 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:26:34 -0400 Subject: [PATCH 03/26] docs: JEP-0017 drop non-public fleet-scale claim from motivation The motivation cited phone-projection fleets in a specific size range across multiple labs. That figure is not public and should not ship in a JEP. Remove the sourcing sentence entirely and keep only the two technical gaps it was there to support -- heterogeneous benches, and pairing two virtual devices -- stated as properties of the problem rather than as reported from any particular fleet. The argument does not depend on the number. Assisted-by: Claude Signed-off-by: Kirk Brauer --- .../JEP-0017-multi-exporter-leases-port-forwarding.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 9c2902af1..2c71bda89 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -80,11 +80,9 @@ holds for Android Auto and for Apple CarPlay, and for a head unit running Android, Linux, or QNX; the pairing-then-handover pattern is a property of projection, not of one vendor's stack. -This is not hypothetical. Teams running phone-projection validation operate -fleets in the ~1000-device range across multiple labs, and the two things -they cannot get from existing tooling are (a) heterogeneous benches — a Linux -or QNX head unit alongside an Android phone — and (b) pairing two *virtual* -devices to each other. +Two capabilities are missing from existing tooling, and projection tests need +both: **heterogeneous benches** — a Linux or QNX head unit alongside an +Android phone — and **pairing two virtual devices to each other**. Running that on Jumpstarter today requires the test author to hand-roll everything the lease layer should provide: From fec5ef04969c3f99959b9efbfcc13b49046e0662 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:37:45 -0400 Subject: [PATCH 04/26] docs: JEP-0017 ground media handling in real drivers, Bumble, and the deployment model Three corrections and additions, all replacing speculation with verified fact. Deployment assumptions. Add an explicit Design Details subsection stating the invariant the design silently relied on: each exporter owns its own network namespace, either a container running one exporter or a single-exporter edge device. JEP-0016 supplies this by construction for virtual targets -- one CVD per Pod under a JEP-0014 ExporterSet, and a Pod is a namespace -- so nobody has to arrange it. Three properties follow: requires-ports can bind fixed local addresses (making zero-driver-changes structural rather than lucky), control-plane blast radius equals lease scope, and DD-4's two transports map onto the two deployment shapes. Note the exception: the cuttlefish container recipe's --network=host is fine for a single-exporter host and is not a supported basis for multiple exporters on one machine. Downgrade the listener collision hazard from a risk to a stated precondition accordingly. Real drivers, not invented ones. Split the media section into data plane (ports a forward carries) and control plane (drivers that configure and observe a medium without carrying it). Drop the invented netsim_bridge port: jumpstarter-driver-netsim (PR #980) is a REST control driver -- radios, patching, reset, pcap capture -- and composes with forwards rather than competing. Cite bt-peer (PR #986, merged) passing its transport string straight to bumble.transport.open_transport as the evidence for the zero-Python-changes claim, which was previously asserted. DD-9 reworked around a shared virtual controller. Forwarding one rootcanal into another joins controller to controller and cannot work; joining two devices needs a component that is a controller to both. Bumble already implements it -- virtual Controller on a link-layer bus, RemoteLink over a WebSocket relay with rooms, and an android-netsim mode=controller documented as replacing netsim outright. The medium becomes a Python driver instead of a placement dependency, and what crosses between exporters is a WebSocket, a forward this JEP already carries. Carry two caveats rather than hiding them: Bumble documents controller mode against the emulator, not Cuttlefish, and a Python medium is unproven for A2DP-class traffic. Narrow the corresponding unresolved question from a design question to a prototype one, name Bumble link relay rooms as the implementation of DD-6's deferred N-way medium, and add netsim pcap capture to Observability. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 235 +++++++++++++----- 1 file changed, 173 insertions(+), 62 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 2c71bda89..73d68b8c7 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -494,28 +494,38 @@ subsystem and becomes a question of which port each stack exposes. Radios are the demanding case, but nothing in the table below is privileged — a CAN bus or a serial cross-over is the same declaration: +First, the **data plane** — ports that a forward carries: + | Stack | Port | Direction | Notes | | --- | --- | --- | --- | +| Shared virtual controller (Bumble) | `controller` | provides | Accepts multiple hosts and mediates between them (DD-9) | +| `jumpstarter-driver-bt-peer` | `controller` | requires | A Bumble `Device` dialing an external controller | | rootcanal (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP; hosts attach to it | -| `bt-peer` (bumble) | `controller` | requires | Dials an HCI controller | -| netsim | `netsim` | provides | gRPC `PacketStreamer`; attaching side must originate `ChipInfo` | | `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | | Gateway exporter (real adapter) | `hci` | provides | A real radio, presented as HCI | | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | -Two consequences worth stating plainly, both of which generalize beyond -radios. First, HCI is **asymmetric**: a host -attaches to a controller. Forwarding one rootcanal's port into another -rootcanal wires controller to controller and nothing happens — which is -precisely why forwards are directional and why a `provides → provides` -forward is rejected (DD-6). Second, netsim's port is not a transparent -splice: the attaching side must speak `StreamPackets` and send `ChipInfo` -first, so the consumer is a protocol-terminating driver rather than a raw -socket. Whether two CVDs can be joined at rootcanal directly, or whether -netsim must sit in the middle because it is the component that presents a -controller to multiple hosts, is an open question the Phase 1 prototype -answers (see Unresolved Questions). +Second, the **control plane** — drivers that configure and observe a medium +without carrying its traffic. `jumpstarter-driver-netsim` is the worked +example: it speaks netsim's REST API (`7681 + netsim_instance_num`) to list +devices, toggle radios, patch state, reset, and start/stop/download **pcap +captures** of the simulated air. It is not a forward endpoint and needs none +of this JEP's machinery; the two compose, with the netsim driver observing +the medium a forward connects. + +Two consequences of the data-plane table are worth stating plainly, and both +generalize beyond radios. First, HCI is **asymmetric**: a host attaches to a +controller. Forwarding one rootcanal's port into another rootcanal wires +controller to controller and nothing happens — which is precisely why +forwards are directional and why a `provides → provides` forward is rejected +(DD-6). It is also why the first row exists: joining two devices needs a +component that *is* a controller to both of them (DD-9). Second, not every +port is a transparent splice — netsim's `PacketStreamer` requires the +attaching side to originate a `StreamPackets` call carrying `ChipInfo` before +traffic flows, so its consumer is a protocol-terminating driver rather than a +raw socket. Forwards carry both kinds; the difference lives in the driver at +the `requires` end. For **physical ↔ virtual** there is no way to relay a real phone's internal HCI: the radio is inside the device. The bridge happens in RF, at a @@ -697,10 +707,11 @@ message DialPeerResponse { - **Wi-Fi frame forwarding is the most latency-sensitive path.** `wmediumd` models RSSI-based delivery and expects medium-like timing; a TCP substrate introduces head-of-line blocking a real air interface does not have (DD-10). -- **Listener collisions.** A `requires` port binds a fixed local address. In - a Pod that is free; on a physical lab host already running its own - rootcanal it is not. This is a real deployment constraint, called out in - Unresolved Questions. +- **Listener addresses.** A `requires` port binds a fixed local address, + which is safe because each exporter owns its network namespace (see + *Deployment assumptions*). The one configuration that breaks this is a + host-networked container sharing a machine with other exporters, which is + outside the supported deployment model. - **Degraded hardware.** If a member's exporter goes offline mid-lease, the lease reports `Ready=False` naming the role and does *not* silently continue with a partial bench (DD-3). @@ -988,36 +999,59 @@ validation and is recorded as such. **Alternatives considered:** -1. **At the simulators' existing sockets** — rootcanal's HCI TCP port, - netsim's `PacketStreamer`, `wmediumd`'s frame socket — each exposed as a - port and joined by an ordinary forward. -2. **A dedicated bridge driver tier** — purpose-built `LinkEndpoint` drivers +1. **Forward one simulator's socket into another** — e.g. one CVD's + rootcanal port forwarded to the other CVD. +2. **A shared virtual controller** that both devices attach to as hosts, + using Bumble's virtual `Controller`. +3. **A dedicated bridge driver tier** — purpose-built `LinkEndpoint` drivers that know about radios. -3. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the +4. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the HAL or socket layer. -**Decision:** Option 1. - -**Rationale:** Every simulator involved is already something a process -connects to over a socket, and `jumpstarter-driver-bt-peer` already proves -the pattern by attaching `bumble` to rootcanal at -`tcp-client:127.0.0.1:7300`. Once forwards exist, "bridge a radio" reduces to -"forward the socket that was always there", and a `TcpNetwork` child pointed -at `127.0.0.1:7300` is configuration rather than code. - -Option 2 was in an earlier draft of this JEP and is now rejected as invented +**Decision:** Option 2, implemented over the ordinary forward machinery. + +**Rationale:** Option 1 is the obvious first idea and it does not work for +the motivating topology. rootcanal is a *controller*; two CVDs each have one; +forwarding either into the other joins controller to controller and nothing +happens. Option 1 remains correct wherever one side genuinely is a host — a +`bt-peer` exporter attaching to a CVD's rootcanal is exactly that, and is the +cheapest thing to build first — but it cannot join two devices that each +already own a controller. + +Option 2 supplies the missing piece: a component that is a controller to +*both* devices. Bumble already implements it. Its virtual `Controller` +attaches to a link-layer bus, several controllers on one bus exchange +broadcast advertising and unicast ACL data, and `RemoteLink` carries that bus +over a WebSocket relay hosting virtual *rooms*. Bumble's `android-netsim` +transport additionally has a `mode=controller` that accepts host connections +speaking netsim's protocol — documented as a way to replace netsim outright, +with `bumble-hci-bridge` as the worked example. So the medium becomes a +Jumpstarter driver in Python rather than a dependency on where upstream +components happen to be placed, and the thing crossing between exporters is a +WebSocket — a plain TCP forward this JEP already carries. + +Bumble is also already a dependency: `jumpstarter-driver-bt-peer` is built on +it and passes its `transport` string directly to `bumble.transport. +open_transport`, then uses the result as `controller_source`/`controller_sink` +for a Bumble `Device`. Any Bumble moniker therefore already works with no +Python change, which is what makes the `requires` side of this design free. +The gap is that the driver only ever constructs a `Device` — a host — so the +controller side is new code, and it is small. + +Option 3 was in an earlier draft of this JEP and is rejected as invented machinery: it added a driver interface, a driver tier, and a medium taxonomy -to express something the existing network drivers plus a direction already -express. Option 3 is rejected because it changes the device under test — a +to express something existing network drivers plus a direction already +express. Option 4 is rejected because it changes the device under test — a guest-side shim means the Bluetooth stack being exercised is not the one that ships, invalidating precisely the pairing and handover behavior these tests exist to verify. -One consequence to be explicit about: not every port is a transparent splice. -netsim's `PacketStreamer` requires the attaching side to originate a -`StreamPackets` call with `ChipInfo` before traffic flows, so its consumer is -a protocol-terminating driver, not a raw socket. Forwards carry both kinds; -the difference lives in the driver at the `requires` end. +Two caveats are carried rather than hidden. Bumble's netsim controller mode +is documented against the Android emulator, not Cuttlefish, so pointing a CVD +at an external controller is unverified (see Unresolved Questions). And a +radio medium in Python is comfortable for advertising, pairing, and control +while being an open question for sustained A2DP-class traffic — which +compounds, rather than relieves, the latency risk already recorded. ### DD-10: Wi-Fi medium forwarding @@ -1121,6 +1155,48 @@ there is operational experience with what benches people actually build. ## Design Details +### Deployment assumptions + +Several properties below depend on one deployment invariant, stated here so a +reviewer deploying differently finds out from the document rather than from a +port conflict: + +> **Each exporter owns its own network namespace.** In practice this is +> either a container running exactly one exporter, or a single-exporter edge +> device. + +Both supported shapes satisfy it. For virtual targets, JEP-0016 supplies it +*by construction*: its `cuttlefish.jumpstarter.dev` provisioner renders one +CVD per Pod under a JEP-0014 `ExporterSet`, and a Pod is a network namespace, +so every provisioned exporter gets a private loopback without anyone having +to arrange it. For physical targets an edge device is a single exporter and +the question does not arise. + +Three consequences follow, and they are why this JEP can be as small as it is: + +- **`requires` ports can bind fixed local addresses.** A driver that dials + `127.0.0.1:7300` is dialing *its own* loopback, so two exporters both using + 7300 never collide. This is what makes the zero-driver-changes property + (DD-6) structural rather than coincidental — under a shared namespace it + would be luck that breaks on the second exporter. +- **Control-plane blast radius equals lease scope.** A simulator control API + scoped to "this host" is scoped to one exporter, hence to one lease. This + matters concretely: the netsim driver's `reset` is documented as affecting + every device on its netsim instance, which would cross lease boundaries if + two exporters shared one. +- **DD-4's two transports map onto the two shapes.** Containers on a common + host are mutually routable, so the direct fast path applies; edge devices + behind NAT are exactly why the router path is the default and why pure + peer-to-peer was rejected. + +The exception to watch is host networking. The `jumpstarter-driver-cuttlefish` +container recipe currently documents `--network=host` (so that netsim and +rootcanal are reachable from outside the container), which places every +exporter on that machine in one namespace and forces port-offset schemes such +as `7681 + instance_num`. That arrangement is fine for a hand-managed +single-exporter host or local development, and is **not** a supported basis +for multiple exporters on one machine under this JEP. + ### Binding: one pass, one write The existing `reconcileStatusExporterRef` generalizes to @@ -1306,6 +1382,16 @@ actually used, direct-dial fallback rate). Time-to-bench (lease create → `Ready`) is the headline metric; it is the existing lease-acquisition metric extended to record the member count. +For simulated media there is a stronger signal available than byte counters. +`jumpstarter-driver-netsim` exposes netsim's pcap capture (start, stop, +download) over its REST control API, so a bench can capture the *over-the-air* +traffic of a pairing and attach it to the run — the interaction itself, not +just the fact that bytes crossed a forward. Because the control API is scoped +to one exporter's namespace, that capture is scoped to the lease. This is the +clearest illustration of the control-plane / data-plane split described under +*Attaching media*: the netsim driver controls and observes the medium, while +a forward carries it. + ## Test Plan ### Unit Tests @@ -1357,8 +1443,10 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): ### Hardware-in-the-Loop Tests - **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in - separate Pods (JEP-0016 `ExporterSet`), forwarded at rootcanal/netsim. - Assert BT discovery and pairing, and — Phase 4 — Wi-Fi association. + separate Pods (JEP-0016 `ExporterSet`), joined through a shared virtual + controller (DD-9). Assert BT discovery and pairing, and — Phase 4 — Wi-Fi + association. `jumpstarter-driver-netsim` supplies the pcap capture used to + evidence what actually crossed the air. Runnable in CI on KVM-capable nodes with no lab hardware. This is the headline result and should run on every merge once it exists. - **Physical ↔ physical**: a physical phone and head unit on two exporters @@ -1414,15 +1502,19 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] Direct fast path is authenticated, falls back automatically, and is observable (mode + fallback-rate metrics) - [ ] `bt-peer` participates as a `requires` endpoint with **no Python - changes** — exporter configuration only + changes** — exporter configuration only, relying on its existing + `open_transport(self.transport)` passthrough +- [ ] A shared virtual controller component (Bumble `Controller` plus link + relay) exists as a driver exposing a `provides` port, and two hosts + attached to it exchange advertising and ACL data - [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` integration test **Topologies** (each a phase gate, in order) -- [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods on - separate nodes complete BR/EDR discovery and pairing over a forwarded - rootcanal/netsim port, in CI +- [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods + on separate nodes complete BR/EDR discovery and pairing through a + shared virtual controller reached by a forward, in CI - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection session (Android Auto in the reference implementation) @@ -1557,10 +1649,13 @@ risk in one field, handled by DD-2. - **Bluetooth timing may be tighter than measured** — pairing may work while A2DP streaming does not. Mitigation: latency characterization is an acceptance criterion whose answer is published, not assumed. -- **`listen` collisions on physical hosts.** A lab host already running the - service a `requires` port wants to shadow cannot host that endpoint. - Mitigation: validation names the conflict explicitly; ephemeral-port - allocation is a Future Possibility. +- **The namespace invariant may be violated in the field.** Fixed `listen` + addresses and lease-scoped control-plane blast radius both assume one + exporter per network namespace. A host-networked deployment silently + breaks both — colliding ports, and simulator resets that reach another + lease's devices. Mitigation: bind failures are reported as `Invalid` + naming the port and address, and the assumption is stated explicitly; + JEP-0016 removes the question entirely for provisioned virtual targets. - **Multi-member leases amplify the pre-existing binding race** — N chances to lose a race, so re-bind churn grows with bench size. Mitigation: `MaxItems=8` bounds it; the real fix is the global scheduler the code @@ -1682,12 +1777,14 @@ risk in one field, handled by DD-2. To resolve during review: -- **Can two CVDs be joined at rootcanal directly, or is netsim required?** - HCI is host-to-controller, and two rootcanals are both controllers — so the - Phase 1 topology may need netsim in the middle as the component that - presents a controller to multiple hosts. This determines whether Phase 1's - forward is `rootcanal → controller` or `netsim → chip`, and it is a - prototype question, not a design one. +- **Confirm the Phase 1 topology against a real CVD.** The design answer is + settled (DD-9: a shared virtual controller, with Bumble the leading + candidate). What is unverified is whether a Cuttlefish guest can be pointed + at an *external* controller speaking the netsim protocol — Bumble documents + `mode=controller` for the Android emulator, and Cuttlefish routes radios + through netsim, but no source ties the two together. If it cannot, the + fallback is a Bumble host attached per device with the relay between them, + which costs an extra hop. This is a prototype question, not a design one. - **Should the scalar and members forms really be mutually exclusive?** The alternative is `spec.selector` as a default for members that omit their own — convenient for homogeneous benches, but two ways to express one thing. @@ -1725,11 +1822,17 @@ Not part of this proposal: - **A global lease scheduler** with a view of all leases and exporters, which the controller already carries a `TODO` for; it would close the binding race for single- and multi-member leases alike. -- **Ephemeral `listen` allocation** for `requires` ports, removing the - collision constraint at the cost of driver coupling. -- **Fan-out forwards** — one `provides` port serving several `requires` ports, - for shared media with more than two participants. Today's forward is - strictly point-to-point. +- **Fan-out forwards** — one `provides` port serving several `requires` + ports, for shared media with more than two participants. Today's forward is + strictly point-to-point. Bumble's link relay already implements the + underlying idea: a WebSocket relay hosting virtual *rooms*, each room a set + of virtual controllers that can advertise and exchange ACL data with one + another. A room is the N-way medium DD-6 deferred, so this is an + integration rather than an invention. +- **Ephemeral `listen` allocation** for `requires` ports. Only needed if the + one-exporter-per-namespace assumption is relaxed; it trades the fixed + address for driver coupling, so it is not worth doing while the assumption + holds. - **Bench-level access policy and quota** (DD-12). - **Per-member early release**, if the all-or-nothing lifetime proves coarse. - **Datagram forwards.** `wmediumd` and other frame-oriented media want @@ -1773,7 +1876,15 @@ Not part of this proposal: - [Cuttlefish: test connectivity of multiple devices](https://source.android.com/docs/devices/cuttlefish/connectivity) - [Mobly](https://github.com/google/mobly) — [testbed tutorial](https://github.com/google/mobly/blob/master/docs/tutorial.md) - [Bumble, a Python Bluetooth stack](https://google.github.io/bumble/) — - [transports](https://google.github.io/bumble/transports/index.html) + [transports](https://google.github.io/bumble/transports/index.html), + [Android / `android-netsim` `mode=controller`](https://google.github.io/bumble/platforms/android.html), + [apps and tools](https://google.github.io/bumble/apps_and_tools/index.html) +- [jumpstarter-dev/jumpstarter#986](https://github.com/jumpstarter-dev/jumpstarter/pull/986) + — `jumpstarter-driver-bt-peer` (merged); the reference `requires`-side + endpoint +- [jumpstarter-dev/jumpstarter#980](https://github.com/jumpstarter-dev/jumpstarter/pull/980) + — `jumpstarter-driver-netsim` (open); the control-plane companion, incl. + pcap capture - [netsim (`platform/tools/netsim`)](https://android.googlesource.com/platform/tools/netsim/) — `proto/netsim/packet_streamer.proto` - [google/android-cuttlefish](https://github.com/google/android-cuttlefish) From 05b97de96cddda838a5300773a9763546d691320 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:44:01 -0400 Subject: [PATCH 05/26] docs: JEP-0017 note the counterparty pattern and defer restbus integration to a future JEP DD-9's shared virtual controller is an instance of a general pattern, not a Bluetooth special case: a bench needs a counterparty component that presents itself to the DUT as whatever the DUT expects on the other side of the medium. For Bluetooth that is a controller shared by two hosts; for a vehicle bus it is a restbus simulating the remaining ECUs, which is long-established automotive practice. Both are ordinary provides-ports, so the same forward machinery serves them and no medium-specific mechanism is needed. Add a short paragraph saying so, kept vendor-neutral. Record concrete vehicle-bus integration as future work rather than scope. A broker exposing CAN, LIN, FlexRay and Automotive Ethernet over gRPC is already a provides-port needing nothing new here, and it would compose with the automotive drivers Jumpstarter already ships as the counterparty they talk to. Name RemotiveLabs as the obvious candidate, including the three-member bench their AAOS VHAL integration suggests, and state plainly that it belongs in its own JEP: the broker is a commercial product, so a driver is an integration against something the user licenses rather than a dependency this project can ship, and that warrants its own design discussion instead of a line in these acceptance criteria. No change to scope, acceptance criteria, or the data-plane table. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 73d68b8c7..77a425e77 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1046,6 +1046,16 @@ guest-side shim means the Bluetooth stack being exercised is not the one that ships, invalidating precisely the pairing and handover behavior these tests exist to verify. +Worth noting that option 2 is an instance of a general pattern rather than a +Bluetooth special case. What a bench needs is a **counterparty component**: +something that presents itself to the DUT as whatever the DUT expects on the +other side of the medium. For Bluetooth that is a controller shared by two +hosts; for a vehicle bus it is a *restbus* — a simulation of the remaining +ECUs, a long-established practice in automotive integration testing. Both are +ordinary `provides` ports under this design, so the same forward machinery +serves them and no medium-specific mechanism is required. Integrating a +concrete restbus is future work (see Future Possibilities). + Two caveats are carried rather than hidden. Bumble's netsim controller mode is documented against the Android emulator, not Cuttlefish, so pointing a CVD at an external controller is unverified (see Unresolved Questions). And a @@ -1840,6 +1850,23 @@ Not part of this proposal: additive `FRAME_TYPE_DATAGRAM` extension to `RouterService.Stream` (and, further out, QUIC unreliable datagrams) is a separate protocol JEP, for which Phase 4 is the most compelling justification. +- **Vehicle-bus counterparty integration.** The restbus pattern described in + DD-9 has mature tooling behind it, and a broker that exposes CAN, LIN, + FlexRay, and Automotive Ethernet over a gRPC socket is already a `provides` + port needing nothing new from this JEP. RemotiveLabs is the obvious + candidate — RemotiveBroker plus RemotiveTopology's DBC/ARXML-driven restbus + — and it composes with the automotive drivers Jumpstarter already ships + (`can`, `doip`, `someip`, `uds`, `xcp`, `obd`) as the counterparty they + talk to rather than a replacement for any of them. Their AAOS emulator + integration, which feeds VHAL from broker signals, suggests a three-member + bench worth building eventually: a virtual head unit driven by real vehicle + signals, a restbus supplying the rest of the vehicle, and a phone for + projection, with every pairwise connection an ordinary forward. **This is + deliberately out of scope here and should get its own JEP** — the broker is + a commercial product, so a driver is an integration against something the + user licenses rather than a dependency this project can ship, and that + distinction deserves its own design discussion rather than a line in this + one's acceptance criteria. - **Jumpstarter under ATS** — a Mobile Harness `Device` or Mobly-controller shim backed by a Jumpstarter lease, so Google's results pipeline keeps working while gaining non-Android and cross-host devices. Complements From 1ae59e1df2178d86f62c1fb2aba6c25955882369 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:53:28 -0400 Subject: [PATCH 06/26] docs: JEP-0017 scope to homogeneous benches, defer mixed physical/virtual Narrow v1 to two virtual devices or two physical devices. Mixing them in one bench is deferred, and DD-11 flips from "how we do it" to "why we are not doing it yet" with the gateway-exporter analysis preserved so it does not have to be redone -- the same shape as JEP-0016's DD-8. The obstacle was never software. Nothing about the lease plane or the forward mechanism distinguishes a mixed bench; a real device's radio is inside the device, so the bridge has to happen in RF, which means a dongle or a 5 GHz radio physically near the device and shared between whoever uses it. Requiring that to accept this JEP couples a software design to a hardware procurement, and homogeneous benches already deliver both headline results: cross-host physical benches, and cross-Pod virtual pairing. Consequences of the narrower scope, all recorded: no new lab hardware is required at all, so that cost leaves Hardware Considerations and the negative consequences entirely; the gateway row leaves the data-plane table; the hybrid HiL test and Phase 3 acceptance gate are gone, with virtual Wi-Fi renumbered Phase 3; and the gateway-modelling unresolved question moves into the deferred Future Possibility where it belongs. Also state the asymmetry between the two supported bench kinds, which the draft never made explicit: for two virtual devices the medium is simulated, so a forward carries it and a shared controller mediates it; for two physical devices the radio medium is the air, so they pair without any forward at all and forwards carry only wired media such as CAN or a serial cross-over. What physical benches need from this JEP is the lease plane -- the part that lets their two exporters live on different hosts. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 119 ++++++++++-------- 1 file changed, 67 insertions(+), 52 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 77a425e77..db34dbd37 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -178,9 +178,11 @@ This JEP targets exactly that seam: refuses. - **Virtual ↔ virtual**: two CVDs in two Pods, on two nodes, pairing over Bluetooth and Wi-Fi — the thing that has not been done anywhere. -- **Physical ↔ virtual (hybrid)**: a real phone in a lab rack paired to a - virtual head unit running in the cluster, via a gateway exporter that owns - a real radio. +**Scope.** This JEP covers **homogeneous benches** — two virtual devices, or +two physical devices. Mixing them in one bench (a real phone paired to a +virtual head unit) is a natural extension and the eventual prize, but it +requires real radio hardware bridging the two worlds and is deferred to keep +v1 tractable (DD-11). The last one is the interesting business case: labs have scarce physical head units and abundant phones (or the reverse), and virtualizing the @@ -502,7 +504,6 @@ First, the **data plane** — ports that a forward carries: | `jumpstarter-driver-bt-peer` | `controller` | requires | A Bumble `Device` dialing an external controller | | rootcanal (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP; hosts attach to it | | `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | -| Gateway exporter (real adapter) | `hci` | provides | A real radio, presented as HCI | | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | @@ -527,10 +528,13 @@ traffic flows, so its consumer is a protocol-terminating driver rather than a raw socket. Forwards carry both kinds; the difference lives in the driver at the `requires` end. -For **physical ↔ virtual** there is no way to relay a real phone's internal -HCI: the radio is inside the device. The bridge happens in RF, at a -**gateway exporter** owning a real adapter and physically near the physical -device (DD-11). This is a lab-hardware requirement, not a software trick. +Note the asymmetry between the two supported bench kinds. For **two virtual +devices** the medium is simulated, so a forward carries it and a shared +controller mediates it. For **two physical devices** the radio medium is the +air: real devices in RF range pair without any forward at all, and forwards +carry only the *wired* media of such a bench — a CAN segment, a serial +cross-over. The lease plane is what physical benches need most, because it is +what lets their two exporters live on different hosts. ### API / Protocol Changes @@ -688,10 +692,10 @@ message DialPeerResponse { ### Hardware Considerations -- **Gateway exporters** for hybrid benches need a real Bluetooth adapter (a - USB HCI dongle is sufficient — `bumble` already supports `usb:0`) and, for - Wi-Fi, a 5 GHz-capable radio in the same RF space as the physical device. - This is new lab hardware, and the JEP does not pretend otherwise. +- **No new lab hardware is required.** Homogeneous benches use what a lab + already has: two physical devices pair over the air as they always have, + and two virtual devices need no radio at all. The radio bridging hardware + that a mixed physical/virtual bench would need is out of scope (DD-11). - **RF isolation.** Multiple physical benches in one room share the air. Labs running more than one need shielded enclosures or channel planning; the controller cannot schedule around collisions it cannot observe. Express RF @@ -851,7 +855,7 @@ sustained throughput. It is retained as an explicit `mode: client-relay` debug transport, not as the architecture. Option 3 has the best data plane and fails at exactly the case this JEP is -for: the hybrid bench, where a lab exporter behind NAT and a cluster Pod are +for: an edge-device exporter behind NAT and a cluster Pod, which are not mutually routable. Jumpstarter's value in that topology is that the controller and router are the only things both sides must reach. @@ -1094,31 +1098,40 @@ for sharing a Wi-Fi medium between separately-launched instances, and hardest thing in this JEP and the most likely to need upstream work — it is scheduled last (Phase 4) and its risk is called out explicitly. -### DD-11: Physical ↔ virtual — gateway exporter +### DD-11: Mixed physical/virtual benches — deferred **Alternatives considered:** -1. **Gateway exporter with a real radio**, presenting the adapter as a - `provides` port; the virtual side attaches as if to any other controller. -2. **Emulate the physical device's peer in software** and never involve RF. -3. **Require both halves to be the same kind** — no hybrid benches. - -**Decision:** Option 1. - -**Rationale:** A physical device's radio is inside the device; its HCI is not -reachable from outside, so no software path puts a real phone and a simulated -head unit on the same medium without a real radio somewhere. Option 1 puts -that radio in exactly one place and keeps it a normal exporter with a normal -driver, so it schedules, leases, and reports like everything else. - -Option 2 is a useful *test double* but is not a hybrid bench: it is a virtual -bench with a hand-written model of the physical device, and it cannot find -bugs in the physical device's stack. Option 3 gives up the differentiator in -the Motivation. - -The consequence is stated plainly: hybrid benches require lab hardware and -physical proximity between the gateway and the device, and the gateway is a -shared RF resource that must be modelled as such. +1. **Defer** — v1 supports homogeneous benches only: two virtual devices or + two physical devices. +2. **In scope now**, via a gateway exporter owning a real radio adapter, + presented as a `provides` port that the virtual side attaches to as it + would to any other controller. + +**Decision:** Option 1 — defer. + +**Rationale:** Nothing about the lease plane or the forward mechanism +distinguishes a mixed bench; the obstacle is entirely physical. A real +device's radio is inside the device, and its HCI is not reachable from +outside, so no software path puts a real phone and a simulated head unit on +the same medium. The bridge has to happen in RF, which means new lab hardware +(a USB HCI dongle suffices for Bluetooth; Wi-Fi needs a 5 GHz radio), +physically near the device, shared between whoever is using it. Requiring +that to accept this JEP couples a software design to a hardware procurement, +and homogeneous benches already deliver both headline results — cross-host +physical benches and cross-Pod virtual pairing. + +The path when it returns is option 2, and the analysis is recorded here so it +does not have to be redone. A gateway exporter stays a normal Jumpstarter +exporter with a normal driver, so it schedules, leases, and reports like +everything else, and its adapter is an ordinary `provides` port — no new +mechanism is needed, only the hardware and a decision about how to model a +shared RF resource (see Future Possibilities). + +Worth naming a tempting non-answer: emulating the physical device's peer in +software and never involving RF. That is a useful *test double*, but it is +not a mixed bench — it is a virtual bench with a hand-written model of the +physical device, and it cannot find bugs in the physical device's stack. ### DD-12: Access policy and port validation timing @@ -1379,9 +1392,9 @@ two-object design would face does not arise. configuration. - **`members` is immutable after creation**, so a bound lease cannot be widened beyond what it was authorized for. -- **Physical RF is not access-controlled.** A gateway exporter's radio is - audible to anything in range; labs must treat RF proximity as a trust - boundary. +- **Physical RF is not access-controlled.** Two physical devices pairing in + a shared lab space are audible to anything in range; labs must treat RF + proximity as a trust boundary. ### Observability @@ -1462,8 +1475,6 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - **Physical ↔ physical**: a physical phone and head unit on two exporters **on different lab hosts** — the allocation ATS's `SimpleScheduler` refuses. Requires lab hardware; runs on a labeled runner. -- **Hybrid**: physical phone + gateway exporter (USB HCI dongle) + virtual - head unit in-cluster. - **Latency characterization**: HCI round-trip through a router forward vs. a direct forward vs. host-local rootcanal, reported as a distribution. Its result determines whether A2DP-class workloads are in scope for router mode. @@ -1528,9 +1539,7 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection session (Android Auto in the reference implementation) -- [ ] **Phase 3 — Hybrid**: a physical phone pairs with a virtual head unit - through a gateway exporter -- [ ] **Phase 4 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a +- [ ] **Phase 3 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a forwarded `mac80211_hwsim`/`wmediumd` medium, and a projection session completes the Bluetooth → Wi-Fi handover end to end - [ ] Measured HCI round-trip latency through a router forward is published, @@ -1618,8 +1627,9 @@ risk in one field, handled by DD-2. form expressible *in CI without a lab*. - Cross-host virtual device pairing, which no shipping tool does, reduces to forwarding a socket. -- Hybrid benches let labs virtualize the abundant half of a bench and keep the - scarce half real. +- Homogeneous benches are the tractable half of the problem and already + deliver both headline results; mixing physical and virtual devices in one + bench needs no new software mechanism, only radio hardware (DD-11). - Nothing here is Android-specific: CAN cross-connects between two ECUs, serial cross-overs, and SOME/IP peer benches are all ports and forwards. - The exporter = DUT invariant survives; JEP-0016 DD-8's option 1 becomes @@ -1646,7 +1656,6 @@ risk in one field, handled by DD-2. optional direct fast-path listener. - **Latency is a first-class risk**, not a footnote, and some timing-sensitive protocols may not work in router mode. -- **Hybrid benches need new lab hardware** and physical proximity. - **Phase 4 depends on upstream behavior** we do not control. ### Risks @@ -1716,8 +1725,8 @@ risk in one field, handled by DD-2. report entries and two parallel key namespaces. - **Client-relayed forwards** — DD-4. Retained as an explicit `mode: client-relay` debug transport, not the architecture. -- **Direct peer-to-peer only** — DD-4. Fails on the hybrid topology that - motivates the design. +- **Direct peer-to-peer only** — DD-4. Fails whenever an edge-device + exporter and a cluster Pod are not mutually routable. - **A peer-specific RPC in `router.proto`** — DD-5. - **Guest-side Bluetooth/Wi-Fi shims** — DD-9. Changes the device under test. - **L4 forwarding of the projection session's socket** — DD-10. Skips the @@ -1797,7 +1806,8 @@ To resolve during review: which costs an extra hop. This is a prototype question, not a design one. - **Should the scalar and members forms really be mutually exclusive?** The alternative is `spec.selector` as a default for members that omit their - own — convenient for homogeneous benches, but two ways to express one thing. + own — convenient when several members share a selector, but two ways to + express one thing. - **How should `listen` collisions be handled?** A fixed address keeps drivers unchanged but can collide on a physical host; an ephemeral port avoids collisions but requires telling the driver its address, reintroducing @@ -1809,9 +1819,6 @@ To resolve during review: forwards from `jmp shell --lease` couple the export to a live session; a long-lived per-member forward managed by the client library is the alternative. -- **How are gateway exporters modeled?** As a member with its own role - (schedulable and auditable, but benches become three members where users - think in two), or as an attribute of the physical member's exporter? To resolve during implementation: @@ -1826,6 +1833,14 @@ To resolve during implementation: Not part of this proposal: +- **Mixed physical/virtual benches** (DD-11) — a real device paired to a + virtual one through a gateway exporter owning a real radio adapter. This + needs no new software mechanism: the adapter is an ordinary `provides` + port and the lease plane is unchanged. What it needs is lab hardware near + the physical device, and a decision about how to model a shared RF + resource — as a member with its own role, which makes it schedulable and + auditable but turns a two-device bench into three members, or as an + attribute of the physical member's exporter. - **Selection-time port validation** via JEP-0015 dynamic exporter labels, so a bench whose ports cannot be satisfied reports `Unsatisfiable` without holding anything (DD-12). From 29600ca42cf100f425421e7d236f7f3e0bd77526 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 16:53:50 -0400 Subject: [PATCH 07/26] docs: JEP-0017 fix stale Phase 4 references after renumbering Dropping the hybrid phase renumbered virtual Wi-Fi from Phase 4 to Phase 3, but six references elsewhere still pointed at Phase 4: DD-10's scheduling note, the HiL test plan, the graduation criteria range, a negative consequence, the Wi-Fi risk mitigation, and the datagram-forwards future possibility. All corrected. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...JEP-0017-multi-exporter-leases-port-forwarding.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index db34dbd37..6592c7bf2 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1096,7 +1096,7 @@ because `--vhost_user_mac80211_hwsim` is the mechanism Cuttlefish documents for sharing a Wi-Fi medium between separately-launched instances, and `wmediumd` is where RSSI and delivery modeling live. Option 1 is also the hardest thing in this JEP and the most likely to need upstream work — it is -scheduled last (Phase 4) and its risk is called out explicitly. +scheduled last (Phase 3) and its risk is called out explicitly. ### DD-11: Mixed physical/virtual benches — deferred @@ -1467,7 +1467,7 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in separate Pods (JEP-0016 `ExporterSet`), joined through a shared virtual - controller (DD-9). Assert BT discovery and pairing, and — Phase 4 — Wi-Fi + controller (DD-9). Assert BT discovery and pairing, and — Phase 3 — Wi-Fi association. `jumpstarter-driver-netsim` supplies the pcap capture used to evidence what actually crossed the air. Runnable in CI on KVM-capable nodes with no lab hardware. This is the @@ -1560,7 +1560,7 @@ JEP-0015 dependency. ### Stable -- Phases 1–4 complete, with Phase 4 green in CI for 30 consecutive days +- Phases 1–3 complete, with Phase 3 green in CI for 30 consecutive days - At least two `requires`-side drivers outside this JEP's reference set (evidence the port model generalizes) - No API changes to `members` / `forwards` / `PortReport` for one release @@ -1656,14 +1656,14 @@ risk in one field, handled by DD-2. optional direct fast-path listener. - **Latency is a first-class risk**, not a footnote, and some timing-sensitive protocols may not work in router mode. -- **Phase 4 depends on upstream behavior** we do not control. +- **Phase 3 depends on upstream behavior** we do not control. ### Risks - **Wi-Fi frame forwarding may not be viable over the router.** `wmediumd` assumes medium-like timing; head-of-line blocking on a TCP substrate may make association flaky or impossible except on the direct fast path. - Mitigation: Phase 4 is last, may conclude "direct mode only", and the + Mitigation: Phase 3 is last, may conclude "direct mode only", and the datagram work in Future Possibilities is the escalation path. - **Bluetooth timing may be tighter than measured** — pairing may work while A2DP streaming does not. Mitigation: latency characterization is an @@ -1864,7 +1864,7 @@ Not part of this proposal: datagram semantics with real boundaries and no head-of-line blocking; the additive `FRAME_TYPE_DATAGRAM` extension to `RouterService.Stream` (and, further out, QUIC unreliable datagrams) is a separate protocol JEP, for - which Phase 4 is the most compelling justification. + which Phase 3 is the most compelling justification. - **Vehicle-bus counterparty integration.** The restbus pattern described in DD-9 has mature tooling behind it, and a broker that exposes CAN, LIN, FlexRay, and Automotive Ethernet over a gRPC socket is already a `provides` From 6d9228cd415f955d6307ffad22bd38ec35265de8 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 17:26:07 -0400 Subject: [PATCH 08/26] docs: JEP-0017 add DD-13 -- infer forward direction, never infer topology Auto-wiring matching ports and resolving which end listens look like the same feature and are opposites. They fall on different sides of the line DD-6 and DD-10 already drew, so record that explicitly. Direction is a property of the drivers. Whether rootcanal is the listening end is a fact about rootcanal, already in the exporter's report and already validated by the controller. Making the author write it asks them to know exporter internals -- exactly what DD-6 removed addresses to avoid -- and it is knowledge that can go stale, since swapping a provides implementation for one that dials would silently invalidate every manifest naming it as `from`. So add a symmetric `between` form listing two endpoints in any order, and have the controller use the role check it already performs to assign rather than merely reject. `from`/`to` stays for wiring that should be pinned regardless of what exporters report, and is then checked against the reports. Topology is a property of the test, per DD-10: the same two exporters are a Bluetooth bench in one test and two independent devices in another. Auto- forwarding every matching pair would make the bench wire itself differently run to run as selectors bind different exporters, turn multi-port members into a bipartite matching problem where a silently wrong answer beats an error, and relax the security property that an exporter can reach only a peer its lease explicitly names. Switch both worked examples to `between`, extend the CEL rules to require exactly one of `between` or `from`+`to`, add direction resolution to the validation steps and two rows to the failure table, and note the CLI shorthand that answers the verbosity complaint client-side without making topology implicit. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 110 +++++++++++++++--- 1 file changed, 95 insertions(+), 15 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 6592c7bf2..b34196801 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -308,12 +308,15 @@ spec: device-type: aaos-headunit forwards: - name: bt - from: { member: headunit, port: rootcanal } # provides - to: { member: phone, port: controller } # requires + between: + - { member: headunit, port: rootcanal } + - { member: phone, port: controller } ``` -No addresses, no ports numbers, no medium taxonomy. The lease names roles and -port names; both sides' internals stay inside their exporters (DD-6). +No addresses, no port numbers, no medium taxonomy — and no statement of which +side listens. The lease names roles and port names; both sides' internals +stay inside their exporters (DD-6), and the controller resolves direction +from the reported ports at bind time (DD-13). `members[].selector` and `members[].exporterRef` are the *same* fields as the top-level `spec.selector` and `spec.exporterRef` — a member is the @@ -339,8 +342,9 @@ spec: selector: { matchLabels: { ecu-role: body-controller } } forwards: - name: powertrain-bus - from: { member: gateway, port: can0 } # provides - to: { member: node, port: can } # requires + between: + - { member: gateway, port: can0 } + - { member: node, port: can } ``` Only the port names differ. The controller applies the same validation — @@ -590,9 +594,20 @@ type LeaseMember struct { } type LeaseForward struct { - Name string `json:"name"` - From ForwardEndpoint `json:"from"` // must resolve to a `provides` port - To ForwardEndpoint `json:"to"` // must resolve to a `requires` port + Name string `json:"name"` + + // Symmetric form (preferred): exactly two endpoints, in any order. The + // controller resolves which is `provides` and which is `requires` from + // the reported ports at bind time (DD-13). + // +kubebuilder:validation:MinItems=2 + // +kubebuilder:validation:MaxItems=2 + Between []ForwardEndpoint `json:"between,omitempty"` + + // Explicit form: use when the wiring should be pinned regardless of what + // the exporters report. Mutually exclusive with Between. + From *ForwardEndpoint `json:"from,omitempty"` // must resolve to `provides` + To *ForwardEndpoint `json:"to,omitempty"` // must resolve to `requires` + // Auto (default) | Router | Direct | ClientRelay Mode string `json:"mode,omitempty"` } @@ -636,7 +651,8 @@ validate forwards against bound exporters. The existing CEL rules are extended, not replaced — the current "one of selector or exporterRef is required" rule gains a `members` arm, plus new rules for mutual exclusion, unique role names, forwards referencing -declared members, and member immutability (mirroring `tags` and `context`). +declared members, member immutability (mirroring `tags` and `context`), and +exactly one of `between` or `from`+`to` per forward. **Protocol** — three additive fields and one new RPC: @@ -683,7 +699,9 @@ message DialPeerResponse { **CLI surface** — existing commands, new flags: -- `jmp create lease --member role=selector --forward name=m.port:m.port` +- `jmp create lease --member role=selector --forward name=m.port,m.port` + (endpoint order is irrelevant — direction is resolved at bind time; where a + lab uses the same port name on both sides, `--forward bt` expands to it) - `jmp get lease[s]` prints per-role exporters; `-o json|yaml|name` unchanged - `jmp get lease -o mobly` - `jmp get exporter ` shows declared ports @@ -1176,6 +1194,64 @@ them. Option 3 is a real requirement but separable, and should be designed once there is operational experience with what benches people actually build. +### DD-13: Infer direction, never infer topology + +**Alternatives considered:** + +1. **Infer direction only.** A forward names its two endpoints in any order + (`between`); the controller decides which is `provides` and which is + `requires` from the reported ports. Which ports are joined stays explicit. +2. **Infer topology too** — auto-forward every `provides`/`requires` pair the + bound exporters happen to expose, with no `forwards` stanza at all. +3. **Infer nothing** — the author states `from` and `to` on every forward. + +**Decision:** Option 1, with option 3 retained as an explicit form. + +**Rationale:** These two kinds of inference look similar and are opposites, +because they sit on different sides of the line DD-6 and DD-10 already drew. + +Direction is a **property of the drivers**, not a choice. Whether rootcanal is +the listening end is a fact about how rootcanal works, already declared in the +exporter's report and already validated by the controller. Making the author +write it down asks them to know exporter internals — exactly what DD-6 +removed addresses to avoid. Worse, it is knowledge that can go stale: a lab +that swaps a `provides` implementation for one that dials would silently +invalidate every lease manifest naming it as `from`. Inferring direction is +therefore not a convenience but a correctness improvement, and it costs +nothing — the controller already checks the reported roles, so option 1 uses +that check to *assign* rather than merely to *reject*. + +Topology is a **property of the test**. DD-10 settled this: the same two +exporters are a Bluetooth bench in one test and two independent devices in +another, so the wiring belongs to whoever wrote the test. Option 2 makes it a +property of whatever drivers happen to be configured on whichever exporters +the selectors happened to bind, which fails in three ways: + +- **Nondeterminism across bindings.** A selector-based member binds to + different exporters on different runs. If those exporters declare different + ports, the bench wires itself differently run to run — the worst property a + reproducible test can have. +- **Ambiguity is not rare.** Two members each exposing several ports turns + matching into a bipartite matching problem — the same one ATS's + `AdhocTestbedSchedulingUtil` solves for device allocation. Tractable, but a + silently wrong wiring is worse than an error. +- **It weakens a security property.** This JEP states that a forward is not a + general exporter-to-exporter tunnel, because an exporter can reach only a + peer *a lease it is bound to explicitly names*. Option 2 relaxes "explicitly + names" to "happened to match", and an unintended pairing of two `console` + ports is a data path nobody requested. + +Option 3 remains available as `from`/`to` for the case where wiring should be +pinned regardless of what the exporters report, and the controller then checks +the stated roles against the reported ones rather than assigning them. + +The ergonomic complaint that motivates option 2 — `member.port:member.port` +is verbose — is better answered in the client. The CLI expands shorthand into +an explicit `spec.forwards[]` entry before submission, so the stored object +stays auditable and `kubectl get lease -o yaml` shows the real wiring. Where a +lab adopts the same port name on both sides, `--forward bt` can expand to it; +that is a naming convention worth offering and not worth depending on. + ## Design Details ### Deployment assumptions @@ -1310,10 +1386,13 @@ exists belongs to a bound exporter, not to a selector (DD-12). For each `spec.forwards[]` entry the controller checks, against `ExporterStatus.Devices[].Ports`: -1. `from.member` and `to.member` name declared members — enforced by CEL at - admission, before this point. +1. Every endpoint names a declared member — enforced by CEL at admission, + before this point. 2. Both named ports exist on the respective bound exporters. -3. `from` resolves to a `PROVIDES` port and `to` to a `REQUIRES` port. +3. **Direction resolves.** For a `between` forward, exactly one endpoint must + report `PROVIDES` and the other `REQUIRES`; the controller assigns the + roles accordingly (DD-13). For an explicit `from`/`to` forward, the stated + roles must match what the exporters report. 4. If both ports declare `protocol`, the values are equal (DD-8). A failure sets `Invalid` with the offending forward and reason named, and @@ -1345,7 +1424,8 @@ Establishment then reuses the existing port-forward primitives: | Failure | Behavior | | --- | --- | | Named port absent on a bound exporter | Lease `Invalid`; members stay bound for inspection | -| `provides → provides` or `requires → requires` | Lease `Invalid` (DD-6) | +| Both endpoints `provides`, or both `requires` | Lease `Invalid`; direction cannot resolve (DD-6, DD-13) | +| Explicit `from`/`to` contradicts the reported directions | Lease `Invalid` naming the forward and the reported roles | | Declared protocols disagree | Lease `Invalid` (DD-8) | | Protocols differ but neither declared | Connects, fails at first byte — accepted (DD-8) | | Forward references an undeclared member | Rejected by CEL at admission; lease never created | From 1a51731e407c121c828107c9ba8743b38fb71736 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 18:52:10 -0400 Subject: [PATCH 09/26] docs: JEP-0017 correct DD-9 against Cuttlefish sources -- three forwardable options Two claims in DD-9 were wrong, both found by reading google/android-cuttlefish rather than reasoning from the driver README's port table. Sharing a rootcanal between devices is a first-class supported configuration, not something we would be inventing. In assemble_cvd all four rootcanal ports derive from rootcanal_instance_num rather than the CVD's own instance number (hci 7300+N, link 7400+N, test 7500+N, link_ble 7600+N), and the flag is documented as "use an existing rootcanal instance which is launched from cuttlefish instance with rootcanal_instance_num". The guest reaches it through a TCP connector on rootcanal_hci_port. Everything host-local about it is a port, which is what this JEP forwards. And "forwarding one rootcanal into another joins controller to controller and nothing happens" was true only of the HCI port. rootcanal is launched with a separate --link_port and --link_ble_port, distinct from the HCI port that hosts attach to, which is what controller-to-controller federation is for. netsim_server takes only --hci_port, so link-layer federation is available on the standalone rootcanal path and not under netsim -- a real constraint, now stated. DD-9 therefore becomes three supported options rather than a Bumble endorsement: federate two rootcanals at the link port (Phase 1 default, symmetric, no new code, native C++), share one rootcanal via HCI (fallback, equally free but asymmetric so one Pod becomes a single point of failure), and Bumble's virtual Controller plus RemoteLink rooms (the general answer, and the only one that works outside Cuttlefish, for N-way media, or when the medium must be scripted or instrumented). The Python-throughput caveat now applies only to the third. Add a rootcanal link-layer row to the data-plane table, correct the controller-to-controller paragraph to scope its claim to HCI, and rewrite the unresolved question as three cheap experiments that need no new code, with whichever works becoming the Phase 1 default. Phase 1 gains an acceptance criterion that it be achieved with no new driver code at all. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 140 +++++++++++------- 1 file changed, 86 insertions(+), 54 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index b34196801..babba8033 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -506,7 +506,8 @@ First, the **data plane** — ports that a forward carries: | --- | --- | --- | --- | | Shared virtual controller (Bumble) | `controller` | provides | Accepts multiple hosts and mediates between them (DD-9) | | `jumpstarter-driver-bt-peer` | `controller` | requires | A Bumble `Device` dialing an external controller | -| rootcanal (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP; hosts attach to it | +| rootcanal HCI (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP (`7300 + rootcanal_instance_num`); hosts attach to it | +| rootcanal link layer | `rootcanal-link` | provides | Controller-to-controller federation (`7400`, `7600` BLE); standalone rootcanal only, not netsim | | `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | @@ -521,11 +522,12 @@ the medium a forward connects. Two consequences of the data-plane table are worth stating plainly, and both generalize beyond radios. First, HCI is **asymmetric**: a host attaches to a -controller. Forwarding one rootcanal's port into another rootcanal wires -controller to controller and nothing happens — which is precisely why -forwards are directional and why a `provides → provides` forward is rejected -(DD-6). It is also why the first row exists: joining two devices needs a -component that *is* a controller to both of them (DD-9). Second, not every +controller, so forwarding one device's *HCI* port into another's achieves +nothing — which is precisely why forwards are directional and why a +`provides → provides` forward is rejected (DD-6). Note this is a statement +about HCI specifically, not about controllers: rootcanal exposes a separate +**link-layer** port for joining controllers to each other, which is why the +table lists both (DD-9). Second, not every port is a transparent splice — netsim's `PacketStreamer` requires the attaching side to originate a `StreamPackets` call carrying `ChipInfo` before traffic flows, so its consumer is a protocol-terminating driver rather than a @@ -1021,36 +1023,60 @@ validation and is recorded as such. **Alternatives considered:** -1. **Forward one simulator's socket into another** — e.g. one CVD's - rootcanal port forwarded to the other CVD. -2. **A shared virtual controller** that both devices attach to as hosts, - using Bumble's virtual `Controller`. -3. **A dedicated bridge driver tier** — purpose-built `LinkEndpoint` drivers +1. **Share one rootcanal** — the second CVD is launched with + `--rootcanal_instance_num` pointing at the first, and a forward supplies + that instance's HCI port. +2. **Federate two rootcanals at the link layer** — each CVD keeps its own + controller, and a forward joins them at rootcanal's `link_port`. +3. **A shared virtual controller from Bumble** — its virtual `Controller` + plus `RemoteLink` relay, replacing the simulator entirely. +4. **A dedicated bridge driver tier** — purpose-built `LinkEndpoint` drivers that know about radios. -4. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the +5. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the HAL or socket layer. -**Decision:** Option 2, implemented over the ordinary forward machinery. - -**Rationale:** Option 1 is the obvious first idea and it does not work for -the motivating topology. rootcanal is a *controller*; two CVDs each have one; -forwarding either into the other joins controller to controller and nothing -happens. Option 1 remains correct wherever one side genuinely is a host — a -`bt-peer` exporter attaching to a CVD's rootcanal is exactly that, and is the -cheapest thing to build first — but it cannot join two devices that each -already own a controller. - -Option 2 supplies the missing piece: a component that is a controller to -*both* devices. Bumble already implements it. Its virtual `Controller` -attaches to a link-layer bus, several controllers on one bus exchange -broadcast advertising and unicast ACL data, and `RemoteLink` carries that bus -over a WebSocket relay hosting virtual *rooms*. Bumble's `android-netsim` -transport additionally has a `mode=controller` that accepts host connections -speaking netsim's protocol — documented as a way to replace netsim outright, -with `bumble-hci-bridge` as the worked example. So the medium becomes a -Jumpstarter driver in Python rather than a dependency on where upstream -components happen to be placed, and the thing crossing between exporters is a -WebSocket — a plain TCP forward this JEP already carries. +**Decision:** Options 1–3 are all ordinary forwards and all supported. +Option 2 is the Phase 1 default, option 1 the fallback, option 3 the general +answer beyond Cuttlefish. + +**Rationale:** Cuttlefish already solves multi-device Bluetooth on one host, +and reading how tells us what to forward. In +`assemble_cvd`, all four rootcanal ports are derived from +`rootcanal_instance_num` rather than from the CVD's own instance number — +`hci 7300+N`, `link 7400+N`, `test 7500+N`, `link_ble 7600+N` — and +`--rootcanal_instance_num` is documented as *"use an existing rootcanal +instance which is launched from cuttlefish instance with +rootcanal_instance_num."* Sharing a controller between devices is therefore a +first-class, supported configuration, and the guest reaches it through a TCP +connector on `rootcanal_hci_port`. Everything host-local about it is a +**port**, which is exactly what this JEP forwards. + +Option 2 is preferred because it is symmetric. Each device keeps its own +controller, so neither exporter's failure removes the other's radio, and +rootcanal exposes `link_port` / `link_ble_port` specifically for joining +controllers to one another — distinct from the HCI port that hosts use. It +needs no new code at all: a `TcpNetwork` child on 7400 and a forward. + +Option 1 is the fallback if link-layer federation does not behave as the flag +names imply. It is equally free of new code, but asymmetric — one exporter's +rootcanal becomes the medium for both, so that Pod becomes a single point of +failure for the bench. + +Option 3 is the general answer, and the only one that works outside +Cuttlefish. Bumble's virtual `Controller` attaches to a link-layer bus, +several controllers on one bus exchange broadcast advertising and unicast ACL +data, and `RemoteLink` carries that bus over a WebSocket relay hosting +virtual *rooms* — the N-way medium DD-6 deferred. Its `android-netsim` +transport has a `mode=controller` documented as replacing netsim outright. +Choose it when a bench includes a non-Cuttlefish device, when more than two +participants share a medium, or when the medium itself needs to be scripted +or instrumented — a Python controller can inject faults a C++ simulator will +not. + +One constraint separates them in practice: `link_port` is passed only to +standalone rootcanal, never to netsim, which takes `--hci_port` alone. A +device configured to route its radios through netsim therefore has options 1 +and 3 available but not option 2. Bumble is also already a dependency: `jumpstarter-driver-bt-peer` is built on it and passes its `transport` string directly to `bumble.transport. @@ -1078,12 +1104,14 @@ ordinary `provides` ports under this design, so the same forward machinery serves them and no medium-specific mechanism is required. Integrating a concrete restbus is future work (see Future Possibilities). -Two caveats are carried rather than hidden. Bumble's netsim controller mode -is documented against the Android emulator, not Cuttlefish, so pointing a CVD -at an external controller is unverified (see Unresolved Questions). And a -radio medium in Python is comfortable for advertising, pairing, and control -while being an open question for sustained A2DP-class traffic — which -compounds, rather than relieves, the latency risk already recorded. +Two caveats are carried rather than hidden. Every option above depends on the +same unverified step — that the relevant port still works when it is a +forward from another host rather than a local socket (see Unresolved +Questions). And for option 3 specifically, a radio medium in Python is +comfortable for advertising, pairing, and control while being an open +question for sustained A2DP-class traffic, which compounds rather than +relieves the latency risk already recorded; options 1 and 2 keep the medium +in native C++ and avoid that entirely. ### DD-10: Wi-Fi medium forwarding @@ -1546,8 +1574,8 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): ### Hardware-in-the-Loop Tests - **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in - separate Pods (JEP-0016 `ExporterSet`), joined through a shared virtual - controller (DD-9). Assert BT discovery and pairing, and — Phase 3 — Wi-Fi + separate Pods (JEP-0016 `ExporterSet`), joined by a forwarded rootcanal + port (DD-9). Assert BT discovery and pairing, and — Phase 3 — Wi-Fi association. `jumpstarter-driver-netsim` supplies the pcap capture used to evidence what actually crossed the air. Runnable in CI on KVM-capable nodes with no lab hardware. This is the @@ -1605,17 +1633,19 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] `bt-peer` participates as a `requires` endpoint with **no Python changes** — exporter configuration only, relying on its existing `open_transport(self.transport)` passthrough -- [ ] A shared virtual controller component (Bumble `Controller` plus link - relay) exists as a driver exposing a `provides` port, and two hosts - attached to it exchange advertising and ACL data +- [ ] Phase 1 is achieved with **no new driver code** — a `TcpNetwork` child + on the rootcanal port plus a forward +- [ ] A Bumble-based shared controller exists as a driver exposing a + `provides` port, for benches outside Cuttlefish and for N-way media - [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` integration test **Topologies** (each a phase gate, in order) - [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods - on separate nodes complete BR/EDR discovery and pairing through a - shared virtual controller reached by a forward, in CI + on separate nodes complete BR/EDR discovery and pairing over a + forwarded rootcanal port (link-layer federation, or a shared HCI + instance), in CI - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection session (Android Auto in the reference implementation) @@ -1876,14 +1906,16 @@ risk in one field, handled by DD-2. To resolve during review: -- **Confirm the Phase 1 topology against a real CVD.** The design answer is - settled (DD-9: a shared virtual controller, with Bumble the leading - candidate). What is unverified is whether a Cuttlefish guest can be pointed - at an *external* controller speaking the netsim protocol — Bumble documents - `mode=controller` for the Android emulator, and Cuttlefish routes radios - through netsim, but no source ties the two together. If it cannot, the - fallback is a Bumble host attached per device with the relay between them, - which costs an extra hop. This is a prototype question, not a design one. +- **Confirm the Phase 1 topology against real CVDs.** DD-9 leaves three + forwardable options, and all three turn on one unverified step: whether a + rootcanal port still works when it is a forward from another Pod rather + than a local socket. Three cheap experiments settle it, none needing new + code: (a) forward `7400` between two CVDs and see whether their controllers + federate; (b) launch the second CVD with `--rootcanal_instance_num` and + forward `7300` from the first; (c) point a CVD at a Bumble + `mode=controller` endpoint, which is documented for the Android emulator + but not for Cuttlefish. Whichever works becomes the Phase 1 default. This + is a prototype question, not a design one. - **Should the scalar and members forms really be mutually exclusive?** The alternative is `spec.selector` as a default for members that omit their own — convenient when several members share a selector, but two ways to From 56c4585f9c71fe839b550c1cff1bd985dc0c74c8 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 20:55:51 -0400 Subject: [PATCH 10/26] docs(jep-0017): record DD-9 verification against real CVDs Both simulated-medium options were exercised by hand with two Cuttlefish exporter Pods (AAOS head unit + phone) on the kind cluster: link-layer federation of two rootcanals via add_remote, and a shared netsim instance reached through --rootcanal_instance_num. Both complete discovery, SSP pairing, and HFP/A2DP/AVRCP; federation also carried a 38 s A2DP stream. Findings that change the design surface: standalone rootcanals collide on BD_ADDR da:4c:10:de:00:00; the federation join is a post-forward control action on the test channel; netsimd binds HCI on loopback only, so the forward endpoint must live inside the Pod and exist before the second CVD boots. Recorded in DD-9, the Phase 1 gate, Unresolved Questions and Implementation History. Also fixes the option numbering in DD-9's rejection paragraph. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 110 +++++++++++++++--- 1 file changed, 91 insertions(+), 19 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index babba8033..f308d2ab4 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1086,10 +1086,10 @@ Python change, which is what makes the `requires` side of this design free. The gap is that the driver only ever constructs a `Device` — a host — so the controller side is new code, and it is small. -Option 3 was in an earlier draft of this JEP and is rejected as invented +Option 4 was in an earlier draft of this JEP and is rejected as invented machinery: it added a driver interface, a driver tier, and a medium taxonomy to express something existing network drivers plus a direction already -express. Option 4 is rejected because it changes the device under test — a +express. Option 5 is rejected because it changes the device under test — a guest-side shim means the Bluetooth stack being exercised is not the one that ships, invalidating precisely the pairing and handover behavior these tests exist to verify. @@ -1104,15 +1104,73 @@ ordinary `provides` ports under this design, so the same forward machinery serves them and no medium-specific mechanism is required. Integrating a concrete restbus is future work (see Future Possibilities). -Two caveats are carried rather than hidden. Every option above depends on the -same unverified step — that the relevant port still works when it is a -forward from another host rather than a local socket (see Unresolved -Questions). And for option 3 specifically, a radio medium in Python is -comfortable for advertising, pairing, and control while being an open -question for sustained A2DP-class traffic, which compounds rather than +One caveat is carried rather than hidden: for option 3 a radio medium in +Python is comfortable for advertising, pairing, and control while being an +open question for sustained A2DP-class traffic, which compounds rather than relieves the latency risk already recorded; options 1 and 2 keep the medium in native C++ and avoid that entirely. +**Verified against real CVDs (2026-09-01).** Options 1 and 2 were both +exercised by hand on a single-node kind cluster with two Cuttlefish exporter +Pods — an `aosp_cf_x86_64_auto` head unit in Pod A and an +`aosp_cf_x86_64_only_phone` in Pod B (AOSP build 16102939, cuttlefish host +package 1.55.1). No Jumpstarter code was involved: the forward endpoint was +played by a 50-line asyncio TCP relay inside each Pod, and Pod-to-Pod +traffic went over the Pod network directly, which is what DD-4's Direct mode +will do. The router path is therefore still unmeasured; everything below is +about whether the *ports* behave when the other end is in another Pod. + +Both options produced the same result, driven purely through `adb` and +`uiautomator`: BR/EDR inquiry lists the phone on the head unit, SSP numeric +comparison shows the same passkey on both screens, the bond completes with a +16-byte link key, and HFP, A2DP and AVRCP connect. Under option 2 the phone +then streamed a ringtone to the head unit over A2DP for ~38 s at ~42 KB/s +on the federated link, with no disconnect. Boot-to-bond is about two minutes +of wall clock, most of it guest boot. + +*Option 2 — link-layer federation.* Both CVDs launched with +`--netsim_bt=false`, so each Pod runs its own rootcanal. Standalone rootcanal +binds **all four** ports (`hci`, `link`, `test`, `link_ble`) on `0.0.0.0`, +so the join needed no relay at all: from Pod B's test channel, +`add_remote 7400 BR_EDR` and `add_remote 7600 LOW_ENERGY` produced a +`link_layer_socket_device` on the BR/EDR and LE phys of *both* models. Three +things the flag names do not tell you: + +- **BD_ADDR collision.** Every standalone rootcanal numbers its first HCI + device `da:4c:10:de:00:00`, so two federated CVDs start with the same + address and cannot pair. The fix used was `set_device_address` on B's test + channel followed by a Bluetooth off/on in the guest so the stack re-reads + its address; `--rootcanal_default_commands_file` is the launch-time + equivalent. Under option 1 a single model numbers every device and the + problem does not arise. +- **The join is an action, not a wiring.** `add_remote` is a runtime + test-channel command that dials outward, so option 2 needs *two* forwarded + ports (BR/EDR and LE link) plus a control step on the `requires` side + after the forward is up — a natural post-establish hook for a rootcanal + driver, but not something a static `forwards[]` entry expresses alone. + Standalone rootcanal is also not the Cuttlefish default; netsim is. +- **Beacons leak.** The remote model's two default LE beacons appear in the + peer's scan results, because the LE phys are joined wholesale. + +*Option 1 — shared medium.* Pod A kept the default (netsim-backed) radio; +Pod B launched with `--netsim_bt=false --rootcanal_instance_num=2`, whose +effect is that B starts no controller at all and its `tcp_connector` dials +`127.0.0.1:7301` for HCI. A relay listening on B's `7301` and delivering to +A's `7300` made netsim in Pod A list both chips in `/v1/devices`. Two facts +matter for the design: netsimd binds its HCI port on **loopback only**, so a +forward endpoint *inside* Pod A is required rather than merely tidy (which +the design already provides); and the guest dials at boot, so the forward +must be up before the second CVD launches — a pre-launch ordering constraint +where option 2 has a post-launch one. The entire discovery-pairing-profile +flow moved ~140 KB phone→controller and ~13 KB back. + +*Both.* crosvm's minijail sandbox cannot mount `/dev` inside a Pod, so both +launches needed `--enable_sandbox=false`; this is a JEP-0016 deployment +detail, recorded here because it cost the most time. Neither experiment +changes the decision — option 2 stays the Phase 1 default on its symmetry — +but the address and join findings above are why the Phase 1 driver work is +"a rootcanal control hook", not "nothing". + ### DD-10: Wi-Fi medium forwarding **Alternatives considered:** @@ -1645,7 +1703,9 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods on separate nodes complete BR/EDR discovery and pairing over a forwarded rootcanal port (link-layer federation, or a shared HCI - instance), in CI + instance), in CI. *Demonstrated by hand for both variants on + 2026-09-01 with two Pods on one node and a stand-in relay (DD-9); + the router path and the multi-node case remain.* - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection session (Android Auto in the reference implementation) @@ -1906,16 +1966,25 @@ risk in one field, handled by DD-2. To resolve during review: -- **Confirm the Phase 1 topology against real CVDs.** DD-9 leaves three - forwardable options, and all three turn on one unverified step: whether a - rootcanal port still works when it is a forward from another Pod rather - than a local socket. Three cheap experiments settle it, none needing new - code: (a) forward `7400` between two CVDs and see whether their controllers - federate; (b) launch the second CVD with `--rootcanal_instance_num` and - forward `7300` from the first; (c) point a CVD at a Bumble - `mode=controller` endpoint, which is documented for the Android emulator - but not for Cuttlefish. Whichever works becomes the Phase 1 default. This - is a prototype question, not a design one. +- **Bumble as a Cuttlefish controller.** DD-9's options 1 and 2 are now + verified against real CVDs; option 3 is not. Pointing a CVD at a Bumble + `mode=controller` endpoint is documented for the Android emulator but not + for Cuttlefish, and it is the only option that reaches beyond Cuttlefish. + One more prototype run settles it. +- **Who issues the link-layer join, and who owns addresses?** The + federation experiment showed that option 2 needs a control step + (`add_remote` on the test channel) after the forward is up, and that + standalone rootcanals collide on `da:4c:10:de:00:00` unless one is + re-addressed. The candidates are a post-establish hook on the `requires` + side driver, a lease-level `forwards[].onEstablish` action, or leaving it + to the test. The first keeps the controller ignorant of Bluetooth, which + DD-8 and DD-13 argue for. +- **Forward-before-launch ordering.** Under option 1 the guest dials its HCI + port at boot, so the forward must exist before the second CVD is created; + under option 2 the join must happen after. The JEP-0016 provisioner creates + the CVD at lease time, so the forward has to be established in the window + between binding and `cvd create`. Whether that is a lease-status condition + the provisioner waits on, or a retrying connector, is open. - **Should the scalar and members forms really be mutually exclusive?** The alternative is `spec.selector` as a default for members that omit their own — convenient when several members share a selector, but two ways to @@ -2009,6 +2078,9 @@ Not part of this proposal: ## Implementation History - 2026-09-01: JEP drafted +- 2026-09-01: DD-9 options 1 and 2 verified by hand with two CVD Pods on a + kind cluster (pairing, HFP/A2DP/AVRCP, A2DP streaming); findings recorded + in DD-9 and Unresolved Questions ## References From c4b31907a5c30abf9f3ba408644e0e762d17eba9 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 22:07:28 -0400 Subject: [PATCH 11/26] docs(jep-0017): record Android Auto projection over a forward (DD-10 option 3) A GMS GSI phone CVD projected a live Android Auto 17.4 session to the Desktop Head Unit in the other exporter Pod through a single forwarded TCP port (protocol 1.7, TLS 1.2, full Coolwalk launcher). Record what this proves for DD-10 and Phase 2, what it does not (the BT->Wi-Fi handover), and the operational findings. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 57 ++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index f308d2ab4..7777d5579 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1202,6 +1202,54 @@ for sharing a Wi-Fi medium between separately-launched instances, and hardest thing in this JEP and the most likely to need upstream work — it is scheduled last (Phase 3) and its risk is called out explicitly. +**Verified against real CVDs (2026-09-01): option 3 works, and it is worth +more than a diagnostic.** With the same two Pods as the DD-9 experiment, a +real Android Auto session was projected from the phone exporter to the head +unit exporter over nothing but one forwarded TCP port. The pieces, all +public: + +- *Phone.* Google's GSI with GMS for x86-64 (`gsi_gms_x86_64`, Android 17, + a `user` build) dropped into the Cuttlefish AOSP image set: `super.img` + rebuilt with the GSI `system.img` over the AOSP vendor partitions, and the + GSI `vbmeta.img` (verification disabled) in place of Cuttlefish's. Being a + `user` build it boots with `adbd` stopped; enabling it meant editing the + ext4 `system.img` in place (`persist.sys.usb.config=adb` in `build.prop`, + the exporter's public key in `/adb_keys`), which is the kind of image + preparation a `cuttlefish` driver would own. Android Auto 17.4 (x86-64 + split APKs) was then sideloaded, and its developer-mode *head unit + server* started on port 5277 — the port the phone `provides`. +- *Head unit.* Google's Desktop Head Unit 2.1 in the AAOS Pod, under Xvfb, + in its `--adb=5277` mode, which is a plain TCP client. This is the only + publicly available Android Auto receiver: the AAOS CVD cannot receive + projection because the receiver is part of GAS, so for projection the + "virtual head unit" is a process inside the head-unit exporter that + `requires` the phone's port. Direction falls out of DD-13 as expected. +- *Forward.* The same stand-in relays as DD-9: DHU → `127.0.0.1:5277` in + Pod A → Pod B → `adb forward tcp:5277` → the phone. + +The session negotiated protocol 1.7, completed TLS 1.2 +(ECDHE-RSA-AES128-GCM-SHA256 — the receiver-lib and GMS authenticate each +other across the forward, so any relay must be byte-transparent), ran +service discovery, walked the phone-side first-run flow (consent, car +authorization, notification access — each driven by `uiautomator`), and +rendered the full Coolwalk launcher on the DHU with Maps, YouTube Music and +the phone app live. Video is phone→head-unit and was about 0.6 MB for three +minutes of a mostly static screen, so the L4 path is not bandwidth-limited +in Direct mode. Two operational findings: the DHU quits on stdin EOF (it has +an interactive console, which is a feature — HU-side input can be scripted +through it), and Android Auto's head unit server does not survive an +aborted handoff (`IllegalStateException: Already connected`), so a forward +that drops mid-session needs the server restarted, not just the client. + +What this does and does not change: DD-10's decision stands — the +Bluetooth→Wi-Fi handover is still not exercised, so this is not Phase 3. +But it is the app-level half of Phase 2 done virtually: the whole +GMS/gearhead/receiver-lib stack is shown to run across two exporters with a +single `forwards[]` entry, which means the physical Phase 2 gate is about +transport and radios, not about whether projection tolerates a proxy. It +also gives Phase 1 and Phase 3 a real workload to sit under rather than a +synthetic one. + ### DD-11: Mixed physical/virtual benches — deferred **Alternatives considered:** @@ -1708,7 +1756,11 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): the router path and the multi-node case remain.* - [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection - session (Android Auto in the reference implementation) + session (Android Auto in the reference implementation). *The + projection half was demonstrated virtually on 2026-09-01: a GMS GSI + phone CVD projected a live Android Auto session to the Desktop Head + Unit in the other Pod over one forwarded port (DD-10); the physical + transport and radios remain.* - [ ] **Phase 3 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a forwarded `mac80211_hwsim`/`wmediumd` medium, and a projection session completes the Bluetooth → Wi-Fi handover end to end @@ -2081,6 +2133,9 @@ Not part of this proposal: - 2026-09-01: DD-9 options 1 and 2 verified by hand with two CVD Pods on a kind cluster (pairing, HFP/A2DP/AVRCP, A2DP streaming); findings recorded in DD-9 and Unresolved Questions +- 2026-09-01: DD-10 option 3 verified by hand: Android Auto 17.4 on a GMS + GSI phone CVD projected to the Desktop Head Unit in the other Pod over a + single forwarded port; findings recorded in DD-10 and Phase 2 ## References From ab396d5103ea9109dab6d2b10fc6febf25cbe130 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 22:19:17 -0400 Subject: [PATCH 12/26] docs(jep-0017): record Wi-Fi substrate findings and L3 projection over the guest NIC Stock Cuttlefish Wi-Fi rides vhost-user into wmediumd, so DD-10 option 1 needs a frame bridge across hosts, not the L4 forward. The guests are IP-reachable over their OpenWrt APs, and the Android Auto session was re-run with the phone end on wlan0 through one forward. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 7777d5579..f684dedf5 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1250,6 +1250,27 @@ transport and radios, not about whether projection tolerates a proxy. It also gives Phase 1 and Phase 3 a real workload to sit under rather than a synthetic one. +**Wi-Fi, same setup, two more facts.** First, the stock Cuttlefish Wi-Fi is +`virtio_mac80211_hwsim` into a per-environment `wmediumd` over a vhost-user +Unix socket, with an OpenWrt VM as the access point. vhost-user is shared +memory and fd passing, not a byte stream, so option 1's "forward the frame +socket" cannot reuse the L4 forward across hosts: it needs a frame-level +bridge between two `wmediumd` instances (or a datagram forward under a +future substrate). That sharpens the Phase 3 risk rather than changing the +decision. Second, the guests' Wi-Fi is nonetheless IP-reachable. Both +guests joined their local OpenWrt AP — the GSI phone only after its +Ethernet network was cut, because Android never asks Wi-Fi to connect while +a validated Ethernet default exists, which a phone-CVD driver has to handle +— and the projection session above was re-run with the phone end on `wlan0` +instead of ADB: DHU → forward → OpenWrt WAN (one host route and one +`wan→wifi0` forwarding rule on the AP) → phone `wlan0:5277`. Same +handshake, same launcher, session established on the phone's Wi-Fi +address. That is still option 3 — no shared medium, no handover — but it is +carried over the guest's real Wi-Fi NIC, which is what a physical Phase 2 +bench looks like at L3, and it shows the per-instance OpenWrt AP is a +workable stand-in for the head unit's Wi-Fi Direct group when the medium +is not simulated. + ### DD-11: Mixed physical/virtual benches — deferred **Alternatives considered:** @@ -1884,7 +1905,9 @@ risk in one field, handled by DD-2. - **Wi-Fi frame forwarding may not be viable over the router.** `wmediumd` assumes medium-like timing; head-of-line blocking on a TCP substrate may - make association flaky or impossible except on the direct fast path. + make association flaky or impossible except on the direct fast path. Its + transport is also vhost-user (shared memory), so a cross-host medium needs + a frame bridge first, not just a forward (DD-10). Mitigation: Phase 3 is last, may conclude "direct mode only", and the datagram work in Future Possibilities is the escalation path. - **Bluetooth timing may be tighter than measured** — pairing may work while @@ -2135,7 +2158,8 @@ Not part of this proposal: in DD-9 and Unresolved Questions - 2026-09-01: DD-10 option 3 verified by hand: Android Auto 17.4 on a GMS GSI phone CVD projected to the Desktop Head Unit in the other Pod over a - single forwarded port; findings recorded in DD-10 and Phase 2 + single forwarded port, then again with the phone end on its Wi-Fi NIC via + the per-instance OpenWrt AP; findings recorded in DD-10, Phase 2 and Risks ## References From 7c88a29be3e4d4a6099c683ba84d88c7f9059a46 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 22:23:48 -0400 Subject: [PATCH 13/26] docs(jep-0017): fold the projection and Wi-Fi findings into the design DD-10 revised: L4 projection forwarding is promoted from diagnostic to the standard projection path (Phase 2), and the Wi-Fi medium is reframed as a frame bridge because Cuttlefish's wmediumd transport is vhost-user. Add the concrete projection bench (GMS GSI phone + dhu receiver), the aa-hu/aa-hu-wifi provided ports, the reference-driver responsibilities, a virtual-projection HIL test, the Google-artifacts risk, and renumber the topology phases 1-4. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 320 ++++++++++++------ 1 file changed, 219 insertions(+), 101 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index f684dedf5..cbd70852c 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -508,7 +508,10 @@ First, the **data plane** — ports that a forward carries: | `jumpstarter-driver-bt-peer` | `controller` | requires | A Bumble `Device` dialing an external controller | | rootcanal HCI (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP (`7300 + rootcanal_instance_num`); hosts attach to it | | rootcanal link layer | `rootcanal-link` | provides | Controller-to-controller federation (`7400`, `7600` BLE); standalone rootcanal only, not netsim | -| `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | Frame socket | +| `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | vhost-user, not a byte stream — reached through a frame bridge (DD-10) | +| Android Auto head unit server (phone, developer mode) | `aa-hu`, `aa-hu-wifi` | provides | Same service, via `adb forward` (USB-like) or the guest's Wi-Fi address (wireless-like); the head unit dials it | +| Android Auto receiver (Desktop Head Unit) | `phone` | requires | Google's public receiver; dials the phone's port (DD-10) | +| Android Auto wireless receiver (GAS head unit) | `aa-wireless` | provides | TCP 5288 on the head unit; the phone dials it after the Bluetooth handover — physical head units only | | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | @@ -542,6 +545,46 @@ carry only the *wired* media of such a bench — a CAN segment, a serial cross-over. The lease plane is what physical benches need most, because it is what lets their two exporters live on different hosts. +### A projection bench, concretely + +The Android Auto bench in the reference implementation has one shape that +was verified end to end (DD-10) and one that follows from it. + +**Virtual.** The phone is a Cuttlefish exporter booting Google's GMS GSI — +the phone stack Google ships, not an AOSP approximation — with Android Auto +installed and its developer-mode head unit server started by the driver. The +head unit is *not* an AAOS CVD, which cannot receive projection because the +receiver is part of GAS; it is Google's Desktop Head Unit, run by a `dhu` +driver as a process inside the head-unit exporter. The DHU is a TCP client, +so it is the `requires` side, and the phone `provides` the port it dials: + +```yaml +spec: + members: + - name: phone + selector: { matchLabels: { device-type: android-phone, gms: "true" } } + - name: headunit + selector: { matchLabels: { device-type: aa-headunit } } + forwards: + - name: projection + between: + - { member: phone, port: aa-hu-wifi } + - { member: headunit, port: phone } +``` + +`aa-hu-wifi` reaches the head unit server through the guest's Wi-Fi +interface — the phone's side of a wireless session — where `aa-hu` would +reach it through `adb forward`, the shape of a USB session. The forward is +the same either way; the port name is the test's statement of which cable it +is pretending to be (DD-13). + +**Physical.** A GAS head unit on one exporter and a phone on another, on +different hosts. The radios are the air, so the handover needs no forward; +the lease plane is what this bench needs, and its forwards carry only wired +media — a CAN segment feeding the head unit's VHAL, a serial console. A +`dhu` member can stand in for the head unit here too, which gives a +physical phone a hardware-free receiver. + ### API / Protocol Changes All changes are **additive fields on existing types**, plus one new RPC. No @@ -730,7 +773,12 @@ message DialPeerResponse { direct fast path exists. - **Wi-Fi frame forwarding is the most latency-sensitive path.** `wmediumd` models RSSI-based delivery and expects medium-like timing; a TCP substrate - introduces head-of-line blocking a real air interface does not have (DD-10). + introduces head-of-line blocking a real air interface does not have, and + its vhost-user transport needs a frame bridge on each side before any + forward is involved (DD-10). +- **No head unit hardware is needed to project.** Google's Desktop Head + Unit is the receiver for virtual benches and can stand in for one on a + physical phone's bench (see *A projection bench, concretely*). - **Listener addresses.** A `requires` port binds a fixed local address, which is safe because each exporter owns its network namespace (see *Deployment assumptions*). The one configuration that breaks this is a @@ -1171,7 +1219,7 @@ changes the decision — option 2 stays the Phase 1 default on its symmetry — but the address and join findings above are why the Phase 1 driver work is "a rootcanal control hook", not "nothing". -### DD-10: Wi-Fi medium forwarding +### DD-10: Wi-Fi and projection — what a forward carries **Alternatives considered:** @@ -1179,34 +1227,68 @@ but the address and join findings above are why the Phase 1 driver work is share one simulated medium. 2. **Attach at netsim's 802.11 MAC chip** — Wi-Fi as another chip kind on the `PacketStreamer` port. -3. **Forward the projection socket at L4** — skip radio simulation and carry - the projection session's TCP connection directly. - -**Decision:** Option 1 as the target, with option 2 taken first where -netsim's Wi-Fi support covers the configuration; option 3 rejected as a -fidelity failure but retained as a diagnostic. - -**Rationale:** Wireless phone projection's defining behavior is the -*handover*: pair over Bluetooth, then move the session to a peer-to-peer -Wi-Fi link. This holds for Android Auto and CarPlay alike, and it is the -reason the medium cannot be skipped. A -test that forwards the projection socket at L4 never exercises the handover, -the Wi-Fi Direct negotiation, or the failure modes that matter — it verifies -that a TCP proxy works. So the medium has to be simulated. +3. **Forward the projection session at L4** — carry the projection's own TCP + connection, over the guest's real Wi-Fi NIC, and simulate no radio. + +**Decision:** Option 3 is the standard projection path and a Phase 2 +deliverable: the session is an ordinary `provides`/`requires` pair and needs +nothing this JEP does not already build. Options 1 and 2 remain the target +for **medium** fidelity — association, RSSI, and the Bluetooth→Wi-Fi +handover — with option 2 first where netsim covers it, and option 1 now +understood to need a frame-bridge component on each exporter rather than +the byte forward. That is Phase 4. + +**Rationale:** An earlier draft rejected option 3 as a fidelity failure — +"it verifies that a TCP proxy works" — and kept it only as a diagnostic. +Running it changed that judgment. With the two Pods of the DD-9 experiment, +a GMS phone image and Google's own receiver projected a live Android Auto +session across a single forwarded port (the verification below). Version +negotiation, the TLS authentication between GMS and the receiver library, +service discovery, the phone-side first-run flow, and the rendered launcher +with video, audio and input all ran unchanged. That is not a proxy check; +it is the whole projection stack running on two exporters. + +What option 3 does not exercise is precisely bounded: the **handover** — +the credential exchange over Bluetooth, the phone joining the head unit's +Wi-Fi Direct group — and the radio-layer failure modes (RSSI, roaming, +channel loss). Those are the reasons the medium must eventually be +simulated, and they are all that Phase 4 is for. Everything downstream of +the handover runs over the forward, so a lab gets a real projection +workload on day one, in CI, on hardware-free nodes, and Phase 4 arrives as +a fidelity upgrade rather than as the first time anything projects. + +Two facts from the same experiment shape how the medium options are built: + +- **The Cuttlefish Wi-Fi medium is not a byte stream.** The guest's + `virtio_mac80211_hwsim` feeds a per-environment `wmediumd` over a + **vhost-user** Unix socket (shared memory plus fd passing), with an + OpenWrt VM as the access point. `--vhost_user_mac80211_hwsim` is + Cuttlefish's documented way to share one medium between instances, but + only on one host. Across hosts, option 1 is therefore a *bridge* — a + component on each exporter that terminates vhost-user locally and + exchanges 802.11 frames with its peer — and the forward is only the pipe + between the two bridges. This is why option 1 is the general answer and + also the one that needs new code and datagram semantics (Future + Possibilities). +- **The guest's Wi-Fi is already IP-reachable.** Each CVD's guest joins its + own OpenWrt AP and reaches the exporter's network through the AP's WAN + link; with one host route and one forwarding rule on the AP, the exporter + reaches the guest's Wi-Fi address in turn. So a Cuttlefish driver can + expose a guest-side port two ways: through `adb forward`, which models + the USB cable, or through the guest's `wlan0` address, which models the + Wi-Fi link. The projection session was run both ways with the same + result. They are two `provides` ports, not two mechanisms, and which one + a test forwards is a topology choice DD-13 leaves to the test. Between 1 and 2, option 2 is far cheaper when it applies, because it reuses -the same forward machinery as Bluetooth. Option 1 is the general answer, -because `--vhost_user_mac80211_hwsim` is the mechanism Cuttlefish documents -for sharing a Wi-Fi medium between separately-launched instances, and -`wmediumd` is where RSSI and delivery modeling live. Option 1 is also the -hardest thing in this JEP and the most likely to need upstream work — it is -scheduled last (Phase 3) and its risk is called out explicitly. - -**Verified against real CVDs (2026-09-01): option 3 works, and it is worth -more than a diagnostic.** With the same two Pods as the DD-9 experiment, a -real Android Auto session was projected from the phone exporter to the head -unit exporter over nothing but one forwarded TCP port. The pieces, all -public: +the same forward machinery as Bluetooth and netsim already owns the frames. +Option 1 is the general answer, because `wmediumd` is where RSSI and +delivery modeling live. Option 1 is also the hardest thing in this JEP and +the most likely to need upstream work — it is scheduled last and its risk +is called out explicitly. + +**Verified against real CVDs (2026-09-01).** Same two Pods, same stand-in +relays, same Direct-mode caveat as DD-9. The pieces, all public: - *Phone.* Google's GSI with GMS for x86-64 (`gsi_gms_x86_64`, Android 17, a `user` build) dropped into the Cuttlefish AOSP image set: `super.img` @@ -1214,62 +1296,32 @@ public: GSI `vbmeta.img` (verification disabled) in place of Cuttlefish's. Being a `user` build it boots with `adbd` stopped; enabling it meant editing the ext4 `system.img` in place (`persist.sys.usb.config=adb` in `build.prop`, - the exporter's public key in `/adb_keys`), which is the kind of image - preparation a `cuttlefish` driver would own. Android Auto 17.4 (x86-64 - split APKs) was then sideloaded, and its developer-mode *head unit - server* started on port 5277 — the port the phone `provides`. + the exporter's public key in `/adb_keys`). Android Auto 17.4 (x86-64 + split APKs) was then sideloaded and its developer-mode *head unit server* + started on port 5277 — the port the phone `provides`. - *Head unit.* Google's Desktop Head Unit 2.1 in the AAOS Pod, under Xvfb, in its `--adb=5277` mode, which is a plain TCP client. This is the only publicly available Android Auto receiver: the AAOS CVD cannot receive - projection because the receiver is part of GAS, so for projection the - "virtual head unit" is a process inside the head-unit exporter that - `requires` the phone's port. Direction falls out of DD-13 as expected. -- *Forward.* The same stand-in relays as DD-9: DHU → `127.0.0.1:5277` in - Pod A → Pod B → `adb forward tcp:5277` → the phone. + projection because the receiver ships with GAS, so for a virtual bench + the head unit's projection endpoint is a process inside the head-unit + exporter that `requires` the phone's port (see *A projection bench, + concretely*). Direction falls out of DD-13 as expected. +- *Forward.* DHU → `127.0.0.1:5277` in Pod A → Pod B → the phone, first via + `adb forward tcp:5277` and then via the guest's `wlan0` address through + the OpenWrt AP. The session negotiated protocol 1.7, completed TLS 1.2 -(ECDHE-RSA-AES128-GCM-SHA256 — the receiver-lib and GMS authenticate each -other across the forward, so any relay must be byte-transparent), ran +(ECDHE-RSA-AES128-GCM-SHA256 — the relay must be byte-transparent), ran service discovery, walked the phone-side first-run flow (consent, car authorization, notification access — each driven by `uiautomator`), and -rendered the full Coolwalk launcher on the DHU with Maps, YouTube Music and -the phone app live. Video is phone→head-unit and was about 0.6 MB for three -minutes of a mostly static screen, so the L4 path is not bandwidth-limited -in Direct mode. Two operational findings: the DHU quits on stdin EOF (it has -an interactive console, which is a feature — HU-side input can be scripted -through it), and Android Auto's head unit server does not survive an -aborted handoff (`IllegalStateException: Already connected`), so a forward -that drops mid-session needs the server restarted, not just the client. - -What this does and does not change: DD-10's decision stands — the -Bluetooth→Wi-Fi handover is still not exercised, so this is not Phase 3. -But it is the app-level half of Phase 2 done virtually: the whole -GMS/gearhead/receiver-lib stack is shown to run across two exporters with a -single `forwards[]` entry, which means the physical Phase 2 gate is about -transport and radios, not about whether projection tolerates a proxy. It -also gives Phase 1 and Phase 3 a real workload to sit under rather than a -synthetic one. - -**Wi-Fi, same setup, two more facts.** First, the stock Cuttlefish Wi-Fi is -`virtio_mac80211_hwsim` into a per-environment `wmediumd` over a vhost-user -Unix socket, with an OpenWrt VM as the access point. vhost-user is shared -memory and fd passing, not a byte stream, so option 1's "forward the frame -socket" cannot reuse the L4 forward across hosts: it needs a frame-level -bridge between two `wmediumd` instances (or a datagram forward under a -future substrate). That sharpens the Phase 3 risk rather than changing the -decision. Second, the guests' Wi-Fi is nonetheless IP-reachable. Both -guests joined their local OpenWrt AP — the GSI phone only after its -Ethernet network was cut, because Android never asks Wi-Fi to connect while -a validated Ethernet default exists, which a phone-CVD driver has to handle -— and the projection session above was re-run with the phone end on `wlan0` -instead of ADB: DHU → forward → OpenWrt WAN (one host route and one -`wan→wifi0` forwarding rule on the AP) → phone `wlan0:5277`. Same -handshake, same launcher, session established on the phone's Wi-Fi -address. That is still option 3 — no shared medium, no handover — but it is -carried over the guest's real Wi-Fi NIC, which is what a physical Phase 2 -bench looks like at L3, and it shows the per-instance OpenWrt AP is a -workable stand-in for the head unit's Wi-Fi Direct group when the medium -is not simulated. +rendered the full launcher with Maps, YouTube Music and the phone app live. +Video is phone→head-unit and was about 0.6 MB for three minutes of a mostly +static screen, so the L4 path is not bandwidth-limited in Direct mode. +Three operational findings are folded into *Reference drivers for +projection* under Design Details: the DHU quits on stdin EOF and needs a +display; Android Auto's head unit server does not survive an aborted +session (`IllegalStateException: Already connected`); and the GSI phone +would not join Wi-Fi at all while it held a validated Ethernet network. ### DD-11: Mixed physical/virtual benches — deferred @@ -1376,9 +1428,11 @@ therefore not a convenience but a correctness improvement, and it costs nothing — the controller already checks the reported roles, so option 1 uses that check to *assign* rather than merely to *reject*. -Topology is a **property of the test**. DD-10 settled this: the same two -exporters are a Bluetooth bench in one test and two independent devices in -another, so the wiring belongs to whoever wrote the test. Option 2 makes it a +Topology is a **property of the test**. DD-10 makes this concrete: the +same phone exporter is a USB-attached phone when a test forwards `aa-hu` +and a wireless one when it forwards `aa-hu-wifi`, and two exporters are a +bench in one test and independent devices in another, so the wiring belongs +to whoever wrote the test. Option 2 makes it a property of whatever drivers happen to be configured on whichever exporters the selectors happened to bind, which fails in three ways: @@ -1590,6 +1644,39 @@ Establishment then reuses the existing port-forward primitives: | A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | | Client releases the lease | Forwards torn down first, then the lease ends normally | +### Reference drivers for projection + +Nothing in the projection bench needs new forward machinery, but the +verification (DD-10) showed that the two reference drivers own real work, +of the same kind as DD-9's rootcanal control hook: + +- **`cuttlefish` (phone).** A GMS GSI is a `user` build: `adbd` is off and + stays off, so the image the driver boots must carry + `persist.sys.usb.config=adb` and the exporter's ADB public key in + `/adb_keys`. That is an image-preparation step the driver documents and + JEP-0016's provisioner can run once per image, not a per-lease action. + For `aa-hu-wifi` the driver brings the guest onto the instance's OpenWrt + AP, adds the host route to the AP's LAN and the `wan→wifi` forwarding + rule on the AP — and cuts the guest's Ethernet first, because Android + never asks Wi-Fi to connect while it holds a validated Ethernet default. + The head unit server is started on request and **restarted whenever the + forward resets**: Android Auto does not survive an aborted session, so a + forward reconnect must reach the driver as an event, not be hidden in the + splice. +- **`dhu` (head unit).** Runs the Desktop Head Unit as a process: an X + display (Xvfb), a dummy audio driver, stdin held open — the DHU exits on + stdin EOF because stdin is its interactive console, which is also the + head-unit-side stimulus API (key presses, day/night, microphone) the + driver exposes. Screenshots come from the display. The DHU dials the + instant it starts, so the driver starts it only on a client call, after + the forward is Ready, which is the same ordering rule `bt-peer` already + follows. + +Both drivers are the reference `requires`/`provides` pair for projection. +What they must not do is know about each other: the phone driver exposes +ports and the head unit driver dials `127.0.0.1:5277`, and the lease is the +only place the two are joined. + ### Concurrency and ordering Reconciliation is single-writer per lease (standard controller-runtime work @@ -1702,11 +1789,16 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in separate Pods (JEP-0016 `ExporterSet`), joined by a forwarded rootcanal - port (DD-9). Assert BT discovery and pairing, and — Phase 3 — Wi-Fi + port (DD-9). Assert BT discovery and pairing, and — Phase 4 — Wi-Fi association. `jumpstarter-driver-netsim` supplies the pcap capture used to evidence what actually crossed the air. Runnable in CI on KVM-capable nodes with no lab hardware. This is the headline result and should run on every merge once it exists. +- **Virtual projection**: a GMS-GSI phone CVD and a `dhu` exporter in + separate Pods, joined by the `projection` forward over `aa-hu-wifi` + (DD-10). Assert the receiver reaches the launcher and a screenshot of the + head-unit display matches. Same CI tier as the Bluetooth test; the two + compose into one bench once Phase 1 and Phase 2 are both green. - **Physical ↔ physical**: a physical phone and head unit on two exporters **on different lab hosts** — the allocation ATS's `SimpleScheduler` refuses. Requires lab hardware; runs on a labeled runner. @@ -1775,16 +1867,19 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): instance), in CI. *Demonstrated by hand for both variants on 2026-09-01 with two Pods on one node and a stand-in relay (DD-9); the router path and the multi-node case remain.* -- [ ] **Phase 2 — Physical ↔ physical across hosts**: a phone and head unit +- [ ] **Phase 2 — Virtual projection**: a GMS phone CVD and a `dhu` head + unit in separate Pods complete an Android Auto session to the + launcher over one forwarded port, in CI, with no lab hardware. + *Demonstrated by hand on 2026-09-01 over both `aa-hu` and + `aa-hu-wifi` with a stand-in relay (DD-10); the drivers, the router + path and CI remain.* +- [ ] **Phase 3 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection - session (Android Auto in the reference implementation). *The - projection half was demonstrated virtually on 2026-09-01: a GMS GSI - phone CVD projected a live Android Auto session to the Desktop Head - Unit in the other Pod over one forwarded port (DD-10); the physical - transport and radios remain.* -- [ ] **Phase 3 — Virtual Wi-Fi**: two CVDs in separate Pods associate over a - forwarded `mac80211_hwsim`/`wmediumd` medium, and a projection - session completes the Bluetooth → Wi-Fi handover end to end + session (Android Auto in the reference implementation) +- [ ] **Phase 4 — Virtual Wi-Fi medium**: two CVDs in separate Pods + associate over a bridged `mac80211_hwsim`/`wmediumd` medium or a + shared netsim 802.11 chip, and a projection session completes the + Bluetooth → Wi-Fi handover end to end - [ ] Measured HCI round-trip latency through a router forward is published, with a documented statement of which workloads it does and does not support @@ -1794,7 +1889,7 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): ### Experimental `members`, `forwards`, and port reporting ship behind a controller feature -gate, with Phases 1–2 complete. Signals sought: do real benches stay at two +gate, with Phases 1–3 complete. Signals sought: do real benches stay at two members or grow; how often does the direct fast path apply; does anyone hit `MaxItems=8`; how often do `listen` collisions occur on physical hosts; does the nil-`status.exporterRef` convention (DD-2) surprise any consumer; is @@ -1803,7 +1898,7 @@ JEP-0015 dependency. ### Stable -- Phases 1–3 complete, with Phase 3 green in CI for 30 consecutive days +- Phases 1–4 complete, with Phase 4 green in CI for 30 consecutive days - At least two `requires`-side drivers outside this JEP's reference set (evidence the port model generalizes) - No API changes to `members` / `forwards` / `PortReport` for one release @@ -1899,7 +1994,7 @@ risk in one field, handled by DD-2. optional direct fast-path listener. - **Latency is a first-class risk**, not a footnote, and some timing-sensitive protocols may not work in router mode. -- **Phase 3 depends on upstream behavior** we do not control. +- **Phase 4 depends on upstream behavior** we do not control. ### Risks @@ -1908,8 +2003,18 @@ risk in one field, handled by DD-2. make association flaky or impossible except on the direct fast path. Its transport is also vhost-user (shared memory), so a cross-host medium needs a frame bridge first, not just a forward (DD-10). - Mitigation: Phase 3 is last, may conclude "direct mode only", and the - datagram work in Future Possibilities is the escalation path. + Mitigation: Phase 4 is last, may conclude "direct mode only", and the + datagram work in Future Possibilities is the escalation path; Phase 2 + already delivers a real projection workload without it. +- **The projection bench depends on Google artifacts this project cannot + ship.** The GMS GSI and the Desktop Head Unit are published by Google + under their own terms; the Android Auto APK is not published outside + Google Play, and the verification sideloaded a mirror copy. A lab must + supply these itself, and CI for Phase 2 needs an artifact path that does + not redistribute them. Mitigation: the drivers take image and APK + locations as configuration, the reference documents where each artifact + comes from, and the Bluetooth phase carries the CI headline on AOSP-only + images. - **Bluetooth timing may be tighter than measured** — pairing may work while A2DP streaming does not. Mitigation: latency characterization is an acceptance criterion whose answer is published, not assumed. @@ -1974,8 +2079,10 @@ risk in one field, handled by DD-2. exporter and a cluster Pod are not mutually routable. - **A peer-specific RPC in `router.proto`** — DD-5. - **Guest-side Bluetooth/Wi-Fi shims** — DD-9. Changes the device under test. -- **L4 forwarding of the projection session's socket** — DD-10. Skips the - handover, which is the thing under test. Kept as a diagnostic. +- **L4 projection forwarding *as the whole answer*** — DD-10. It is the + Phase 2 deliverable and runs the full projection stack, but it skips the + Bluetooth → Wi-Fi handover, which is why medium simulation stays on the + roadmap as Phase 4. - **N independently-leased devices behind one exporter.** Ruled out by JEP-0016's exporter = DUT invariant. A group as a single composite DUT (JEP-0016 DD-8 option 2) remains legitimate and orthogonal. @@ -2082,7 +2189,12 @@ To resolve during implementation: - Whether the fast path should race the router dial or attempt direct first with a short timeout — a measurable question. - Whether forward reconnect should preserve the medium's logical state or - force a fresh pairing; likely protocol-specific. + force a fresh pairing; likely protocol-specific. The projection experiment + gives one data point: Android Auto's head unit server must be restarted + after a dropped session, so at minimum the reconnect has to be visible to + the endpoint drivers. +- How Phase 2's CI obtains the GMS GSI, the Desktop Head Unit and the + Android Auto APK without redistributing them (see Risks). - How `jmp get leases` renders N exporters in a table column readably. ## Future Possibilities @@ -2120,7 +2232,9 @@ Not part of this proposal: datagram semantics with real boundaries and no head-of-line blocking; the additive `FRAME_TYPE_DATAGRAM` extension to `RouterService.Stream` (and, further out, QUIC unreliable datagrams) is a separate protocol JEP, for - which Phase 3 is the most compelling justification. + which Phase 4 is the most compelling justification. The frame bridge + DD-10 describes is the consumer: it terminates vhost-user on each + exporter and needs a datagram pipe between the two halves. - **Vehicle-bus counterparty integration.** The restbus pattern described in DD-9 has mature tooling behind it, and a broker that exposes CAN, LIN, FlexRay, and Automotive Ethernet over a gRPC socket is already a `provides` @@ -2159,7 +2273,11 @@ Not part of this proposal: - 2026-09-01: DD-10 option 3 verified by hand: Android Auto 17.4 on a GMS GSI phone CVD projected to the Desktop Head Unit in the other Pod over a single forwarded port, then again with the phone end on its Wi-Fi NIC via - the per-instance OpenWrt AP; findings recorded in DD-10, Phase 2 and Risks + the per-instance OpenWrt AP +- 2026-09-01: DD-10 decision revised on that evidence — L4 projection + promoted from diagnostic to the Phase 2 deliverable, the Wi-Fi medium + reframed as a frame bridge; projection bench, `aa-hu`/`aa-hu-wifi` ports, + `dhu` driver and phase renumbering (1–4) added ## References From 754517b2d4703102c0d26653db9139099d0a3ae0 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Tue, 1 Sep 2026 23:15:36 -0400 Subject: [PATCH 14/26] =?UTF-8?q?docs(jep-0017):=20record=20DD-9=20second?= =?UTF-8?q?=20pass=20=E2=80=94=20GMS=20profiles,=20bt-peer=20driver,=20roo?= =?UTF-8?q?tcanal=20limits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified by hand on the two-Pod CVD bench: - Regular pairing between the GMS GSI phone and the AAOS head unit over federated rootcanals now brings up the full classic profile stack (A2DP sink, AVRCP controller with browsing/cover art, HFP client, PBAP client) and streams AAC over A2DP. The GSI ships only LE-audio bluetooth.profile.* properties; the phone-side classic set had to be added to /system/build.prop, recorded as an image-preparation duty of the cuttlefish phone driver. - The merged jumpstarter-driver-bt-peer ran unchanged as the requires-side host against a CVD's rootcanal through a port-forward: discovered, SSP-paired, AVDTP opened. Demonstrates the "driver untouched" claim. - rootcanal aborts when a test-channel client closes early; the guest's HCI link does not recover without cvd restart. Addresses are assigned by attachment order and reused. Every Cuttlefish guest has the identical address plan. Folded into DD-9, Reference drivers, Security, Risks, Unresolved Questions and a new "Virtual <-> peer" HIL test. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 120 +++++++++++++++++- 1 file changed, 117 insertions(+), 3 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index cbd70852c..eb920f870 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1219,6 +1219,85 @@ changes the decision — option 2 stays the Phase 1 default on its symmetry — but the address and join findings above are why the Phase 1 driver work is "a rootcanal control hook", not "nothing". +**Verified again with a GMS phone and the merged `bt-peer` driver +(2026-09-01, second pass).** The same two Pods, with Pod B now running the +GMS GSI phone of DD-10 instead of the AOSP phone, and the peer role played +by real Jumpstarter code for the first time. + +*Regular pairing with a GMS phone.* The UI-driven flow — inquiry, SSP +numeric comparison, bond — succeeded as before, then stalled: no profile +connected, the head unit logged `CachedBluetoothDevice: No profiles`, and +the ACL dropped on `l2c_link_timeout` after SDP. The medium was not the +cause. The GSI enables only the LE-audio `bluetooth.profile.*` properties; +the classic set — A2DP source, AVRCP target, HFP AG, MAP and PBAP server, +PAN, HID, OPP — is product configuration in `/system/build.prop` (the AAOS +image carries its own sink-side set there), and the GSI's `system.img` +replaced it. Injecting the phone-side set into the GSI's `build.prop`, the +same in-place ext4 edit as the ADB fix, produced the full stack: A2DP sink, +AVRCP controller with browsing and BIP cover art, HFP client carrying the +phone's network and subscriber indicators, and PBAP client pulling the +phonebook over L2CAP, all `Connected`. YouTube Music on the phone then +streamed AAC 44.1 kHz stereo to the head unit over the federated link, and +the phone reconnected on its own after a Bluetooth off/on. Which profiles a +bench has is decided by the phone image, which makes this a second +image-preparation duty for the `cuttlefish` phone driver next to the ADB +one (*Reference drivers for projection*). + +*`jumpstarter-driver-bt-peer` as the `requires` side.* The driver's +`BtPeer` class was run unchanged with +`transport: "tcp-client:127.0.0.1:17300"`, where `17300` was a +`kubectl port-forward` to Pod A's rootcanal `hci_port` — byte for byte what +a `bt=headunit.rootcanal:phone.controller` forward delivers, with the peer +end outside the cluster. rootcanal attached it as a second HCI device on the +head unit's own medium; the head unit discovered *Bumble-Phone*, SSP-paired +it (bond, encrypted ACL), listed it *Connected* with Media active, and +opened AVDTP to the peer's SBC source (`avdtp_connected` in the driver's +event log). That is the claim under *Ports* — "`bt-peer`'s Python is +untouched" — demonstrated on the merged driver. It exercises Bumble as a +*host*; the open question about Bumble as the *controller* (option 3) is +unchanged. + +*Three constraints that change the driver's duties.* + +- **The test channel is fragile, and the guest does not recover.** A + latency probe that opened and closed rootcanal's `test_port` twenty times + aborted rootcanal in *both* Pods — `test_channel_transport.cc: Check + failed: written == size, errno = 32` in `SendResponse`: a client that + closes before the banner is written is fatal. `process_restarter` + respawned rootcanal within a second, but the guest's side of the HCI link + (`tcp_connector`) took `SIGPIPE`, is not under the restarter, and the + guest's Bluetooth sat in `BLE_TURNING_ON` until `cvd restart` (about + 20 s; userdata, bonds and installed apps survive it, where `cvd rm` + discards them). So: the test channel is never a `provides` port — the + rootcanal driver keeps it private and forwards only the HCI and link + ports; a forward endpoint must not verify liveness by connect-and-close, + which is why forward state comes from the splice and not from probing; + and a controller crash on one exporter strands every peer's guest with a + dead controller while the lease still looks bound, so the driver watches + its controller and reports the forward `Failed` and the lease `Degraded` + rather than leaving a bench that cannot pair. +- **Addresses follow attachment order, and numbers are reused.** rootcanal + names the *n*-th live HCI device `da:4c:10:de:00:` per model, so the + Bumble peer took `:00:01` on attaching — the address the federated phone + already held. The BD_ADDR finding above therefore generalises from "two + CVDs" to "any host that attaches": the rootcanal driver assigns a + per-member address (from the member index, say) with `set_device_address` + in its post-establish hook, and before the host powers on, because SC + pairing binds the address into the key derivation. +- **Every Cuttlefish guest has the same address plan.** Both instances came + up with mobile data at `192.168.97.2`, Ethernet on `192.168.98.0/24` and + Wi-Fi on `192.168.99.0/25` behind an AP built from the same OpenWrt + rootfs. An L4 forward never sees this; anything that bridges guests at L2 + or L3 — the Phase 4 frame bridge, a Wi-Fi Direct emulation — must NAT or + re-address, one more reason DD-10's option 3 goes first. Pod-to-Pod TCP + connect on this single-node cluster was ~0.1 ms median; cross-node is + still unmeasured. And because standalone rootcanal binds every port on + `0.0.0.0`, it is the cluster's network policy, not the simulator, that + keeps another Pod from `add_remote`-ing into a bench (*Security*). + +The Phase 1 rootcanal hook is therefore three concrete things: a private +test channel, per-member addresses, and a controller watch. + ### DD-10: Wi-Fi and projection — what a forward carries **Alternatives considered:** @@ -1653,8 +1732,13 @@ of the same kind as DD-9's rootcanal control hook: - **`cuttlefish` (phone).** A GMS GSI is a `user` build: `adbd` is off and stays off, so the image the driver boots must carry `persist.sys.usb.config=adb` and the exporter's ADB public key in - `/adb_keys`. That is an image-preparation step the driver documents and - JEP-0016's provisioner can run once per image, not a per-lease action. + `/adb_keys`. The same image decides which Bluetooth profiles the phone + offers — a GSI ships only the LE-audio `bluetooth.profile.*` properties, + and the classic phone-side set has to be added to `/system/build.prop` + (DD-9, second pass). Both are image-preparation steps the driver + documents and JEP-0016's provisioner can run once per image, not + per-lease actions. Recovery is `cvd restart`, which keeps userdata — + bonds and sideloaded apps included — where `cvd rm` does not. For `aa-hu-wifi` the driver brings the guest onto the instance's OpenWrt AP, adds the host route to the AP's LAN and the `wan→wifi` forwarding rule on the AP — and cuts the guest's Ethernet first, because Android @@ -1717,6 +1801,13 @@ two-object design would face does not arise. - **Physical RF is not access-controlled.** Two physical devices pairing in a shared lab space are audible to anything in range; labs must treat RF proximity as a trust boundary. +- **Simulator control ports are not access-controlled either.** Standalone + rootcanal binds its HCI, link and test-channel ports on `0.0.0.0` and + accepts any client; the test channel can re-address devices, join + models, and — by accident — crash the controller (DD-9, second pass). A + driver exposes only the HCI and link ports as `provides`, never the test + channel, and the exporter's network policy is what limits who can reach + them; JEP-0016 should ship that policy with the Pod. ### Observability @@ -1794,6 +1885,12 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): evidence what actually crossed the air. Runnable in CI on KVM-capable nodes with no lab hardware. This is the headline result and should run on every merge once it exists. +- **Virtual ↔ peer**: one `jumpstarter-driver-cuttlefish` exporter and one + `jumpstarter-driver-bt-peer` exporter, joined by + `bt=headunit.rootcanal:phone.controller`. Assert the CVD discovers and + pairs the peer and the driver reports `avdtp_connected` — the smallest + bench, no second guest to boot, and the one the second pass of DD-9 ran + by hand. Cheapest CI tier; runs on every merge. - **Virtual projection**: a GMS-GSI phone CVD and a `dhu` exporter in separate Pods, joined by the `projection` forward over `aa-hu-wifi` (DD-10). Assert the receiver reaches the launcher and a screenshot of the @@ -2018,6 +2115,13 @@ risk in one field, handled by DD-2. - **Bluetooth timing may be tighter than measured** — pairing may work while A2DP streaming does not. Mitigation: latency characterization is an acceptance criterion whose answer is published, not assumed. +- **The simulators are less robust than a lab needs.** rootcanal aborts + when a test-channel client disconnects early, its restart does not + reattach the guest, and a crash on one exporter strands every bench + member's controller (DD-9, second pass). Mitigation: the rootcanal + driver keeps the test channel private, never probes, watches its + controller and reports `Degraded`, and owns `cvd restart` as the recovery + action; the abort itself is an upstream fix worth contributing. - **The namespace invariant may be violated in the field.** Fixed `listen` addresses and lease-scoped control-plane blast radius both assume one exporter per network namespace. A host-networked deployment silently @@ -2157,7 +2261,9 @@ To resolve during review: federation experiment showed that option 2 needs a control step (`add_remote` on the test channel) after the forward is up, and that standalone rootcanals collide on `da:4c:10:de:00:00` unless one is - re-addressed. The candidates are a post-establish hook on the `requires` + re-addressed — and the second pass showed any attaching host collides + too, since numbering is per model and reused. The candidates are a + post-establish hook on the `requires` side driver, a lease-level `forwards[].onEstablish` action, or leaving it to the test. The first keeps the controller ignorant of Bluetooth, which DD-8 and DD-13 argue for. @@ -2278,6 +2384,14 @@ Not part of this proposal: promoted from diagnostic to the Phase 2 deliverable, the Wi-Fi medium reframed as a frame bridge; projection bench, `aa-hu`/`aa-hu-wifi` ports, `dhu` driver and phase renumbering (1–4) added +- 2026-09-01: DD-9 second pass — regular pairing and the full classic + profile stack (plus A2DP streaming) between the GMS GSI phone and the + AAOS CVD over federated rootcanals, after adding the phone-side + `bluetooth.profile.*` set to the GSI; the merged + `jumpstarter-driver-bt-peer` run unchanged as a `requires`-side host + against a CVD's rootcanal through a port-forward; rootcanal test-channel + crash, address reuse and the identical guest address plan recorded as + driver duties, a Security bullet and a Risk ## References From 0fb03708df947fb05d1119c407b63edc1bf52c08 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 00:06:41 -0400 Subject: [PATCH 15/26] docs(jep-0017): fold in the router-carried Bluetooth bench evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third verification pass for DD-9, this time over Jumpstarter's own data path instead of a kubectl port-forward, and with Bumble standing in for the phone. A hook-less exporter identity in the head unit's Pod exports rootcanal's HCI port as a TcpNetwork; a bt-peer Pod runs the merged driver against 127.0.0.1:7300, where a lease-holding forward endpoint splices it through RouterService.Stream. The head unit paired with the peer and listed it Connected with Phone and Media on. Adding an HFP Audio Gateway to the peer (bumble.rfcomm + hfp.AgProtocol) let it ring the head unit: AAOS Telecom logged a ringing call via HfpClientConnectionService. So the phone-facing half of a head-unit bench needs no second guest. Measured: 0.70 ms median round-trip through the forward against 0.05 ms direct Pod-to-Pod, and ~45 ms for an HCI command either way — rootcanal's own scheduling, not the network. DD-4's fast path has nothing to optimize for simulated Bluetooth. Three findings land on the forward endpoint. An ingress reload drains its workers and cuts every long-lived stream — the exporter re-dialed its status stream, the forward did not, and the bench stayed bound with a dead HCI link — so reconnection is the endpoint's job, not each driver's. Two exporter processes under one identity make connections fail with a KeyError in RouterService.Stream. Version skew across a splice is silent. Recorded alongside a bt-peer `profiles:` duty, a failure-mode row and the Virtual-peer HIL test now asserting a ringing call. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 107 +++++++++++++++++- 1 file changed, 105 insertions(+), 2 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index eb920f870..36b80df45 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -1298,6 +1298,91 @@ unchanged. The Phase 1 rootcanal hook is therefore three concrete things: a private test channel, per-member addresses, and a controller watch. +**Verified over a real router forward, with Bumble as the phone +(2026-09-02, third pass).** The second pass reached rootcanal through a +`kubectl port-forward`; this pass replaced it with Jumpstarter's own data +path and removed the phone CVD entirely. Three Pods: the head unit's Pod +gained a second, hook-less exporter identity that exports its rootcanal +`hci_port` as a `TcpNetwork` (plus an echo port, for measurement); a bt-peer +Pod ran the merged driver with `transport: "tcp-client:127.0.0.1:7300"`; and +a sidecar in that Pod held a lease on the rootcanal exporter and published +both ports locally with `TcpPortforwardAdapter` — a hand-rolled stand-in for +the forward endpoint this JEP specifies, with the same shape: one lease, a +listener on the `requires` side, `RouterService.Stream` in the middle. Every +HCI byte the peer sent crossed bt-peer Pod → router → head unit Pod, and the +head unit paired with it: discovered *Bumble-Phone*, SSP-bonded, encrypted +ACL, listed `Connected` with **Phone and Media both on**. + +*A phone, not just a headset.* The merged `BtPeer` presents an A2DP source, +which lights Media. Subclassing it to add an HFP **Audio Gateway** — a +`bumble.rfcomm` server, `hfp.AgProtocol` with the call/callsetup/callheld, +service, signal, roam and battery indicators, and `hfp.make_ag_sdp_records` +in the device's SDP database — lit Phone as well: `hfp_slc_complete`, +codecs `CVSD`/`MSBC` negotiated, `avdtp_connected`, and the head unit's +`HeadsetClientStateMachine` `Connected`. Ringing the head unit from the peer +(`callsetup=1` + `RING` + `+CLIP`) put a call into AAOS Telecom — +`SET_RINGING (successful incoming call)` against +`HfpClientConnectionService`, phone account `HFP …:00:01`. So the phone-facing +half of a head-unit bench needs no second guest at all: the cheapest bench +is one CVD and a Python process, and the `bt-peer` driver should grow a +`profiles:` config (`a2dp-source`, `hfp-ag`, later AVRCP target and PBAP +server) rather than a second driver. + +*What the router costs.* On the same endpoint, from the same Pod: a +round-trip through the forward was **0.70 ms median** (p90 0.91 ms, n=100) +against **0.05 ms** direct Pod-to-Pod — about 0.65 ms of router. An HCI +command round-trip to rootcanal was **~45 ms median through the forward and +~45 ms direct**: the simulator's own scheduling dominates by two orders of +magnitude, and the router is ~1.5% of an HCI exchange. For simulated +Bluetooth, DD-4's fast path is an optimization with nothing to optimize; +`mode: router` is the sensible default, and the *Latency characterization* +test exists to find where that stops being true (A2DP-class throughput, +cross-node, physical controllers). + +*Three operational constraints, all in the forward endpoint's lap.* + +- **An ingress reload cuts every long-lived stream, and nothing re-dials.** + This cluster's nginx ingress reloads on any `Ingress` change — 85 times in + one log — and each reload drains its old workers with + `worker_shutdown_timeout 240s`. The arithmetic is visible end to end: a + reload at 04:00:13, the exporter's controller status stream cut at + 04:04:13.6, the forwarded HCI splice broken at 04:04:55, and the client's + next driver call failing `UNAVAILABLE: Socket closed`. The exporter + reconnected its own status stream in half a second; the *forward* did not, + so the peer's HCI transport stayed dead and the bench looked bound while + the simulated phone was off the air. Immediately after such an event the + first new connection to the listener is also reset, and the next one + succeeds. Nothing about this is Bluetooth-specific or exotic — any + cluster whose proxy reloads has it — so reconnection belongs to the + forward endpoint, not to each driver: the `requires` side re-dials its + peer stream and re-splices transparently, because a driver that opens its + transport once at start, as `bt-peer` and every HCI or serial-style client + does, has no way to notice or recover. Drivers that genuinely must know — + the head unit server of DD-10 — get the reset as an event, which is the + same rule *Reference drivers for projection* already states. +- **One identity, one process.** Running the exporter twice under the same + identity — trivially, a Deployment scaled to two — leaves both registered; + connections then land on whichever process the router picks, and the one + that does not own the lease's session answers `RouterService.Stream` with + a `KeyError` on the driver UUID and closes. It cost an hour of chasing a + "flaky tunnel". A member exporter is single-writer: the deployment runs + one replica, and a duplicate registration is worth detecting rather than + tolerating. +- **Both ends must be the same build.** A forward endpoint running a newer + `jumpstarter-driver-network` than the exporter's runtime accepted + connections and returned EOF on the first byte, with no error on either + side. Version skew between the two halves of a splice is silent, which is + an argument for the forward endpoint being the exporter's own code — + as specified — rather than an arbitrary client-side helper. + +Two smaller notes for whoever writes the tests. Bumble's bonds live in +memory unless a keystore is configured, so restarting the peer invalidates +the DUT's stored link key and the DUT must forget the bond before re-pairing +— the driver should persist its keystore per lease, or the test should +forget first. And deleting a `Lease` object out from under a waiting client +leaves that client retrying `not found` forever instead of re-queueing: +leases are released, not deleted (*Lease state*). + ### DD-10: Wi-Fi and projection — what a forward carries **Alternatives considered:** @@ -1719,6 +1804,7 @@ Establishment then reuses the existing port-forward primitives: | Forward references an undeclared member | Rejected by CEL at admission; lease never created | | `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | | Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | +| Ingress/proxy reload cuts the peer stream | `requires` side re-dials and re-splices transparently; the first attempt after the cut is expected to fail. Observed in DD-9's third pass, and invisible to drivers that dial once | | Direct dial fails (fast path) | Silent fallback to the router path; recorded as a metric | | A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | | Client releases the lease | Forwards torn down first, then the lease ends normally | @@ -1756,6 +1842,13 @@ of the same kind as DD-9's rootcanal control hook: the forward is Ready, which is the same ordering rule `bt-peer` already follows. +- **`bt-peer` (phone).** The third pass of DD-9 showed the driver is one + config away from being a phone rather than an audio source: with an HFP + Audio Gateway alongside its A2DP source it satisfies both of a head unit's + phone-facing profiles, and can ring it. That belongs in the driver as a + `profiles:` list, with the bond keystore persisted for the life of the + lease so a peer restart does not strand the DUT's link key. + Both drivers are the reference `requires`/`provides` pair for projection. What they must not do is know about each other: the phone driver exposes ports and the head unit driver dials `127.0.0.1:5277`, and the lease is the @@ -1889,8 +1982,11 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): `jumpstarter-driver-bt-peer` exporter, joined by `bt=headunit.rootcanal:phone.controller`. Assert the CVD discovers and pairs the peer and the driver reports `avdtp_connected` — the smallest - bench, no second guest to boot, and the one the second pass of DD-9 ran - by hand. Cheapest CI tier; runs on every merge. + bench, no second guest to boot. With the peer's HFP AG enabled, assert the + head unit's `HeadsetClientConnectionService` takes a ringing call from it, + which covers the phone-facing profiles without a phone. DD-9's third pass + ran exactly this by hand, over the router. Cheapest CI tier; runs on every + merge. - **Virtual projection**: a GMS-GSI phone CVD and a `dhu` exporter in separate Pods, joined by the `projection` forward over `aa-hu-wifi` (DD-10). Assert the receiver reaches the launcher and a screenshot of the @@ -2392,6 +2488,13 @@ Not part of this proposal: against a CVD's rootcanal through a port-forward; rootcanal test-channel crash, address reuse and the identical guest address plan recorded as driver duties, a Security bullet and a Risk +- 2026-09-02: DD-9 third pass — the same bench over Jumpstarter's own data + path: a rootcanal `TcpNetwork` exporter, a bt-peer Pod, and a lease-holding + forward endpoint splicing them through `RouterService.Stream`, with a + Bumble HFP-AG + A2DP peer standing in for the phone (pairing, Phone and + Media, a ringing call into AAOS Telecom). Router overhead measured at + 0.65 ms against ~45 ms of rootcanal; proxy-reload stream loss, duplicate + registration and version skew recorded as forward-endpoint duties ## References From 81cf21f24f72476dad190ce564f3704551950a63 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 00:56:35 -0400 Subject: [PATCH 16/26] docs(jep-0017): make the third-pass findings normative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The router-carried bench produced three results that belong in the design sections, not only in DD-9's evidence log. DD-4 now states what the fast path is worth for simulated media: 0.65 ms of router against a ~45 ms rootcanal command round-trip, so `mode: router` is the honest default and the fast path is justified by workloads where the medium is not the bottleneck. Forward establishment gains "Reconnection belongs to the endpoint" — a cut peer stream is routine in a cluster whose proxy reloads on unrelated Ingress changes, and the drivers this JEP is for dial their transport once at start, so re-dialing cannot be delegated to them. Reconnects become an event as well as a counter, which is also how DD-10's head unit server gets its mandatory restart. Deployment assumptions gain a second invariant: one process per exporter identity. Two registrations under one identity split connections and fail with a KeyError on a driver UUID, presenting as a flaky forward. Also: a cluster-data-path Risk covering stream loss and silent version skew, a Forward resilience HIL test, two acceptance criteria, the Phase 1 gate updated to record what the router path has now carried, and the reconnect Unresolved Question extended with Bumble's in-memory bonds. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 107 ++++++++++++++++-- 1 file changed, 97 insertions(+), 10 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 36b80df45..06cbe0d53 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -934,6 +934,17 @@ fast path is an *optimization*, not a requirement: a bench that works over the router works everywhere, and `mode: router` makes the slow path explicit for tests that need reproducible timing. +How much of an optimization is now measured, for one workload. Carrying a +CVD's HCI port between two Pods (DD-9, third pass), the router added +**0.65 ms** to a round-trip — 0.70 ms through the forward against 0.05 ms +direct to the same endpoint — while an HCI command took **~45 ms either +way**, because rootcanal answers on its own schedule. For simulated +Bluetooth the fast path has nothing to optimize, and `mode: router` is the +honest default. The fast path is therefore justified by the workloads where +the medium is *not* the bottleneck — physical controllers, projection +throughput, and the Phase 4 frame bridge — which is also where the +*Latency characterization* test should point next. + ### DD-5: Router changes — none **Alternatives considered:** @@ -1661,6 +1672,20 @@ Three consequences follow, and they are why this JEP can be as small as it is: behind NAT are exactly why the router path is the default and why pure peer-to-peer was rejected. +A second invariant is narrower, and cost a verification session before it +was noticed, so it is stated rather than assumed: + +> **Each exporter identity is served by exactly one process.** + +Two processes sharing one identity both register and both look healthy; the +controller hands a lease to one of them, and connections that the router +routes to the other are answered with a `KeyError` on a driver UUID that +process has never heard of, closing the stream with no useful error on +either side. The symptom presents as a flaky forward, not as a +misconfiguration. A member exporter therefore runs one replica, and a second +registration under a live identity is worth detecting and refusing rather +than tolerating. + The exception to watch is host networking. The `jumpstarter-driver-cuttlefish` container recipe currently documents `--network=host` (so that netsim and rootcanal are reachable from outside the container), which places every @@ -1792,6 +1817,22 @@ Establishment then reuses the existing port-forward primitives: a weaker trust boundary. `mode: direct` fails rather than falling back; `mode: router` never attempts it. +**Reconnection belongs to the endpoint.** A forward outlives any single +stream: an ingress reload, a router restart, or a node's network blip cuts +the peer stream, and in a cluster whose proxy reloads on every unrelated +`Ingress` change that is a routine event, not an incident (DD-9, third +pass). The `requires` side therefore re-dials with backoff and re-splices +accepted connections without tearing down its listener, and the first +attempt immediately after a cut is expected to fail and be retried. The +alternative — surfacing the reset to the driver — does not work, because the +drivers this JEP is for open their transport once at start: `bt-peer` dials +its HCI port when the peer is created, and after a silent cut the bench +still reports `Ready` while the simulated device is off the air. Drivers +that must know still learn of it: a reconnect is an event on the forward +(*Observability*), which is how the head unit server of DD-10 gets its +mandatory restart. Forward state comes from the splice, never from probing +the far end (DD-9, second pass). + **Failure modes and handling:** | Failure | Behavior | @@ -1907,9 +1948,12 @@ two-object design would face does not arise. JEP-0013 telemetry gains `lease.member` as a span attribute wherever `lease.name` already appears, so per-role activity is separable within one lease, plus per-forward counters (bytes each direction, reconnects, mode -actually used, direct-dial fallback rate). Time-to-bench (lease create → -`Ready`) is the headline metric; it is the existing lease-acquisition metric -extended to record the member count. +actually used, direct-dial fallback rate). Reconnects are also an *event* on +the forward, not only a counter: an endpoint driver that must re-establish +protocol state after a cut (DD-10's head unit server) subscribes to it, and +a bench whose forward is re-dialing is visibly degraded rather than quietly +deaf. Time-to-bench (lease create → `Ready`) is the headline metric; it is +the existing lease-acquisition metric extended to record the member count. For simulated media there is a stronger signal available than byte counters. `jumpstarter-driver-netsim` exposes netsim's pcap capture (start, stop, @@ -1998,6 +2042,14 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - **Latency characterization**: HCI round-trip through a router forward vs. a direct forward vs. host-local rootcanal, reported as a distribution. Its result determines whether A2DP-class workloads are in scope for router mode. + The by-hand version (DD-9, third pass) found the simulator, not the + transport, to be the cost on a single node; the test exists to find where + that inverts — cross-node, sustained A2DP, and physical controllers. +- **Forward resilience**: cut the peer stream of a live bench (restart the + router, or reload the ingress in front of it) and assert the forward + re-establishes, the reconnect is counted and emitted, and a driver that + dialed its transport once at start is still talking to the far end + afterwards. Runs in the same CI tier as the Bluetooth bench. ### Manual Verification @@ -2049,6 +2101,12 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): on the rootcanal port plus a forward - [ ] A Bumble-based shared controller exists as a driver exposing a `provides` port, for benches outside Cuttlefish and for N-way media +- [ ] A forward survives a cut peer stream: the endpoint re-dials and + re-splices with no driver involvement, the reconnect is counted and + emitted as an event, and a driver that dialed once at start keeps + working +- [ ] A second exporter process registering under a live identity is + detected and refused rather than silently splitting connections - [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` integration test @@ -2059,7 +2117,10 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): forwarded rootcanal port (link-layer federation, or a shared HCI instance), in CI. *Demonstrated by hand for both variants on 2026-09-01 with two Pods on one node and a stand-in relay (DD-9); - the router path and the multi-node case remain.* + on 2026-09-02 the router path itself carried a bench — a CVD's + rootcanal to a `bt-peer` Pod through `RouterService.Stream`, pairing + and profiles included. CVD ↔ CVD over the router and the multi-node + case remain.* - [ ] **Phase 2 — Virtual projection**: a GMS phone CVD and a `dhu` head unit in separate Pods complete an Android Auto session to the launcher over one forwarded port, in CI, with no lab hardware. @@ -2075,7 +2136,9 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): Bluetooth → Wi-Fi handover end to end - [ ] Measured HCI round-trip latency through a router forward is published, with a documented statement of which workloads it does and does not - support + support. *First data point, 2026-09-02: 0.65 ms of router against a + ~45 ms rootcanal command round-trip, single node (DD-9, third pass); + cross-node and A2DP-class throughput remain.* ## Graduation Criteria @@ -2218,6 +2281,19 @@ risk in one field, handled by DD-2. driver keeps the test channel private, never probes, watches its controller and reports `Degraded`, and owns `cvd restart` as the recovery action; the abort itself is an upstream fix worth contributing. +- **The cluster data path is less stable than a bench assumes.** Long-lived + streams do not survive ordinary cluster events: an nginx ingress reloads + on any `Ingress` change and drains its workers on a timer, cutting every + gRPC stream through it — measured on the verification cluster as a reload + at 04:00:13 and the exporter's controller stream cut at 04:04:13, with + the forwarded HCI splice going down 40 s later (DD-9, third pass). A + multi-hour bench will meet this repeatedly. Mitigation: reconnection is + the forward endpoint's responsibility, reconnects are observable, and the + HIL suite includes a bench that survives a deliberate stream cut. A + related hazard is version skew across a splice — a newer client-side + network driver against an older exporter closed every forwarded + connection at the first byte with no error — which is an argument for the + endpoint being the exporter's own code, as specified. - **The namespace invariant may be violated in the field.** Fixed `listen` addresses and lease-scoped control-plane blast radius both assume one exporter per network namespace. A host-networked deployment silently @@ -2389,12 +2465,18 @@ To resolve during implementation: - Exact `ListenResponse` variant shape for forward setup instructions. - Whether the fast path should race the router dial or attempt direct first - with a short timeout — a measurable question. + with a short timeout — a measurable question, and one whose stakes are now + bounded for simulated media: the router costs ~0.65 ms where the simulator + costs ~45 ms (DD-4). - Whether forward reconnect should preserve the medium's logical state or - force a fresh pairing; likely protocol-specific. The projection experiment - gives one data point: Android Auto's head unit server must be restarted - after a dropped session, so at minimum the reconnect has to be visible to - the endpoint drivers. + force a fresh pairing; likely protocol-specific. Two data points so far: + Android Auto's head unit server must be restarted after a dropped session, + so the reconnect has to be visible to the endpoint drivers; and a Bumble + peer whose transport is cut loses its in-memory bonds, so the DUT's stored + link key is stale and pairing has to be redone unless the driver persists + its keystore for the life of the lease. The Bluetooth answer may simply be + "persist the keystore and re-attach", which would make reconnect + invisible for that medium. - How Phase 2's CI obtains the GMS GSI, the Desktop Head Unit and the Android Auto APK without redistributing them (see Risks). - How `jmp get leases` renders N exporters in a table column readably. @@ -2495,6 +2577,11 @@ Not part of this proposal: Media, a ringing call into AAOS Telecom). Router overhead measured at 0.65 ms against ~45 ms of rootcanal; proxy-reload stream loss, duplicate registration and version skew recorded as forward-endpoint duties +- 2026-09-02: those findings made normative — DD-4's fast path reframed as + an optimization with a measured price, a single-process invariant per + exporter identity, reconnection assigned to the forward endpoint with + reconnects as events, a cluster-data-path Risk, a *Forward resilience* + HIL test, and two acceptance criteria ## References From 4a77fc1b7cebca4cddda29b9971f517b49db4ec3 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 01:11:13 -0400 Subject: [PATCH 17/26] docs(jep-0017): prefer direct peer connections inside a cluster MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Auto` previously meant "race a direct dial against the router and keep whichever answers first", with the router as the honest default. For two virtual targets in one cluster — the case this JEP is mostly for, and the one JEP-0016 produces by construction — the direct path should simply win. DD-4 now says so and gives three reasons: the router is a shared component while bench traffic is sustained, so N benches funnelling media through one deployment builds a bottleneck the CNI would carry for free; the router path in a typical install leaves the cluster and comes back, inheriting the ingress failure modes that cut a live bench's HCI splice during verification; and the workloads still ahead — the Phase 4 frame bridge, sustained projection — are the ones a 14x round-trip difference decides. The measured 0.65 ms of router against a ~45 ms rootcanal command is what makes the preference safe rather than a gamble: falling back is undetectable on the path measured so far. Mechanically: ExporterStatus gains optional PeerEndpoint and NetworkZone, DialPeerResponse gains prefer_direct, establishment dials the peer first with a short bounded timeout instead of racing, and a Direct eligibility rule says the controller offers it only when both members report the same zone. Zone is configuration, not inference — an edge exporter that reports none keeps the router path with no per-lease setup. Also updated: the Security bullet on the peer listener (preferring direct makes JEP-0016's NetworkPolicy load-bearing), the fallback rate as a health signal rather than a curiosity, the sample `j forward status` output, the failure-mode row, an acceptance criterion, the integration test, and the Unresolved Question about racing (now decided) replaced by the timeout and how a zone is established. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 197 +++++++++++++----- 1 file changed, 144 insertions(+), 53 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 06cbe0d53..e0bc0212a 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -388,7 +388,7 @@ Pixel 8 jumpstarter ⚡ android-auto-bench ➤ j headunit power on jumpstarter ⚡ android-auto-bench ➤ j forward status NAME FROM TO MODE STATE A→B B→A -bt headunit.rootcanal phone.controller router connected 1.2 MiB 0.9 MiB +bt headunit.rootcanal phone.controller direct connected 1.2 MiB 0.9 MiB ``` In Python, the existing `lease()` context manager grows `members` and @@ -466,9 +466,13 @@ with one substitution — a router peer stream in place of the client stream: it to the stream with `forward_stream`. The `requires` side opens a `TemporaryTcpListener` on its declared `listen` address and splices each accepted connection to the stream. -5. **Fast path**: if `DialPeer` returned a peer hint, each exporter races a - direct dial against the router path and keeps whichever completes first - (DD-4). +5. **Fast path**: if `DialPeer` says the pair is direct-eligible — both + members in one network zone, which two exporter Pods in a cluster are — + the `requires` side dials the peer directly first and uses the router + stream only if that does not complete quickly (DD-4). Two virtual targets + in a cluster therefore talk over the Pod network, and the same lease keeps + working unchanged when one member is an edge device the other cannot + reach. ```text ┌──────── Lease: android-auto-bench (one object) ─────────┐ @@ -653,7 +657,10 @@ type LeaseForward struct { From *ForwardEndpoint `json:"from,omitempty"` // must resolve to `provides` To *ForwardEndpoint `json:"to,omitempty"` // must resolve to `requires` - // Auto (default) | Router | Direct | ClientRelay + // Auto (default) | Router | Direct | ClientRelay. + // Auto prefers a direct peer connection when both members are in the + // same network zone and falls back to the router; Direct fails rather + // than falling back; Router never attempts a direct dial (DD-4). Mode string `json:"mode,omitempty"` } @@ -691,7 +698,17 @@ type LeaseForwardStatus struct { ``` `ExporterStatus.Devices[]` gains the reported ports so the controller can -validate forwards against bound exporters. +validate forwards against bound exporters. `ExporterStatus` itself gains two +optional fields used only to decide direct eligibility: + +```go + // Address peers in the same zone can dial for a direct forward, if this + // exporter runs a peer listener. Never a device port (see Security). + PeerEndpoint string `json:"peerEndpoint,omitempty"` + // Opaque reachability domain. Two exporters are candidates for a direct + // forward only if both report the same value. + NetworkZone string `json:"networkZone,omitempty"` +``` The existing CEL rules are extended, not replaced — the current "one of selector or exporterRef is required" rule gains a `members` arm, plus @@ -734,8 +751,9 @@ message DialPeerRequest { message DialPeerResponse { string router_endpoint = 1; string router_token = 2; // `stream` claim shared by both ends - optional string peer_endpoint = 3; // fast-path hint + optional string peer_endpoint = 3; // set when the pair is direct-eligible optional string peer_token = 4; // authenticates a direct dial + bool prefer_direct = 5; // Auto resolved to direct-first } ``` @@ -769,8 +787,9 @@ message DialPeerResponse { - **Latency budgets.** Bluetooth HCI is timing-sensitive: supervision timeouts are seconds, but L2CAP/HCI flow control and A2DP jitter buffers are far tighter. A router-relayed forward adds two gRPC hops; measuring - that budget is an acceptance criterion, and it is the main reason the - direct fast path exists. + that budget is an acceptance criterion, and it is why an in-cluster bench + takes the direct path by default (DD-4) and keeps the router as the + fallback that works everywhere. - **Wi-Fi frame forwarding is the most latency-sensitive path.** `wmediumd` models RSSI-based delivery and expects medium-like timing; a TCP substrate introduces head-of-line blocking a real air interface does not have, and @@ -929,21 +948,42 @@ controller and router are the only things both sides must reach. Option 1 keeps one authentication model — a forward is authorized by the lease that names it, exactly as a stream is authorized by the lease that -names it — while permitting the fast data plane where it is available. The -fast path is an *optimization*, not a requirement: a bench that works over -the router works everywhere, and `mode: router` makes the slow path explicit -for tests that need reproducible timing. - -How much of an optimization is now measured, for one workload. Carrying a -CVD's HCI port between two Pods (DD-9, third pass), the router added -**0.65 ms** to a round-trip — 0.70 ms through the forward against 0.05 ms -direct to the same endpoint — while an HCI command took **~45 ms either -way**, because rootcanal answers on its own schedule. For simulated -Bluetooth the fast path has nothing to optimize, and `mode: router` is the -honest default. The fast path is therefore justified by the workloads where -the medium is *not* the bottleneck — physical controllers, projection -throughput, and the Phase 4 frame bridge — which is also where the -*Latency characterization* test should point next. +names it — while permitting the fast data plane where it is available. A +bench that works over the router works everywhere, and `mode: router` makes +the slow path explicit for tests that need reproducible timing. + +**Direct is preferred, not merely permitted, when both members sit in the +same network zone.** For two virtual targets in one cluster — the case this +JEP is mostly for, and the case JEP-0016 produces by construction, since its +provisioner renders both benches as Pods under one `ExporterSet` — `Auto` +resolves to a direct Pod-to-Pod connection and falls back to the router only +if that dial does not complete. Three reasons, in order of weight: + +1. **The router is a shared component and bench traffic is sustained.** A + projection session or an A2DP stream is not a burst; N benches funnelling + media through one router deployment makes a central bottleneck out of + something the CNI would carry for free, and in a cloud install it can + also mean paying to leave and re-enter the network. +2. **The router path in a typical install leaves the cluster and comes + back.** It therefore inherits the ingress's failure modes — the reload + that cut a live bench's HCI splice in DD-9's third pass was an ingress + worker drain, not a Jumpstarter fault. A direct forward between two Pods + stays inside the CNI and never meets that machinery. +3. **The workloads that are latency-sensitive are the ones still ahead.** + Measured on the bench, the router costs **0.65 ms** on a round-trip + (0.70 ms through the forward against 0.05 ms direct to the same + endpoint), while an HCI command takes **~45 ms either way** because + rootcanal answers on its own schedule. Simulated Bluetooth pairing + therefore cannot tell the two apart — which is exactly why preferring + direct is free — but the Phase 4 frame bridge, where `wmediumd` expects + medium-like timing, and sustained projection throughput are where a 14× + round-trip difference starts to decide whether a bench works at all. + +That last measurement is what makes the preference safe rather than a +gamble: falling back to the router costs sub-millisecond on the path that +has been measured, so a bench that cannot get a direct connection is slower +in a way nothing so far can detect. Eligibility is a property of the pair, +not a user decision — see *Direct eligibility* under Design Details. ### DD-5: Router changes — none @@ -1344,11 +1384,13 @@ round-trip through the forward was **0.70 ms median** (p90 0.91 ms, n=100) against **0.05 ms** direct Pod-to-Pod — about 0.65 ms of router. An HCI command round-trip to rootcanal was **~45 ms median through the forward and ~45 ms direct**: the simulator's own scheduling dominates by two orders of -magnitude, and the router is ~1.5% of an HCI exchange. For simulated -Bluetooth, DD-4's fast path is an optimization with nothing to optimize; -`mode: router` is the sensible default, and the *Latency characterization* -test exists to find where that stops being true (A2DP-class throughput, -cross-node, physical controllers). +magnitude, and the router is ~1.5% of an HCI exchange. Simulated Bluetooth +pairing therefore cannot tell the two transports apart — which is what makes +DD-4's preference for a direct in-cluster connection free rather than a +gamble, since falling back to the router is undetectable on this path. The +*Latency characterization* test exists to find where that stops being true +(A2DP-class throughput, cross-node, physical controllers, and the Phase 4 +frame bridge, where it is expected to). *Three operational constraints, all in the forward endpoint's lap.* @@ -1667,10 +1709,13 @@ Three consequences follow, and they are why this JEP can be as small as it is: matters concretely: the netsim driver's `reset` is documented as affecting every device on its netsim instance, which would cross lease boundaries if two exporters shared one. -- **DD-4's two transports map onto the two shapes.** Containers on a common - host are mutually routable, so the direct fast path applies; edge devices - behind NAT are exactly why the router path is the default and why pure - peer-to-peer was rejected. +- **DD-4's two transports map onto the two shapes.** Exporter Pods in one + cluster are mutually routable, so the direct path applies and is preferred; + edge devices behind NAT are exactly why the router path is the universal + fallback and why pure peer-to-peer was rejected. The split is not a + user-visible choice: a virtual bench gets the fast path because of where it + runs, and the same lease spec keeps working when one member moves to a + bench on someone's desk. A second invariant is narrower, and cost a verification session before it was noticed, so it is stated rather than assumed: @@ -1811,11 +1856,28 @@ Establishment then reuses the existing port-forward primitives: 5. The `provides` side dials its local service and splices with `forward_stream`. The `requires` side opens a `TemporaryTcpListener` on its declared `listen` address and splices each accepted connection. -6. **Fast path**: with a `peer_endpoint`/`peer_token`, each exporter races a - direct dial against the router path; first handshake wins, loser is reset. +6. **Fast path**: with `prefer_direct` set, the `requires` side dials + `peer_endpoint` first and gives it a short bounded timeout (a couple of + round-trips, not seconds), falling back to the already-minted router + stream if it does not complete; the router dial is not raced, so an + eligible pair does not pay for two connections on every establishment. The direct listener authenticates `peer_token`, so a direct forward is not a weaker trust boundary. `mode: direct` fails rather than falling back; - `mode: router` never attempts it. + `mode: router` never attempts it. Which transport a forward ended up on is + in `LeaseForwardStatus.Mode`, and a fallback records why. + +**Direct eligibility.** The controller offers `prefer_direct` when both +bound members report the same non-empty `NetworkZone` and the `provides` +side reports a `PeerEndpoint` (DD-4). Zone is deliberately opaque: an +in-cluster exporter takes it from the deployment — one value per cluster +network, which JEP-0016's provisioner sets for every Pod it renders — and an +edge exporter that reports none is never a direct candidate, so it keeps the +router path without any per-lease configuration. This keeps reachability a +statement someone made about the deployment rather than something the +controller infers from addresses it cannot test, which is the same reasoning +DD-13 applies to topology. A pair that claims a shared zone but cannot +actually connect is not a failure mode the user sees: the dial times out and +the router stream, already minted, carries the forward. **Reconnection belongs to the endpoint.** A forward outlives any single stream: an ingress reload, a router restart, or a node's network blip cuts @@ -1846,7 +1908,7 @@ the far end (DD-9, second pass). | `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | | Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | | Ingress/proxy reload cuts the peer stream | `requires` side re-dials and re-splices transparently; the first attempt after the cut is expected to fail. Observed in DD-9's third pass, and invisible to drivers that dial once | -| Direct dial fails (fast path) | Silent fallback to the router path; recorded as a metric | +| Direct dial fails or times out | Fall back to the already-minted router stream; recorded as a metric, and conspicuous for a same-zone pair that should have connected | | A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | | Client releases the lease | Forwards torn down first, then the lease ends normally | @@ -1882,7 +1944,6 @@ of the same kind as DD-9's rootcanal control hook: instant it starts, so the driver starts it only on a client call, after the forward is Ready, which is the same ordering rule `bt-peer` already follows. - - **`bt-peer` (phone).** The third pass of DD-9 showed the driver is one config away from being a phone rather than an audio source: with an HFP Audio Gateway alongside its A2DP source it satisfies both of a head unit's @@ -1890,7 +1951,8 @@ of the same kind as DD-9's rootcanal control hook: `profiles:` list, with the bond keystore persisted for the life of the lease so a peer restart does not strand the DUT's link key. -Both drivers are the reference `requires`/`provides` pair for projection. +The `cuttlefish` and `dhu` drivers are the reference `requires`/`provides` +pair for projection. What they must not do is know about each other: the phone driver exposes ports and the head unit driver dials `127.0.0.1:5277`, and the lease is the only place the two are joined. @@ -1929,7 +1991,13 @@ two-object design would face does not arise. - **The fast path is authenticated** — a direct peer connection presents `peer_token`; it is not trusted for being on the same network. Exporters supporting it open a listener, disabled by default and enabled per exporter - configuration. + configuration. Preferring it in-cluster (DD-4) does not widen what is + exposed: the peer listener is one authenticated port per exporter, distinct + from any device port, so a cluster can keep a `NetworkPolicy` that blocks + Pod-to-Pod access to simulator ports — which are *not* access-controlled — + while allowing the peer port between exporters. Shipping that policy with + the Pod is JEP-0016's job, and preferring direct makes it load-bearing + rather than advisory. - **`members` is immutable after creation**, so a bound lease cannot be widened beyond what it was authorized for. - **Physical RF is not access-controlled.** Two physical devices pairing in @@ -1948,7 +2016,11 @@ two-object design would face does not arise. JEP-0013 telemetry gains `lease.member` as a span attribute wherever `lease.name` already appears, so per-role activity is separable within one lease, plus per-forward counters (bytes each direction, reconnects, mode -actually used, direct-dial fallback rate). Reconnects are also an *event* on +actually used, direct-dial fallback rate). Because `Auto` now prefers direct +for same-zone pairs (DD-4), the fallback rate is a health signal rather than +a curiosity: an in-cluster bench silently running over the router means the +peer listener, the zone configuration or a `NetworkPolicy` is wrong, and it +should be visible as such. Reconnects are also an *event* on the forward, not only a counter: an endpoint driver that must re-establish protocol state after a cut (DD-10's head unit server) subscribes to it, and a bench whose forward is re-dialing is visibly degraded rather than quietly @@ -2006,7 +2078,10 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): every lease is either fully bound or holding nothing. - Lease expiry, explicit release, and client disconnect; assert no leaked router streams, no exporters left claimed, and no listeners left bound. -- Router-mode vs. direct-mode selection, including forced fallback. +- Router-mode vs. direct-mode selection: `Auto` picks direct for two + same-zone exporters, falls back to the router when the peer dial is + blocked, records the mode actually used, and never falls back under + `mode: direct`. - `jmp get lease -o mobly` output validated against Mobly's testbed schema. - **Compatibility**: an N-1 client against an N controller for the full single-exporter workflow; an N client issuing a single-exporter lease @@ -2094,6 +2169,9 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] Forwards establish over `RouterService` with no `router.proto` change - [ ] Direct fast path is authenticated, falls back automatically, and is observable (mode + fallback-rate metrics) +- [ ] `Auto` resolves to a direct peer connection for two same-zone + in-cluster exporters, and the fallback to the router is a measurable + event rather than the silent normal case - [ ] `bt-peer` participates as a `requires` endpoint with **no Python changes** — exporter configuration only, relying on its existing `open_transport(self.transport)` passthrough @@ -2146,7 +2224,8 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): `members`, `forwards`, and port reporting ship behind a controller feature gate, with Phases 1–3 complete. Signals sought: do real benches stay at two -members or grow; how often does the direct fast path apply; does anyone hit +members or grow; how often does the direct path actually apply, and how +often does a same-zone pair fall back; does anyone hit `MaxItems=8`; how often do `listen` collisions occur on physical hosts; does the nil-`status.exporterRef` convention (DD-2) surprise any consumer; is bind-time port validation (DD-12) painful enough to justify accelerating the @@ -2289,7 +2368,9 @@ risk in one field, handled by DD-2. the forwarded HCI splice going down 40 s later (DD-9, third pass). A multi-hour bench will meet this repeatedly. Mitigation: reconnection is the forward endpoint's responsibility, reconnects are observable, and the - HIL suite includes a bench that survives a deliberate stream cut. A + HIL suite includes a bench that survives a deliberate stream cut. An + in-cluster bench also avoids the machinery altogether, which is part of + why `Auto` prefers a direct connection there (DD-4). A related hazard is version skew across a splice — a newer client-side network driver against an older exporter closed every forwarded connection at the first byte with no error — which is an argument for the @@ -2464,10 +2545,15 @@ To resolve during review: To resolve during implementation: - Exact `ListenResponse` variant shape for forward setup instructions. -- Whether the fast path should race the router dial or attempt direct first - with a short timeout — a measurable question, and one whose stakes are now - bounded for simulated media: the router costs ~0.65 ms where the simulator - costs ~45 ms (DD-4). +- What the direct-dial timeout should be before falling back to the router. + Direct-first rather than racing is now the decision (DD-4); the number is + measurable and the stakes are bounded, since falling back costs ~0.65 ms on + the path measured so far. +- How an exporter's network zone is established: taken from deployment + configuration (the assumption in *Direct eligibility*), derived by the + controller from where the registration arrived, or probed. Configuration is + the least clever and the only one that works for an edge exporter that is + routable from the cluster but not the reverse. - Whether forward reconnect should preserve the medium's logical state or force a fresh pairing; likely protocol-specific. Two data points so far: Android Auto's head unit server must be restarted after a dropped session, @@ -2577,11 +2663,16 @@ Not part of this proposal: Media, a ringing call into AAOS Telecom). Router overhead measured at 0.65 ms against ~45 ms of rootcanal; proxy-reload stream loss, duplicate registration and version skew recorded as forward-endpoint duties -- 2026-09-02: those findings made normative — DD-4's fast path reframed as - an optimization with a measured price, a single-process invariant per - exporter identity, reconnection assigned to the forward endpoint with - reconnects as events, a cluster-data-path Risk, a *Forward resilience* - HIL test, and two acceptance criteria +- 2026-09-02: those findings made normative — DD-4's fast path given a + measured price, a single-process invariant per exporter identity, + reconnection assigned to the forward endpoint with reconnects as events, a + cluster-data-path Risk, a *Forward resilience* HIL test, and two acceptance + criteria +- 2026-09-02: `Auto` changed to **prefer a direct peer connection between + same-zone members** rather than defaulting to the router — DD-4 rewritten, + `PeerEndpoint`/`NetworkZone` added to `ExporterStatus`, `prefer_direct` + added to `DialPeerResponse`, direct-first-with-fallback replacing the + raced dial, and a *Direct eligibility* rule added to Design Details ## References From 6c24e986b8885b1be59273945bb2c63c9b9eab42 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 12:26:26 -0400 Subject: [PATCH 18/26] docs(jep-0017): say why a lease binds exporters, not devices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Motivation explained what is missing but never why the lease's unit is the exporter, which makes the proposal read like a granularity change. It is not one. A bench with several devices bolted to one harness is already an exporter: they share cabling, power and a host, one composite driver exposes them, and one lease hands over the whole assembly. Leasing a device inside that would let two clients hold opposite ends of one cable. The case this JEP is for is the opposite arrangement — a phone racked on one side of the lab and a head unit bench on the other, two exporters possibly on two hosts, with nothing physical between them and no reason to add it. Hence the framing: the unit stays the exporter, only the count changes, and the gain is combinatorial — N phones and M head unit benches give N×M benches out of N+M exporters, any pairing gang-scheduled, none pre-wired. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index e0bc0212a..4eb654295 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -52,6 +52,23 @@ That is the right primitive for the majority of HiL work — one board, one harness — but an entire class of tests is about *interaction between devices*, and today Jumpstarter cannot express it. +### Why the exporter, not the device, is the unit of a lease + +That granularity is deliberate, and this JEP keeps it. An exporter is a +bench: a DUT plus its harness, and often several devices bolted to that +harness and wired to each other. They share cabling, power and a host, so one +composite driver exposes them and one lease hands over the whole assembly. +Leasing a device *inside* such a bench would let two clients hold opposite +ends of one cable. + +The gap is the opposite arrangement: a phone racked on one side of the lab +and a head unit bench on the other — two exporters, possibly on two hosts (a +host can run several), with nothing physical between them and no reason to +add it. Only the scheduler and a data path can join those. So the unit stays +the exporter and only the count changes, which is what makes the gain +combinatorial: N phones and M head unit benches give N×M benches out of N+M +exporters, any pairing gang-scheduled, none pre-wired. + ### The concrete problem: devices that must talk to each other A large class of tests is not about a device but about an *interaction* @@ -2668,6 +2685,10 @@ Not part of this proposal: reconnection assigned to the forward endpoint with reconnects as events, a cluster-data-path Risk, a *Forward resilience* HIL test, and two acceptance criteria +- 2026-09-02: Motivation gained *Why the exporter, not the device, is the + unit of a lease* — harnessed multi-device benches are already one exporter + and stay that way; the gap is physically separate benches on separate + hosts, and the gain is gang-scheduling any combination of them - 2026-09-02: `Auto` changed to **prefer a direct peer connection between same-zone members** rather than defaulting to the router — DD-4 rewritten, `PeerEndpoint`/`NetworkZone` added to `ExporterStatus`, `prefer_direct` From 3aec37daa4a5e3f7117a3aac08bd671193a8b634 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 12:53:19 -0400 Subject: [PATCH 19/26] docs(jep-0017): de-vendor the JEP and tighten the prose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The document argued its case through one vendor's projection stack and a point-by-point comparison with one vendor's test framework. Both narrow the JEP: the mechanism is not specific to either, and a reviewer should not need to know either to evaluate it. Vendor naming is gone. The worked examples are now generic phone projection and Bluetooth pairing between devices: projection ports are `projection`/`projection-wifi`/`projection-rx`, the receiver driver is `projection-rx`, and the prior-art discussion describes what existing multi-device frameworks can and cannot do — gang-schedule device sets, but only on one lab host, with no path between devices — without naming or dissecting a particular one. Prose is tighter throughout. DD-9's three verification passes are consolidated into one result plus the duties it puts on drivers and the forward endpoint; DD-10 keeps its decision and evidence and sheds the recipe detail; Motivation, DD-13, Consequences, Risks, Future Possibilities, the test plan and the implementation history are all compressed. Every design decision, measurement and duty survives — what went is repetition and reproduction steps, which live in the experiment notes rather than the JEP. 2737 lines to 2422. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 1492 +++++++---------- 1 file changed, 590 insertions(+), 902 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index 4eb654295..e51de0912 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -85,24 +85,20 @@ between two of them. The shape recurs across domains: Every one of these needs the same two things Jumpstarter cannot give: **two devices held at once**, and **a path between them**. -**Phone projection is the worked example** used throughout this JEP, because -it exercises the hardest version of both requirements. It is a two-device -protocol by construction: the phone and the head unit first pair over -**Bluetooth**, then the head unit hands the session off to a peer-to-peer -**Wi-Fi** link, because Bluetooth lacks the bandwidth for continuous video. -Validating it means holding both devices *simultaneously*, keeping them -connected, driving both, and asserting on both — a phone-only or -head-unit-only lease cannot observe the handover at all. The same structure -holds for Android Auto and for Apple CarPlay, and for a head unit running -Android, Linux, or QNX; the pairing-then-handover pattern is a property of -projection, not of one vendor's stack. - -Two capabilities are missing from existing tooling, and projection tests need -both: **heterogeneous benches** — a Linux or QNX head unit alongside an -Android phone — and **pairing two virtual devices to each other**. - -Running that on Jumpstarter today requires the test author to hand-roll -everything the lease layer should provide: +**Phone projection is the worked example** throughout this JEP, because it +exercises the hardest version of both requirements. It is a two-device +protocol by construction: phone and head unit pair over **Bluetooth**, then +the head unit hands the session to a peer-to-peer **Wi-Fi** link, Bluetooth +having nowhere near the bandwidth for continuous video. Validating it means +holding both devices at once, driving both and asserting on both — a +phone-only lease cannot observe the handover at all. That structure holds for +every projection protocol in use and for head units running Android, Linux or +QNX; it is a property of projection, not of one vendor's stack. It also needs +two things existing tooling lacks: **heterogeneous benches**, and **pairing +two virtual devices to each other**. + +Running that on Jumpstarter today means hand-rolling everything the lease +layer should provide: - **No atomic acquisition.** The client requests two independent leases. If the second selector is unsatisfiable the first is already held, so the @@ -125,86 +121,60 @@ ergonomics change. For virtual devices the connectivity problem is sharper. Cuttlefish and the Android emulator can already pair virtual devices to each other — but only -within one host: - -- Cuttlefish shares its virtual radio media between instances launched from - a single `launch_cvd --num_instances=n`, or from separate `launch_cvd` - invocations that are pointed at the *same* `--vhost_user_mac80211_hwsim` - socket path and the same rootcanal/netsim daemon. Both mechanisms are - host-local Unix sockets and loopback TCP ports. -- The Android emulator's new networking backplane (36.5) is explicitly - described as bridging "all running instances on the same host machine". -- `podcvd`, Google's container wrapper, publishes each container's ports on - its own IP but keeps radio simulation inside the container group. - -So the state of the art for virtual multi-device Android testing is: *both -devices must live on one machine*. That constraint is exactly what a -cluster-scheduled, autoscaled pool of one-CVD-per-Pod exporters (JEP-0016) -breaks — and it is why JEP-0016's DD-8 deferred multi-CVD groups pending -"a cross-Pod virtual-radio story… real upstream-facing work." +within one host. Cuttlefish shares its virtual radio media between instances +launched together, or launched separately against the *same* `wmediumd` +socket and rootcanal/netsim daemon; both are host-local Unix sockets and +loopback TCP ports. The emulator's networking backplane is explicitly scoped +to "all running instances on the same host machine", and container wrappers +publish each container's ports while keeping radio simulation inside the +container group. + +So the state of the art for virtual multi-device testing is: *both devices +must live on one machine*. That is exactly the constraint a cluster-scheduled +pool of one-device-per-Pod exporters (JEP-0016) breaks, and why its DD-8 +deferred multi-device groups pending "a cross-Pod virtual-radio story… real +upstream-facing work." The encouraging part is that these simulators are reached over ordinary -sockets. `rootcanal`, the virtual Bluetooth controller, accepts HCI on a TCP -port — which is why `jumpstarter-driver-bt-peer` can already attach a -`bumble` peer to a CVD with `transport: "tcp-client:127.0.0.1:7300"`. -`netsim`, the Rust simulator that orchestrates Bluetooth, BLE, Wi-Fi and UWB -for both Cuttlefish and the emulator, accepts virtual chips over a -bidirectional gRPC stream (`PacketStreamer.StreamPackets`, whose first -`PacketRequest` carries a `ChipInfo` naming the device and chip kind). -`wmediumd` speaks over a frame socket. Every one of them is something a -process connects to. Nothing about "same host" is fundamental; it is an -artifact of the fact that nobody has forwarded those sockets between -machines under a common lease. - -### Prior art's ceiling, precisely - -Google's OmniLab Android Test Station is the closest existing system, and -being exact about what it can and cannot do matters, because the naive -version of this comparison is wrong. - -ATS 2.0 runs on Mobile Harness (open-sourced as `google/device-infra`), and -it **does** do multi-device. Ad-hoc testbeds gang-schedule N devices for one -job: `AdhocTestbedSchedulingUtil.findSubDevicesSupportingJob` performs -maximum-cardinality bipartite matching of `SubDeviceSpec`s against idle -devices, and `TestbedDevice`, `CompositeDevice`, and -`SubDeviceSynchronizationDriver` all exist. It also has a multi-host mode. -So "ATS cannot do multi-device" and "ATS is single-host" are both too strong. - -The actual constraint is one check in -`infra/controller/scheduler/simple/SimpleScheduler.java`. In `allocate`, -every device locator in an allocation must share a single `LabLocator`; -otherwise the allocation is refused with *"Lab locators do not match. Can not -create allocation"*. **A multi-device job is therefore confined to one lab -host.** Multi-host mode is one controller with N workers for fleet-level -pooling — it does not make a single job span hosts. And in either case there -is no bridged medium between roles: Tradefed states its own version of the -gap verbatim — *"No APIs yet exist to conduct operations from one device to -another, such as `device1.sync(device2)`."* - -Two further limits shape the opportunity. Mobile Harness's `Device` -abstraction is ADB-shaped and runs inside the lab-server JVM next to the -device, and its open-source platform coverage is `platform/android`, -`platform/androiddesktop`, and `platform/testbed` — there is no Linux, QNX, -serial, power, or CAN support, and extending it means in-tree Java and Bazel -rather than an out-of-tree package. +sockets: rootcanal accepts HCI on TCP — which is why +`jumpstarter-driver-bt-peer` can already attach a `bumble` peer with +`transport: "tcp-client:127.0.0.1:7300"` — netsim accepts virtual chips over +a bidirectional gRPC stream, and `wmediumd` speaks over a frame socket. +Nothing about "same host" is fundamental; it is an artifact of nobody having +forwarded those sockets between machines under a common lease. + +### Prior art's ceiling + +Existing multi-device test frameworks stop in the same place, and the limit +is structural rather than accidental. They *can* gang-schedule several +devices into one job — role declarations, composite devices and coordinated +setup steps are all standard — but the devices must hang off a single lab +host: allocation refuses a job whose devices are attached to different hosts, +and where a multi-host mode exists it pools jobs across hosts rather than +letting one job span them. None of them offers a path *between* two devices; +the frameworks say so themselves, noting the absence of any API for +conducting an operation from one device against another. Their device +abstractions also tend to be single-platform in practice, so a bench pairing +a phone with a Linux or QNX head unit is out of reach before scheduling is +even a question. This JEP targets exactly that seam: -- **Physical ↔ physical**: a phone and a head unit on two exporters, on - **different lab hosts**, in one lease — the allocation `SimpleScheduler` - refuses. -- **Virtual ↔ virtual**: two CVDs in two Pods, on two nodes, pairing over - Bluetooth and Wi-Fi — the thing that has not been done anywhere. +- **Physical ↔ physical**: two devices on two exporters on **different lab + hosts**, in one lease. +- **Virtual ↔ virtual**: two emulated devices in two Pods, on two nodes, + pairing over Bluetooth and handing a session to Wi-Fi. + **Scope.** This JEP covers **homogeneous benches** — two virtual devices, or two physical devices. Mixing them in one bench (a real phone paired to a virtual head unit) is a natural extension and the eventual prize, but it -requires real radio hardware bridging the two worlds and is deferred to keep -v1 tractable (DD-11). +needs real radio hardware bridging the two worlds and is deferred to keep v1 +tractable (DD-11). -The last one is the interesting business case: labs have scarce physical -head units and abundant phones (or the reverse), and virtualizing the -abundant half while keeping the scarce half real is a direct cost and -throughput win no host-local scheduler can offer. +The virtual case is the interesting business case: labs have scarce physical +head units and abundant phones, or the reverse, and virtualizing the abundant +half while keeping the scarce half real is a cost and throughput win no +host-local scheduler can offer. ### User Stories @@ -278,9 +248,8 @@ export: protocol: hci-h4 # optional ``` -`requires` is the genuinely new concept: a declaration that the driver will -dial a local address, and that the exporter should bind an inbound forward -there. +`requires` is the genuinely new concept: the driver will dial a local +address, and the exporter binds an inbound forward there. ```yaml export: @@ -295,11 +264,10 @@ export: protocol: hci-h4 # optional ``` -Note what did *not* change: `bt-peer`'s Python is untouched, and its -`transport` string still points at `127.0.0.1:7300`. The lease simply makes -that address resolve to a rootcanal in another Pod on another node. A driver -that can talk to a local service can now talk to a remote one without -knowing the difference. +Note what did *not* change: `bt-peer`'s Python is untouched and its +`transport` still points at `127.0.0.1:7300`. The lease makes that address +resolve to a rootcanal in another Pod on another node — a driver that can +talk to a local service now talks to a remote one without knowing. ### Declaring a bench @@ -307,7 +275,7 @@ knowing the difference. apiVersion: jumpstarter.dev/v1alpha1 kind: Lease metadata: - name: android-auto-bench + name: projection-bench namespace: jumpstarter-lab spec: clientRef: @@ -376,34 +344,34 @@ command set. ```console $ jmp create lease \ - --member phone=device-type=android-phone \ - --member headunit=device-type=aaos-headunit \ + --member phone=device-type=phone \ + --member headunit=device-type=headunit \ --forward bt=headunit.rootcanal:phone.controller \ --duration 45m -android-auto-bench +projection-bench -$ jmp get lease android-auto-bench +$ jmp get lease projection-bench NAME ENDED CLIENT EXPORTER AGE -android-auto-bench false ci-runner phone=rack3-pixel8, 12s - headunit=cf-auto-7b2c +projection-bench false ci-runner phone=rack3-phone-4, 12s + headunit=virt-hu-7b2c ``` Port names are discoverable rather than tribal knowledge, which is the point of putting them in the report (DD-7): ```console -$ jmp get exporter cf-auto-7b2c -o json | jq '.status.devices[].ports' +$ jmp get exporter virt-hu-7b2c -o json | jq '.status.devices[].ports' [{"name":"rootcanal","direction":"provides","protocol":"hci-h4"}] ``` In a shell, roles become top-level names alongside the usual driver clients: ```console -$ jmp shell --lease android-auto-bench -jumpstarter ⚡ android-auto-bench ➤ j phone adb shell getprop ro.product.model -Pixel 8 -jumpstarter ⚡ android-auto-bench ➤ j headunit power on -jumpstarter ⚡ android-auto-bench ➤ j forward status +$ jmp shell --lease projection-bench +jumpstarter ⚡ projection-bench ➤ j phone adb shell getprop ro.product.model +phone-under-test +jumpstarter ⚡ projection-bench ➤ j headunit power on +jumpstarter ⚡ projection-bench ➤ j forward status NAME FROM TO MODE STATE A→B B→A bt headunit.rootcanal phone.controller direct connected 1.2 MiB 0.9 MiB ``` @@ -418,7 +386,7 @@ from jumpstarter.config.client import ClientConfigV1Alpha1 config = ClientConfigV1Alpha1.load("default") with config.lease( - members={"phone": "device-type=android-phone", "headunit": "device-type=aaos-headunit"}, + members={"phone": "device-type=phone", "headunit": "device-type=headunit"}, forwards=[Forward("bt", frm=("headunit", "rootcanal"), to=("phone", "controller"))], duration=timedelta(minutes=45), ) as lease: @@ -434,7 +402,7 @@ with config.lease( phone.bt_peer.start({"name": "Bumble-Phone"}) phone.bt_peer.wait_connection(timeout=60) - assert phone.adb.shell("dumpsys activity | grep -c CarProjection") == "1" + assert phone.adb.shell("dumpsys bluetooth_manager | grep -c Connected") == "1" ``` `config.lease(selector=...)` without `members` behaves exactly as it does @@ -451,17 +419,17 @@ projected into the config formats existing Android multi-device tests already consume: ```console -$ jmp get lease android-auto-bench -o mobly > testbed.yml -$ mobly_test.py -c testbed.yml --test_bed android-auto-bench +$ jmp get lease projection-bench -o mobly > testbed.yml +$ mobly_test.py -c testbed.yml --test_bed projection-bench ``` which emits a Mobly testbed whose `AndroidDevice` controllers point at the per-member ADB endpoints Jumpstarter has forwarded locally, with the role -name carried through as the Mobly device label. A Mobile Harness -`SubDeviceSpec` mapping follows the same shape. This is the integration seam -with ATS/OmniLab: the tests and the results pipeline do not change, the -*bench* changes from "devices sharing one `LabLocator`" to "any mix of -physical and virtual devices anywhere the controller can reach." +name carried through as the Mobly device label; other frameworks' device-set +formats map the same way. This is the integration seam: the tests and the +results pipeline do not change, while the *bench* changes from "devices +sharing one lab host" to "any mix of physical and virtual devices anywhere +the controller can reach." ### How a forward comes up @@ -492,14 +460,14 @@ with one substitution — a router peer stream in place of the client stream: reach. ```text - ┌──────── Lease: android-auto-bench (one object) ─────────┐ + ┌──────── Lease: projection-bench (one object) ─────────┐ │ status.members: │ - │ phone → Exporter/rack3-pixel8 │ - │ headunit → Exporter/cf-auto-7b2c │ + │ phone → Exporter/rack3-phone-4 │ + │ headunit → Exporter/virt-hu-7b2c │ └───────┬─────────────────────────────────┬───────────────┘ │ │ ┌────────────▼────────────┐ ┌────────────▼────────────┐ - │ Exporter: rack3-pixel8 │ │ Exporter: cf-auto-7b2c │ + │ Exporter: rack3-phone-4 │ │ Exporter: virt-hu-7b2c │ │ bt_peer │ │ cuttlefish │ │ requires: controller │ │ provides: rootcanal │ │ listener 127.0.0.1:7300│ │ dials 127.0.0.1:7300 │ @@ -530,81 +498,74 @@ First, the **data plane** — ports that a forward carries: | rootcanal HCI (Cuttlefish, emulator) | `rootcanal` | provides | HCI on TCP (`7300 + rootcanal_instance_num`); hosts attach to it | | rootcanal link layer | `rootcanal-link` | provides | Controller-to-controller federation (`7400`, `7600` BLE); standalone rootcanal only, not netsim | | `wmediumd` / `mac80211_hwsim` | `hwsim` | provides | vhost-user, not a byte stream — reached through a frame bridge (DD-10) | -| Android Auto head unit server (phone, developer mode) | `aa-hu`, `aa-hu-wifi` | provides | Same service, via `adb forward` (USB-like) or the guest's Wi-Fi address (wireless-like); the head unit dials it | -| Android Auto receiver (Desktop Head Unit) | `phone` | requires | Google's public receiver; dials the phone's port (DD-10) | -| Android Auto wireless receiver (GAS head unit) | `aa-wireless` | provides | TCP 5288 on the head unit; the phone dials it after the Bluetooth handover — physical head units only | +| Projection server on a phone (developer mode) | `projection`, `projection-wifi` | provides | Same service, reached over ADB (USB-like) or the guest's Wi-Fi address (wireless-like); the receiver dials it | +| Projection receiver (desktop or head unit) | `phone` | requires | Dials the phone's port (DD-10) | +| Wireless projection receiver on a head unit | `projection-rx` | provides | A TCP port on the head unit; the phone dials it after the Bluetooth handover — physical head units only | | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | Second, the **control plane** — drivers that configure and observe a medium without carrying its traffic. `jumpstarter-driver-netsim` is the worked -example: it speaks netsim's REST API (`7681 + netsim_instance_num`) to list -devices, toggle radios, patch state, reset, and start/stop/download **pcap -captures** of the simulated air. It is not a forward endpoint and needs none -of this JEP's machinery; the two compose, with the netsim driver observing -the medium a forward connects. - -Two consequences of the data-plane table are worth stating plainly, and both -generalize beyond radios. First, HCI is **asymmetric**: a host attaches to a -controller, so forwarding one device's *HCI* port into another's achieves -nothing — which is precisely why forwards are directional and why a -`provides → provides` forward is rejected (DD-6). Note this is a statement -about HCI specifically, not about controllers: rootcanal exposes a separate -**link-layer** port for joining controllers to each other, which is why the -table lists both (DD-9). Second, not every -port is a transparent splice — netsim's `PacketStreamer` requires the -attaching side to originate a `StreamPackets` call carrying `ChipInfo` before +example: it speaks netsim's REST API to list devices, toggle radios, patch +state, reset, and start/stop/download **pcap captures** of the simulated air. +It is not a forward endpoint and needs none of this JEP's machinery; the two +compose, with the netsim driver observing the medium a forward connects. + +Two consequences of the table generalize beyond radios. HCI is +**asymmetric** — a host attaches to a controller, so forwarding one device's +HCI port into another's achieves nothing, which is why forwards are +directional and `provides → provides` is rejected (DD-6); joining two +controllers is a different port, which is why the table lists both (DD-9). +And not every port is a transparent splice: netsim's `PacketStreamer` +requires the attaching side to originate a call carrying `ChipInfo` before traffic flows, so its consumer is a protocol-terminating driver rather than a raw socket. Forwards carry both kinds; the difference lives in the driver at the `requires` end. -Note the asymmetry between the two supported bench kinds. For **two virtual -devices** the medium is simulated, so a forward carries it and a shared -controller mediates it. For **two physical devices** the radio medium is the -air: real devices in RF range pair without any forward at all, and forwards -carry only the *wired* media of such a bench — a CAN segment, a serial -cross-over. The lease plane is what physical benches need most, because it is -what lets their two exporters live on different hosts. +The two bench kinds are asymmetric too. For **two virtual devices** the +medium is simulated, so a forward carries it. For **two physical devices** +the medium is the air: devices in RF range pair with no forward at all, and +forwards carry only that bench's wired media — a CAN segment, a serial +cross-over. What a physical bench needs is the lease plane, which is what +lets its two exporters live on different hosts. ### A projection bench, concretely -The Android Auto bench in the reference implementation has one shape that -was verified end to end (DD-10) and one that follows from it. - -**Virtual.** The phone is a Cuttlefish exporter booting Google's GMS GSI — -the phone stack Google ships, not an AOSP approximation — with Android Auto -installed and its developer-mode head unit server started by the driver. The -head unit is *not* an AAOS CVD, which cannot receive projection because the -receiver is part of GAS; it is Google's Desktop Head Unit, run by a `dhu` -driver as a process inside the head-unit exporter. The DHU is a TCP client, -so it is the `requires` side, and the phone `provides` the port it dials: +**Virtual.** The phone is a Cuttlefish exporter booting a vendor phone image +— the stack that ships on retail devices, not an AOSP approximation — with +the projection app installed and its developer-mode server started by the +driver. The head unit is a *receiver* run as a process inside the head-unit +exporter by a `projection-rx` driver, because an automotive Android CVD +cannot receive a session: the receiver is part of the vendor stack, not of +AOSP. The receiver is a TCP client, so it is the `requires` side and the +phone `provides` the port it dials: ```yaml spec: members: - name: phone - selector: { matchLabels: { device-type: android-phone, gms: "true" } } + selector: { matchLabels: { device-type: phone, projection: "true" } } - name: headunit - selector: { matchLabels: { device-type: aa-headunit } } + selector: { matchLabels: { device-type: projection-receiver } } forwards: - - name: projection + - name: session between: - - { member: phone, port: aa-hu-wifi } + - { member: phone, port: projection-wifi } - { member: headunit, port: phone } ``` -`aa-hu-wifi` reaches the head unit server through the guest's Wi-Fi -interface — the phone's side of a wireless session — where `aa-hu` would -reach it through `adb forward`, the shape of a USB session. The forward is -the same either way; the port name is the test's statement of which cable it -is pretending to be (DD-13). +`projection-wifi` reaches the phone's server through the guest's Wi-Fi +interface — the shape of a wireless session — where `projection` reaches it +over ADB, the shape of a USB session. The forward is the same either way; the +port name is the test's statement of which cable it is pretending to be +(DD-13). -**Physical.** A GAS head unit on one exporter and a phone on another, on +**Physical.** A head unit on one exporter and a phone on another, on different hosts. The radios are the air, so the handover needs no forward; -the lease plane is what this bench needs, and its forwards carry only wired -media — a CAN segment feeding the head unit's VHAL, a serial console. A -`dhu` member can stand in for the head unit here too, which gives a -physical phone a hardware-free receiver. +what this bench needs is the lease plane, and its forwards carry only wired +media — a CAN segment feeding the head unit, a serial console. A software +receiver can stand in for the head unit here too, giving a physical phone a +hardware-free counterpart. ### API / Protocol Changes @@ -983,7 +944,7 @@ if that dial does not complete. Three reasons, in order of weight: also mean paying to leave and re-enter the network. 2. **The router path in a typical install leaves the cluster and comes back.** It therefore inherits the ingress's failure modes — the reload - that cut a live bench's HCI splice in DD-9's third pass was an ingress + that cut a live bench's HCI splice during verification was an ingress worker drain, not a Jumpstarter fault. A direct forward between two Pods stays inside the CNI and never meets that machinery. 3. **The workloads that are latency-sensitive are the ones still ahead.** @@ -1156,302 +1117,149 @@ Option 2 is the Phase 1 default, option 1 the fallback, option 3 the general answer beyond Cuttlefish. **Rationale:** Cuttlefish already solves multi-device Bluetooth on one host, -and reading how tells us what to forward. In -`assemble_cvd`, all four rootcanal ports are derived from -`rootcanal_instance_num` rather than from the CVD's own instance number — -`hci 7300+N`, `link 7400+N`, `test 7500+N`, `link_ble 7600+N` — and -`--rootcanal_instance_num` is documented as *"use an existing rootcanal -instance which is launched from cuttlefish instance with -rootcanal_instance_num."* Sharing a controller between devices is therefore a -first-class, supported configuration, and the guest reaches it through a TCP -connector on `rootcanal_hci_port`. Everything host-local about it is a -**port**, which is exactly what this JEP forwards. - -Option 2 is preferred because it is symmetric. Each device keeps its own -controller, so neither exporter's failure removes the other's radio, and -rootcanal exposes `link_port` / `link_ble_port` specifically for joining -controllers to one another — distinct from the HCI port that hosts use. It -needs no new code at all: a `TcpNetwork` child on 7400 and a forward. - -Option 1 is the fallback if link-layer federation does not behave as the flag -names imply. It is equally free of new code, but asymmetric — one exporter's -rootcanal becomes the medium for both, so that Pod becomes a single point of -failure for the bench. - -Option 3 is the general answer, and the only one that works outside -Cuttlefish. Bumble's virtual `Controller` attaches to a link-layer bus, -several controllers on one bus exchange broadcast advertising and unicast ACL -data, and `RemoteLink` carries that bus over a WebSocket relay hosting -virtual *rooms* — the N-way medium DD-6 deferred. Its `android-netsim` -transport has a `mode=controller` documented as replacing netsim outright. -Choose it when a bench includes a non-Cuttlefish device, when more than two -participants share a medium, or when the medium itself needs to be scripted -or instrumented — a Python controller can inject faults a C++ simulator will -not. - -One constraint separates them in practice: `link_port` is passed only to -standalone rootcanal, never to netsim, which takes `--hci_port` alone. A -device configured to route its radios through netsim therefore has options 1 -and 3 available but not option 2. - -Bumble is also already a dependency: `jumpstarter-driver-bt-peer` is built on -it and passes its `transport` string directly to `bumble.transport. -open_transport`, then uses the result as `controller_source`/`controller_sink` -for a Bumble `Device`. Any Bumble moniker therefore already works with no -Python change, which is what makes the `requires` side of this design free. -The gap is that the driver only ever constructs a `Device` — a host — so the -controller side is new code, and it is small. - -Option 4 was in an earlier draft of this JEP and is rejected as invented -machinery: it added a driver interface, a driver tier, and a medium taxonomy -to express something existing network drivers plus a direction already -express. Option 5 is rejected because it changes the device under test — a -guest-side shim means the Bluetooth stack being exercised is not the one that -ships, invalidating precisely the pairing and handover behavior these tests -exist to verify. - -Worth noting that option 2 is an instance of a general pattern rather than a -Bluetooth special case. What a bench needs is a **counterparty component**: -something that presents itself to the DUT as whatever the DUT expects on the -other side of the medium. For Bluetooth that is a controller shared by two -hosts; for a vehicle bus it is a *restbus* — a simulation of the remaining -ECUs, a long-established practice in automotive integration testing. Both are -ordinary `provides` ports under this design, so the same forward machinery -serves them and no medium-specific mechanism is required. Integrating a -concrete restbus is future work (see Future Possibilities). - -One caveat is carried rather than hidden: for option 3 a radio medium in -Python is comfortable for advertising, pairing, and control while being an -open question for sustained A2DP-class traffic, which compounds rather than -relieves the latency risk already recorded; options 1 and 2 keep the medium -in native C++ and avoid that entirely. - -**Verified against real CVDs (2026-09-01).** Options 1 and 2 were both -exercised by hand on a single-node kind cluster with two Cuttlefish exporter -Pods — an `aosp_cf_x86_64_auto` head unit in Pod A and an -`aosp_cf_x86_64_only_phone` in Pod B (AOSP build 16102939, cuttlefish host -package 1.55.1). No Jumpstarter code was involved: the forward endpoint was -played by a 50-line asyncio TCP relay inside each Pod, and Pod-to-Pod -traffic went over the Pod network directly, which is what DD-4's Direct mode -will do. The router path is therefore still unmeasured; everything below is -about whether the *ports* behave when the other end is in another Pod. - -Both options produced the same result, driven purely through `adb` and -`uiautomator`: BR/EDR inquiry lists the phone on the head unit, SSP numeric -comparison shows the same passkey on both screens, the bond completes with a -16-byte link key, and HFP, A2DP and AVRCP connect. Under option 2 the phone -then streamed a ringtone to the head unit over A2DP for ~38 s at ~42 KB/s -on the federated link, with no disconnect. Boot-to-bond is about two minutes -of wall clock, most of it guest boot. - -*Option 2 — link-layer federation.* Both CVDs launched with -`--netsim_bt=false`, so each Pod runs its own rootcanal. Standalone rootcanal -binds **all four** ports (`hci`, `link`, `test`, `link_ble`) on `0.0.0.0`, -so the join needed no relay at all: from Pod B's test channel, -`add_remote 7400 BR_EDR` and `add_remote 7600 LOW_ENERGY` produced a -`link_layer_socket_device` on the BR/EDR and LE phys of *both* models. Three -things the flag names do not tell you: - -- **BD_ADDR collision.** Every standalone rootcanal numbers its first HCI - device `da:4c:10:de:00:00`, so two federated CVDs start with the same - address and cannot pair. The fix used was `set_device_address` on B's test - channel followed by a Bluetooth off/on in the guest so the stack re-reads - its address; `--rootcanal_default_commands_file` is the launch-time - equivalent. Under option 1 a single model numbers every device and the - problem does not arise. -- **The join is an action, not a wiring.** `add_remote` is a runtime - test-channel command that dials outward, so option 2 needs *two* forwarded - ports (BR/EDR and LE link) plus a control step on the `requires` side - after the forward is up — a natural post-establish hook for a rootcanal - driver, but not something a static `forwards[]` entry expresses alone. - Standalone rootcanal is also not the Cuttlefish default; netsim is. -- **Beacons leak.** The remote model's two default LE beacons appear in the - peer's scan results, because the LE phys are joined wholesale. - -*Option 1 — shared medium.* Pod A kept the default (netsim-backed) radio; -Pod B launched with `--netsim_bt=false --rootcanal_instance_num=2`, whose -effect is that B starts no controller at all and its `tcp_connector` dials -`127.0.0.1:7301` for HCI. A relay listening on B's `7301` and delivering to -A's `7300` made netsim in Pod A list both chips in `/v1/devices`. Two facts -matter for the design: netsimd binds its HCI port on **loopback only**, so a -forward endpoint *inside* Pod A is required rather than merely tidy (which -the design already provides); and the guest dials at boot, so the forward -must be up before the second CVD launches — a pre-launch ordering constraint -where option 2 has a post-launch one. The entire discovery-pairing-profile -flow moved ~140 KB phone→controller and ~13 KB back. - -*Both.* crosvm's minijail sandbox cannot mount `/dev` inside a Pod, so both -launches needed `--enable_sandbox=false`; this is a JEP-0016 deployment -detail, recorded here because it cost the most time. Neither experiment -changes the decision — option 2 stays the Phase 1 default on its symmetry — -but the address and join findings above are why the Phase 1 driver work is -"a rootcanal control hook", not "nothing". - -**Verified again with a GMS phone and the merged `bt-peer` driver -(2026-09-01, second pass).** The same two Pods, with Pod B now running the -GMS GSI phone of DD-10 instead of the AOSP phone, and the peer role played -by real Jumpstarter code for the first time. - -*Regular pairing with a GMS phone.* The UI-driven flow — inquiry, SSP -numeric comparison, bond — succeeded as before, then stalled: no profile -connected, the head unit logged `CachedBluetoothDevice: No profiles`, and -the ACL dropped on `l2c_link_timeout` after SDP. The medium was not the -cause. The GSI enables only the LE-audio `bluetooth.profile.*` properties; -the classic set — A2DP source, AVRCP target, HFP AG, MAP and PBAP server, -PAN, HID, OPP — is product configuration in `/system/build.prop` (the AAOS -image carries its own sink-side set there), and the GSI's `system.img` -replaced it. Injecting the phone-side set into the GSI's `build.prop`, the -same in-place ext4 edit as the ADB fix, produced the full stack: A2DP sink, -AVRCP controller with browsing and BIP cover art, HFP client carrying the -phone's network and subscriber indicators, and PBAP client pulling the -phonebook over L2CAP, all `Connected`. YouTube Music on the phone then -streamed AAC 44.1 kHz stereo to the head unit over the federated link, and -the phone reconnected on its own after a Bluetooth off/on. Which profiles a -bench has is decided by the phone image, which makes this a second -image-preparation duty for the `cuttlefish` phone driver next to the ADB -one (*Reference drivers for projection*). - -*`jumpstarter-driver-bt-peer` as the `requires` side.* The driver's -`BtPeer` class was run unchanged with -`transport: "tcp-client:127.0.0.1:17300"`, where `17300` was a -`kubectl port-forward` to Pod A's rootcanal `hci_port` — byte for byte what -a `bt=headunit.rootcanal:phone.controller` forward delivers, with the peer -end outside the cluster. rootcanal attached it as a second HCI device on the -head unit's own medium; the head unit discovered *Bumble-Phone*, SSP-paired -it (bond, encrypted ACL), listed it *Connected* with Media active, and -opened AVDTP to the peer's SBC source (`avdtp_connected` in the driver's -event log). That is the claim under *Ports* — "`bt-peer`'s Python is -untouched" — demonstrated on the merged driver. It exercises Bumble as a -*host*; the open question about Bumble as the *controller* (option 3) is -unchanged. - -*Three constraints that change the driver's duties.* - -- **The test channel is fragile, and the guest does not recover.** A - latency probe that opened and closed rootcanal's `test_port` twenty times - aborted rootcanal in *both* Pods — `test_channel_transport.cc: Check - failed: written == size, errno = 32` in `SendResponse`: a client that - closes before the banner is written is fatal. `process_restarter` - respawned rootcanal within a second, but the guest's side of the HCI link - (`tcp_connector`) took `SIGPIPE`, is not under the restarter, and the - guest's Bluetooth sat in `BLE_TURNING_ON` until `cvd restart` (about - 20 s; userdata, bonds and installed apps survive it, where `cvd rm` - discards them). So: the test channel is never a `provides` port — the - rootcanal driver keeps it private and forwards only the HCI and link - ports; a forward endpoint must not verify liveness by connect-and-close, - which is why forward state comes from the splice and not from probing; - and a controller crash on one exporter strands every peer's guest with a - dead controller while the lease still looks bound, so the driver watches - its controller and reports the forward `Failed` and the lease `Degraded` - rather than leaving a bench that cannot pair. -- **Addresses follow attachment order, and numbers are reused.** rootcanal - names the *n*-th live HCI device `da:4c:10:de:00:` per model, so the - Bumble peer took `:00:01` on attaching — the address the federated phone - already held. The BD_ADDR finding above therefore generalises from "two - CVDs" to "any host that attaches": the rootcanal driver assigns a - per-member address (from the member index, say) with `set_device_address` - in its post-establish hook, and before the host powers on, because SC - pairing binds the address into the key derivation. -- **Every Cuttlefish guest has the same address plan.** Both instances came - up with mobile data at `192.168.97.2`, Ethernet on `192.168.98.0/24` and - Wi-Fi on `192.168.99.0/25` behind an AP built from the same OpenWrt - rootfs. An L4 forward never sees this; anything that bridges guests at L2 - or L3 — the Phase 4 frame bridge, a Wi-Fi Direct emulation — must NAT or - re-address, one more reason DD-10's option 3 goes first. Pod-to-Pod TCP - connect on this single-node cluster was ~0.1 ms median; cross-node is - still unmeasured. And because standalone rootcanal binds every port on - `0.0.0.0`, it is the cluster's network policy, not the simulator, that - keeps another Pod from `add_remote`-ing into a bench (*Security*). - -The Phase 1 rootcanal hook is therefore three concrete things: a private -test channel, per-member addresses, and a controller watch. - -**Verified over a real router forward, with Bumble as the phone -(2026-09-02, third pass).** The second pass reached rootcanal through a -`kubectl port-forward`; this pass replaced it with Jumpstarter's own data -path and removed the phone CVD entirely. Three Pods: the head unit's Pod -gained a second, hook-less exporter identity that exports its rootcanal -`hci_port` as a `TcpNetwork` (plus an echo port, for measurement); a bt-peer -Pod ran the merged driver with `transport: "tcp-client:127.0.0.1:7300"`; and -a sidecar in that Pod held a lease on the rootcanal exporter and published -both ports locally with `TcpPortforwardAdapter` — a hand-rolled stand-in for -the forward endpoint this JEP specifies, with the same shape: one lease, a -listener on the `requires` side, `RouterService.Stream` in the middle. Every -HCI byte the peer sent crossed bt-peer Pod → router → head unit Pod, and the -head unit paired with it: discovered *Bumble-Phone*, SSP-bonded, encrypted -ACL, listed `Connected` with **Phone and Media both on**. - -*A phone, not just a headset.* The merged `BtPeer` presents an A2DP source, -which lights Media. Subclassing it to add an HFP **Audio Gateway** — a -`bumble.rfcomm` server, `hfp.AgProtocol` with the call/callsetup/callheld, -service, signal, roam and battery indicators, and `hfp.make_ag_sdp_records` -in the device's SDP database — lit Phone as well: `hfp_slc_complete`, -codecs `CVSD`/`MSBC` negotiated, `avdtp_connected`, and the head unit's -`HeadsetClientStateMachine` `Connected`. Ringing the head unit from the peer -(`callsetup=1` + `RING` + `+CLIP`) put a call into AAOS Telecom — -`SET_RINGING (successful incoming call)` against -`HfpClientConnectionService`, phone account `HFP …:00:01`. So the phone-facing -half of a head-unit bench needs no second guest at all: the cheapest bench -is one CVD and a Python process, and the `bt-peer` driver should grow a -`profiles:` config (`a2dp-source`, `hfp-ag`, later AVRCP target and PBAP -server) rather than a second driver. - -*What the router costs.* On the same endpoint, from the same Pod: a -round-trip through the forward was **0.70 ms median** (p90 0.91 ms, n=100) -against **0.05 ms** direct Pod-to-Pod — about 0.65 ms of router. An HCI -command round-trip to rootcanal was **~45 ms median through the forward and -~45 ms direct**: the simulator's own scheduling dominates by two orders of -magnitude, and the router is ~1.5% of an HCI exchange. Simulated Bluetooth -pairing therefore cannot tell the two transports apart — which is what makes -DD-4's preference for a direct in-cluster connection free rather than a -gamble, since falling back to the router is undetectable on this path. The -*Latency characterization* test exists to find where that stops being true -(A2DP-class throughput, cross-node, physical controllers, and the Phase 4 -frame bridge, where it is expected to). - -*Three operational constraints, all in the forward endpoint's lap.* - -- **An ingress reload cuts every long-lived stream, and nothing re-dials.** - This cluster's nginx ingress reloads on any `Ingress` change — 85 times in - one log — and each reload drains its old workers with - `worker_shutdown_timeout 240s`. The arithmetic is visible end to end: a - reload at 04:00:13, the exporter's controller status stream cut at - 04:04:13.6, the forwarded HCI splice broken at 04:04:55, and the client's - next driver call failing `UNAVAILABLE: Socket closed`. The exporter - reconnected its own status stream in half a second; the *forward* did not, - so the peer's HCI transport stayed dead and the bench looked bound while - the simulated phone was off the air. Immediately after such an event the - first new connection to the listener is also reset, and the next one - succeeds. Nothing about this is Bluetooth-specific or exotic — any - cluster whose proxy reloads has it — so reconnection belongs to the - forward endpoint, not to each driver: the `requires` side re-dials its - peer stream and re-splices transparently, because a driver that opens its - transport once at start, as `bt-peer` and every HCI or serial-style client - does, has no way to notice or recover. Drivers that genuinely must know — - the head unit server of DD-10 — get the reset as an event, which is the - same rule *Reference drivers for projection* already states. -- **One identity, one process.** Running the exporter twice under the same - identity — trivially, a Deployment scaled to two — leaves both registered; - connections then land on whichever process the router picks, and the one - that does not own the lease's session answers `RouterService.Stream` with - a `KeyError` on the driver UUID and closes. It cost an hour of chasing a - "flaky tunnel". A member exporter is single-writer: the deployment runs - one replica, and a duplicate registration is worth detecting rather than - tolerating. -- **Both ends must be the same build.** A forward endpoint running a newer - `jumpstarter-driver-network` than the exporter's runtime accepted - connections and returned EOF on the first byte, with no error on either - side. Version skew between the two halves of a splice is silent, which is - an argument for the forward endpoint being the exporter's own code — - as specified — rather than an arbitrary client-side helper. - -Two smaller notes for whoever writes the tests. Bumble's bonds live in -memory unless a keystore is configured, so restarting the peer invalidates -the DUT's stored link key and the DUT must forget the bond before re-pairing -— the driver should persist its keystore per lease, or the test should -forget first. And deleting a `Lease` object out from under a waiting client -leaves that client retrying `not found` forever instead of re-queueing: -leases are released, not deleted (*Lease state*). +and how it does so says what to forward: all four rootcanal ports derive from +`rootcanal_instance_num` rather than the CVD's own instance number +(`hci 7300+N`, `link 7400+N`, `test 7500+N`, `link_ble 7600+N`), and sharing +one controller between devices is a documented, first-class configuration +that the guest reaches through a TCP connector. Everything host-local about +it is a **port**, which is what this JEP forwards. + +Option 2 is preferred for symmetry: each device keeps its own controller, so +neither exporter's failure removes the other's radio, and rootcanal exposes +`link_port`/`link_ble_port` specifically for joining controllers to each +other. It needs no new code — a `TcpNetwork` child and a forward. Option 1 is +the fallback if federation misbehaves; equally free, but asymmetric, since one +Pod becomes the medium and therefore a single point of failure. One +constraint separates them: `link_port` exists only on standalone rootcanal, +never on netsim, so a device routing its radios through netsim has options 1 +and 3 but not 2. + +Option 3 is the general answer and the only one that works outside +Cuttlefish. Bumble's virtual `Controller` attaches to a link-layer bus that +several controllers can share, and its `RemoteLink` carries that bus over a +relay — the N-way medium DD-6 deferred. Choose it for a non-Cuttlefish +device, for more than two participants, or when the medium itself must be +scripted or fault-injected, which a Python controller does and a C++ +simulator does not. Bumble is already a dependency: +`jumpstarter-driver-bt-peer` passes its `transport` string straight to +`bumble.transport.open_transport`, so any transport moniker works with no +Python change — that is what makes the `requires` side free. The driver only +ever builds a `Device`, a host, so the controller side is new code, and small. + +Options 4 and 5 are rejected. Option 4 invents machinery — a driver +interface, a driver tier and a medium taxonomy — to express what network +drivers plus a direction already express. Option 5 changes the device under +test: a guest-side shim means the stack being exercised is not the one that +ships, invalidating the pairing and handover behavior these tests exist to +verify. + +Option 2 is an instance of a general pattern rather than a Bluetooth special +case. What a bench needs is a **counterparty**: something presenting itself +to the DUT as whatever it expects on the other side of the medium. For +Bluetooth that is a shared controller; for a vehicle bus it is a *restbus*, +simulating the remaining ECUs. Both are ordinary `provides` ports, so the +same machinery serves them (integrating a real restbus is future work). + +One caveat is carried rather than hidden: a Python medium is comfortable for +advertising, pairing and control but an open question for sustained +A2DP-class traffic, which compounds the latency risk already recorded. +Options 1 and 2 keep the medium in native C++ and avoid it. + +**Verified on real devices (2026-09-01 and 2026-09-02).** Options 1 and 2 +were exercised by hand on a single-node kind cluster with two Cuttlefish +exporter Pods — an automotive head unit in one, a phone in the other — first +with a 50-line TCP relay standing in for the forward endpoint, then over +Jumpstarter's own router path with the merged `jumpstarter-driver-bt-peer` +playing the phone. Driven through `adb` alone, every variant produced the +same result: inquiry, SSP numeric comparison showing one passkey on both +screens, a bond, and HFP, A2DP and AVRCP `Connected`, with music streaming +continuously over the link and the phone reconnecting by itself after a +Bluetooth off/on. Boot-to-bond is about two minutes, most of it guest boot. +Which profiles appear at all is decided by the phone *image*, not by the +medium: a plain GSI enables only the LE-audio profile properties, and the +classic set had to be added to the image before anything beyond the bond +connected. None of this changes the decision — option 2 stays the Phase 1 +default on its symmetry — but it turns "nothing to build" into a specific +list of duties. + +*What the ports actually require.* Under option 2 both CVDs run standalone +rootcanal (`--netsim_bt=false`), which binds all four ports on `0.0.0.0`; +the join itself is a runtime `add_remote` command on the *test* channel, so +federation needs two forwarded link ports **plus a control step after the +forward is up** — a post-establish hook, not something a static `forwards[]` +entry expresses. It also joins the LE phys wholesale, so the peer's beacons +appear in local scans. Under option 1 the second CVD starts no controller and +its guest dials the shared HCI port at boot, so the forward must exist +*before* that CVD launches — a pre-launch ordering constraint where option 2 +has a post-launch one — and because netsim binds its HCI port on loopback +only, the forward endpoint must live inside the owning Pod rather than +merely nearby, which is what this design already provides. + +*The peer side is free, and can be the whole phone.* `bt-peer` ran unchanged +as the `requires` end, its `transport` pointed at a forwarded HCI port: the +head unit discovered it, SSP-paired it, and opened AVDTP to its A2DP source +— the claim under *Ports* that the driver's Python is untouched, demonstrated +on merged code. Adding an HFP Audio Gateway alongside that source (a +`bumble.rfcomm` server plus `hfp.AgProtocol` and its SDP record) made the +head unit show both Phone and Media, and a simulated inbound call from the +peer arrived in the head unit's telephony stack as a ringing call. The +cheapest useful bench is therefore one virtual device and a Python process, +and `bt-peer` should grow a `profiles:` config rather than a sibling driver. +This exercises Bumble as a *host*; option 3, Bumble as the *controller*, is +still unverified. + +*What the router costs.* Measured from one Pod against the same endpoint: a +round-trip through a router-carried forward was **0.70 ms** median (p90 +0.91 ms, n=100) against **0.05 ms** direct Pod-to-Pod, while an HCI command +to rootcanal took **~45 ms either way** — the simulator's own scheduling +dominates by two orders of magnitude. Pairing cannot tell the transports +apart, which is what makes DD-4's preference for a direct in-cluster +connection free rather than a gamble. Where that stops being true is what the +*Latency characterization* test is for. + +*Duties this puts on the drivers and the forward endpoint.* + +- **Keep the test channel private, and watch the controller.** A client that + connects to rootcanal's test port and closes before its banner is written + aborts the process (`Check failed: written == size, errno = 32`). + `process_restarter` brings rootcanal back, but the guest's connector took + `SIGPIPE`, is not restarted, and its Bluetooth stays half-up until a `cvd + restart`. So the test channel is never a `provides` port, a forward + endpoint never probes liveness by connect-and-close — state comes from the + splice — and a driver whose controller dies reports the forward `Failed` + and the lease `Degraded` instead of leaving a bench that cannot pair. +- **Assign addresses per member, before the host powers on.** rootcanal names + the *n*-th live device on a model `da:4c:10:de:00:` and reuses numbers, + so two federated CVDs start life with the same BD_ADDR and any attaching + peer collides with an existing one. The rootcanal driver sets an address + per member in its post-establish hook, ahead of power-on, because secure + pairing binds the address into key derivation. +- **Expect identical guest networks.** Every Cuttlefish guest boots the same + address plan behind an identical AP, so anything bridging guests at L2 or + L3 — the Phase 4 frame bridge, a Wi-Fi Direct emulation — must NAT or + re-address. An L4 forward never sees it. +- **Reconnect, and do it in the endpoint.** The verification cluster's + ingress reloads on any unrelated `Ingress` change and drains its workers + 240 s later, which cut the exporter's controller stream and the forwarded + HCI splice within seconds of each other; the exporter re-dialed its own + stream, the forward did not, and the bench stayed `Ready` with the + simulated phone off the air. A driver that opens its transport once at + start cannot notice this, so the `requires` side re-dials and re-splices + transparently and the reconnect reaches drivers that care as an event. +- **One process per exporter identity.** Two processes registered under one + identity split connections between them; the one that does not own the + lease's session fails `RouterService.Stream` with a `KeyError` and closes, + which presents as a flaky forward rather than a misconfiguration. +- **Match versions across a splice.** A forward endpoint built against a + newer network driver than the exporter's runtime accepted connections and + returned EOF at the first byte, silently — an argument for the endpoint + being the exporter's own code, as specified. + +Two notes for whoever writes the tests: a Bumble peer keeps its bonds in +memory unless a keystore is configured, so restarting it invalidates the +DUT's link key and the DUT must forget the bond first; and deleting a `Lease` +object out from under a waiting client leaves that client retrying +`not found` forever — leases are released, not deleted (*Lease state*). ### DD-10: Wi-Fi and projection — what a forward carries @@ -1474,45 +1282,41 @@ the byte forward. That is Phase 4. **Rationale:** An earlier draft rejected option 3 as a fidelity failure — "it verifies that a TCP proxy works" — and kept it only as a diagnostic. -Running it changed that judgment. With the two Pods of the DD-9 experiment, -a GMS phone image and Google's own receiver projected a live Android Auto -session across a single forwarded port (the verification below). Version -negotiation, the TLS authentication between GMS and the receiver library, -service discovery, the phone-side first-run flow, and the rendered launcher -with video, audio and input all ran unchanged. That is not a proxy check; -it is the whole projection stack running on two exporters. - -What option 3 does not exercise is precisely bounded: the **handover** — -the credential exchange over Bluetooth, the phone joining the head unit's -Wi-Fi Direct group — and the radio-layer failure modes (RSSI, roaming, -channel loss). Those are the reasons the medium must eventually be -simulated, and they are all that Phase 4 is for. Everything downstream of -the handover runs over the forward, so a lab gets a real projection -workload on day one, in CI, on hardware-free nodes, and Phase 4 arrives as -a fidelity upgrade rather than as the first time anything projects. +Running it changed that judgment. With the two Pods of the DD-9 experiment, a +vendor phone image and a publicly available receiver projected a live session +across a single forwarded port (the verification below): version negotiation, +the TLS authentication between the phone stack and the receiver, service +discovery, the phone-side first-run flow, and the rendered launcher with +video, audio and input all ran unchanged. That is not a proxy check; it is +the whole projection stack running on two exporters. + +What option 3 does not exercise is precisely bounded: the **handover** — the +credential exchange over Bluetooth and the phone joining the head unit's +Wi-Fi Direct group — and radio-layer failure modes such as RSSI, roaming and +channel loss. Those are what Phase 4 is for. Everything downstream of the +handover runs over the forward, so a lab gets a real projection workload on +day one, in CI, on hardware-free nodes, and Phase 4 arrives as a fidelity +upgrade rather than as the first time anything projects. Two facts from the same experiment shape how the medium options are built: - **The Cuttlefish Wi-Fi medium is not a byte stream.** The guest's `virtio_mac80211_hwsim` feeds a per-environment `wmediumd` over a - **vhost-user** Unix socket (shared memory plus fd passing), with an - OpenWrt VM as the access point. `--vhost_user_mac80211_hwsim` is - Cuttlefish's documented way to share one medium between instances, but - only on one host. Across hosts, option 1 is therefore a *bridge* — a - component on each exporter that terminates vhost-user locally and - exchanges 802.11 frames with its peer — and the forward is only the pipe - between the two bridges. This is why option 1 is the general answer and - also the one that needs new code and datagram semantics (Future - Possibilities). -- **The guest's Wi-Fi is already IP-reachable.** Each CVD's guest joins its - own OpenWrt AP and reaches the exporter's network through the AP's WAN - link; with one host route and one forwarding rule on the AP, the exporter - reaches the guest's Wi-Fi address in turn. So a Cuttlefish driver can - expose a guest-side port two ways: through `adb forward`, which models - the USB cable, or through the guest's `wlan0` address, which models the - Wi-Fi link. The projection session was run both ways with the same - result. They are two `provides` ports, not two mechanisms, and which one - a test forwards is a topology choice DD-13 leaves to the test. + **vhost-user** Unix socket (shared memory plus fd passing), with an OpenWrt + VM as the access point. `--vhost_user_mac80211_hwsim` shares one medium + between instances on one host only. Across hosts, option 1 is therefore a + *bridge* — a component on each exporter terminating vhost-user locally and + exchanging 802.11 frames with its peer — and the forward is only the pipe + between the two bridges. Hence option 1 is both the general answer and the + one needing new code and datagram semantics (Future Possibilities). +- **The guest's Wi-Fi is already IP-reachable.** Each guest joins its own + OpenWrt AP and reaches the exporter's network through the AP's WAN link; + one host route and one forwarding rule make the guest's Wi-Fi address + reachable in return. So a Cuttlefish driver can expose a guest-side port + two ways — over ADB, which models the USB cable, or at the guest's `wlan0` + address, which models the Wi-Fi link. Both were run with the same result. + They are two `provides` ports, not two mechanisms, and which one a test + forwards is a topology choice DD-13 leaves to the test. Between 1 and 2, option 2 is far cheaper when it applies, because it reuses the same forward machinery as Bluetooth and netsim already owns the frames. @@ -1521,41 +1325,25 @@ delivery modeling live. Option 1 is also the hardest thing in this JEP and the most likely to need upstream work — it is scheduled last and its risk is called out explicitly. -**Verified against real CVDs (2026-09-01).** Same two Pods, same stand-in -relays, same Direct-mode caveat as DD-9. The pieces, all public: - -- *Phone.* Google's GSI with GMS for x86-64 (`gsi_gms_x86_64`, Android 17, - a `user` build) dropped into the Cuttlefish AOSP image set: `super.img` - rebuilt with the GSI `system.img` over the AOSP vendor partitions, and the - GSI `vbmeta.img` (verification disabled) in place of Cuttlefish's. Being a - `user` build it boots with `adbd` stopped; enabling it meant editing the - ext4 `system.img` in place (`persist.sys.usb.config=adb` in `build.prop`, - the exporter's public key in `/adb_keys`). Android Auto 17.4 (x86-64 - split APKs) was then sideloaded and its developer-mode *head unit server* - started on port 5277 — the port the phone `provides`. -- *Head unit.* Google's Desktop Head Unit 2.1 in the AAOS Pod, under Xvfb, - in its `--adb=5277` mode, which is a plain TCP client. This is the only - publicly available Android Auto receiver: the AAOS CVD cannot receive - projection because the receiver ships with GAS, so for a virtual bench - the head unit's projection endpoint is a process inside the head-unit - exporter that `requires` the phone's port (see *A projection bench, - concretely*). Direction falls out of DD-13 as expected. -- *Forward.* DHU → `127.0.0.1:5277` in Pod A → Pod B → the phone, first via - `adb forward tcp:5277` and then via the guest's `wlan0` address through - the OpenWrt AP. - -The session negotiated protocol 1.7, completed TLS 1.2 -(ECDHE-RSA-AES128-GCM-SHA256 — the relay must be byte-transparent), ran -service discovery, walked the phone-side first-run flow (consent, car -authorization, notification access — each driven by `uiautomator`), and -rendered the full launcher with Maps, YouTube Music and the phone app live. -Video is phone→head-unit and was about 0.6 MB for three minutes of a mostly -static screen, so the L4 path is not bandwidth-limited in Direct mode. -Three operational findings are folded into *Reference drivers for -projection* under Design Details: the DHU quits on stdin EOF and needs a -display; Android Auto's head unit server does not survive an aborted -session (`IllegalStateException: Already connected`); and the GSI phone -would not join Wi-Fi at all while it held a validated Ethernet network. +**Verified against real devices (2026-09-01).** Same two Pods and stand-in +relays as DD-9. The phone was a Cuttlefish exporter running a vendor phone +image — a `user` build, so enabling ADB meant preparing the image rather than +configuring the guest — with the projection app sideloaded and its +developer-mode server started on the port the phone `provides`. The head unit +was a desktop receiver process in the other Pod, a plain TCP client and hence +the `requires` side; direction fell out of DD-13 as expected. The forward ran +first over ADB and then over the guest's Wi-Fi address through its AP. + +The session negotiated its protocol version, completed a TLS handshake (so +the path must be byte-transparent), ran service discovery, walked the +phone-side first-run flow, and rendered the full launcher with maps, media +and telephony live. Video is phone→head-unit and cost about 0.6 MB for three +minutes of a mostly static screen, so an L4 forward is not the bandwidth +constraint. Three operational findings are folded into *Reference drivers for +projection*: the receiver quits on stdin EOF and needs a display; the phone's +projection server does not survive an aborted session and must be restarted; +and the phone would not join Wi-Fi while it held a validated Ethernet +network. ### DD-11: Mixed physical/virtual benches — deferred @@ -1648,52 +1436,46 @@ there is operational experience with what benches people actually build. **Decision:** Option 1, with option 3 retained as an explicit form. -**Rationale:** These two kinds of inference look similar and are opposites, -because they sit on different sides of the line DD-6 and DD-10 already drew. - -Direction is a **property of the drivers**, not a choice. Whether rootcanal is -the listening end is a fact about how rootcanal works, already declared in the -exporter's report and already validated by the controller. Making the author -write it down asks them to know exporter internals — exactly what DD-6 -removed addresses to avoid. Worse, it is knowledge that can go stale: a lab -that swaps a `provides` implementation for one that dials would silently -invalidate every lease manifest naming it as `from`. Inferring direction is -therefore not a convenience but a correctness improvement, and it costs -nothing — the controller already checks the reported roles, so option 1 uses -that check to *assign* rather than merely to *reject*. - -Topology is a **property of the test**. DD-10 makes this concrete: the -same phone exporter is a USB-attached phone when a test forwards `aa-hu` -and a wireless one when it forwards `aa-hu-wifi`, and two exporters are a -bench in one test and independent devices in another, so the wiring belongs -to whoever wrote the test. Option 2 makes it a -property of whatever drivers happen to be configured on whichever exporters -the selectors happened to bind, which fails in three ways: +**Rationale:** The two kinds of inference look similar and are opposites. + +Direction is a **property of the drivers**. Whether rootcanal is the listening +end is a fact about rootcanal, already in the exporter's report and already +validated. Making the author restate it asks them to know exporter internals +— what DD-6 removed addresses to avoid — and the knowledge goes stale: swap a +`provides` implementation for one that dials, and every manifest naming it as +`from` is silently wrong. Inferring direction is a correctness improvement +that costs nothing, since the controller already checks the reported roles; +option 1 uses that check to *assign* rather than only to *reject*. + +Topology is a **property of the test**. The same phone exporter is a +USB-attached phone when a test forwards `projection` and a wireless one when +it forwards `projection-wifi`; two exporters are a bench in one test and +independent devices in another. Option 2 makes that a property of whichever +drivers happen to be configured on whichever exporters the selectors bound, +which fails in three ways: - **Nondeterminism across bindings.** A selector-based member binds to different exporters on different runs. If those exporters declare different ports, the bench wires itself differently run to run — the worst property a reproducible test can have. - **Ambiguity is not rare.** Two members each exposing several ports turns - matching into a bipartite matching problem — the same one ATS's - `AdhocTestbedSchedulingUtil` solves for device allocation. Tractable, but a - silently wrong wiring is worse than an error. + matching into a bipartite matching problem. Tractable, but a silently wrong + wiring is worse than an error. - **It weakens a security property.** This JEP states that a forward is not a general exporter-to-exporter tunnel, because an exporter can reach only a peer *a lease it is bound to explicitly names*. Option 2 relaxes "explicitly names" to "happened to match", and an unintended pairing of two `console` ports is a data path nobody requested. -Option 3 remains available as `from`/`to` for the case where wiring should be -pinned regardless of what the exporters report, and the controller then checks -the stated roles against the reported ones rather than assigning them. +Option 3 stays available as `from`/`to` for wiring that should be pinned +regardless of what the exporters report; the controller then checks the stated +roles against the reported ones instead of assigning them. -The ergonomic complaint that motivates option 2 — `member.port:member.port` -is verbose — is better answered in the client. The CLI expands shorthand into -an explicit `spec.forwards[]` entry before submission, so the stored object -stays auditable and `kubectl get lease -o yaml` shows the real wiring. Where a -lab adopts the same port name on both sides, `--forward bt` can expand to it; -that is a naming convention worth offering and not worth depending on. +The ergonomic complaint behind option 2 — `member.port:member.port` is +verbose — is better answered in the client: the CLI expands shorthand into an +explicit `spec.forwards[]` entry before submission, so the stored object stays +auditable. Where a lab uses one port name on both sides, `--forward bt` can +expand to it — a convention worth offering, not worth depending on. ## Design Details @@ -1735,18 +1517,15 @@ Three consequences follow, and they are why this JEP can be as small as it is: bench on someone's desk. A second invariant is narrower, and cost a verification session before it -was noticed, so it is stated rather than assumed: +was noticed: > **Each exporter identity is served by exactly one process.** -Two processes sharing one identity both register and both look healthy; the -controller hands a lease to one of them, and connections that the router -routes to the other are answered with a `KeyError` on a driver UUID that -process has never heard of, closing the stream with no useful error on -either side. The symptom presents as a flaky forward, not as a -misconfiguration. A member exporter therefore runs one replica, and a second -registration under a live identity is worth detecting and refusing rather -than tolerating. +Two processes under one identity both register and both look healthy, but +only one owns the lease's session; connections routed to the other die on a +`KeyError` for a driver UUID it has never heard of, with no useful error on +either side. It presents as a flaky forward, not a misconfiguration. A member +exporter runs one replica, and a duplicate registration is worth refusing. The exception to watch is host networking. The `jumpstarter-driver-cuttlefish` container recipe currently documents `--network=host` (so that netsim and @@ -1791,16 +1570,15 @@ Two properties follow, and they are the reason for DD-1: resolving to the same exporter — which is what a `phone` + `phone` two-handset bench needs. -The single-exporter path is the same code with one member: `spec.selector` is -normalized into a synthetic member at admission, and on success the result is -written to `status.exporterRef` rather than `status.members` (DD-2). There is -one selection implementation, not two. +The single-exporter path is the same code with one member: `spec.selector` +normalizes into a synthetic member at admission and the result is written to +`status.exporterRef` instead of `status.members` (DD-2). One selection +implementation, not two. **What is unchanged, and still imperfect.** Exclusivity is still a read-scan of other leases' claims against a possibly-stale cache, so two leases can -race for one exporter and both write, because they write different objects. -This is pre-existing and neither improved nor worsened here; the loser is -detected on a later reconcile and re-bound. +race for one exporter and both write. Pre-existing, neither improved nor +worsened here; the loser is detected on a later reconcile and re-bound. ### Lease state @@ -1883,34 +1661,30 @@ Establishment then reuses the existing port-forward primitives: `mode: router` never attempts it. Which transport a forward ended up on is in `LeaseForwardStatus.Mode`, and a fallback records why. -**Direct eligibility.** The controller offers `prefer_direct` when both -bound members report the same non-empty `NetworkZone` and the `provides` -side reports a `PeerEndpoint` (DD-4). Zone is deliberately opaque: an -in-cluster exporter takes it from the deployment — one value per cluster -network, which JEP-0016's provisioner sets for every Pod it renders — and an -edge exporter that reports none is never a direct candidate, so it keeps the -router path without any per-lease configuration. This keeps reachability a -statement someone made about the deployment rather than something the -controller infers from addresses it cannot test, which is the same reasoning -DD-13 applies to topology. A pair that claims a shared zone but cannot -actually connect is not a failure mode the user sees: the dial times out and -the router stream, already minted, carries the forward. +**Direct eligibility.** The controller offers `prefer_direct` when both bound +members report the same non-empty `NetworkZone` and the `provides` side +reports a `PeerEndpoint` (DD-4). Zone is deliberately opaque and comes from +deployment configuration — one value per cluster network, which JEP-0016's +provisioner sets on every Pod it renders — so an edge exporter reporting none +is never a direct candidate and keeps the router path with no per-lease +setup. Reachability stays a statement someone made about the deployment +rather than something the controller infers from addresses it cannot test, +the same reasoning DD-13 applies to topology. A pair that claims a zone but +cannot connect is not a user-visible failure: the dial times out and the +router stream, already minted, carries the forward. **Reconnection belongs to the endpoint.** A forward outlives any single -stream: an ingress reload, a router restart, or a node's network blip cuts -the peer stream, and in a cluster whose proxy reloads on every unrelated -`Ingress` change that is a routine event, not an incident (DD-9, third -pass). The `requires` side therefore re-dials with backoff and re-splices -accepted connections without tearing down its listener, and the first -attempt immediately after a cut is expected to fail and be retried. The -alternative — surfacing the reset to the driver — does not work, because the -drivers this JEP is for open their transport once at start: `bt-peer` dials -its HCI port when the peer is created, and after a silent cut the bench -still reports `Ready` while the simulated device is off the air. Drivers -that must know still learn of it: a reconnect is an event on the forward -(*Observability*), which is how the head unit server of DD-10 gets its -mandatory restart. Forward state comes from the splice, never from probing -the far end (DD-9, second pass). +stream: an ingress reload, a router restart or a network blip cuts the peer +stream, and in a cluster whose proxy reloads on unrelated `Ingress` changes +that is routine rather than exceptional (DD-9). The `requires` side therefore +re-dials with backoff and re-splices without tearing down its listener, and +the first attempt right after a cut is expected to fail and be retried. +Surfacing the reset to the driver instead does not work: these drivers open +their transport once at start, so after a silent cut the bench still reports +`Ready` while the device is off the air. Drivers that must know still learn +of it — a reconnect is an event on the forward (*Observability*), which is +how DD-10's projection server gets its mandatory restart. Forward state comes +from the splice, never from probing the far end. **Failure modes and handling:** @@ -1924,7 +1698,7 @@ the far end (DD-9, second pass). | Forward references an undeclared member | Rejected by CEL at admission; lease never created | | `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | | Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | -| Ingress/proxy reload cuts the peer stream | `requires` side re-dials and re-splices transparently; the first attempt after the cut is expected to fail. Observed in DD-9's third pass, and invisible to drivers that dial once | +| Ingress/proxy reload cuts the peer stream | `requires` side re-dials and re-splices transparently; the first attempt after the cut is expected to fail. Observed in verification, and invisible to drivers that dial once | | Direct dial fails or times out | Fall back to the already-minted router stream; recorded as a metric, and conspicuous for a same-zone pair that should have connected | | A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | | Client releases the lease | Forwards torn down first, then the lease ends normally | @@ -1935,42 +1709,35 @@ Nothing in the projection bench needs new forward machinery, but the verification (DD-10) showed that the two reference drivers own real work, of the same kind as DD-9's rootcanal control hook: -- **`cuttlefish` (phone).** A GMS GSI is a `user` build: `adbd` is off and - stays off, so the image the driver boots must carry - `persist.sys.usb.config=adb` and the exporter's ADB public key in - `/adb_keys`. The same image decides which Bluetooth profiles the phone - offers — a GSI ships only the LE-audio `bluetooth.profile.*` properties, - and the classic phone-side set has to be added to `/system/build.prop` - (DD-9, second pass). Both are image-preparation steps the driver - documents and JEP-0016's provisioner can run once per image, not - per-lease actions. Recovery is `cvd restart`, which keeps userdata — - bonds and sideloaded apps included — where `cvd rm` does not. - For `aa-hu-wifi` the driver brings the guest onto the instance's OpenWrt - AP, adds the host route to the AP's LAN and the `wan→wifi` forwarding - rule on the AP — and cuts the guest's Ethernet first, because Android - never asks Wi-Fi to connect while it holds a validated Ethernet default. - The head unit server is started on request and **restarted whenever the - forward resets**: Android Auto does not survive an aborted session, so a - forward reconnect must reach the driver as an event, not be hidden in the +- **`cuttlefish` (phone).** A retail phone image is a `user` build, so the + image the driver boots must be prepared rather than configured: ADB enabled + and the exporter's key installed, and the classic Bluetooth profile + properties present, since a plain GSI ships only the LE-audio set (DD-9). + Both are per-image steps JEP-0016's provisioner can run once, not per-lease + actions; recovery is `cvd restart`, which keeps userdata where `cvd rm` + does not. For `projection-wifi` the driver brings the guest onto the + instance's AP, routes to it, and cuts the guest's Ethernet first, because + Android will not join Wi-Fi while it holds a validated Ethernet default. + The projection server is started on request and **restarted whenever the + forward resets**, because a projection session does not survive an abort — + so a reconnect must reach the driver as an event, not be hidden in the splice. -- **`dhu` (head unit).** Runs the Desktop Head Unit as a process: an X - display (Xvfb), a dummy audio driver, stdin held open — the DHU exits on - stdin EOF because stdin is its interactive console, which is also the - head-unit-side stimulus API (key presses, day/night, microphone) the - driver exposes. Screenshots come from the display. The DHU dials the - instant it starts, so the driver starts it only on a client call, after - the forward is Ready, which is the same ordering rule `bt-peer` already - follows. -- **`bt-peer` (phone).** The third pass of DD-9 showed the driver is one +- **`projection-rx` (head unit).** Runs a receiver as a process: a display, + a dummy audio device, and its console held open, since receivers typically + exit on stdin EOF and use that console as their stimulus API (key presses, + day/night, microphone). Screenshots come from the display. The receiver + dials the instant it starts, so the driver starts it only on a client call, + after the forward is Ready — the same ordering rule `bt-peer` follows. +- **`bt-peer` (phone).** DD-9's verification showed the driver is one config away from being a phone rather than an audio source: with an HFP Audio Gateway alongside its A2DP source it satisfies both of a head unit's phone-facing profiles, and can ring it. That belongs in the driver as a `profiles:` list, with the bond keystore persisted for the life of the lease so a peer restart does not strand the DUT's link key. -The `cuttlefish` and `dhu` drivers are the reference `requires`/`provides` -pair for projection. -What they must not do is know about each other: the phone driver exposes +The `cuttlefish` and `projection-rx` drivers are the reference +`provides`/`requires` pair for projection. What they must not do is know +about each other: the phone driver exposes ports and the head unit driver dials `127.0.0.1:5277`, and the lease is the only place the two are joined. @@ -2023,7 +1790,7 @@ two-object design would face does not arise. - **Simulator control ports are not access-controlled either.** Standalone rootcanal binds its HCI, link and test-channel ports on `0.0.0.0` and accepts any client; the test channel can re-address devices, join - models, and — by accident — crash the controller (DD-9, second pass). A + models, and — by accident — crash the controller (DD-9). A driver exposes only the HCI and link ports as `provides`, never the test channel, and the exporter's network policy is what limits who can reach them; JEP-0016 should ship that policy with the Pod. @@ -2110,38 +1877,32 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in separate Pods (JEP-0016 `ExporterSet`), joined by a forwarded rootcanal port (DD-9). Assert BT discovery and pairing, and — Phase 4 — Wi-Fi - association. `jumpstarter-driver-netsim` supplies the pcap capture used to - evidence what actually crossed the air. - Runnable in CI on KVM-capable nodes with no lab hardware. This is the - headline result and should run on every merge once it exists. -- **Virtual ↔ peer**: one `jumpstarter-driver-cuttlefish` exporter and one - `jumpstarter-driver-bt-peer` exporter, joined by - `bt=headunit.rootcanal:phone.controller`. Assert the CVD discovers and - pairs the peer and the driver reports `avdtp_connected` — the smallest - bench, no second guest to boot. With the peer's HFP AG enabled, assert the - head unit's `HeadsetClientConnectionService` takes a ringing call from it, - which covers the phone-facing profiles without a phone. DD-9's third pass - ran exactly this by hand, over the router. Cheapest CI tier; runs on every - merge. -- **Virtual projection**: a GMS-GSI phone CVD and a `dhu` exporter in - separate Pods, joined by the `projection` forward over `aa-hu-wifi` - (DD-10). Assert the receiver reaches the launcher and a screenshot of the - head-unit display matches. Same CI tier as the Bluetooth test; the two - compose into one bench once Phase 1 and Phase 2 are both green. + association, with `jumpstarter-driver-netsim` supplying a pcap of what + actually crossed the air. Runs in CI on KVM-capable nodes with no lab + hardware: the headline result, on every merge once it exists. +- **Virtual ↔ peer**: a `cuttlefish` exporter and a `bt-peer` exporter joined + by `bt=headunit.rootcanal:phone.controller`. Assert the CVD discovers and + pairs the peer and the driver reports `avdtp_connected`, and — with the + peer's HFP AG enabled — that a ringing call from it reaches the head unit's + telephony stack, covering the phone-facing profiles without a phone. The + smallest bench, no second guest to boot; DD-9's verification ran it by hand + over the router. Cheapest CI tier, every merge. +- **Virtual projection**: a phone CVD and a `projection-rx` exporter in + separate Pods joined over `projection-wifi` (DD-10). Assert the receiver + reaches the launcher and a screenshot matches. Same CI tier; the two + compose into one bench once Phases 1 and 2 are green. - **Physical ↔ physical**: a physical phone and head unit on two exporters - **on different lab hosts** — the allocation ATS's `SimpleScheduler` - refuses. Requires lab hardware; runs on a labeled runner. -- **Latency characterization**: HCI round-trip through a router forward vs. a - direct forward vs. host-local rootcanal, reported as a distribution. Its - result determines whether A2DP-class workloads are in scope for router mode. - The by-hand version (DD-9, third pass) found the simulator, not the - transport, to be the cost on a single node; the test exists to find where - that inverts — cross-node, sustained A2DP, and physical controllers. -- **Forward resilience**: cut the peer stream of a live bench (restart the - router, or reload the ingress in front of it) and assert the forward - re-establishes, the reconnect is counted and emitted, and a driver that - dialed its transport once at start is still talking to the far end - afterwards. Runs in the same CI tier as the Bluetooth bench. + **on different lab hosts** — the allocation no existing framework will + make. Requires lab hardware; runs on a labeled runner. +- **Latency characterization**: HCI round-trip through a router forward, a + direct forward and host-local rootcanal, reported as a distribution, which + decides whether A2DP-class workloads are in scope for router mode. By hand + the simulator dominated on a single node (DD-9); the test looks for where + that inverts — cross-node, sustained A2DP, physical controllers. +- **Forward resilience**: cut a live bench's peer stream (restart the router, + or reload the ingress in front of it) and assert the forward re-establishes, + the reconnect is counted and emitted, and a driver that dialed once at start + is still talking to the far end. ### Manual Verification @@ -2216,15 +1977,15 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): rootcanal to a `bt-peer` Pod through `RouterService.Stream`, pairing and profiles included. CVD ↔ CVD over the router and the multi-node case remain.* -- [ ] **Phase 2 — Virtual projection**: a GMS phone CVD and a `dhu` head - unit in separate Pods complete an Android Auto session to the +- [ ] **Phase 2 — Virtual projection**: a phone CVD and a `projection-rx` + head unit in separate Pods complete a projection session to the launcher over one forwarded port, in CI, with no lab hardware. - *Demonstrated by hand on 2026-09-01 over both `aa-hu` and - `aa-hu-wifi` with a stand-in relay (DD-10); the drivers, the router + *Demonstrated by hand on 2026-09-01 over both `projection` and + `projection-wifi` with a stand-in relay (DD-10); the drivers, the router path and CI remain.* - [ ] **Phase 3 — Physical ↔ physical across hosts**: a phone and head unit on exporters on different lab hosts complete a phone projection - session (Android Auto in the reference implementation) + session - [ ] **Phase 4 — Virtual Wi-Fi medium**: two CVDs in separate Pods associate over a bridged `mac80211_hwsim`/`wmediumd` medium or a shared netsim 802.11 chip, and a projection session completes the @@ -2232,7 +1993,7 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] Measured HCI round-trip latency through a router forward is published, with a documented statement of which workloads it does and does not support. *First data point, 2026-09-02: 0.65 ms of router against a - ~45 ms rootcanal command round-trip, single node (DD-9, third pass); + ~45 ms rootcanal command round-trip, single node (DD-9); cross-node and A2DP-class throughput remain.* ## Graduation Criteria @@ -2268,18 +2029,16 @@ risk in one field, handled by DD-2. existing field changes type, meaning, or default. `Exporter`, `ExporterAccessPolicy`, `ExporterSet`, and `VirtualTargetClass` are otherwise untouched. Every lease that exists today validates unchanged. -- **`status.exporterRef`**: unchanged for single-exporter leases — every - lease created before this feature and every one created after that does not - pass `members`. Multi-member leases leave it nil, which existing consumers - already read as "not bound yet" (DD-2). The JEP-0016 façade, - `jmp get leases`, the `Dial` path, and JEP-0013 telemetry need no changes - to keep working correctly; they need changes only to *support* benches. -- **Driver report**: `ports` is a new optional repeated field. An exporter - built before this JEP reports nothing and is treated as non-forwardable — - the correct answer. No version negotiation, no migration. -- **Drivers**: unchanged. Ports are declared in exporter configuration; the - reference `requires` endpoint (`bt-peer`) needs no Python changes at all. - A driver with no ports is simply not a forward endpoint. +- **`status.exporterRef`**: unchanged for every lease that does not pass + `members`. Multi-member leases leave it nil, which existing consumers + already read as "not bound yet" (DD-2), so the JEP-0016 façade, + `jmp get leases`, `Dial` and JEP-0013 telemetry keep working; they change + only to *support* benches. +- **Driver report and drivers**: `ports` is a new optional repeated field, so + an exporter built before this JEP reports none and is treated as + non-forwardable — the correct answer, with no version negotiation or + migration. Ports are declared in exporter configuration, and the reference + `requires` endpoint (`bt-peer`) needs no Python changes at all. - **Protocol**: three new fields on existing messages and one new RPC. Unknown fields are ignored by proto3, so an N-1 client talks to an N controller unchanged. An N client requesting `members` from an N-1 @@ -2296,34 +2055,27 @@ risk in one field, handled by DD-2. ### Positive -- Multi-device testing becomes a first-class Jumpstarter concept, with atomic - acquisition, shared lifetime, and no deadlock. -- **Acquisition atomicity is structural, not engineered.** All of a lease's - claims are in one object written once, so there is no partial-hold state, - no acquisition timeout, no release-and-retry, and no gang scheduler. -- **A bench can span hosts** — the specific allocation ATS's `SimpleScheduler` - refuses, and the reason multi-device testing is capped at one lab machine - everywhere today. -- **The data plane is existing code.** Forwards are `TemporaryTcpListener` + - `forward_stream` with a router peer stream substituted for a client stream; +- Multi-device testing becomes a first-class concept, with atomic + acquisition, shared lifetime and no deadlock — and the atomicity is + structural, not engineered: all of a lease's claims are one object written + once, so there is no partial-hold state, no acquisition timeout, no + release-and-retry and no gang scheduler. +- **A bench can span hosts** — the allocation existing frameworks refuse, and + the reason multi-device testing is capped at one lab machine today. +- **The data plane is existing code**: `TemporaryTcpListener` + + `forward_stream` with a router peer stream substituted for a client stream. `router.proto` is untouched and no driver changes. - A bench is a lease, so `jmp create lease`, expiry, `spec.release`, access - policy, telemetry, and `kubectl get leases` all apply unchanged. No second - resource, no second RBAC surface, no second lifecycle. -- **Ports are discoverable**, so a client can construct a forward from - `jmp get exporter` output instead of tribal knowledge or someone's YAML. -- Phone projection becomes expressible — Android Auto, CarPlay, and - head units on Android, Linux, or QNX alike — and in the virtual-to-virtual - form expressible *in CI without a lab*. -- Cross-host virtual device pairing, which no shipping tool does, reduces to - forwarding a socket. -- Homogeneous benches are the tractable half of the problem and already - deliver both headline results; mixing physical and virtual devices in one - bench needs no new software mechanism, only radio hardware (DD-11). -- Nothing here is Android-specific: CAN cross-connects between two ECUs, - serial cross-overs, and SOME/IP peer benches are all ports and forwards. -- The exporter = DUT invariant survives; JEP-0016 DD-8's option 1 becomes - available. + policy, telemetry and `kubectl get leases` apply unchanged — no second + resource, RBAC surface or lifecycle. Ports are discoverable, so a forward + can be built from `jmp get exporter` output rather than tribal knowledge. +- Phone projection and Bluetooth pairing become expressible for any protocol + and any head unit OS, and in virtual-to-virtual form expressible *in CI + without a lab* — cross-host virtual device pairing, which no shipping tool + does, reduces to forwarding a socket. +- Nothing here is Android-specific: CAN cross-connects, serial cross-overs + and SOME/IP peer benches are all ports and forwards. The exporter = DUT + invariant survives, and JEP-0016 DD-8's option 1 becomes available. ### Negative @@ -2358,40 +2110,33 @@ risk in one field, handled by DD-2. Mitigation: Phase 4 is last, may conclude "direct mode only", and the datagram work in Future Possibilities is the escalation path; Phase 2 already delivers a real projection workload without it. -- **The projection bench depends on Google artifacts this project cannot - ship.** The GMS GSI and the Desktop Head Unit are published by Google - under their own terms; the Android Auto APK is not published outside - Google Play, and the verification sideloaded a mirror copy. A lab must - supply these itself, and CI for Phase 2 needs an artifact path that does - not redistribute them. Mitigation: the drivers take image and APK - locations as configuration, the reference documents where each artifact - comes from, and the Bluetooth phase carries the CI headline on AOSP-only - images. +- **The projection bench depends on vendor artifacts this project cannot + ship** — the phone image, the projection app and the receiver, one of which + is not distributed outside an app store. A lab must supply them, and Phase + 2 CI needs an artifact path that does not redistribute them. Mitigation: + the drivers take artifact locations as configuration, and the Bluetooth + phase carries the CI headline on freely available images. - **Bluetooth timing may be tighter than measured** — pairing may work while A2DP streaming does not. Mitigation: latency characterization is an acceptance criterion whose answer is published, not assumed. -- **The simulators are less robust than a lab needs.** rootcanal aborts - when a test-channel client disconnects early, its restart does not - reattach the guest, and a crash on one exporter strands every bench - member's controller (DD-9, second pass). Mitigation: the rootcanal - driver keeps the test channel private, never probes, watches its - controller and reports `Degraded`, and owns `cvd restart` as the recovery - action; the abort itself is an upstream fix worth contributing. +- **The simulators are less robust than a lab needs.** rootcanal aborts when + a test-channel client disconnects early, its restart does not reattach the + guest, and a crash on one exporter strands every member's controller + (DD-9). Mitigation: the driver keeps the test channel private, never + probes, watches its controller and reports `Degraded`, and owns the restart + as a recovery action; the abort is an upstream fix worth contributing. - **The cluster data path is less stable than a bench assumes.** Long-lived streams do not survive ordinary cluster events: an nginx ingress reloads on any `Ingress` change and drains its workers on a timer, cutting every gRPC stream through it — measured on the verification cluster as a reload at 04:00:13 and the exporter's controller stream cut at 04:04:13, with - the forwarded HCI splice going down 40 s later (DD-9, third pass). A - multi-hour bench will meet this repeatedly. Mitigation: reconnection is - the forward endpoint's responsibility, reconnects are observable, and the - HIL suite includes a bench that survives a deliberate stream cut. An - in-cluster bench also avoids the machinery altogether, which is part of - why `Auto` prefers a direct connection there (DD-4). A - related hazard is version skew across a splice — a newer client-side - network driver against an older exporter closed every forwarded - connection at the first byte with no error — which is an argument for the - endpoint being the exporter's own code, as specified. + the forwarded HCI splice going down 40 s later (DD-9). A + multi-hour bench will meet this repeatedly. Mitigation: reconnection is the + forward endpoint's responsibility, reconnects are observable, the HIL suite + includes a bench that survives a deliberate stream cut, and an in-cluster + bench avoids the machinery altogether (DD-4). A related hazard is silent + version skew across a splice — an argument for the endpoint being the + exporter's own code, as specified. - **The namespace invariant may be violated in the field.** Fixed `listen` addresses and lease-scoped control-plane blast radius both assume one exporter per network namespace. A host-networked deployment silently @@ -2461,36 +2206,22 @@ risk in one field, handled by DD-2. JEP-0016's exporter = DUT invariant. A group as a single composite DUT (JEP-0016 DD-8 option 2) remains legitimate and orthogonal. - **Building a Jumpstarter multi-device *test runner*.** This JEP stops at the - bench. Tradefed, Mobly, and pytest already run multi-device tests well; - Jumpstarter's contribution is the bench they run against. -- **Adopting ATS/OmniLab as the fleet layer.** Mobile Harness's `Device` - abstraction is ADB-shaped and runs in the lab-server JVM beside the device, - its OSS platform coverage is Android-only, and extending it means in-tree - Java and Bazel. Adopting it would import the one-`LabLocator` constraint - this JEP removes. The complementary direction — Jumpstarter *under* ATS via - a Mobile Harness `Device` or Mobly-controller shim, so existing results - pipelines keep working — is a Future Possibility, not a rejection. + bench. Existing runners already run multi-device tests well; Jumpstarter's + contribution is the bench they run against. +- **Adopting an existing mobile test framework as the fleet layer.** Their + device abstractions are Android-shaped and run beside the device on the lab + host, and adopting one would import the single-host allocation constraint + this JEP removes. The complementary direction — Jumpstarter *under* such a + framework via a device shim, so existing results pipelines keep working — is + a Future Possibility, not a rejection. ## Prior Art -- **OmniLab Android Test Station / Mobile Harness** (`google/device-infra`) - is the closest system and the one to be precise about. It **does** - gang-schedule multi-device jobs: `AdhocTestbedSchedulingUtil` performs - maximum-cardinality bipartite matching of `SubDeviceSpec`s onto idle - devices, with `TestbedDevice`, `CompositeDevice`, and - `SubDeviceSynchronizationDriver` all present, and it has a multi-host mode - (one controller, N workers) for fleet pooling. The structural limit is in - `SimpleScheduler.allocate`: every device locator in an allocation must - share one `LabLocator`, so a multi-device job cannot span lab hosts. That - single check is the gap this JEP targets. *Terminology note for reviewers:* - a Mobile Harness `Driver` is a **test runner** (TradefedTest, MoblyTest), - not a device abstraction — its `Device`/`BaseDevice` is the analog of a - Jumpstarter driver. -- **Tradefed multi-device** contributes the role/requirement declaration - pattern (``) and `multi_target_preparer` for coordinated - setup such as Bluetooth pairing, and documents its own gap verbatim: *"No - APIs yet exist to conduct operations from one device to another, such as - `device1.sync(device2)`."* +- **Android/mobile multi-device test frameworks** contribute the role + declaration pattern (a job naming `phone`, `headunit`) and coordinated + setup steps such as pairing two devices before a test. They gang-schedule + device sets, but allocation is confined to one lab host and none of them + bridges a medium between roles — the gap this JEP targets. - **LAVA MultiNode** is the closest HiL prior art: one job spans multiple devices with named roles, and — notably — makes the multi-device unit *the job itself* rather than a wrapper around N jobs, which independently @@ -2522,27 +2253,21 @@ risk in one field, handled by DD-2. To resolve during review: -- **Bumble as a Cuttlefish controller.** DD-9's options 1 and 2 are now - verified against real CVDs; option 3 is not. Pointing a CVD at a Bumble - `mode=controller` endpoint is documented for the Android emulator but not - for Cuttlefish, and it is the only option that reaches beyond Cuttlefish. - One more prototype run settles it. -- **Who issues the link-layer join, and who owns addresses?** The - federation experiment showed that option 2 needs a control step - (`add_remote` on the test channel) after the forward is up, and that - standalone rootcanals collide on `da:4c:10:de:00:00` unless one is - re-addressed — and the second pass showed any attaching host collides - too, since numbering is per model and reused. The candidates are a - post-establish hook on the `requires` - side driver, a lease-level `forwards[].onEstablish` action, or leaving it - to the test. The first keeps the controller ignorant of Bluetooth, which - DD-8 and DD-13 argue for. -- **Forward-before-launch ordering.** Under option 1 the guest dials its HCI - port at boot, so the forward must exist before the second CVD is created; - under option 2 the join must happen after. The JEP-0016 provisioner creates - the CVD at lease time, so the forward has to be established in the window - between binding and `cvd create`. Whether that is a lease-status condition - the provisioner waits on, or a retrying connector, is open. +- **Bumble as a Cuttlefish controller.** DD-9's options 1 and 2 are verified; + option 3 is not. `mode=controller` is documented for the Android emulator + but not for Cuttlefish, and it is the only option that reaches beyond + Cuttlefish. One prototype run settles it. +- **Who issues the link-layer join, and who owns addresses?** Option 2 needs + a control step after the forward is up, and rootcanal's address numbering + collides for any attaching host (DD-9). Candidates: a post-establish hook + on the `requires`-side driver, a lease-level `forwards[].onEstablish` + action, or leaving it to the test. The first keeps the controller ignorant + of Bluetooth, which DD-8 and DD-13 argue for. +- **Forward-before-launch ordering.** Option 1's guest dials at boot, so the + forward must exist before the second device is created; option 2's join + must happen after. JEP-0016's provisioner creates the device at lease time, + so the forward has to come up between binding and creation — as a + lease-status condition the provisioner waits on, or a retrying connector. - **Should the scalar and members forms really be mutually exclusive?** The alternative is `spec.selector` as a default for members that omit their own — convenient when several members share a selector, but two ways to @@ -2572,16 +2297,14 @@ To resolve during implementation: the least clever and the only one that works for an edge exporter that is routable from the cluster but not the reverse. - Whether forward reconnect should preserve the medium's logical state or - force a fresh pairing; likely protocol-specific. Two data points so far: - Android Auto's head unit server must be restarted after a dropped session, - so the reconnect has to be visible to the endpoint drivers; and a Bumble - peer whose transport is cut loses its in-memory bonds, so the DUT's stored - link key is stale and pairing has to be redone unless the driver persists - its keystore for the life of the lease. The Bluetooth answer may simply be - "persist the keystore and re-attach", which would make reconnect - invisible for that medium. -- How Phase 2's CI obtains the GMS GSI, the Desktop Head Unit and the - Android Auto APK without redistributing them (see Risks). + force a fresh pairing; likely protocol-specific. Two data points: a + projection server must be restarted after a dropped session, so the + reconnect must be visible to endpoint drivers; and a Bumble peer loses its + in-memory bonds when its transport is cut, stranding the DUT's link key + unless the driver persists a keystore. For Bluetooth the answer may simply + be "persist and re-attach", making reconnect invisible. +- How Phase 2's CI obtains the phone image, projection app and receiver + without redistributing them (see Risks). - How `jmp get leases` renders N exporters in a table column readably. ## Future Possibilities @@ -2589,111 +2312,81 @@ To resolve during implementation: Not part of this proposal: - **Mixed physical/virtual benches** (DD-11) — a real device paired to a - virtual one through a gateway exporter owning a real radio adapter. This - needs no new software mechanism: the adapter is an ordinary `provides` - port and the lease plane is unchanged. What it needs is lab hardware near - the physical device, and a decision about how to model a shared RF - resource — as a member with its own role, which makes it schedulable and - auditable but turns a two-device bench into three members, or as an - attribute of the physical member's exporter. -- **Selection-time port validation** via JEP-0015 dynamic exporter labels, so - a bench whose ports cannot be satisfied reports `Unsatisfiable` without - holding anything (DD-12). -- **A global lease scheduler** with a view of all leases and exporters, which - the controller already carries a `TODO` for; it would close the binding race - for single- and multi-member leases alike. + virtual one through a gateway exporter owning a real radio adapter. No new + software mechanism: the adapter is an ordinary `provides` port. What it + needs is lab hardware and a decision on how to model a shared RF resource — + as its own member, schedulable and auditable but making a two-device bench + three, or as an attribute of the physical member's exporter. +- **Selection-time port validation** via JEP-0015 dynamic labels, so a bench + whose ports cannot be satisfied reports `Unsatisfiable` without holding + anything (DD-12). +- **A global lease scheduler**, which the controller already carries a `TODO` + for; it closes the binding race for single- and multi-member leases alike. - **Fan-out forwards** — one `provides` port serving several `requires` - ports, for shared media with more than two participants. Today's forward is - strictly point-to-point. Bumble's link relay already implements the - underlying idea: a WebSocket relay hosting virtual *rooms*, each room a set - of virtual controllers that can advertise and exchange ACL data with one - another. A room is the N-way medium DD-6 deferred, so this is an - integration rather than an invention. -- **Ephemeral `listen` allocation** for `requires` ports. Only needed if the - one-exporter-per-namespace assumption is relaxed; it trades the fixed - address for driver coupling, so it is not worth doing while the assumption - holds. -- **Bench-level access policy and quota** (DD-12). -- **Per-member early release**, if the all-or-nothing lifetime proves coarse. -- **Datagram forwards.** `wmediumd` and other frame-oriented media want - datagram semantics with real boundaries and no head-of-line blocking; the - additive `FRAME_TYPE_DATAGRAM` extension to `RouterService.Stream` (and, - further out, QUIC unreliable datagrams) is a separate protocol JEP, for - which Phase 4 is the most compelling justification. The frame bridge - DD-10 describes is the consumer: it terminates vhost-user on each - exporter and needs a datagram pipe between the two halves. -- **Vehicle-bus counterparty integration.** The restbus pattern described in - DD-9 has mature tooling behind it, and a broker that exposes CAN, LIN, - FlexRay, and Automotive Ethernet over a gRPC socket is already a `provides` - port needing nothing new from this JEP. RemotiveLabs is the obvious - candidate — RemotiveBroker plus RemotiveTopology's DBC/ARXML-driven restbus - — and it composes with the automotive drivers Jumpstarter already ships - (`can`, `doip`, `someip`, `uds`, `xcp`, `obd`) as the counterparty they - talk to rather than a replacement for any of them. Their AAOS emulator - integration, which feeds VHAL from broker signals, suggests a three-member - bench worth building eventually: a virtual head unit driven by real vehicle - signals, a restbus supplying the rest of the vehicle, and a phone for - projection, with every pairwise connection an ordinary forward. **This is - deliberately out of scope here and should get its own JEP** — the broker is - a commercial product, so a driver is an integration against something the - user licenses rather than a dependency this project can ship, and that - distinction deserves its own design discussion rather than a line in this - one's acceptance criteria. -- **Jumpstarter under ATS** — a Mobile Harness `Device` or Mobly-controller - shim backed by a Jumpstarter lease, so Google's results pipeline keeps - working while gaining non-Android and cross-host devices. Complements - JEP-0016's Cloud Orchestrator seam. + ports, for media with more than two participants. Bumble's link relay + already implements the idea as virtual *rooms*, so this is an integration + rather than an invention. +- **Ephemeral `listen` allocation**, needed only if the + one-exporter-per-namespace assumption is relaxed. +- **Bench-level access policy and quota** (DD-12), and **per-member early + release** if the all-or-nothing lifetime proves coarse. +- **Datagram forwards.** Frame-oriented media want datagram semantics with + real boundaries and no head-of-line blocking; an additive datagram frame + type on `RouterService.Stream` (and, further out, QUIC datagrams) is a + separate protocol JEP that Phase 4's frame bridge would consume. +- **Vehicle-bus counterparty integration.** A broker exposing CAN, LIN, + FlexRay and Automotive Ethernet over a socket is already a `provides` port + needing nothing new here, and composes with the automotive drivers + Jumpstarter ships (`can`, `doip`, `someip`, `uds`, `xcp`, `obd`) as the + counterparty they talk to. It suggests a three-member bench — a virtual + head unit driven by real vehicle signals, a restbus supplying the rest of + the vehicle, and a phone — but the mature options are commercial products, + so integrating one is an out-of-scope decision deserving its own JEP. +- **Jumpstarter under an existing test framework** — a device or + Mobly-controller shim backed by a lease, so an established results pipeline + keeps working while gaining non-Android and cross-host devices. - **A `Bench` template CRD** — a reusable named topology instantiated by reference, the way `VirtualTargetClass` is to `ExporterSet`. A *template*, not a second lease-like object, so it does not reopen DD-1. -- **Spawned-on-lease members** — a member satisfied by provisioning a new - JEP-0014 pool instance on demand, making a bench elastic in its virtual half. +- **Spawned-on-lease members** — a member satisfied by provisioning a + JEP-0014 pool instance on demand, making a bench elastic in its virtual + half. - **Agent-facing bench skills** — an agent leasing a two-device bench to reproduce an interaction bug, following JEP-0016's agent-native framing. ## Implementation History -- 2026-09-01: JEP drafted +- 2026-09-01: JEP drafted. - 2026-09-01: DD-9 options 1 and 2 verified by hand with two CVD Pods on a - kind cluster (pairing, HFP/A2DP/AVRCP, A2DP streaming); findings recorded - in DD-9 and Unresolved Questions -- 2026-09-01: DD-10 option 3 verified by hand: Android Auto 17.4 on a GMS - GSI phone CVD projected to the Desktop Head Unit in the other Pod over a - single forwarded port, then again with the phone end on its Wi-Fi NIC via - the per-instance OpenWrt AP -- 2026-09-01: DD-10 decision revised on that evidence — L4 projection - promoted from diagnostic to the Phase 2 deliverable, the Wi-Fi medium - reframed as a frame bridge; projection bench, `aa-hu`/`aa-hu-wifi` ports, - `dhu` driver and phase renumbering (1–4) added -- 2026-09-01: DD-9 second pass — regular pairing and the full classic - profile stack (plus A2DP streaming) between the GMS GSI phone and the - AAOS CVD over federated rootcanals, after adding the phone-side - `bluetooth.profile.*` set to the GSI; the merged - `jumpstarter-driver-bt-peer` run unchanged as a `requires`-side host - against a CVD's rootcanal through a port-forward; rootcanal test-channel - crash, address reuse and the identical guest address plan recorded as - driver duties, a Security bullet and a Risk + kind cluster — pairing, HFP/A2DP/AVRCP, A2DP streaming. +- 2026-09-01: DD-10 option 3 verified by hand — a phone CVD projected a live + session to a receiver in another Pod over one forwarded port, then again + with the phone end on its Wi-Fi NIC. The decision was revised on that + evidence: L4 projection promoted from diagnostic to the Phase 2 + deliverable, the Wi-Fi medium reframed as a frame bridge, and the + projection bench, its ports, the `projection-rx` driver and the phase + renumbering added. +- 2026-09-01: DD-9 second pass — full classic profile stack between a + retail-image phone CVD and the head unit over federated rootcanals, and + `jumpstarter-driver-bt-peer` run unchanged as a `requires` endpoint. + rootcanal's test-channel crash, address reuse and the identical guest + address plan became driver duties, a Security bullet and a Risk. - 2026-09-02: DD-9 third pass — the same bench over Jumpstarter's own data - path: a rootcanal `TcpNetwork` exporter, a bt-peer Pod, and a lease-holding - forward endpoint splicing them through `RouterService.Stream`, with a - Bumble HFP-AG + A2DP peer standing in for the phone (pairing, Phone and - Media, a ringing call into AAOS Telecom). Router overhead measured at - 0.65 ms against ~45 ms of rootcanal; proxy-reload stream loss, duplicate - registration and version skew recorded as forward-endpoint duties -- 2026-09-02: those findings made normative — DD-4's fast path given a - measured price, a single-process invariant per exporter identity, - reconnection assigned to the forward endpoint with reconnects as events, a - cluster-data-path Risk, a *Forward resilience* HIL test, and two acceptance - criteria -- 2026-09-02: Motivation gained *Why the exporter, not the device, is the - unit of a lease* — harnessed multi-device benches are already one exporter - and stay that way; the gap is physically separate benches on separate - hosts, and the gain is gang-scheduling any combination of them + path, with a Bumble HFP-AG + A2DP peer standing in for the phone. Router + overhead measured at 0.65 ms against ~45 ms of rootcanal; proxy-reload + stream loss, duplicate registration and version skew became + forward-endpoint duties, then normative text: a measured price on DD-4's + fast path, one process per exporter identity, endpoint-owned reconnection + with reconnects as events, a cluster-data-path Risk, a *Forward resilience* + test and two acceptance criteria. - 2026-09-02: `Auto` changed to **prefer a direct peer connection between - same-zone members** rather than defaulting to the router — DD-4 rewritten, - `PeerEndpoint`/`NetworkZone` added to `ExporterStatus`, `prefer_direct` - added to `DialPeerResponse`, direct-first-with-fallback replacing the - raced dial, and a *Direct eligibility* rule added to Design Details + same-zone members** rather than defaulting to the router — + `PeerEndpoint`/`NetworkZone`, `prefer_direct`, direct-first-with-fallback + and a *Direct eligibility* rule. +- 2026-09-02: Motivation gained *Why the exporter, not the device, is the + unit of a lease*; vendor-specific naming replaced with generic phone + projection and Bluetooth pairing throughout, and the verification + narratives consolidated. ## References @@ -2707,11 +2400,6 @@ Not part of this proposal: - [JEP-0013: Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) - [JEP-0011: Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) — the introspection direction port reporting extends -- [google/device-infra](https://github.com/google/device-infra) — Mobile - Harness, the ATS 2.0 engine (`SimpleScheduler`, `AdhocTestbedSchedulingUtil`) -- [OmniLab Android Test Station user guide](https://source.android.com/docs/core/tests/development/android-test-station/ats-user-guide) -- [Virtual devices in OmniLab ATS](https://source.android.com/docs/core/tests/development/android-test-station/ats-virtual-devices) -- [Tradefed: run tests with multiple devices](https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/multi-device) - [Cuttlefish: test connectivity of multiple devices](https://source.android.com/docs/devices/cuttlefish/connectivity) - [Mobly](https://github.com/google/mobly) — [testbed tutorial](https://github.com/google/mobly/blob/master/docs/tutorial.md) - [Bumble, a Python Bluetooth stack](https://google.github.io/bumble/) — From b8eb3f6517a9c2584f3d0bba616886ca8a01ebff Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Wed, 2 Sep 2026 12:56:24 -0400 Subject: [PATCH 20/26] docs(jep-0017): soften the novelty claims about the virtual case The virtual-case section asserted that nobody had shipped cross-host virtual device pairing, and the Consequences repeated it. Absolute claims about what does not exist are the easiest thing in a JEP to be wrong about, and they are not load-bearing: the design stands on the fact that these simulators are reached over ordinary sockets, whoever else has noticed that. The section is now "The virtual case: host-local by construction". It reports what the surveyed tooling assumes rather than what the world has never done, invites correction in review, and notes that a prior solution to the cross-host case would change the build-or-adopt calculation. "Nothing about same-host is fundamental" becomes an observation about these interfaces rather than a verdict. The physical-bench and Consequences claims are scoped to the frameworks actually surveyed. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...7-multi-exporter-leases-port-forwarding.md | 30 +++++++++++-------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md index e51de0912..0c9e787a5 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md @@ -117,7 +117,7 @@ layer should provide: The last point is the hard one, and it is what makes this more than an ergonomics change. -### The virtual case: what nobody has shipped yet +### The virtual case: host-local by construction For virtual devices the connectivity problem is sharper. Cuttlefish and the Android emulator can already pair virtual devices to each other — but only @@ -129,19 +129,22 @@ to "all running instances on the same host machine", and container wrappers publish each container's ports while keeping radio simulation inside the container group. -So the state of the art for virtual multi-device testing is: *both devices -must live on one machine*. That is exactly the constraint a cluster-scheduled -pool of one-device-per-Pod exporters (JEP-0016) breaks, and why its DD-8 -deferred multi-device groups pending "a cross-Pod virtual-radio story… real -upstream-facing work." +The common thread in the tooling we surveyed is that virtual multi-device +testing assumes *both devices live on one machine*. That is exactly the +assumption a cluster-scheduled pool of one-device-per-Pod exporters +(JEP-0016) breaks, and why its DD-8 deferred multi-device groups pending "a +cross-Pod virtual-radio story… real upstream-facing work." If someone has +solved the cross-host case in a way this survey missed, that is worth raising +in review — it would change the build-or-adopt calculation. The encouraging part is that these simulators are reached over ordinary sockets: rootcanal accepts HCI on TCP — which is why `jumpstarter-driver-bt-peer` can already attach a `bumble` peer with `transport: "tcp-client:127.0.0.1:7300"` — netsim accepts virtual chips over a bidirectional gRPC stream, and `wmediumd` speaks over a frame socket. -Nothing about "same host" is fundamental; it is an artifact of nobody having -forwarded those sockets between machines under a common lease. +Nothing about "same host" looks fundamental to these interfaces; it appears +to be an artifact of where the sockets are reachable from, rather than a +property of the simulators themselves. ### Prior art's ceiling @@ -1892,7 +1895,7 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): reaches the launcher and a screenshot matches. Same CI tier; the two compose into one bench once Phases 1 and 2 are green. - **Physical ↔ physical**: a physical phone and head unit on two exporters - **on different lab hosts** — the allocation no existing framework will + **on different lab hosts** — the allocation the surveyed frameworks do not make. Requires lab hardware; runs on a labeled runner. - **Latency characterization**: HCI round-trip through a router forward, a direct forward and host-local rootcanal, reported as a distribution, which @@ -2060,8 +2063,9 @@ risk in one field, handled by DD-2. structural, not engineered: all of a lease's claims are one object written once, so there is no partial-hold state, no acquisition timeout, no release-and-retry and no gang scheduler. -- **A bench can span hosts** — the allocation existing frameworks refuse, and - the reason multi-device testing is capped at one lab machine today. +- **A bench can span hosts** — the allocation the frameworks surveyed here + decline to make, and the usual reason multi-device testing stays on one lab + machine. - **The data plane is existing code**: `TemporaryTcpListener` + `forward_stream` with a router peer stream substituted for a client stream. `router.proto` is untouched and no driver changes. @@ -2071,8 +2075,8 @@ risk in one field, handled by DD-2. can be built from `jmp get exporter` output rather than tribal knowledge. - Phone projection and Bluetooth pairing become expressible for any protocol and any head unit OS, and in virtual-to-virtual form expressible *in CI - without a lab* — cross-host virtual device pairing, which no shipping tool - does, reduces to forwarding a socket. + without a lab* — cross-host virtual device pairing reduces to forwarding a + socket. - Nothing here is Android-specific: CAN cross-connects, serial cross-overs and SOME/IP peer benches are all ports and forwards. The exporter = DUT invariant survives, and JEP-0016 DD-8's option 1 becomes available. From ab1de467832071932dc88bd4ba6c2686bf57df5d Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Thu, 3 Sep 2026 15:48:16 -0400 Subject: [PATCH 21/26] docs(jep-0015): renumber from 0017 to the next available number Numbering follows what is claimed by merged JEPs and open PRs, not by branches. On main: 0000, 0010, 0011, 0013, 0014. Open PRs claim 0012 (lease lifecycle FSM) and, mislabelled, 0014 (admin API). The jep-0015-exporterclass branch has no PR and so claims nothing. The next available number is therefore 0015. The two drafts this JEP leans on are not upstream yet, so their numbers follow from this one: the Cuttlefish orchestration JEP keeps 0016 as the next number after this, and dynamic exporter labels becomes 0017. Both are marked in References as drafts not yet submitted, so a reviewer is not left looking for documents that do not exist. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...-multi-exporter-leases-port-forwarding.md} | 26 +++++++++---------- docs/source/contributing/jeps/index.md | 4 +-- 2 files changed, 15 insertions(+), 15 deletions(-) rename docs/source/contributing/jeps/{JEP-0017-multi-exporter-leases-port-forwarding.md => JEP-0015-multi-exporter-leases-port-forwarding.md} (99%) diff --git a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md similarity index 99% rename from docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md rename to docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index 0c9e787a5..f077ccfd8 100644 --- a/docs/source/contributing/jeps/JEP-0017-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -1,14 +1,14 @@ -# JEP-0017: Multi-Exporter Leases and Inter-Exporter Port Forwarding +# JEP-0015: Multi-Exporter Leases and Inter-Exporter Port Forwarding | Field | Value | | ----------------- | -------------------------------------------------------- | -| **JEP** | 0017 | +| **JEP** | 0015 | | **Title** | Multi-Exporter Leases and Inter-Exporter Port Forwarding | | **Author(s)** | @kirkbrauer (Kirk Brauer) | | **Status** | Draft | | **Type** | Standards Track | | **Created** | 2026-09-01 | -| **Updated** | 2026-09-01 | +| **Updated** | 2026-09-03 | | **Discussion** | *TBD (PR link)* | | **Requires** | JEP-0014 | | **Supersedes** | | @@ -1395,7 +1395,7 @@ physical device, and it cannot find bugs in the physical device's stack. 3. **A bench-shaped policy CRD** with rules over lease shape, size, and count. **Decision:** Option 1 for v1; option 2 recorded as a follow-on that depends -on JEP-0015; option 3 deferred. +on JEP-0017; option 3 deferred. **Rationale:** Per-member evaluation is the least surprising and most secure default: a multi-member lease can never reach an exporter its client could @@ -1419,7 +1419,7 @@ Option 2 is strictly better and worth doing: it would make a bench whose ports cannot be satisfied report `Unsatisfiable` **without holding anything**, consistent with DD-1's all-or-nothing property. It requires reported device information to be reconciled into selectable exporter labels, -which is precisely the mechanism proposed in JEP-0015. Hand-maintained CR +which is precisely the mechanism proposed in JEP-0017. Hand-maintained CR labels are a stopgap but drift from reality, so this JEP does not depend on them. @@ -2010,7 +2010,7 @@ often does a same-zone pair fall back; does anyone hit `MaxItems=8`; how often do `listen` collisions occur on physical hosts; does the nil-`status.exporterRef` convention (DD-2) surprise any consumer; is bind-time port validation (DD-12) painful enough to justify accelerating the -JEP-0015 dependency. +JEP-0017 dependency. ### Stable @@ -2019,7 +2019,7 @@ JEP-0015 dependency. (evidence the port model generalizes) - No API changes to `members` / `forwards` / `PortReport` for one release cycle -- Selection-time port validation either shipped on JEP-0015 or explicitly +- Selection-time port validation either shipped on JEP-0017 or explicitly deferred with a rationale ## Backward Compatibility @@ -2094,7 +2094,7 @@ risk in one field, handled by DD-2. - **Protocol mismatch can fail late** when neither port declares `protocol` (DD-8) — a deliberate retreat from admission-time validation. - **Port validation happens after binding** (DD-12), so an unsatisfiable - forward holds devices while reporting `Invalid`, until JEP-0015 makes ports + forward holds devices while reporting `Invalid`, until JEP-0017 makes ports selectable. - **No per-member early release.** A client finished with one device cannot hand it back without ending the lease. @@ -2321,7 +2321,7 @@ Not part of this proposal: needs is lab hardware and a decision on how to model a shared RF resource — as its own member, schedulable and auditable but making a two-device bench three, or as an attribute of the physical member's exporter. -- **Selection-time port validation** via JEP-0015 dynamic labels, so a bench +- **Selection-time port validation** via JEP-0017 dynamic labels, so a bench whose ports cannot be satisfied reports `Unsatisfiable` without holding anything (DD-12). - **A global lease scheduler**, which the controller already carries a `TODO` @@ -2397,10 +2397,10 @@ Not part of this proposal: - [JEP-0014: Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) — "Composite leases — multiple exporters linked into one logical lease" (Future Possibilities), which this JEP realizes -- JEP-0015: Dynamic Exporter Labels — the mechanism selection-time port - validation depends on (DD-12) -- JEP-0016: Cuttlefish Kubernetes-Native Orchestration — DD-8, whose option 1 - this JEP implements +- JEP-0016: Cuttlefish Kubernetes-Native Orchestration (draft, not yet + submitted) — DD-8, whose option 1 this JEP implements +- JEP-0017: Dynamic Exporter Labels (draft, not yet submitted) — the + mechanism selection-time port validation depends on (DD-12) - [JEP-0013: Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) - [JEP-0011: Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) — the introspection direction port reporting extends diff --git a/docs/source/contributing/jeps/index.md b/docs/source/contributing/jeps/index.md index 567afe4e8..70247a7b7 100644 --- a/docs/source/contributing/jeps/index.md +++ b/docs/source/contributing/jeps/index.md @@ -38,7 +38,7 @@ For the full process definition, see [JEP-0000](JEP-0000-jep-process.md). | 0011 | [Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) | Accepted | @kirkbrauer (Kirk Brauer) | | 0013 | [Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) | Accepted | @mangelajo (Miguel Angel Ajo Pelayo) | | 0014 | [Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) | Approved | @mangelajo (Miguel Angel Ajo Pelayo) | -| 0017 | [Multi-Exporter Leases and Inter-Exporter Port Forwarding](JEP-0017-multi-exporter-leases-port-forwarding.md) | Draft | @kirkbrauer (Kirk Brauer) | +| 0015 | [Multi-Exporter Leases and Inter-Exporter Port Forwarding](JEP-0015-multi-exporter-leases-port-forwarding.md) | Draft | @kirkbrauer (Kirk Brauer) | ### Informational JEPs @@ -73,5 +73,5 @@ JEP-0010-renode-integration.md JEP-0011-protobuf-introspection-interface-generation.md JEP-0013-observability-telemetry-logs.md JEP-0014-virtual-scalable-exporters.md -JEP-0017-multi-exporter-leases-port-forwarding.md +JEP-0015-multi-exporter-leases-port-forwarding.md ``` From 26a2df078953b0701bc6221414bc2b05c984b86f Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Thu, 3 Sep 2026 15:49:11 -0400 Subject: [PATCH 22/26] docs(jep-0015): the netsim driver has merged #980 landed on main as the netsim driver, so the reference to it as an open PR is stale. The control-plane companion this JEP composes with now exists in-tree. Assisted-by: Claude Signed-off-by: Kirk Brauer --- .../jeps/JEP-0015-multi-exporter-leases-port-forwarding.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index f077ccfd8..7f9a1a90f 100644 --- a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -2414,8 +2414,8 @@ Not part of this proposal: — `jumpstarter-driver-bt-peer` (merged); the reference `requires`-side endpoint - [jumpstarter-dev/jumpstarter#980](https://github.com/jumpstarter-dev/jumpstarter/pull/980) - — `jumpstarter-driver-netsim` (open); the control-plane companion, incl. - pcap capture + — `jumpstarter-driver-netsim` (merged); the control-plane companion, + incl. pcap capture - [netsim (`platform/tools/netsim`)](https://android.googlesource.com/platform/tools/netsim/) — `proto/netsim/packet_streamer.proto` - [google/android-cuttlefish](https://github.com/google/android-cuttlefish) From 3bf8b893162116f45017f1fbdefc8f665f021931 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Fri, 4 Sep 2026 23:52:46 -0400 Subject: [PATCH 23/26] docs(jep-0015): tighten proposal and convert diagrams to Mermaid Signed-off-by: Kirk Brauer --- ...5-multi-exporter-leases-port-forwarding.md | 2277 ++++++----------- docs/source/contributing/jeps/index.md | 2 +- 2 files changed, 735 insertions(+), 1544 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index 7f9a1a90f..c75ece2cd 100644 --- a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -5,10 +5,10 @@ | **JEP** | 0015 | | **Title** | Multi-Exporter Leases and Inter-Exporter Port Forwarding | | **Author(s)** | @kirkbrauer (Kirk Brauer) | -| **Status** | Draft | +| **Status** | Discussion | | **Type** | Standards Track | | **Created** | 2026-09-01 | -| **Updated** | 2026-09-03 | +| **Updated** | 2026-09-04 | | **Discussion** | *TBD (PR link)* | | **Requires** | JEP-0014 | | **Supersedes** | | @@ -18,201 +18,73 @@ ## Abstract -This JEP extends the existing `Lease` to bind more than one exporter, and adds -automatic port forwarding between those exporters so devices in one lease can -talk to each other. A lease gains an optional `spec.members[]` — each member a -role name (`phone`, `headunit`) plus the same selector fields a single-exporter -lease already uses — and an optional `spec.forwards[]` joining a **named port** -on one member to a named port on another. Ports are declared by drivers and -carried in the exporter's existing report, so a lease references -`headunit.rootcanal → phone.controller` and never learns an address; the -exporter resolves that locally. Because every member's exporter claim lives in -one object's status, binding a bench is a single atomic write: a lease holds -all of its devices or none. The data plane is the existing -`TcpPortforwardAdapter` with one substitution — a router peer stream in place -of a client stream — so `RouterService` and every driver work unchanged. This -realizes JEP-0014's deferred "composite leases — multiple exporters linked into -one logical lease" literally, and takes option 1 of JEP-0016's DD-8. The -worked example throughout is phone projection — a phone and a head unit -pairing over Bluetooth, then handing off to Wi-Fi — but nothing in the -mechanism is radio- or platform-specific: two ECUs joined over CAN, a BLE -peripheral and its gateway, or a plain serial cross-over are the same -declaration with different port names. +This JEP extends `Lease` to acquire multiple exporters together and forward +traffic between their named driver ports. Optional `spec.members[]` assigns +roles to exporters, and `spec.forwards[]` declares connections between them +without exposing local addresses. All member claims are committed in one +status update, and the members share a lease lifetime. Forwards reuse the +existing port-forwarding primitives and router, with authenticated direct +connections preferred within a network zone. The examples focus on phone +projection, but the same model supports CAN, serial, and other socket-based +connections. ## Motivation -Jumpstarter's lease is the unit of exclusive access to *one* exporter. -`LeaseSpec` carries a single `selector` or a single `exporterRef`, and -`LeaseStatus` records a single `status.exporterRef`. Every layer above -inherits that shape: `RequestLease`/`Dial` are keyed by one lease, -`jmp shell` exports one `JUMPSTARTER_HOST`, and `JumpstarterTest` acquires -"a lease for a single exporter using the selector annotation". - -That is the right primitive for the majority of HiL work — one board, one -harness — but an entire class of tests is about *interaction between -devices*, and today Jumpstarter cannot express it. - -### Why the exporter, not the device, is the unit of a lease - -That granularity is deliberate, and this JEP keeps it. An exporter is a -bench: a DUT plus its harness, and often several devices bolted to that -harness and wired to each other. They share cabling, power and a host, so one -composite driver exposes them and one lease hands over the whole assembly. -Leasing a device *inside* such a bench would let two clients hold opposite -ends of one cable. - -The gap is the opposite arrangement: a phone racked on one side of the lab -and a head unit bench on the other — two exporters, possibly on two hosts (a -host can run several), with nothing physical between them and no reason to -add it. Only the scheduler and a data path can join those. So the unit stays -the exporter and only the count changes, which is what makes the gain -combinatorial: N phones and M head unit benches give N×M benches out of N+M -exporters, any pairing gang-scheduled, none pre-wired. - -### The concrete problem: devices that must talk to each other - -A large class of tests is not about a device but about an *interaction* -between two of them. The shape recurs across domains: - -| Domain | Bench | What is exercised | -| --- | --- | --- | -| Phone projection | Phone + head unit | Pairing, then session handover to a high-bandwidth link | -| Wireless peripherals | Peripheral + host, or peripheral + gateway | Advertising, pairing, reconnection, roaming | -| Automotive networks | Two ECUs, or ECU + gateway | Bus arbitration, routing, diagnostics across a segment | -| Device-to-device apps | Two handsets | Discovery, transfer, sync over Bluetooth or Wi-Fi Direct | -| Serial / console harnesses | DUT + companion | Protocol conformance over a cross-over link | - -Every one of these needs the same two things Jumpstarter cannot give: **two -devices held at once**, and **a path between them**. - -**Phone projection is the worked example** throughout this JEP, because it -exercises the hardest version of both requirements. It is a two-device -protocol by construction: phone and head unit pair over **Bluetooth**, then -the head unit hands the session to a peer-to-peer **Wi-Fi** link, Bluetooth -having nowhere near the bandwidth for continuous video. Validating it means -holding both devices at once, driving both and asserting on both — a -phone-only lease cannot observe the handover at all. That structure holds for -every projection protocol in use and for head units running Android, Linux or -QNX; it is a property of projection, not of one vendor's stack. It also needs -two things existing tooling lacks: **heterogeneous benches**, and **pairing -two virtual devices to each other**. - -Running that on Jumpstarter today means hand-rolling everything the lease -layer should provide: - -- **No atomic acquisition.** The client requests two independent leases. If - the second selector is unsatisfiable the first is already held, so the - test either blocks holding scarce hardware or unwinds by hand. Under - contention, two concurrent benches can each take one half of the pair and - deadlock until expiry — the classic gang-scheduling failure. -- **No shared lifetime.** Two leases expire independently. Half a bench - disappearing mid-run produces a failure that looks like a device fault. -- **No correlation for policy or observability.** `ExporterAccessPolicy` - sees two unrelated requests; JEP-0013 traces see two unrelated leases. - Nothing records that these devices were *a bench*. -- **No path between the devices.** Even with both leases in hand, the two - exporters have no way to exchange device-level traffic. Every Jumpstarter - stream is client↔exporter; there is no exporter↔exporter path. - -The last point is the hard one, and it is what makes this more than an -ergonomics change. - -### The virtual case: host-local by construction - -For virtual devices the connectivity problem is sharper. Cuttlefish and the -Android emulator can already pair virtual devices to each other — but only -within one host. Cuttlefish shares its virtual radio media between instances -launched together, or launched separately against the *same* `wmediumd` -socket and rootcanal/netsim daemon; both are host-local Unix sockets and -loopback TCP ports. The emulator's networking backplane is explicitly scoped -to "all running instances on the same host machine", and container wrappers -publish each container's ports while keeping radio simulation inside the -container group. - -The common thread in the tooling we surveyed is that virtual multi-device -testing assumes *both devices live on one machine*. That is exactly the -assumption a cluster-scheduled pool of one-device-per-Pod exporters -(JEP-0016) breaks, and why its DD-8 deferred multi-device groups pending "a -cross-Pod virtual-radio story… real upstream-facing work." If someone has -solved the cross-host case in a way this survey missed, that is worth raising -in review — it would change the build-or-adopt calculation. - -The encouraging part is that these simulators are reached over ordinary -sockets: rootcanal accepts HCI on TCP — which is why -`jumpstarter-driver-bt-peer` can already attach a `bumble` peer with -`transport: "tcp-client:127.0.0.1:7300"` — netsim accepts virtual chips over -a bidirectional gRPC stream, and `wmediumd` speaks over a frame socket. -Nothing about "same host" looks fundamental to these interfaces; it appears -to be an artifact of where the sockets are reachable from, rather than a -property of the simulators themselves. - -### Prior art's ceiling - -Existing multi-device test frameworks stop in the same place, and the limit -is structural rather than accidental. They *can* gang-schedule several -devices into one job — role declarations, composite devices and coordinated -setup steps are all standard — but the devices must hang off a single lab -host: allocation refuses a job whose devices are attached to different hosts, -and where a multi-host mode exists it pools jobs across hosts rather than -letting one job span them. None of them offers a path *between* two devices; -the frameworks say so themselves, noting the absence of any API for -conducting an operation from one device against another. Their device -abstractions also tend to be single-platform in practice, so a bench pairing -a phone with a Linux or QNX head unit is out of reach before scheduling is -even a question. - -This JEP targets exactly that seam: - -- **Physical ↔ physical**: two devices on two exporters on **different lab - hosts**, in one lease. -- **Virtual ↔ virtual**: two emulated devices in two Pods, on two nodes, - pairing over Bluetooth and handing a session to Wi-Fi. - -**Scope.** This JEP covers **homogeneous benches** — two virtual devices, or -two physical devices. Mixing them in one bench (a real phone paired to a -virtual head unit) is a natural extension and the eventual prize, but it -needs real radio hardware bridging the two worlds and is deferred to keep v1 -tractable (DD-11). - -The virtual case is the interesting business case: labs have scarce physical -head units and abundant phones, or the reverse, and virtualizing the abundant -half while keeping the scarce half real is a cost and throughput win no -host-local scheduler can offer. +A Jumpstarter lease currently grants exclusive access to one exporter. +Tests involving several exporters must acquire separate leases, coordinate +their lifetimes, and arrange any connections between devices themselves. +If one request succeeds while another waits, a test holds hardware it cannot +use; concurrent tests can each hold a device the other needs. + +An exporter remains the unit of allocation. It owns a DUT and its harness, +which may include several physically connected devices. Leasing those devices +separately could give different clients control of the same assembly. This +proposal instead joins independently managed exporters into one lease, +called a **bench**. A pool of N phones and M head units can then support N×M +pairings without pre-wiring each pair. + +Phone projection illustrates the need: a test must control both a phone and +a head unit, pair them over Bluetooth, and observe the handover to Wi-Fi. +The devices may run different operating systems and belong to exporters on +different hosts. Other examples include two ECUs testing a CAN gateway, a +BLE peripheral and its central, or a DUT and a serial companion. + +Separate leases leave four gaps: + +- **Acquisition:** there is no all-or-nothing claim across exporters. +- **Lifetime:** devices can expire or be released independently. +- **Policy and observability:** requests have no shared bench identity. +- **Connectivity:** existing streams connect clients to exporters, with no + managed exporter-to-exporter path. + +For virtual devices, connectivity is also constrained by host-local simulator +interfaces. Cuttlefish instances can share rootcanal, netsim, and `wmediumd` +on one host. Separately scheduled exporter Pods need a path between those +interfaces. HCI over TCP can use a byte forward; interfaces such as +vhost-user require a local bridge as well (DD-9, DD-10). + +The initial scope covers two virtual devices or two physical devices. +Physical radio peers must remain within RF range even when their exporters +run on different hosts. Connecting physical and simulated radios requires +additional hardware and is deferred (DD-11). ### User Stories -- **As a** phone-projection QA engineer, **I want to** lease a phone and a - head unit as one bench with roles, **so that** my test either gets both - devices or waits — never half a bench, never a deadlock against a - concurrent run. -- **As an** automotive platform developer, **I want to** pair two *virtual* - devices running in different cluster Pods over Bluetooth and then hand off - to Wi-Fi, **so that** I can validate wireless projection in CI without a - physical lab or a single fat host. -- **As a** lab operator whose devices are in different racks or different - buildings, **I want** one lease to span them, **so that** my bench is not - limited to devices plugged into the same machine. -- **As a** test author with a mixed bench, **I want** a QNX or Linux head - unit alongside an Android phone, **so that** device type is a driver - choice rather than a platform limit. -- **As an** embedded engineer testing a CAN gateway, **I want** two ECU - exporters joined over a forwarded bus, **so that** I can exercise routing - between segments without physically cabling them to one machine. -- **As a** BLE peripheral developer, **I want** my DUT leased alongside a - central acting as its phone, **so that** pairing and reconnection are - covered in CI rather than by hand at a desk. -- **As a** Jumpstarter user who already knows leases, **I want** a bench to - be *a lease*, **so that** everything I know about `jmp create lease`, - expiry, release, and access policy carries over without learning a second - resource. +- A projection test acquires a phone and head unit together, controls both + by role, and retains access to surviving devices for diagnostics if one + fails. +- A CI test pairs virtual devices in separate Pods and runs a projection + session without physical phone or head unit hardware. +- A lab test joins exporters on different hosts, such as two ECUs connected + through a CAN-over-TCP bridge. ## Proposal -Three additions, none of them a new resource kind: +The proposal adds three concepts to existing resources: - **Members.** A lease gains optional `spec.members[]`, each a role name plus the same `selector` / `exporterRef` fields a single-exporter lease already - uses. A lease with members binds every one of them or none. + uses. A lease binds all required members or holds none. - **Ports.** A driver may declare named ports it **provides** (a service listening locally) or **requires** (a socket it will dial). Ports travel in the exporter's existing report; addresses never leave the exporter. @@ -220,22 +92,20 @@ Three additions, none of them a new resource kind: provided port on one member to a required port on another. The exporters establish the forward themselves over the existing router. -A lease with members is called a **bench** informally, but it is not a new -kind of object: it is a `Lease`, it appears in `jmp get leases`, it expires -the way leases expire, and `spec.release` ends it. +A bench appears in `jmp get leases` and uses the existing expiry and +`spec.release` behavior. ### Ports -A port is a named connection point on a driver. There are exactly two -directions, and the direction is what makes a forward well-formed. +A port is a named connection point on a driver. Each forward connects a +`provides` port to a `requires` port. | Direction | Meaning | Example | | --- | --- | --- | | `provides` | A service is listening; something may be forwarded *from* it | `rootcanal` on a Cuttlefish exporter — HCI on `127.0.0.1:7300` | | `requires` | The driver will dial a local address; a forward may be delivered *to* it | `controller` on a `bt-peer` exporter — where its bumble stack expects an HCI controller | -`provides` needs almost nothing new, because a `TcpNetwork` child already -*is* a named provided port — the child's name is the port name: +A `TcpNetwork` child can expose a provided port: ```yaml export: @@ -251,8 +121,8 @@ export: protocol: hci-h4 # optional ``` -`requires` is the genuinely new concept: the driver will dial a local -address, and the exporter binds an inbound forward there. +For a `requires` port, the exporter binds a local listener that the driver +dials: ```yaml export: @@ -267,10 +137,9 @@ export: protocol: hci-h4 # optional ``` -Note what did *not* change: `bt-peer`'s Python is untouched and its -`transport` still points at `127.0.0.1:7300`. The lease makes that address -resolve to a rootcanal in another Pod on another node — a driver that can -talk to a local service now talks to a remote one without knowing. +The `bt-peer` driver still dials `127.0.0.1:7300`. The exporter forwards that +connection to the remote rootcanal; using the forward requires no changes to +the driver's Python code. ### Declaring a bench @@ -301,25 +170,19 @@ spec: - { member: phone, port: controller } ``` -No addresses, no port numbers, no medium taxonomy — and no statement of which -side listens. The lease names roles and port names; both sides' internals -stay inside their exporters (DD-6), and the controller resolves direction -from the reported ports at bind time (DD-13). +The lease names members and ports. Exporters resolve addresses locally, and +the controller determines direction from reported ports at bind time (DD-6, +DD-13). -`members[].selector` and `members[].exporterRef` are the *same* fields as -the top-level `spec.selector` and `spec.exporterRef` — a member is the -existing lease request with a role name attached. The single-exporter form is -unchanged and exactly equivalent to a one-member lease, so today's leases are -already the degenerate case rather than a legacy shape to migrate (DD-2). +Each member uses the existing `selector` or `exporterRef` fields with a role +name. The scalar lease form remains supported (DD-2). -A member may be marked `optional: true`, in which case the lease binds -without it and the role resolves to `None` on the client. +A member marked `optional: true` may be omitted at binding; its role then +resolves to `None` on the client. -#### A bench with no radios in it +#### CAN example -Nothing above is projection-specific. The same two fields express two ECUs -sharing a CAN segment, where one exporter fronts the bus over TCP (via -`socketcand` or an equivalent bridge) and the other dials it: +Two ECUs can share a CAN segment through `socketcand` or another TCP bridge: ```yaml spec: @@ -335,10 +198,8 @@ spec: - { member: node, port: can } ``` -Only the port names differ. The controller applies the same validation — -both ports exist, `provides` meets `requires`, declared protocols agree — and -the same forward machinery carries the bytes. A serial cross-over between a -DUT and a companion board is the same shape again. +The controller checks that both ports exist, their directions complement +each other, and any declared protocols agree. ### Acquiring and using a bench @@ -359,8 +220,7 @@ projection-bench false ci-runner phone=rack3-phone-4, 12s headunit=virt-hu-7b2c ``` -Port names are discoverable rather than tribal knowledge, which is the point -of putting them in the report (DD-7): +Users can discover port names from the exporter report (DD-7): ```console $ jmp get exporter virt-hu-7b2c -o json | jq '.status.devices[].ports' @@ -400,7 +260,7 @@ with config.lease( headunit.power.on() phone.adb.wait_for_device() - # The forward is already up; drive the protocol, not the plumbing. + # Wait for the forward before starting the peer. bench.forwards["bt"].wait_connected(timeout=30) phone.bt_peer.start({"name": "Bumble-Phone"}) phone.bt_peer.wait_connection(timeout=60) @@ -408,91 +268,61 @@ with config.lease( assert phone.adb.shell("dumpsys bluetooth_manager | grep -c Connected") == "1" ``` -`config.lease(selector=...)` without `members` behaves exactly as it does -today and yields a single client — the one-member case is not special-cased -in the API, only in what `connect()` returns (DD-2). +`config.lease(selector=...)` keeps its existing behavior, with `connect()` +yielding a single driver client (DD-2). `JumpstarterTest` grows `members` and `forwards` class variables next to `selector`, so existing pytest suites extend without a second base class. ### Running existing multi-device suites -Because a bench is a set of roles with ADB-capable members, it can be -projected into the config formats existing Android multi-device tests already -consume: +A bench with ADB-capable members can be exported as a Mobly testbed: ```console $ jmp get lease projection-bench -o mobly > testbed.yml $ mobly_test.py -c testbed.yml --test_bed projection-bench ``` -which emits a Mobly testbed whose `AndroidDevice` controllers point at the -per-member ADB endpoints Jumpstarter has forwarded locally, with the role -name carried through as the Mobly device label; other frameworks' device-set -formats map the same way. This is the integration seam: the tests and the -results pipeline do not change, while the *bench* changes from "devices -sharing one lab host" to "any mix of physical and virtual devices anywhere -the controller can reach." +The exported `AndroidDevice` controllers use locally forwarded ADB endpoints +and retain the member names as device labels. Existing tests and results +pipelines can use these endpoints. The lifetime of the local ADB forwards +remains an open question. ### How a forward comes up -The data plane already exists. `adapters/portforward.py` implements -`TcpPortforwardAdapter` as `TemporaryTcpListener` + `client.stream_async()` + -`forward_stream()`. The inter-exporter version is the same three primitives -with one substitution — a router peer stream in place of the client stream: - -1. Once the lease is bound, the controller validates both ports against the - exporters' reports and instructs each exporter over its existing `Listen` - stream. -2. Each exporter calls the new `DialPeer` RPC and receives a router endpoint - plus a token whose `stream` claim is the **same** for both ends. -3. Both call `RouterService.Stream`. The router already "connects caller to - another caller of the same stream" — a symmetric rendezvous that needs no - protocol change to pair two exporters instead of a client and an exporter - (DD-5). -4. The `provides` side dials its local service (`127.0.0.1:7300`) and splices - it to the stream with `forward_stream`. The `requires` side opens a - `TemporaryTcpListener` on its declared `listen` address and splices each - accepted connection to the stream. -5. **Fast path**: if `DialPeer` says the pair is direct-eligible — both - members in one network zone, which two exporter Pods in a cluster are — - the `requires` side dials the peer directly first and uses the router - stream only if that does not complete quickly (DD-4). Two virtual targets - in a cluster therefore talk over the Pod network, and the same lease keeps - working unchanged when one member is an edge device the other cannot - reach. - -```text - ┌──────── Lease: projection-bench (one object) ─────────┐ - │ status.members: │ - │ phone → Exporter/rack3-phone-4 │ - │ headunit → Exporter/virt-hu-7b2c │ - └───────┬─────────────────────────────────┬───────────────┘ - │ │ - ┌────────────▼────────────┐ ┌────────────▼────────────┐ - │ Exporter: rack3-phone-4 │ │ Exporter: virt-hu-7b2c │ - │ bt_peer │ │ cuttlefish │ - │ requires: controller │ │ provides: rootcanal │ - │ listener 127.0.0.1:7300│ │ dials 127.0.0.1:7300 │ - └────────────┬────────────┘ └────────────┬────────────┘ - │ │ - │ ┌───────────────────┐ │ - └─────►│ RouterService │◄─────┘ (default) - │ same stream id │ - └───────────────────┘ - └───────── direct peer ─────────┘ (fast path) +The exporter reuses `TemporaryTcpListener` and `forward_stream()` from +`TcpPortforwardAdapter`, replacing the client stream with a peer stream: + +1. After binding, the controller validates the ports and sends setup + instructions over each exporter's existing `Listen` stream. +2. Each exporter calls `DialPeer` for connection details and credentials. +3. For a direct-eligible pair, the `requires` side first attempts an + authenticated direct connection with a bounded timeout. Otherwise, or if + that attempt fails in `Auto` mode, both endpoints call + `RouterService.Stream` with tokens sharing one `stream` claim (DD-4, DD-5). +4. The `provides` side dials its local service. The `requires` side listens + on its configured address. Both splice local connections to the peer + stream using `forward_stream()`. + +```{mermaid} +flowchart TD + lease["Lease: projection-bench"] + phone["Exporter: rack3-phone-4
bt_peer · requires: controller
Listens on 127.0.0.1:7300"] + headunit["Exporter: virt-hu-7b2c
cuttlefish · provides: rootcanal
Dials 127.0.0.1:7300"] + router["RouterService"] + + lease -.->|"status.members: phone"| phone + lease -.->|"status.members: headunit"| headunit + phone <-->|"Direct peer: preferred in same zone"| headunit + phone <-->|"Router fallback"| router + router <--> headunit ``` The client is not in the data path. ### Attaching media, simulated and physical -With forwards as the mechanism, "bridging a medium" stops being a special -subsystem and becomes a question of which port each stack exposes. Radios are -the demanding case, but nothing in the table below is privileged — a CAN bus -or a serial cross-over is the same declaration: - -First, the **data plane** — ports that a forward carries: +Drivers expose the connection points that forwards carry: | Stack | Port | Direction | Notes | | --- | --- | --- | --- | @@ -507,41 +337,28 @@ First, the **data plane** — ports that a forward carries: | `socketcand` / CAN-over-TCP bridge | `can` | provides | A CAN segment reachable as a socket | | Serial bridge (pty or TCP) | `console` | requires/provides | Cross-over between a DUT and a companion | -Second, the **control plane** — drivers that configure and observe a medium -without carrying its traffic. `jumpstarter-driver-netsim` is the worked -example: it speaks netsim's REST API to list devices, toggle radios, patch -state, reset, and start/stop/download **pcap captures** of the simulated air. -It is not a forward endpoint and needs none of this JEP's machinery; the two -compose, with the netsim driver observing the medium a forward connects. - -Two consequences of the table generalize beyond radios. HCI is -**asymmetric** — a host attaches to a controller, so forwarding one device's -HCI port into another's achieves nothing, which is why forwards are -directional and `provides → provides` is rejected (DD-6); joining two -controllers is a different port, which is why the table lists both (DD-9). -And not every port is a transparent splice: netsim's `PacketStreamer` -requires the attaching side to originate a call carrying `ChipInfo` before -traffic flows, so its consumer is a protocol-terminating driver rather than a -raw socket. Forwards carry both kinds; the difference lives in the driver at -the `requires` end. - -The two bench kinds are asymmetric too. For **two virtual devices** the -medium is simulated, so a forward carries it. For **two physical devices** -the medium is the air: devices in RF range pair with no forward at all, and -forwards carry only that bench's wired media — a CAN segment, a serial -cross-over. What a physical bench needs is the lease plane, which is what -lets its two exporters live on different hosts. - -### A projection bench, concretely - -**Virtual.** The phone is a Cuttlefish exporter booting a vendor phone image -— the stack that ships on retail devices, not an AOSP approximation — with -the projection app installed and its developer-mode server started by the -driver. The head unit is a *receiver* run as a process inside the head-unit -exporter by a `projection-rx` driver, because an automotive Android CVD -cannot receive a session: the receiver is part of the vendor stack, not of -AOSP. The receiver is a TCP client, so it is the `requires` side and the -phone `provides` the port it dials: +Control drivers configure and observe a medium separately from its forwarded +traffic. For example, `jumpstarter-driver-netsim` uses the REST API to list +devices, toggle radios, reset state, and collect pcap captures. It is not a +forward endpoint. + +HCI connects a host to a controller. Joining two controllers requires a +link-layer port instead; joining two HCI `provides` ports is invalid (DD-6, +DD-9). Some interfaces also need protocol handling at the endpoint: netsim's +`PacketStreamer` requires a gRPC call carrying `ChipInfo`, so its consumer +must implement that handshake. + +Virtual radio benches forward simulator traffic. Physical radio peers +communicate over the air and may need only the shared lease; forwards can +still carry wired traffic such as CAN or serial. + +### Projection example + +**Virtual.** A Cuttlefish phone exporter runs a vendor phone image with the +projection app and its developer-mode server. A `projection-rx` driver runs +a receiver process in the head unit exporter, since AOSP automotive images +do not include that receiver. The phone provides the port and the receiver +dials it: ```yaml spec: @@ -557,23 +374,18 @@ spec: - { member: headunit, port: phone } ``` -`projection-wifi` reaches the phone's server through the guest's Wi-Fi -interface — the shape of a wireless session — where `projection` reaches it -over ADB, the shape of a USB session. The forward is the same either way; the -port name is the test's statement of which cable it is pretending to be -(DD-13). +`projection-wifi` reaches the server through the guest's Wi-Fi interface; +`projection` reaches it over ADB. The test chooses which path to exercise +(DD-13). Neither forward alone tests the Bluetooth-to-Wi-Fi handover (DD-10). -**Physical.** A head unit on one exporter and a phone on another, on -different hosts. The radios are the air, so the handover needs no forward; -what this bench needs is the lease plane, and its forwards carry only wired -media — a CAN segment feeding the head unit, a serial console. A software -receiver can stand in for the head unit here too, giving a physical phone a -hardware-free counterpart. +**Physical.** A phone and head unit on different exporters pair over the air. +The lease coordinates their access and lifetime; forwards can carry wired +connections such as a CAN segment feeding the head unit. ### API / Protocol Changes -All changes are **additive fields on existing types**, plus one new RPC. No -new CRD, no new API group, no existing field changes meaning. +The API adds fields to existing types and one RPC. Existing fields retain +their meanings. **Driver report** — ports are optional, and an exporter that reports none simply cannot participate in forwards (DD-7): @@ -697,7 +509,7 @@ new rules for mutual exclusion, unique role names, forwards referencing declared members, member immutability (mirroring `tags` and `context`), and exactly one of `between` or `from`+`to` per forward. -**Protocol** — three additive fields and one new RPC: +**Protocol** — additions to existing messages and one new RPC: ```protobuf message RequestLeaseRequest { @@ -754,39 +566,25 @@ message DialPeerResponse { ### Hardware Considerations -- **No new lab hardware is required.** Homogeneous benches use what a lab - already has: two physical devices pair over the air as they always have, - and two virtual devices need no radio at all. The radio bridging hardware - that a mixed physical/virtual bench would need is out of scope (DD-11). -- **RF isolation.** Multiple physical benches in one room share the air. Labs - running more than one need shielded enclosures or channel planning; the - controller cannot schedule around collisions it cannot observe. Express RF - domains as exporter labels and let selectors keep benches apart. -- **Physical proximity is a scheduling constraint** for physical ↔ physical - radio benches, modeled with ordinary labels (`rf-domain: rack-3`), not new - machinery. -- **Latency budgets.** Bluetooth HCI is timing-sensitive: supervision - timeouts are seconds, but L2CAP/HCI flow control and A2DP jitter buffers - are far tighter. A router-relayed forward adds two gRPC hops; measuring - that budget is an acceptance criterion, and it is why an in-cluster bench - takes the direct path by default (DD-4) and keeps the router as the - fallback that works everywhere. -- **Wi-Fi frame forwarding is the most latency-sensitive path.** `wmediumd` - models RSSI-based delivery and expects medium-like timing; a TCP substrate - introduces head-of-line blocking a real air interface does not have, and - its vhost-user transport needs a frame bridge on each side before any - forward is involved (DD-10). -- **No head unit hardware is needed to project.** Google's Desktop Head - Unit is the receiver for virtual benches and can stand in for one on a - physical phone's bench (see *A projection bench, concretely*). -- **Listener addresses.** A `requires` port binds a fixed local address, - which is safe because each exporter owns its network namespace (see - *Deployment assumptions*). The one configuration that breaks this is a - host-networked container sharing a machine with other exporters, which is - outside the supported deployment model. -- **Degraded hardware.** If a member's exporter goes offline mid-lease, the - lease reports `Ready=False` naming the role and does *not* silently - continue with a partial bench (DD-3). +- **Hardware:** virtual benches need KVM-capable hosts but no physical + radios. Physical benches use existing devices and harnesses. Mixed radio + benches require additional adapters and are deferred (DD-11). +- **RF range and isolation:** physical radio peers must be within range. + Shared labs may need shielded enclosures or channel planning. Exporter + labels such as `rf-domain: rack-3` express placement constraints for + selectors; the controller does not measure RF interference. +- **Bluetooth latency:** HCI flow control and audio buffering can be more + sensitive than supervision timeouts. Measure both router and direct paths + against the intended workloads (DD-4). +- **Wi-Fi simulation:** vhost-user needs a frame bridge, and TCP adds + head-of-line blocking that may affect medium timing (DD-10). +- **Projection receiver:** virtual benches use a software receiver, such as + Google's Desktop Head Unit (see *Projection example*). +- **Listener isolation:** fixed `requires` addresses rely on each exporter + owning its network namespace. Multiple host-networked exporters sharing + one machine are unsupported. +- **Member loss:** report `Ready=False` with the failed role and retain + surviving members for diagnostics (DD-3). ## Design Decisions @@ -795,55 +593,29 @@ message DialPeerResponse { **Alternatives considered:** 1. **Extend `Lease`** with `spec.members[]` / `status.members[]`; a - single-exporter lease is the one-member degenerate case. + single-exporter lease uses the same selection logic. 2. **A new `LeaseGroup` CR owning N child `Lease` CRs.** 3. **Client-side coordination only** — the client acquires N leases and correlates them by tag. **Decision:** Option 1 — extend `Lease`. -**Rationale:** The deciding factor is how exporter exclusivity is actually -implemented. In `lease_controller.go`, an exporter is claimed by writing -`lease.Status.ExporterRef`, and exclusivity is enforced by scanning other -active leases for a claim on the same exporter -(`ListActiveLeases` → `attachExistingLeases` → `filterOutLeasedExporters`). -The claim is persisted by a **single** `r.Status().Update(ctx, &lease)`. - -All of a lease's exporter claims therefore live in one object's status, and -binding N of them is one write. Either every member's claim lands or none -does. **Partial acquisition is structurally impossible**, so there is nothing -to time out, nothing to release-and-retry, and no gang scheduler. - -Option 2 gets the opposite property. With N child leases, the existing -reconciler binds each independently with its own `Status().Update`, so "three -of four members bound" is a real, durable state in etcd. Everything needed to -recover from it — admission gating, an acquisition timeout, -release-all-on-timeout, jittered backoff, and a contention test to -demonstrate no deadlock — exists purely to clean up after a state option 1 -never enters. That is a large amount of new machinery in the most -correctness-critical controller in the project, in exchange for a separation -that buys little: JEP-0014 already described the goal as "multiple exporters -linked into **one logical lease**." - -Option 2's genuine advantage is that it does not touch `Lease`. That matters -less than it appears, because the compatibility risk concentrates in exactly -one field, `status.exporterRef`, which DD-2 handles directly. Option 2 also -does not avoid protocol work: `Dial(lease_name)` has no way to say *which* -device, so a member selector must be added either way. - -Option 3 provides no atomicity, no co-scheduling, no deadlock avoidance, and -leaves policy unable to reason about a bench — most of the motivation. - -**One thing this decision does not fix.** Atomic means no partial *holds*; it -does not mean two leases can never race for the same exporter. The -exclusivity scan reads a possibly-stale cache, and two leases racing for one -exporter write to *different* objects, so optimistic concurrency does not -catch it. That race exists today for single-exporter leases and is already -acknowledged in the code (*"we could have multiple clients trying to lease -the same exporters… we will need to construct a lease scheduler with the view -of all leases and exporters"*). This JEP neither worsens nor fixes it. A -multi-member lease does fit that future scheduler better than N correlated -objects would: one lease is one scheduling unit, the way one Pod is. +**Rationale:** An exporter claim is stored on the lease. The current controller writes +`lease.Status.ExporterRef` and checks other active leases through +`ListActiveLeases` → `attachExistingLeases` → `filterOutLeasedExporters`. +Writing all member claims in one `Status().Update` prevents a lease from +persisting a partial acquisition. + +Child leases would bind independently and require acquisition timeouts, +release-and-retry behavior, and contention handling. Client-side coordination +has the same partial-acquisition problem. Both approaches also need a way to +select a member when dialing. + +This atomic write does **not** guarantee exclusivity across leases. Two +reconcilers can read stale claims and write conflicting selections to +different lease objects. That race already exists; larger benches create +more opportunities to encounter it. A global scheduler remains separate +work, as noted in the controller's existing TODO. ### DD-2: Keep `status.exporterRef` scalar; add `status.members[]` alongside @@ -857,32 +629,19 @@ objects would: one lease is one scheduling unit, the way one Pod is. **Decision:** Option 1. -**Rationale:** This is the whole compatibility story of DD-1. Every existing -reader of `status.exporterRef` — the `Dial` path, the `Exporter` print -column, JEP-0013 telemetry, and the JEP-0016 Host Orchestrator façade -(explicitly *a view over ordinary Lease CRs*) — keeps working -**byte-identically** for single-exporter leases, which is every lease that -exists today. - -For multi-member leases, option 1 makes old readers see a nil -`status.exporterRef`, which they already interpret as "not bound yet". That -is a **fail-safe** degradation: an unaware consumer shows an unbound lease -rather than confidently operating on one arbitrary device out of several. -Option 3 is the fail-dangerous version — the façade would present a -two-device bench as a single host and route Host Orchestrator calls to -whichever member sorted first. Option 2 is honest but forces a migration on -every consumer for a feature most will never see. - -The same rule applies on the wire: `GetLeaseResponse.exporter_uuid` stays set -only for single-member leases. In the client library, `lease.connect()` -returns a driver client directly for a single-member lease and a role mapping -for a multi-member one — a difference the caller opted into by passing -`members`. - -`Dial` is the one place where silence is unacceptable, because a multi-member -lease has no defensible default device. `DialRequest` gains an optional -`member_name`; calling `Dial` on a multi-member lease without it returns -`INVALID_ARGUMENT` naming the available roles rather than guessing. +**Rationale:** Existing consumers retain the scalar field for single-exporter leases. +For multi-member leases, an old reader sees nil and treats the lease as +unbound, rather than selecting an arbitrary member. Populating the scalar +with the first member would misroute consumers such as the JEP-0016 Host +Orchestrator façade. Replacing it with a list would require every consumer +to migrate. + +`GetLeaseResponse.exporter_uuid` follows the same convention. The client +returns a driver client for a single-member lease and a role mapping for a +multi-member lease. + +`DialRequest.member_name` selects a role. Omitting it for a multi-member +lease returns `INVALID_ARGUMENT` listing the available roles. ### DD-3: Partial-bench behavior when a member is lost mid-lease @@ -895,16 +654,12 @@ lease has no defensible default device. `DialRequest` gains an optional **Decision:** Option 1. -**Rationale:** Option 3 is how a two-device test turns into a confusing -one-device pass; it is never right. Between 1 and 2, keeping the survivors -held respects the test: a client that has just lost its head unit usually -wants to collect logs and artifacts from the phone before releasing, and an -abrupt teardown destroys the evidence needed to diagnose the failure. The -condition is loud, and the client library raises on the next call into the -failed role rather than returning stale data. - -This concerns a member lost *after* binding — distinct from acquisition, -which DD-1 makes all-or-nothing. +**Rationale:** Keeping surviving members leased lets the client collect logs and +artifacts before release. The lease reports the failed role with +`Ready=False`, and the client raises on the next call into that role. +Immediate release would remove that diagnostic access; silently continuing +could hide an incomplete test. This behavior applies after binding; +acquisition remains all-or-nothing. ### DD-4: Forward transport — router peer streams vs. client relay vs. pure P2P @@ -916,55 +671,22 @@ which DD-1 makes all-or-nothing. **Decision:** Option 1. -**Rationale:** Option 2 needs *zero* protocol change and could be prototyped -immediately — but it doubles RTT on a latency-sensitive path, routes lab -traffic through whatever machine ran the test, and breaks for headless CI and -sustained throughput. It is retained as an explicit `mode: client-relay` -debug transport, not as the architecture. - -Option 3 has the best data plane and fails at exactly the case this JEP is -for: an edge-device exporter behind NAT and a cluster Pod, which are -not mutually routable. Jumpstarter's value in that topology is that the -controller and router are the only things both sides must reach. - -Option 1 keeps one authentication model — a forward is authorized by the -lease that names it, exactly as a stream is authorized by the lease that -names it — while permitting the fast data plane where it is available. A -bench that works over the router works everywhere, and `mode: router` makes -the slow path explicit for tests that need reproducible timing. - -**Direct is preferred, not merely permitted, when both members sit in the -same network zone.** For two virtual targets in one cluster — the case this -JEP is mostly for, and the case JEP-0016 produces by construction, since its -provisioner renders both benches as Pods under one `ExporterSet` — `Auto` -resolves to a direct Pod-to-Pod connection and falls back to the router only -if that dial does not complete. Three reasons, in order of weight: - -1. **The router is a shared component and bench traffic is sustained.** A - projection session or an A2DP stream is not a burst; N benches funnelling - media through one router deployment makes a central bottleneck out of - something the CNI would carry for free, and in a cloud install it can - also mean paying to leave and re-enter the network. -2. **The router path in a typical install leaves the cluster and comes - back.** It therefore inherits the ingress's failure modes — the reload - that cut a live bench's HCI splice during verification was an ingress - worker drain, not a Jumpstarter fault. A direct forward between two Pods - stays inside the CNI and never meets that machinery. -3. **The workloads that are latency-sensitive are the ones still ahead.** - Measured on the bench, the router costs **0.65 ms** on a round-trip - (0.70 ms through the forward against 0.05 ms direct to the same - endpoint), while an HCI command takes **~45 ms either way** because - rootcanal answers on its own schedule. Simulated Bluetooth pairing - therefore cannot tell the two apart — which is exactly why preferring - direct is free — but the Phase 4 frame bridge, where `wmediumd` expects - medium-like timing, and sustained projection throughput are where a 14× - round-trip difference starts to decide whether a bench works at all. - -That last measurement is what makes the preference safe rather than a -gamble: falling back to the router costs sub-millisecond on the path that -has been measured, so a bench that cannot get a direct connection is slower -in a way nothing so far can detect. Eligibility is a property of the pair, -not a user decision — see *Direct eligibility* under Design Details. +**Rationale:** The router connects exporters that cannot reach each other, including +edge devices behind NAT. Direct connections avoid router load and reduce +latency where peers are reachable. Client relay adds a dependency on the +client's network and lifetime; it remains an explicit debug mode. + +In `Auto` mode, same-zone pairs attempt a direct connection first, then fall +back to the router after a bounded timeout. `Router` forces the router path +for testing, and `Direct` fails if a direct connection cannot be established. +Both paths authenticate against the lease. + +Direct connections also avoid ingress-related stream failures where the +router route passes through an ingress. In the single-node prototype, a +router forward had a median round-trip time of 0.70 ms versus 0.05 ms direct; +rootcanal HCI commands took about 45 ms either way (DD-9). These measurements +do not establish cross-node or sustained-throughput limits. Phase 4's Wi-Fi +frame bridge requires separate latency testing. ### DD-5: Router changes — none @@ -976,15 +698,10 @@ not a user decision — see *Direct eligibility* under Design Details. **Decision:** Option 1 — no `router.proto` change. -**Rationale:** `RouterService.Stream`'s documented contract is already -"Stream connects caller to another caller of the same stream", and its token -claims (`sub: jumpstarter client/exporter`, `stream: stream id`) already -accommodate an exporter as either party. The router is a symmetric rendezvous -that never needed to know which side was the client. A peer RPC would encode -an asymmetry that does not exist and fork the data path — two implementations -to keep correct, two places to fix flow-control bugs, and a second surface -for the datagram work in Future Possibilities. Everything peer-specific -belongs in token issuance, which is the controller's job. +**Rationale:** `RouterService.Stream` pairs callers with the same stream claim and +already accepts exporter identities. Peer-specific authorization belongs in +the controller's token issuance. A second router RPC would duplicate the +stream path without changing its behavior. ### DD-6: Named ports with direction, not addresses @@ -998,26 +715,15 @@ belongs in token issuance, which is the controller's job. **Decision:** Option 1. -**Rationale:** Option 2 forces the *client* to know the internal architecture -of an exporter it did not configure: that `bt-peer` expects its controller on -7300, that the address is on loopback, that nothing else is bound there. That -is exporter-private detail leaking into a lease manifest, and it breaks the -moment a lab reconfigures a port. With named ports the lease says -`headunit.rootcanal → phone.controller` and each exporter resolves its own -address locally. - -Direction is what makes option 1 strictly better than option 3, and it took -three attempts to see why. A forward is inherently asymmetric — one side has -a listening service, the other gets a local listener — and that asymmetry is -exactly the relationship the underlying protocols have. HCI has a host side -and a controller side: `bt-peer` attaches to rootcanal *as a host*. Forward -one rootcanal's port into another rootcanal and you have wired controller to -controller, and nothing happens. A `provides → provides` forward is therefore -rejected structurally, with no knowledge of Bluetooth anywhere in the -controller. - -Direction also gives the validation something real to check (DD-7), which -name-matching alone cannot. +**Rationale:** Named ports let exporter configuration own addresses. A lease can +reference `headunit.rootcanal` and `phone.controller` without knowing their +loopback addresses or port numbers. + +Direction allows the controller to reject invalid connections without +understanding the device protocol. For example, a Bumble host can attach to +a rootcanal HCI controller; connecting two rootcanal HCI services cannot +provide that relationship. The controller rejects the latter because both +ports declare `provides`. ### DD-7: Ports are an optional part of the exporter report @@ -1030,37 +736,21 @@ name-matching alone cannot. 3. **No reporting** — ports live only in exporter config, and forwards fail at connect time if misconfigured. -**Decision:** Option 1, and optional in the strong sense: an exporter that -reports no ports simply cannot participate in forwards. - -**Rationale:** Option 3 loses the two things that make named ports usable. -Without reporting, the controller cannot validate a forward before -establishing it, and — more importantly — a client has no way to *discover* -port names, which just relocates the tribal-knowledge problem DD-6 set out to -solve. Ports in the report make `jmp get exporter` the answer to "what can I -wire up here", the same way it already answers "what drivers are here". That -also sits naturally alongside JEP-0011's introspection direction rather than -inventing a parallel discovery channel. - -Option 2 is genuinely tempting: `DriverInstanceReport.labels` already exists -and already reaches `ExporterStatus.Devices`, so ports could ship with no -schema change at all. It fails on the `provides`/`requires` asymmetry. A -`provides` port is already a driver instance — a `TcpNetwork` child with a -uuid in the report — so labelling it annotates something real. A `requires` -port **has no backing driver instance**; it is a declared need, so option 2 -would require synthesizing a phantom report entry with no methods purely to -carry two labels, plus two parallel key namespaces (`port.jumpstarter.dev/*` -and `port-protocol.jumpstarter.dev/*`) to keep in sync. - -Being optional is what makes this free: `repeated` defaults to empty, so -every existing exporter reports nothing new, no version negotiation is -needed, and an old exporter against a new controller is simply not a forward -endpoint — the correct answer anyway. - -**What is deliberately not reported:** the `listen` address of a `requires` -port. It is local to the exporter and no other component needs it (DD-6). -Keeping it out means the report carries only label-safe scalars, and means a -lab can re-address a port without touching anything outside that exporter. +**Decision:** Option 1. Exporters without reported ports can join leases +but cannot participate in forwards. + +**Rationale:** Reported ports support discovery through `jmp get exporter` and +validation before a forward starts. Configuration alone would defer errors +to connection time and leave clients unable to discover available names. + +Labels would overload driver metadata: a provided port can correspond to a +`TcpNetwork` child, while a required port describes a connection the driver +needs. A structured field represents both without synthetic driver entries +or separate label conventions. + +An absent `ports` field defaults to an empty list, so old exporters continue +to register and serve ordinary leases. The local `listen` address stays in +exporter configuration and is not reported. ### DD-8: No protocol taxonomy; direction plus an optional tag @@ -1074,30 +764,16 @@ lab can re-address a port without touching anything outside that exporter. **Decision:** Option 3. -**Rationale:** Option 1 validates the wrong axis. `medium: bluetooth` would -approve a forward between a raw-HCI rootcanal port and a netsim -`PacketStreamer` port — both are "bluetooth", neither can talk to the other, -because one is raw bytes and the other is length-delimited protobuf behind a -`ChipInfo` handshake. It describes what traffic *represents*, not what the -bytes *are*, so it approves the one pairing that most needs catching. - -Option 2 fixes the axis but is still insufficient on its own: `hci-h4` on -both ends is exactly the controller-to-controller wiring DD-6 rejects. The -thing that had to be checked was never the payload format but the -*relationship*, and once direction is modelled, most of the value is already -captured without any taxonomy to define, version, or argue about. - -An optional `protocol` string remains useful for the residue — it catches -wiring a `vnc` provides-port into an HCI requires-port — but comparing it -only when both ends declare it keeps it opt-in. Driver authors who want the -check get it; nobody is forced to classify anything, and there is no closed -enum to maintain. - -The honest cost: a forward whose two ends speak different protocols and -declare no `protocol` tag will connect and then fail at the first byte. This -is the same guarantee `kubectl port-forward` gives, which nobody treats as a -defect, but it is a deliberate retreat from admission-time protocol -validation and is recorded as such. +**Rationale:** A medium name does not establish wire compatibility: rootcanal HCI +and netsim `PacketStreamer` both carry Bluetooth traffic but use different +protocols. A wire-format token alone also misses direction errors such as +connecting two HCI controllers. + +Direction is required. A free-form `protocol` adds an optional compatibility +check when both ends declare it, without a centrally maintained taxonomy. +If either end omits the tag, the forward may connect despite incompatible +protocols and fail when data is exchanged. Forwarding guarantees byte +transport, not protocol compatibility. ### DD-9: Where simulated media attach @@ -1115,154 +791,69 @@ validation and is recorded as such. 5. **Inside the guest** — a shim in Android proxying Bluetooth/Wi-Fi at the HAL or socket layer. -**Decision:** Options 1–3 are all ordinary forwards and all supported. -Option 2 is the Phase 1 default, option 1 the fallback, option 3 the general -answer beyond Cuttlefish. - -**Rationale:** Cuttlefish already solves multi-device Bluetooth on one host, -and how it does so says what to forward: all four rootcanal ports derive from -`rootcanal_instance_num` rather than the CVD's own instance number -(`hci 7300+N`, `link 7400+N`, `test 7500+N`, `link_ble 7600+N`), and sharing -one controller between devices is a documented, first-class configuration -that the guest reaches through a TCP connector. Everything host-local about -it is a **port**, which is what this JEP forwards. - -Option 2 is preferred for symmetry: each device keeps its own controller, so -neither exporter's failure removes the other's radio, and rootcanal exposes -`link_port`/`link_ble_port` specifically for joining controllers to each -other. It needs no new code — a `TcpNetwork` child and a forward. Option 1 is -the fallback if federation misbehaves; equally free, but asymmetric, since one -Pod becomes the medium and therefore a single point of failure. One -constraint separates them: `link_port` exists only on standalone rootcanal, -never on netsim, so a device routing its radios through netsim has options 1 -and 3 but not 2. - -Option 3 is the general answer and the only one that works outside -Cuttlefish. Bumble's virtual `Controller` attaches to a link-layer bus that -several controllers can share, and its `RemoteLink` carries that bus over a -relay — the N-way medium DD-6 deferred. Choose it for a non-Cuttlefish -device, for more than two participants, or when the medium itself must be -scripted or fault-injected, which a Python controller does and a C++ -simulator does not. Bumble is already a dependency: -`jumpstarter-driver-bt-peer` passes its `transport` string straight to -`bumble.transport.open_transport`, so any transport moniker works with no -Python change — that is what makes the `requires` side free. The driver only -ever builds a `Device`, a host, so the controller side is new code, and small. - -Options 4 and 5 are rejected. Option 4 invents machinery — a driver -interface, a driver tier and a medium taxonomy — to express what network -drivers plus a direction already express. Option 5 changes the device under -test: a guest-side shim means the stack being exercised is not the one that -ships, invalidating the pairing and handover behavior these tests exist to -verify. - -Option 2 is an instance of a general pattern rather than a Bluetooth special -case. What a bench needs is a **counterparty**: something presenting itself -to the DUT as whatever it expects on the other side of the medium. For -Bluetooth that is a shared controller; for a vehicle bus it is a *restbus*, -simulating the remaining ECUs. Both are ordinary `provides` ports, so the -same machinery serves them (integrating a real restbus is future work). - -One caveat is carried rather than hidden: a Python medium is comfortable for -advertising, pairing and control but an open question for sustained -A2DP-class traffic, which compounds the latency risk already recorded. -Options 1 and 2 keep the medium in native C++ and avoid it. - -**Verified on real devices (2026-09-01 and 2026-09-02).** Options 1 and 2 -were exercised by hand on a single-node kind cluster with two Cuttlefish -exporter Pods — an automotive head unit in one, a phone in the other — first -with a 50-line TCP relay standing in for the forward endpoint, then over -Jumpstarter's own router path with the merged `jumpstarter-driver-bt-peer` -playing the phone. Driven through `adb` alone, every variant produced the -same result: inquiry, SSP numeric comparison showing one passkey on both -screens, a bond, and HFP, A2DP and AVRCP `Connected`, with music streaming -continuously over the link and the phone reconnecting by itself after a -Bluetooth off/on. Boot-to-bond is about two minutes, most of it guest boot. -Which profiles appear at all is decided by the phone *image*, not by the -medium: a plain GSI enables only the LE-audio profile properties, and the -classic set had to be added to the image before anything beyond the bond -connected. None of this changes the decision — option 2 stays the Phase 1 -default on its symmetry — but it turns "nothing to build" into a specific -list of duties. - -*What the ports actually require.* Under option 2 both CVDs run standalone -rootcanal (`--netsim_bt=false`), which binds all four ports on `0.0.0.0`; -the join itself is a runtime `add_remote` command on the *test* channel, so -federation needs two forwarded link ports **plus a control step after the -forward is up** — a post-establish hook, not something a static `forwards[]` -entry expresses. It also joins the LE phys wholesale, so the peer's beacons -appear in local scans. Under option 1 the second CVD starts no controller and -its guest dials the shared HCI port at boot, so the forward must exist -*before* that CVD launches — a pre-launch ordering constraint where option 2 -has a post-launch one — and because netsim binds its HCI port on loopback -only, the forward endpoint must live inside the owning Pod rather than -merely nearby, which is what this design already provides. - -*The peer side is free, and can be the whole phone.* `bt-peer` ran unchanged -as the `requires` end, its `transport` pointed at a forwarded HCI port: the -head unit discovered it, SSP-paired it, and opened AVDTP to its A2DP source -— the claim under *Ports* that the driver's Python is untouched, demonstrated -on merged code. Adding an HFP Audio Gateway alongside that source (a -`bumble.rfcomm` server plus `hfp.AgProtocol` and its SDP record) made the -head unit show both Phone and Media, and a simulated inbound call from the -peer arrived in the head unit's telephony stack as a ringing call. The -cheapest useful bench is therefore one virtual device and a Python process, -and `bt-peer` should grow a `profiles:` config rather than a sibling driver. -This exercises Bumble as a *host*; option 3, Bumble as the *controller*, is -still unverified. - -*What the router costs.* Measured from one Pod against the same endpoint: a -round-trip through a router-carried forward was **0.70 ms** median (p90 -0.91 ms, n=100) against **0.05 ms** direct Pod-to-Pod, while an HCI command -to rootcanal took **~45 ms either way** — the simulator's own scheduling -dominates by two orders of magnitude. Pairing cannot tell the transports -apart, which is what makes DD-4's preference for a direct in-cluster -connection free rather than a gamble. Where that stops being true is what the -*Latency characterization* test is for. - -*Duties this puts on the drivers and the forward endpoint.* - -- **Keep the test channel private, and watch the controller.** A client that - connects to rootcanal's test port and closes before its banner is written - aborts the process (`Check failed: written == size, errno = 32`). - `process_restarter` brings rootcanal back, but the guest's connector took - `SIGPIPE`, is not restarted, and its Bluetooth stays half-up until a `cvd - restart`. So the test channel is never a `provides` port, a forward - endpoint never probes liveness by connect-and-close — state comes from the - splice — and a driver whose controller dies reports the forward `Failed` - and the lease `Degraded` instead of leaving a bench that cannot pair. -- **Assign addresses per member, before the host powers on.** rootcanal names - the *n*-th live device on a model `da:4c:10:de:00:` and reuses numbers, - so two federated CVDs start life with the same BD_ADDR and any attaching - peer collides with an existing one. The rootcanal driver sets an address - per member in its post-establish hook, ahead of power-on, because secure - pairing binds the address into key derivation. -- **Expect identical guest networks.** Every Cuttlefish guest boots the same - address plan behind an identical AP, so anything bridging guests at L2 or - L3 — the Phase 4 frame bridge, a Wi-Fi Direct emulation — must NAT or - re-address. An L4 forward never sees it. -- **Reconnect, and do it in the endpoint.** The verification cluster's - ingress reloads on any unrelated `Ingress` change and drains its workers - 240 s later, which cut the exporter's controller stream and the forwarded - HCI splice within seconds of each other; the exporter re-dialed its own - stream, the forward did not, and the bench stayed `Ready` with the - simulated phone off the air. A driver that opens its transport once at - start cannot notice this, so the `requires` side re-dials and re-splices - transparently and the reconnect reaches drivers that care as an event. -- **One process per exporter identity.** Two processes registered under one - identity split connections between them; the one that does not own the - lease's session fails `RouterService.Stream` with a `KeyError` and closes, - which presents as a flaky forward rather than a misconfiguration. -- **Match versions across a splice.** A forward endpoint built against a - newer network driver than the exporter's runtime accepted connections and - returned EOF at the first byte, silently — an argument for the endpoint - being the exporter's own code, as specified. - -Two notes for whoever writes the tests: a Bumble peer keeps its bonds in -memory unless a keystore is configured, so restarting it invalidates the -DUT's link key and the DUT must forget the bond first; and deleting a `Lease` -object out from under a waiting client leaves that client retrying -`not found` forever — leases are released, not deleted (*Lease state*). +**Decision:** Use option 2 for Phase 1, with option 1 as a fallback. +Option 3 is the proposed extension beyond Cuttlefish; its use as a Cuttlefish +controller remains unverified. + +**Rationale:** Cuttlefish reaches rootcanal through TCP ports derived from +`rootcanal_instance_num`: HCI `7300+N`, link `7400+N`, test `7500+N`, and +BLE link `7600+N`. These provide two attachment choices: + +- Sharing one controller requires the forward before the second CVD boots. + That controller becomes a shared failure point. +- Federating controllers lets each CVD keep its own radio controller. It + requires standalone rootcanal (`--netsim_bt=false`), two forwarded link + ports, and an `add_remote` command on the private test channel after the + forwards are established. Netsim does not expose these link ports. + +Bumble's virtual `Controller` and `RemoteLink` offer a programmable shared +medium. This requires a controller driver; the existing `bt-peer` creates a +host `Device` and can use a forwarded HCI transport without Python changes. +Sustained A2DP performance with a Python controller still needs testing. + +A dedicated bridge-driver interface is unnecessary for these socket +connections. Guest-side shims would change the stack under test. Protocol +setup remains the responsibility of endpoint drivers. + +**Prototype results (2026-09-01–02).** Manual tests used two Cuttlefish Pods +on one kind node with stand-in TCP relays, exercising shared and federated +rootcanal configurations. Tests covered discovery, SSP pairing, HFP, A2DP, +AVRCP, audio streaming, and reconnection after toggling Bluetooth. A later +test connected a CVD to a Bumble `bt-peer` over Jumpstarter's router. Adding +an HFP Audio Gateway to the peer also delivered a simulated incoming call +to the head unit. These tests exercised Bumble as a host, not as a controller. + +The measured router-forward round-trip was 0.70 ms median (p90 0.91 ms, +n=100), versus 0.05 ms direct to the same endpoint. Rootcanal HCI commands +took about 45 ms on either path. Multi-node tests remain required. + +The prototypes identified these implementation requirements: + +- **Controller recovery:** connecting to rootcanal's test port and closing + before its banner is written can abort rootcanal. Its process restart did + not restore the guest connector; recovery required `cvd restart`. Keep + the test channel private, derive forward health from the stream rather + than connect-and-close probes, and report controller loss as `Failed` and + lease `Degraded`. +- **Unique Bluetooth addresses:** rootcanal assigns and reuses addresses + such as `da:4c:10:de:00:`. Federated controllers and attaching peers can + collide. Assign member addresses before host power-on and pairing. +- **Guest addressing:** Cuttlefish guests use identical network address + plans. L2/L3 bridges require NAT or re-addressing; L4 forwards do not. +- **Reconnection:** an ingress reload followed by a 240-second worker drain + cut the prototype's controller and HCI streams. The exporter reconnected + its controller stream but the stand-in forward stayed down. Forward + endpoints must reconnect, update readiness, and notify drivers that need + to restore protocol state. +- **Exporter identity and versions:** duplicate exporter processes split + sessions and caused driver-UUID `KeyError` failures. A separately built + forward endpoint with a mismatched network-driver version returned EOF. + Run one process per identity and keep forwarding in the exporter runtime. +- **Image and bond state:** the tested GSI needed classic Bluetooth profile + properties enabled. Bumble bonds need a keystore to survive peer restarts; + otherwise the DUT must forget the old bond. Release leases through their + lifecycle API; deleting a lease during acquisition left the prototype + client retrying `not found`. ### DD-10: Wi-Fi and projection — what a forward carries @@ -1275,78 +866,40 @@ object out from under a waiting client leaves that client retrying 3. **Forward the projection session at L4** — carry the projection's own TCP connection, over the guest's real Wi-Fi NIC, and simulate no radio. -**Decision:** Option 3 is the standard projection path and a Phase 2 -deliverable: the session is an ordinary `provides`/`requires` pair and needs -nothing this JEP does not already build. Options 1 and 2 remain the target -for **medium** fidelity — association, RSSI, and the Bluetooth→Wi-Fi -handover — with option 2 first where netsim covers it, and option 1 now -understood to need a frame-bridge component on each exporter rather than -the byte forward. That is Phase 4. - -**Rationale:** An earlier draft rejected option 3 as a fidelity failure — -"it verifies that a TCP proxy works" — and kept it only as a diagnostic. -Running it changed that judgment. With the two Pods of the DD-9 experiment, a -vendor phone image and a publicly available receiver projected a live session -across a single forwarded port (the verification below): version negotiation, -the TLS authentication between the phone stack and the receiver, service -discovery, the phone-side first-run flow, and the rendered launcher with -video, audio and input all ran unchanged. That is not a proxy check; it is -the whole projection stack running on two exporters. - -What option 3 does not exercise is precisely bounded: the **handover** — the -credential exchange over Bluetooth and the phone joining the head unit's -Wi-Fi Direct group — and radio-layer failure modes such as RSSI, roaming and -channel loss. Those are what Phase 4 is for. Everything downstream of the -handover runs over the forward, so a lab gets a real projection workload on -day one, in CI, on hardware-free nodes, and Phase 4 arrives as a fidelity -upgrade rather than as the first time anything projects. - -Two facts from the same experiment shape how the medium options are built: - -- **The Cuttlefish Wi-Fi medium is not a byte stream.** The guest's - `virtio_mac80211_hwsim` feeds a per-environment `wmediumd` over a - **vhost-user** Unix socket (shared memory plus fd passing), with an OpenWrt - VM as the access point. `--vhost_user_mac80211_hwsim` shares one medium - between instances on one host only. Across hosts, option 1 is therefore a - *bridge* — a component on each exporter terminating vhost-user locally and - exchanging 802.11 frames with its peer — and the forward is only the pipe - between the two bridges. Hence option 1 is both the general answer and the - one needing new code and datagram semantics (Future Possibilities). -- **The guest's Wi-Fi is already IP-reachable.** Each guest joins its own - OpenWrt AP and reaches the exporter's network through the AP's WAN link; - one host route and one forwarding rule make the guest's Wi-Fi address - reachable in return. So a Cuttlefish driver can expose a guest-side port - two ways — over ADB, which models the USB cable, or at the guest's `wlan0` - address, which models the Wi-Fi link. Both were run with the same result. - They are two `provides` ports, not two mechanisms, and which one a test - forwards is a topology choice DD-13 leaves to the test. - -Between 1 and 2, option 2 is far cheaper when it applies, because it reuses -the same forward machinery as Bluetooth and netsim already owns the frames. -Option 1 is the general answer, because `wmediumd` is where RSSI and -delivery modeling live. Option 1 is also the hardest thing in this JEP and -the most likely to need upstream work — it is scheduled last and its risk -is called out explicitly. - -**Verified against real devices (2026-09-01).** Same two Pods and stand-in -relays as DD-9. The phone was a Cuttlefish exporter running a vendor phone -image — a `user` build, so enabling ADB meant preparing the image rather than -configuring the guest — with the projection app sideloaded and its -developer-mode server started on the port the phone `provides`. The head unit -was a desktop receiver process in the other Pod, a plain TCP client and hence -the `requires` side; direction fell out of DD-13 as expected. The forward ran -first over ADB and then over the guest's Wi-Fi address through its AP. - -The session negotiated its protocol version, completed a TLS handshake (so -the path must be byte-transparent), ran service discovery, walked the -phone-side first-run flow, and rendered the full launcher with maps, media -and telephony live. Video is phone→head-unit and cost about 0.6 MB for three -minutes of a mostly static screen, so an L4 forward is not the bandwidth -constraint. Three operational findings are folded into *Reference drivers for -projection*: the receiver quits on stdin EOF and needs a display; the phone's -projection server does not survive an aborted session and must be restarted; -and the phone would not join Wi-Fi while it held a validated Ethernet -network. +**Decision:** Deliver option 3 in Phase 2. Defer simulated Wi-Fi and the +Bluetooth-to-Wi-Fi handover to Phase 4, using netsim where supported or a +frame bridge for `mac80211_hwsim`/`wmediumd`. + +**Rationale:** An L4 forward exercises projection version negotiation, TLS, +service discovery, video, audio, and input. It does not exercise Bluetooth +credential exchange, Wi-Fi Direct association, RSSI, roaming, or channel +loss. This provides a useful projection test before medium simulation is +available. + +Cuttlefish's `virtio_mac80211_hwsim` connects to `wmediumd` through vhost-user, +which uses shared memory and file-descriptor passing. That connection cannot +be forwarded as a byte stream across hosts. Option 1 needs a bridge on each +exporter to terminate the local interface and exchange 802.11 frames, with +datagram support considered separately. Option 2 can reuse netsim's packet +transport where its Wi-Fi support is sufficient. + +For L4 forwarding, a route and forwarding rule expose the guest's Wi-Fi +address through its OpenWrt AP. The driver can therefore provide the same +projection server through ADB (`projection`) or the guest's Wi-Fi address +(`projection-wifi`). The test selects the path (DD-13). + +**Prototype results (2026-09-01).** A vendor phone image in one Cuttlefish Pod +projected to a desktop receiver in another Pod through a stand-in relay, +first over ADB and then over the guest's Wi-Fi address. The session completed +version negotiation, TLS, service discovery, and the phone's first-run flow, +and displayed the launcher with maps, media, and telephony. A mostly static +screen transferred about 0.6 MB of video in three minutes; this does not +establish a sustained-throughput limit. + +The receiver required a display and open stdin. An aborted session required +a phone-side server restart. The phone joined Wi-Fi only after its validated +Ethernet connection was removed. These requirements belong in the reference +drivers. ### DD-11: Mixed physical/virtual benches — deferred @@ -1360,28 +913,14 @@ network. **Decision:** Option 1 — defer. -**Rationale:** Nothing about the lease plane or the forward mechanism -distinguishes a mixed bench; the obstacle is entirely physical. A real -device's radio is inside the device, and its HCI is not reachable from -outside, so no software path puts a real phone and a simulated head unit on -the same medium. The bridge has to happen in RF, which means new lab hardware -(a USB HCI dongle suffices for Bluetooth; Wi-Fi needs a 5 GHz radio), -physically near the device, shared between whoever is using it. Requiring -that to accept this JEP couples a software design to a hardware procurement, -and homogeneous benches already deliver both headline results — cross-host -physical benches and cross-Pod virtual pairing. - -The path when it returns is option 2, and the analysis is recorded here so it -does not have to be redone. A gateway exporter stays a normal Jumpstarter -exporter with a normal driver, so it schedules, leases, and reports like -everything else, and its adapter is an ordinary `provides` port — no new -mechanism is needed, only the hardware and a decision about how to model a -shared RF resource (see Future Possibilities). - -Worth naming a tempting non-answer: emulating the physical device's peer in -software and never involving RF. That is a useful *test double*, but it is -not a mixed bench — it is a virtual bench with a hand-written model of the -physical device, and it cannot find bugs in the physical device's stack. +**Rationale:** Joining physical and simulated radios requires a nearby hardware +adapter and a way to allocate that shared RF resource. Bluetooth may use a +USB HCI adapter; Wi-Fi needs suitable radio hardware. Requiring this now +would add hardware integration to the lease and forwarding work. + +A future gateway exporter can own the adapter and expose a provided port. +Its allocation model remains open (Future Possibilities). A software model +of a physical peer is useful but does not test the physical device's stack. ### DD-12: Access policy and port validation timing @@ -1397,34 +936,20 @@ physical device, and it cannot find bugs in the physical device's stack. **Decision:** Option 1 for v1; option 2 recorded as a follow-on that depends on JEP-0017; option 3 deferred. -**Rationale:** Per-member evaluation is the least surprising and most secure -default: a multi-member lease can never reach an exporter its client could -not have leased directly, which makes the feature non-escalating by -construction. The lease's effective priority is the minimum across members -and its maximum duration the minimum of the per-member `maximumDuration` -values, so a bench never outlives or outranks its most restricted member. -`status.members[].priority` records each member's own value so the -aggregation is auditable. - -Port validation is bind-time in v1 because of a concrete constraint: lease -selection matches `labels.Set(exporter.Labels)` — **CR metadata labels** — -while reported ports live in `ExporterStatus.Devices[]`. Ports are therefore -visible to the controller *after* binding but not selectable *before* it. The -bind-time check (both named ports exist, directions are complementary, -declared protocols agree) needs no new machinery and catches every -misconfiguration, at the cost of holding the devices while it reports -`Invalid` — which is arguably better for debugging anyway. - -Option 2 is strictly better and worth doing: it would make a bench whose -ports cannot be satisfied report `Unsatisfiable` **without holding -anything**, consistent with DD-1's all-or-nothing property. It requires -reported device information to be reconciled into selectable exporter labels, -which is precisely the mechanism proposed in JEP-0017. Hand-maintained CR -labels are a stopgap but drift from reality, so this JEP does not depend on -them. - -Option 3 is a real requirement but separable, and should be designed once -there is operational experience with what benches people actually build. +**Rationale:** Each member must satisfy the same access policy as an independent +lease request. Lease priority is the minimum member priority, and duration +is bounded by the minimum per-member `maximumDuration`. +`status.members[].priority` preserves the individual values for inspection. + +The existing selector pipeline matches exporter CR metadata labels, while +ports are reported in `ExporterStatus.Devices[]`. V1 therefore validates +ports after selecting and binding exporters. An invalid forward leaves the +members held for inspection until release or expiry. + +Selection-time validation would avoid holding exporters whose ports cannot +satisfy the request. JEP-0017 proposes exposing reported device information +through selectable labels. Bench-level policy and quota are deferred until +there is operational experience with multi-member leases. ### DD-13: Infer direction, never infer topology @@ -1439,186 +964,113 @@ there is operational experience with what benches people actually build. **Decision:** Option 1, with option 3 retained as an explicit form. -**Rationale:** The two kinds of inference look similar and are opposites. - -Direction is a **property of the drivers**. Whether rootcanal is the listening -end is a fact about rootcanal, already in the exporter's report and already -validated. Making the author restate it asks them to know exporter internals -— what DD-6 removed addresses to avoid — and the knowledge goes stale: swap a -`provides` implementation for one that dials, and every manifest naming it as -`from` is silently wrong. Inferring direction is a correctness improvement -that costs nothing, since the controller already checks the reported roles; -option 1 uses that check to *assign* rather than only to *reject*. - -Topology is a **property of the test**. The same phone exporter is a -USB-attached phone when a test forwards `projection` and a wireless one when -it forwards `projection-wifi`; two exporters are a bench in one test and -independent devices in another. Option 2 makes that a property of whichever -drivers happen to be configured on whichever exporters the selectors bound, -which fails in three ways: - -- **Nondeterminism across bindings.** A selector-based member binds to - different exporters on different runs. If those exporters declare different - ports, the bench wires itself differently run to run — the worst property a - reproducible test can have. -- **Ambiguity is not rare.** Two members each exposing several ports turns - matching into a bipartite matching problem. Tractable, but a silently wrong - wiring is worse than an error. -- **It weakens a security property.** This JEP states that a forward is not a - general exporter-to-exporter tunnel, because an exporter can reach only a - peer *a lease it is bound to explicitly names*. Option 2 relaxes "explicitly - names" to "happened to match", and an unintended pairing of two `console` - ports is a data path nobody requested. - -Option 3 stays available as `from`/`to` for wiring that should be pinned -regardless of what the exporters report; the controller then checks the stated -roles against the reported ones instead of assigning them. - -The ergonomic complaint behind option 2 — `member.port:member.port` is -verbose — is better answered in the client: the CLI expands shorthand into an -explicit `spec.forwards[]` entry before submission, so the stored object stays -auditable. Where a lab uses one port name on both sides, `--forward bt` can -expand to it — a convention worth offering, not worth depending on. +**Rationale:** Port direction is already in the exporter report. The +controller can resolve it without requiring the lease author to repeat it. +The test must still choose which ports to connect: forwarding `projection` +and forwarding `projection-wifi` exercise different paths. + +Automatic topology would make connections depend on the ports exposed by +whichever exporters were selected. Multiple possible matches would be +ambiguous and could create data paths the test did not request. + +Explicit `from`/`to` remains available when a test requires a particular +direction; the controller validates it against the report. CLI shorthand may +expand into explicit `spec.forwards[]` entries before submission so the +stored topology remains inspectable. ## Design Details ### Deployment assumptions -Several properties below depend on one deployment invariant, stated here so a -reviewer deploying differently finds out from the document rather than from a -port conflict: - -> **Each exporter owns its own network namespace.** In practice this is -> either a container running exactly one exporter, or a single-exporter edge -> device. - -Both supported shapes satisfy it. For virtual targets, JEP-0016 supplies it -*by construction*: its `cuttlefish.jumpstarter.dev` provisioner renders one -CVD per Pod under a JEP-0014 `ExporterSet`, and a Pod is a network namespace, -so every provisioned exporter gets a private loopback without anyone having -to arrange it. For physical targets an edge device is a single exporter and -the question does not arise. - -Three consequences follow, and they are why this JEP can be as small as it is: - -- **`requires` ports can bind fixed local addresses.** A driver that dials - `127.0.0.1:7300` is dialing *its own* loopback, so two exporters both using - 7300 never collide. This is what makes the zero-driver-changes property - (DD-6) structural rather than coincidental — under a shared namespace it - would be luck that breaks on the second exporter. -- **Control-plane blast radius equals lease scope.** A simulator control API - scoped to "this host" is scoped to one exporter, hence to one lease. This - matters concretely: the netsim driver's `reset` is documented as affecting - every device on its netsim instance, which would cross lease boundaries if - two exporters shared one. -- **DD-4's two transports map onto the two shapes.** Exporter Pods in one - cluster are mutually routable, so the direct path applies and is preferred; - edge devices behind NAT are exactly why the router path is the universal - fallback and why pure peer-to-peer was rejected. The split is not a - user-visible choice: a virtual bench gets the fast path because of where it - runs, and the same lease spec keeps working when one member moves to a - bench on someone's desk. - -A second invariant is narrower, and cost a verification session before it -was noticed: - -> **Each exporter identity is served by exactly one process.** - -Two processes under one identity both register and both look healthy, but -only one owns the lease's session; connections routed to the other die on a -`KeyError` for a driver UUID it has never heard of, with no useful error on -either side. It presents as a flaky forward, not a misconfiguration. A member -exporter runs one replica, and a duplicate registration is worth refusing. - -The exception to watch is host networking. The `jumpstarter-driver-cuttlefish` -container recipe currently documents `--network=host` (so that netsim and -rootcanal are reachable from outside the container), which places every -exporter on that machine in one namespace and forces port-offset schemes such -as `7681 + instance_num`. That arrangement is fine for a hand-managed -single-exporter host or local development, and is **not** a supported basis -for multiple exporters on one machine under this JEP. +Each exporter must own its network namespace and run one process per +exporter identity. + +A virtual exporter can run in its own Pod, as proposed in JEP-0016. A +physical exporter can run on a dedicated edge device or in an isolated +container. This allows `requires` ports to use fixed loopback addresses and +keeps host-scoped simulator operations, such as netsim reset, within one +exporter's lease. + +Multiple host-networked exporters on one machine are unsupported: their +listeners can collide and simulator control operations can cross lease +boundaries. Host networking remains usable for a single-exporter host or +local development. + +Duplicate processes under one identity can both register but hold different +driver sessions, causing routed calls to fail. Each member exporter runs one +replica; duplicate registration must be rejected. + +Network-zone configuration determines direct eligibility. Same-zone peers +attempt direct connections; exporters without a suitable peer route use the +router (DD-4). ### Binding: one pass, one write The existing `reconcileStatusExporterRef` generalizes to `reconcileStatusMembers`, keeping its selection pipeline intact per member: -```text -for each member in spec.members: - approved = policy-approved exporters matching member.selector - online = filter offline - unleased = filter out exporters claimed by other active leases - AND exporters already picked earlier in this same pass - ready = filter out exporters still cleaning up a prior lease - candidate[member] = best(ready) # in memory only, nothing written yet - -if any required member has no candidate: - set Pending/Unsatisfiable with the failing role named - write NO member claims - requeue - -else: - status.members = candidates # all of them - status.priority = min(member priorities) - single Status().Update() # atomic +```{mermaid} +flowchart TD + select["Select policy-approved exporters
matching the member selector"] + filter["Exclude offline exporters, active claims,
earlier picks, and exporters still cleaning up"] + candidate["Keep the best candidate in memory
Write no claims yet"] + more{"More members?"} + complete{"Every required member
has a candidate?"} + pending["Set Pending / Unsatisfiable
Name the failing role; write no claims"] + requeue["Requeue"] + bind["Set status.members to all candidates
Set priority to the minimum member priority"] + commit["Commit one atomic Status().Update()"] + + select --> filter --> candidate --> more + more -->|"Yes: next member"| select + more -->|No| complete + complete -->|No| pending --> requeue + complete -->|Yes| bind --> commit ``` -Two properties follow, and they are the reason for DD-1: +Candidates remain in memory until every required member resolves. The +selection pass excludes exporters already assigned to another member, so +two roles with the same selector receive distinct exporters. -- **No partial holds.** Candidates live in memory until every required member - resolves. A lease that cannot complete writes no claims at all, so it holds - nothing while it waits and there is nothing to release on timeout. -- **No self-collision.** Because the pass excludes exporters already chosen - for an earlier member, two roles can share a selector without both - resolving to the same exporter — which is what a `phone` + `phone` - two-handset bench needs. +The scalar path uses the same selection code with a synthetic member and +writes the result to `status.exporterRef` (DD-2). -The single-exporter path is the same code with one member: `spec.selector` -normalizes into a synthetic member at admission and the result is written to -`status.exporterRef` instead of `status.members` (DD-2). One selection -implementation, not two. - -**What is unchanged, and still imperfect.** Exclusivity is still a read-scan -of other leases' claims against a possibly-stale cache, so two leases can -race for one exporter and both write. Pre-existing, neither improved nor -worsened here; the loser is detected on a later reconcile and re-bound. +The existing cross-lease race remains: reconcilers can read stale claims +and commit conflicting selections to different lease objects. A later +reconcile detects the conflict and rebinds; a global scheduler is separate +work (DD-1). ### Lease state -```text - ┌─────────┐ - │ Pending │◄────────────┐ requeue: some member unavailable - └────┬────┘ │ (nothing held) - │ │ - ┌──────┴───────┐ │ - ▼ ▼ │ -┌──────────────┐ ┌────────────────────┐ -│Unsatisfiable │ │ all members bound │──┘ -└──────────────┘ │ (one status write) │ - └─────────┬──────────┘ - │ forwards validated + requested - ▼ - ┌────────────┐ forward failure ┌──────────┐ - │ ForwardsUp │──────────────────►│ Degraded │ - └─────┬──────┘ └──────────┘ - │ all forwards connected - ▼ - ┌────────────┐ member lost ┌──────────┐ - │ Ready │───────────────►│ Degraded │ - └─────┬──────┘ └──────────┘ - │ release / expiry / spec.release - ▼ - ┌────────────┐ - │ Ended │ - └────────────┘ +```{mermaid} +flowchart TD + pending["Pending"] + available{"All required members available?"} + unsatisfiable["Unsatisfiable
No exporters held"] + bound["All member claims committed
in one status write"] + forwards["ForwardsUp"] + ready["Ready"] + degraded["Degraded"] + ended["Ended"] + + pending --> available + available -->|No| unsatisfiable + unsatisfiable -->|Requeue| pending + available -->|Yes| bound + bound -->|"Forwards validated and requested"| forwards + forwards -->|"All forwards connected"| ready + forwards -->|"Forward failure"| degraded + ready -->|"Member lost or forward failure"| degraded + ready -->|"Release or expiry"| ended + degraded -->|"Release or expiry"| ended ``` Conditions reuse the existing `LeaseConditionType` values — `Pending`, `Ready`, `Unsatisfiable`, `Invalid` — with `ForwardsReady` and `Degraded` added. `Ready` for a multi-member lease requires all required members bound *and* all declared forwards connected. Expiry, `status.ended`, the -`jumpstarter.dev/lease-ended` label, and `spec.release` behave exactly as for -a single-exporter lease, because it is the same object and the same code. +`jumpstarter.dev/lease-ended` label, and `spec.release` retain their existing +behavior. ### Forward validation and establishment @@ -1636,58 +1088,32 @@ exists belongs to a bound exporter, not to a selector (DD-12). For each roles must match what the exporters report. 4. If both ports declare `protocol`, the values are equal (DD-8). -A failure sets `Invalid` with the offending forward and reason named, and -leaves the members bound so the user can inspect the bench rather than -staring at a lease that silently refuses to bind. - -Establishment then reuses the existing port-forward primitives: - -1. The controller instructs both exporters over their existing `Listen` - streams. -2. Each calls `DialPeer(lease, forward, member)`. -3. The controller mints two tokens with an identical `stream` claim derived - from `(lease UID, forward name)` — a UUIDv5 in a fixed namespace, so the - value is stable across reconciles and both ends compute the same - rendezvous without coordination. `aud: jumpstarter router`, - `sub: jumpstarter exporter`, expiry clamped to the lease's end time. -4. Both call `RouterService.Stream`. -5. The `provides` side dials its local service and splices with - `forward_stream`. The `requires` side opens a `TemporaryTcpListener` on - its declared `listen` address and splices each accepted connection. -6. **Fast path**: with `prefer_direct` set, the `requires` side dials - `peer_endpoint` first and gives it a short bounded timeout (a couple of - round-trips, not seconds), falling back to the already-minted router - stream if it does not complete; the router dial is not raced, so an - eligible pair does not pay for two connections on every establishment. - The direct listener authenticates `peer_token`, so a direct forward is not - a weaker trust boundary. `mode: direct` fails rather than falling back; - `mode: router` never attempts it. Which transport a forward ended up on is - in `LeaseForwardStatus.Mode`, and a fallback records why. - -**Direct eligibility.** The controller offers `prefer_direct` when both bound -members report the same non-empty `NetworkZone` and the `provides` side -reports a `PeerEndpoint` (DD-4). Zone is deliberately opaque and comes from -deployment configuration — one value per cluster network, which JEP-0016's -provisioner sets on every Pod it renders — so an edge exporter reporting none -is never a direct candidate and keeps the router path with no per-lease -setup. Reachability stays a statement someone made about the deployment -rather than something the controller infers from addresses it cannot test, -the same reasoning DD-13 applies to topology. A pair that claims a zone but -cannot connect is not a user-visible failure: the dial times out and the -router stream, already minted, carries the forward. - -**Reconnection belongs to the endpoint.** A forward outlives any single -stream: an ingress reload, a router restart or a network blip cuts the peer -stream, and in a cluster whose proxy reloads on unrelated `Ingress` changes -that is routine rather than exceptional (DD-9). The `requires` side therefore -re-dials with backoff and re-splices without tearing down its listener, and -the first attempt right after a cut is expected to fail and be retried. -Surfacing the reset to the driver instead does not work: these drivers open -their transport once at start, so after a silent cut the bench still reports -`Ready` while the device is off the air. Drivers that must know still learn -of it — a reconnect is an event on the forward (*Observability*), which is -how DD-10's projection server gets its mandatory restart. Forward state comes -from the splice, never from probing the far end. +A failure sets `Invalid`, names the forward and reason, and leaves members +bound for inspection. + +Setup follows *How a forward comes up*. The controller derives the router +`stream` claim from `(lease UID, forward name)` using UUIDv5 in a fixed +namespace, keeping it stable across reconciles. Both tokens use +`aud: jumpstarter router`, `sub: jumpstarter exporter`, and an expiry bounded +by the lease end time. + +The direct attempt has a short, bounded timeout and is not raced with a +router connection. The peer listener authenticates `peer_token`. `Direct` +fails without fallback; `Router` skips the direct attempt. Status records the +transport used and the reason for any fallback. + +**Direct eligibility.** Both members must report the same non-empty +`NetworkZone`, and the `provides` side must report a `PeerEndpoint`. The zone +is an opaque value supplied by deployment configuration, such as one value +per cluster network. The controller does not infer reachability from IP +addresses. An unreachable peer falls back to the router in `Auto` mode. + +**Reconnection.** The `requires` endpoint keeps its listener open and +reconnects the peer stream with backoff after a network interruption, router +restart, or ingress reload. Forward state comes from the stream, without +probing the service. Reconnect events notify drivers that must restore +protocol state, such as restarting a projection server. Protocol-specific +recovery remains necessary where a new stream cannot preserve the session. **Failure modes and handling:** @@ -1697,52 +1123,37 @@ from the splice, never from probing the far end. | Both endpoints `provides`, or both `requires` | Lease `Invalid`; direction cannot resolve (DD-6, DD-13) | | Explicit `from`/`to` contradicts the reported directions | Lease `Invalid` naming the forward and the reported roles | | Declared protocols disagree | Lease `Invalid` (DD-8) | -| Protocols differ but neither declared | Connects, fails at first byte — accepted (DD-8) | +| Protocols differ and at least one tag is absent | Validation passes; protocol errors may occur when data is exchanged (DD-8) | | Forward references an undeclared member | Rejected by CEL at admission; lease never created | | `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | | Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | -| Ingress/proxy reload cuts the peer stream | `requires` side re-dials and re-splices transparently; the first attempt after the cut is expected to fail. Observed in verification, and invisible to drivers that dial once | -| Direct dial fails or times out | Fall back to the already-minted router stream; recorded as a metric, and conspicuous for a same-zone pair that should have connected | +| Ingress/proxy reload cuts the peer stream | Reconnect with backoff and notify endpoint drivers | +| Direct dial fails or times out | In `Auto`, fall back to the router and record the reason; in `Direct`, fail | | A member's exporter disappears | Peer's stream resets; lease `Degraded` naming the role (DD-3) | | Client releases the lease | Forwards torn down first, then the lease ends normally | ### Reference drivers for projection -Nothing in the projection bench needs new forward machinery, but the -verification (DD-10) showed that the two reference drivers own real work, -of the same kind as DD-9's rootcanal control hook: - -- **`cuttlefish` (phone).** A retail phone image is a `user` build, so the - image the driver boots must be prepared rather than configured: ADB enabled - and the exporter's key installed, and the classic Bluetooth profile - properties present, since a plain GSI ships only the LE-audio set (DD-9). - Both are per-image steps JEP-0016's provisioner can run once, not per-lease - actions; recovery is `cvd restart`, which keeps userdata where `cvd rm` - does not. For `projection-wifi` the driver brings the guest onto the - instance's AP, routes to it, and cuts the guest's Ethernet first, because - Android will not join Wi-Fi while it holds a validated Ethernet default. - The projection server is started on request and **restarted whenever the - forward resets**, because a projection session does not survive an abort — - so a reconnect must reach the driver as an event, not be hidden in the - splice. -- **`projection-rx` (head unit).** Runs a receiver as a process: a display, - a dummy audio device, and its console held open, since receivers typically - exit on stdin EOF and use that console as their stimulus API (key presses, - day/night, microphone). Screenshots come from the display. The receiver - dials the instant it starts, so the driver starts it only on a client call, - after the forward is Ready — the same ordering rule `bt-peer` follows. -- **`bt-peer` (phone).** DD-9's verification showed the driver is one - config away from being a phone rather than an audio source: with an HFP - Audio Gateway alongside its A2DP source it satisfies both of a head unit's - phone-facing profiles, and can ring it. That belongs in the driver as a - `profiles:` list, with the bond keystore persisted for the life of the - lease so a peer restart does not strand the DUT's link key. - -The `cuttlefish` and `projection-rx` drivers are the reference -`provides`/`requires` pair for projection. What they must not do is know -about each other: the phone driver exposes -ports and the head unit driver dials `127.0.0.1:5277`, and the lease is the -only place the two are joined. +The reference drivers handle device setup and protocol recovery: + +- **`cuttlefish` (phone).** Prepare the image with ADB enabled, the exporter + key installed, and the Bluetooth profiles needed by the test. These are + per-image preparation steps. Use `cvd restart` for recovery that retains + userdata. For `projection-wifi`, disable the guest's Ethernet connection, + join the instance's AP, and configure the route to the guest. Start the + projection server on request and restart it on forward-reset events. +- **`projection-rx` (head unit).** Run the receiver with a display, dummy + audio device, and open console for input commands. Capture screenshots + from the display. Start the receiver on a client call after the forward + listener is available, since it dials immediately on startup. + +- **`bt-peer` (phone).** Add a `profiles:` list to configure HFP Audio + Gateway alongside A2DP, and persist the bond keystore for the lease lifetime + so restarting the peer does not invalidate the DUT's link key. + +The drivers configure only local endpoints. The phone exposes named ports; +`projection-rx` dials its local listener, such as `127.0.0.1:5277`. The lease +specifies the connection between them. ### Concurrency and ordering @@ -1752,77 +1163,52 @@ computation followed by one write. Forward splicing on the exporter side runs in the existing per-driver task group, so a stalled forward cannot block driver calls on other children. -Ordering between forwards and drivers is safe by construction: forwards come -up before `Ready`, and a `requires`-side driver only dials when the client -invokes it — `bt-peer` connects its transport on `start()`, well after the -lease is Ready. - -Because binding is a single status write, the read-your-writes hazard a -two-object design would face does not arise. +For client-started drivers such as `bt-peer`, the listener must be available +before `start()` dials it. Shared-rootcanal guests need the forward before +boot, while federated rootcanals need a join after establishment (DD-9). +Provisioner ordering remains an unresolved question. ### Security -- **No privilege escalation by construction** (DD-12): every member is - evaluated against `ExporterAccessPolicy` exactly as a direct request would - be, so a multi-member lease reaches only exporters its client already could. -- **Forward authorization is lease-scoped**: `DialPeer` verifies the calling - exporter is currently bound to the named member of the named lease and that - the named forward lists it. Tokens expire with the lease. -- **A forward is a data path between two devices under one client's control**, - not a general exporter-to-exporter tunnel: an exporter can reach only a peer - a lease it is bound to explicitly names, and only through the named port. -- **Ports are opt-in surface.** A driver that declares no ports cannot be - forwarded to or from. A `requires` port is the only new inbound socket, it - binds an address the exporter chose, and it exists only while the lease - holds it. -- **The fast path is authenticated** — a direct peer connection presents - `peer_token`; it is not trusted for being on the same network. Exporters - supporting it open a listener, disabled by default and enabled per exporter - configuration. Preferring it in-cluster (DD-4) does not widen what is - exposed: the peer listener is one authenticated port per exporter, distinct - from any device port, so a cluster can keep a `NetworkPolicy` that blocks - Pod-to-Pod access to simulator ports — which are *not* access-controlled — - while allowing the peer port between exporters. Shipping that policy with - the Pod is JEP-0016's job, and preferring direct makes it load-bearing - rather than advisory. -- **`members` is immutable after creation**, so a bound lease cannot be - widened beyond what it was authorized for. -- **Physical RF is not access-controlled.** Two physical devices pairing in - a shared lab space are audible to anything in range; labs must treat RF - proximity as a trust boundary. -- **Simulator control ports are not access-controlled either.** Standalone - rootcanal binds its HCI, link and test-channel ports on `0.0.0.0` and - accepts any client; the test channel can re-address devices, join - models, and — by accident — crash the controller (DD-9). A - driver exposes only the HCI and link ports as `provides`, never the test - channel, and the exporter's network policy is what limits who can reach - them; JEP-0016 should ship that policy with the Pod. +- **Access policy:** evaluate each member against `ExporterAccessPolicy` as + for a direct request (DD-12). +- **Forward authorization:** `DialPeer` verifies that the caller is the + exporter bound to the named member and that the forward includes it. + Tokens expire with the lease. Access is limited to the explicitly named + peer and port. +- **Port exposure:** only declared ports can participate in forwards. A + `requires` listener uses an exporter-configured address and exists only + while leased. +- **Direct authentication:** the optional peer listener is disabled by + default and requires `peer_token`. It is separate from device ports. + Network policies should allow the authenticated peer port while blocking + peer access to unauthenticated simulator ports. JEP-0016 is expected to + supply this policy with exporter Pods. +- **Membership:** `members` is immutable after creation. +- **Physical RF:** devices in a shared lab are audible to others in range; + lease authorization does not isolate radio traffic. +- **Simulator control:** standalone rootcanal listens on `0.0.0.0` without + client authentication. Keep its test channel private because it can + re-address devices, join controllers, and trigger the crash described in + DD-9. Drivers may expose HCI and link ports through authorized forwards; + network policy must block direct access to the underlying ports. ### Observability -JEP-0013 telemetry gains `lease.member` as a span attribute wherever -`lease.name` already appears, so per-role activity is separable within one -lease, plus per-forward counters (bytes each direction, reconnects, mode -actually used, direct-dial fallback rate). Because `Auto` now prefers direct -for same-zone pairs (DD-4), the fallback rate is a health signal rather than -a curiosity: an in-cluster bench silently running over the router means the -peer listener, the zone configuration or a `NetworkPolicy` is wrong, and it -should be visible as such. Reconnects are also an *event* on -the forward, not only a counter: an endpoint driver that must re-establish -protocol state after a cut (DD-10's head unit server) subscribes to it, and -a bench whose forward is re-dialing is visibly degraded rather than quietly -deaf. Time-to-bench (lease create → `Ready`) is the headline metric; it is -the existing lease-acquisition metric extended to record the member count. - -For simulated media there is a stronger signal available than byte counters. -`jumpstarter-driver-netsim` exposes netsim's pcap capture (start, stop, -download) over its REST control API, so a bench can capture the *over-the-air* -traffic of a pairing and attach it to the run — the interaction itself, not -just the fact that bytes crossed a forward. Because the control API is scoped -to one exporter's namespace, that capture is scoped to the lease. This is the -clearest illustration of the control-plane / data-plane split described under -*Attaching media*: the netsim driver controls and observes the medium, while -a forward carries it. +JEP-0013 telemetry adds `lease.member` alongside `lease.name` and records +member count on the lease-acquisition metric (creation to `Ready`). +Per-forward telemetry records bytes in each direction, reconnect count, +selected transport, and direct-dial fallback rate. Same-zone fallback can +indicate an unavailable peer listener, incorrect zone configuration, or a +blocking `NetworkPolicy`. + +Reconnects also emit events for drivers that must restore protocol state, +such as the phone's projection server. Forward status exposes reconnection +and degradation. + +Where netsim supplies the simulated medium, `jumpstarter-driver-netsim` can +start, stop, and download pcap captures through its REST control API. These +captures can be attached to test results alongside forward metrics. ## Test Plan @@ -1877,35 +1263,24 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): ### Hardware-in-the-Loop Tests -- **Virtual ↔ virtual**: two `jumpstarter-driver-cuttlefish` exporters in - separate Pods (JEP-0016 `ExporterSet`), joined by a forwarded rootcanal - port (DD-9). Assert BT discovery and pairing, and — Phase 4 — Wi-Fi - association, with `jumpstarter-driver-netsim` supplying a pcap of what - actually crossed the air. Runs in CI on KVM-capable nodes with no lab - hardware: the headline result, on every merge once it exists. -- **Virtual ↔ peer**: a `cuttlefish` exporter and a `bt-peer` exporter joined - by `bt=headunit.rootcanal:phone.controller`. Assert the CVD discovers and - pairs the peer and the driver reports `avdtp_connected`, and — with the - peer's HFP AG enabled — that a ringing call from it reaches the head unit's - telephony stack, covering the phone-facing profiles without a phone. The - smallest bench, no second guest to boot; DD-9's verification ran it by hand - over the router. Cheapest CI tier, every merge. -- **Virtual projection**: a phone CVD and a `projection-rx` exporter in - separate Pods joined over `projection-wifi` (DD-10). Assert the receiver - reaches the launcher and a screenshot matches. Same CI tier; the two - compose into one bench once Phases 1 and 2 are green. -- **Physical ↔ physical**: a physical phone and head unit on two exporters - **on different lab hosts** — the allocation the surveyed frameworks do not - make. Requires lab hardware; runs on a labeled runner. -- **Latency characterization**: HCI round-trip through a router forward, a - direct forward and host-local rootcanal, reported as a distribution, which - decides whether A2DP-class workloads are in scope for router mode. By hand - the simulator dominated on a single node (DD-9); the test looks for where - that inverts — cross-node, sustained A2DP, physical controllers. -- **Forward resilience**: cut a live bench's peer stream (restart the router, - or reload the ingress in front of it) and assert the forward re-establishes, - the reconnect is counted and emitted, and a driver that dialed once at start - is still talking to the far end. +- **Virtual devices:** two Cuttlefish exporters in separate Pods pair over + forwarded rootcanal ports. Phase 4 adds Wi-Fi association and netsim pcap + capture where supported. Run on KVM-capable CI nodes. +- **Virtual device and peer:** connect a Cuttlefish exporter to `bt-peer`. + Assert pairing and `avdtp_connected`; with HFP Audio Gateway enabled, + assert that an incoming call reaches the head unit's telephony stack. + This test needs only one guest and should run on each merge. +- **Virtual projection:** connect a phone CVD and `projection-rx` in separate + Pods through `projection-wifi`. Assert that the receiver reaches the + launcher and matches an expected screenshot. +- **Physical devices:** run a phone and head unit on exporters on different + lab hosts, using a labeled runner with the required hardware. +- **Latency:** publish HCI round-trip distributions for router, direct, and + host-local paths. Include cross-node, sustained A2DP, and physical-controller + tests to establish supported workloads. +- **Forward resilience:** cut a live stream through a router restart or + ingress reload. Assert reconnection, metrics and events, and recovery of + communication for a driver that opened its transport at startup. ### Manual Verification @@ -1951,13 +1326,12 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - [ ] Direct fast path is authenticated, falls back automatically, and is observable (mode + fallback-rate metrics) - [ ] `Auto` resolves to a direct peer connection for two same-zone - in-cluster exporters, and the fallback to the router is a measurable - event rather than the silent normal case + in-cluster exporters with peer listeners; router fallback is recorded - [ ] `bt-peer` participates as a `requires` endpoint with **no Python changes** — exporter configuration only, relying on its existing `open_transport(self.transport)` passthrough -- [ ] Phase 1 is achieved with **no new driver code** — a `TcpNetwork` child - on the rootcanal port plus a forward +- [ ] Phase 1 reuses network drivers for byte forwarding, with rootcanal + join, address assignment, and recovery handled by endpoint drivers (DD-9) - [ ] A Bumble-based shared controller exists as a driver exposing a `provides` port, for benches outside Cuttlefish and for N-way media - [ ] A forward survives a cut peer stream: the endpoint re-dials and @@ -1965,52 +1339,37 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): emitted as an event, and a driver that dialed once at start keeps working - [ ] A second exporter process registering under a live identity is - detected and refused rather than silently splitting connections + detected and refused - [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` integration test **Topologies** (each a phase gate, in order) -- [ ] **Phase 1 — Virtual ↔ virtual Bluetooth**: two CVDs in separate Pods - on separate nodes complete BR/EDR discovery and pairing over a - forwarded rootcanal port (link-layer federation, or a shared HCI - instance), in CI. *Demonstrated by hand for both variants on - 2026-09-01 with two Pods on one node and a stand-in relay (DD-9); - on 2026-09-02 the router path itself carried a bench — a CVD's - rootcanal to a `bt-peer` Pod through `RouterService.Stream`, pairing - and profiles included. CVD ↔ CVD over the router and the multi-node - case remain.* -- [ ] **Phase 2 — Virtual projection**: a phone CVD and a `projection-rx` - head unit in separate Pods complete a projection session to the - launcher over one forwarded port, in CI, with no lab hardware. - *Demonstrated by hand on 2026-09-01 over both `projection` and - `projection-wifi` with a stand-in relay (DD-10); the drivers, the router - path and CI remain.* -- [ ] **Phase 3 — Physical ↔ physical across hosts**: a phone and head unit - on exporters on different lab hosts complete a phone projection - session -- [ ] **Phase 4 — Virtual Wi-Fi medium**: two CVDs in separate Pods - associate over a bridged `mac80211_hwsim`/`wmediumd` medium or a - shared netsim 802.11 chip, and a projection session completes the - Bluetooth → Wi-Fi handover end to end -- [ ] Measured HCI round-trip latency through a router forward is published, - with a documented statement of which workloads it does and does not - support. *First data point, 2026-09-02: 0.65 ms of router against a - ~45 ms rootcanal command round-trip, single node (DD-9); - cross-node and A2DP-class throughput remain.* +- [ ] **Phase 1 — Virtual Bluetooth:** two CVDs in Pods on separate nodes + complete BR/EDR discovery and pairing through federated rootcanals or + a shared HCI instance, in CI +- [ ] **Phase 2 — Virtual projection:** a phone CVD and `projection-rx` in + separate Pods reach the projection launcher over a forwarded port, + in CI without lab hardware +- [ ] **Phase 3 — Physical devices across hosts:** a phone and head unit on + exporters on different lab hosts complete a projection session +- [ ] **Phase 4 — Virtual Wi-Fi medium:** two CVDs in separate Pods associate + through bridged `mac80211_hwsim`/`wmediumd` or a shared netsim 802.11 + chip and complete the Bluetooth-to-Wi-Fi projection handover +- [ ] Publish router-forward HCI latency measurements and supported workload + limits, including cross-node and sustained A2DP tests + +DD-9 and DD-10 record the manual prototype results. Automated CI, CVD-to-CVD +forwarding over the router, and multi-node verification remain outstanding. ## Graduation Criteria ### Experimental `members`, `forwards`, and port reporting ship behind a controller feature -gate, with Phases 1–3 complete. Signals sought: do real benches stay at two -members or grow; how often does the direct path actually apply, and how -often does a same-zone pair fall back; does anyone hit -`MaxItems=8`; how often do `listen` collisions occur on physical hosts; does -the nil-`status.exporterRef` convention (DD-2) surprise any consumer; is -bind-time port validation (DD-12) painful enough to justify accelerating the -JEP-0017 dependency. +gate after Phases 1–3 are complete. Collect feedback on member counts, +direct-connection and fallback rates, the eight-member limit, listener +collisions, scalar-field compatibility, and bind-time port validation. ### Stable @@ -2024,8 +1383,8 @@ JEP-0017 dependency. ## Backward Compatibility -This proposal is **additive**, and deliberately concentrates compatibility -risk in one field, handled by DD-2. +The schema and protocol changes are additive. DD-2 defines compatibility +for `status.exporterRef`. - **CRD**: `Lease` gains two optional spec lists and two optional status lists; `ExporterStatus.Devices[]` gains an optional `ports` list. No @@ -2039,18 +1398,18 @@ risk in one field, handled by DD-2. only to *support* benches. - **Driver report and drivers**: `ports` is a new optional repeated field, so an exporter built before this JEP reports none and is treated as - non-forwardable — the correct answer, with no version negotiation or - migration. Ports are declared in exporter configuration, and the reference - `requires` endpoint (`bt-peer`) needs no Python changes at all. -- **Protocol**: three new fields on existing messages and one new RPC. + unable to participate in forwards. Ports are declared in exporter + configuration; `bt-peer` needs no Python changes to use a forwarded HCI + endpoint. +- **Protocol**: new fields on existing messages and one new RPC. Unknown fields are ignored by proto3, so an N-1 client talks to an N controller unchanged. An N client requesting `members` from an N-1 controller has them silently dropped — so the client probes for `DialPeer` (or a controller version) and fails with a clear message rather than acquiring a one-device lease it will misuse. - **Operator upgrade**: a CRD schema addition, a standard bundle bump with no - conversion webhook. Rolling back removes multi-member leases; single-exporter - leases are unaffected, because they are ordinary leases in every respect. + conversion webhook. Multi-member leases must be removed before rollback; + single-exporter leases remain compatible. - **Coexistence**: single- and multi-member leases share one exporter pool, one scheduler, and one selection implementation. @@ -2058,345 +1417,177 @@ risk in one field, handled by DD-2. ### Positive -- Multi-device testing becomes a first-class concept, with atomic - acquisition, shared lifetime and no deadlock — and the atomicity is - structural, not engineered: all of a lease's claims are one object written - once, so there is no partial-hold state, no acquisition timeout, no - release-and-retry and no gang scheduler. -- **A bench can span hosts** — the allocation the frameworks surveyed here - decline to make, and the usual reason multi-device testing stays on one lab - machine. -- **The data plane is existing code**: `TemporaryTcpListener` + - `forward_stream` with a router peer stream substituted for a client stream. - `router.proto` is untouched and no driver changes. -- A bench is a lease, so `jmp create lease`, expiry, `spec.release`, access - policy, telemetry and `kubectl get leases` apply unchanged — no second - resource, RBAC surface or lifecycle. Ports are discoverable, so a forward - can be built from `jmp get exporter` output rather than tribal knowledge. -- Phone projection and Bluetooth pairing become expressible for any protocol - and any head unit OS, and in virtual-to-virtual form expressible *in CI - without a lab* — cross-host virtual device pairing reduces to forwarding a - socket. -- Nothing here is Android-specific: CAN cross-connects, serial cross-overs - and SOME/IP peer benches are all ports and forwards. The exporter = DUT - invariant survives, and JEP-0016 DD-8's option 1 becomes available. +- One lease acquires all required members in one status write and gives them + a shared lifetime. +- Exporters can run on different hosts while retaining role-based access, + lease policy, and telemetry. +- Forwards reuse existing stream primitives and `RouterService`; named ports + are discoverable through exporter reports. +- The port model supports Bluetooth, projection, CAN, and serial connections + without adding protocol-specific logic to the controller. ### Negative -- **`LeaseSpec` now has two shapes.** Scalar and members forms are mutually - exclusive, enforced by CEL, on the most heavily validated object in the API. - Every future change to lease semantics must reason about both. -- **`status.exporterRef` becomes conditional** — set only for single-exporter - leases. A documented, fail-safe convention (DD-2), but a subtlety every new - consumer must learn. -- **Ports are a third place to configure things**, alongside driver config and - labels, and a `requires` port hard-codes a local address that can collide. -- **Protocol mismatch can fail late** when neither port declares `protocol` - (DD-8) — a deliberate retreat from admission-time validation. -- **Port validation happens after binding** (DD-12), so an unsatisfiable - forward holds devices while reporting `Invalid`, until JEP-0017 makes ports - selectable. -- **No per-member early release.** A client finished with one device cannot - hand it back without ending the lease. -- **A new inbound socket on exporters** — the `requires` listener, plus the - optional direct fast-path listener. -- **Latency is a first-class risk**, not a footnote, and some timing-sensitive - protocols may not work in router mode. -- **Phase 4 depends on upstream behavior** we do not control. +- Consumers must handle scalar and member-list lease forms. +- Port declarations add configuration, and fixed listeners depend on network + namespace isolation. +- Bind-time port validation holds devices even when a forward is invalid. + Omitted protocol tags allow compatibility errors to surface at runtime. +- Individual members cannot be released early. +- Forwarding adds local listeners and, optionally, an authenticated peer + listener to exporters. +- Timing-sensitive protocols and Wi-Fi simulation require further testing + and may need upstream changes. ### Risks -- **Wi-Fi frame forwarding may not be viable over the router.** `wmediumd` - assumes medium-like timing; head-of-line blocking on a TCP substrate may - make association flaky or impossible except on the direct fast path. Its - transport is also vhost-user (shared memory), so a cross-host medium needs - a frame bridge first, not just a forward (DD-10). - Mitigation: Phase 4 is last, may conclude "direct mode only", and the - datagram work in Future Possibilities is the escalation path; Phase 2 - already delivers a real projection workload without it. -- **The projection bench depends on vendor artifacts this project cannot - ship** — the phone image, the projection app and the receiver, one of which - is not distributed outside an app store. A lab must supply them, and Phase - 2 CI needs an artifact path that does not redistribute them. Mitigation: - the drivers take artifact locations as configuration, and the Bluetooth - phase carries the CI headline on freely available images. -- **Bluetooth timing may be tighter than measured** — pairing may work while - A2DP streaming does not. Mitigation: latency characterization is an - acceptance criterion whose answer is published, not assumed. -- **The simulators are less robust than a lab needs.** rootcanal aborts when - a test-channel client disconnects early, its restart does not reattach the - guest, and a crash on one exporter strands every member's controller - (DD-9). Mitigation: the driver keeps the test channel private, never - probes, watches its controller and reports `Degraded`, and owns the restart - as a recovery action; the abort is an upstream fix worth contributing. -- **The cluster data path is less stable than a bench assumes.** Long-lived - streams do not survive ordinary cluster events: an nginx ingress reloads - on any `Ingress` change and drains its workers on a timer, cutting every - gRPC stream through it — measured on the verification cluster as a reload - at 04:00:13 and the exporter's controller stream cut at 04:04:13, with - the forwarded HCI splice going down 40 s later (DD-9). A - multi-hour bench will meet this repeatedly. Mitigation: reconnection is the - forward endpoint's responsibility, reconnects are observable, the HIL suite - includes a bench that survives a deliberate stream cut, and an in-cluster - bench avoids the machinery altogether (DD-4). A related hazard is silent - version skew across a splice — an argument for the endpoint being the - exporter's own code, as specified. -- **The namespace invariant may be violated in the field.** Fixed `listen` - addresses and lease-scoped control-plane blast radius both assume one - exporter per network namespace. A host-networked deployment silently - breaks both — colliding ports, and simulator resets that reach another - lease's devices. Mitigation: bind failures are reported as `Invalid` - naming the port and address, and the assumption is stated explicitly; - JEP-0016 removes the question entirely for provisioned virtual targets. -- **Multi-member leases amplify the pre-existing binding race** — N chances - to lose a race, so re-bind churn grows with bench size. Mitigation: - `MaxItems=8` bounds it; the real fix is the global scheduler the code - already has a `TODO` for. -- **The scalar/list convention may leak** — some consumer may assume a bound - lease always has `status.exporterRef`. Mitigation: fail-safe by design, and - the compatibility matrix covers known consumers. -- **netsim and `vhost_user_mac80211_hwsim` are internal-ish interfaces** that - upstream may change. Mitigation: attach at documented seams, pin the - runtime image, treat divergence as a contribution opportunity. -- **Bench sprawl.** Mitigation: per-member policy limits reach, `MaxItems=8` - bounds size, bench-level quota is an explicit Future Possibility. +- **Wi-Fi transport:** vhost-user requires a frame bridge, and TCP + head-of-line blocking may prevent reliable medium simulation. Phase 4 may + require direct mode or separate datagram support (DD-10). +- **Projection artifacts:** the phone image, app, and receiver must be + supplied by the lab. Phase 2 CI needs an artifact source that does not + require project redistribution. Drivers accept artifact locations as + configuration. +- **Bluetooth latency:** single-node pairing results do not establish + cross-node or sustained A2DP performance. Publish workload-specific + latency measurements. +- **Simulator failures:** the rootcanal test-channel crash and guest recovery + behavior require private control ports, explicit health reporting, and + driver-owned recovery (DD-9). +- **Stream interruption:** ingress reloads and router restarts can cut + long-lived forwards. Endpoints reconnect and emit recovery events; tests + must cover deliberate stream loss. Direct connections avoid the ingress + path where available. +- **Deployment isolation:** shared network namespaces can cause listener + collisions and simulator resets across leases. Enforce the documented + deployment assumptions and report bind failures clearly. +- **Binding contention:** larger benches have more opportunities to hit the + existing cross-lease race. `MaxItems=8` bounds bench size; a global scheduler + remains the full solution. +- **Consumer compatibility:** readers may assume every bound lease has + `status.exporterRef`. Cover known consumers in compatibility tests. +- **Upstream interfaces:** netsim and vhost-user integration may change. + Pin runtime images and track upstream compatibility. +- **Capacity:** per-member policy and the eight-member limit bound access; + bench-level quota remains future work. ## Rejected Alternatives -- **Not doing this.** Multi-device testing stays a client-side workaround with - no atomicity and no device-to-device path, JEP-0016 DD-8 stays blocked, and - Jumpstarter inherits the same one-host ceiling as every existing tool. -- **A separate `LeaseGroup` CR owning N child leases** — DD-1. An earlier - draft proposed exactly this; it was rejected once the binding path was - examined, because N members in one object bind in one write whereas N child - leases make partial acquisition a durable state requiring machinery that - the single-object design never needs. -- **Naming it `LeaseSet`.** `Set` already means ReplicaSet/HPA semantics - here: `ExporterSet` (JEP-0014) is N *identical* instances from a template - with `minReplicas`/`maxReplicas`. A bench is heterogeneous members addressed - by role, so `LeaseSet` beside `ExporterSet` would actively mislead. (The - Kubernetes name for gang-scheduled heterogeneous members is `PodGroup`, - hence the earlier `LeaseGroup` — but DD-1 removes the need for any new kind.) -- **Promoting `status.exporterRef` to a list** — DD-2. -- **Raw addresses in the lease spec** — DD-6. Leaks exporter-private - architecture into a client-authored manifest and breaks when a lab - re-addresses a service. -- **Untyped endpoints without direction** — DD-6. Cannot reject the - controller-to-controller wiring that is the most likely mistake. -- **A `medium` taxonomy** (`bluetooth | wifi | uwb | …`) — DD-8. Validates - what traffic represents rather than what it is, and would approve a - raw-HCI-to-netsim forward, the one pairing most needing rejection. -- **A mandatory wire-format token** — DD-8. Right axis, but still approves - controller-to-controller, and imposes a taxonomy to define and version for - a check that direction already makes. -- **A dedicated `LinkEndpoint` driver interface and bridge-driver tier** — - DD-9. Invented machinery for something existing network drivers plus a - direction already express; a `TcpNetwork` child at `127.0.0.1:7300` is - configuration, not code. -- **Encoding ports as driver labels** — DD-7. Zero schema change, but a - `requires` port has no backing driver instance to label, forcing phantom - report entries and two parallel key namespaces. -- **Client-relayed forwards** — DD-4. Retained as an explicit - `mode: client-relay` debug transport, not the architecture. -- **Direct peer-to-peer only** — DD-4. Fails whenever an edge-device - exporter and a cluster Pod are not mutually routable. -- **A peer-specific RPC in `router.proto`** — DD-5. -- **Guest-side Bluetooth/Wi-Fi shims** — DD-9. Changes the device under test. -- **L4 projection forwarding *as the whole answer*** — DD-10. It is the - Phase 2 deliverable and runs the full projection stack, but it skips the - Bluetooth → Wi-Fi handover, which is why medium simulation stays on the - roadmap as Phase 4. -- **N independently-leased devices behind one exporter.** Ruled out by - JEP-0016's exporter = DUT invariant. A group as a single composite DUT - (JEP-0016 DD-8 option 2) remains legitimate and orthogonal. -- **Building a Jumpstarter multi-device *test runner*.** This JEP stops at the - bench. Existing runners already run multi-device tests well; Jumpstarter's - contribution is the bench they run against. -- **Adopting an existing mobile test framework as the fleet layer.** Their - device abstractions are Android-shaped and run beside the device on the lab - host, and adopting one would import the single-host allocation constraint - this JEP removes. The complementary direction — Jumpstarter *under* such a - framework via a device shim, so existing results pipelines keep working — is - a Future Possibility, not a rejection. +DD-1 through DD-13 record the API and transport alternatives. Higher-level +alternatives are: + +- **Keep client-managed leases:** leaves partial acquisition, independent + lifetimes, and unmanaged device connections. +- **Add `LeaseGroup` or `LeaseSet`:** child leases require partial-acquisition + recovery (DD-1). `LeaseSet` also suggests the interchangeable replicas of + `ExporterSet`, rather than members with distinct roles. +- **Lease devices independently inside one exporter:** can divide control + of a physically connected harness. A composite DUT behind one exporter + remains supported. +- **Build a multi-device test runner:** existing runners can consume the + leased devices. This proposal supplies allocation and connectivity. +- **Adopt a mobile test framework as the fleet layer:** this would require + adapting its device and allocation model to Jumpstarter. Integration + through a device or Mobly controller shim remains possible. ## Prior Art -- **Android/mobile multi-device test frameworks** contribute the role - declaration pattern (a job naming `phone`, `headunit`) and coordinated - setup steps such as pairing two devices before a test. They gang-schedule - device sets, but allocation is confined to one lab host and none of them - bridges a medium between roles — the gap this JEP targets. -- **LAVA MultiNode** is the closest HiL prior art: one job spans multiple - devices with named roles, and — notably — makes the multi-device unit *the - job itself* rather than a wrapper around N jobs, which independently - supports DD-1. Its synchronization primitives (`lava-sync`, `lava-send`, - `lava-wait`) coordinate *test scripts*, not devices; there is no bridged - medium between roles. -- **Mobly** contributes the testbed format this JEP exports to. -- **`kubectl port-forward`** is the mental model for a forward, including its - guarantee — it delivers bytes to a socket and does not verify the consumer - speaks the protocol (DD-8). -- **Cuttlefish multi-instance connectivity** (`--num_instances`, shared - `--vhost_user_mac80211_hwsim`, netsim, rootcanal, `wmediumd_control`) - defines what "correct" looks like for the virtual media this JEP forwards. -- **The Android emulator networking backplane (36.5)** independently - validates the demand, and its same-host scope defines the boundary this JEP - moves. -- **`bumble`** contributes the virtual-controller model and is already a - Jumpstarter dependency via `jumpstarter-driver-bt-peer`, whose existing - `tcp-client:127.0.0.1:7300` transport is the proof that a driver needs no - changes to become a `requires` endpoint. -- **Kubernetes gang scheduling** (Volcano `PodGroup`, coscheduling, Kueue - `Workload`) is the reference for what DD-1's option 2 would have required. - Worth noting *why* Kubernetes needs it and this JEP does not: a Pod's - placement is recorded on the Pod, so N Pods are N objects and gang - scheduling must be layered on top. A Jumpstarter lease records its claims on - itself, so N devices can be one object. +- **LAVA MultiNode** assigns named device roles within one job and provides + `lava-sync`, `lava-send`, and `lava-wait` for test-script coordination. + It is a reference for grouping devices within the existing work unit. +- **Mobly** provides the testbed format used by the proposed export command. +- **Cuttlefish multi-instance connectivity** supplies the local simulator + interfaces considered in DD-9 and DD-10. +- **Android emulator networking** provides another model for multi-device + connectivity within one host. +- **Bumble** supplies virtual hosts, controllers, and link relays. The existing + `jumpstarter-driver-bt-peer` demonstrates the required-port model. +- **Kubernetes gang scheduling** illustrates the coordination needed when + claims live on separate objects. Lease member claims instead live in one + status update, with the cross-lease race described in DD-1. +- **`kubectl port-forward`** provides a comparable byte-transport guarantee + without checking application protocol compatibility. ## Unresolved Questions To resolve during review: -- **Bumble as a Cuttlefish controller.** DD-9's options 1 and 2 are verified; - option 3 is not. `mode=controller` is documented for the Android emulator - but not for Cuttlefish, and it is the only option that reaches beyond - Cuttlefish. One prototype run settles it. -- **Who issues the link-layer join, and who owns addresses?** Option 2 needs - a control step after the forward is up, and rootcanal's address numbering - collides for any attaching host (DD-9). Candidates: a post-establish hook - on the `requires`-side driver, a lease-level `forwards[].onEstablish` - action, or leaving it to the test. The first keeps the controller ignorant - of Bluetooth, which DD-8 and DD-13 argue for. -- **Forward-before-launch ordering.** Option 1's guest dials at boot, so the - forward must exist before the second device is created; option 2's join - must happen after. JEP-0016's provisioner creates the device at lease time, - so the forward has to come up between binding and creation — as a - lease-status condition the provisioner waits on, or a retrying connector. -- **Should the scalar and members forms really be mutually exclusive?** The - alternative is `spec.selector` as a default for members that omit their - own — convenient when several members share a selector, but two ways to - express one thing. -- **How should `listen` collisions be handled?** A fixed address keeps drivers - unchanged but can collide on a physical host; an ephemeral port avoids - collisions but requires telling the driver its address, reintroducing - coupling. Currently fixed-and-validated. -- **Should a lease-level "sync" primitive exist?** LAVA provides `lava-sync`. - Cross-role barriers are implementable in the client library, but a - controller-mediated barrier would work across independently-driven roles. -- **Where does the exported Mobly testbed get its ADB endpoints?** Local - forwards from `jmp shell --lease` couple the export to a live session; a - long-lived per-member forward managed by the client library is the - alternative. +- **Bumble controller:** validate option 3 from DD-9 with Cuttlefish. +- **Link-layer setup:** decide whether rootcanal join and address assignment + run through a driver post-establish hook, a lease-level action, or the + test. A driver hook keeps Bluetooth handling outside the controller. +- **Launch ordering:** define how the provisioner waits for a shared HCI + forward before booting the second CVD, and how federation runs its join + after the link forwards are available. +- **Readiness:** distinguish a forward listener being available from an + application connection being established. Client-started drivers need + the former before they can create the latter. +- **Single-member form:** clarify whether explicit `members` with one entry + uses scalar status and a bare client or member status and a role mapping. +- **Optional members:** define forward behavior when an optional endpoint's + member is omitted at binding. +- **Scalar/member exclusion:** retain mutual exclusion or allow a top-level + selector to act as a default for members without their own selector. +- **Listener allocation:** keep fixed addresses with collision validation, + or allocate ephemeral ports and pass the address to the driver. +- **Synchronization:** determine whether client-side barriers are sufficient + or independently controlled roles need a controller-mediated barrier. +- **Mobly endpoints:** decide whether exported ADB endpoints depend on a live + shell session or on longer-lived client-managed forwards. To resolve during implementation: -- Exact `ListenResponse` variant shape for forward setup instructions. -- What the direct-dial timeout should be before falling back to the router. - Direct-first rather than racing is now the decision (DD-4); the number is - measurable and the stakes are bounded, since falling back costs ~0.65 ms on - the path measured so far. -- How an exporter's network zone is established: taken from deployment - configuration (the assumption in *Direct eligibility*), derived by the - controller from where the registration arrived, or probed. Configuration is - the least clever and the only one that works for an edge exporter that is - routable from the cluster but not the reverse. -- Whether forward reconnect should preserve the medium's logical state or - force a fresh pairing; likely protocol-specific. Two data points: a - projection server must be restarted after a dropped session, so the - reconnect must be visible to endpoint drivers; and a Bumble peer loses its - in-memory bonds when its transport is cut, stranding the DUT's link key - unless the driver persists a keystore. For Bluetooth the answer may simply - be "persist and re-attach", making reconnect invisible. -- How Phase 2's CI obtains the phone image, projection app and receiver - without redistributing them (see Risks). -- How `jmp get leases` renders N exporters in a table column readably. +- The `ListenResponse` variant for forward setup instructions. +- The direct-dial timeout before router fallback. +- How deployments supply and validate `NetworkZone`; the proposed default + is deployment configuration. +- Protocol-specific reconnect behavior, including projection-server restart + and persistence of Bumble bond keys. +- How Phase 2 CI obtains vendor artifacts without redistributing them. +- Readable multi-member output in `jmp get leases`. ## Future Possibilities -Not part of this proposal: - -- **Mixed physical/virtual benches** (DD-11) — a real device paired to a - virtual one through a gateway exporter owning a real radio adapter. No new - software mechanism: the adapter is an ordinary `provides` port. What it - needs is lab hardware and a decision on how to model a shared RF resource — - as its own member, schedulable and auditable but making a two-device bench - three, or as an attribute of the physical member's exporter. -- **Selection-time port validation** via JEP-0017 dynamic labels, so a bench - whose ports cannot be satisfied reports `Unsatisfiable` without holding - anything (DD-12). -- **A global lease scheduler**, which the controller already carries a `TODO` - for; it closes the binding race for single- and multi-member leases alike. -- **Fan-out forwards** — one `provides` port serving several `requires` - ports, for media with more than two participants. Bumble's link relay - already implements the idea as virtual *rooms*, so this is an integration - rather than an invention. -- **Ephemeral `listen` allocation**, needed only if the - one-exporter-per-namespace assumption is relaxed. -- **Bench-level access policy and quota** (DD-12), and **per-member early - release** if the all-or-nothing lifetime proves coarse. -- **Datagram forwards.** Frame-oriented media want datagram semantics with - real boundaries and no head-of-line blocking; an additive datagram frame - type on `RouterService.Stream` (and, further out, QUIC datagrams) is a - separate protocol JEP that Phase 4's frame bridge would consume. -- **Vehicle-bus counterparty integration.** A broker exposing CAN, LIN, - FlexRay and Automotive Ethernet over a socket is already a `provides` port - needing nothing new here, and composes with the automotive drivers - Jumpstarter ships (`can`, `doip`, `someip`, `uds`, `xcp`, `obd`) as the - counterparty they talk to. It suggests a three-member bench — a virtual - head unit driven by real vehicle signals, a restbus supplying the rest of - the vehicle, and a phone — but the mature options are commercial products, - so integrating one is an out-of-scope decision deserving its own JEP. -- **Jumpstarter under an existing test framework** — a device or - Mobly-controller shim backed by a lease, so an established results pipeline - keeps working while gaining non-Android and cross-host devices. -- **A `Bench` template CRD** — a reusable named topology instantiated by - reference, the way `VirtualTargetClass` is to `ExporterSet`. A *template*, - not a second lease-like object, so it does not reopen DD-1. -- **Spawned-on-lease members** — a member satisfied by provisioning a - JEP-0014 pool instance on demand, making a bench elastic in its virtual - half. -- **Agent-facing bench skills** — an agent leasing a two-device bench to - reproduce an interaction bug, following JEP-0016's agent-native framing. +These are outside the initial scope: + +- **Mixed radio benches:** a gateway exporter owns a physical radio adapter. + Decide whether it is a separate leased member or part of the physical + device's exporter (DD-11). +- **Selection-time port validation:** expose reported ports through JEP-0017 + labels so an unsatisfiable request holds no exporters (DD-12). +- **Global scheduling:** resolve the existing cross-lease binding race. +- **Fan-out forwards:** connect one provided port to several required ports, + including shared media such as Bumble relay rooms. +- **Ephemeral listeners:** allocate addresses dynamically if shared network + namespaces are supported later. +- **Bench policy and quota:** add limits across members and support individual + member release if needed. +- **Datagram transport:** define a separate protocol extension for framed + traffic, avoiding TCP head-of-line blocking in the Phase 4 bridge. +- **Vehicle-bus simulation:** integrate a restbus simulator as a provided + socket alongside existing CAN, DoIP, SOME/IP, UDS, XCP, and OBD drivers. +- **Test-framework integration:** supply a device or Mobly-controller shim + backed by a lease. +- **Bench templates:** define reusable named topologies that instantiate + ordinary leases. +- **On-demand members:** provision JEP-0014 pool instances to satisfy a lease. ## Implementation History -- 2026-09-01: JEP drafted. -- 2026-09-01: DD-9 options 1 and 2 verified by hand with two CVD Pods on a - kind cluster — pairing, HFP/A2DP/AVRCP, A2DP streaming. -- 2026-09-01: DD-10 option 3 verified by hand — a phone CVD projected a live - session to a receiver in another Pod over one forwarded port, then again - with the phone end on its Wi-Fi NIC. The decision was revised on that - evidence: L4 projection promoted from diagnostic to the Phase 2 - deliverable, the Wi-Fi medium reframed as a frame bridge, and the - projection bench, its ports, the `projection-rx` driver and the phase - renumbering added. -- 2026-09-01: DD-9 second pass — full classic profile stack between a - retail-image phone CVD and the head unit over federated rootcanals, and - `jumpstarter-driver-bt-peer` run unchanged as a `requires` endpoint. - rootcanal's test-channel crash, address reuse and the identical guest - address plan became driver duties, a Security bullet and a Risk. -- 2026-09-02: DD-9 third pass — the same bench over Jumpstarter's own data - path, with a Bumble HFP-AG + A2DP peer standing in for the phone. Router - overhead measured at 0.65 ms against ~45 ms of rootcanal; proxy-reload - stream loss, duplicate registration and version skew became - forward-endpoint duties, then normative text: a measured price on DD-4's - fast path, one process per exporter identity, endpoint-owned reconnection - with reconnects as events, a cluster-data-path Risk, a *Forward resilience* - test and two acceptance criteria. -- 2026-09-02: `Auto` changed to **prefer a direct peer connection between - same-zone members** rather than defaulting to the router — - `PeerEndpoint`/`NetworkZone`, `prefer_direct`, direct-first-with-fallback - and a *Direct eligibility* rule. -- 2026-09-02: Motivation gained *Why the exporter, not the device, is the - unit of a lease*; vendor-specific naming replaced with generic phone - projection and Bluetooth pairing throughout, and the verification - narratives consolidated. +- 2026-09-01: Drafted the JEP. Manually verified shared and federated + rootcanal configurations and L4 projection with stand-in relays (DD-9, + DD-10). +- 2026-09-02: Verified a CVD-to-Bumble bench over Jumpstarter's router, + measured single-node latency, and identified stream-recovery requirements. + Updated `Auto` to prefer direct connections for same-zone peers. +- 2026-09-04: Consolidated rationale and prototype notes; clarified scope, + implementation requirements, and remaining verification. ## References - [JEP-0014: Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) — "Composite leases — multiple exporters linked into one logical lease" - (Future Possibilities), which this JEP realizes + (Future Possibilities) - JEP-0016: Cuttlefish Kubernetes-Native Orchestration (draft, not yet submitted) — DD-8, whose option 1 this JEP implements - JEP-0017: Dynamic Exporter Labels (draft, not yet submitted) — the diff --git a/docs/source/contributing/jeps/index.md b/docs/source/contributing/jeps/index.md index 70247a7b7..1c1ad6439 100644 --- a/docs/source/contributing/jeps/index.md +++ b/docs/source/contributing/jeps/index.md @@ -38,7 +38,7 @@ For the full process definition, see [JEP-0000](JEP-0000-jep-process.md). | 0011 | [Protobuf Introspection and Interface Generation](JEP-0011-protobuf-introspection-interface-generation.md) | Accepted | @kirkbrauer (Kirk Brauer) | | 0013 | [Metrics, Tracing, and Log Observability](JEP-0013-observability-telemetry-logs.md) | Accepted | @mangelajo (Miguel Angel Ajo Pelayo) | | 0014 | [Virtual Scalable Exporters](JEP-0014-virtual-scalable-exporters.md) | Approved | @mangelajo (Miguel Angel Ajo Pelayo) | -| 0015 | [Multi-Exporter Leases and Inter-Exporter Port Forwarding](JEP-0015-multi-exporter-leases-port-forwarding.md) | Draft | @kirkbrauer (Kirk Brauer) | +| 0015 | [Multi-Exporter Leases and Inter-Exporter Port Forwarding](JEP-0015-multi-exporter-leases-port-forwarding.md) | Discussion | @kirkbrauer (Kirk Brauer) | ### Informational JEPs From de26f58c6809c0e87acada2bd7fadf2a3a2396c3 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Fri, 4 Sep 2026 23:54:12 -0400 Subject: [PATCH 24/26] docs(jep-0015): link proposal discussion Signed-off-by: Kirk Brauer --- .../jeps/JEP-0015-multi-exporter-leases-port-forwarding.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index c75ece2cd..92f63446d 100644 --- a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -9,7 +9,7 @@ | **Type** | Standards Track | | **Created** | 2026-09-01 | | **Updated** | 2026-09-04 | -| **Discussion** | *TBD (PR link)* | +| **Discussion** | [PR #1069](https://github.com/jumpstarter-dev/jumpstarter/pull/1069) | | **Requires** | JEP-0014 | | **Supersedes** | | | **Superseded-By** | | @@ -1582,6 +1582,8 @@ These are outside the initial scope: Updated `Auto` to prefer direct connections for same-zone peers. - 2026-09-04: Consolidated rationale and prototype notes; clarified scope, implementation requirements, and remaining verification. +- 2026-09-04: Submitted for discussion in + [PR #1069](https://github.com/jumpstarter-dev/jumpstarter/pull/1069). ## References From 111d36038792129ab7910dbf0125048d12f108c9 Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Sat, 5 Sep 2026 14:11:28 -0400 Subject: [PATCH 25/26] docs(jep-0015): address review feedback Resolve the API, forwarding, security, compatibility, and recovery ambiguities raised during review. Define the wire messages and make the acceptance criteria match the selected behavior. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...5-multi-exporter-leases-port-forwarding.md | 495 +++++++++++++----- 1 file changed, 355 insertions(+), 140 deletions(-) diff --git a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index 92f63446d..dd82f2dad 100644 --- a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -4,11 +4,11 @@ | ----------------- | -------------------------------------------------------- | | **JEP** | 0015 | | **Title** | Multi-Exporter Leases and Inter-Exporter Port Forwarding | -| **Author(s)** | @kirkbrauer (Kirk Brauer) | +| **Author(s)** | @kirkbrauer (Kirk Brauer, kbrauer@hatci.com) | | **Status** | Discussion | | **Type** | Standards Track | | **Created** | 2026-09-01 | -| **Updated** | 2026-09-04 | +| **Updated** | 2026-09-05 | | **Discussion** | [PR #1069](https://github.com/jumpstarter-dev/jumpstarter/pull/1069) | | **Requires** | JEP-0014 | | **Supersedes** | | @@ -23,8 +23,8 @@ traffic between their named driver ports. Optional `spec.members[]` assigns roles to exporters, and `spec.forwards[]` declares connections between them without exposing local addresses. All member claims are committed in one status update, and the members share a lease lifetime. Forwards reuse the -existing port-forwarding primitives and router, with authenticated direct -connections preferred within a network zone. The examples focus on phone +existing port-forwarding primitives and router, with mutually authenticated, +encrypted direct connections preferred within a network zone. The examples focus on phone projection, but the same model supports CAN, serial, and other socket-based connections. @@ -178,7 +178,11 @@ Each member uses the existing `selector` or `exporterRef` fields with a role name. The scalar lease form remains supported (DD-2). A member marked `optional: true` may be omitted at binding; its role then -resolves to `None` on the client. +resolves to `None` on the client. A forward that references an omitted +optional member is `Disabled`, with a status message naming that member. It +is not established and does not prevent the rest of the lease from becoming +`Ready`. If the optional member binds, every forward that references it is +required to connect normally. #### CAN example @@ -240,8 +244,11 @@ bt headunit.rootcanal phone.controller direct connected 1.2 MiB ``` In Python, the existing `lease()` context manager grows `members` and -`forwards`, and a multi-member lease yields a mapping of roles to the same -client objects a single-exporter lease yields directly: +`forwards`, and a lease requested through `members` yields a bench whose +`members` mapping contains the same client objects a scalar lease yields +directly. Roles are deliberately not installed as arbitrary object +attributes: mapping access safely supports names such as `head-unit` without +allowing a role such as `__class__` or `forwards` to shadow client state: ```python from jumpstarter.config.client import ClientConfigV1Alpha1 @@ -254,8 +261,8 @@ with config.lease( duration=timedelta(minutes=45), ) as lease: with lease.connect() as bench: - phone = bench.phone - headunit = bench.headunit + phone = bench.members["phone"] + headunit = bench.members["headunit"] headunit.power.on() phone.adb.wait_for_device() @@ -268,8 +275,10 @@ with config.lease( assert phone.adb.shell("dumpsys bluetooth_manager | grep -c Connected") == "1" ``` -`config.lease(selector=...)` keeps its existing behavior, with `connect()` -yielding a single driver client (DD-2). +`config.lease(selector=...)` keeps its existing scalar behavior, with +`connect()` yielding a single driver client. Explicit `members`, including a +list with exactly one entry, always uses `status.members` and yields the bench +shape with a role mapping (DD-2). `JumpstarterTest` grows `members` and `forwards` class variables next to `selector`, so existing pytest suites extend without a second base class. @@ -296,13 +305,15 @@ The exporter reuses `TemporaryTcpListener` and `forward_stream()` from 1. After binding, the controller validates the ports and sends setup instructions over each exporter's existing `Listen` stream. 2. Each exporter calls `DialPeer` for connection details and credentials. -3. For a direct-eligible pair, the `requires` side first attempts an - authenticated direct connection with a bounded timeout. Otherwise, or if - that attempt fails in `Auto` mode, both endpoints call - `RouterService.Stream` with tokens sharing one `stream` claim (DD-4, DD-5). +3. For a direct-eligible pair, the `requires` side first establishes mTLS + with a bounded timeout. Both peers validate their controller-issued, + per-forward certificate identities; only then does the requiring side send + `peer_token` inside the encrypted channel. Otherwise, or if that attempt + fails in `Auto` mode, both endpoints call `RouterService.Stream` with + tokens sharing one unique per-forward subject (DD-4, DD-5). 4. The `provides` side dials its local service. The `requires` side listens - on its configured address. Both splice local connections to the peer - stream using `forward_stream()`. + on its configured address. Both splice each accepted local connection to + that connection's peer stream using `forward_stream()`. ```{mermaid} flowchart TD @@ -409,17 +420,26 @@ enum PortDirection { } ``` -The `listen` address of a `requires` port is deliberately **absent**: it is -local to the exporter and no other component needs it (DD-6). +Port names are unique across all `DriverInstanceReport` entries from one +exporter, not merely within one driver instance. The exporter validates this +before registration, and the controller rejects a registration containing a +duplicate with `INVALID_ARGUMENT`. A `(member, port)` therefore resolves to +exactly one driver UUID; the resolved UUID is included in setup instructions +so the exporter never selects a local endpoint by name alone. The `listen` +address of a `requires` port is deliberately **absent**: it is local to the +exporter and no other component needs it (DD-6). **`LeaseSpec`** gains two optional lists: ```go +// Member and forward names must each be unique before binding or token creation. +// +kubebuilder:validation:XValidation:rule="self.members.all(m, self.members.filter(x, x.name == m.name).size() == 1)",message="member names must be unique" +// +kubebuilder:validation:XValidation:rule="self.forwards.all(f, self.forwards.filter(x, x.name == f.name).size() == 1)",message="forward names must be unique" type LeaseSpec struct { // ... all existing fields unchanged ... - // Members of a multi-exporter lease. When empty, the lease binds a - // single exporter using Selector/ExporterRef exactly as before. + // Members of a member-form lease. When empty, the lease binds a single + // exporter using the top-level Selector/ExporterRef exactly as before. // +kubebuilder:validation:MaxItems=8 Members []LeaseMember `json:"members,omitempty"` @@ -427,7 +447,14 @@ type LeaseSpec struct { Forwards []LeaseForward `json:"forwards,omitempty"` } +// Exactly one non-empty selection source is required for every member. +// +kubebuilder:validation:XValidation:rule="((((has(self.selector.matchLabels) && size(self.selector.matchLabels) > 0) || (has(self.selector.matchExpressions) && size(self.selector.matchExpressions) > 0)) ? 1 : 0) + ((has(self.exporterRef) && has(self.exporterRef.name) && size(self.exporterRef.name) > 0) ? 1 : 0)) == 1",message="exactly one of selector or exporterRef.name is required" +// +kubebuilder:validation:XValidation:rule="self.name != 'forward' && self.name != 'forwards'",message="member name is reserved" type LeaseMember struct { + // DNS-label syntax keeps names usable in the CLI and generated formats. + // Python accesses them only through bench.members[name]. + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` Name string `json:"name"` Selector metav1.LabelSelector `json:"selector,omitempty"` ExporterRef *corev1.LocalObjectReference `json:"exporterRef,omitempty"` @@ -459,7 +486,7 @@ type LeaseForward struct { type ForwardEndpoint struct { Member string `json:"member"` - Port string `json:"port"` + Port string `json:"port"` // exporter-wide unique port name } ``` @@ -468,8 +495,8 @@ type ForwardEndpoint struct { ```go type LeaseStatus struct { // ... all existing fields unchanged ... - // ExporterRef stays authoritative for single-exporter leases and is - // left nil for multi-member leases (DD-2). + // ExporterRef stays authoritative for scalar leases and is left nil for + // every lease requested through Members, including one-member lists. Members []LeaseMemberStatus `json:"members,omitempty"` Forwards []LeaseForwardStatus `json:"forwards,omitempty"` @@ -484,8 +511,8 @@ type LeaseMemberStatus struct { type LeaseForwardStatus struct { Name string `json:"name"` - State string `json:"state"` // Pending|Connecting|Connected|Reconnecting|Failed - Mode string `json:"mode,omitempty"` // transport actually in use + State string `json:"state"` // Pending|Disabled|Connecting|Connected|Reconnecting|Failed + Mode string `json:"mode,omitempty"` Message string `json:"message,omitempty"` } ``` @@ -503,31 +530,136 @@ optional fields used only to decide direct eligibility: NetworkZone string `json:"networkZone,omitempty"` ``` -The existing CEL rules are extended, not replaced — the current -"one of selector or exporterRef is required" rule gains a `members` arm, plus -new rules for mutual exclusion, unique role names, forwards referencing -declared members, member immutability (mirroring `tags` and `context`), and -exactly one of `between` or `from`+`to` per forward. +The existing CEL rules are extended, not replaced. The current top-level +"one of selector or exporterRef is required" rule gains a `members` arm and +mutual exclusion. Per-member CEL requires exactly one *non-empty* `selector` +or `exporterRef.name`; both set and both unset are rejected. Additional rules +enforce unique and immutable member names, unique forward names, forwards +referencing declared members, member immutability (mirroring `tags` and +`context`), and exactly one of `between` or `from`+`to` per forward. Duplicate +forward names are thus rejected before the controller derives stream subjects +or creates status maps. -**Protocol** — additions to existing messages and one new RPC: +**Protocol** — additions to existing messages and one new RPC. These are the +wire definitions; the similarly named Go structs above describe the CRD only: ```protobuf +enum LeaseForwardMode { + LEASE_FORWARD_MODE_UNSPECIFIED = 0; // Auto + LEASE_FORWARD_MODE_AUTO = 1; + LEASE_FORWARD_MODE_ROUTER = 2; + LEASE_FORWARD_MODE_DIRECT = 3; + LEASE_FORWARD_MODE_CLIENT_RELAY = 4; +} + +enum LeaseForwardState { + LEASE_FORWARD_STATE_UNSPECIFIED = 0; + LEASE_FORWARD_STATE_PENDING = 1; + LEASE_FORWARD_STATE_DISABLED = 2; + LEASE_FORWARD_STATE_CONNECTING = 3; + LEASE_FORWARD_STATE_CONNECTED = 4; + LEASE_FORWARD_STATE_RECONNECTING = 5; + LEASE_FORWARD_STATE_FAILED = 6; +} + +enum ForwardSide { + FORWARD_SIDE_UNSPECIFIED = 0; + FORWARD_SIDE_PROVIDES = 1; + FORWARD_SIDE_REQUIRES = 2; +} + +message LeaseMember { + string name = 1; + oneof selection { + LabelSelector selector = 2; + string exporter_name = 3; + } + bool optional = 4; + bool allow_disabled = 5; +} + +message ForwardEndpoint { + string member_name = 1; + string port_name = 2; // Unique within the member exporter. +} + +message LeaseForwardBetween { + repeated ForwardEndpoint endpoints = 1; // Exactly two. +} + +message LeaseForwardDirected { + ForwardEndpoint from = 1; // Must resolve to PROVIDES. + ForwardEndpoint to = 2; // Must resolve to REQUIRES. +} + +message LeaseForward { + string name = 1; + oneof topology { + LeaseForwardBetween between = 2; + LeaseForwardDirected directed = 3; + } + LeaseForwardMode mode = 4; +} + +message LeaseMemberStatus { + string name = 1; + optional string exporter_uuid = 2; // Absent when an optional member is omitted. + int32 priority = 3; + bool spot_access = 4; +} + +message LeaseForwardStatus { + string name = 1; + LeaseForwardState state = 2; + LeaseForwardMode mode = 3; // Transport actually in use when connected. + optional string message = 4; +} + message RequestLeaseRequest { - google.protobuf.Duration duration = 1; // unchanged - LabelSelector selector = 2; // unchanged - repeated LeaseMember members = 3; // NEW - repeated LeaseForward forwards = 4; // NEW + google.protobuf.Duration duration = 1; // unchanged + LabelSelector selector = 2; // unchanged; scalar form only + repeated LeaseMember members = 3; // NEW + repeated LeaseForward forwards = 4; // NEW } message GetLeaseResponse { - // ... fields 1-6 unchanged; exporter_uuid set only when single-member ... + // ... fields 1-6 unchanged; exporter_uuid set only for scalar leases ... repeated LeaseMemberStatus members = 7; // NEW repeated LeaseForwardStatus forwards = 8; // NEW } message DialRequest { - string lease_name = 1; // unchanged - optional string member_name = 2; // NEW: required for multi-member leases + string lease_name = 1; // unchanged + optional string member_name = 2; // NEW: required for member-form leases +} + +// Listen is a server stream from the controller to an authenticated exporter. +// Fields 1 and 2 retain the existing client-connection instruction. Exactly +// one instruction is populated. Forward setup is idempotent by +// (lease_uid, forward_name, member_name). +message ListenResponse { + string router_endpoint = 1; // unchanged + string router_token = 2; // unchanged + optional ForwardSetup forward_setup = 3; // NEW + optional ForwardTeardown forward_teardown = 4; // NEW +} + +message ForwardSetup { + string lease_name = 1; + string lease_uid = 2; + string forward_name = 3; + string member_name = 4; + string peer_member_name = 5; + ForwardSide side = 6; + string local_driver_uuid = 7; // Resolved from the exporter-wide unique port. + string local_port_name = 8; + string peer_port_name = 9; + LeaseForwardMode mode = 10; +} + +message ForwardTeardown { + string lease_uid = 1; + string forward_name = 2; } service ControllerService { @@ -541,17 +673,38 @@ message DialPeerRequest { string member_name = 3; } +message DirectPeerParameters { + string endpoint = 1; // Dial target for REQUIRES; empty for PROVIDES. + bytes ca_certificate = 2; // Trust root for the opposite endpoint. + bytes certificate = 3; // This endpoint's short-lived certificate. + bytes private_key = 4; // This endpoint's short-lived private key. + string expected_peer_identity = 5; +} + message DialPeerResponse { string router_endpoint = 1; - string router_token = 2; // `stream` claim shared by both ends - optional string peer_endpoint = 3; // set when the pair is direct-eligible - optional string peer_token = 4; // authenticates a direct dial - bool prefer_direct = 5; // Auto resolved to direct-first + string router_token = 2; + optional DirectPeerParameters direct = 3; + optional string peer_token = 4; + bool prefer_direct = 5; } ``` -`ReleaseLeaseRequest`, `ListLeasesRequest`, `RouterService`, and -`router.proto` are untouched. +Forward credentials are not carried on `Listen`. After receiving +`ForwardSetup`, each exporter calls authenticated `DialPeer`; the controller +returns side-specific, short-lived router and (when eligible) per-forward mTLS +credentials. The provider uses them for its configured peer listener, and the +requiring side uses them to dial and verify that listener. Private keys remain +inside the authenticated controller channel. + +For an ordinary client connection, fields 1 and 2 are both populated and +fields 3 and 4 are absent. For setup or teardown, only the corresponding +optional message is populated. An old exporter reports no ports, so it cannot +be selected for a forward and never receives fields 3 or 4 of +`ListenResponse`. A new exporter checks those fields before treating fields 1 +and 2 as a client connection. Unknown fields remain safe under proto3. `ReleaseLeaseRequest` and `ListLeasesRequest` are +untouched. `RouterService.Stream` and its protobuf remain unchanged, but its +token validation changes as described in DD-5. **CLI surface** — existing commands, new flags: @@ -622,25 +775,28 @@ work, as noted in the controller's existing TODO. **Alternatives considered:** 1. **Keep the scalar, add a parallel list.** `status.exporterRef` stays - authoritative for single-exporter leases and is left **nil** for - multi-member leases, which populate `status.members[]` instead. + authoritative for scalar-form leases and is left **nil** for every + member-form lease, which populates `status.members[]` instead. 2. **Promote the scalar to a list** and migrate every reader. 3. **Always populate both**, setting `status.exporterRef` to the first member. **Decision:** Option 1. -**Rationale:** Existing consumers retain the scalar field for single-exporter leases. -For multi-member leases, an old reader sees nil and treats the lease as -unbound, rather than selecting an arbitrary member. Populating the scalar -with the first member would misroute consumers such as the JEP-0016 Host -Orchestrator façade. Replacing it with a list would require every consumer -to migrate. - -`GetLeaseResponse.exporter_uuid` follows the same convention. The client -returns a driver client for a single-member lease and a role mapping for a -multi-member lease. - -`DialRequest.member_name` selects a role. Omitting it for a multi-member +**Rationale:** Existing consumers retain the scalar field for leases requested +through the top-level selector or exporter reference. For a lease requested +through `members`, even when the list contains exactly one entry, an old +reader sees nil and treats the lease as unbound rather than selecting an +arbitrary role. Populating the scalar with the first member would misroute +consumers such as the JEP-0016 Host Orchestrator façade. Replacing it with a +list would require every consumer to migrate. + +`GetLeaseResponse.exporter_uuid` follows the same convention. A scalar-form +lease returns a bare driver client. An explicit one-member or multi-member +lease returns a bench with `members[role]` and member status. The request +form, rather than the number of bound exporters, therefore determines a +stable response shape. + +`DialRequest.member_name` selects a role. Omitting it for any member-form lease returns `INVALID_ARGUMENT` listing the available roles. ### DD-3: Partial-bench behavior when a member is lost mid-lease @@ -688,20 +844,34 @@ rootcanal HCI commands took about 45 ms either way (DD-9). These measurements do not establish cross-node or sustained-throughput limits. Phase 4's Wi-Fi frame bridge requires separate latency testing. -### DD-5: Router changes — none +### DD-5: Keep the router protobuf; bind pairing to forward claims **Alternatives considered:** -1. **Reuse `RouterService.Stream` unchanged**, issuing both endpoints a token - bearing the same `stream` claim. +1. **Reuse the `RouterService.Stream` RPC and forwarding path**, while + strengthening its JWT validation for peer-pair claims. 2. **Add a peer-specific RPC** to `router.proto` with explicit A/B roles. - -**Decision:** Option 1 — no `router.proto` change. - -**Rationale:** `RouterService.Stream` pairs callers with the same stream claim and -already accepts exporter identities. Peer-specific authorization belongs in -the controller's token issuance. A second router RPC would duplicate the -stream path without changing its behavior. +3. **Reuse the current shared exporter subject unchanged**, relying only on + controller-side token issuance. + +**Decision:** Option 1 — no `router.proto` change, with authorization changes +inside `RouterService.Stream`. + +**Rationale:** The current router uses the JWT `sub` as its pending-stream key; +a shared subject such as `jumpstarter exporter` would allow unrelated +forwards to collide. For each forward, the controller instead sets `sub` to +the stable UUIDv5 derived from `(lease UID, forward name)`. Each signed token +also carries `lease_uid`, `forward_name`, `source_exporter`, `target_exporter`, +`source_member`, `target_member`, and `side` (`provides` or `requires`). + +The router parses these claims, keys pending streams by the unique subject, +and pairs only two tokens whose exporter/member fields are reciprocal and +whose sides are complementary. A duplicate token from the same side is +rejected rather than paired. `DialPeer` has already authenticated the caller +as the bound source exporter before issuing its token, and token expiry is +bounded by the lease. This preserves the existing byte-forwarding RPC while +preventing cross-forward and same-side pairing. A second RPC would duplicate +the stream path without improving these checks. ### DD-6: Named ports with direction, not addresses @@ -749,8 +919,10 @@ needs. A structured field represents both without synthetic driver entries or separate label conventions. An absent `ports` field defaults to an empty list, so old exporters continue -to register and serve ordinary leases. The local `listen` address stays in -exporter configuration and is not reported. +to register and serve ordinary leases. Reported names are exporter-wide +unique; duplicate names across driver instances reject registration, making a +lease endpoint unambiguous. The local `listen` address stays in exporter +configuration and is not reported. ### DD-8: No protocol taxonomy; direction plus an optional tag @@ -1067,10 +1239,11 @@ flowchart TD Conditions reuse the existing `LeaseConditionType` values — `Pending`, `Ready`, `Unsatisfiable`, `Invalid` — with `ForwardsReady` and `Degraded` -added. `Ready` for a multi-member lease requires all required members bound -*and* all declared forwards connected. Expiry, `status.ended`, the -`jumpstarter.dev/lease-ended` label, and `spec.release` retain their existing -behavior. +added. `Ready` for a member-form lease requires all required members bound and every +non-disabled forward connected. A forward whose optional endpoint was omitted +has explicit `Disabled` status and does not gate readiness. Expiry, +`status.ended`, the `jumpstarter.dev/lease-ended` label, and `spec.release` +retain their existing behavior. ### Forward validation and establishment @@ -1080,8 +1253,11 @@ exists belongs to a bound exporter, not to a selector (DD-12). For each `ExporterStatus.Devices[].Ports`: 1. Every endpoint names a declared member — enforced by CEL at admission, - before this point. -2. Both named ports exist on the respective bound exporters. + before this point. If either role is an omitted optional member, validation + stops for that entry and records `Disabled` with the omitted role named. +2. Both named ports exist on the respective bound exporters, and each port + resolves to one driver UUID because report registration enforces + exporter-wide unique names. 3. **Direction resolves.** For a `between` forward, exactly one endpoint must report `PROVIDES` and the other `REQUIRES`; the controller assigns the roles accordingly (DD-13). For an explicit `from`/`to` forward, the stated @@ -1089,18 +1265,23 @@ exists belongs to a bound exporter, not to a selector (DD-12). For each 4. If both ports declare `protocol`, the values are equal (DD-8). A failure sets `Invalid`, names the forward and reason, and leaves members -bound for inspection. +bound for inspection. A disabled optional-member forward is not a validation +failure and receives no setup instruction or token. -Setup follows *How a forward comes up*. The controller derives the router -`stream` claim from `(lease UID, forward name)` using UUIDv5 in a fixed -namespace, keeping it stable across reconciles. Both tokens use -`aud: jumpstarter router`, `sub: jumpstarter exporter`, and an expiry bounded -by the lease end time. +Setup follows *How a forward comes up*. The controller derives a router +subject from `(lease UID, forward name)` using UUIDv5 in a fixed namespace, +keeping it stable across reconciles. Both tokens use that value as `sub`, use +`aud: https://jumpstarter.dev/router`, carry reciprocal member, exporter, and +side claims, and expire no later than the lease. `RouterService.Stream` +validates those claims as described in DD-5. The direct attempt has a short, bounded timeout and is not raced with a -router connection. The peer listener authenticates `peer_token`. `Direct` -fails without fallback; `Router` skips the direct attempt. Status records the -transport used and the reason for any fallback. +router connection. Direct mode requires controller-issued per-forward mTLS: +both sides validate the peer certificate identity before the requiring side +transmits `peer_token` inside the encrypted channel. A plaintext or +server-authentication-only connection is rejected. `Direct` fails without +fallback; `Router` skips the direct attempt. Status records the transport +used and the reason for any fallback. **Direct eligibility.** Both members must report the same non-empty `NetworkZone`, and the `provides` side must report a `PeerEndpoint`. The zone @@ -1109,11 +1290,14 @@ per cluster network. The controller does not infer reachability from IP addresses. An unreachable peer falls back to the router in `Auto` mode. **Reconnection.** The `requires` endpoint keeps its listener open and -reconnects the peer stream with backoff after a network interruption, router -restart, or ingress reload. Forward state comes from the stream, without -probing the service. Reconnect events notify drivers that must restore -protocol state, such as restarting a projection server. Protocol-specific -recovery remains necessary where a new stream cannot preserve the session. +re-establishes the peer path with backoff after a network interruption, router +restart, or ingress reload. One accepted local TCP connection and one peer +stream form a single splice: if the peer stream closes, the exporter closes +that local connection and never attaches a replacement stream to it. A driver +that reconnects gets a fresh splice. Reconnect events invoke an explicit +endpoint-driver recovery hook for protocols that dial only once or must +restore state, such as restarting a projection server. Forward state comes +from the stream, without probing the service. **Failure modes and handling:** @@ -1125,6 +1309,8 @@ recovery remains necessary where a new stream cannot preserve the session. | Declared protocols disagree | Lease `Invalid` (DD-8) | | Protocols differ and at least one tag is absent | Validation passes; protocol errors may occur when data is exchanged (DD-8) | | Forward references an undeclared member | Rejected by CEL at admission; lease never created | +| Forward references an omitted optional member | Forward `Disabled` naming the role; no setup or token; lease may become `Ready` | +| Duplicate port names in one exporter's reports | Exporter registration rejected with `INVALID_ARGUMENT` | | `listen` address already bound on the exporter | Lease `Invalid` naming the port and address | | Router stream drops mid-lease | Re-dial with backoff; `Reconnecting`; `Degraded` after a grace period | | Ingress/proxy reload cuts the peer stream | Reconnect with backoff and notify endpoint drivers | @@ -1179,11 +1365,14 @@ Provisioner ordering remains an unresolved question. - **Port exposure:** only declared ports can participate in forwards. A `requires` listener uses an exporter-configured address and exists only while leased. -- **Direct authentication:** the optional peer listener is disabled by - default and requires `peer_token`. It is separate from device ports. - Network policies should allow the authenticated peer port while blocking - peer access to unauthenticated simulator ports. JEP-0016 is expected to - supply this policy with exporter Pods. +- **Direct authentication and confidentiality:** the optional peer listener + is disabled by default and accepts only controller-issued, short-lived + per-forward mTLS credentials. Both sides verify the expected exporter and + member identity from the certificate before `peer_token` is sent inside + the encrypted channel. It is separate from device ports. Network policies + should allow the authenticated peer port while blocking peer access to + unauthenticated simulator ports. JEP-0016 is expected to supply this policy + with exporter Pods. - **Membership:** `members` is immutable after creation. - **Physical RF:** devices in a shared lab are audible to others in range; lease authorization does not isolate radio traffic. @@ -1214,31 +1403,42 @@ captures can be attached to test results alongside forward metrics. ### Unit Tests -- `LeaseSpec` CEL validation: extended one-of rule, members/scalar mutual - exclusion, role-name uniqueness, forwards referencing declared members, - member immutability. +- `LeaseSpec` CEL validation: extended top-level one-of rule, + members/scalar mutual exclusion, per-member rejection when both or neither + of `selector` and `exporterRef` are usable, DNS-label and reserved role-name + checks, unique member and forward names, forwards referencing declared + members, and member immutability. - Member selection: all-or-nothing binding, no self-collision, correct `Unsatisfiable` role naming, and — the property DD-1 rests on — that a reconcile which cannot satisfy every member writes **no** member claims. -- Single-exporter regression: existing lease controller tests pass - unmodified, and a one-member lease produces the same `status.exporterRef` - as the equivalent scalar lease. +- Scalar-form regression: existing lease controller tests pass unmodified + and retain `status.exporterRef`. An explicit one-member lease instead + populates one `status.members` entry and leaves `status.exporterRef` nil. - Aggregation: priority = min(member priorities), duration clamped to min(member `maximumDuration`). - Port report round-trip: driver-declared ports reach `ExporterStatus.Devices[].Ports`; an exporter reporting none is treated as - non-forwardable; a report with no `ports` field is accepted unchanged. + non-forwardable; a report with no `ports` field is accepted unchanged; and + duplicate names across two driver reports reject registration. - Forward validation: missing port, `provides→provides`, `requires→requires`, protocol disagreement, and `listen` collision each - produce `Invalid` with the offending forward named. -- `Dial` without `member_name` on a multi-member lease returns + produce `Invalid` with the offending forward named. A forward with an + omitted optional endpoint instead becomes `Disabled` and does not gate + lease readiness. +- `Dial` without `member_name` on any member-form lease returns `INVALID_ARGUMENT` listing roles; with a valid role, routes correctly. -- `DialPeer` token issuance: identical `stream` claim for both ends, - stability across reconciles, expiry clamped to lease end, rejection when - the caller is not bound to the named member. -- Python client: role attribute access, `connect()` returning a bare client - for single-member and a role mapping for multi-member, - `bench.forwards[...]` state, raising on a `Degraded` role. +- `DialPeer` token issuance: identical unique `sub` for both ends, reciprocal + member/exporter and complementary side claims, stability across reconciles, + expiry clamped to lease end, and rejection when the caller is not bound to + the named member. Router tests reject unrelated or same-side tokens that + carry the same subject. +- `ListenResponse`: forward setup and teardown decode alongside the unchanged + client-connection fields; setup carries the resolved local driver UUID; + old exporters never receive forward instructions. +- Python client: scalar `connect()` returns a bare client; explicit one-member + and multi-member requests both return a bench with `members[...]`; arbitrary + role attributes are not exposed; `bench.forwards[...]` reports state; and + calls into a `Degraded` role raise. ### Integration Tests @@ -1252,9 +1452,10 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): - Lease expiry, explicit release, and client disconnect; assert no leaked router streams, no exporters left claimed, and no listeners left bound. - Router-mode vs. direct-mode selection: `Auto` picks direct for two - same-zone exporters, falls back to the router when the peer dial is - blocked, records the mode actually used, and never falls back under - `mode: direct`. + same-zone exporters only after mutual TLS identity verification, never + sends `peer_token` before the encrypted handshake, falls back to the router + when the peer dial is blocked, records the mode actually used, and never + falls back under `mode: direct`. - `jmp get lease -o mobly` output validated against Mobly's testbed schema. - **Compatibility**: an N-1 client against an N controller for the full single-exporter workflow; an N client issuing a single-exporter lease @@ -1279,8 +1480,10 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): host-local paths. Include cross-node, sustained A2DP, and physical-controller tests to establish supported workloads. - **Forward resilience:** cut a live stream through a router restart or - ingress reload. Assert reconnection, metrics and events, and recovery of - communication for a driver that opened its transport at startup. + ingress reload. Assert that the affected local TCP connection closes, a + later local connection receives a new peer stream, metrics and events are + emitted, and an endpoint-driver recovery hook restores protocols that do + not reconnect themselves. ### Manual Verification @@ -1308,23 +1511,37 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): cannot reach an exporter its client could not lease directly - [ ] Lease priority = min(member priorities); duration clamped to min(member `maximumDuration`) -- [ ] `status.exporterRef` unchanged for single-exporter leases and nil for - multi-member ones; existing lease controller tests pass unmodified -- [ ] `Dial` without `member_name` on a multi-member lease returns +- [ ] `status.exporterRef` unchanged for scalar-form leases and nil for every + member-form lease, including an explicit one-member list; existing + scalar controller tests pass unmodified +- [ ] Scalar Python connections return a bare client; explicit one-member and + multi-member connections return `bench.members[...]` +- [ ] `Dial` without `member_name` on a member-form lease returns `INVALID_ARGUMENT` naming the roles **Ports and forwards** - [ ] `PortReport` is optional on `DriverInstanceReport`; exporters reporting no ports register and operate unchanged +- [ ] Port names are exporter-wide unique; duplicate names across driver + instances reject registration, and setup names the resolved driver UUID - [ ] Declared ports appear in `ExporterStatus.Devices[].Ports` and in `jmp get exporter` output - [ ] `listen` addresses never appear in any report, CR status, or lease spec +- [ ] Admission rejects duplicate forward names before deriving router + subjects, and rejects member selection with both/neither source set - [ ] Forward validation rejects missing ports, `provides→provides`, `requires→requires`, and declared-protocol mismatch, naming the forward -- [ ] Forwards establish over `RouterService` with no `router.proto` change -- [ ] Direct fast path is authenticated, falls back automatically, and is - observable (mode + fallback-rate metrics) +- [ ] A forward whose optional endpoint is omitted is `Disabled`, receives no + credentials, and does not prevent lease readiness +- [ ] Additive `ListenResponse` setup and teardown instructions identify the + lease, forward, side, local driver UUID, and ports; credentials are + returned only by authenticated `DialPeer`; old exporters receive neither +- [ ] Forwards establish over `RouterService` with no `router.proto` change; + its authorization pairs only reciprocal claims under a unique subject +- [ ] Direct fast path uses per-forward mTLS, verifies peer identity before + sending `peer_token`, falls back automatically, and is observable + (mode + fallback-rate metrics) - [ ] `Auto` resolves to a direct peer connection for two same-zone in-cluster exporters with peer listeners; router fallback is recorded - [ ] `bt-peer` participates as a `requires` endpoint with **no Python @@ -1334,10 +1551,11 @@ Against a kind cluster with the controller and mock exporters (`e2e/`): join, address assignment, and recovery handled by endpoint drivers (DD-9) - [ ] A Bumble-based shared controller exists as a driver exposing a `provides` port, for benches outside Cuttlefish and for N-way media -- [ ] A forward survives a cut peer stream: the endpoint re-dials and - re-splices with no driver involvement, the reconnect is counted and - emitted as an event, and a driver that dialed once at start keeps - working +- [ ] After a peer-stream cut, the endpoint closes the associated local TCP + connection and re-establishes the peer path with backoff; a reconnecting + driver obtains a new splice, while protocols without reconnect behavior + recover through an explicit endpoint-driver hook. No replacement stream + is attached to an existing local TCP connection - [ ] A second exporter process registering under a live identity is detected and refused - [ ] Byte fidelity and reset semantics verified by the `EchoNetwork` @@ -1392,10 +1610,10 @@ for `status.exporterRef`. `ExporterAccessPolicy`, `ExporterSet`, and `VirtualTargetClass` are otherwise untouched. Every lease that exists today validates unchanged. - **`status.exporterRef`**: unchanged for every lease that does not pass - `members`. Multi-member leases leave it nil, which existing consumers - already read as "not bound yet" (DD-2), so the JEP-0016 façade, - `jmp get leases`, `Dial` and JEP-0013 telemetry keep working; they change - only to *support* benches. + `members`. Every member-form lease, including an explicit one-member list, + leaves it nil, which existing consumers already read as "not bound yet" + (DD-2), so the JEP-0016 façade, `jmp get leases`, `Dial` and JEP-0013 + telemetry keep working; they change only to *support* benches. - **Driver report and drivers**: `ports` is a new optional repeated field, so an exporter built before this JEP reports none and is treated as unable to participate in forwards. Ports are declared in exporter @@ -1408,9 +1626,9 @@ for `status.exporterRef`. (or a controller version) and fails with a clear message rather than acquiring a one-device lease it will misuse. - **Operator upgrade**: a CRD schema addition, a standard bundle bump with no - conversion webhook. Multi-member leases must be removed before rollback; - single-exporter leases remain compatible. -- **Coexistence**: single- and multi-member leases share one exporter pool, + conversion webhook. Member-form leases must be removed before rollback; + scalar leases remain compatible. +- **Coexistence**: scalar- and member-form leases share one exporter pool, one scheduler, and one selection implementation. ## Consequences @@ -1522,12 +1740,6 @@ To resolve during review: - **Readiness:** distinguish a forward listener being available from an application connection being established. Client-started drivers need the former before they can create the latter. -- **Single-member form:** clarify whether explicit `members` with one entry - uses scalar status and a bare client or member status and a role mapping. -- **Optional members:** define forward behavior when an optional endpoint's - member is omitted at binding. -- **Scalar/member exclusion:** retain mutual exclusion or allow a top-level - selector to act as a default for members without their own selector. - **Listener allocation:** keep fixed addresses with collision validation, or allocate ephemeral ports and pass the address to the driver. - **Synchronization:** determine whether client-side barriers are sufficient @@ -1537,7 +1749,6 @@ To resolve during review: To resolve during implementation: -- The `ListenResponse` variant for forward setup instructions. - The direct-dial timeout before router fallback. - How deployments supply and validate `NetworkZone`; the proposed default is deployment configuration. @@ -1584,6 +1795,10 @@ These are outside the initial scope: implementation requirements, and remaining verification. - 2026-09-04: Submitted for discussion in [PR #1069](https://github.com/jumpstarter-dev/jumpstarter/pull/1069). +- 2026-09-05: Resolved review questions around optional endpoints, explicit + one-member response shape, port and name uniqueness, protobuf setup + messages, router claim binding, direct-path encryption, and reconnect + semantics. ## References From 9af2525d95db5c6d4be043e088f54c131eb5c55e Mon Sep 17 00:00:00 2001 From: Kirk Brauer Date: Sat, 5 Sep 2026 14:13:33 -0400 Subject: [PATCH 26/26] docs(jep-0015): note Kubernetes pod certificate path Record Kubernetes 1.37 Pod Certificates as a future source of exporter workload identity, and call out mTLS as a broader exporter authentication enhancement. Assisted-by: Claude Signed-off-by: Kirk Brauer --- ...5-multi-exporter-leases-port-forwarding.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md index dd82f2dad..0a7c07ad9 100644 --- a/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md +++ b/docs/source/contributing/jeps/JEP-0015-multi-exporter-leases-port-forwarding.md @@ -1373,6 +1373,15 @@ Provisioner ordering remains an unresolved question. should allow the authenticated peer port while blocking peer access to unauthenticated simulator ports. JEP-0016 is expected to supply this policy with exporter Pods. +- **Broader exporter authentication:** mTLS is also a stronger future + authentication mechanism for exporter-to-controller and exporter-to-router + connections. Unlike the current bearer token, it proves possession of a + private key while protecting the channel and can bind the certificate + identity to one `Exporter`. Lease and forward authorization still apply; + possession of an exporter certificate alone never grants access to a port. + Kubernetes Pod Certificates are one possible source of this workload + identity, while physical and non-Kubernetes exporters require an equivalent + issuer and enrollment path. - **Membership:** `members` is immutable after creation. - **Physical RF:** devices in a shared lab are audible to others in range; lease authorization does not isolate radio traffic. @@ -1773,6 +1782,16 @@ These are outside the initial scope: namespaces are supported later. - **Bench policy and quota:** add limits across members and support individual member release if needed. +- **Kubernetes-native mTLS identity:** Kubernetes 1.37 graduates Pod + Certificates and ClusterTrustBundles to Stable. Exporter Pods could mount a + signer-issued, automatically rotated X.509 identity and trust bundle for + direct peer, controller, and router mTLS. This would keep the workload + private key generated and managed by the kubelet instead of returning a + private key from `DialPeer`; the per-forward token would remain as + lease-scoped authorization after workload authentication. Adoption requires + a configured signer (Kubernetes 1.37 does not ship a production signer in + core), live certificate reload, an identity-to-`Exporter` binding, and a + platform-neutral fallback for exporters outside Kubernetes. - **Datagram transport:** define a separate protocol extension for framed traffic, avoiding TCP head-of-line blocking in the Phase 4 bridge. - **Vehicle-bus simulation:** integrate a restbus simulator as a provided @@ -1798,7 +1817,8 @@ These are outside the initial scope: - 2026-09-05: Resolved review questions around optional endpoints, explicit one-member response shape, port and name uniqueness, protobuf setup messages, router claim binding, direct-path encryption, and reconnect - semantics. + semantics. Recorded Kubernetes Pod Certificates and broader exporter mTLS + authentication as future paths. ## References @@ -1827,6 +1847,9 @@ These are outside the initial scope: - [netsim (`platform/tools/netsim`)](https://android.googlesource.com/platform/tools/netsim/) — `proto/netsim/packet_streamer.proto` - [google/android-cuttlefish](https://github.com/google/android-cuttlefish) +- [Kubernetes 1.37: Pod Certificates and Cluster Trust Bundles](https://kubernetes.io/blog/2026/08/28/kubernetes-v1-37-pod-certificates-and-cluster-trust-bundles/) + — Stable projected workload certificates and trust anchors (KEP-4317 and + KEP-3257) - [Test Multi-Device Interactions with the Android Emulator](https://android-developers.googleblog.com/2026/04/Test-Multi-Device-Interactions-with-the-Android-Emulator.html) - [LAVA MultiNode](https://docs.lavasoftware.org/lava/multinode.html)