docs(l2): add the multiprover (SP1 GPU + TDX) release test - #7243
docs(l2): add the multiprover (SP1 GPU + TDX) release test#7243ilitteri wants to merge 5 commits into
Conversation
Every L2 check in the release process exercises one prover against an `OnChainProposer` that requires only that one. A production rollup runs with two, and nothing verified that a batch can satisfy both — so a regression in the two-prover path would have shipped unnoticed. Adds the runbook and a checklist entry. Validated against v25.0.0-rc.2: 2494 batches verified with `REQUIRE_SP1_PROOF` and `REQUIRE_TDX_PROOF` both true on chain, real SP1 GPU proofs, and no `InvalidTdxProof` reverts. The runbook documents four environment pins, each of which fails in a way that does not point at its cause, and all four were hit while bringing the test up: - solc must be exactly 0.8.31 (`TDXVerifier.sol` pins the pragma) - the deploy is not idempotent; it needs a fresh chain - `--watcher.watch-interval 1000`, or commits revert with `InvalidPrivilegedTransactionLogs()` on an instantly-mining dev L1 - both provers must attach before the first batch, and the L1 and L2 datadirs must be reset together, or a prover deadlocks asking for a pruned batch It also records why the TDX guest must be a plain QEMU VM rather than a real TDX guest: dev-mode `register()` reads the signer from the quote's first 20 bytes, which only holds for a dev quote. Booting a real TDX guest registers quote-header bytes as the signer and every `verifyBatch` reverts with `InvalidTdxProof()`, while registration appears to succeed. Real attestation needs `ETHREX_TDX_DEV_MODE=false`, which the DCAP registry blocks on a dev chain id — noted as future work rather than papered over.
|
🤖 Kimi Code ReviewAutomated review by Kimi (Moonshot AI) |
🤖 Codex Code Review
No code-level correctness or security findings beyond the operational/documentation issues above; the diff is documentation-only. I did not execute the full procedure. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
The prose was hard-wrapped at ~80 columns, the way a commit body is written. Every other page under docs/developers keeps one unwrapped line per paragraph and lets the renderer do the wrapping, so this stood out in the source and produced needless diff noise on any future edit. Reflowed the paragraphs, bullets and blockquotes in the new page and in the release-process section it adds. Code blocks and tables are untouched. Two things the reflow surfaced are fixed as well: the environment-pin list said "three things" after a fourth had been added, and the first procedure step was still called "Prepare the TDX host" even though the test runs against a plain guest and needs no TDX-capable CPU.
The checklist item did not say where to run it, unlike every other entry in that list, which names its host. It now names both: the SP1 prover on `l2-gpu`, and the L1, deploy, sequencer and TDX prover VM on `ethrex-tdx-baremetal`. The runbook said "any Linux host with QEMU/KVM" for the TDX side, which is true of the dev-mode configuration but is not the operational answer. `ethrex-tdx-baremetal` is the designated host and the one where extending this to real attestation would happen, so the page names it, keeps the note that dev mode does not depend on its TDX hardware, and names both hosts in the step headings. Also notes that both machines are shared, so the stack should be torn down afterwards, and that on `l2-gpu` this test contends with the SP1 GPU integration test for the same GPU, ports and datadirs.
The test was written around `ETHREX_TDX_DEV_MODE=true`, which is what CI uses. In that mode `TDXVerifier.register()` short-circuits: it takes the signing address from the quote's first 20 bytes and never calls `verifyAndAttestOnChain`. So the quote is trusted rather than verified, and running it on TDX hardware buys nothing — the guest even has to be a plain VM, because a real quote's header bytes are not an address. Switch it to `ETHREX_TDX_DEV_MODE=false` with a real TDX guest, so a quote signed by the host's silicon is verified on chain. Confirmed working on `ethrex-tdx-baremetal`: `ProverSetup received for TDX` followed by `ProverSetupACK sent` with dev mode off is the verifier accepting a genuine quote. Three prerequisites the deploy does not handle, each of which fails opaquely: - `_validateReport` compares MRTD and RTMR0-2 against constants compiled into `TDXVerifier.sol`, which pin one historical image; the release under test measures differently and registration reverts with `MRTD mismatch`. Pin them to the image being released, read out of a real quote. - `CA.SIGNING` — the Intel TCB Signing certificate — is never loaded, only `CA.ROOT` and `CA.PLATFORM`, so the DAO cannot check collateral signatures and rejects the upsert with `TCB_Cert_Expired`. Nothing has expired. - The TCB info and QE identity for the platform have to be upserted. ethrex shells out to `automata-dcap-qpl-tool` for this, which resolves addresses from a chain-id-keyed registry and rejects the dev L1 with `Unsupported chain_id: 9` — silently, since its exit status is discarded. Fetch them from the host's PCCS service and upsert them directly. Also records that TD_QE identity needs version 4 or 5 rather than the version inside the JSON, that these calls have to be sent as raw calldata because cast's tuple parser splits on the commas in the JSON payload, and that `run-qemu` from `hypervisor.nix` cannot be detached because it hardcodes `-serial mon:stdio`. Dev mode stays documented as the fallback when no TDX host is available.
…rover test Both proofs are required per batch, so verification advances at the slower prover's rate. An SP1 proof on the GPU host measures a steady 106s, so the 15s commit time the runbook carried outran proving by roughly 7x and the gap never closed: a 63-hour run reached batch 10983 committed against 2127 verified. The integration suite's withdrawal tests block until the batch holding the withdrawal is verified, so in that state step 9 cannot pass however long it runs. Raises the commit time to 120s, says why, and adds the check to step 9 plus a troubleshooting row for the symptom it produces.
1 participant
Motivation
Every L2 check in the release process exercises a single prover against an
OnChainProposerconfigured to require only that one. A production rollup runs with two, and nothing verified that a batch can satisfy both — so a regression in the two-prover verification path would ship unnoticed.Description
Adds
docs/developers/l2/multiprover-test.mdand a release-process checklist entry for a deployment whereREQUIRE_SP1_PROOFandREQUIRE_TDX_PROOFare both set, solastVerifiedBatchonly advances once a batch has satisfied both.The TDX side runs with
ETHREX_TDX_DEV_MODE=falseon real TDX hardware, so the quote is checked on chain byverifyAndAttestOnChain. This is deliberately stronger than CI, which uses dev mode — thereTDXVerifier.register()short-circuits, taking the signing address from the quote's first 20 bytes and never verifying anything. Dev mode is documented as the fallback when no TDX host is available.Validation
Run against
v25.0.0-rc.2, SP1 prover onl2-gpu, everything else onethrex-tdx-baremetal:TDX_VERIFIER_ADDRESS.isDevMode()false— real verification, not the dev short-circuitREQUIRE_SP1_PROOF()/REQUIRE_TDX_PROOF()truelastVerifiedBatchInvalidTdxProof()revertsMRTD()on chain0x4e2d467d…, the measurement of the image under testauthorizedSignature()ProverSetup received for TDX→ProverSetupACK sentwith dev mode off is the verifier accepting a quote signed by that machine's silicon.Prerequisites the deploy does not handle
Real attestation needed three things, each of which fails opaquely and none of which were documented:
MRTD mismatch_validateReportcompares MRTD/RTMR0-2 against constants compiled intoTDXVerifier.solthat pin one historical image build; they must be pinned to the image being releasedTCB_Cert_Expired(0xea8cd522)CA.ROOTandCA.PLATFORMbut neverCA.SIGNING, so the DAO cannot check collateral signaturesUnsupported chain_id: 9automata-dcap-qpl-toolresolves addresses from a chain-id-keyed registry and cannot serve a dev chain; its exit status is discarded byprepare_quote_prerequisites, so the first symptom is an unrelated revert laterAlso recorded: TD_QE identity needs version 4 or 5, not the
versioninside the JSON (Incorrect_Enclave_Id_Version); these upserts must be sent as raw calldata because cast's tuple parser splits on the commas in the JSON payload; andrun-qemufromhypervisor.nixcannot be detached because it hardcodes-serial mon:stdio.Environment pins
Four more, each hit while bringing the test up:
Source file requires different compiler version—TDXVerifier.solpins the pragmadeploy-p256reverts with no reason string (CREATE2 addresses already populated)--watcher.watch-interval 1000InvalidPrivilegedTransactionLogs()(0x9e6e5638) on an instantly-mining dev L1No blocks to prove, asking forever for a batch whose input was pruned, whilelastVerifiedBatchcannot advance without itFollow-ups this surfaced (not in this PR)
prepare_quote_prerequisitesdiscards the qpl tool's exit status, so a failed collateral load only surfaces as an unrelated revert much later.TDXVerifier's measurements are storage variables with no setter, so authorising a new prover image means redeploying the verifier.Checklist
STORE_SCHEMA_VERSION(crates/storage/lib.rs) if the PR includes breaking changes to theStorerequiring a re-sync.