feat: add complete Flock Stage 3 compression for aggregate roots - #604
Open
johnchandlerburnham wants to merge 26 commits into
Open
feat: add complete Flock Stage 3 compression for aggregate roots#604johnchandlerburnham wants to merge 26 commits into
johnchandlerburnham wants to merge 26 commits into
Conversation
Add the deterministic lift/flat/structural activation matrix and keep dummy calls deferred. Split aggregate recursion commitment/FRI configuration from IxVM defaults, share it between proving and verification, and pin the future cache encoding without changing active protocol parameters.
Precompute versioned per-slot cache keys, persist lift and join wrappers in the content-addressed store, and reuse entries only after exact claim and native outer-proof verification. Add safe proof decoding, corruption recovery, and --no-cache.
Execute ready lift and join slots as a dependency DAG under explicit job and RAM admission. Add calibration-pending slot weights, failure draining, CLI controls, and serial/parallel scheduler and proof-equivalence gates.
Add an opt-in two-child benchmark that proves singleton CheckEnv shards, lifts both proofs, and measures a verified flat join. Wire join metrics through reporting and dashboards, and record the current pre-E2 lift-size baseline.
- rust-toolchain.toml channel: 1.92 → 1.98, with the matching fenix toolchain hash in flake.nix. The pinned fenix already carries the 1.98 release manifest, so flake.lock needs no change (fenix's nixpkgs stays pinned via its lean4-nix follows, so the Lean toolchain is untouched). - Drop clippy::from_iter_instead_of_collect from the workspace lints: removed in clippy 1.98 and now warns as unknown. - Fix the warnings new clippy 1.98 lints surface across the workspace: chunks_exact(N) → as_chunks::<N>() where the chunk size is constant, descending sort_by → sort_by_key(Reverse(..)), iteration over map values via .values(), map().unwrap_or() → map_or(), a checked division, an unwrap-after-is_some restructured into if-let, and assorted redundant-reference/pattern cleanups (mostly cargo clippy --fix). The two byte-gadget files keep their chunks_exact warnings until the next commit, which rewrites those regions anyway. CI derives its Rust version from rust-toolchain.toml, so no workflow changes are needed.
Companion to multi-stark's update-p3 branch (c72d321 → 249b740), which carries four soundness/robustness fixes and the Plonky3 v0.6.0 bump (pruned FRI Merkle multiproofs: ~2x faster verification, 40-70% smaller proofs; canonical Goldilocks serde removes proof-byte malleability). Proofs and verifying keys are not compatible with the previous pin. Integration: - Lookup gained max_multiplicity, a declared per-row bound on the multiplicity's integer magnitude feeding the newly enforced logUp height bound Σ wᵢ·hᵢ + |claims| < p. Function-circuit slots accumulate mutually-exclusive branch selectors, so they declare 1; committed count columns (function return slots, the memory circuit, the byte gadget tables) declare the new COUNT_COLUMN_BUDGET (2^32 queries per entry). - The VK wire format carries the bound: u64 LE max_multiplicity per lookup, between the multiplicity node id and the arg count. The in-circuit VK deserializer (Ix/MultiStark/SystemDeserialize.lean) parses past it; the value is bound through the vk digest but the height bound itself is not yet enforced in-circuit (the native verifier enforces it). - Message fingerprints are width-bound by default upstream (the slot width seeds the Horner fold), which is incompatible with aiur's branch-shared lookup slots: mutually exclusive branches superpose messages of different natural widths into one slot at the maximum width, so a narrow call is sent zero-padded to a width its callee's return slot never provides, and proving fails with UnbalancedChannel (pinned as the prove_verify_mismatched_call_widths regression). Aiur instead declares WidthBinding::ByConstruction — the plain Horner fold, restoring zero-padding transparency — and takes on the prefix-freeness contract that makes it sound: every message's natural width is a function of its constant-constrained leading prefix (channel tag plus discriminator: fun_idx fixes 2+in+out, the memory size coordinate fixes 3+size, each gadget tag fixes its table width), so zero-extension can only equate a padded message with its own natural form. The contract is documented at the channel constants in lib.rs; the declaration is applied in AiurSystem::build and mirrored in the vk_codec decoder so decoded VKs replay the same transcript. ByConstruction is also exactly the fold the in-circuit verifier's logup_fingerprint already computes, so the recursive verifier needs no fingerprint change. - The policy is Fiat-Shamir-bound as the first observe_shape word; the in-circuit transcript replay prepends the matching limb. - aiur_multi_stark.rs regenerated (ix codegen) for the deserializer and transcript changes. - P3 v0.6.0 ships FRI query openings as pruned Merkle multiproofs, while the in-circuit verifier consumes one authentication path per query (its per-query control flow is a far smaller circuit than the amortized multiproof walk). Rather than porting the walk into the DSL, the proof advice stays in the per-query transport: multi-stark's new advice module re-encodes a natively-verified proof by running p3's own verification with a recording compression function and reading each query's path back out of the recorded digest map. The advice bytes are untrusted verifier input, never digest-bound — the transcript binds the commitments and every expanded sibling is authenticated against them per query — so pruning vs expansion is pure transport and the encoding choice is sound. AiurSystem gains proof_to_advice_bytes (FFI: AiurSystem.proofToAdviceBytes); the recursive-verifier test feeds it instead of Proof.toBytes, whose native wire format is still round-tripped separately. The Lean-side proof grammar and the codegen'd verifier are byte-identical to before — no in-circuit changes. Claim layout, the VK wire format above and aiur's public semantics are otherwise unchanged; the policy adds no prover or verifier work over the previous pin. Still open, native-verifier-only: the logUp height bound is parsed past but not yet enforced in-circuit (a wide-arithmetic check, tracked separately).
The Lean v4.33.1 update pinned both dependencies at revisions that predate their Rust 1.98 bumps; their heads now carry those bumps, which this workspace needs since rust-toolchain.toml moved to 1.98. Both revisions stay on leanprover/lean4:v4.33.1. - Blake3.lean 1b0fbd2 → e6e908b (Rust 1.98, plus a case-insensitive source-directory fix), updated in lakefile.lean, lake-manifest.json and the blake3-lean flake input. The revision keeps the `blake3_rs_shared` target the `ix_native_decide_dynlib` pin requires. The inherited entry in Benchmarks/Compile/lake-manifest.json was still on the pre-4.33.1 revision and now tracks the root pin. - lean-ffi 2a9c91e → 93c7e52 (Rust 1.98). Only bignat reaches the sp1 and zisk workspaces, so their lock files move that one package.
The !benchmark recursive phase reported n/a for every fri-verifier metric: Benchmarks/Typecheck.lean still fed Proof.toBytes — the pruned multiproof wire format — to executeMultiStark/proveMultiStark, so the in-circuit verifier rejected on parse and the harness (correctly) left the recursive fields absent rather than emit a fake datum. The in-circuit verifier consumes the per-query advice transport (AiurSystem.proofToAdviceBytes); proofBytes stays the reported proof-size metric. bench-recursion-debug had the same advice-format gap plus a stale claim recipe: it still built the public input as 32 raw digest bytes, predating the ClaimHarness.packedDigestKey packing bench-typecheck uses (its own out-of-circuit sanity check failed with InvalidPowWitness — a wrong claim diverges every challenge — and the advice re-encoder refused the proof for the same reason). Both aligned with the typecheck flow. Validated end-to-end at production parameters (numQueries 100, query PoW 20, blowup 2) on Nat.add_comm: inner prove, advice re-encoding, and the codegen'd in-circuit verifier accepting.
Keep compact proof bytes at storage and cache boundaries, expand them only for recursive lift/join advice, reject unsupported zero-query benchmarks, and decode legacy store proofs without panicking. Refresh tests, benchmark pins, and Rust 1.98 lint compatibility.
Add Ix/Aggr, a recursive aggregation system for IxVM shard proofs that keeps Ix/MultiStark untouched and Ix-agnostic. One entrypoint, ix_aggr, subsumes lifting and joining: a one-byte advice shape selects wrap or binary join over any mix of IxVM and ix_aggr children, so shard proofs enter the recursion system directly as join children and the dedicated lift stage disappears. Circuit (Ix/Aggr/Circuit.lean): every shape verifies its children in full (verify + ood_verify from the shared Multi-STARK verifier modules) against the vk its hinted kind demands. Identity is one 80-byte digest-bound blob - blake3(ixvm vk) || verify_claim idx || blake3(self vk) || ix_aggr idx - carried unchanged at every node; self children must bind the identical blob digest, pinning both vks and both entrypoint indices transitively. Claims are a uniform 18-word [0, aggr_idx, allowed(8), checkEnv(8)] at every depth, so proofs of different tree levels combine freely. Wrap shapes bind the output digest to the child CheckEnv digest directly; pair shapes open both CheckEnv preimages, re-root the canonical subject/assumption trees, and prove subjects = L ∪ R, assumptions = (asmL ∪ asmR) ∖ subjects with linear sorted merges. Toplevel (Ix/Aggr.lean): ixAggr = MultiStark.multiStarkFull + circuit, pruned to ix_aggr, so verify_multi_stark_proof and other unrelated entries no longer pad aggregate proofs. The host half of the wire contracts (allowed blob, public input packing, shape codes, keyed preimage/tree blob framing, interpreter IO assembly) lives beside the toplevel; Ix/Aggr/Host.lean folds CheckEnvTrees statements. Native path: ix codegen gains the ix-aggr target (crates/ixvm-codegen/src/aiur_ix_aggr.rs, 244 fns); its runner builds the seven-channel IO buffer natively and routes execution through the generated code. New FFI rs_aiur_ix_aggr_execute/_prove (executeIxAggr/proveIxAggr) pass proofs, vks, claims, and the compact count/key/length preimage/tree blobs without per-byte boxing, plus Proof.ofBytesChecked for store-boundary decoding. CLI: ix aggr --ixe E --ixes M <shard-proof>... reconstructs every nonempty shard statement from the env, matches wrappers by claim digest, natively pre-verifies them, folds a balanced bisection (the canonical fold makes the root claim independent of tree shape), wraps single-shard roots so the persisted root is always an ix_aggr proof, checks the root closes over the env canonical tree with no residual assumptions, and persists the wrapper. Tests (lake test -- ix-aggr, 17 cases): all five shapes accept over real Multi-STARK stand-in child proofs from two distinct-vk systems; codegen'd execution matches the interpreter on output and per-circuit query counts for wrap and pair; negatives break one binding each - lying shape hint, tampered proof, foreign identity, wrap statement drift, dropped assumption, padded subject set, and tree advice not reproducing its keyed root.
Replace the legacy lift/join entrypoints with the heterogeneous ix_aggr system across aggregate, verification, cache, and codegen paths. Preserve wrap-first and direct-join policies behind one proof identity and cache namespace. Add the converged 91-check semantic suite and a deterministic 132-case activation audit covering shapes 0 through 9 twice, with no unobserved circuits.
Replace the legacy two-shard three-entrypoint benchmark with a four-shard manifest-subtree harness for production ix_aggr wrap-first and direct policies. Enforce the q=100 serialized no-cache profile, natively verify every input and recursive output, persist ordinary aggregate wrappers, and emit resumable per-slot JSON metrics. Port bench-typecheck --join to ix_aggr direct shape 2 while preserving the stable join metric schema. Remove the final legacy slot-spec/preimage shim and pin both M1-f four-shard plans in the focused suite.
Remove the fixed eight-round FRI ceiling, add deep-round relation regressions, and expose aggregate-root preflight diagnostics. Add the optional Flock CLI/FFI bridge with shared canonical root preparation, statement pinning, proof verification, and atomic artifact output.
Instrument the complete production regression across setup, proving, artifact transport, valid verification, and negative checks. Document the measured 524.315-second breakdown and clarify that corrupted-proof rejection performs a second full verifier run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
This PR is stacked on
jcb/sp1-compressorand should target that branch:The base branches supply the shard-to-root aggregation pipeline, canonical
Stage 2 aggregate root, and the existing SP1 terminal-compression experiment.
This PR adds an independent no-RISC-V Stage 3 backend: it verifies the complete
specialised Stage 2 relation directly in Flock over
F128with BLAKE3.Summary
The resulting pipeline is:
Stage 3 is intentionally an off-chain intermediate proof. It removes the
general-purpose RISC-V verifier overhead and leaves Stage 4 with one fixed
Flock verification relation suitable for a universal-setup PLONK/FFLONK
development backend. A circuit-specific Groth16 endpoint remains available
once that relation stabilises and sub-kilobyte settlement is the overriding
constraint.
The complete production relation now proves and verifies successfully. The
latest prover work also removes padded witness generation from the critical
path: the release CLI-shaped round trip now takes approximately 2.12
seconds, versus 39.62 seconds for the previous release proving path,
without changing the 326,019-byte Stage 3 artifact.
Protocol configuration
The backend pins the proof-system configuration as part of the Stage 3
statement:
b310f35f35f68095537150a1c8c0a43caca9a29e;F128binary field;Fast128Ligerito profile;Changing the upstream revision or any of these parameters is a protocol
change and changes the configuration digest. Poseidon2 is not used anywhere
in this path.
Complete Stage 2 verifier relation
The Flock relation constrains all eleven registered verifier phases,
including:
recombination;
final-polynomial check; and
The BLAKE3 lowering supports multi-block rows and messages beyond one
1,024-byte chunk. Transcript sampling matches Plonky3 rejection sampling,
including the constrained chained refill.
Native Stage 2 verification remains an inexpensive fail-fast guard before
allocating the production relation. It is not part of the soundness boundary:
the Flock relation repeats the relevant verification constraints.
Statement and artifact binding
Stage3StatementV1binds three independent values:Verification requires an externally expected Stage 3 statement. In
particular, the verifier does not accept a relation digest supplied only by
the prover. Deployments and the eventual Stage 4 verifier must pin the
expected relation digest.
Stage3ArtifactV1adds strict versioned framing and bounded parsing. Theproduction payload carries the canonical Stage 2 transport, compiled-circuit
digest, and Flock proof bundle needed to reconstruct and verify the relation.
Production aggregate integration
The SP1 and Flock root commands share one canonical
AggregateRootInputspreparation path. This prevents the two compression backends from drifting
on:
A new optional
flockFFI feature exposes the backend without adding Flock tothe normal Ix build.
Preflight a persisted aggregate root with:
Preflight performs all of the following without starting the cryptographic
Flock prover:
CheckEnvaggregate;nu, gate-rowcounts, and all relevant digests.
After successful preflight, generate an artifact with:
Proof mode preflights, proves the complete relation, checks that the resulting
statement exactly matches the preflighted root/relation/configuration,
cryptographically verifies the artifact, and only then installs it through an
atomic rename.
The Lake build traces the nested
flock-stage3Rust sources so changes to theoptional connector cannot silently reuse a stale static archive. The Nix
inputs also use public, immutable sources so CI does not depend on an
interactive GitHub credential.
Full FRI schedules
This removes the former eight-round implementation ceiling.
The accepted commit-phase length is derived from the initial FRI height and
remains bounded by
MAX_LOG_HEIGHT = 31. With the current binary FRIproduction parameters, schedules through the current 30-round maximum are
supported.
Circuit-level regressions construct and evaluate valid transcript-bound
relations at 9, 16, and 30 rounds. Negative tests reject a missing final round
and a corrupted deepest Merkle path.
Prover architecture and optimization
The initial complete prover was correct but spent almost all of its time
materialising padded Boolean witnesses. Every table has uniform
2^nucapacity, while the current complete fixture uses only a small live prefix.
The allocating path constructed logical
z,Az, andBzvectors across theentire capacity, packed them, copied them into the union, and later cleared the
large buffers.
This PR now uses a generic live-row in-place Boolean driver:
stripe layout;
z,Az, andBzare written directly into Flock's union buffers;(Az) * (Bz) = zbefore proving;stripe equality, including a partial final group.
Additional setup improvements include:
and verification;
TableTypeinstead of deep-copied;proving, and post-hoc verification; and
IX_FLOCK_TIMING=1, with Flock's internal phasetracing available under
PCS_TRACE=1.The relation cache compares the complete witness for equality rather than
using a digest, so a hit cannot substitute a relation for different proof
values. It retains only the most recent relation and therefore cannot grow
without bound.
Measured complete proof
The ignored complete regression uses a real canonical multi-STARK proof with:
The artifact remains byte-for-byte the same size after the prover rewrite:
The latest release run, shaped like the real CLI (
preflight → prove → verify → negative checks), reported:Within proof generation, the Flock prover itself took 177.84 ms. The largest
improvements versus the preceding release trace were:
Counting both one-time relation setup and proof generation, the previous
39.615-second release proving path is now approximately 2.076 seconds: about
a 19× end-to-end improvement. The distinction between setup and proof is
reported explicitly because the immutable relation can be reused, while a
one-shot process must still pay setup once.
This fixture is deliberately complete but small; it is not a substitute for
the pending full production
ix_aggrcapacity and memory run. Stage 3 is alsonot intended for Ethereum calldata. Stage 4 must compress verification of this
fixed relation into the sub-kilobyte terminal proof.
Run the expensive regression explicitly with:
Validation performed
The final optimization pass was validated with:
cargo check -p flock-stage3-host;cargo test -p flock-stage3-host:cargo fmt --all; andgit diff --check.The broader branch integration also includes:
cargo clippy -p flock-stage3-host --all-targets -- -D warnings;cargo check -p flock-stage3-host --all-targets;cargo check -p ix-ffi;cargo check -p ix-ffi --features flock;cargo clippy -p ix-ffi --features flock -- -D warnings;nix develop --command lake build Ix.Cli.FlockRootCmd Main;IX_FLOCK=1 nix develop --command lake exe ix flock-root --help; andflock-rootandcompress-root.Scope and remaining work
The relation is deliberately specialised to the current Ix protocol:
Before freezing a deployment relation, we still need to:
ix_aggrroot;profile; and
The remaining local setup bottleneck is immutable shape construction: in the
latest run, table declaration and final circuit wiring consumed almost all of
the 1.896-second preflight. The exact-witness cache eliminates repeated work
for the CLI's preflight/prove/verify sequence. Reuse across different aggregate
proof values with the same structural layout will require separating
per-proof input/public extraction from Flock's immutable
CircuitShapesetup.The next architectural boundary is Stage 4: compile verification of this fixed
Flock relation into the universal-setup FFLONK/PLONK development backend,
measure constraints and Ethereum gas, and retain the ability to switch the
same stable statement to a circuit-specific Groth16 endpoint if required.