Skip to content

feat: add complete Flock Stage 3 compression for aggregate roots - #604

Open
johnchandlerburnham wants to merge 26 commits into
jcb/sp1-compressorfrom
jcb/flock-stage3
Open

feat: add complete Flock Stage 3 compression for aggregate roots#604
johnchandlerburnham wants to merge 26 commits into
jcb/sp1-compressorfrom
jcb/flock-stage3

Conversation

@johnchandlerburnham

@johnchandlerburnham johnchandlerburnham commented Aug 31, 2026

Copy link
Copy Markdown
Member

Stack

This PR is stacked on jcb/sp1-compressor and should target that branch:

main
  └── jcb/aggregate-first
        └── jcb/sp1-compressor
              └── jcb/flock-stage3  ← this PR

The base branches supply the shard-to-root aggregation pipeline, canonical
Stage 2 aggregate root, and the existing SP1 terminal-compression experiment.
This PR adds an independent no-RISC-V Stage 3 backend: it verifies the complete
specialised Stage 2 relation directly in Flock over F128 with BLAKE3.

Summary

The resulting pipeline is:

closed ix_aggr root
  → canonical Stage 2 vk / claim / compact proof transport
  → typed AIR / logUp / PCS / FRI verifier witness
  → specialised Flock Fast128 relation
  → versioned, verified Stage 3 artifact
  → future Stage 4 Ethereum SNARK

Stage 3 is intentionally an off-chain intermediate proof. It removes the
general-purpose RISC-V verifier overhead and leaves Stage 4 with one fixed
Flock verification relation suitable for a universal-setup PLONK/FFLONK
development backend. A circuit-specific Groth16 endpoint remains available
once that relation stabilises and sub-kilobyte settlement is the overriding
constraint.

The complete production relation now proves and verifies successfully. The
latest prover work also removes padded witness generation from the critical
path: the release CLI-shaped round trip now takes approximately 2.12
seconds
, versus 39.62 seconds for the previous release proving path,
without changing the 326,019-byte Stage 3 artifact.

Protocol configuration

The backend pins the proof-system configuration as part of the Stage 3
statement:

  • Flock revision b310f35f35f68095537150a1c8c0a43caca9a29e;
  • the F128 binary field;
  • the Fast128 Ligerito profile;
  • BLAKE3 Merkle commitments;
  • a chained-BLAKE3 Fiat–Shamir transcript; and
  • domain-separated production and conformance transcripts.

Changing the upstream revision or any of these parameters is a protocol
change and changes the configuration digest. Poseidon2 is not used anywhere
in this path.

Complete Stage 2 verifier relation

The Flock relation constrains all eleven registered verifier phases,
including:

  • canonical verifier-key, claim, and proof encodings;
  • typed witness shape, sparse circuit activation, and trace heights;
  • the specialised Aiur AIR and verifier-key metadata;
  • all 18 canonical Goldilocks claim words;
  • lookup-message inversion and the complete logUp balance;
  • exact chained-BLAKE3 transcript replay and field rejection sampling;
  • Goldilocks and quadratic-extension arithmetic;
  • AIR selectors, DAG evaluation, OOD composition, and quotient
    recombination;
  • multi-matrix and multi-height PCS openings;
  • BLAKE3 MMCS leaf hashing and authentication paths;
  • every binary-FRI challenge, grinding draw, query index, fold, roll-in, and
    final-polynomial check; and
  • the published Stage 2 root shared by the statement and proof constraints.

The BLAKE3 lowering supports multi-block rows and messages beyond one
1,024-byte chunk. Transcript sampling matches Plonky3 rejection sampling,
including the constrained chained refill.

Native Stage 2 verification remains an inexpensive fail-fast guard before
allocating the production relation. It is not part of the soundness boundary:
the Flock relation repeats the relevant verification constraints.

Statement and artifact binding

Stage3StatementV1 binds three independent values:

  1. the canonical Stage 2 root digest;
  2. the complete relation-manifest digest; and
  3. the pinned Flock configuration digest.

Verification requires an externally expected Stage 3 statement. In
particular, the verifier does not accept a relation digest supplied only by
the prover. Deployments and the eventual Stage 4 verifier must pin the
expected relation digest.

Stage3ArtifactV1 adds strict versioned framing and bounded parsing. The
production payload carries the canonical Stage 2 transport, compiled-circuit
digest, and Flock proof bundle needed to reconstruct and verify the relation.

Production aggregate integration

The SP1 and Flock root commands share one canonical AggregateRootInputs
preparation path. This prevents the two compression backends from drifting
on:

  • aggregate claim construction;
  • recursion verifying-key selection;
  • outer-claim serialization;
  • compact-proof extraction; and
  • FRI parameters.

A new optional flock FFI feature exposes the backend without adding Flock to
the normal Ix build.

Preflight a persisted aggregate root with:

IX_FLOCK=1 nix develop --command lake exe ix flock-root ROOT_ADDRESS \
  --mode preflight

Preflight performs all of the following without starting the cryptographic
Flock prover:

  • validates that the root is a closed CheckEnv aggregate;
  • natively verifies and expands the compact Stage 2 proof;
  • constructs the complete typed AIR/PCS/FRI witness;
  • builds and evaluates every Flock gate;
  • constructs the relation manifest and Stage 3 statement; and
  • reports transport sizes, advice geometry, table capacity, nu, gate-row
    counts, and all relevant digests.

After successful preflight, generate an artifact with:

IX_FLOCK=1 nix develop --command lake exe ix flock-root ROOT_ADDRESS \
  --mode prove --output root.stage3.flock

Proof mode preflights, proves the complete relation, checks that the resulting
statement exactly matches the preflighted root/relation/configuration,
cryptographically verifies the artifact, and only then installs it through an
atomic rename.

The Lake build traces the nested flock-stage3 Rust sources so changes to the
optional connector cannot silently reuse a stale static archive. The Nix
inputs also use public, immutable sources so CI does not depend on an
interactive GitHub credential.

Full FRI schedules

This removes the former eight-round implementation ceiling.

The accepted commit-phase length is derived from the initial FRI height and
remains bounded by MAX_LOG_HEIGHT = 31. With the current binary FRI
production parameters, schedules through the current 30-round maximum are
supported.

Circuit-level regressions construct and evaluate valid transcript-bound
relations at 9, 16, and 30 rounds. Negative tests reject a missing final round
and a corrupted deepest Merkle path.

Prover architecture and optimization

The initial complete prover was correct but spent almost all of its time
materialising padded Boolean witnesses. Every table has uniform 2^nu
capacity, while the current complete fixture uses only a small live prefix.
The allocating path constructed logical z, Az, and Bz vectors across the
entire capacity, packed them, copied them into the union, and later cleared the
large buffers.

This PR now uses a generic live-row in-place Boolean driver:

  • rows are evaluated in parallel groups of eight, matching the lincheck
    stripe layout;
  • z, Az, and Bz are written directly into Flock's union buffers;
  • only declared rows and useful columns are evaluated;
  • padding writes are skipped when the merged union declares padding unread;
  • live rows still check (Az) * (Bz) = z before proving;
  • all eleven production table slots use the in-place path; and
  • an allocating-vs-in-place differential test checks observable buffer and
    stripe equality, including a partial final group.

Additional setup improvements include:

  • compiled CSC lincheck circuits cached once per process and reused by proving
    and verification;
  • Boolean table plans cached where their construction is nontrivial;
  • freshly built R1CS matrices moved into TableType instead of deep-copied;
  • a bounded one-entry exact-witness relation cache shared by preflight,
    proving, and post-hoc verification; and
  • opt-in nested timing under IX_FLOCK_TIMING=1, with Flock's internal phase
    tracing available under PCS_TRACE=1.

The relation cache compares the complete witness for equality rather than
using a digest, so a hit cannot substitute a relation for different proof
values. It retains only the most recent relation and therefore cannot grow
without bound.

Measured complete proof

The ignored complete regression uses a real canonical multi-STARK proof with:

  • an inactive leading circuit;
  • active circuits at heights 8 and 4;
  • an active preprocessed matrix;
  • an 18-word claim lookup;
  • nontrivial first-row and transition constraints; and
  • two FRI queries.

The artifact remains byte-for-byte the same size after the prover rewrite:

Output Size
Stage 3 artifact 326,019 bytes
Encoded production payload 325,893 bytes

The latest release run, shaped like the real CLI (preflight → prove → verify → negative checks), reported:

Phase Time
Fixture setup 0.005 s
Relation preflight/setup 1.896 s
Proof generation after preflight 0.180 s
Artifact encode/decode <0.001 s
Valid cryptographic verification 0.017 s
Corrupted-proof rejection 0.018 s
Complete round trip 2.116 s

Within proof generation, the Flock prover itself took 177.84 ms. The largest
improvements versus the preceding release trace were:

Internal phase Before After
Live/padded witness generation 566.46 ms 34.57 ms
Witness-buffer give-back 159.70 ms 1.47 ms
Complete Flock prove 1,140.48 ms 177.84 ms

Counting both one-time relation setup and proof generation, the previous
39.615-second release proving path is now approximately 2.076 seconds: about
a 19× end-to-end improvement. The distinction between setup and proof is
reported explicitly because the immutable relation can be reused, while a
one-shot process must still pay setup once.

This fixture is deliberately complete but small; it is not a substitute for
the pending full production ix_aggr capacity and memory run. Stage 3 is also
not intended for Ethereum calldata. Stage 4 must compress verification of this
fixed relation into the sub-kilobyte terminal proof.

Run the expensive regression explicitly with:

IX_FLOCK_TIMING=1 PCS_TRACE=1 \
  cargo test -p flock-stage3-host --release \
    real_stage2_production_artifact_round_trip -- --ignored --nocapture

Validation performed

The final optimization pass was validated with:

  • cargo check -p flock-stage3-host;
  • cargo test -p flock-stage3-host:
    • 54 passed;
    • 13 expensive tests ignored;
  • the release complete-production regression above;
  • successful valid proof verification and corrupted-proof rejection;
  • the allocating-vs-in-place differential witness test;
  • cargo fmt --all; and
  • git diff --check.

The broader branch integration also includes:

  • deep binary-FRI relation evaluation at 9, 16, and 30 rounds;
  • real Stage 2 root lowering through the public preflight API;
  • cargo clippy -p flock-stage3-host --all-targets -- -D warnings;
  • cargo check -p flock-stage3-host --all-targets;
  • cargo check -p ix-ffi;
  • cargo check -p ix-ffi --features flock;
  • cargo clippy -p ix-ffi --features flock -- -D warnings;
  • nix develop --command lake build Ix.Cli.FlockRootCmd Main;
  • IX_FLOCK=1 nix develop --command lake exe ix flock-root --help; and
  • graceful invalid-address handling for both flock-root and
    compress-root.

Scope and remaining work

The relation is deliberately specialised to the current Ix protocol:

  • 18 claim words;
  • binary FRI;
  • cap height zero;
  • constant final-polynomial configuration; and
  • an exact circuit activation and height shape.

Before freezing a deployment relation, we still need to:

  • run preflight and proof generation on a full production ix_aggr root;
  • record peak memory, proving time, relation capacity, and artifact size;
  • measure capacity across a representative aggregate-proof corpus;
  • add production-sized differential vectors with nonzero grinding;
  • independently review the local Boolean R1CS tables and Flock soundness
    profile; and
  • export canonical fixed-verifier inputs for Stage 4 witness generation.

The remaining local setup bottleneck is immutable shape construction: in the
latest run, table declaration and final circuit wiring consumed almost all of
the 1.896-second preflight. The exact-witness cache eliminates repeated work
for the CLI's preflight/prove/verify sequence. Reuse across different aggregate
proof values with the same structural layout will require separating
per-proof input/public extraction from Flock's immutable CircuitShape setup.

The next architectural boundary is Stage 4: compile verification of this fixed
Flock relation into the universal-setup FFLONK/PLONK development backend,
measure constraints and Ethereum gas, and retain the ability to switch the
same stable statement to a circuit-specific Groth16 endpoint if required.

johnchandlerburnham and others added 26 commits August 27, 2026 16:21
Add the deterministic lift/flat/structural activation matrix and keep dummy calls deferred. Split aggregate recursion commitment/FRI configuration from IxVM defaults, share it between proving and verification, and pin the future cache encoding without changing active protocol parameters.
Precompute versioned per-slot cache keys, persist lift and join wrappers in the content-addressed store, and reuse entries only after exact claim and native outer-proof verification. Add safe proof decoding, corruption recovery, and --no-cache.
Execute ready lift and join slots as a dependency DAG under explicit job and RAM admission. Add calibration-pending slot weights, failure draining, CLI controls, and serial/parallel scheduler and proof-equivalence gates.
Add an opt-in two-child benchmark that proves singleton CheckEnv shards, lifts both proofs, and measures a verified flat join. Wire join metrics through reporting and dashboards, and record the current pre-E2 lift-size baseline.
- rust-toolchain.toml channel: 1.92 → 1.98, with the matching fenix
  toolchain hash in flake.nix. The pinned fenix already carries the 1.98
  release manifest, so flake.lock needs no change (fenix's nixpkgs stays
  pinned via its lean4-nix follows, so the Lean toolchain is untouched).
- Drop clippy::from_iter_instead_of_collect from the workspace lints:
  removed in clippy 1.98 and now warns as unknown.
- Fix the warnings new clippy 1.98 lints surface across the workspace:
  chunks_exact(N) → as_chunks::<N>() where the chunk size is constant,
  descending sort_by → sort_by_key(Reverse(..)), iteration over map
  values via .values(), map().unwrap_or() → map_or(), a checked
  division, an unwrap-after-is_some restructured into if-let, and
  assorted redundant-reference/pattern cleanups (mostly cargo clippy
  --fix). The two byte-gadget files keep their chunks_exact warnings
  until the next commit, which rewrites those regions anyway.

CI derives its Rust version from rust-toolchain.toml, so no workflow
changes are needed.
Companion to multi-stark's update-p3 branch (c72d321 → 249b740), which
carries four soundness/robustness fixes and the Plonky3 v0.6.0 bump
(pruned FRI Merkle multiproofs: ~2x faster verification, 40-70% smaller
proofs; canonical Goldilocks serde removes proof-byte malleability).
Proofs and verifying keys are not compatible with the previous pin.

Integration:

- Lookup gained max_multiplicity, a declared per-row bound on the
  multiplicity's integer magnitude feeding the newly enforced logUp
  height bound Σ wᵢ·hᵢ + |claims| < p. Function-circuit slots accumulate
  mutually-exclusive branch selectors, so they declare 1; committed
  count columns (function return slots, the memory circuit, the byte
  gadget tables) declare the new COUNT_COLUMN_BUDGET (2^32 queries per
  entry).
- The VK wire format carries the bound: u64 LE max_multiplicity per
  lookup, between the multiplicity node id and the arg count. The
  in-circuit VK deserializer (Ix/MultiStark/SystemDeserialize.lean)
  parses past it; the value is bound through the vk digest but the
  height bound itself is not yet enforced in-circuit (the native
  verifier enforces it).
- Message fingerprints are width-bound by default upstream (the slot
  width seeds the Horner fold), which is incompatible with aiur's
  branch-shared lookup slots: mutually exclusive branches superpose
  messages of different natural widths into one slot at the maximum
  width, so a narrow call is sent zero-padded to a width its callee's
  return slot never provides, and proving fails with
  UnbalancedChannel (pinned as the prove_verify_mismatched_call_widths
  regression). Aiur instead declares WidthBinding::ByConstruction —
  the plain Horner fold, restoring zero-padding transparency — and
  takes on the prefix-freeness contract that makes it sound: every
  message's natural width is a function of its constant-constrained
  leading prefix (channel tag plus discriminator: fun_idx fixes
  2+in+out, the memory size coordinate fixes 3+size, each gadget tag
  fixes its table width), so zero-extension can only equate a padded
  message with its own natural form. The contract is documented at the
  channel constants in lib.rs; the declaration is applied in
  AiurSystem::build and mirrored in the vk_codec decoder so decoded
  VKs replay the same transcript. ByConstruction is also exactly the
  fold the in-circuit verifier's logup_fingerprint already computes,
  so the recursive verifier needs no fingerprint change.
- The policy is Fiat-Shamir-bound as the first observe_shape word; the
  in-circuit transcript replay prepends the matching limb.
- aiur_multi_stark.rs regenerated (ix codegen) for the deserializer
  and transcript changes.

- P3 v0.6.0 ships FRI query openings as pruned Merkle multiproofs,
  while the in-circuit verifier consumes one authentication path per
  query (its per-query control flow is a far smaller circuit than the
  amortized multiproof walk). Rather than porting the walk into the
  DSL, the proof advice stays in the per-query transport: multi-stark's
  new advice module re-encodes a natively-verified proof by running
  p3's own verification with a recording compression function and
  reading each query's path back out of the recorded digest map. The
  advice bytes are untrusted verifier input, never digest-bound —
  the transcript binds the commitments and every expanded sibling is
  authenticated against them per query — so pruning vs expansion is
  pure transport and the encoding choice is sound. AiurSystem gains
  proof_to_advice_bytes (FFI: AiurSystem.proofToAdviceBytes); the
  recursive-verifier test feeds it instead of Proof.toBytes, whose
  native wire format is still round-tripped separately. The Lean-side
  proof grammar and the codegen'd verifier are byte-identical to
  before — no in-circuit changes.

Claim layout, the VK wire format above and aiur's public semantics are
otherwise unchanged; the policy adds no prover or verifier work over
the previous pin.

Still open, native-verifier-only: the logUp height bound is parsed
past but not yet enforced in-circuit (a wide-arithmetic check, tracked
separately).
The Lean v4.33.1 update pinned both dependencies at revisions that
predate their Rust 1.98 bumps; their heads now carry those bumps, which
this workspace needs since rust-toolchain.toml moved to 1.98. Both
revisions stay on leanprover/lean4:v4.33.1.

- Blake3.lean 1b0fbd2 → e6e908b (Rust 1.98, plus a case-insensitive
  source-directory fix), updated in lakefile.lean, lake-manifest.json
  and the blake3-lean flake input. The revision keeps the
  `blake3_rs_shared` target the `ix_native_decide_dynlib` pin requires.
  The inherited entry in Benchmarks/Compile/lake-manifest.json was still
  on the pre-4.33.1 revision and now tracks the root pin.
- lean-ffi 2a9c91e → 93c7e52 (Rust 1.98). Only bignat reaches the sp1
  and zisk workspaces, so their lock files move that one package.
The !benchmark recursive phase reported n/a for every fri-verifier
metric: Benchmarks/Typecheck.lean still fed Proof.toBytes — the pruned
multiproof wire format — to executeMultiStark/proveMultiStark, so the
in-circuit verifier rejected on parse and the harness (correctly) left
the recursive fields absent rather than emit a fake datum. The
in-circuit verifier consumes the per-query advice transport
(AiurSystem.proofToAdviceBytes); proofBytes stays the reported
proof-size metric.

bench-recursion-debug had the same advice-format gap plus a stale
claim recipe: it still built the public input as 32 raw digest bytes,
predating the ClaimHarness.packedDigestKey packing bench-typecheck
uses (its own out-of-circuit sanity check failed with
InvalidPowWitness — a wrong claim diverges every challenge — and the
advice re-encoder refused the proof for the same reason). Both aligned
with the typecheck flow.

Validated end-to-end at production parameters (numQueries 100,
query PoW 20, blowup 2) on Nat.add_comm: inner prove, advice
re-encoding, and the codegen'd in-circuit verifier accepting.
Keep compact proof bytes at storage and cache boundaries, expand them only for recursive lift/join advice, reject unsupported zero-query benchmarks, and decode legacy store proofs without panicking. Refresh tests, benchmark pins, and Rust 1.98 lint compatibility.
Add Ix/Aggr, a recursive aggregation system for IxVM shard proofs that
keeps Ix/MultiStark untouched and Ix-agnostic. One entrypoint, ix_aggr,
subsumes lifting and joining: a one-byte advice shape selects wrap or
binary join over any mix of IxVM and ix_aggr children, so shard proofs
enter the recursion system directly as join children and the dedicated
lift stage disappears.

Circuit (Ix/Aggr/Circuit.lean): every shape verifies its children in
full (verify + ood_verify from the shared Multi-STARK verifier modules)
against the vk its hinted kind demands. Identity is one 80-byte
digest-bound blob - blake3(ixvm vk) || verify_claim idx ||
blake3(self vk) || ix_aggr idx - carried unchanged at every node; self
children must bind the identical blob digest, pinning both vks and both
entrypoint indices transitively. Claims are a uniform 18-word
[0, aggr_idx, allowed(8), checkEnv(8)] at every depth, so proofs of
different tree levels combine freely. Wrap shapes bind the output
digest to the child CheckEnv digest directly; pair shapes open both
CheckEnv preimages, re-root the canonical subject/assumption trees, and
prove subjects = L ∪ R, assumptions = (asmL ∪ asmR) ∖ subjects with
linear sorted merges.

Toplevel (Ix/Aggr.lean): ixAggr = MultiStark.multiStarkFull + circuit,
pruned to ix_aggr, so verify_multi_stark_proof and other unrelated
entries no longer pad aggregate proofs. The host half of the wire
contracts (allowed blob, public input packing, shape codes, keyed
preimage/tree blob framing, interpreter IO assembly) lives beside the
toplevel; Ix/Aggr/Host.lean folds CheckEnvTrees statements.

Native path: ix codegen gains the ix-aggr target
(crates/ixvm-codegen/src/aiur_ix_aggr.rs, 244 fns); its runner builds
the seven-channel IO buffer natively and routes execution through the
generated code. New FFI rs_aiur_ix_aggr_execute/_prove
(executeIxAggr/proveIxAggr) pass proofs, vks, claims, and the compact
count/key/length preimage/tree blobs without per-byte boxing, plus
Proof.ofBytesChecked for store-boundary decoding.

CLI: ix aggr --ixe E --ixes M <shard-proof>... reconstructs every
nonempty shard statement from the env, matches wrappers by claim
digest, natively pre-verifies them, folds a balanced bisection (the
canonical fold makes the root claim independent of tree shape), wraps
single-shard roots so the persisted root is always an ix_aggr proof,
checks the root closes over the env canonical tree with no residual
assumptions, and persists the wrapper.

Tests (lake test -- ix-aggr, 17 cases): all five shapes accept over
real Multi-STARK stand-in child proofs from two distinct-vk systems;
codegen'd execution matches the interpreter on output and per-circuit
query counts for wrap and pair; negatives break one binding each -
lying shape hint, tampered proof, foreign identity, wrap statement
drift, dropped assumption, padded subject set, and tree advice not
reproducing its keyed root.
Replace the legacy lift/join entrypoints with the heterogeneous ix_aggr system across aggregate, verification, cache, and codegen paths. Preserve wrap-first and direct-join policies behind one proof identity and cache namespace.

Add the converged 91-check semantic suite and a deterministic 132-case activation audit covering shapes 0 through 9 twice, with no unobserved circuits.
Replace the legacy two-shard three-entrypoint benchmark with a four-shard manifest-subtree harness for production ix_aggr wrap-first and direct policies. Enforce the q=100 serialized no-cache profile, natively verify every input and recursive output, persist ordinary aggregate wrappers, and emit resumable per-slot JSON metrics.

Port bench-typecheck --join to ix_aggr direct shape 2 while preserving the stable join metric schema. Remove the final legacy slot-spec/preimage shim and pin both M1-f four-shard plans in the focused suite.
Remove the fixed eight-round FRI ceiling, add deep-round relation regressions, and expose aggregate-root preflight diagnostics.

Add the optional Flock CLI/FFI bridge with shared canonical root preparation, statement pinning, proof verification, and atomic artifact output.
Instrument the complete production regression across setup, proving, artifact transport, valid verification, and negative checks.

Document the measured 524.315-second breakdown and clarify that corrupted-proof rejection performs a second full verifier run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants