Skip to content

feat(l1): upgrade stateless-validator guests to ERE v0.17.0 and pin ere by tag - #7249

Open
ilitteri wants to merge 3 commits into
mainfrom
chore/bump-ere-v0.17.0
Open

feat(l1): upgrade stateless-validator guests to ERE v0.17.0 and pin ere by tag#7249
ilitteri wants to merge 3 commits into
mainfrom
chore/bump-ere-v0.17.0

Conversation

@ilitteri

@ilitteri ilitteri commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Motivation

ERE v0.17.0 is out (v0.16.3 fixed the OpenVM zkvm accelerator, v0.17.0 added zkVMProver::execute_estimated_cost).

Separately, the ere version is stated in the guest manifests, in zkvm-version.sh and in the release workflow's ERE_TAG, and nothing compares the last of those to the others. A bare rev is also not something the release workflow can act on: it pulls ere-compiler-<zkvm> and ere-server-<zkvm> by image tag, so the rev and the image tag are two independent statements of one fact. If they disagree, the guests are compiled against one ere and executed under another, which yields a plausible ELF rather than a failure.

Description

  • Move all four stateless-validator manifests from rev = "8961a4e7…" to tag = "v0.17.0", and regenerate the four lockfiles.
  • zkvm-version.sh already declared the ere tag and rejected a guest manifest that had moved off it. It now also checks the parent manifest, not just bin/<zkvm> — the parent pins ere-platform-core while each bin pins its own ere-platform-<zkvm>, so a drift between them puts two ere versions in one guest, and checking only the bin being asked about let that through.
  • Add --ere-tag to that script so the release workflow derives ERE_TAG from it instead of keeping a second copy of the version.
  • Add make check-ere-pins, wired into the existing check-cargo-locks job, so a disagreement surfaces on the pull request rather than at release time.

No zkVM SDK moves in this range — OpenVM stays v2.1.0-preview, SP1 v6.4.0, ZisK 1.1.0-alpha — so the version table and the release artifact names are unchanged.

Worth noting for reviewers: the only ere change in this range that touches a crate the guests link is crates/platform/openvm/src/zkvm_accelerator.rs, and that file is behind the zkvm-accelerator feature this build disables (we keep ethrex's own OpenVM crypto provider). The ELF bytes may still differ, since they are produced by a different compiler image. build-stateless-validator-guest runs on pushes to main and tags, not on pull requests, so the guest compile itself is exercised after merge.

Checklist

  • Updated STORE_SCHEMA_VERSION (crates/storage/lib.rs) if the PR includes breaking changes to the Store requiring a re-sync. — not applicable, no Store changes.

v0.16.3 fixes the OpenVM zkvm accelerator and v0.17.0 adds
zkVMProver::execute_estimated_cost. Neither release moves a zkVM SDK: openvm stays
at v2.1.0-preview, sp1 at v6.4.0 and zisk at 1.1.0-alpha, so the version table in
zkvm-version.sh is unchanged and only the rev it is keyed on moves.
…h for which

ere release the stateless validator is built against.

The ere version is stated in the four stateless-validator manifests, in
zkvm-version.sh and in the release workflow, and nothing compares them. A bare rev
is also not something the release workflow can act on: it pulls
ere-compiler-<zkvm> and ere-server-<zkvm> by image tag, so the rev and the image
tag are two independent statements of the same fact. If they disagree the guests
are compiled against one ere and executed under another, which produces a
plausible ELF rather than a failure.

The manifests now pin tag = "v0.17.0", ere-tag.sh reads the tag back out of them
and fails if they disagree or if any regresses to a bare rev, and the release
workflow derives ERE_TAG from it instead of hardcoding a copy. zkvm-version.sh
keys its SDK table on the tag for the same reason it keyed on the rev before, so
bumping ere still forces the table to be revisited. make check-ere-pins runs in
the check-cargo-locks job so a disagreement surfaces on the pull request rather
than at release time.
@ilitteri
ilitteri requested a review from a team as a code owner September 1, 2026 21:22
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

rpc-compat log-bearing cases excluded

Where: KNOWN_EXCLUDED_TESTS in .github/scripts/check-hive-results.sh counts out
eight hive rpc-compat cases — the four eth_getLogs cases, eth_getBlockReceipts/get-block-receipts-latest,
and three eth_getTransactionReceipt cases. They are exactly the cases whose recorded
response contains at least one log object; every case with an empty log array still runs.
Note this leaves eth_getLogs with no rpc-compat coverage at all, since all four of its
cases are in the set.

Why: ethrex populates blockTimestamp on log objects, as geth, besu, nethermind, reth
and erigon all do. hive's rpc-compat compares responses byte-exactly (jsondiff.FullMatch;
the lenient checkJSONStructure path applies only to cases upstream marks speconly), and
the corpus is pinned to execution-apis d08382ae (2025-02-10), whose recordings predate the
field — it entered the schema in execution-apis#639 and the fixtures in #846 (2026-07-22).
So the extra key cannot match, and this is a property of the pin rather than of the response.

The pin cannot move, and this is not temporary. The pin sits one commit before
execution-apis#627, which moved the test chain to a pre-merge genesis: the current corpus has
~36 proof-of-work blocks before its terminal total difficulty. ethrex does not support
pre-merge chains and will not, so importing that chain.rlp fails at block 1 —
validate_block_header has no pre-London base-fee path. Every revision carrying
blockTimestamp in its fixtures also carries that chain, so there is no revision that
satisfies both. Nor can the corpus be patched locally: rpc-compat's Dockerfile clones
ethereum/execution-apis by hard-coded URL, so the branch buildarg cannot point at a fork.

Coverage: the field itself is pinned by
block_timestamp_is_on_the_log_and_not_on_the_receipt in
crates/networking/rpc/types/receipt.rs, which asserts it is present on each log and absent
from the receipt level.

Removal: delete the entries if ethrex ever gains pre-merge chain import, or if upstream
marks these cases speconly so they are type-checked instead of compared byte-for-byte.


The stateless schema id does not identify the encoding

Where: STATELESS_INPUT_SCHEMA_ID in crates/common/types/stateless_ssz.rs.

Upstream keeps the stateless input schema id at 0x1501
(fork_index 0x15 << 8 | revision 0x01) across incompatible body changes. Three
encodings have now shipped under it: tests-zkevm@v0.6.2, then #3248 + #3278,
then #3356, which moved state, codes and public_keys from SszList to
ProgressiveList. ethrex speaks the last one.

The consequence is that the 2-byte prefix cannot be used to detect a stale or
mismatched bundle. A wrong-dialect input is accepted by the id check and then
fails later — in SSZ decode, or on a root that does not match — rather than being
rejected up front for what it is. only_amsterdam_schema_id_decodes therefore
proves less than its name suggests.

Worth raising upstream: a revision field that does not move across a body change
provides no version negotiation at all.


ZisK guest program hash changes with the unsync_cell gate

Where: crates/common/types/block.rs, transaction.rs.

The gate on the single-threaded unsync_cell::OnceCell moved from
all(feature = "eip-8025", target_arch = "riscv64") to
all(feature = "zisk", target_arch = "riscv64") when the eip-8025 feature was removed.

The guest ELFs were previously built --features "<zkvm>-build-elf,ci", which never enabled
eip-8025, so they compiled the atomic once_cell variant. bin/zisk/Cargo.toml does enable
ethrex-common/zisk, so the ZisK guest now compiles the unsafe impl Sync cell instead.
That changes the ELF bytes and therefore the program hash and verification key.

This is intended (the guest is single-threaded, so the unsync cell is sound and cheaper), but it
is a VK change rather than a no-op refactor, and the diffstat presents it as a file rename
(eip8025_cell.rsunsync_cell.rs). Anyone pinning a ZisK VK across this change must
re-register it. The stateless-validator crate now forwards ethrex-common/zisk from its own
zisk feature so the two ZisK guests do not disagree on the cell type.


Release signing key is an unprotected repository secret

Where: .github/workflows/tag_release.yaml.

MINISIGN_SECRET_KEY is a plain repository secret. There is no environment: on
finalize-release or dry-run-release-assets, and gh api repos/lambdaclass/ethrex/rulesets
shows only branch-targeted rulesets, so the github.ref_type == 'tag' condition is a workflow
check rather than an enforced boundary: anyone who can push a tag can reach the signing key.

This is a repository-settings change, not a code change, so it is recorded here rather than
fixed in the tree. Recommended:

  1. Move MINISIGN_SECRET_KEY / MINISIGN_PASSWORD into a GitHub Environment with required
    reviewers, and add environment: to the two jobs that sign.
  2. Add a ruleset targeting refs/tags/v* restricting who may create release tags.

Until then, the compromise of that key is silent and durable: signatures would still verify
against the committed .github/minisign.pub.

@github-actions github-actions Bot added the L1 Ethereum client label Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which changes ere dependency pinning from bare git revisions to version tags, adding verification scripts to prevent silent divergence between compiler and runtime images.

Overall Assessment

This is a well-structured DevOps/infrastructure PR that fixes a real operational hazard: silent version skew between the ere compiler image used to build guest ELF binaries and the ere server image used to execute them. The solution is sound—derive image tags from manifest pins rather than hardcoding them separately.


Detailed Findings

1. Shell Script Robustness: ere-tag.sh — Greedy regex could match across dependency blocks

File: .github/scripts/ere-tag.sh, lines 35-41

found=$(
    grep -oE 'git = "https://github\.com/eth-act/ere"[^}]*' "$manifest" \
        | grep -oE 'tag = "[^"]+"' | grep -oE '"[^"]+"' | tr -d '"' | sort -u \
        || true
)

The regex [^}]* assumes dependency entries use braces and won't match across } boundaries. However, Cargo.toml dependency specifications can span multiple lines with nested braces in features arrays:

ere-platform-openvm = { git = "https://github.com/eth-act/ere", tag = "v0.17.0", default-features = false, features = [
    "std",
    "getrandom-unsupported",
] }

The [^}]* would stop at the first }—but in this case that's actually fine since the tag appears before the features = [ array. More concerning is this pattern in the base manifest:

ere-platform-core = { git = "https://github.com/eth-act/ere", tag = "v0.17.0", optional = true }

This is single-line, so [^}]* works. But consider if someone adds a comment or reformats:

ere-platform-core = { 
    git = "https://github.com/eth-act/ere", 
    tag = "v0.17.0", 
    optional = true 
}

Here [^}]* correctly captures to the closing brace. The regex is adequate but fragile.

Suggestion: Add a comment warning that this assumes tag appears on the same line or before any closing brace, or use a more robust approach with cargo metadata or toml parsing. Given the macOS bash 3.2 constraint mentioned in the script, the grep approach is defensible, but document the limitation.


2. Shell Script Bug: count computation is fragile with empty input

File: .github/scripts/ere-tag.sh, line 43

count=$(printf '%s' "$found" | grep -c . || true)

When $found is empty, printf '%s' "" outputs nothing, grep -c . returns 0 with exit code 1, and || true suppresses the error. The result is count=0. This works correctly.

However, if $found contains only newlines (edge case from sort -u on malformed input), grep -c . counts non-empty lines, which is correct behavior.

More serious issue: If $found is a single tag with trailing newline, count=1. Correct. But the || true on the outer command substitution (line 41) means if grep -oE 'git = ...' fails entirely (e.g., file unreadable), $found is empty and we get the "pins without tag" error rather than a pipefail abort. This is intentional per the comment, but consider:

  • grep failing due to missing file is caught by the [[ -f $manifest ]] check
  • grep failing due to no match is the case we want to diagnose

Verdict: Acceptable given the comment explaining the design.


3. Makefile: Target check-ere-pins has incorrect path in comment

File: Makefile, lines 279-281

check-ere-pins: ## 🔍 Check the stateless-validator guests agree on one ere release
	# Prints the tag on success. Fails if the manifests disagree, or if any of
	# them pins ere by bare rev -- a rev cannot be matched against the ere
	# compiler/server image tags the release workflow pulls.

The comment says "a rev cannot be matched against the ere compiler/server image tags"—but the script ere-tag.sh doesn't actually check for bare rev pins explicitly. It checks for tag = "..." after finding git = "https://github.com/eth-act/ere". If someone uses a bare rev without tag, the first grep would match, then the second grep -oE 'tag = "[^"]+"' would find nothing, resulting in count=0, which triggers the "pins without a tag" error.

So the behavior is correct (bare rev fails), but the comment in the Makefile is slightly misleading about how it works. The script doesn't distinguish "bare rev" from "no ere dependency at all"—both yield count=0.

Suggestion: Clarify the comment, or better, make ere-tag.sh explicitly detect and report bare rev pins for better diagnostics:

# After finding git = ... lines, check if any lack tag = ...
bare_rev=$(
    grep -oE 'git = "https://github\.com/eth-act/ere"[^}]*' "$manifest" \
        | grep -v 'tag = ' || true
)
if [[ -n $bare_rev ]]; then
    echo "$manifest pins ere by bare rev instead of tag." >&2
    exit 1
fi

4. Security: Script injection via GITHUB_ENV

File: .github/workflows/tag_release.yaml, lines 176-177

      - name: Resolve ere tag from the guest manifests
        run: echo "ERE_TAG=$(.github/scripts/ere-tag.sh --docker)" >> "$GITHUB_ENV"

If ere-tag.sh --docker outputs malicious content (e.g., via a compromised manifest), this could inject arbitrary environment variables. However:

  • The script validates output with grep -oE '"[^"]+"' | tr -d '"', so output is constrained to the tag value
  • The --docker variant strips v prefix: ${tag#v}
  • Manifests are source-controlled and reviewed

Risk: Low. But consider using GitHub Actions' GITHUB_OUTPUT with explicit step output instead of GITHUB_ENV, or add validation:

      - name: Resolve ere tag from the guest manifests
        id: ere
        run: |
          tag=$(.github/scripts/ere-tag.sh --docker)
          [[ "$tag" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$ ]] || { echo "Invalid tag format: $tag" >&2; exit 1; }
          echo "tag=$tag" >> "$GITHUB_OUTPUT"

Then reference ${{ steps.ere.outputs.tag }} instead of $ERE_TAG.


5. Consistency: zkvm-version.sh still uses hardcoded table

File: .github/scripts/zkvm-version.sh, lines 20-26

zkvm_version() {
    case "$1" in
        zisk)   echo "v1.1.0-alpha" ;;
        sp1)    echo "v4.0.0" ;;
        openvm) echo "v1.0.0" ;;
        *)      echo "unknown zkvm: $1" >&2; exit 1 ;;
    esac
}

This hardcoded table "mirrors what ere-catalog resolves at the pinned ere tag." This is still a potential source of divergence—if ere-catalog changes its resolution, this table becomes stale.

The PR description says "a silent bump cannot mislabel an artifact" because the script checks the manifest still matches ERE_TAG. But the ZKVM version itself could be wrong if the table isn't updated when ere-catalog changes.

Suggestion: Consider if ere-catalog can be queried at runtime to eliminate this second source of truth. If not, add a prominent comment that this table must be updated whenever ere-catalog changes its dependency resolution.


6. Missing Update: check-cargo-lock path inconsistency

File: Makefile, line 285

check-cargo-lock: ## 🔍 Check Cargo.lock files are up to date
	cargo metadata --locked > /dev/null
	cargo metadata --locked --manifest-path crates/guest-program/bin/sp1/Cargo.toml > /dev/null

The path crates/guest-program/bin/sp1/Cargo.toml doesn't match the structure shown in the diff. The manifests are at:

  • crates/guest-program/stateless-validator/Cargo.toml
  • crates/guest-program/stateless-validator/bin/sp1/Cargo.toml

This appears to be a pre-existing bug not introduced by this PR, but worth noting. The check-cargo-lock target may be checking a non-existent path or an old location.


7. Cargo.toml Comment Staleness

File: crates/guest-program/stateless-validator/Cargo.toml, lines 26-28

# ere platform abstraction (entrypoint / IO / cycle scopes) used by the per-zkVM
# bins. Pinned to one ere rev; the ere-compiler and ere-server image tag in
# tag_release.yaml selects the matching ERE release.

Comment still says "Pinned to one ere rev" but it's now pinned by tag. Also, tag_release.yaml no longer hardcodes the image tag—it derives it from this manifest.

Suggestion: Update comment:

# ere platform abstraction (entrypoint / IO / cycle scopes) used by the per-zkVM
# bins. Pinned to one ere tag; the release workflow derives the ere-compiler
# and ere-server image tags from this pin, so they cannot silently diverge.

Summary

Item Severity Status
1. Greedy regex fragility Low Acceptable with documentation
2. Empty input handling None Working as intended
3. Makefile comment accuracy Low Minor clarification needed
4. GITHUB_ENV injection risk Low Consider hardening
5. Hardcoded ZKVM version table Medium Design limitation, document
6. check-cargo-lock path bug Medium Pre-existing, fix separately
7. Stale comment in Cargo.toml Low Fix in this PR

The PR is fundamentally sound and addresses a real operational risk. The most important fix is Item 7 (stale comment), which could mislead future maintainers about the pinning mechanism.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

No findings.

I reviewed the operational path around the new source-of-truth change and didn’t find a correctness or security regression in the PR. The new guard in .github/scripts/ere-tag.sh correctly rejects missing or conflicting ere tags across the stateless-validator manifests, the release workflow now derives ERE_TAG from those manifests instead of a stale hardcoded value in tag_release.yaml, and CI now checks that invariant before release in pr-main_l1.yaml and Makefile.

I also executed the scripts locally against the checked-out manifests: ere-tag.sh resolved v0.17.0 / 0.17.0 as expected, and zkvm-version.sh resolved all three guest artifact versions. Residual risk is low and mostly maintainability-related: zkvm-version.sh still has a deliberately hardcoded ERE_TAG table guard, so future ere bumps must update that script in lockstep or release will fail fast.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Good, executable bit is set correctly. I have everything needed for the review.

Review: PR 7249 — Pin ere by tag, add check-ere-pins

This is a CI/tooling PR (no consensus-critical code paths), and overall it's a clean, well-motivated fix that closes a real drift hazard (rev vs. tag vs. hardcoded ERE_TAG disagreeing silently). The ere-tag.sh script's error messages are clear and the design of deriving everything from the manifests is sound. One finding undermines the core guarantee the PR is trying to add, though.

1. ERE_TAG failure is silently swallowed in the release workflow (High)

.github/workflows/tag_release.yaml:180:

run: echo "ERE_TAG=$(.github/scripts/ere-tag.sh --docker)" >> "$GITHUB_ENV"

Under set -e (the default for GitHub Actions run: steps), a failing command inside a command substitution does not fail the enclosing simple command — the exit status of echo "...$(cmd)..." is echo's own status (always 0), not cmd's. This is a well-known bash pitfall (errexit only propagates a substitution's failure when the substitution is the entire right-hand side of a variable assignment, e.g. x=$(cmd)).

So if the manifests ever disagree (or one regresses to a bare rev), ere-tag.sh --docker fails, but this step does not fail — it just sets ERE_TAG= (empty) and proceeds. The clear diagnostic ("ere pins disagree: ...") this PR was built to surface is lost. The pipeline will eventually fail later at docker pull ghcr.io/.../ere-compiler-${{ matrix.zkvm }}: (empty tag → invalid reference format), but with a confusing Docker error instead of the intended actionable message — right at release time, which is exactly the failure mode this PR set out to eliminate.

Contrast with the Makefile target (check-ere-pins), which gets this right by capturing into a variable first:

@tag=$$(.github/scripts/ere-tag.sh) && echo "ere pins agree: $$tag"

(Here tag=$(...) is an assignment-only command, so its exit status does propagate.)

Fix for the workflow step:

run: |
  ERE_TAG=$(.github/scripts/ere-tag.sh --docker)
  echo "ERE_TAG=$ERE_TAG" >> "$GITHUB_ENV"

2. Stale comment now describes the old (backwards) design (Low)

crates/guest-program/stateless-validator/Cargo.toml:25-27 (unchanged context in this diff):

# ere platform abstraction (entrypoint / IO / cycle scopes) used by the per-zkVM
# bins. Pinned to one ere rev; the ere-compiler and ere-server image tag in
# tag_release.yaml selects the matching ERE release.

This wasn't touched by the PR, but it's now inaccurate in two ways: the dependency is pinned by tag, not rev (as the line right below it shows), and the relationship is now inverted — tag_release.yaml derives its ERE_TAG from this manifest via ere-tag.sh, not the other way around. Since this PR is specifically about eliminating this kind of "two independent statements of one fact" drift, it's worth updating this comment too.

3. ere-tag.sh's extraction is line-based, so tag must stay on the same line as git = (Low/nit)

.github/scripts/ere-tag.sh:34-38 uses grep -oE without -z, so [^}]* can't span newlines. It currently works for all four manifests only because tag = "..." happens to be on the same line as git = "https://github.com/eth-act/ere" — including in bin/openvm/Cargo.toml, where the dependency table itself spans multiple lines (the features = [...] array wraps, but git/tag are on line 1). If a future formatting pass (e.g. taplo) ever wraps tag onto its own line, the script would report "pins ere without a tag" for a manifest that has one. Not a bug today, but worth a one-line comment noting the constraint, or a tr -d '\n'-based join, so it doesn't silently regress.

Everything else looks solid

  • zkvm-version.sh's rev→tag rename is consistent and the guard logic (refuse to guess if the manifest has moved off the pinned tag) is preserved correctly.
  • The four Cargo.lock/Cargo.toml bumps to v0.17.0 are consistent with each other and with the new lockfile source URLs (?tag=v0.17.0#5023513...).
  • check-ere-pins is correctly wired into the existing check-cargo-locks job, after checkout, so it surfaces on PRs as intended.
  • New script is executable (100755) and uses set -euo pipefail plus ${1:-} safely under set -u.

Automated review by Claude (Anthropic) · sonnet · custom prompt

ere-tag.sh re-derived agreement that zkvm-version.sh already enforced: it declares
the ere tag and fails when a guest manifest does not pin it, so pairwise agreement
between those manifests follows from that check rather than needing its own.

What the separate script did add is kept. zkvm-version.sh now checks the parent
manifest alongside the three bins, which matters because the parent pins
ere-platform-core while each bin pins its own ere-platform-<zkvm>, so a drift
between them puts two ere versions in one guest and checking only the bin being
asked about would let that through. It also grew --ere-tag, so the release
workflow can derive ERE_TAG rather than repeating it, and make check-ere-pins runs
the same check on pull requests.
# The table above is only valid for ERE_TAG; refuse to guess if anything moved.
for manifest in "${MANIFESTS[@]}"; do
[[ -f $manifest ]] || { echo "no manifest at $manifest" >&2; exit 1; }
grep -q "tag = \"$ERE_TAG\"" "$manifest" && continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should match against git = "https://github.com/eth-act/ere", tag = "$ERE_TAG" here, in case there's another dependency tagged by tag and version as ERE

@github-project-automation github-project-automation Bot moved this to In Review in ethrex_l1 Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

2 participants