fix: sync ream with lighthouse in a kurtosis devnet - #18
Open
perfogic wants to merge 17 commits into
Open
Conversation
perfogic
changed the title
perfogic
marked this pull request as ready for review
August 8, 2026 14:06
perfogic
force-pushed
the
feat/kurtosis-integration
branch
from
August 12, 2026 08:34
434a1c7 to
7a7c9dc
Compare
The `ubuntu` base ships no trust store, and reqwest panics while *building* a client when no system roots load, so the node died right after database initialization and never reached the network. This bites even on a plain-HTTP deployment such as a local Kurtosis devnet, because the panic happens in `Client::new()` rather than on a request.
EIP-4399 repurposed the execution header's `mixHash` slot to hold the beacon chain's `prev_randao`; the two layers just spell the same value differently. Rebuilding the header with a zero `mix_hash` made the recomputed hash differ from every real block, so `is_valid_block_hash` rejected each payload before the execution engine was ever consulted — the node imported nothing and its EL never saw a single `newPayload`, while the logs showed only a generic condition failure. Add golden vectors built from a real block so a dropped or wrongly mapped header field fails loudly instead of silently stalling the node. Existing end-to-end tests could not catch this: they run against the mock execution engine, which never exercises this path.
A devnet with a genesis delay routinely starts nodes ahead of genesis, and two separate paths treated that as impossible: `on_tick` computed `time - genesis_time` on unsigned values. In a release build that wraps rather than panics, so `tick_slot` became ~u64::MAX and the slot catch-up loop effectively never ended. The network manager task sat inside `process_tick` forever: no gossip was processed at all, and the store's slot climbed quadratically — `/eth/v1/node/syncing` reported a sync distance in the tens of thousands on a chain that was a dozen slots long. `slot_n_days_ago` panicked outright. It runs on the block range syncer's task, and `current_epoch` now funnels through it as well, so the panic reaches gossip topic construction and message validation rather than one lone caller. Return early and clamp respectively, matching how other clients model pre-genesis: a node that starts early is at slot 0, not in an error state. Note that clamping is only safe here because `on_tick` returns before storing anything — writing the pre-genesis time would walk the store's slot backwards and hang startup.
Starting before genesis is normal and deliberate — p2p, discovery and the HTTP API all stay up so the gossip mesh is formed by slot 0 — but until now the node looked indistinguishable from one that was wedged. The tick keeps doing its work while waiting: `on_tick` is already a no-op before genesis, and skipping the rest freezes the store's slot clock, which makes blocks arriving just after genesis look future-dated and drops them. Read genesis time from the store rather than the spec's MIN_GENESIS_TIME, so the countdown stays correct when GENESIS_DELAY is non-zero.
A fork digest that differs by one byte is invisible locally but fatal on the wire: peers reject our Status and we subscribe to the wrong gossip topics. Cover the EIP-7892 masking against Lighthouse's own vector, plus the two boundary behaviours that are easy to regress — the digest only changing on a BPO entry, and no masking at all before Fulu.
Payload transactions arrive as EIP-2718 envelopes and belong in the transactions trie exactly as they are. Encoding them through `Encodable` wrapped each one in a second RLP string header, so the computed root — and with it the header hash — matched no other client. Only blocks carrying at least one transaction were affected, which is why this survived alongside the `mix_hash` fix: an empty block hashes to the empty-trie root either way, so the node would import the first block of a devnet and then reject every block after it, reporting nothing beyond a parent-not-found cascade. Cover it with the transactions root of a real block, since the shape of the bug is that our answer is self-consistent and simply not the one everyone else computes.
`/eth/v1/node/syncing` nested the status one level deeper than the Beacon API spec, under an extra `sync_status` object. Spec-compliant consumers cannot parse that: assertoor reported "get node syncing not supported" and listed the node as offline while it was in fact fully synced and importing blocks.
`engine_newPayload` only hands a block to the execution layer for validation. Until a forkchoice update arrives it keeps its previous head, reports itself as syncing and answers `eth_*` queries from a chain that never advances — so a node that only follows the chain, rather than proposing on it, left its execution layer parked at genesis. Until now the call was made solely from the block production paths. Notify on each of the three import paths (gossip, and both routes where a block that was pending data availability becomes available). The forkchoice state is built while the store guard is held and the guard released before the request, so the lock is never held across the network call, and a failed notification only logs: the block is already imported and reverting that would be worse.
The endpoint returned four hardcoded entries where other clients return ~190. Tooling diffs this across clients to decide whether they are on the same chain, so the gaps read as disagreement rather than omission: assertoor reported "invalid node specs: spec mismatch" over a dozen fields and marked the node offline while it was following the chain. Serialize the configured network spec and add the preset and domain constants the node was compiled against, quoting every integer the way the Beacon API does. Gloas is reported as far future, which is how a client says "not scheduled" — leaving it out reads as a disagreement instead. Serializing the spec for the first time exposed `b32_hex` prefixing its output with `0x` on top of the `0x` that `hex::encode` already adds. Fork versions went out as `0x0x10000038`, which still passes for hex at a glance but is rejected wholesale.
Both were rejected with a 404, so `/eth/v1/beacon/headers/head` — the ordinary way to ask a node where it is — returned "This ID type is currently not supported" and any consumer polling it concluded the node was offline. Head resolves through fork choice, which reads entirely out of the database, and genesis through the slot index.
tosynthegeek
force-pushed
the
feat/kurtosis-integration
branch
from
August 18, 2026 08:02
7a7c9dc to
6574066
Compare
There was a problem hiding this comment.
Pull request overview
Improves Ream interoperability with Lighthouse in Kurtosis devnets.
Changes:
- Fixes pre-genesis timing, custody metadata, and container trust-store setup.
- Corrects execution payload hashing and adds execution fork-choice updates.
- Aligns Beacon API responses with expected formats and adds regression vectors.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
Dockerfile |
Installs CA certificates. |
crates/rpc/beacon/src/handlers/syncing.rs |
Fixes syncing response shape. |
crates/rpc/beacon/src/handlers/config.rs |
Expands specification response. |
crates/rpc/beacon/src/handlers/block.rs |
Supports head and genesis identifiers. |
crates/networking/p2p/src/network/beacon/mod.rs |
Advertises supernode custody. |
crates/networking/manager/src/service.rs |
Configures custody and pre-genesis logging. |
crates/common/network_spec/src/networks/beacon.rs |
Adds serialization and safe pre-genesis slots. |
crates/common/network_spec/src/b32_hex.rs |
Fixes duplicated hexadecimal prefixes. |
crates/common/network_spec/Cargo.toml |
Adds test dependency. |
crates/common/fork_choice/beacon/src/store.rs |
Prevents pre-genesis underflow. |
crates/common/fork_choice/beacon/src/handlers.rs |
Makes pre-genesis ticks no-ops. |
crates/common/execution/rpc_types/src/electra/execution_payload.rs |
Corrects execution-header and transaction roots. |
crates/common/execution/rpc_types/resources/transactions_root_block.txt |
Adds real transaction-root fixture. |
crates/common/consensus/misc/src/fork_data.rs |
Adds fork-digest vectors. |
crates/common/chain/beacon/src/beacon_chain.rs |
Sends execution fork-choice updates. |
crates/common/chain/beacon/Cargo.toml |
Adds execution RPC types dependency. |
Cargo.lock |
Records dependency changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ition/engine calls
Start validators and voluntary-exit epoch calculation at slot zero when the current time is before genesis.
Expose SECONDS_PER_SLOT, serialize terminal difficulty as decimal, and quote nested integer values.
Recompute the latest state under an ordering guard and deduplicate updates from blocks, attestations, slashings, and slot ticks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable Ream to interoperate with Lighthouse in a one-Ream/one-Lighthouse Kurtosis devnet.
prev_randaointo the execution headermix_hash. (we have some mismatch in EIP)B32fork versions with exactly one0xprefix./node/syncingand/config/spec.headandgenesisblock IDs.Scope
This PR targets Lighthouse interoperability. The devnet runs without blob load, so data availability is not exercised. Import throughput and finality performance remain follow-up work.
Result
To-Do