diff --git a/bin/mega-evme/AGENTS.md b/bin/mega-evme/AGENTS.md index 7dc0e946..6f010ef0 100644 --- a/bin/mega-evme/AGENTS.md +++ b/bin/mega-evme/AGENTS.md @@ -1,7 +1,7 @@ # AGENTS.md ## OVERVIEW -CLI toolbox for direct MegaEVM execution (`run`, `tx`, `replay`) with optional forking, tracing, and state dump workflows. +CLI toolbox for direct MegaEVM execution (`run`, `tx`, `replay`, `cache`) with optional forking, tracing, and state dump workflows. ## STRUCTURE - `src/main.rs`: CLI bootstrap and panic hook. @@ -9,7 +9,8 @@ CLI toolbox for direct MegaEVM execution (`run`, `tx`, `replay`) with optional f - `src/common/`: shared CLI args, state loading, tracing, tx parsing, output printers. - `src/run/`: bytecode execution command. - `src/tx/`: full transaction execution command with raw-tx override support. -- `src/replay/`: RPC-backed historical transaction replay through block executor. +- `src/replay/`: RPC-backed historical transaction replay through block executor, plus the batch driver. +- `src/cache/`: cache-file merge utilities (provider-cache and capture-envelope JSON shapes) backing the `cache merge` subcommand and the lock-protected merge-on-persist, plus the sidecar advisory lock every cache-file writer takes. ## KEY PATTERNS - Shared argument groups are flattened from `run` argument structs into sibling commands. @@ -31,3 +32,6 @@ CLI toolbox for direct MegaEVM execution (`run`, `tx`, `replay`) with optional f - Change state-forking or prestate merge semantics: `src/common/state.rs`. - Change replay hardfork/spec selection: `src/replay/{cmd.rs,hardforks.rs}`. - Change receipt/summary formatting: `src/common/outcome.rs` and printer helpers. +- Change cache merge behavior (CLI or merge-on-persist): `src/cache/{mod.rs,merge.rs}`. +- Change how cache files are locked against concurrent writers: `src/cache/lock.rs` — the one place a cache-file write may acquire its lock, and every caller must fail closed when it cannot. +- Change process exit classification: `src/common/exit.rs` — the single exit site for command results. diff --git a/bin/mega-evme/src/cache/lock.rs b/bin/mega-evme/src/cache/lock.rs new file mode 100644 index 00000000..1a4310d6 --- /dev/null +++ b/bin/mega-evme/src/cache/lock.rs @@ -0,0 +1,90 @@ +//! Advisory locking for cache files shared by concurrent processes. +//! +//! Every writer of a cache file — clean-exit persist and the offline +//! `cache merge` subcommand alike — takes the exclusive lock on that file's +//! sidecar before it re-reads the file, merges, and renames the result into +//! place. A writer that skips the lock can only be correct by luck: two +//! read-modify-write cycles that interleave lose whichever side renamed first. +//! +//! The lock lives on a sidecar (`.lock`) rather than the cache file +//! itself because the target is replaced by rename on every write, and a lock +//! held on the replaced inode protects nothing. + +use std::{ + fs, + fs::OpenOptions, + path::{Path, PathBuf}, +}; + +/// Path of the advisory lock sidecar for `target` (`.lock`). +pub(crate) fn lock_sidecar_path(target: &Path) -> PathBuf { + let mut os = target.as_os_str().to_owned(); + os.push(".lock"); + PathBuf::from(os) +} + +/// RAII exclusive lock on the sidecar file for a cache target. +/// +/// The lock is released when this guard is dropped (file handle closed). +/// The sidecar file itself is left on disk. +#[derive(Debug)] +pub(crate) struct ExclusiveFileLock { + _file: fs::File, +} + +/// Acquire an exclusive advisory lock on `.lock`, blocking until held. +/// +/// The sidecar is created if missing and left in place after unlock. +/// +/// Callers must fail closed on `Err`: an unlocked write is exactly the +/// lost-update race the lock exists to prevent, so a failed acquisition means +/// "do not write", never "write anyway". +pub(crate) fn acquire_exclusive_lock(target: &Path) -> std::io::Result { + let lock_path = lock_sidecar_path(target); + if let Some(parent) = lock_path.parent() { + fs::create_dir_all(parent)?; + } + // truncate(false): the sidecar is only a flock target; keep any existing bytes. + let file = + OpenOptions::new().create(true).read(true).write(true).truncate(false).open(&lock_path)?; + // Blocking exclusive advisory lock. + file.lock()?; + Ok(ExclusiveFileLock { _file: file }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_lock_sidecar_path_suffix() { + let p = Path::new("/tmp/rpc-cache-1.json"); + assert_eq!(lock_sidecar_path(p), PathBuf::from("/tmp/rpc-cache-1.json.lock")); + } + + /// The sidecar is created on acquisition and left in place after unlock. + #[test] + fn test_acquire_exclusive_lock_creates_and_keeps_sidecar() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("rpc-cache-1.json"); + let sidecar = lock_sidecar_path(&target); + assert!(!sidecar.exists()); + + let guard = acquire_exclusive_lock(&target).expect("acquire"); + assert!(sidecar.exists(), "sidecar created while held"); + drop(guard); + assert!(sidecar.exists(), "sidecar left in place after unlock"); + } + + /// An un-openable sidecar path surfaces as an error rather than a silent + /// unlocked write. + #[test] + fn test_acquire_exclusive_lock_reports_unopenable_sidecar() { + let dir = tempfile::tempdir().expect("tempdir"); + let target = dir.path().join("rpc-cache-1.json"); + // A directory in the sidecar's place cannot be opened as a file. + fs::create_dir(lock_sidecar_path(&target)).expect("occupy sidecar path"); + + acquire_exclusive_lock(&target).expect_err("un-openable sidecar must not silently succeed"); + } +} diff --git a/bin/mega-evme/src/cache/merge.rs b/bin/mega-evme/src/cache/merge.rs new file mode 100644 index 00000000..19386d34 --- /dev/null +++ b/bin/mega-evme/src/cache/merge.rs @@ -0,0 +1,1115 @@ +//! Pure merge helpers for provider-cache and capture-envelope JSON shapes. +//! +//! Used by the `cache merge` subcommand and by lock-protected merge-on-persist +//! in [`crate::common::provider`]'s cache store. + +use std::{ + collections::BTreeMap, + fs, + io::Write as _, + path::{Path, PathBuf}, +}; + +use alloy_primitives::B256; +use serde::{Deserialize, Serialize}; + +use crate::common::{EvmeError, Result}; + +/// Current on-disk envelope schema version (must match capture/replay). +pub(crate) const ENVELOPE_VERSION: u32 = 1; + +/// One `{key, value}` entry shared by provider-cache files and envelope `cache` arrays. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct CacheKv { + /// Request fingerprint (typically `keccak256` of method + params). + pub key: B256, + /// Serialized JSON-RPC response body. + pub value: String, +} + +/// Detected on-disk shape of a cache file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum CacheShape { + /// JSON array of `{key, value}` (provider `--rpc.cache-dir` files). + Provider, + /// `{version, chain_id, cache, external_env?}` capture envelope. + Envelope, +} + +/// Minimal envelope view used for merge (independent of the store type). +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub(crate) struct EnvelopeDoc { + /// Schema version (must match [`ENVELOPE_VERSION`] for this build). + pub version: u32, + /// Chain ID recorded at capture time. + pub chain_id: u64, + /// Transport-level cache entries. + pub cache: Vec, + /// Optional external-env snapshot (SALT buckets, …). + #[serde(default)] + pub external_env: Option, +} + +/// External-env snapshot fields needed for envelope merge. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct ExternalEnvDoc { + /// SALT bucket capacity pairs `(bucket_id, capacity)`. + #[serde(default)] + pub bucket_capacities: Vec<(u32, u64)>, +} + +impl ExternalEnvDoc { + /// Canonical form used for equality and on-disk writes. + /// + /// Deduplicates by bucket id with last-wins (matching runtime map-insert + /// semantics when applying `--bucket-capacity`), then sorts by bucket id so + /// two workers with the same effective capacities never conflict solely + /// because of CLI order. + pub(crate) fn canonicalized(&self) -> Self { + Self { bucket_capacities: canonicalize_bucket_capacities(&self.bucket_capacities) } + } +} + +/// Deduplicate by bucket id (last-wins), then sort by bucket id. +pub(crate) fn canonicalize_bucket_capacities(caps: &[(u32, u64)]) -> Vec<(u32, u64)> { + let mut map = BTreeMap::new(); + for &(id, capacity) in caps { + map.insert(id, capacity); + } + map.into_iter().collect() +} + +/// Parse `rpc-cache-{chain_id}.json` from a path's file name. +/// +/// Returns `None` when the name does not match the per-chain provider-cache +/// convention (so callers can warn that chain identity cannot be validated). +pub(crate) fn parse_rpc_cache_filename_chain_id(path: &Path) -> Option { + let name = path.file_name()?.to_str()?; + let rest = name.strip_prefix("rpc-cache-")?.strip_suffix(".json")?; + if rest.is_empty() || !rest.chars().all(|c| c.is_ascii_digit()) { + return None; + } + // Reject leading zeros (except the single digit `0`) so `rpc-cache-01.json` + // is not treated as chain 1 under a different spelling. + if rest.len() > 1 && rest.starts_with('0') { + return None; + } + rest.parse().ok() +} + +/// Union `base` with `overlay` by key; overlay wins on collision. +/// +/// Output is sorted by key for deterministic files. +pub(crate) fn merge_kv_entries(base: Vec, overlay: Vec) -> Vec { + let mut map: BTreeMap = BTreeMap::new(); + for e in base { + map.insert(e.key, e.value); + } + for e in overlay { + map.insert(e.key, e.value); + } + map.into_iter().map(|(key, value)| CacheKv { key, value }).collect() +} + +/// Merge `ours` over `on_disk` for a provider cache, keeping at most `cap` +/// entries. +/// +/// `--rpc.cache-max-entries` bounds what a run persists, so the union of a +/// sibling's file and ours must be bounded too: runs that share a cache +/// directory but touch disjoint RPC keys would otherwise grow the file without +/// limit, and every later start would parse all of it before the in-memory LRU +/// could evict anything. +/// +/// This process's entries are kept first — they are already LRU-bounded by the +/// same cap, and they are the ones this run just proved it needs. On-disk +/// entries then fill whatever room is left. +pub(crate) fn merge_provider_entries_capped( + on_disk: Vec, + ours: Vec, + cap: usize, +) -> Vec { + let mut map: BTreeMap = BTreeMap::new(); + for e in ours.into_iter().take(cap) { + map.insert(e.key, e.value); + } + for e in on_disk { + if map.len() >= cap { + break; + } + map.entry(e.key).or_insert(e.value); + } + map.into_iter().map(|(key, value)| CacheKv { key, value }).collect() +} + +/// Detect whether `value` is a provider-cache array or a capture envelope. +pub(crate) fn detect_shape(value: &serde_json::Value, path: &Path) -> Result { + if value.is_array() { + // Validate array elements look like {key, value} when non-empty. + if let Some(arr) = value.as_array() { + for (i, el) in arr.iter().enumerate() { + if !el.is_object() || el.get("key").is_none() || el.get("value").is_none() { + return Err(EvmeError::InvalidInput(format!( + "Provider-cache entry {i} in '{}' is not a {{key, value}} object", + path.display() + ))); + } + } + } + return Ok(CacheShape::Provider); + } + if let Some(obj) = value.as_object() { + if obj.contains_key("version") && obj.contains_key("chain_id") && obj.contains_key("cache") + { + return Ok(CacheShape::Envelope); + } + } + Err(EvmeError::InvalidInput(format!( + "Unrecognized cache file shape in '{}': expected a JSON array of {{key, value}} \ + or a capture envelope {{version, chain_id, cache, ...}}", + path.display() + ))) +} + +/// Read and parse a provider-cache file (JSON array). Missing file → empty vec. +/// +/// Corrupt / unreadable content returns `Err` so callers can degrade or hard-fail. +/// +/// Production writers use [`reread_provider_cache_for_merge`] (typed hard vs +/// degradable). This helper remains for tests and call sites that only need the +/// provider-array parse and treat any other shape as an error. +#[cfg_attr(not(test), allow(dead_code))] +pub(crate) fn read_provider_cache(path: &Path) -> Result> { + if !path.exists() { + return Ok(Vec::new()); + } + let content = fs::read_to_string(path).map_err(|e| { + EvmeError::InvalidInput(format!("Failed to read cache file {}: {e}", path.display())) + })?; + let value: serde_json::Value = serde_json::from_str(&content).map_err(|e| { + EvmeError::InvalidInput(format!("Failed to parse cache file {}: {e}", path.display())) + })?; + match detect_shape(&value, path)? { + CacheShape::Provider => serde_json::from_value(value).map_err(|e| { + EvmeError::InvalidInput(format!( + "Failed to decode provider-cache entries in {}: {e}", + path.display() + )) + }), + CacheShape::Envelope => Err(EvmeError::InvalidInput(format!( + "Expected provider-cache array in '{}', found capture envelope", + path.display() + ))), + } +} + +/// Classification of the file already at a provider-shape merge's output. +/// +/// The counterpart of [`EnvelopeReread`] for the other shape, and typed for the +/// same reason: content that cannot be parsed at all is safe to replace, while +/// a file that parses into something this merge cannot fold is a file the merge +/// must not silently destroy. +#[derive(Debug)] +pub(crate) enum ProviderReread { + /// The output holds provider-cache entries (or does not exist yet). + Ok(Vec), + /// Corrupt, unreadable, or undecodable content — safe to warn and replace. + Degradable(String), + /// Readable, but not a provider cache: refusing beats overwriting. + Hard(EvmeError), +} + +/// Re-read the existing merge output for a provider-shape merge. +pub(crate) fn reread_provider_cache_for_merge(path: &Path) -> ProviderReread { + if !path.exists() { + return ProviderReread::Ok(Vec::new()); + } + let content = match fs::read_to_string(path) { + Ok(c) => c, + Err(e) => { + return ProviderReread::Degradable(format!( + "Failed to read cache file {}: {e}", + path.display() + )); + } + }; + let value: serde_json::Value = match serde_json::from_str(&content) { + Ok(v) => v, + Err(e) => { + return ProviderReread::Degradable(format!( + "Failed to parse cache file {}: {e}", + path.display() + )); + } + }; + match detect_shape(&value, path) { + Ok(CacheShape::Provider) => match serde_json::from_value(value) { + Ok(entries) => ProviderReread::Ok(entries), + Err(e) => ProviderReread::Degradable(format!( + "Failed to decode provider-cache entries in {}: {e}", + path.display() + )), + }, + Ok(CacheShape::Envelope) => ProviderReread::Hard(EvmeError::InvalidInput(format!( + "Expected provider-cache array in '{}', found capture envelope", + path.display() + ))), + // An unrecognized shape is still structured JSON somebody wrote: it is + // not this merge's output to overwrite. + Err(e) => ProviderReread::Hard(e), + } +} + +/// Classification of an envelope re-read during concurrent persist merge. +/// +/// Typed so hard identity failures (version / `chain_id` / wrong shape) are not +/// confused with corrupt JSON merely because a path or message contains those +/// substrings. +#[derive(Debug)] +pub(crate) enum EnvelopeReread { + /// Successfully parsed and version-validated envelope. + Ok(EnvelopeDoc), + /// Corrupt, unreadable, or undecodable content — safe to warn and replace. + Degradable(String), + /// Schema / identity failure that must abort the capture persist. + Hard(EvmeError), +} + +/// Re-read an on-disk envelope for the lock-protected merge-on-persist path. +/// +/// Distinguishes hard identity failures from degradable corrupt content without +/// substring-searching formatted messages. +pub(crate) fn reread_envelope_for_merge(path: &Path) -> EnvelopeReread { + let content = match fs::read_to_string(path) { + Ok(c) => c, + Err(e) => { + return EnvelopeReread::Degradable(format!( + "Failed to read envelope {}: {e}", + path.display() + )); + } + }; + let value: serde_json::Value = match serde_json::from_str(&content) { + Ok(v) => v, + Err(e) => { + return EnvelopeReread::Degradable(format!( + "Failed to parse envelope {}: {e}", + path.display() + )); + } + }; + let shape = match detect_shape(&value, path) { + Ok(s) => s, + Err(e) => { + // Unrecognized shape is a hard identity/schema failure: the on-disk + // file is not a capture envelope this build can merge into. + return EnvelopeReread::Hard(EvmeError::FixtureError(e.to_string())); + } + }; + match shape { + CacheShape::Envelope => { + let doc: EnvelopeDoc = match serde_json::from_value(value) { + Ok(d) => d, + Err(e) => { + return EnvelopeReread::Degradable(format!( + "Failed to decode envelope {}: {e}", + path.display() + )); + } + }; + if doc.version != ENVELOPE_VERSION { + return EnvelopeReread::Hard(EvmeError::FixtureError(format!( + "Unsupported cache file version {} in '{}'; expected {ENVELOPE_VERSION}", + doc.version, + path.display(), + ))); + } + EnvelopeReread::Ok(doc) + } + CacheShape::Provider => EnvelopeReread::Hard(EvmeError::FixtureError(format!( + "Expected capture envelope in '{}', found provider-cache array", + path.display() + ))), + } +} + +/// Merge two provider-cache entry lists (overlay wins). +pub(crate) fn merge_provider_lists(base: Vec, overlay: Vec) -> Vec { + merge_kv_entries(base, overlay) +} + +/// Merge `ours` over `on_disk` for envelope persist (ours wins on key collision). +/// +/// Returns an error if `chain_id` or `version` disagree. +/// +/// `external_env` uses optimistic concurrency against `loaded_external_env` — the +/// snapshot observed when this process opened the capture file (or `None` when +/// the file was absent / had no snapshot). See +/// [`resolve_external_env_for_persist`] for the full decision table; only a true +/// concurrent conflict is a hard error, and the message then names all three +/// values (loaded, ours, on-disk). +pub(crate) fn merge_envelope_for_persist( + on_disk: &EnvelopeDoc, + ours: &EnvelopeDoc, + loaded_external_env: Option<&ExternalEnvDoc>, + path: &Path, +) -> Result { + if on_disk.version != ours.version { + return Err(EvmeError::FixtureError(format!( + "Envelope version mismatch when merging '{}': on-disk {}, ours {}", + path.display(), + on_disk.version, + ours.version, + ))); + } + if on_disk.chain_id != ours.chain_id { + return Err(EvmeError::FixtureError(format!( + "Envelope chain_id mismatch when merging '{}': on-disk {}, ours {}", + path.display(), + on_disk.chain_id, + ours.chain_id, + ))); + } + let external_env = resolve_external_env_for_persist( + &ours.external_env, + &on_disk.external_env, + loaded_external_env, + path, + )?; + Ok(EnvelopeDoc { + version: ours.version, + chain_id: ours.chain_id, + cache: merge_kv_entries(on_disk.cache.clone(), ours.cache.clone()), + external_env, + }) +} + +/// Resolve the envelope `external_env` under optimistic concurrency. +/// +/// Three inputs decide the outcome: `loaded` (the snapshot this process observed +/// when it opened the file), `ours` (what this process would write), and +/// `on_disk` (what the locked re-read found). All three are canonicalized first, +/// so CLI ordering alone never decides anything. +/// +/// | `ours` | `on_disk` | relation | result | why | +/// | ------ | --------- | ---------------------------- | --------- | ---------------------------------------------------------------- | +/// | `None` | any | — | `on_disk` | this run has no snapshot to contribute | +/// | `Some` | `None` | — | `ours` | nothing on disk to disagree with | +/// | `Some` | `Some` | equal | `ours` | no decision to make | +/// | `Some` | `Some` | differ, `ours == loaded` | `on_disk` | this run changed nothing: sibling's refresh wins | +/// | `Some` | `Some` | differ, `on_disk == loaded` | `ours` | nobody wrote since load: our intentional refresh wins | +/// | `Some` | `Some` | differ, neither | `Err` | true conflict: two runs changed the same snapshot differently | +/// +/// The last two conditions are mutually exclusive, so their order does not +/// matter: if both held, `ours` and `on_disk` would each equal `loaded` and +/// therefore each other, contradicting "differ". +/// +/// Row four carries as much weight as row six. A capture run given no +/// `--bucket-capacity` carries the previous snapshot forward verbatim, so +/// `ours == loaded` means "changed nothing", not "chose this value". Calling +/// that a conflict fails the persist — and because capture persistence is a +/// hard error, that discards every RPC response the run captured, over metadata +/// the run never had a stake in. +/// +/// The predicate is value equality, not "was the flag passed": persist has no +/// record of the caller's argv, so re-asserting the values already in force is +/// indistinguishable from carrying them forward, and both yield. +fn resolve_external_env_for_persist( + ours: &Option, + on_disk: &Option, + loaded: Option<&ExternalEnvDoc>, + path: &Path, +) -> Result> { + let ours_c = ours.as_ref().map(ExternalEnvDoc::canonicalized); + let disk_c = on_disk.as_ref().map(ExternalEnvDoc::canonicalized); + let loaded_c = loaded.map(ExternalEnvDoc::canonicalized); + + match (&ours_c, &disk_c) { + (Some(o), Some(d)) if o != d => { + let ours_carried_forward = loaded_c.as_ref() == Some(o); + let disk_unchanged_since_load = loaded_c.as_ref() == Some(d); + if ours_carried_forward { + Ok(disk_c) + } else if disk_unchanged_since_load { + Ok(ours_c) + } else { + Err(EvmeError::FixtureError(format!( + "Conflicting external_env snapshots when merging '{}': \ + loaded {loaded_env:?}, ours {ours_env:?}, on-disk {on_disk_env:?}", + path.display(), + loaded_env = loaded_c, + ours_env = ours_c, + on_disk_env = disk_c, + ))) + } + } + (Some(_), _) => Ok(ours_c), + (None, disk) => Ok(disk.clone()), + } +} + +/// Merge multiple envelope inputs for the `cache merge` subcommand. +/// +/// All inputs must share `version` and `chain_id`. Later inputs win on cache +/// key collision. Non-null `external_env` values must be identical (after +/// canonicalization) when more than one is present; the written snapshot is +/// always the canonical form. +pub(crate) fn merge_envelopes_cli(docs: &[(PathBuf, EnvelopeDoc)]) -> Result { + let Some((_, first)) = docs.first() else { + return Err(EvmeError::InvalidInput("cache merge requires at least one input file".into())); + }; + let version = first.version; + let chain_id = first.chain_id; + if version != ENVELOPE_VERSION { + return Err(EvmeError::InvalidInput(format!( + "Unsupported envelope version {version} in '{}'; expected {ENVELOPE_VERSION}", + docs[0].0.display(), + ))); + } + + let mut cache = Vec::new(); + let mut external_env: Option = None; + + for (path, doc) in docs { + if doc.version != version { + return Err(EvmeError::InvalidInput(format!( + "Envelope version mismatch: '{}' has version {}, expected {version}", + path.display(), + doc.version, + ))); + } + if doc.chain_id != chain_id { + return Err(EvmeError::InvalidInput(format!( + "Envelope chain_id mismatch: '{}' has chain_id {}, expected {chain_id}", + path.display(), + doc.chain_id, + ))); + } + cache = merge_kv_entries(cache, doc.cache.clone()); + if let Some(ref ext) = doc.external_env { + let canon = ext.canonicalized(); + match &external_env { + None => external_env = Some(canon), + Some(prev) if prev != &canon => { + return Err(EvmeError::InvalidInput(format!( + "Conflicting external_env snapshots while merging '{}'", + path.display(), + ))); + } + Some(_) => {} + } + } + } + + Ok(EnvelopeDoc { version, chain_id, cache, external_env }) +} + +/// Fold the envelope already on disk at the merge output into `merged_inputs`. +/// +/// Called by `cache merge` under the output's exclusive lock, so `on_disk` is +/// whatever a concurrent writer left behind while this merge waited. It joins +/// the union as one more input under the same rules the CLI merge applies to +/// its inputs: identity must agree, entries union by key, and non-null +/// `external_env` snapshots must be identical after canonicalization. +/// +/// The merge's own inputs win on key collision — the operator named those +/// files, and this matches the ours-win rule the persist path uses for the +/// same read-modify-write cycle. Errors name the output path, since that is +/// the file the caller did not list on the command line. +pub(crate) fn fold_output_envelope( + output: &Path, + on_disk: EnvelopeDoc, + merged_inputs: EnvelopeDoc, +) -> Result { + if on_disk.version != merged_inputs.version { + return Err(EvmeError::InvalidInput(format!( + "Envelope version mismatch with existing output '{}': output has version {}, \ + inputs have version {}", + output.display(), + on_disk.version, + merged_inputs.version, + ))); + } + if on_disk.chain_id != merged_inputs.chain_id { + return Err(EvmeError::InvalidInput(format!( + "Envelope chain_id mismatch with existing output '{}': output has chain_id {}, \ + inputs have chain_id {}", + output.display(), + on_disk.chain_id, + merged_inputs.chain_id, + ))); + } + + let external_env = match (&on_disk.external_env, &merged_inputs.external_env) { + (Some(disk), Some(ours)) => { + let (disk_c, ours_c) = (disk.canonicalized(), ours.canonicalized()); + if disk_c != ours_c { + return Err(EvmeError::InvalidInput(format!( + "Conflicting external_env snapshots with existing output '{}': \ + output {disk_c:?}, inputs {ours_c:?}", + output.display(), + ))); + } + Some(ours_c) + } + (Some(disk), None) => Some(disk.canonicalized()), + (None, Some(ours)) => Some(ours.canonicalized()), + (None, None) => None, + }; + + Ok(EnvelopeDoc { + version: merged_inputs.version, + chain_id: merged_inputs.chain_id, + cache: merge_kv_entries(on_disk.cache, merged_inputs.cache), + external_env, + }) +} + +/// Atomically write `entries` as a provider-cache JSON array to `path`. +pub(crate) fn write_provider_cache_atomic(path: &Path, entries: &[CacheKv]) -> Result<()> { + let dir = path.parent().unwrap_or_else(|| Path::new(".")); + fs::create_dir_all(dir).map_err(|e| { + EvmeError::InvalidInput(format!("Failed to create directory {}: {e}", dir.display())) + })?; + let serialized = serde_json::to_vec(entries) + .map_err(|e| EvmeError::InvalidInput(format!("Failed to serialize provider cache: {e}")))?; + write_bytes_atomic(path, &serialized) + .map_err(|e| EvmeError::InvalidInput(format!("Failed to write {}: {e}", path.display()))) +} + +/// Atomically write an envelope document (pretty-printed, matching capture). +pub(crate) fn write_envelope_atomic(path: &Path, doc: &EnvelopeDoc) -> Result<()> { + let dir = path.parent().unwrap_or_else(|| Path::new(".")); + fs::create_dir_all(dir).map_err(|e| { + EvmeError::FixtureError(format!( + "Failed to create cache file directory {}: {e}", + dir.display() + )) + })?; + let serialized = serde_json::to_string_pretty(doc).map_err(|e| { + EvmeError::FixtureError(format!("Failed to serialize envelope for {}: {e}", path.display())) + })?; + write_bytes_atomic(path, serialized.as_bytes()).map_err(|e| { + EvmeError::FixtureError(format!("Failed to persist envelope to {}: {e}", path.display())) + }) +} + +/// Temp-file + rename write. +pub(crate) fn write_bytes_atomic(path: &Path, bytes: &[u8]) -> std::io::Result<()> { + let dir = path.parent().unwrap_or_else(|| Path::new(".")); + let mut tmp = tempfile::NamedTempFile::new_in(dir).map_err(|e| { + std::io::Error::other(format!("failed to create temp file in {}: {e}", dir.display())) + })?; + tmp.write_all(bytes)?; + tmp.flush()?; + // flush() only clears the userspace buffer. Without sync_all() a crash + // between write and rename can publish a truncated file under the target + // name — the rename is atomic, the contents are not. + tmp.as_file().sync_all()?; + tmp.persist(path).map_err(|e| { + std::io::Error::other(format!( + "failed to rename temp file into {}: {}", + path.display(), + e.error, + )) + })?; + Ok(()) +} + +/// Load any supported cache file and return its shape + entry count. +pub(crate) fn load_cache_file(path: &Path) -> Result<(CacheShape, LoadedCache)> { + let content = fs::read_to_string(path) + .map_err(|e| EvmeError::InvalidInput(format!("Failed to read {}: {e}", path.display())))?; + let value: serde_json::Value = serde_json::from_str(&content) + .map_err(|e| EvmeError::InvalidInput(format!("Failed to parse {}: {e}", path.display())))?; + let shape = detect_shape(&value, path)?; + match shape { + CacheShape::Provider => { + let entries: Vec = serde_json::from_value(value).map_err(|e| { + EvmeError::InvalidInput(format!( + "Failed to decode provider cache {}: {e}", + path.display() + )) + })?; + Ok((shape, LoadedCache::Provider(entries))) + } + CacheShape::Envelope => { + let doc: EnvelopeDoc = serde_json::from_value(value).map_err(|e| { + EvmeError::InvalidInput(format!( + "Failed to decode envelope {}: {e}", + path.display() + )) + })?; + if doc.version != ENVELOPE_VERSION { + return Err(EvmeError::InvalidInput(format!( + "Unsupported cache file version {} in '{}'; expected {ENVELOPE_VERSION}", + doc.version, + path.display(), + ))); + } + Ok((shape, LoadedCache::Envelope(doc))) + } + } +} + +/// Parsed cache file payload. +#[derive(Debug)] +pub(crate) enum LoadedCache { + Provider(Vec), + Envelope(EnvelopeDoc), +} + +impl LoadedCache { + pub(crate) fn entry_count(&self) -> usize { + match self { + Self::Provider(e) => e.len(), + Self::Envelope(d) => d.cache.len(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn kv(byte: u8, val: &str) -> CacheKv { + CacheKv { key: B256::repeat_byte(byte), value: val.to_string() } + } + + #[test] + fn test_merge_kv_union_and_ours_wins() { + let base = vec![kv(1, "a"), kv(2, "b")]; + let overlay = vec![kv(2, "B"), kv(3, "c")]; + let merged = merge_kv_entries(base, overlay); + assert_eq!(merged.len(), 3); + assert_eq!(merged[0], kv(1, "a")); + assert_eq!(merged[1], kv(2, "B")); // overlay wins + assert_eq!(merged[2], kv(3, "c")); + } + + #[test] + fn test_detect_shape_provider_and_envelope() { + let arr = serde_json::json!([{"key": B256::ZERO, "value": "x"}]); + assert_eq!(detect_shape(&arr, Path::new("p.json")).unwrap(), CacheShape::Provider); + let env = serde_json::json!({ + "version": 1, + "chain_id": 1, + "cache": [] + }); + assert_eq!(detect_shape(&env, Path::new("e.json")).unwrap(), CacheShape::Envelope); + } + + #[test] + fn test_merge_envelope_for_persist_union_and_ext() { + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10)] }), + }; + let ours = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "ours"), kv(2, "new")], + external_env: None, + }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, None, Path::new("x.json")).unwrap(); + assert_eq!(merged.cache, vec![kv(1, "ours"), kv(2, "new")]); + assert_eq!(merged.external_env, Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10)] })); + } + + #[test] + fn test_merge_envelope_for_persist_chain_id_mismatch() { + let on_disk = EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: None }; + let ours = EnvelopeDoc { version: 1, chain_id: 2, cache: vec![], external_env: None }; + let err = + merge_envelope_for_persist(&on_disk, &ours, None, Path::new("x.json")).unwrap_err(); + assert!(err.to_string().contains("chain_id")); + } + + /// Intentional refresh: loaded A, ours B, disk still A → B wins. + #[test] + fn test_merge_envelope_for_persist_intentional_refresh_wins() { + let loaded = ExternalEnvDoc { bucket_capacities: vec![(1, 10)] }; + let ours_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 99)] }; + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(loaded.clone()), + }; + let ours = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(2, "ours")], + external_env: Some(ours_ext.clone()), + }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, Some(&loaded), Path::new("capture.json")) + .expect("intentional A→B refresh must succeed"); + assert_eq!(merged.external_env, Some(ours_ext.canonicalized())); + assert_eq!(merged.cache, vec![kv(1, "disk"), kv(2, "ours")]); + } + + /// The merged provider cache never exceeds the configured cap, and this + /// process's entries survive the truncation. + /// + /// Runs sharing a cache directory touch disjoint RPC keys, so an uncapped + /// union grows the file without limit no matter how small each run's LRU is. + #[test] + fn test_merge_provider_entries_capped_bounds_the_union() { + let on_disk: Vec = (0..10).map(|i| kv(i, "disk")).collect(); + let ours: Vec = (100..104).map(|i| kv(i, "ours")).collect(); + + let merged = merge_provider_entries_capped(on_disk, ours.clone(), 6); + assert_eq!(merged.len(), 6, "the union is capped, not the sum of both sides"); + for entry in &ours { + assert!( + merged.iter().any(|m| m.key == entry.key && m.value == entry.value), + "this run's entries survive truncation: {:?}", + entry.key + ); + } + } + + /// A cap larger than the union keeps everything, and ours win on collision. + #[test] + fn test_merge_provider_entries_capped_keeps_all_below_the_cap() { + let on_disk = vec![kv(1, "disk"), kv(2, "disk")]; + let ours = vec![kv(2, "ours"), kv(3, "ours")]; + + let merged = merge_provider_entries_capped(on_disk, ours, 16); + assert_eq!(merged, vec![kv(1, "disk"), kv(2, "ours"), kv(3, "ours")]); + } + + /// No opinion: loaded A, ours A (carried forward), disk now B → B wins and + /// our cache entries still merge. A run given no `--bucket-capacity` reaches + /// persist with `ours == loaded`; treating that as a conflict would fail the + /// persist and throw away everything the run captured. + #[test] + fn test_merge_envelope_for_persist_carried_forward_snapshot_yields_to_sibling_refresh() { + let loaded = ExternalEnvDoc { bucket_capacities: vec![(1, 10)] }; + let disk_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 20)] }; + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(disk_ext.clone()), + }; + // No `--bucket-capacity` on this run: the loaded snapshot is carried + // forward verbatim, so `ours` is byte-identical to `loaded`. + let ours = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(2, "ours")], + external_env: Some(loaded.clone()), + }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, Some(&loaded), Path::new("capture.json")) + .expect("a run that expressed no opinion must not conflict"); + assert_eq!(merged.external_env, Some(disk_ext.canonicalized())); + assert_eq!(merged.cache, vec![kv(1, "disk"), kv(2, "ours")]); + } + + /// Table row one (`ours = None`): a run with no snapshot of its own keeps + /// the on-disk one and still merges its cache entries. + /// + /// This row was always correct; it is pinned so the table has a test per + /// row rather than only where a bug was found. + #[test] + fn test_merge_envelope_for_persist_no_snapshot_yields_to_sibling_refresh() { + let disk_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 20)] }; + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(disk_ext.clone()), + }; + let ours = + EnvelopeDoc { version: 1, chain_id: 7, cache: vec![kv(2, "ours")], external_env: None }; + let merged = merge_envelope_for_persist(&on_disk, &ours, None, Path::new("capture.json")) + .expect("a run with no snapshot must not conflict"); + assert_eq!(merged.external_env, Some(disk_ext)); + assert_eq!(merged.cache, vec![kv(1, "disk"), kv(2, "ours")]); + } + + /// True concurrent conflict: loaded A, ours B, disk now C≠B → hard error naming A/B/C. + #[test] + fn test_merge_envelope_for_persist_rejects_true_concurrent_conflict() { + let loaded = ExternalEnvDoc { bucket_capacities: vec![(1, 10)] }; + let ours_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 99)] }; + let disk_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 42)] }; + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(disk_ext), + }; + let ours = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(2, "ours")], + external_env: Some(ours_ext), + }; + let err = + merge_envelope_for_persist(&on_disk, &ours, Some(&loaded), Path::new("capture.json")) + .unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!(msg.contains("loaded"), "msg={msg}"); + assert!(msg.contains("ours"), "msg={msg}"); + assert!(msg.contains("on-disk"), "msg={msg}"); + // All three snapshots named (Debug form of bucket capacities). + assert!(msg.contains("10") && msg.contains("99") && msg.contains("42"), "msg={msg}"); + } + + /// Loaded none, ours B, disk now C≠B → hard error (file gained a foreign snapshot). + #[test] + fn test_merge_envelope_for_persist_rejects_conflict_when_loaded_none() { + let ours_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 99)] }; + let disk_ext = ExternalEnvDoc { bucket_capacities: vec![(1, 42)] }; + let on_disk = + EnvelopeDoc { version: 1, chain_id: 7, cache: vec![], external_env: Some(disk_ext) }; + let ours = + EnvelopeDoc { version: 1, chain_id: 7, cache: vec![], external_env: Some(ours_ext) }; + let err = merge_envelope_for_persist(&on_disk, &ours, None, Path::new("capture.json")) + .unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!(msg.contains("loaded"), "msg={msg}"); + assert!(msg.contains("99") && msg.contains("42"), "msg={msg}"); + } + + /// Identical non-null `external_env` snapshots merge successfully. + #[test] + fn test_merge_envelope_for_persist_identical_external_env() { + let ext = ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }; + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "disk")], + external_env: Some(ext.clone()), + }; + let ours = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(2, "ours")], + external_env: Some(ext.clone()), + }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, Some(&ext), Path::new("x.json")).unwrap(); + assert_eq!(merged.cache, vec![kv(1, "disk"), kv(2, "ours")]); + assert_eq!(merged.external_env, Some(ext.canonicalized())); + } + + /// Same capacities in different order are not a conflict (canonical equality). + #[test] + fn test_merge_envelope_for_persist_order_insensitive_external_env() { + let a = ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }; + let b = ExternalEnvDoc { bucket_capacities: vec![(2, 20), (1, 10)] }; + let on_disk = EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: Some(a) }; + let ours = EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: Some(b) }; + // Concurrent writer used the same effective capacities in different CLI order. + let merged = merge_envelope_for_persist( + &on_disk, + &ours, + Some(&ExternalEnvDoc { bucket_capacities: vec![(9, 9)] }), + Path::new("x.json"), + ) + .expect("order-only difference must not conflict"); + assert_eq!( + merged.external_env, + Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }) + ); + } + + /// Duplicate bucket ids collapse with last-wins before sort. + #[test] + fn test_canonicalize_bucket_capacities_last_wins_and_sorts() { + let caps = canonicalize_bucket_capacities(&[(2, 20), (1, 10), (2, 99), (1, 11)]); + assert_eq!(caps, vec![(1, 11), (2, 99)]); + let doc = ExternalEnvDoc { bucket_capacities: vec![(3, 1), (1, 2), (3, 9)] }; + assert_eq!(doc.canonicalized().bucket_capacities, vec![(1, 2), (3, 9)]); + } + + /// One-sided `external_env` propagates the non-null snapshot (either side). + #[test] + fn test_merge_envelope_for_persist_one_sided_external_env() { + let ext = ExternalEnvDoc { bucket_capacities: vec![(3, 30)] }; + // Ours None, disk Some → disk propagates (covered by existing union test + // for the reverse orientation; re-assert disk-propagates here). + let on_disk = + EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: Some(ext.clone()) }; + let ours = + EnvelopeDoc { version: 1, chain_id: 1, cache: vec![kv(1, "a")], external_env: None }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, None, Path::new("x.json")).unwrap(); + assert_eq!(merged.external_env, Some(ext.canonicalized())); + + // Ours Some, disk None → ours kept. + let on_disk = EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: None }; + let ours = EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: Some(ext.clone()), + }; + let merged = + merge_envelope_for_persist(&on_disk, &ours, None, Path::new("x.json")).unwrap(); + assert_eq!(merged.external_env, Some(ext.canonicalized())); + } + + #[test] + fn test_merge_envelopes_cli_conflict_external_env() { + let a = EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 1)] }), + }; + let b = EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(2, "b")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 2)] }), + }; + let docs = vec![(PathBuf::from("a.json"), a), (PathBuf::from("b.json"), b)]; + let err = merge_envelopes_cli(&docs).unwrap_err(); + assert!(err.to_string().contains("external_env")); + } + + /// CLI merge treats equal capacities in different order as identical. + #[test] + fn test_merge_envelopes_cli_order_insensitive_external_env() { + let a = EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(2, 20), (1, 10)] }), + }; + let b = EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(2, "b")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }), + }; + let docs = vec![(PathBuf::from("a.json"), a), (PathBuf::from("b.json"), b)]; + let merged = merge_envelopes_cli(&docs).expect("order-only difference must merge"); + assert_eq!( + merged.external_env, + Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }) + ); + } + + /// Folding the existing output unions its entries in, with the named inputs + /// winning on key collision. + #[test] + fn test_fold_output_envelope_unions_with_inputs_winning() { + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "output"), kv(9, "concurrent")], + external_env: None, + }; + let inputs = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![kv(1, "inputs"), kv(2, "inputs")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(2, 20), (1, 10)] }), + }; + let folded = + fold_output_envelope(Path::new("out.json"), on_disk, inputs).expect("fold output"); + assert_eq!(folded.cache, vec![kv(1, "inputs"), kv(2, "inputs"), kv(9, "concurrent")]); + // The written snapshot is canonical. + assert_eq!( + folded.external_env, + Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }) + ); + } + + /// A snapshot on the existing output that disagrees with the inputs is a + /// conflict, exactly as it is between two inputs. + #[test] + fn test_fold_output_envelope_rejects_conflicting_external_env() { + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 42)] }), + }; + let inputs = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 99)] }), + }; + let err = + fold_output_envelope(Path::new("out.json"), on_disk, inputs).expect_err("conflict"); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!(msg.contains("out.json"), "msg={msg}"); + assert!(msg.contains("42") && msg.contains("99"), "msg={msg}"); + } + + /// Same effective capacities in different order are not a conflict. + #[test] + fn test_fold_output_envelope_order_insensitive_external_env() { + let on_disk = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(2, 20), (1, 10)] }), + }; + let inputs = EnvelopeDoc { + version: 1, + chain_id: 7, + cache: vec![], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }), + }; + let folded = fold_output_envelope(Path::new("out.json"), on_disk, inputs) + .expect("order-only difference must not conflict"); + assert_eq!( + folded.external_env, + Some(ExternalEnvDoc { bucket_capacities: vec![(1, 10), (2, 20)] }) + ); + } + + /// Identity failures against the existing output name that file. + #[test] + fn test_fold_output_envelope_rejects_identity_mismatch() { + let inputs = EnvelopeDoc { version: 1, chain_id: 7, cache: vec![], external_env: None }; + + let other_chain = + EnvelopeDoc { version: 1, chain_id: 8, cache: vec![], external_env: None }; + let err = fold_output_envelope(Path::new("out.json"), other_chain, inputs.clone()) + .expect_err("chain_id mismatch"); + let msg = err.to_string(); + assert!(msg.contains("chain_id") && msg.contains("out.json"), "msg={msg}"); + + let other_version = + EnvelopeDoc { version: 2, chain_id: 7, cache: vec![], external_env: None }; + let err = fold_output_envelope(Path::new("out.json"), other_version, inputs) + .expect_err("version mismatch"); + let msg = err.to_string(); + assert!(msg.contains("version") && msg.contains("out.json"), "msg={msg}"); + } + + /// Filename-derived chain id for the standard provider-cache naming scheme. + #[test] + fn test_parse_rpc_cache_filename_chain_id() { + assert_eq!(parse_rpc_cache_filename_chain_id(Path::new("rpc-cache-1.json")), Some(1)); + assert_eq!( + parse_rpc_cache_filename_chain_id(Path::new("/tmp/rpc-cache-4326.json")), + Some(4326) + ); + assert_eq!( + parse_rpc_cache_filename_chain_id(Path::new("worker/rpc-cache-11155420.json")), + Some(11_155_420) + ); + // Non-matching names cannot be validated from the filename alone. + assert_eq!(parse_rpc_cache_filename_chain_id(Path::new("out.json")), None); + assert_eq!(parse_rpc_cache_filename_chain_id(Path::new("rpc-cache.json")), None); + assert_eq!(parse_rpc_cache_filename_chain_id(Path::new("rpc-cache-abc.json")), None); + assert_eq!(parse_rpc_cache_filename_chain_id(Path::new("cache-4326.json")), None); + } +} diff --git a/bin/mega-evme/src/cache/mod.rs b/bin/mega-evme/src/cache/mod.rs new file mode 100644 index 00000000..53862a08 --- /dev/null +++ b/bin/mega-evme/src/cache/mod.rs @@ -0,0 +1,705 @@ +//! Top-level `cache` subcommand group (`mega-evme cache …`). +//! +//! Currently ships `cache merge` for consolidating per-worker provider-cache +//! files or capture envelopes after historical sharded campaigns. + +mod lock; +mod merge; + +use std::{fmt, path::PathBuf}; + +use clap::{Parser, Subcommand}; + +use crate::common::{EvmeError, Result}; + +pub(crate) use lock::{acquire_exclusive_lock, lock_sidecar_path}; +pub(crate) use merge::{ + merge_envelope_for_persist, merge_provider_entries_capped, merge_provider_lists, + parse_rpc_cache_filename_chain_id, reread_envelope_for_merge, reread_provider_cache_for_merge, + write_bytes_atomic, write_envelope_atomic, write_provider_cache_atomic, CacheKv, EnvelopeDoc, + EnvelopeReread, ExternalEnvDoc, ProviderReread, ENVELOPE_VERSION, +}; + +// Used by unit tests that assert the provider-array on-disk shape after a merge. +#[cfg(test)] +pub(crate) use merge::read_provider_cache; + +use merge::{fold_output_envelope, load_cache_file, merge_envelopes_cli, CacheShape, LoadedCache}; +use tracing::warn; + +/// `mega-evme cache` — offline cache-file utilities. +#[derive(Parser, Debug)] +pub struct Cmd { + /// Cache utility subcommand (`merge`, …). + #[command(subcommand)] + pub command: CacheCommands, +} + +/// Cache utility subcommands. +#[derive(Subcommand, Debug)] +pub enum CacheCommands { + /// Merge provider-cache files or capture envelopes into one output file. + Merge(MergeArgs), +} + +/// Arguments for `mega-evme cache merge`. +#[derive(Parser, Debug)] +pub struct MergeArgs { + /// Input cache files (provider-cache arrays or capture envelopes; not mixed). + #[arg(required = true, num_args = 1.., value_name = "INPUT")] + pub inputs: Vec, + + /// Destination path for the merged file (written atomically via temp + rename). + #[arg(long, short = 'o', value_name = "FILE")] + pub output: PathBuf, +} + +impl Cmd { + /// Dispatch the cache subcommand. + pub fn run(self) -> Result<()> { + match self.command { + CacheCommands::Merge(args) => args.run(), + } + } +} + +/// Emit a diagnostic that protects the user from a silently wrong merge or +/// persist decision, on stderr unconditionally and through the structured log +/// sinks. +/// +/// The CLI leaves the tracing filter at `off` unless `-v` flags or `RUST_LOG` +/// raise it, so a `warn!`-only diagnostic reaches nobody on a default command +/// line: a safeguard reporting that it could not run, or a write about to +/// discard data already on disk, would be announced into a disabled subscriber. +/// stderr therefore carries the human line regardless of verbosity, and the +/// tracing event still carries it to a `--log.file` sink. At raised verbosity +/// without `--log.file` both channels land on stderr and the line appears +/// twice, which is preferable to dropping either one. +/// +/// Reserved for warnings a user must act on; ordinary progress reporting stays +/// on `tracing` alone. +pub(crate) fn warn_user(message: fmt::Arguments<'_>) { + eprintln!("warning: {message}"); + warn!("{message}"); +} + +/// Validate that provider-cache paths agreeing with `rpc-cache-{id}.json` all +/// name the same chain id. +/// +/// Paths that do not match the pattern warn the user (chain identity cannot be +/// validated for them) and are otherwise ignored. Two or more matching paths +/// with different ids are a hard error naming the conflicting files. +pub(crate) fn check_provider_cache_chain_identity<'a>( + paths: impl IntoIterator, +) -> Result<()> { + let mut seen: Option<(u64, PathBuf)> = None; + for path in paths { + match parse_rpc_cache_filename_chain_id(path) { + None => { + warn_user(format_args!( + "Provider-cache path '{}' does not match rpc-cache-{{id}}.json; \ + chain identity cannot be validated for this file", + path.display(), + )); + } + Some(id) => match &seen { + None => seen = Some((id, path.to_path_buf())), + Some((prev_id, prev_path)) if *prev_id != id => { + return Err(EvmeError::InvalidInput(format!( + "Provider-cache chain identity mismatch: '{}' is chain {prev_id}, \ + but '{}' is chain {id}. Merge only caches from the same chain.", + prev_path.display(), + path.display(), + ))); + } + Some(_) => {} + }, + } + } + Ok(()) +} + +impl MergeArgs { + /// Merge inputs into `--output` and print a one-line summary. + pub fn run(self) -> Result<()> { + if self.inputs.is_empty() { + return Err(EvmeError::InvalidInput( + "cache merge requires at least one input file".into(), + )); + } + + let mut loaded: Vec<(PathBuf, CacheShape, LoadedCache)> = + Vec::with_capacity(self.inputs.len()); + for path in &self.inputs { + let (shape, data) = load_cache_file(path)?; + loaded.push((path.clone(), shape, data)); + } + + let first_shape = loaded[0].1; + for (path, shape, _) in &loaded { + if *shape != first_shape { + return Err(EvmeError::InvalidInput(format!( + "Mixed cache file shapes: '{}' is {:?}, but the first input is {:?}. \ + Merge provider-cache files and capture envelopes in separate invocations.", + path.display(), + shape, + first_shape, + ))); + } + } + + let total_in: usize = loaded.iter().map(|(_, _, d)| d.entry_count()).sum(); + let input_count = loaded.len(); + + if first_shape == CacheShape::Provider { + // Provider-cache files carry chain identity only in the + // `rpc-cache-{id}.json` filename. Reject merges that would + // union different chains; warn when a path cannot be checked. + // Checked before the output is locked so a doomed merge leaves no + // sidecar behind. + check_provider_cache_chain_identity( + loaded + .iter() + .map(|(p, _, _)| p.as_path()) + .chain(std::iter::once(self.output.as_path())), + )?; + } + + // The output is a shared file: a live run may be persisting to the same + // path under the same sidecar lock. Take that lock and hold it across + // read-merge-rename, so neither side's entries are lost to whichever + // rename lands last. + let _output_lock = acquire_exclusive_lock(&self.output).map_err(|e| { + EvmeError::InvalidInput(format!( + "Failed to acquire the cache lock {} for output '{}': {e}. \ + Refusing to merge without it: an unlocked write would silently drop \ + entries written by a concurrent process.", + lock_sidecar_path(&self.output).display(), + self.output.display(), + )) + })?; + + // Entries the output file already held when the lock was granted. + let mut folded_in = 0usize; + + let unique_out = match first_shape { + CacheShape::Provider => { + let mut acc = Vec::new(); + for (_, _, data) in loaded { + let LoadedCache::Provider(entries) = data else { unreachable!() }; + // Later inputs win on collision. + acc = merge_provider_lists(acc, entries); + } + + // Whatever is at the output now joins the union as one more + // input: a concurrent writer may have landed entries there + // while this merge waited for the lock. + let on_disk = match reread_provider_cache_for_merge(&self.output) { + ProviderReread::Ok(entries) => entries, + ProviderReread::Hard(err) => return Err(err), + ProviderReread::Degradable(msg) => { + // Replacing the output drops whatever it held: the user + // must hear about it whatever the verbosity is. + warn_user(format_args!( + "{msg}. Replacing the existing merge output '{}' with the \ + merged inputs; any entries it held are discarded", + self.output.display(), + )); + Vec::new() + } + }; + folded_in = on_disk.len(); + // The named inputs win over the output's prior entries. + let acc = merge_provider_lists(on_disk, acc); + + let unique = acc.len(); + write_provider_cache_atomic(&self.output, &acc)?; + unique + } + CacheShape::Envelope => { + let docs: Vec<(PathBuf, EnvelopeDoc)> = loaded + .into_iter() + .map(|(path, _, data)| { + let LoadedCache::Envelope(doc) = data else { unreachable!() }; + (path, doc) + }) + .collect(); + let merged = merge_envelopes_cli(&docs)?; + + let merged = if self.output.exists() { + // Typed classification: identity/schema failures must not be + // papered over by overwriting the file we cannot read. + match reread_envelope_for_merge(&self.output) { + EnvelopeReread::Ok(on_disk) => { + folded_in = on_disk.cache.len(); + fold_output_envelope(&self.output, on_disk, merged)? + } + EnvelopeReread::Hard(err) => return Err(err), + EnvelopeReread::Degradable(msg) => { + // Same data loss as the provider shape above. + warn_user(format_args!( + "{msg}. Replacing the existing merge output '{}' with the \ + merged inputs; any entries it held are discarded", + self.output.display(), + )); + merged + } + } + } else { + merged + }; + + let unique = merged.cache.len(); + write_envelope_atomic(&self.output, &merged)?; + unique + } + }; + + // Name the folded-in entries so the arithmetic still adds up when the + // output already held some. + let folded = if folded_in > 0 { + format!(" + {folded_in} already in the output") + } else { + String::new() + }; + println!( + "Merged {input_count} inputs ({total_in} entries in{folded}) \ + → {unique_out} unique entries out" + ); + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use std::fs; + + use alloy_primitives::B256; + use tempfile::tempdir; + + use super::*; + use crate::cache::merge::{merge_envelopes_cli, CacheKv, EnvelopeDoc, ExternalEnvDoc}; + + fn write(path: &std::path::Path, content: &str) { + fs::write(path, content).expect("write"); + } + + fn kv(byte: u8, val: &str) -> CacheKv { + CacheKv { key: B256::repeat_byte(byte), value: val.to_string() } + } + + #[test] + fn test_cache_merge_provider_union_later_wins() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + let entries_a = vec![kv(1, "from-a"), kv(2, "a")]; + let entries_b = vec![kv(2, "from-b"), kv(3, "b")]; + write(&a, &serde_json::to_string(&entries_a).unwrap()); + write(&b, &serde_json::to_string(&entries_b).unwrap()); + + MergeArgs { inputs: vec![a, b], output: out.clone() }.run().expect("merge"); + + let merged: Vec = + serde_json::from_str(&fs::read_to_string(&out).unwrap()).unwrap(); + assert_eq!(merged, vec![kv(1, "from-a"), kv(2, "from-b"), kv(3, "b")]); + } + + #[test] + fn test_cache_merge_envelope_union() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + let env_a = EnvelopeDoc { + version: 1, + chain_id: 4326, + cache: vec![kv(1, "a")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 100)] }), + }; + let env_b = EnvelopeDoc { + version: 1, + chain_id: 4326, + cache: vec![kv(1, "b"), kv(2, "b")], + external_env: None, + }; + write(&a, &serde_json::to_string_pretty(&env_a).unwrap()); + write(&b, &serde_json::to_string_pretty(&env_b).unwrap()); + + MergeArgs { inputs: vec![a, b], output: out.clone() }.run().expect("merge"); + + let merged: EnvelopeDoc = serde_json::from_str(&fs::read_to_string(&out).unwrap()).unwrap(); + assert_eq!(merged.chain_id, 4326); + assert_eq!(merged.cache, vec![kv(1, "b"), kv(2, "b")]); + assert_eq!(merged.external_env, Some(ExternalEnvDoc { bucket_capacities: vec![(1, 100)] })); + } + + #[test] + fn test_cache_merge_rejects_mixed_shapes() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "a")]).unwrap()); + write( + &b, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![], + external_env: None, + }) + .unwrap(), + ); + + let err = MergeArgs { inputs: vec![a, b], output: out }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("Mixed") || msg.contains("shape"), "msg={msg}"); + } + + #[test] + fn test_cache_merge_rejects_chain_id_mismatch() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + write( + &a, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![], + external_env: None, + }) + .unwrap(), + ); + write( + &b, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 2, + cache: vec![], + external_env: None, + }) + .unwrap(), + ); + + let err = MergeArgs { inputs: vec![a, b], output: out }.run().unwrap_err(); + assert!(err.to_string().contains("chain_id")); + } + + #[test] + fn test_cache_merge_rejects_version_mismatch() { + let docs = vec![ + ( + PathBuf::from("a.json"), + EnvelopeDoc { version: 1, chain_id: 1, cache: vec![], external_env: None }, + ), + ( + PathBuf::from("b.json"), + EnvelopeDoc { version: 2, chain_id: 1, cache: vec![], external_env: None }, + ), + ]; + let err = merge_envelopes_cli(&docs).unwrap_err(); + assert!(err.to_string().contains("version")); + } + + /// Mismatched `rpc-cache-{id}.json` filenames hard-error naming both files. + #[test] + fn test_cache_merge_rejects_provider_chain_id_filename_mismatch() { + let dir = tempdir().unwrap(); + let a = dir.path().join("rpc-cache-1.json"); + let b = dir.path().join("rpc-cache-4326.json"); + let out = dir.path().join("rpc-cache-4326-out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "a")]).unwrap()); + write(&b, &serde_json::to_string(&vec![kv(2, "b")]).unwrap()); + + let err = MergeArgs { inputs: vec![a.clone(), b.clone()], output: out }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("chain identity") || msg.contains("chain"), "msg={msg}"); + assert!(msg.contains("chain 1") && msg.contains("chain 4326"), "msg={msg}"); + assert!( + msg.contains(a.file_name().unwrap().to_str().unwrap()) || + msg.contains("rpc-cache-1.json"), + "msg={msg}" + ); + assert!( + msg.contains(b.file_name().unwrap().to_str().unwrap()) || + msg.contains("rpc-cache-4326.json"), + "msg={msg}" + ); + } + + /// Output path is included in the chain-identity check. + #[test] + fn test_cache_merge_rejects_provider_output_chain_id_mismatch() { + let dir = tempdir().unwrap(); + let a = dir.path().join("rpc-cache-1.json"); + let out = dir.path().join("rpc-cache-4326.json"); + write(&a, &serde_json::to_string(&vec![kv(1, "a")]).unwrap()); + + let err = MergeArgs { inputs: vec![a], output: out }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("chain 1") && msg.contains("chain 4326"), "msg={msg}"); + } + + /// A provider-shaped output already on disk joins the union as one more + /// input: entries a concurrent writer left there survive the merge, and the + /// named inputs win where the keys collide. + #[test] + fn test_cache_merge_folds_the_existing_provider_output() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()); + write(&b, &serde_json::to_string(&vec![kv(2, "from-b")]).unwrap()); + // The output already holds a sibling's entry plus a stale copy of key 2. + write( + &out, + &serde_json::to_string(&vec![kv(2, "from-output"), kv(9, "concurrent")]).unwrap(), + ); + + MergeArgs { inputs: vec![a, b], output: out.clone() }.run().expect("merge"); + + let merged: Vec = + serde_json::from_str(&fs::read_to_string(&out).unwrap()).unwrap(); + assert_eq!(merged, vec![kv(1, "from-a"), kv(2, "from-b"), kv(9, "concurrent")]); + } + + /// The envelope shape folds its existing output the same way. + #[test] + fn test_cache_merge_folds_the_existing_envelope_output() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + let env_a = EnvelopeDoc { + version: 1, + chain_id: 4326, + cache: vec![kv(1, "from-a"), kv(2, "from-a")], + external_env: None, + }; + let existing = EnvelopeDoc { + version: 1, + chain_id: 4326, + cache: vec![kv(2, "from-output"), kv(9, "concurrent")], + external_env: Some(ExternalEnvDoc { bucket_capacities: vec![(1, 100)] }), + }; + write(&a, &serde_json::to_string_pretty(&env_a).unwrap()); + write(&out, &serde_json::to_string_pretty(&existing).unwrap()); + + MergeArgs { inputs: vec![a], output: out.clone() }.run().expect("merge"); + + let merged: EnvelopeDoc = serde_json::from_str(&fs::read_to_string(&out).unwrap()).unwrap(); + assert_eq!(merged.cache, vec![kv(1, "from-a"), kv(2, "from-a"), kv(9, "concurrent")]); + // The output's snapshot is preserved when the inputs carry none. + assert_eq!(merged.external_env, Some(ExternalEnvDoc { bucket_capacities: vec![(1, 100)] })); + } + + /// An unreadable provider output degrades to the merged inputs (warned), + /// matching the persist path's handling of a corrupt on-disk file. + #[test] + fn test_cache_merge_replaces_corrupt_provider_output() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()); + write(&out, "not-json{{{"); + + MergeArgs { inputs: vec![a], output: out.clone() }.run().expect("merge"); + + let merged: Vec = + serde_json::from_str(&fs::read_to_string(&out).unwrap()).unwrap(); + assert_eq!(merged, vec![kv(1, "from-a")]); + } + + /// An existing envelope output on another chain is an identity failure, not + /// something to overwrite. + #[test] + fn test_cache_merge_rejects_existing_envelope_output_on_another_chain() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write( + &a, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: None, + }) + .unwrap(), + ); + let existing = serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 2, + cache: vec![kv(9, "concurrent")], + external_env: None, + }) + .unwrap(); + write(&out, &existing); + + let err = MergeArgs { inputs: vec![a], output: out.clone() }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("chain_id"), "msg={msg}"); + assert!(msg.contains("out.json"), "the output must be named: msg={msg}"); + assert_eq!(fs::read_to_string(&out).unwrap(), existing, "output left untouched"); + } + + /// A capture envelope sitting at a provider merge's output is a hard error + /// for the same reason as the mirrored case below: a mistyped `--output` + /// must not destroy a file the merge cannot fold. + #[test] + fn test_cache_merge_rejects_wrong_shaped_existing_provider_output() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()); + let existing = serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(9, "concurrent")], + external_env: None, + }) + .unwrap(); + write(&out, &existing); + + let err = MergeArgs { inputs: vec![a], output: out.clone() }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("envelope"), "msg={msg}"); + assert_eq!(fs::read_to_string(&out).unwrap(), existing, "output left untouched"); + } + + /// A provider-shaped file sitting at an envelope merge's output is a hard + /// error: it cannot be folded, and overwriting it would destroy it. + #[test] + fn test_cache_merge_rejects_wrong_shaped_existing_output() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write( + &a, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: None, + }) + .unwrap(), + ); + let existing = serde_json::to_string(&vec![kv(9, "concurrent")]).unwrap(); + write(&out, &existing); + + let err = MergeArgs { inputs: vec![a], output: out.clone() }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("envelope"), "msg={msg}"); + assert_eq!(fs::read_to_string(&out).unwrap(), existing, "output left untouched"); + } + + /// Provider merge fails closed when the output lock cannot be acquired: the + /// existing output is left exactly as it was. + #[test] + fn test_cache_merge_provider_fails_closed_when_the_output_lock_is_unavailable() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write(&a, &serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()); + let existing = serde_json::to_string(&vec![kv(9, "concurrent")]).unwrap(); + write(&out, &existing); + // A directory in the sidecar's place makes the lock un-acquirable. + fs::create_dir(lock_sidecar_path(&out)).expect("occupy sidecar path"); + + let err = MergeArgs { inputs: vec![a], output: out.clone() }.run().unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("lock"), "msg={msg}"); + assert!(msg.contains("out.json.lock"), "the lock path must be named: msg={msg}"); + assert_eq!(fs::read_to_string(&out).unwrap(), existing, "no unlocked write happened"); + } + + /// Envelope merge fails closed on the same condition. + #[test] + fn test_cache_merge_envelope_fails_closed_when_the_output_lock_is_unavailable() { + let dir = tempdir().unwrap(); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + write( + &a, + &serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(1, "a")], + external_env: None, + }) + .unwrap(), + ); + let existing = serde_json::to_string_pretty(&EnvelopeDoc { + version: 1, + chain_id: 1, + cache: vec![kv(9, "concurrent")], + external_env: None, + }) + .unwrap(); + write(&out, &existing); + fs::create_dir(lock_sidecar_path(&out)).expect("occupy sidecar path"); + + let err = MergeArgs { inputs: vec![a], output: out.clone() }.run().unwrap_err(); + assert!(err.to_string().contains("lock"), "msg={err}"); + assert_eq!(fs::read_to_string(&out).unwrap(), existing, "no unlocked write happened"); + } + + /// Unit-testable predicate: non-matching filename cannot supply chain identity. + #[test] + fn test_provider_cache_chain_identity_non_matching_filename_is_none() { + assert_eq!(parse_rpc_cache_filename_chain_id(std::path::Path::new("merged.json")), None); + assert_eq!( + parse_rpc_cache_filename_chain_id(std::path::Path::new("rpc-cache-1.json")), + Some(1) + ); + } + + /// Same chain id across matching names is accepted (including output). + #[test] + fn test_check_provider_cache_chain_identity_same_id_ok() { + let paths = [ + std::path::Path::new("worker0/rpc-cache-4326.json"), + std::path::Path::new("worker1/rpc-cache-4326.json"), + std::path::Path::new("rpc-cache-4326.json"), + ]; + check_provider_cache_chain_identity(paths).expect("same chain ok"); + } + + /// Different ids hard-error; non-matching names alone do not. + #[test] + fn test_check_provider_cache_chain_identity_mismatch_and_non_matching() { + let paths = [ + std::path::Path::new("rpc-cache-1.json"), + std::path::Path::new("out.json"), // non-matching → warn only + std::path::Path::new("rpc-cache-4326.json"), + ]; + let err = check_provider_cache_chain_identity(paths).unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("rpc-cache-1.json") && msg.contains("rpc-cache-4326.json"), "{msg}"); + + // Only non-matching names: no chain id to disagree on → ok (with warns). + let only_free = [ + std::path::Path::new("a.json"), + std::path::Path::new("b.json"), + std::path::Path::new("merged.json"), + ]; + check_provider_cache_chain_identity(only_free).expect("no ids to conflict"); + } +} diff --git a/bin/mega-evme/src/cmd.rs b/bin/mega-evme/src/cmd.rs index 736f257a..f33fc473 100644 --- a/bin/mega-evme/src/cmd.rs +++ b/bin/mega-evme/src/cmd.rs @@ -1,5 +1,4 @@ use clap::{Parser, Subcommand}; -use tracing::error; use crate::common::LogArgs; @@ -26,6 +25,8 @@ pub enum Commands { Tx(crate::tx::Cmd), /// Replay a transaction from RPC Replay(crate::replay::Cmd), + /// Offline RPC cache utilities (`cache merge`, …) + Cache(crate::cache::Cmd), } /// Error types for the main command system @@ -40,29 +41,33 @@ pub enum Error { } impl MainCmd { - /// Execute the main command + /// Execute the main command. + /// + /// Failures are returned, never reported here: the binary hands the result + /// to [`crate::common::report_command_result`], which owns the single + /// failure report and the process exit code. pub async fn run(self) -> Result<(), Error> { // Initialize logging first self.log.init(); match self.command { - Commands::Run(cmd) => { - cmd.run().await?; - Ok(()) - } - Commands::Tx(cmd) => { - cmd.run().await?; - Ok(()) - } - Commands::Replay(cmd) => { - cmd.run().await?; - Ok(()) - } + Commands::Run(cmd) => cmd.run().await.map_err(Error::from), + Commands::Tx(cmd) => cmd.run().await.map_err(Error::from), + Commands::Replay(cmd) => cmd.run().await.map_err(Error::from), + Commands::Cache(cmd) => cmd.run().map_err(Error::from), + } + } + + /// Whether the selected subcommand was asked for machine-readable output. + /// + /// Read before [`Self::run`] consumes the command, so a failure is reported + /// in the output mode the user asked for. `cache` has no `--json` mode. + pub const fn json_output(&self) -> bool { + match &self.command { + Commands::Run(cmd) => cmd.output_args.json, + Commands::Tx(cmd) => cmd.output_args.json, + Commands::Replay(cmd) => cmd.output_args.json, + Commands::Cache(_) => false, } - .inspect_err(|e| { - error!(err = ?e, "Error executing command"); - eprintln!("{e}"); - std::process::exit(1); - }) } } diff --git a/bin/mega-evme/src/common/error.rs b/bin/mega-evme/src/common/error.rs index 69ca5bc0..f32932a1 100644 --- a/bin/mega-evme/src/common/error.rs +++ b/bin/mega-evme/src/common/error.rs @@ -5,6 +5,25 @@ use mega_evm::{ revm::{bytecode::BytecodeDecodeError, database_interface::bal::EvmDatabaseError}, }; +/// Stable `Display` prefix of [`EvmeError::RpcTransportError`]. +/// +/// Pre-block system calls (EIP-2935 / EIP-4788) and similar mega-evm wrappers +/// render a database failure into a message string with `to_string()`, so the +/// typed variant is gone by the time exit classification runs. The classifier +/// recovers the RPC class by stripping recognized outer wrappers and requiring +/// the remainder to start with this exact prefix. The `#[error(...)]` text on +/// the variant must keep the same prefix; the round-trip unit test in `exit` +/// enforces that. +pub const RPC_TRANSPORT_ERROR_PREFIX: &str = "RPC transport error: "; + +/// Stable `Display` prefix of [`EvmeError::RpcError`]. +/// +/// Same recovery contract as [`RPC_TRANSPORT_ERROR_PREFIX`]: fork-state reads +/// map transport/cache failures into this variant, and stringified block errors +/// still start with this prefix (after wrapper strip) so exit classification +/// can treat them as unanswered questions rather than execution failures. +pub const RPC_ERROR_PREFIX: &str = "RPC error: "; + /// Error types for the replay command #[derive(Debug, thiserror::Error)] pub enum EvmeError { @@ -16,6 +35,29 @@ pub enum EvmeError { #[error("Transaction not found: {0}")] TransactionNotFound(TxHash), + /// The block body listed this hash, but `eth_getTransactionByHash` returned null. + /// + /// That answer contradicts data the endpoint already served (the block body), + /// so the endpoint is inconsistent — typically a reorg or load-balanced + /// divergent views — rather than a definitive "unknown transaction". + #[error("Block body lists transaction {0} but the endpoint resolves it to null")] + BlockBodyTransactionNull(TxHash), + + /// The block body listed this hash, but fetching the transaction failed. + /// + /// A transport error, an offline cache miss, or a served transaction that + /// fails authentication against the requested hash is the same class as a + /// null answer: the endpoint failed to deliver a transaction it claimed to + /// include, rather than answering "unknown hash" about a user query. The + /// hash is carried so abort output can name the failing fetch. + #[error("Block body lists transaction {tx_hash} but fetching it failed: {message}")] + BlockBodyTransactionFetch { + /// Hash the block body listed and the lookup failed for. + tx_hash: TxHash, + /// Transport or cache-miss detail from the failed lookup. + message: String, + }, + /// Block not found #[error("Block not found: {0}")] BlockNotFound(BlockNumber), @@ -56,6 +98,30 @@ pub enum EvmeError { #[error("Unsupported transaction type: {0}")] UnsupportedTxType(u8), + /// A `replay --verify-receipt` run found at least one local replay that did + /// not reproduce the on-chain receipt. + /// + /// Distinct from the infrastructure error variants so a verification + /// mismatch can be told apart from a target that could not be replayed or + /// verified at all. + #[error( + "Receipt verification mismatch: {mismatched} of {total} verified transaction(s) did \ + not reproduce the on-chain receipt" + )] + VerificationMismatch { + /// Number of verified transactions whose replay diverged. + mismatched: usize, + /// Number of transactions that were verified. + total: usize, + }, + + /// A batch replay in which at least one target did not come out clean. + /// + /// Carries the counts by failure class so the exit-code mapping resolves the + /// batch precedence from data instead of parsing this message. + #[error("{0}")] + BatchFailed(BatchFailureCounts), + /// Code hash mismatch #[error("Code hash mismatch: expected {expected}, computed {computed}")] CodeHashMismatch { @@ -70,6 +136,73 @@ pub enum EvmeError { Other(String), } +/// Exit-code floor contributed by a non-target mid-block abort. +/// +/// Per-target failure counters stay strictly about reported targets. When the +/// aborting transaction is not itself a target, this floor still ranks the run +/// exit by the abort's root cause without inflating the "N of M target(s) +/// failed" totals. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum BatchExitFloor { + /// No non-target abort contributed a floor. + #[default] + None, + /// A non-target abort was execution-class (setup, executor rejection, …). + Execution, + /// A non-target abort was rpc-class (transport, cache miss, …). + Rpc, +} + +/// How many targets of a batch replay failed, by failure class. +/// +/// A batch run reports every target on its own output line and then fails once +/// with this summary, so the exit-code mapping can apply the batch precedence +/// (execution before RPC before mismatch) without re-reading the per-target +/// lines or parsing an error message. +/// +/// [`Self::execution`], [`Self::rpc`], [`Self::mismatched`], and [`Self::total`] +/// count only reported targets. [`Self::exit_floor`] is consulted solely for +/// exit ranking when a non-target abort's class is not carried by any target. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct BatchFailureCounts { + /// Targets that failed for an execution, setup, or definitive-answer reason + /// (unknown or pending transaction, block executor rejection, a fixture the + /// run was asked to write and could not). + pub execution: usize, + /// Targets whose question went unanswered because an RPC call failed. + pub rpc: usize, + /// Targets that replayed but did not reproduce their on-chain receipt. + pub mismatched: usize, + /// Targets the run reported on. + pub total: usize, + /// Non-target abort class that floors the run exit without being a target + /// failure count. Display ignores this for "N of M"; exit mapping uses it. + pub exit_floor: BatchExitFloor, +} + +impl core::fmt::Display for BatchFailureCounts { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + // Totals stay per-target: a non-target abort floor must not print + // "3 of 2 target transaction(s) failed". + write!( + f, + "{} of {} target transaction(s) failed ({} execution, {} rpc)", + self.execution + self.rpc, + self.total, + self.execution, + self.rpc, + )?; + if self.mismatched > 0 { + write!( + f, + "; {} replayed transaction(s) did not reproduce the on-chain receipt", + self.mismatched, + )?; + } + Ok(()) + } +} + // Implement DBErrorMarker to allow EvmeError to be used as Database error type impl mega_evm::revm::database::DBErrorMarker for EvmeError {} diff --git a/bin/mega-evme/src/common/exit.rs b/bin/mega-evme/src/common/exit.rs new file mode 100644 index 00000000..22a81421 --- /dev/null +++ b/bin/mega-evme/src/common/exit.rs @@ -0,0 +1,796 @@ +//! Central exit-code taxonomy for the `mega-evme` CLI. +//! +//! Verification pipelines branch on the process status, so every failure the +//! CLI can reach maps onto exactly one documented code, and every exit flows +//! through this module: the binary hands its top-level result to +//! [`report_command_result`] and returns the code it produces. Every *command +//! result* becomes a status here; the only other exit is the panic hook, which +//! reports an execution error for a failure no command result can describe. +//! +//! | Code | Class | Meaning | +//! | ---- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +//! | `0` | success | The command completed; with `--verify-receipt`, every verification matched. | +//! | `1` | `execution-error` | Execution or internal error: an EVM/setup failure, bad input, or a definitive negative answer such as an unknown transaction or block. | +//! | `2` | `verification-mismatch` | The run completed, but at least one replay did not reproduce its on-chain receipt. | +//! | `3` | `rpc-failure` | An RPC/transport call failed (endpoint unreachable, transport error, offline replay cache miss): the question went unanswered rather than answered no. | +//! +//! A batch run reports every target individually and then fails once with the +//! counts by failure class ([`BatchFailureCounts`]), which +//! [`ExitCode::from_batch_failures`] resolves by precedence: any +//! execution/internal failure yields `1`, else any RPC failure yields `3`, else +//! any mismatch yields `2`. +//! +//! Extension rule: a new failure class gets a new discriminant. The meaning of +//! an existing code never changes and a retired code is never reused, because +//! callers pin these numbers in scripts. The mapping matches the error enums +//! exhaustively, so a new error variant does not compile until it has been +//! assigned a class. + +use mega_evm::{ + alloy_evm::block::{BlockExecutionError, BlockValidationError}, + alloy_op_evm::OpTxError, + revm::{context::result::EVMError, database_interface::bal::EvmDatabaseError}, +}; +use serde::Serialize; +use tracing::error; + +use crate::{ + cmd::Error, + common::{BatchFailureCounts, EvmeError, RPC_ERROR_PREFIX, RPC_TRANSPORT_ERROR_PREFIX}, +}; + +/// The concrete EVM error a `mega-evme` block executor produces. +/// +/// The executor runs the `MegaEvmFactory` EVM (transaction error [`OpTxError`]) +/// over revm's `State<_>` wrapper around [`crate::EvmeState`], whose database +/// error is `EvmDatabaseError`. A fatal EVM error is boxed as +/// `dyn Error` inside the block error, so recovering the cause needs this exact +/// type. +type BlockEvmError = EVMError, OpTxError>; + +/// The database failure behind a block execution error, if it has one. +/// +/// A state read that fails mid-execution (offline cache miss, transport error) +/// is fatal, so the block executor keeps it in one of its internal branches as +/// a boxed `dyn Error`. Which wrapper it arrives in depends on where the read +/// happened — inside the EVM, or in the executor around it — so the cause is +/// recovered by downcasting to each concrete type it can be boxed as. Every +/// step is typed: the rendered message is never inspected. +/// +/// Not every block error carries its cause this way. A read that fails inside +/// the pre-block system calls (EIP-4788 beacon root, EIP-2935 block hashes) +/// is stringified into a [`BlockValidationError`] message field by mega-evm +/// before the error reaches here. For that path, see +/// [`stringified_rpc_failure`]. The keyless-deploy sandbox erases the cause +/// entirely (selector-only `InternalError`); that path cannot be recovered +/// bin-side. +fn database_cause(err: &BlockExecutionError) -> Option<&EvmeError> { + let boxed = match err { + BlockExecutionError::Internal(internal) => { + internal.as_evm().map(|(_, error)| error).or_else(|| internal.as_other())? + } + // Validation::EVM / Other box a non-tx failure the same way Internal does. + BlockExecutionError::Validation( + BlockValidationError::EVM { error, .. } | BlockValidationError::Other(error), + ) => error.as_ref(), + BlockExecutionError::Validation(_) => return None, + }; + + if let Some(evm_error) = boxed.downcast_ref::() { + return match evm_error { + EVMError::Database(database_error) => external_cause(database_error), + _ => None, + }; + } + if let Some(database_error) = boxed.downcast_ref::>() { + return external_cause(database_error); + } + boxed.downcast_ref::() +} + +/// The external database error behind a revm database error, if it is one. +const fn external_cause(err: &EvmDatabaseError) -> Option<&EvmeError> { + match err { + EvmDatabaseError::Database(cause) => Some(cause), + // A block access list error is the executor's own bookkeeping. + EvmDatabaseError::Bal(_) => None, + } +} + +/// Outer `Display` wrapper of alloy-evm [`BlockValidationError::BlockHashContractCall`]. +/// +/// Producer: alloy-evm's `#[error("failed to apply blockhash contract call: {message}")]` +/// on that variant; mega-evm's EIP-2935 pre-block path +/// (`transact_blockhashes_contract_call`) fills `message` with `e.to_string()`. +/// The classifier usually sees only the inner `message` field, but this constant +/// is also stripped so a full rendered validation string classifies the same way. +const BLOCKHASH_CONTRACT_CALL_WRAPPER: &str = "failed to apply blockhash contract call: "; + +/// Outer `Display` stem of alloy-evm [`BlockValidationError::BeaconRootContractCall`]. +/// +/// Producer: alloy-evm's beacon-root validation error / mega-evm's EIP-4788 +/// pre-block path (`transact_beacon_root_contract_call`). The full `Display` +/// inserts `at {parent_beacon_block_root}:` between this stem and the message; +/// the variant's `message` field itself does not carry this wrapper. +const BEACON_ROOT_CONTRACT_CALL_WRAPPER: &str = "failed to apply beacon root contract call: "; + +/// revm [`EvmDatabaseError::Database`] `Display` layer around the external DB error. +/// +/// Producer: revm-database-interface `EvmDatabaseError` formats +/// `Database error: {error}`; mega-evm pre-block helpers stringify the system-call +/// failure with `e.to_string()`, so this layer sits immediately outside the +/// crate-owned RPC `Display` prefix in the validation `message` field. +const DATABASE_ERROR_WRAPPER: &str = "Database error: "; + +/// Strip every recognized outer wrapper produced on the pre-block stringification +/// path, leaving the cause boundary for prefix matching. +/// +/// Only the wrappers documented above are removed, and only from the front of +/// the string (repeatedly). Anything else — including an RPC-looking substring +/// that appears mid-message — is left alone so incidental embeds stay +/// execution-class. +fn strip_recognized_wrappers(message: &str) -> &str { + let mut rest = message; + loop { + if let Some(stripped) = rest.strip_prefix(BLOCKHASH_CONTRACT_CALL_WRAPPER) { + rest = stripped; + continue; + } + if let Some(stripped) = rest.strip_prefix(BEACON_ROOT_CONTRACT_CALL_WRAPPER) { + rest = stripped; + continue; + } + if let Some(stripped) = rest.strip_prefix(DATABASE_ERROR_WRAPPER) { + rest = stripped; + continue; + } + break; + } + rest +} + +/// Whether a rendered message is an RPC-class failure at the cause boundary. +/// +/// Used only after typed recovery fails. Recognized mega-evm / revm / alloy-evm +/// outer wrappers are stripped first; the remainder must then +/// [`str::starts_with`] a stable [`EvmeError`] RPC [`Display`] prefix this crate +/// owns. A mere `contains` would misclassify an execution failure whose message +/// incidentally embeds `"RPC error: "` (revert data, user input, etc.). +fn message_carries_rpc_class(message: &str) -> bool { + let cause = strip_recognized_wrappers(message); + cause.starts_with(RPC_ERROR_PREFIX) || cause.starts_with(RPC_TRANSPORT_ERROR_PREFIX) +} + +/// Recover an RPC-class failure that mega-evm stringified into a validation +/// message, losing the typed [`EvmeError`] chain. +/// +/// Pre-block EIP-2935 / EIP-4788 helpers map a system-call database error to +/// [`BlockValidationError::BlockHashContractCall`] / +/// [`BlockValidationError::BeaconRootContractCall`] with `message: e.to_string()`. +/// The type is gone, but after stripping the recognized outer wrappers the +/// remainder still starts with this crate's RPC [`Display`] prefixes. Other +/// validation variants either keep a typed box (handled by [`database_cause`]) +/// or are genuine consensus/execution failures. +fn stringified_rpc_failure(err: &BlockExecutionError) -> bool { + let BlockExecutionError::Validation(validation) = err else { + return false; + }; + let message = match validation { + BlockValidationError::BlockHashContractCall { message } | + BlockValidationError::BeaconRootContractCall { message, .. } | + BlockValidationError::WithdrawalRequestsContractCall { message } | + BlockValidationError::ConsolidationRequestsContractCall { message } => message.as_str(), + // Typed boxes are recovered by `database_cause`; consensus-only variants + // never start with an RPC `Display` prefix after wrapper strip. + _ => return false, + }; + message_carries_rpc_class(message) +} + +/// Process exit status of a `mega-evme` run. +/// +/// The discriminants are the wire contract with calling scripts; see the module +/// documentation for the taxonomy and the rule for extending it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +pub enum ExitCode { + /// The command completed and every check it ran passed. + Success = 0, + /// Execution or internal error, bad input, or a definitive negative answer. + ExecutionError = 1, + /// The run completed but at least one receipt verification mismatched. + VerificationMismatch = 2, + /// An RPC or transport call failed, so the question went unanswered. + RpcFailure = 3, +} + +impl ExitCode { + /// The numeric status this class exits with. + pub const fn code(self) -> u8 { + self as u8 + } + + /// Kebab-case name of the class, used as the `kind` of the structured error + /// object printed in `--json` mode. + /// + /// This namespace describes the run as a whole and is distinct from the + /// per-target `kind` on a batch NDJSON error line (`not_found`, `pending`, + /// `rpc`, `execution`), which reports why one target failed. + pub const fn kind(self) -> &'static str { + match self { + Self::Success => "success", + Self::ExecutionError => "execution-error", + Self::VerificationMismatch => "verification-mismatch", + Self::RpcFailure => "rpc-failure", + } + } + + /// Map a top-level command error onto its class. + pub fn from_command_error(err: &Error) -> Self { + match err { + Error::Custom(_) => Self::ExecutionError, + Error::Evme(err) => Self::from_evme_error(err), + } + } + + /// Map a command failure onto its class. + /// + /// The match is exhaustive on purpose: a new [`EvmeError`] variant must + /// pick a class here rather than inherit one by default. + pub fn from_evme_error(err: &EvmeError) -> Self { + match err { + // The endpoint never answered: unreachable, transport-level + // failure, or an offline replay file without the response. + // `BlockBodyTransactionNull` / `BlockBodyTransactionFetch` are the + // same class: the block body already listed the hash, so a null or + // failed lookup is an inconsistent endpoint rather than a + // definitive unknown transaction. + EvmeError::RpcTransportError(_) | + EvmeError::RpcError(_) | + EvmeError::BlockBodyTransactionNull(_) | + EvmeError::BlockBodyTransactionFetch { .. } => Self::RpcFailure, + // A block error the EVM raised because a state read failed is that + // read's failure, not an execution result: classify it by its + // cause when the type survives, or by the stable RPC Display + // prefixes when mega-evm stringified the failure into a validation + // message (pre-block EIP-2935 / EIP-4788 system calls). + EvmeError::BlockExecutionError(err) => { + if let Some(cause) = database_cause(err) { + Self::from_evme_error(cause) + } else if stringified_rpc_failure(err) { + Self::RpcFailure + } else { + Self::ExecutionError + } + } + // Answered, definitively negative. + EvmeError::TransactionNotFound(_) | + EvmeError::BlockNotFound(_) | + // Execution, setup, input, and internal failures. + EvmeError::InvalidBytecode(_) | + EvmeError::FileRead(_) | + EvmeError::InvalidHex(_) | + EvmeError::ExecutionError(_) | + EvmeError::InvalidInput(_) | + EvmeError::FixtureError(_) | + EvmeError::UnsupportedTxType(_) | + EvmeError::CodeHashMismatch { .. } | + EvmeError::Other(_) => Self::ExecutionError, + // The run completed; the replay diverged from the chain. + EvmeError::VerificationMismatch { .. } => Self::VerificationMismatch, + EvmeError::BatchFailed(counts) => Self::from_batch_failures(counts), + } + } + + /// Resolve the class of a batch run from its failure counts. + /// + /// Precedence: an execution/internal failure outranks an RPC failure, which + /// outranks a mismatch. A target that never replayed was also never + /// verified, so reporting such a run as a mismatch would overstate what it + /// found. + /// + /// [`BatchFailureCounts::exit_floor`] ranks with the same precedence when a + /// non-target abort's class is not carried by any reported target: an + /// execution floor outranks target-only rpc failures, and an rpc floor still + /// yields rpc when every target was clean of infrastructure failures. + /// + /// Counts that record no failure at all reach this mapping only through a + /// batch aggregation bug, since the run reports a failure precisely when it + /// counted one. That is an internal error, not a success: a failure can + /// never produce exit `0`. + pub const fn from_batch_failures(counts: &BatchFailureCounts) -> Self { + use crate::common::BatchExitFloor; + + if counts.execution > 0 || matches!(counts.exit_floor, BatchExitFloor::Execution) { + Self::ExecutionError + } else if counts.rpc > 0 || matches!(counts.exit_floor, BatchExitFloor::Rpc) { + Self::RpcFailure + } else if counts.mismatched > 0 { + Self::VerificationMismatch + } else { + Self::ExecutionError + } + } +} + +impl From for std::process::ExitCode { + fn from(code: ExitCode) -> Self { + Self::from(code.code()) + } +} + +/// The structured failure object `--json` runs print as their last stdout line. +#[derive(Debug, Serialize)] +struct ErrorEnvelope<'a> { + error: ErrorBody<'a>, +} + +/// Payload of an [`ErrorEnvelope`]. +#[derive(Debug, Serialize)] +struct ErrorBody<'a> { + /// The process exit code this failure produces. + code: u8, + /// Kebab-case failure class, see [`ExitCode::kind`]. + kind: &'static str, + /// The error's `Display` text, never its `Debug` form. + message: &'a str, +} + +/// Report a finished command and return the code the process exits with. +/// +/// A successful run prints nothing. A failure is reported exactly once on +/// stderr as `error: `, always `Display`-formatted (the message itself +/// may carry continuation lines, such as an RPC error's re-capture hint), and +/// in `--json` mode additionally as the structured object on stdout — the last +/// line, so a machine-readable run never ends with empty stdout, and in batch +/// mode the object follows the per-target lines. The `tracing` event carries +/// the same text and stays silent unless `-v` was given. +pub fn report_command_result(result: Result<(), Error>, json: bool) -> ExitCode { + let Err(err) = result else { + return ExitCode::Success; + }; + + let code = ExitCode::from_command_error(&err); + error!(err = %err, exit_code = code.code(), "Command failed"); + + let message = err.to_string(); + eprintln!("error: {message}"); + if json { + print_json_error(code, &message); + } + + code +} + +/// Print the structured failure object of a `--json` run on stdout. +/// +/// Also used by failures that never become a command result — an argument +/// parsing error is reported by `clap` itself, but a machine-readable run must +/// still end its stdout with the object the taxonomy promises. +/// +/// Writes are fallible: a closed stdout is ignored rather than panicking. +/// The panic hook relies on this so a broken-pipe panic during normal output +/// still reaches `exit(1)` instead of aborting inside the hook. With an open +/// stdout the bytes match a successful `println!` of the same envelope. +pub fn print_json_error(code: ExitCode, message: &str) { + use std::io::Write; + + let envelope = + ErrorEnvelope { error: ErrorBody { code: code.code(), kind: code.kind(), message } }; + // Serialization of this envelope cannot fail for ordinary messages; still + // avoid `.expect` so a hook-path write never panics on its way to exit(1). + if let Ok(json) = serde_json::to_string(&envelope) { + let _ = writeln!(std::io::stdout(), "{json}"); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy_primitives::B256; + use mega_evm::alloy_evm::block::BlockValidationError; + + /// Every failure class the taxonomy defines keeps its documented code. + #[test] + fn test_exit_codes_are_stable() { + assert_eq!(ExitCode::Success.code(), 0); + assert_eq!(ExitCode::ExecutionError.code(), 1); + assert_eq!(ExitCode::VerificationMismatch.code(), 2); + assert_eq!(ExitCode::RpcFailure.code(), 3); + } + + /// The classifier recovers stringified RPC failures by the same prefixes + /// [`EvmeError`]'s `Display` emits — a drift between the two would silently + /// reclassify pre-block cache misses as execution errors. Outer wrappers + /// from the pre-block path are stripped before the `starts_with` check. + #[test] + fn test_rpc_display_prefixes_round_trip_with_classifier() { + let rpc = EvmeError::RpcError("cache miss in offline replay file".to_string()); + let rpc_text = rpc.to_string(); + assert!( + rpc_text.starts_with(RPC_ERROR_PREFIX), + "RpcError Display must start with RPC_ERROR_PREFIX, got: {rpc_text}" + ); + assert!(message_carries_rpc_class(&rpc_text)); + + let transport = EvmeError::RpcTransportError( + alloy_provider::transport::TransportErrorKind::custom_str("connection refused"), + ); + let transport_text = transport.to_string(); + assert!( + transport_text.starts_with(RPC_TRANSPORT_ERROR_PREFIX), + "RpcTransportError Display must start with RPC_TRANSPORT_ERROR_PREFIX, got: \ + {transport_text}" + ); + assert!(message_carries_rpc_class(&transport_text)); + + // Nested the way mega-evm stringifies a system-call DB failure into the + // validation message field (`Database error: RPC error: …`). + let nested = format!("{DATABASE_ERROR_WRAPPER}{rpc_text}"); + assert!(message_carries_rpc_class(&nested)); + assert_eq!(strip_recognized_wrappers(&nested), rpc_text.as_str()); + + // Full alloy-evm Display of BlockHashContractCall around that message. + let with_blockhash = format!("{BLOCKHASH_CONTRACT_CALL_WRAPPER}{nested}"); + assert!(message_carries_rpc_class(&with_blockhash)); + assert_eq!(strip_recognized_wrappers(&with_blockhash), rpc_text.as_str()); + + // Beacon-root stem (message field path has no stem; full Display may). + let with_beacon = format!("{BEACON_ROOT_CONTRACT_CALL_WRAPPER}{nested}"); + assert!(message_carries_rpc_class(&with_beacon)); + + // Execution-class pre-block halt: recognized wrapper, no RPC cause. + assert!(!message_carries_rpc_class(&format!("{BLOCKHASH_CONTRACT_CALL_WRAPPER}halt"))); + } + + /// An execution-class message that merely *contains* an RPC prefix substring + /// (e.g. revert data or user-echoed text) must not classify as exit 3. + /// Only a cause that *starts with* the prefix after wrapper strip is RPC. + #[test] + fn test_embedded_rpc_prefix_in_execution_message_maps_to_one() { + // Bare embed: contains the prefix, does not start with it. + let embedded = "system contract reverted: payload embeds RPC error: spoofed"; + assert!( + embedded.contains(RPC_ERROR_PREFIX) && !embedded.starts_with(RPC_ERROR_PREFIX), + "fixture must contain but not start with the RPC prefix" + ); + assert!(!message_carries_rpc_class(embedded)); + + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BlockHashContractCall { message: embedded.to_string() }, + )); + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "embedded RPC substring must stay execution-class: {err}" + ); + + // Same embed behind the Database-error wrapper: after strip the cause + // still does not start with the RPC prefix. + let wrapped_embed = + format!("{DATABASE_ERROR_WRAPPER}OutOfGas while echoing {RPC_ERROR_PREFIX}spoofed"); + assert!( + wrapped_embed.contains(RPC_ERROR_PREFIX) && + !strip_recognized_wrappers(&wrapped_embed).starts_with(RPC_ERROR_PREFIX), + "post-strip cause must not start with the RPC prefix" + ); + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BlockHashContractCall { message: wrapped_embed }, + )); + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "wrapped embed must stay execution-class: {err}" + ); + } + + /// The `kind` namespace is kebab-case and one name per class. + #[test] + fn test_exit_code_kinds_are_kebab_case() { + for code in [ + ExitCode::Success, + ExitCode::ExecutionError, + ExitCode::VerificationMismatch, + ExitCode::RpcFailure, + ] { + let kind = code.kind(); + assert!( + kind.chars().all(|c| c.is_ascii_lowercase() || c == '-'), + "kind must be kebab-case, got: {kind}" + ); + } + } + + /// An unanswered question is an RPC failure, whatever shape it arrived in. + #[test] + fn test_rpc_class_errors_map_to_three() { + for err in [ + EvmeError::RpcError("cache miss in offline replay file".to_string()), + EvmeError::RpcTransportError( + alloy_provider::transport::TransportErrorKind::custom_str("connection refused"), + ), + EvmeError::BlockBodyTransactionNull(B256::ZERO), + EvmeError::BlockBodyTransactionFetch { + tx_hash: B256::ZERO, + message: "cache miss in offline replay file".to_string(), + }, + ] { + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::RpcFailure, + "unexpected class for {err}" + ); + } + } + + /// Definitive answers, bad input, and internal failures all exit 1. + #[test] + fn test_execution_class_errors_map_to_one() { + for err in [ + EvmeError::TransactionNotFound(B256::ZERO), + EvmeError::BlockNotFound(7), + EvmeError::ExecutionError("halted".to_string()), + EvmeError::InvalidInput("no transaction hashes".to_string()), + EvmeError::FixtureError("unsupported transaction".to_string()), + EvmeError::UnsupportedTxType(0x7e), + EvmeError::CodeHashMismatch { expected: B256::ZERO, computed: B256::ZERO }, + EvmeError::FileRead(std::io::Error::other("boom")), + EvmeError::InvalidHex(alloy_primitives::hex::FromHexError::OddLength), + EvmeError::Other("something else".to_string()), + ] { + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "unexpected class for {err}" + ); + } + } + + /// A state read that failed mid-execution is an unanswered question, even + /// though it surfaced as a block execution error. + #[test] + fn test_block_execution_error_caused_by_a_database_failure_maps_to_three() { + let evm_err: BlockEvmError = EVMError::Database(EvmDatabaseError::Database( + EvmeError::RpcError("cache miss in offline replay file".to_string()), + )); + let err = EvmeError::BlockExecutionError(BlockExecutionError::evm(evm_err, B256::ZERO)); + + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::RpcFailure, + "unexpected class: {err}" + ); + } + + /// A block error with no database cause stays an execution failure. + #[test] + fn test_block_execution_error_without_a_database_cause_maps_to_one() { + let err = EvmeError::BlockExecutionError(BlockExecutionError::msg("gas limit reached")); + + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "unexpected class: {err}" + ); + } + + /// Pre-block EIP-2935 stringifies the system-call error into + /// `BlockHashContractCall { message }`. The typed cause is gone, but after + /// stripping the Database-error wrapper the remainder starts with this + /// crate's RPC Display prefix — classify as unanswered, not as an + /// execution rejection. + #[test] + fn test_stringified_blockhash_contract_call_rpc_failure_maps_to_three() { + let message = format!( + "{DATABASE_ERROR_WRAPPER}{}", + EvmeError::RpcError("Failed to fetch storage for history slot: cache miss".into()) + ); + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BlockHashContractCall { message }, + )); + + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::RpcFailure, + "unexpected class: {err}" + ); + } + + /// Pre-block EIP-4788 uses the same stringification path with a different + /// validation variant; the classifier must treat it the same way. + #[test] + fn test_stringified_beacon_root_contract_call_rpc_failure_maps_to_three() { + let message = format!( + "{DATABASE_ERROR_WRAPPER}{}", + EvmeError::RpcError("Failed to fetch storage for beacon root: cache miss".into()) + ); + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BeaconRootContractCall { + parent_beacon_block_root: Box::new(B256::ZERO), + message, + }, + )); + + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::RpcFailure, + "unexpected class: {err}" + ); + } + + /// A pre-block system-call failure that is not an unanswered RPC question + /// (for example a genuine EVM halt during the call) stays execution-class. + #[test] + fn test_stringified_blockhash_contract_call_without_rpc_prefix_maps_to_one() { + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BlockHashContractCall { message: "OutOfGas".to_string() }, + )); + + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "unexpected class: {err}" + ); + } + + /// Keyless-deploy sandbox DB failures are erased to a selector-only + /// `InternalError` inside mega-evm before any message reaches bin-side + /// classification. A synthetic stringified sandbox path that still carried + /// the RPC Display prefix (as `SandboxDbError` does before that final + /// erasure) is classified as RPC when it appears in a message-bearing + /// wrapper — pinning the prefix rule used for pre-block recovery. + #[test] + fn test_stringified_sandbox_style_rpc_prefix_maps_to_three() { + // SandboxDbError(e.to_string()) where e is EvmeError::RpcError(...). + let sandbox_db_message = + EvmeError::RpcError("Failed to fetch account during sandbox read".into()).to_string(); + assert!( + sandbox_db_message.starts_with(RPC_ERROR_PREFIX), + "sandbox stringification preserves the RPC prefix: {sandbox_db_message}" + ); + // If that message were embedded in a validation wrapper the same way + // pre-block helpers do, classification would recover it. + let err = EvmeError::BlockExecutionError(BlockExecutionError::Validation( + BlockValidationError::BlockHashContractCall { message: sandbox_db_message }, + )); + assert_eq!(ExitCode::from_evme_error(&err), ExitCode::RpcFailure); + } + + /// A completed run whose replay diverged from the chain exits 2. + #[test] + fn test_verification_mismatch_maps_to_two() { + let err = EvmeError::VerificationMismatch { mismatched: 1, total: 3 }; + assert_eq!(ExitCode::from_evme_error(&err), ExitCode::VerificationMismatch); + } + + /// The top-level wrapper adds no class of its own beyond the internal one. + #[test] + fn test_command_error_classes() { + assert_eq!( + ExitCode::from_command_error(&Error::Custom("bad state")), + ExitCode::ExecutionError + ); + assert_eq!( + ExitCode::from_command_error(&Error::Evme(EvmeError::RpcError("down".to_string()))), + ExitCode::RpcFailure + ); + } + + /// Batch precedence: execution beats rpc beats mismatch. + #[test] + fn test_batch_failure_precedence() { + let mixed = BatchFailureCounts { + execution: 1, + rpc: 2, + mismatched: 3, + total: 6, + ..Default::default() + }; + assert_eq!(ExitCode::from_batch_failures(&mixed), ExitCode::ExecutionError); + + let rpc_only = BatchFailureCounts { + execution: 0, + rpc: 2, + mismatched: 3, + total: 6, + ..Default::default() + }; + assert_eq!(ExitCode::from_batch_failures(&rpc_only), ExitCode::RpcFailure); + + let mismatch_only = BatchFailureCounts { + execution: 0, + rpc: 0, + mismatched: 3, + total: 6, + ..Default::default() + }; + assert_eq!(ExitCode::from_batch_failures(&mismatch_only), ExitCode::VerificationMismatch); + } + + /// A non-target execution abort floors exit 1 even when every reported + /// target failure is rpc (swept unanswered). + #[test] + fn test_batch_exit_floor_execution_outranks_target_rpc() { + use crate::common::BatchExitFloor; + + let counts = BatchFailureCounts { + execution: 0, + rpc: 2, + mismatched: 0, + total: 2, + exit_floor: BatchExitFloor::Execution, + }; + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::ExecutionError); + assert_eq!( + counts.to_string(), + "2 of 2 target transaction(s) failed (0 execution, 2 rpc)", + "floor must not inflate the target totals" + ); + } + + /// A non-target rpc abort floors exit 3 when targets alone would not. + #[test] + fn test_batch_exit_floor_rpc_when_targets_clean_of_infra() { + use crate::common::BatchExitFloor; + + let counts = BatchFailureCounts { + execution: 0, + rpc: 0, + mismatched: 1, + total: 1, + exit_floor: BatchExitFloor::Rpc, + }; + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::RpcFailure); + } + + /// A batch failure that counted nothing is an internal error, never a + /// success: an error variant must not be able to produce exit 0. + #[test] + fn test_batch_failure_without_counts_is_not_a_success() { + let counts = BatchFailureCounts::default(); + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::ExecutionError); + assert_eq!( + ExitCode::from_evme_error(&EvmeError::BatchFailed(counts)), + ExitCode::ExecutionError, + ); + } + + /// The aggregate error carries its counts through the top-level mapping. + #[test] + fn test_batch_failed_error_maps_by_counts() { + let rpc_only = EvmeError::BatchFailed(BatchFailureCounts { + execution: 0, + rpc: 1, + ..Default::default() + }); + assert_eq!(ExitCode::from_evme_error(&rpc_only), ExitCode::RpcFailure); + + let with_execution = EvmeError::BatchFailed(BatchFailureCounts { + execution: 1, + rpc: 1, + mismatched: 1, + total: 3, + ..Default::default() + }); + assert_eq!(ExitCode::from_evme_error(&with_execution), ExitCode::ExecutionError); + } + + /// The serialized object is a single compact line with the documented shape. + #[test] + fn test_error_envelope_shape() { + let code = ExitCode::RpcFailure; + let envelope = ErrorEnvelope { + error: ErrorBody { code: code.code(), kind: code.kind(), message: "endpoint down" }, + }; + let line = serde_json::to_string(&envelope).expect("serialize"); + + assert!(!line.contains('\n'), "the object must be a single line: {line}"); + let value: serde_json::Value = serde_json::from_str(&line).expect("valid JSON"); + assert_eq!( + value, + serde_json::json!({ + "error": { "code": 3, "kind": "rpc-failure", "message": "endpoint down" } + }) + ); + } +} diff --git a/bin/mega-evme/src/common/hardfork.rs b/bin/mega-evme/src/common/hardfork.rs index 3ffb9488..73a75dc2 100644 --- a/bin/mega-evme/src/common/hardfork.rs +++ b/bin/mega-evme/src/common/hardfork.rs @@ -1,23 +1,44 @@ +use core::any::Any; + use mega_evm::{ alloy_hardforks::{EthereumHardfork, ForkCondition}, alloy_op_hardforks::{EthereumHardforks, OpHardfork, OpHardforks}, - MegaHardfork, MegaHardforks, MegaSpecId, + MegaHardfork, MegaHardforkConfig, MegaHardforks, MegaSpecId, }; /// Fixed hardfork configuration for replay +/// +/// Activation follows the fixed spec alone: every hardfork whose spec is included in `spec` is +/// active at timestamp 0, and every later hardfork never activates. This is how mega-evme +/// expresses "a chain running spec N" without a real activation schedule. +/// +/// Per-fork parameters are *not* synthesized. A fixed-spec world still needs the chain's +/// parameters (the Rex5+ `SequencerRegistry` seeds are chain-specific data, not spec data), so an +/// optional parameter source can be attached with [`with_params_from`](Self::with_params_from). +/// Without one, parameter lookups return `None` as before. #[derive(Debug, Clone, Copy)] -pub struct FixedHardfork { +pub struct FixedHardfork<'a> { spec: MegaSpecId, + params: Option<&'a MegaHardforkConfig>, } -impl FixedHardfork { +impl<'a> FixedHardfork<'a> { /// Create a new [`FixedHardfork`] with the given `spec` pub fn new(spec: MegaSpecId) -> Self { - Self { spec } + Self { spec, params: None } + } + + /// Delegates per-fork parameter lookups to `config` while activation stays fixed. + /// + /// The delegation is wholesale rather than per parameter type: a parameter query carries no + /// activation check, so forwarding the whole lookup keeps every parameter type reachable — + /// including ones added later, which a hand-listed forwarding would silently drop. + pub fn with_params_from(self, config: &'a MegaHardforkConfig) -> Self { + Self { params: Some(config), ..self } } } -impl EthereumHardforks for FixedHardfork { +impl EthereumHardforks for FixedHardfork<'_> { fn ethereum_fork_activation(&self, fork: EthereumHardfork) -> ForkCondition { if fork <= EthereumHardfork::Prague { ForkCondition::Timestamp(0) @@ -27,7 +48,7 @@ impl EthereumHardforks for FixedHardfork { } } -impl OpHardforks for FixedHardfork { +impl OpHardforks for FixedHardfork<'_> { fn op_fork_activation(&self, fork: OpHardfork) -> ForkCondition { if fork <= OpHardfork::Isthmus { ForkCondition::Timestamp(0) @@ -37,7 +58,7 @@ impl OpHardforks for FixedHardfork { } } -impl MegaHardforks for FixedHardfork { +impl MegaHardforks for FixedHardfork<'_> { fn mega_fork_activation(&self, fork: MegaHardfork) -> ForkCondition { let mapped_spec = fork.spec_id(); if mapped_spec <= self.spec { @@ -46,4 +67,119 @@ impl MegaHardforks for FixedHardfork { ForkCondition::Never } } + + fn fork_params_any(&self, fork: MegaHardfork) -> Option<&(dyn Any + Send + Sync)> { + self.params?.fork_params_any(fork) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use mega_evm::{ + hardfork_schedule, SequencerRegistryConfig, SequencerRegistryRex6Config, MAINNET_CHAIN_ID, + }; + + /// A chain configuration carrying both parameter types, so the delegation can be checked for a + /// type the published schedules do not (yet) attach. + fn config_with_all_params() -> MegaHardforkConfig { + // The unknown-chain fallback attaches both the Rex5 and the Rex6 registry parameters. + hardfork_schedule(0xdead_beef) + } + + /// Activation is a function of the fixed spec alone, in both directions: everything up to the + /// spec is active at timestamp 0, everything above it never activates — no matter which + /// timestamp is asked about, and no matter what the attached parameter source schedules. + #[test] + fn test_activation_follows_the_fixed_spec_only() { + let chain = hardfork_schedule(MAINNET_CHAIN_ID); + let fixed = FixedHardfork::new(MegaSpecId::REX5).with_params_from(&chain); + + for fork in MegaHardfork::VARIANTS { + let expected = if fork.spec_id() <= MegaSpecId::REX5 { + ForkCondition::Timestamp(0) + } else { + ForkCondition::Never + }; + assert_eq!(fixed.mega_fork_activation(*fork), expected, "{fork:?}"); + } + + // The chain's own schedule puts Rex5 far in the future; the fixed world ignores it. + assert_eq!(fixed.spec_id(0), MegaSpecId::REX5); + assert_eq!(fixed.spec_id(u64::MAX), MegaSpecId::REX5); + } + + /// Every spec on the ladder resolves to itself, including the patch hardforks that map back to + /// an earlier spec. + #[test] + fn test_every_spec_resolves_to_itself() { + for spec in [ + MegaSpecId::EQUIVALENCE, + MegaSpecId::MINI_REX, + MegaSpecId::REX, + MegaSpecId::REX1, + MegaSpecId::REX2, + MegaSpecId::REX3, + MegaSpecId::REX4, + MegaSpecId::REX5, + MegaSpecId::REX6, + MegaSpecId::REX7, + ] { + assert_eq!(FixedHardfork::new(spec).spec_id(0), spec, "{spec:?}"); + } + } + + /// Without a parameter source, parameter lookups stay empty — the behavior the `run` / `tx` + /// commands rely on. + #[test] + fn test_bare_fixed_hardfork_has_no_params() { + let fixed = FixedHardfork::new(MegaSpecId::REX6); + assert!(fixed.fork_params::().is_none()); + assert!(fixed.fork_params::().is_none()); + } + + /// With a parameter source, lookups return the source's values verbatim — for every parameter + /// type it carries, not just the one the current deploy path happens to need. + #[test] + fn test_params_are_delegated_to_the_chain_config() { + let chain = config_with_all_params(); + let fixed = FixedHardfork::new(MegaSpecId::REX7).with_params_from(&chain); + + assert_eq!( + fixed.fork_params::(), + chain.fork_params::(), + ); + assert_eq!( + fixed.fork_params::(), + chain.fork_params::(), + ); + assert!(fixed.fork_params::().is_some()); + assert!(fixed.fork_params::().is_some()); + } + + /// The mainnet schedule's Rex5 parameters survive the swap: this is what keeps a Rex5+ + /// override from failing closed in the pre-block `SequencerRegistry` deploy. + #[test] + fn test_mainnet_rex5_params_survive_the_swap() { + let chain = hardfork_schedule(MAINNET_CHAIN_ID); + let fixed = FixedHardfork::new(MegaSpecId::REX5).with_params_from(&chain); + + assert_eq!( + fixed.fork_params::(), + chain.fork_params::(), + ); + assert!(fixed.fork_params::().is_some()); + } + + /// Parameter lookups do not consult activation: a parameter attached to a fork the fixed spec + /// never activates is still returned. Delegating wholesale therefore cannot depend on the + /// order in which parameters and specs were chosen. + #[test] + fn test_params_lookup_is_independent_of_activation() { + let chain = config_with_all_params(); + let fixed = FixedHardfork::new(MegaSpecId::EQUIVALENCE).with_params_from(&chain); + + assert_eq!(fixed.mega_fork_activation(MegaHardfork::Rex5), ForkCondition::Never); + assert!(fixed.fork_params::().is_some()); + } } diff --git a/bin/mega-evme/src/common/mod.rs b/bin/mega-evme/src/common/mod.rs index 96ddf3e2..7555f366 100644 --- a/bin/mega-evme/src/common/mod.rs +++ b/bin/mega-evme/src/common/mod.rs @@ -1,5 +1,6 @@ mod env; mod error; +mod exit; mod hardfork; mod hex; mod logging; @@ -12,6 +13,7 @@ mod tx_override; pub use env::*; pub use error::*; +pub use exit::*; pub use hardfork::*; pub use hex::*; pub use logging::*; diff --git a/bin/mega-evme/src/common/outcome.rs b/bin/mega-evme/src/common/outcome.rs index 71b37c41..2fd1ab09 100644 --- a/bin/mega-evme/src/common/outcome.rs +++ b/bin/mega-evme/src/common/outcome.rs @@ -5,7 +5,7 @@ use std::{path::Path, time::Duration}; use super::{EvmeError, StateDumpArgs, TraceArgs}; use alloy_consensus::{Eip658Value, Receipt}; -use alloy_primitives::{hex, Address, BlockHash, Bytes, TxHash, B256}; +use alloy_primitives::{hex, Address, BlockHash, Bytes, TxHash}; use alloy_rpc_types_eth::TransactionReceipt; use alloy_sol_types::{Panic, Revert, SolError}; use clap::Parser; @@ -68,6 +68,11 @@ impl EvmeOutcome { } /// Convert an [`OpReceiptEnvelope`] to an OP transaction receipt. +/// +/// `first_log_index` is the block-global log index of this receipt's first log +/// (the cumulative log count of all preceding receipts in the block). Each +/// inner log is stamped with the same block/tx identity as the outer receipt so +/// the JSON is self-consistent. #[allow(clippy::too_many_arguments)] pub fn op_receipt_to_tx_receipt( receipt: &OpReceiptEnvelope, @@ -81,17 +86,24 @@ pub fn op_receipt_to_tx_receipt( transaction_hash: Option, // only used for replay command where tx hash is known block_hash: Option, // only used for replay command where block hash is known transaction_index: u64, + first_log_index: u64, ) -> OpTxReceipt { - // Map logs to include block/tx metadata - let mut log_index = 0; + // Resolve the effective tx hash once so the outer receipt and every inner + // log agree: a missing hash becomes `B256::ZERO` on both sides (the outer + // field is non-optional on `TransactionReceipt`). + let effective_tx_hash = transaction_hash.unwrap_or_default(); + let stamped_tx_hash = Some(effective_tx_hash); + + // Map logs to include block/tx metadata matching the outer receipt. + let mut log_index = first_log_index; let inner = receipt.clone().map_logs(|log| { let log = alloy_rpc_types_eth::Log { inner: log, - block_hash: None, + block_hash, block_number: Some(block_number), block_timestamp: Some(block_timestamp), - transaction_hash: Some(B256::ZERO), - transaction_index: Some(0), + transaction_hash: stamped_tx_hash, + transaction_index: Some(transaction_index), log_index: Some(log_index), removed: false, }; @@ -101,7 +113,7 @@ pub fn op_receipt_to_tx_receipt( TransactionReceipt { inner, - transaction_hash: transaction_hash.unwrap_or_default(), + transaction_hash: effective_tx_hash, transaction_index: Some(transaction_index), block_hash, block_number: Some(block_number), @@ -275,6 +287,10 @@ pub struct ExecutionSummary { /// Transaction receipt (present only for `tx` command) #[serde(skip_serializing_if = "Option::is_none")] pub receipt: Option, + /// On-chain receipt verification verdict (present only for `replay + /// --verify-receipt`) + #[serde(skip_serializing_if = "Option::is_none")] + pub verification: Option, } impl ExecutionSummary { @@ -353,7 +369,7 @@ impl ExecutionSummary { #[cfg(test)] mod tests { use super::*; - use alloy_primitives::Bytes; + use alloy_primitives::{address, b256, Bytes, Log as PrimitiveLog, LogData, B256}; use alloy_sol_types::SolError; #[test] @@ -379,4 +395,101 @@ mod tests { let raw = Bytes::from(vec![0xde, 0xad]); assert_eq!(decode_revert_reason(&raw), "0xdead"); } + + /// Inner logs carry the same block/tx identity as the outer receipt, and + /// `log_index` is the block-global index starting at `first_log_index`. + #[test] + fn test_op_receipt_to_tx_receipt_stamps_inner_log_metadata() { + let addr = address!("0x00000000000000000000000000000000000000aa"); + let topic = b256!("0x000000000000000000000000000000000000000000000000000000000000000a"); + let log = PrimitiveLog { + address: addr, + data: LogData::new(vec![topic], Bytes::from(vec![0xde, 0xad])).expect("topics"), + }; + let receipt = OpReceiptEnvelope::Legacy( + alloy_consensus::Receipt { + status: Eip658Value::Eip658(true), + cumulative_gas_used: 21_000, + logs: vec![log.clone(), log], + } + .with_bloom(), + ); + let tx_hash = b256!("0x1111111111111111111111111111111111111111111111111111111111111111"); + let block_hash = + b256!("0x2222222222222222222222222222222222222222222222222222222222222222"); + let from = address!("0x00000000000000000000000000000000000000bb"); + + let tx_receipt = op_receipt_to_tx_receipt( + &receipt, + 42, + 1_700_000_000, + from, + Some(addr), + None, + 1, + 21_000, + Some(tx_hash), + Some(block_hash), + 3, + 7, // two preceding receipts already emitted 7 logs in this block + ); + + assert_eq!(tx_receipt.transaction_hash, tx_hash); + assert_eq!(tx_receipt.block_hash, Some(block_hash)); + assert_eq!(tx_receipt.transaction_index, Some(3)); + let logs = tx_receipt.inner.logs(); + assert_eq!(logs.len(), 2); + for (i, log) in logs.iter().enumerate() { + assert_eq!(log.block_hash, Some(block_hash), "log {i} block_hash"); + assert_eq!(log.transaction_hash, Some(tx_hash), "log {i} transaction_hash"); + assert_eq!(log.transaction_index, Some(3), "log {i} transaction_index"); + assert_eq!(log.log_index, Some(7 + i as u64), "log {i} block-global log_index"); + assert_eq!(log.block_number, Some(42)); + } + } + + /// A missing transaction hash becomes `B256::ZERO` on the outer receipt and + /// the same value on every inner log (not `None` on logs / zero only outside). + #[test] + fn test_op_receipt_to_tx_receipt_missing_tx_hash_stamps_outer_and_inner_consistently() { + let addr = address!("0x00000000000000000000000000000000000000aa"); + let topic = b256!("0x000000000000000000000000000000000000000000000000000000000000000a"); + let log = PrimitiveLog { + address: addr, + data: LogData::new(vec![topic], Bytes::from(vec![0xbe, 0xef])).expect("topics"), + }; + let receipt = OpReceiptEnvelope::Legacy( + alloy_consensus::Receipt { + status: Eip658Value::Eip658(true), + cumulative_gas_used: 21_000, + logs: vec![log], + } + .with_bloom(), + ); + let from = address!("0x00000000000000000000000000000000000000bb"); + + let tx_receipt = op_receipt_to_tx_receipt( + &receipt, + 1, + 1, + from, + Some(addr), + None, + 1, + 21_000, + None, + None, + 0, + 0, + ); + + assert_eq!(tx_receipt.transaction_hash, B256::ZERO); + let logs = tx_receipt.inner.logs(); + assert_eq!(logs.len(), 1); + assert_eq!( + logs[0].transaction_hash, + Some(B256::ZERO), + "inner log must use the same effective hash as the outer receipt" + ); + } } diff --git a/bin/mega-evme/src/common/provider/cache_store.rs b/bin/mega-evme/src/common/provider/cache_store.rs index ef7c717e..25133fe9 100644 --- a/bin/mega-evme/src/common/provider/cache_store.rs +++ b/bin/mega-evme/src/common/provider/cache_store.rs @@ -8,10 +8,23 @@ //! //! The envelope is v1. Forward-incompatible changes bump `ENVELOPE_VERSION`; //! additive fields use `#[serde(default)]` instead. +//! +//! # Concurrent cache-dir sharing +//! +//! Persist takes an exclusive advisory lock on a sidecar `.lock`, re-reads +//! the target file, merges in-memory entries over on-disk ones (ours win on key +//! collision), then writes via temp-file + atomic rename. Multiple processes may +//! therefore share one `--rpc.cache-dir` without losing each other's entries. +//! The lock sidecar is left in place after the process exits (the flock is released +//! when the lock file handle is closed). +//! +//! Persist fails closed on the lock: if the lock cannot be acquired, nothing is +//! written. Writing unlocked would reintroduce exactly the lost-update race the +//! lock exists to prevent, and it would do so silently — a sibling process's +//! entries would vanish under our rename. use std::{ fmt, fs, - io::Write as _, path::{Path, PathBuf}, }; @@ -20,13 +33,21 @@ use serde::{Deserialize, Serialize}; use tracing::{info, warn}; use super::transport::TransportCache; -use crate::common::{EvmeError, Result}; +use crate::{ + cache::{ + acquire_exclusive_lock, lock_sidecar_path, merge_envelope_for_persist, + merge_provider_entries_capped, reread_envelope_for_merge, reread_provider_cache_for_merge, + warn_user, write_bytes_atomic, write_envelope_atomic, CacheKv, EnvelopeDoc, EnvelopeReread, + ExternalEnvDoc, ProviderReread, ENVELOPE_VERSION, + }, + common::{EvmeError, Result}, +}; /// Clean-exit cache persistence handle. /// -/// An `RpcCacheStore` may internally have nothing to persist — non-fork run, -/// `--rpc.cache-size 0`, or `--rpc.no-cache-file`. In any of those cases -/// `persist()` is a no-op. Callers do not and must not branch on whether +/// An `RpcCacheStore` may internally have nothing to persist — a non-fork run, +/// or `--rpc.no-cache-file`. In either case `persist()` is a no-op. Callers do +/// not and must not branch on whether /// a given store is real or no-op; the whole point of this type is a single /// uniform persistence entry point. /// @@ -59,6 +80,11 @@ enum RpcCacheStoreInner { /// by the command layer once it has computed the effective value /// from CLI + prior envelope. external_env: Option, + /// Snapshot observed when the capture file was loaded (or `None` when + /// the file was absent / carried no snapshot). Used for optimistic + /// concurrency at persist: intentional A→B refreshes are accepted when + /// the locked re-read is still A; only a true concurrent change conflicts. + loaded_external_env: Option, }, } @@ -74,15 +100,27 @@ impl RpcCacheStore { /// Construct a store backed by a transport-level fixture envelope file. /// /// `pub(super)` to keep the `TransportCache` parameter from leaking out of - /// this module. The snapshot field starts empty; callers inject it later + /// this module. The write snapshot starts empty; callers inject it later /// via [`Self::set_external_env`]. - pub(super) fn new_envelope(cache: TransportCache, path: PathBuf, chain_id: u64) -> Self { + /// + /// `loaded_external_env` is the load-time baseline already observed by the + /// caller when it opened the capture file (if any). Persist compares the + /// locked re-read against this value so intentional A→B refreshes are + /// accepted when the on-disk snapshot is still A. The baseline must come + /// from that first load — this constructor must not re-read the file. + pub(super) fn new_envelope( + cache: TransportCache, + path: PathBuf, + chain_id: u64, + loaded_external_env: Option, + ) -> Self { Self { inner: Some(RpcCacheStoreInner::FixtureCapture { cache, path, chain_id, external_env: None, + loaded_external_env, }), } } @@ -145,6 +183,11 @@ impl RpcCacheStore { /// For fixture-capture stores, any `external_env` snapshot previously /// attached via [`Self::set_external_env`] is written into the envelope. /// + /// Persist takes an exclusive advisory lock on `.lock`, re-reads the + /// on-disk file (a sibling process may have written since load), and merges + /// our in-memory entries over the on-disk ones (ours win on key collision) + /// before the atomic write. + /// /// - **`ProviderCache`**: best-effort — failures are warn-logged and swallowed. /// - **`FixtureCapture`**: hard error — the fixture is the primary output of capture mode. /// - **No-op**: returns `Ok(())`. @@ -153,7 +196,9 @@ impl RpcCacheStore { match inner { RpcCacheStoreInner::ProviderCache { cache, path } => { match save_cache_atomic(&cache, &path) { - Ok(()) => info!(path = %path.display(), "Persisted RPC cache"), + Ok(true) => info!(path = %path.display(), "Persisted RPC cache"), + // Intentional skip (e.g. foreign on-disk shape) already warned inside. + Ok(false) => {} Err(err) => warn!( path = %path.display(), error = %err, @@ -162,9 +207,16 @@ impl RpcCacheStore { } Ok(()) } - RpcCacheStoreInner::FixtureCapture { cache, path, chain_id, external_env } => { + RpcCacheStoreInner::FixtureCapture { + cache, + path, + chain_id, + external_env, + loaded_external_env, + } => { let entry_count = cache.len(); - CacheFileEnvelope::new(&cache, chain_id, external_env.as_ref()).save(&path)?; + CacheFileEnvelope::new(&cache, chain_id, external_env.as_ref()) + .save(&path, loaded_external_env.as_ref())?; info!( path = %path.display(), entries = entry_count, @@ -189,40 +241,107 @@ impl fmt::Debug for RpcCacheStore { } } -/// Atomically persist `cache` to `target` via a temp file + rename. +/// Atomically persist `cache` to `target` via lock + re-read-merge + temp rename. /// -/// All error paths include `target` in the returned [`std::io::Error`] so the -/// warn-log in [`RpcCacheStore::persist`] identifies which file failed. -fn save_cache_atomic(cache: &SharedCache, target: &Path) -> std::io::Result<()> { +/// Returns `Ok(true)` when the file was written, `Ok(false)` when the write was +/// intentionally skipped (a recognizable foreign on-disk shape), and `Err` on +/// lock/IO failure. All error paths include `target` in the returned +/// [`std::io::Error`] so the warn-log in [`RpcCacheStore::persist`] identifies +/// which file failed. +/// +/// Lock acquisition failure aborts the persist: the provider cache is a +/// best-effort artifact, so skipping it costs a re-fetch, while an unlocked +/// write can silently delete a sibling process's entries. +/// +/// On-disk re-read is typed (same classification as `cache merge`): +/// - missing / provider array → merge and write; +/// - corrupt / unreadable → degrade to ours-only (with a `warn!`); +/// - recognizable foreign shape (capture envelope, …) → skip the write with a visible warning so a +/// mispointed cache-dir cannot destroy the foreign file. +fn save_cache_atomic(cache: &SharedCache, target: &Path) -> std::io::Result { + let _guard = acquire_exclusive_lock(target).map_err(|e| { + std::io::Error::other(format!( + "failed to acquire the cache lock {} for {}: {e}; \ + cache entries were not saved (an unlocked write could drop a \ + concurrent process's entries)", + lock_sidecar_path(target).display(), + target.display(), + )) + })?; + let dir = target.parent().unwrap_or_else(|| Path::new(".")); - let tmp = tempfile::NamedTempFile::new_in(dir).map_err(|e| { - std::io::Error::other(format!("failed to create temp file in {}: {e}", dir.display())) + fs::create_dir_all(dir).map_err(|e| { + std::io::Error::other(format!("failed to create directory {}: {e}", dir.display())) })?; - let tmp_path = tmp.path().to_path_buf(); - // alloy's save_cache takes a PathBuf, not a Write. - cache.save_cache(tmp_path).map_err(|e| { + // SharedCache has no iteration API — dump our entries to a temp file and re-read. + let our_tmp = tempfile::NamedTempFile::new_in(dir).map_err(|e| { + std::io::Error::other(format!("failed to create temp file in {}: {e}", dir.display())) + })?; + let our_tmp_path = our_tmp.path().to_path_buf(); + cache.save_cache(our_tmp_path.clone()).map_err(|e| { std::io::Error::other(format!("failed to save cache for {}: {e}", target.display())) })?; - // Atomic rename. persist() consumes the NamedTempFile without deleting it. - tmp.persist(target).map_err(|e| { + let our_entries: Vec = match fs::read_to_string(&our_tmp_path) + .map_err(|e| e.to_string()) + .and_then(|s| serde_json::from_str(&s).map_err(|e| e.to_string())) + { + Ok(entries) => entries, + Err(err) => { + return Err(std::io::Error::other(format!( + "failed to re-read our cache dump for {}: {err}", + target.display() + ))); + } + }; + // Drop the NamedTempFile so it is unlinked; we only needed the dump bytes. + drop(our_tmp); + + let disk_entries = match reread_provider_cache_for_merge(target) { + ProviderReread::Ok(entries) => entries, + ProviderReread::Degradable(msg) => { + warn!( + path = %target.display(), + error = %msg, + "Failed to re-read on-disk RPC cache during merge; persisting our entries only", + ); + Vec::new() + } + ProviderReread::Hard(err) => { + // Best-effort provider persist must not destroy a foreign file that + // a shared-dir misconfiguration pointed it at (e.g. a capture + // envelope). Skip the write so the foreign content survives. + // stderr via `warn_user`: default CLI tracing is off, so a + // `warn!`-only line would never reach the operator who needs to + // fix the shared-dir misconfiguration. + warn_user(format_args!( + "Skipping RPC cache persist to '{}': {err}. On-disk file is not a \ + provider cache; leaving it intact", + target.display(), + )); + return Ok(false); + } + }; + + // The union must respect the configured cap: a sibling's file plus ours can + // otherwise exceed what either run was allowed to keep. + let merged = + merge_provider_entries_capped(disk_entries, our_entries, cache.max_items() as usize); + let serialized = serde_json::to_vec(&merged).map_err(|e| { std::io::Error::other(format!( - "failed to rename temp file into {}: {}", - target.display(), - e.error, + "failed to serialize merged cache for {}: {e}", + target.display() )) })?; - Ok(()) + write_bytes_atomic(target, &serialized)?; + Ok(true) } -/// Envelope version accepted by this build. -const ENVELOPE_VERSION: u32 = 1; - /// On-disk envelope format shared by `--rpc.capture-file` (write) and /// `--rpc.replay-file` (read). Contains a transport-level cache dump, /// chain ID, and optional external environment snapshot. -#[derive(Debug, Serialize, Deserialize)] +#[derive(Debug, Clone, Serialize, Deserialize)] pub(super) struct CacheFileEnvelope { /// Schema version (currently always 1, reserved for future format changes). version: u32, @@ -274,41 +393,90 @@ impl CacheFileEnvelope { Ok(envelope) } - /// Atomically write this envelope to `path`. - pub(super) fn save(&self, path: &Path) -> Result<()> { - let dir = path.parent().unwrap_or_else(|| Path::new(".")); - fs::create_dir_all(dir).map_err(|e| { + /// Atomically write this envelope to `path` under a lock, merging with any + /// on-disk envelope already present (ours win on cache key collision). + /// + /// `loaded_external_env` is the snapshot observed when this process opened + /// the capture file. Persist accepts an intentional A→B refresh when the + /// locked re-read is still A (or absent); a concurrent change to a third + /// snapshot C hard-errors and names loaded/ours/on-disk. See + /// [`merge_envelope_for_persist`]. + /// + /// On-disk re-read failures are typed: identity/schema mismatches hard-fail; + /// corrupt JSON degrades to ours-only with a warning. + /// + /// Lock contention blocks until the lock is free. Failure to create/acquire + /// the lock is a hard error — the envelope is the primary output of capture + /// mode, so an unlocked write that silently drops a concurrent writer's + /// entries is worse than a failed run. Write failures remain hard errors. + pub(super) fn save( + &self, + path: &Path, + loaded_external_env: Option<&ExternalEnvSnapshot>, + ) -> Result<()> { + let _guard = acquire_exclusive_lock(path).map_err(|e| { EvmeError::FixtureError(format!( - "Failed to create cache file directory {}: {e}", - dir.display() + "Failed to acquire the cache lock {} for envelope '{}': {e}. \ + Refusing to write it unlocked: a concurrent writer's entries would be lost.", + lock_sidecar_path(path).display(), + path.display(), )) })?; - let serialized = serde_json::to_string_pretty(self).map_err(|e| { - EvmeError::FixtureError(format!( - "Failed to serialize envelope for {}: {e}", - path.display() - )) - })?; + let ours = self.to_merge_doc()?; + let loaded_doc = loaded_external_env + .map(|e| ExternalEnvDoc { bucket_capacities: e.bucket_capacities.clone() }); + let to_write = if path.exists() { + // Typed hard vs degradable: no substring matching on formatted messages. + match reread_envelope_for_merge(path) { + EnvelopeReread::Ok(on_disk) => { + merge_envelope_for_persist(&on_disk, &ours, loaded_doc.as_ref(), path)? + } + EnvelopeReread::Hard(err) => return Err(err), + EnvelopeReread::Degradable(msg) => { + warn!( + path = %path.display(), + error = %msg, + "Failed to re-read on-disk envelope during merge; persisting our entries only", + ); + // Still write the canonical form when replacing corrupt content. + canonicalize_envelope_external_env(ours) + } + } + } else { + canonicalize_envelope_external_env(ours) + }; - let mut tmp = tempfile::NamedTempFile::new_in(dir).map_err(|e| { - EvmeError::FixtureError(format!("Failed to create temp file in {}: {e}", dir.display())) - })?; - tmp.write_all(serialized.as_bytes()) - .map_err(|e| EvmeError::FixtureError(format!("Failed to write envelope: {e}")))?; - tmp.persist(path).map_err(|e| { - EvmeError::FixtureError(format!( - "Failed to persist envelope to {}: {e}", - path.display() - )) + write_envelope_atomic(path, &to_write) + } + + fn to_merge_doc(&self) -> Result { + let cache: Vec = serde_json::from_value(self.cache.clone()).map_err(|e| { + EvmeError::FixtureError(format!("Failed to decode envelope cache entries: {e}")) })?; + Ok(EnvelopeDoc { + version: self.version, + chain_id: self.chain_id, + cache, + external_env: self + .external_env + .as_ref() + .map(|e| ExternalEnvDoc { bucket_capacities: e.bucket_capacities.clone() }), + }) + } +} - Ok(()) +/// Canonicalize `external_env` on a merge doc about to be written alone (no +/// on-disk merge). Merge path already returns a canonical snapshot. +fn canonicalize_envelope_external_env(mut doc: EnvelopeDoc) -> EnvelopeDoc { + if let Some(ext) = doc.external_env.take() { + doc.external_env = Some(ext.canonicalized()); } + doc } /// Snapshot of mega-evm external environment inputs not derivable from RPC. -#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)] pub struct ExternalEnvSnapshot { /// SALT bucket capacity pairs `(bucket_id, capacity)`. #[serde(default)] @@ -317,7 +485,8 @@ pub struct ExternalEnvSnapshot { #[cfg(test)] mod tests { - use alloy_primitives::keccak256; + use alloy_primitives::{keccak256, B256}; + use alloy_provider::layers::CacheLayer; use super::*; @@ -339,7 +508,7 @@ mod tests { .expect("seed cache"); let ext = ExternalEnvSnapshot { bucket_capacities: vec![(1, 100), (2, 200)] }; - CacheFileEnvelope::new(&cache, 4326, Some(&ext)).save(&path).expect("save envelope"); + CacheFileEnvelope::new(&cache, 4326, Some(&ext)).save(&path, None).expect("save envelope"); let envelope = CacheFileEnvelope::load(&path).expect("load envelope"); assert_eq!(envelope.version, 1); @@ -392,4 +561,546 @@ mod tests { let msg = format!("{err}"); assert!(msg.contains("parse"), "error should mention parse: {msg}"); } + + /// Interleaving: A holds only key A in memory; B persists key B; A then + /// Persisting into a shared cache directory keeps the file within the + /// configured cap. + /// + /// Runs that share a directory touch disjoint RPC keys, so merging a + /// sibling's file in wholesale would grow it past what either run was + /// allowed to keep, and every later start would parse all of it. + #[test] + fn test_provider_cache_persist_respects_the_configured_cap() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + + // A sibling with a bigger budget fills the file first. + let sibling = CacheLayer::new(64).cache(); + for i in 0..20u8 { + sibling.put(B256::repeat_byte(i), format!(r#"{{"result":"{i}"}}"#)).expect("put"); + } + RpcCacheStore::new(sibling, path.clone()).persist().expect("persist sibling"); + + // Ours is capped at 4 and holds keys the sibling never saw. + let ours = CacheLayer::new(4).cache(); + let mine: Vec = (100..104u8).map(B256::repeat_byte).collect(); + for key in &mine { + ours.put(*key, r#"{"result":"mine"}"#.to_string()).expect("put"); + } + RpcCacheStore::new(ours, path.clone()).persist().expect("persist ours"); + + let entries = crate::cache::read_provider_cache(&path).expect("read merged cache"); + assert!( + entries.len() <= 4, + "the merged file must respect this run's cap, got {} entries", + entries.len() + ); + for key in &mine { + assert!(entries.iter().any(|e| e.key == *key), "this run's entries survive: {key}"); + } + } + + /// persists — on-disk file must contain the union (B's entries survive). + #[test] + fn test_provider_cache_persist_merges_interleaved_disk_entries() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + + let key_a = B256::repeat_byte(0xaa); + let key_b = B256::repeat_byte(0xbb); + let val_a = r#"{"result":"a"}"#.to_string(); + let val_b = r#"{"result":"b"}"#.to_string(); + + // Process B persists first. + let cache_b = CacheLayer::new(64).cache(); + cache_b.put(key_b, val_b.clone()).expect("put b"); + RpcCacheStore::new(cache_b, path.clone()).persist().expect("persist b"); + + // Process A never loaded B's write; only has key_a in memory. + let cache_a = CacheLayer::new(64).cache(); + cache_a.put(key_a, val_a.clone()).expect("put a"); + RpcCacheStore::new(cache_a, path.clone()).persist().expect("persist a"); + + let loaded = CacheLayer::new(64).cache(); + loaded.load_cache(path).expect("load"); + assert_eq!(loaded.get(&key_a).as_deref(), Some(val_a.as_str())); + assert_eq!(loaded.get(&key_b).as_deref(), Some(val_b.as_str())); + } + + /// On collision, the process that persists last wins for that key. + #[test] + fn test_provider_cache_persist_ours_wins_on_collision() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + let key = B256::repeat_byte(0x01); + + let cache_b = CacheLayer::new(64).cache(); + cache_b.put(key, "from-b".into()).expect("put"); + RpcCacheStore::new(cache_b, path.clone()).persist().expect("persist b"); + + let cache_a = CacheLayer::new(64).cache(); + cache_a.put(key, "from-a".into()).expect("put"); + RpcCacheStore::new(cache_a, path.clone()).persist().expect("persist a"); + + let loaded = CacheLayer::new(64).cache(); + loaded.load_cache(path).expect("load"); + assert_eq!(loaded.get(&key).as_deref(), Some("from-a")); + } + + /// Lock sidecar `.lock` is created on persist and left in place. + #[test] + fn test_provider_cache_persist_creates_lock_sidecar_left_in_place() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-9.json"); + let lock = lock_sidecar_path(&path); + assert!(!lock.exists()); + + let cache = CacheLayer::new(16).cache(); + cache.put(B256::repeat_byte(1), "v".into()).expect("put"); + RpcCacheStore::new(cache, path.clone()).persist().expect("persist"); + + assert!(path.exists(), "cache file written"); + assert!(lock.exists(), "lock sidecar left in place"); + // Sidecar is an empty (or near-empty) lock file, not the cache payload. + let lock_meta = fs::metadata(&lock).expect("lock meta"); + assert!(lock_meta.len() == 0 || lock_meta.is_file()); + } + + /// Envelope persist merges on-disk entries the same way, with `chain_id` check. + #[test] + fn test_envelope_persist_merges_interleaved_disk_entries() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let key_a = keccak256("a"); + let key_b = keccak256("b"); + + let cache_b = TransportCache::new(); + cache_b + .merge(&serde_json::json!([{ + "key": key_b, + "value": r#"{"result":"b"}"#, + }])) + .expect("seed b"); + CacheFileEnvelope::new(&cache_b, 99, None).save(&path, None).expect("save b"); + + let cache_a = TransportCache::new(); + cache_a + .merge(&serde_json::json!([{ + "key": key_a, + "value": r#"{"result":"a"}"#, + }])) + .expect("seed a"); + CacheFileEnvelope::new(&cache_a, 99, None).save(&path, None).expect("save a"); + + let env = CacheFileEnvelope::load(&path).expect("load"); + let loaded = TransportCache::from_value(&env.cache).expect("from_value"); + assert_eq!(loaded.len(), 2); + assert_eq!(env.chain_id, 99); + } + + /// Envelope persist hard-errors on `chain_id` mismatch with on-disk file. + #[test] + fn test_envelope_persist_rejects_chain_id_mismatch() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let cache_b = TransportCache::new(); + CacheFileEnvelope::new(&cache_b, 1, None).save(&path, None).expect("save b"); + + let cache_a = TransportCache::new(); + let err = + CacheFileEnvelope::new(&cache_a, 2, None).save(&path, None).expect_err("mismatch"); + assert!(err.to_string().contains("chain_id")); + } + + /// Corrupt on-disk envelope under a path whose name contains `chain_id` is + /// degradable (warn + replace), not a hard error — classification is typed, + /// not substring-based on the formatted message / path. + #[test] + fn test_envelope_persist_degrades_on_corrupt_disk_path_containing_chain_id() { + let dir = tempfile::tempdir().expect("tempdir"); + // Path deliberately contains the substrings the old classifier matched. + let path = dir.path().join("chain_id_version_capture.json"); + fs::write(&path, "not-json{{{").expect("corrupt"); + + let cache = TransportCache::new(); + cache + .merge(&serde_json::json!([{ + "key": keccak256("eth_blockNumber"), + "value": r#"{"id":0,"jsonrpc":"2.0","result":"0x1"}"#, + }])) + .expect("seed"); + CacheFileEnvelope::new(&cache, 7, None) + .save(&path, None) + .expect("corrupt disk with chain_id in path must degrade, not hard-fail"); + + let env = CacheFileEnvelope::load(&path).expect("ours written"); + assert_eq!(env.chain_id, 7); + assert_eq!(TransportCache::from_value(&env.cache).expect("from_value").len(), 1); + } + + /// Genuine `chain_id` mismatch remains a hard error (typed path via merge). + #[test] + fn test_envelope_persist_hard_errors_on_genuine_chain_id_mismatch() { + let dir = tempfile::tempdir().expect("tempdir"); + // Same path naming trap as the corrupt-file test: must not flip classification. + let path = dir.path().join("chain_id_version_capture.json"); + + let cache_b = TransportCache::new(); + CacheFileEnvelope::new(&cache_b, 1, None).save(&path, None).expect("save b"); + + let cache_a = TransportCache::new(); + let err = CacheFileEnvelope::new(&cache_a, 2, None) + .save(&path, None) + .expect_err("chain_id mismatch must hard-fail"); + let msg = err.to_string(); + assert!(msg.contains("chain_id"), "msg={msg}"); + } + + /// Sequential capture refresh: loaded A, ours B, disk still A → B is written. + #[test] + fn test_envelope_persist_intentional_external_env_refresh() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let loaded = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10)] }; + let cache_a = TransportCache::new(); + CacheFileEnvelope::new(&cache_a, 7, Some(&loaded)).save(&path, None).expect("seed A"); + + let ours = ExternalEnvSnapshot { bucket_capacities: vec![(1, 99)] }; + let cache_b = TransportCache::new(); + CacheFileEnvelope::new(&cache_b, 7, Some(&ours)) + .save(&path, Some(&loaded)) + .expect("intentional A→B refresh"); + + let env = CacheFileEnvelope::load(&path).expect("load"); + let written = env.external_env.expect("external_env written"); + assert_eq!(written.bucket_capacities, vec![(1, 99)]); + } + + /// Store constructed with baseline A; disk still A; ours B → B wins + /// (intentional refresh through `set_external_env` + `persist`). + #[test] + fn test_store_persist_intentional_external_env_refresh() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let a = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&a)) + .save(&path, None) + .expect("seed A"); + + // Baseline A is passed in (same object the caller loaded); no re-read. + let mut store = + RpcCacheStore::new_envelope(TransportCache::new(), path.clone(), 7, Some(a)); + store.set_external_env(ExternalEnvSnapshot { bucket_capacities: vec![(1, 99)] }); + store.persist().expect("store-level intentional A→B refresh must succeed"); + + let env = CacheFileEnvelope::load(&path).expect("load"); + assert_eq!( + env.external_env.expect("external_env written").bucket_capacities, + vec![(1, 99)] + ); + } + + /// Store constructed with baseline A; on-disk mutated to C before persist; + /// ours B derived from A → hard conflict naming loaded/ours/on-disk. + /// + /// Regression for the double-read defect: the store must use the caller's + /// loaded baseline, not re-read the file at construction (which would + /// observe C and treat ours-from-A as an intentional refresh of C). + #[test] + fn test_store_persist_rejects_concurrent_external_env_conflict() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let a = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&a)) + .save(&path, None) + .expect("seed A"); + + // Baseline A from the first load — not re-read from disk at construction. + let mut store = + RpcCacheStore::new_envelope(TransportCache::new(), path.clone(), 7, Some(a.clone())); + + // Concurrent writer lands C (≠A, ≠B) after our load, before our persist. + let c = ExternalEnvSnapshot { bucket_capacities: vec![(1, 42)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&c)) + .save(&path, Some(&a)) + .expect("concurrent C"); + + store.set_external_env(ExternalEnvSnapshot { bucket_capacities: vec![(1, 99)] }); + let err = store.persist().expect_err("true concurrent conflict via store"); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!( + msg.contains("loaded") && msg.contains("ours") && msg.contains("on-disk"), + "msg={msg}" + ); + assert!(msg.contains("10") && msg.contains("99") && msg.contains("42"), "msg={msg}"); + } + + /// If the store re-read the file at construction, a concurrent C would be + /// mistaken for the baseline and an A-derived B would silently overwrite C. + /// Passing baseline A while disk is already C must still conflict. + #[test] + fn test_store_persist_uses_passed_baseline_not_disk_at_construction() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + // Disk already holds C when the store is constructed (simulates a + // writer that landed between the caller's first load and store build). + let a = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10)] }; + let c = ExternalEnvSnapshot { bucket_capacities: vec![(1, 42)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&c)) + .save(&path, None) + .expect("disk is C"); + + let mut store = RpcCacheStore::new_envelope(TransportCache::new(), path, 7, Some(a)); + store.set_external_env(ExternalEnvSnapshot { bucket_capacities: vec![(1, 99)] }); + let err = store + .persist() + .expect_err("passed baseline A must not be replaced by on-disk C at construction"); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!( + msg.contains("loaded") && msg.contains("ours") && msg.contains("on-disk"), + "msg={msg}" + ); + assert!(msg.contains("10") && msg.contains("99") && msg.contains("42"), "msg={msg}"); + } + + /// Concurrent conflict through save: loaded A, ours B, disk C → hard error. + #[test] + fn test_envelope_persist_rejects_concurrent_external_env_conflict() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let c = ExternalEnvSnapshot { bucket_capacities: vec![(1, 42)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&c)) + .save(&path, None) + .expect("seed C"); + + let loaded = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10)] }; + let ours = ExternalEnvSnapshot { bucket_capacities: vec![(1, 99)] }; + let err = CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&ours)) + .save(&path, Some(&loaded)) + .expect_err("true concurrent conflict"); + let msg = err.to_string(); + assert!(msg.contains("external_env"), "msg={msg}"); + assert!( + msg.contains("loaded") && msg.contains("ours") && msg.contains("on-disk"), + "msg={msg}" + ); + assert!(msg.contains("10") && msg.contains("99") && msg.contains("42"), "msg={msg}"); + } + + /// Same effective capacities in different order do not conflict at save. + #[test] + fn test_envelope_persist_order_insensitive_external_env() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + let a = ExternalEnvSnapshot { bucket_capacities: vec![(1, 10), (2, 20)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&a)) + .save(&path, None) + .expect("seed"); + + let b = ExternalEnvSnapshot { bucket_capacities: vec![(2, 20), (1, 10)] }; + // Pretend we loaded something else so equality is the only thing that + // would save us from a false concurrent-conflict report. + let foreign = ExternalEnvSnapshot { bucket_capacities: vec![(9, 9)] }; + CacheFileEnvelope::new(&TransportCache::new(), 7, Some(&b)) + .save(&path, Some(&foreign)) + .expect("order-only difference must not conflict"); + + let env = CacheFileEnvelope::load(&path).expect("load"); + assert_eq!(env.external_env.expect("present").bucket_capacities, vec![(1, 10), (2, 20)]); + } + + /// Typed re-read: corrupt content is Degradable even when path mentions `chain_id`. + #[test] + fn test_reread_envelope_classifies_corrupt_vs_identity() { + let dir = tempfile::tempdir().expect("tempdir"); + + let corrupt = dir.path().join("chain_id_and_version.json"); + fs::write(&corrupt, "{not valid").expect("write"); + match reread_envelope_for_merge(&corrupt) { + EnvelopeReread::Degradable(msg) => { + assert!(msg.contains("parse") || msg.contains("Failed"), "{msg}"); + } + other => panic!("corrupt must be Degradable, got {other:?}"), + } + + let bad_version = dir.path().join("env.json"); + fs::write(&bad_version, r#"{"version":99,"chain_id":1,"cache":[]}"#).unwrap(); + match reread_envelope_for_merge(&bad_version) { + EnvelopeReread::Hard(err) => { + assert!(err.to_string().contains("Unsupported") || err.to_string().contains("99")); + } + other => panic!("unsupported version must be Hard, got {other:?}"), + } + + let ok_path = dir.path().join("ok.json"); + fs::write(&ok_path, r#"{"version":1,"chain_id":5,"cache":[]}"#).unwrap(); + match reread_envelope_for_merge(&ok_path) { + EnvelopeReread::Ok(doc) => assert_eq!(doc.chain_id, 5), + other => panic!("valid envelope must be Ok, got {other:?}"), + } + } + + /// Provider persist fails closed when the lock cannot be acquired: nothing + /// is written, and the file a sibling process left behind is intact. + /// + /// The store swallows the failure (the provider cache is best-effort), so + /// the observable contract is the untouched file, not the return value. + #[test] + fn test_provider_cache_persist_skips_when_the_lock_is_unavailable() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + + // A sibling's file is already on disk. + let sibling = CacheLayer::new(16).cache(); + sibling.put(B256::repeat_byte(0xbb), "from-sibling".into()).expect("put"); + RpcCacheStore::new(sibling, path.clone()).persist().expect("persist sibling"); + let before = fs::read_to_string(&path).expect("read sibling file"); + + // A directory in the sidecar's place makes the lock un-acquirable. + fs::remove_file(lock_sidecar_path(&path)).expect("remove sidecar"); + fs::create_dir(lock_sidecar_path(&path)).expect("occupy sidecar path"); + + let ours = CacheLayer::new(16).cache(); + ours.put(B256::repeat_byte(0xaa), "ours".into()).expect("put"); + RpcCacheStore::new(ours, path.clone()) + .persist() + .expect("provider persist stays best-effort"); + + assert_eq!(fs::read_to_string(&path).unwrap(), before, "no unlocked write happened"); + } + + /// The skipped persist reports why, naming the lock and stating that the + /// entries were not saved. + #[test] + fn test_save_cache_atomic_reports_the_lock_failure() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + fs::create_dir(lock_sidecar_path(&path)).expect("occupy sidecar path"); + + let cache = CacheLayer::new(16).cache(); + cache.put(B256::repeat_byte(0xaa), "ours".into()).expect("put"); + let err = save_cache_atomic(&cache, &path).expect_err("lock failure must not write"); + let msg = err.to_string(); + assert!(msg.contains("rpc-cache-1.json.lock"), "msg={msg}"); + assert!(msg.contains("were not saved"), "msg={msg}"); + assert!(!path.exists(), "nothing was written"); + } + + /// Persisting a provider cache onto a capture envelope must leave the + /// envelope intact: a shared-dir misconfiguration must not destroy a + /// foreign file the provider shape cannot fold. + #[test] + fn test_provider_cache_persist_skips_foreign_envelope_target() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + + let envelope = serde_json::json!({ + "version": 1, + "chain_id": 7, + "cache": [], + "external_env": null, + }); + let before = serde_json::to_string_pretty(&envelope).unwrap(); + fs::write(&path, &before).expect("seed envelope"); + + let cache = CacheLayer::new(16).cache(); + cache.put(B256::repeat_byte(0xaa), "ours".into()).expect("put"); + let wrote = save_cache_atomic(&cache, &path).expect("skip is Ok(false), not Err"); + assert!(!wrote, "foreign shape must not be overwritten"); + assert_eq!(fs::read_to_string(&path).unwrap(), before, "envelope left intact"); + + // Store path is best-effort: same skip, same intact file, no hard error. + let store_cache = CacheLayer::new(16).cache(); + store_cache.put(B256::repeat_byte(0xbb), "store".into()).expect("put"); + RpcCacheStore::new(store_cache, path.clone()) + .persist() + .expect("provider persist stays best-effort on foreign skip"); + assert_eq!(fs::read_to_string(&path).unwrap(), before, "store path also leaves envelope"); + } + + /// An unrecognized structured JSON shape is also foreign: skip, do not replace. + #[test] + fn test_provider_cache_persist_skips_unrecognized_foreign_shape() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + let before = r#"{"not":"a-provider-cache","nor":"an-envelope"}"#; + fs::write(&path, before).expect("seed foreign"); + + let cache = CacheLayer::new(16).cache(); + cache.put(B256::repeat_byte(0xaa), "ours".into()).expect("put"); + let wrote = save_cache_atomic(&cache, &path).expect("skip is Ok"); + assert!(!wrote); + assert_eq!(fs::read_to_string(&path).unwrap(), before); + } + + /// Envelope persist hard-errors when the lock cannot be acquired: the + /// capture is the primary output, so a silently unlocked write is worse + /// than a failed run. + #[test] + fn test_envelope_persist_errors_when_the_lock_is_unavailable() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + + CacheFileEnvelope::new(&TransportCache::new(), 7, None).save(&path, None).expect("seed"); + let before = fs::read_to_string(&path).expect("read seeded envelope"); + + fs::remove_file(lock_sidecar_path(&path)).expect("remove sidecar"); + fs::create_dir(lock_sidecar_path(&path)).expect("occupy sidecar path"); + + let cache = TransportCache::new(); + cache + .merge(&serde_json::json!([{ + "key": keccak256("a"), + "value": r#"{"result":"a"}"#, + }])) + .expect("seed ours"); + let err = CacheFileEnvelope::new(&cache, 7, None) + .save(&path, None) + .expect_err("lock failure must abort the capture persist"); + let msg = err.to_string(); + assert!(msg.contains("capture.json.lock"), "msg={msg}"); + assert!(msg.contains("unlocked"), "msg={msg}"); + assert_eq!(fs::read_to_string(&path).unwrap(), before, "no unlocked write happened"); + } + + /// The store surfaces the envelope lock failure to its caller. + #[test] + fn test_store_envelope_persist_errors_when_the_lock_is_unavailable() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("capture.json"); + fs::create_dir(lock_sidecar_path(&path)).expect("occupy sidecar path"); + + let store = RpcCacheStore::new_envelope(TransportCache::new(), path.clone(), 7, None); + let err = store.persist().expect_err("capture persist must fail closed"); + assert!(err.to_string().contains("lock"), "msg={err}"); + assert!(!path.exists(), "nothing was written"); + } + + /// Corrupt on-disk provider cache during re-read does not abort; ours are written. + #[test] + fn test_provider_cache_persist_degrades_on_corrupt_disk() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("rpc-cache-1.json"); + fs::write(&path, "not-json{{{").expect("corrupt"); + + let key = B256::repeat_byte(0xcc); + let cache = CacheLayer::new(16).cache(); + cache.put(key, "ok".into()).expect("put"); + let wrote = save_cache_atomic(&cache, &path).expect("corrupt degrades to write"); + assert!(wrote, "corrupt target is replaced with ours"); + + let loaded = CacheLayer::new(16).cache(); + loaded.load_cache(path).expect("load"); + assert_eq!(loaded.get(&key).as_deref(), Some("ok")); + } } diff --git a/bin/mega-evme/src/common/provider/mod.rs b/bin/mega-evme/src/common/provider/mod.rs index 94369a6f..70846621 100644 --- a/bin/mega-evme/src/common/provider/mod.rs +++ b/bin/mega-evme/src/common/provider/mod.rs @@ -18,6 +18,7 @@ mod transport; use std::{ fs, path::{Path, PathBuf}, + time::Duration, }; use alloy_provider::{ @@ -38,6 +39,7 @@ use self::{ transport::{CachingTransport, ReplayTransport, TransportCache}, }; use super::{EvmeError, Result}; +use crate::cache::{acquire_exclusive_lock, lock_sidecar_path}; /// OP-stack provider type used throughout mega-evme. pub type OpProvider = DynProvider; @@ -45,11 +47,11 @@ pub type OpProvider = DynProvider; /// Return value of the `RpcArgs::build_*_provider` methods. #[derive(Debug)] pub struct BuildProviderOutput { - /// Configured OP-stack provider. Already wrapped with the retry layer and (unless - /// the cache is disabled) the in-memory cache layer. + /// Configured OP-stack provider. Already wrapped with the retry layer and the + /// in-memory cache layer. pub provider: OpProvider, /// Clean-exit cache persistence handle. Call [`RpcCacheStore::persist`] on the - /// success path; no-op when the cache is disabled. + /// success path; no-op when on-disk persistence is disabled (`--rpc.no-cache-file`). pub cache_store: RpcCacheStore, /// Chain id resolved during provider construction. Always populated — /// comes from `eth_chainId` (standard/capture) or the envelope (replay). @@ -75,12 +77,12 @@ pub struct RpcArgs { /// If the file already exists, its entries are loaded and merged; /// missing entries are fetched via the RPC endpoint and persisted on clean exit. /// Cannot be used with --rpc.replay-file, --rpc.cache-dir, --rpc.clear-cache, - /// --rpc.no-cache-file, or --rpc.cache-size. + /// --rpc.no-cache-file, or --rpc.cache-max-entries. #[arg( long = "rpc.capture-file", value_parser = parse_non_empty_path, requires = "rpc_url", - conflicts_with_all = ["replay_file", "cache_dir", "clear_cache", "no_cache_file", "cache_size"], + conflicts_with_all = ["replay_file", "cache_dir", "clear_cache", "no_cache_file", "cache_max_entries"], )] pub capture_file: Option, @@ -88,18 +90,20 @@ pub struct RpcArgs { /// Cannot be used with `--rpc`. /// Any RPC miss is a hard error; the file is never written. /// Cannot be used with --rpc.capture-file, --rpc.cache-dir, --rpc.clear-cache, - /// --rpc.no-cache-file, or --rpc.cache-size. + /// --rpc.no-cache-file, or --rpc.cache-max-entries. #[arg( long = "rpc.replay-file", value_parser = parse_non_empty_path, - conflicts_with_all = ["rpc_url", "capture_file", "cache_dir", "clear_cache", "no_cache_file", "cache_size"], + conflicts_with_all = ["rpc_url", "capture_file", "cache_dir", "clear_cache", "no_cache_file", "cache_max_entries"], )] pub replay_file: Option, - /// Maximum number of items to keep in the in-memory RPC LRU cache. - /// Set to 0 to disable the cache layer entirely. - #[arg(id = "cache_size", long = "rpc.cache-size", default_value_t = 10_000)] - pub cache_size: u32, + /// Maximum number of items in the in-memory RPC LRU cache (and therefore what + /// gets persisted to the cache file). `0` = effectively unlimited (caps at + /// 1,048,576 entries; the cache index is preallocated proportional to the + /// cap). Default is `0`. + #[arg(id = "cache_max_entries", long = "rpc.cache-max-entries", default_value_t = 0)] + pub cache_max_entries: u32, /// Directory for per-chain RPC cache files. /// @@ -108,25 +112,33 @@ pub struct RpcArgs { /// /// Defaults to the platform cache directory (`$XDG_CACHE_HOME/mega-evme/rpc` on /// Linux, `~/Library/Caches/mega-evme/rpc` on macOS). Pass `--rpc.no-cache-file` - /// to disable on-disk persistence entirely. + /// to disable on-disk persistence entirely. Batch replay (`--tx-file` / `--block`) + /// uses the on-disk cache only when this flag or `--rpc.clear-cache` is passed + /// explicitly. #[arg(long = "rpc.cache-dir", value_parser = parse_non_empty_path)] pub cache_dir: Option, - /// Disable on-disk cache persistence. The in-memory LRU cache still applies — use - /// `--rpc.cache-size 0` to disable that too. + /// Disable on-disk cache persistence. The in-memory LRU cache still applies. + /// Takes precedence over `--rpc.clear-cache`: with no cache file in play there is + /// nothing to delete, load, or persist. This is already the default for batch replay + /// (`--tx-file` / `--block`) unless `--rpc.cache-dir` or `--rpc.clear-cache` is passed. #[arg(long = "rpc.no-cache-file")] pub no_cache_file: bool, /// Delete the current chain's cache file before loading it. Recovery path for a /// polluted or corrupt cache file. If the unlink itself fails (e.g. insufficient /// permissions), `mega-evme` aborts rather than silently reloading the stale file. + /// Passing this flag engages the on-disk cache, including in batch replay + /// (`--tx-file` / `--block`), where it is otherwise off by default. Has no effect + /// alongside `--rpc.no-cache-file`. #[arg(long = "rpc.clear-cache")] pub clear_cache: bool, /// Maximum number of times the transport layer will retry a failing RPC request. /// Retries trigger on HTTP 429 / 503, JSON-RPC rate-limit error responses, and /// transport failures surfaced as `TransportErrorKind::Custom` (connection refused, - /// DNS failure, TLS handshake, etc.). Set to 0 to disable retries entirely. + /// DNS failure, TLS handshake, request timeout, etc.). Set to 0 to disable retries + /// entirely. #[arg(long = "rpc.max-retries", default_value_t = 5)] pub max_retries: u32, @@ -135,9 +147,18 @@ pub struct RpcArgs { #[arg(long = "rpc.backoff-ms", default_value_t = 1_000)] pub backoff_ms: u64, - /// Compute units per second budget passed to the retry layer's rate-limit accounting. - #[arg(long = "rpc.rate-limit", default_value_t = 660)] + /// Compute-unit budget (CU/s) for the retry layer's rate-limit accounting. + /// This is NOT requests per second: each RPC method costs multiple compute units. + /// A single-digit value will heavily self-throttle. Default (660) matches typical + /// public-endpoint budgets. + #[arg(long = "rpc.cu-per-sec", visible_alias = "rpc.rate-limit", default_value_t = 660)] pub compute_units_per_sec: u64, + + /// Total per-HTTP-request timeout in seconds (connect + response). + /// `0` disables the timeout (previous behavior: a hung endpoint can block forever). + /// A non-zero timeout surfaces a hung endpoint as a retryable transport error. + #[arg(long = "rpc.request-timeout", default_value_t = 30)] + pub request_timeout: u64, } impl RpcArgs { @@ -152,45 +173,67 @@ impl RpcArgs { })?; let url: reqwest::Url = rpc_url_str.parse().map_err(|e| { - EvmeError::RpcError(format!("Invalid RPC URL '{}': {}", rpc_url_str, e)) + EvmeError::InvalidInput(format!("Invalid RPC URL '{}': {}", rpc_url_str, e)) })?; + // Once per provider build (not per client: resolve_chain_id also builds a client). + self.maybe_warn_low_cu_per_sec(); + // 1. Resolve chain id (always needed by downstream consumers). let chain_id = self.resolve_chain_id(url.clone()).await?; - // 2. Fast path: cache fully disabled. - if self.cache_size == 0 { - let provider = build_bare_op_provider(self.build_retry_client(url)); - info!( - rpc_url = %rpc_url_str, - max_retries = self.max_retries, - backoff_ms = self.backoff_ms, - "Built RPC provider (cache disabled)", - ); - return Ok(BuildProviderOutput { - provider, - cache_store: RpcCacheStore::noop(), - chain_id, - external_env: None, - }); - } - - // 3. Resolve on-disk cache path (None when disk persistence is disabled). + // 2. Resolve on-disk cache path (None when disk persistence is disabled). let cache_path = if self.no_cache_file { None } else { Some(resolve_cache_path(self.cache_dir.as_deref(), chain_id)?) }; - // 4. Build the cache layer and (optionally) the disk store. - let cache_layer = CacheLayer::new(self.cache_size); + // 3. Build the cache layer and (optionally) the disk store. + // Cache layer is always installed; 0 max entries maps to + // EFFECTIVELY_UNLIMITED_CACHE_ENTRIES. + let max_items = cache_max_entries_capacity(self.cache_max_entries); + let cache_layer = CacheLayer::new(max_items); let cache = cache_layer.cache(); let cache_store = match cache_path { Some(path) => { + // Same sidecar lock as persist / `cache merge`. Held for the + // whole clear critical section: acquire → unlink → exists-check + // → load (or the decision that nothing is on disk to load). + // Releasing after unlink but before load leaves a window where a + // concurrent locked writer can recreate the file with the + // entries the user asked to remove, and this invocation then + // loads them. Fail closed if the lock cannot be acquired — the + // user asked for a deletion that is not safe to do unlocked. + // + // Lock ordering with same-process persist: this guard lives only + // for provider build and is dropped before `BuildProviderOutput` + // returns; clean-exit `RpcCacheStore::persist` acquires later. + // The two critical sections never overlap in one process, so + // clear cannot deadlock against its own later persist. + // Local filesystem failures below are the operator's + // environment refusing the requested operation — retrying or + // switching the endpoint cannot fix them, so they classify as + // execution-class input failures (exit 1), not as the endpoint + // failing to answer (exit 3). Same class as `cache merge`'s + // lock failure. + let clear_lock = if self.clear_cache { + Some(acquire_exclusive_lock(&path).map_err(|e| { + EvmeError::InvalidInput(format!( + "Failed to acquire the cache lock {} for clear-cache of {}: {e}. \ + Refusing to clear without it: a concurrent writer could race \ + the unlink and silently recreate or rely on the file.", + lock_sidecar_path(&path).display(), + path.display(), + )) + })?) + } else { + None + }; if self.clear_cache { if let Err(e) = fs::remove_file(&path) { if e.kind() != std::io::ErrorKind::NotFound { - return Err(EvmeError::RpcError(format!( + return Err(EvmeError::InvalidInput(format!( "Failed to clear RPC cache at {}: {e}", path.display(), ))); @@ -209,6 +252,10 @@ impl RpcArgs { } } if path.exists() { + // Oversized-load warning is intentionally omitted: alloy's + // `SharedCache` has no public entry-count API (`len` / loaded-count), + // so a file-with-N-entries-over-cap warning cannot be emitted + // without re-implementing load or file-size heuristics (rejected). if let Err(err) = cache.load_cache(path.clone()) { warn!( path = %path.display(), @@ -217,12 +264,16 @@ impl RpcArgs { ); } } + // Release after unlink + exists + load; later provider construction + // and exit-time persist run without this guard (they never overlap + // it in the same process — see lock-ordering note above). + drop(clear_lock); RpcCacheStore::new(cache, path) } None => RpcCacheStore::noop(), }; - // 5. Build the cached provider. + // 4. Build the cached provider. let client = self.build_retry_client(url); let provider = ProviderBuilder::new() .disable_recommended_fillers() @@ -232,7 +283,8 @@ impl RpcArgs { info!( rpc_url = %rpc_url_str, - cache_size = self.cache_size, + cache_max_entries = self.cache_max_entries, + cache_capacity = max_items, max_retries = self.max_retries, backoff_ms = self.backoff_ms, "Built RPC provider", @@ -301,9 +353,12 @@ impl RpcArgs { let rpc_url_str = self.rpc_url.as_ref().expect("capture mode requires --rpc"); let url: reqwest::Url = rpc_url_str.parse().map_err(|e| { - EvmeError::RpcError(format!("Invalid RPC URL '{}': {}", rpc_url_str, e)) + EvmeError::InvalidInput(format!("Invalid RPC URL '{}': {}", rpc_url_str, e)) })?; + // Once per provider build (capture builds a single client; keep the same entry point). + self.maybe_warn_low_cu_per_sec(); + // Load existing envelope if the file exists. let existing_envelope = if path.exists() { let env = CacheFileEnvelope::load(path)?; @@ -324,7 +379,7 @@ impl RpcArgs { // envelope first, a stale eth_chainId entry would short-circuit the // cross-chain validation. let transport_cache = TransportCache::new(); - let http = alloy_transport_http::Http::new(url.clone()); + let http = self.build_http_transport(url.clone()); let caching = CachingTransport::new(http, transport_cache.clone()); let client = self.build_client(caching, &url); let provider = ProviderBuilder::new() @@ -361,24 +416,68 @@ impl RpcArgs { "Built RPC provider (capture to cache file)", ); + // Same snapshot whose entries were merged above — carry it into the + // store as the OCC load-time baseline. Do not re-read the file: a + // concurrent writer between this load and store construction would + // make the later write treat C as baseline and silently overwrite it. let prev_external_env = existing_envelope.and_then(|e| e.external_env); Ok(BuildProviderOutput { provider: DynProvider::new(provider), - cache_store: RpcCacheStore::new_envelope(transport_cache, path.clone(), chain_id), + cache_store: RpcCacheStore::new_envelope( + transport_cache, + path.clone(), + chain_id, + prev_external_env.clone(), + ), chain_id, external_env: prev_external_env, }) } + /// Build the HTTP transport, applying [`Self::request_timeout`] when non-zero. + /// + /// All networked `Http` construction (standard provider, capture provider, and + /// the throwaway chain-id probe) must go through this helper so the timeout is + /// applied uniformly. Offline replay (`--rpc.replay-file`) never constructs + /// an HTTP transport. + /// + /// When `request_timeout == 0`, uses the default reqwest client (no total + /// request timeout). When non-zero, builds a client with both `.timeout` and + /// `.connect_timeout` set to the same duration so a hung endpoint surfaces as + /// a `TransportErrorKind::Custom` error (retryable under the policy below) + /// instead of hanging the process forever. + fn build_http_transport( + &self, + url: reqwest::Url, + ) -> alloy_transport_http::Http { + if self.request_timeout == 0 { + return alloy_transport_http::Http::new(url); + } + let timeout = Duration::from_secs(self.request_timeout); + // `Client::builder` with only timeout settings cannot fail under normal + // conditions (failure requires a broken TLS backend). + let client = reqwest::Client::builder() + .timeout(timeout) + .connect_timeout(timeout) + .build() + .expect("reqwest Client with timeout settings must build"); + alloy_transport_http::Http::with_client(client, url) + } + /// Build an `RpcClient` over HTTP, wired with the configured retry layer. fn build_retry_client(&self, url: reqwest::Url) -> RpcClient { - self.build_client(alloy_transport_http::Http::new(url.clone()), &url) + self.build_client(self.build_http_transport(url.clone()), &url) } /// Build an `RpcClient` over an arbitrary transport, wired with the configured /// retry layer (or bare when `max_retries == 0`). `url` is used only to detect /// whether the endpoint is local. + /// + /// Retry coverage: `RateLimitRetryPolicy` handles HTTP 429/503 and JSON-RPC + /// rate-limit bodies. The `TransportErrorKind::Custom` predicate additionally + /// retries transport failures, including reqwest request timeouts (mapped by + /// `alloy-transport-http` via `TransportErrorKind::custom`). fn build_client( &self, transport: T, @@ -387,6 +486,8 @@ impl RpcArgs { let is_local = url.host_str().is_some_and(|h| h == "localhost" || h == "127.0.0.1" || h == "::1"); if self.max_retries > 0 { + // Reqwest timeouts, connection refused, DNS failure, TLS handshake, + // etc. all arrive as `TransportErrorKind::Custom` and are retryable. let policy = RateLimitRetryPolicy::default().or(|err: &TransportError| { matches!(err, RpcError::Transport(TransportErrorKind::Custom(_))) }); @@ -402,6 +503,17 @@ impl RpcArgs { } } + /// Emit the low CU/s warning at most once per networked provider build. + /// + /// Not placed in [`Self::build_client`]: the standard path builds a throwaway + /// client for chain-id resolution and then the real client, so a warning there + /// would fire twice. + fn maybe_warn_low_cu_per_sec(&self) { + if let Some(msg) = cu_per_sec_warning(self.max_retries, self.compute_units_per_sec) { + warn!("{msg}"); + } + } + /// Resolve the chain ID by issuing `eth_chainId` against a throwaway /// cache-less provider using the configured retry policy. async fn resolve_chain_id(&self, url: reqwest::Url) -> Result { @@ -413,6 +525,23 @@ impl RpcArgs { } } +/// Threshold below which a configured CU/s budget is considered dangerously low. +/// Values under this with retries enabled produce a one-shot warning at provider build. +const CU_PER_SEC_WARN_THRESHOLD: u64 = 100; + +/// Return a warning message when the retry layer is enabled and the CU/s budget is +/// below [`CU_PER_SEC_WARN_THRESHOLD`]. Used so the trigger rule is unit-testable +/// without capturing log output. +fn cu_per_sec_warning(max_retries: u32, compute_units_per_sec: u64) -> Option { + (max_retries > 0 && compute_units_per_sec < CU_PER_SEC_WARN_THRESHOLD).then(|| { + format!( + "--rpc.cu-per-sec is set to {compute_units_per_sec}, which is a compute-unit \ + budget (CU/s) for the retry layer's rate-limit accounting, NOT requests per \ + second; such a low budget will heavily self-throttle RPC traffic" + ) + }) +} + /// `clap` value parser that rejects empty and whitespace-only path arguments /// at parse time. /// @@ -429,12 +558,35 @@ fn parse_non_empty_path(s: &str) -> std::result::Result { } } +/// Cap used when `--rpc.cache-max-entries 0` ("effectively unlimited") is requested. +/// +/// Alloy's `SharedCache` preallocates its LRU hash table to full capacity +/// (`lru::LruCache::with_hasher` → `HashMap::with_capacity_and_hasher`), so a true +/// `u32::MAX` mapping would preallocate ~2^33 hash buckets and balloon RSS by multi-GB +/// on every default-config online run. 2^20 entries preallocates ~2^21 pointer-sized +/// buckets (tens of MB) while covering ~2,600 blocks' worth of RPC entries per process +/// (a full mainnet block is ~200 entries; the largest real merged corpus to date was +/// 15,294). +const EFFECTIVELY_UNLIMITED_CACHE_ENTRIES: u32 = 1_048_576; + +/// Map `--rpc.cache-max-entries` to the capacity passed to [`CacheLayer::new`]. +/// +/// `0` means effectively unlimited and is approximated by +/// [`EFFECTIVELY_UNLIMITED_CACHE_ENTRIES`] (see that constant's rationale). +/// Nonzero values pass through unchanged. +fn cache_max_entries_capacity(cache_max_entries: u32) -> u32 { + if cache_max_entries == 0 { + EFFECTIVELY_UNLIMITED_CACHE_ENTRIES + } else { + cache_max_entries + } +} + /// Build a cache-less [`OpProvider`] from an already-configured `RpcClient`. /// -/// Used by the cache-disabled fast path and by the throwaway chain-id fetch. -/// The cache-enabled path builds its provider inline because the cache layer -/// has to be inserted into the `ProviderBuilder` chain before the client is -/// attached. +/// Used by the throwaway chain-id fetch. The standard path builds its provider +/// inline because the cache layer has to be inserted into the `ProviderBuilder` +/// chain before the client is attached. fn build_bare_op_provider(client: RpcClient) -> OpProvider { DynProvider::new( ProviderBuilder::new() @@ -495,4 +647,55 @@ mod tests { let expected = expected_root.join("mega-evme").join("rpc").join("rpc-cache-11155420.json"); assert_eq!(path, expected); } + + /// Warn when retries are on and CU/s is below the threshold. + #[test] + fn test_cu_per_sec_warning_fires_below_threshold_with_retries() { + let msg = cu_per_sec_warning(5, 99).expect("should warn at 99 with retries on"); + assert!(msg.contains("99"), "message should include the configured value: {msg}"); + assert!(msg.contains("NOT requests per second") || msg.contains("NOT requests"), "{msg}"); + assert!(msg.contains("self-throttle"), "{msg}"); + } + + /// Silent at the threshold boundary (100) and at the production default (660). + #[test] + fn test_cu_per_sec_warning_silent_at_or_above_threshold() { + assert!(cu_per_sec_warning(5, 100).is_none()); + assert!(cu_per_sec_warning(5, 660).is_none()); + } + + /// Silent when the retry layer is disabled, even with a low CU/s budget. + #[test] + fn test_cu_per_sec_warning_silent_when_retries_disabled() { + assert!(cu_per_sec_warning(0, 1).is_none()); + assert!(cu_per_sec_warning(0, 99).is_none()); + } + + /// `0` maps to the effectively-unlimited cap; nonzero values pass through. + #[test] + fn test_cache_max_entries_capacity_mapping() { + assert_eq!(cache_max_entries_capacity(0), EFFECTIVELY_UNLIMITED_CACHE_ENTRIES); + assert_eq!(cache_max_entries_capacity(0), 1_048_576); + assert_eq!(cache_max_entries_capacity(1), 1); + assert_eq!(cache_max_entries_capacity(256), 256); + assert_eq!(cache_max_entries_capacity(10_000), 10_000); + assert_eq!(cache_max_entries_capacity(u32::MAX), u32::MAX); + } + + /// Construction-reality check: actually allocate the cache at the mapped "unlimited" + /// capacity, insert, and read back. Preallocating `u32::MAX` would hang/OOM here + /// (hashbrown ctrl-byte memset over ~2^33 buckets); this test must complete quickly. + #[test] + fn test_cache_layer_constructs_at_effectively_unlimited_capacity() { + let layer = CacheLayer::new(cache_max_entries_capacity(0)); + assert_eq!(layer.max_items(), EFFECTIVELY_UNLIMITED_CACHE_ENTRIES); + + let cache = layer.cache(); + assert_eq!(cache.max_items(), EFFECTIVELY_UNLIMITED_CACHE_ENTRIES); + + let key = alloy_primitives::B256::repeat_byte(0xab); + let value = r#"{"result":"0x1"}"#.to_string(); + cache.put(key, value.clone()).expect("put into SharedCache"); + assert_eq!(cache.get(&key).as_deref(), Some(value.as_str())); + } } diff --git a/bin/mega-evme/src/common/provider/transport.rs b/bin/mega-evme/src/common/provider/transport.rs index 07739555..810a04b7 100644 --- a/bin/mega-evme/src/common/provider/transport.rs +++ b/bin/mega-evme/src/common/provider/transport.rs @@ -128,6 +128,15 @@ fn transport_cache_key(method: &str, params: Option<&serde_json::value::RawValue keccak256(format!("{method}\x00{params_str}")) } +/// Whether a success response carries a JSON `null` result. +/// +/// A null is never load-bearing for offline replay — every consumer fails its +/// target or run on it — so capture must not bake it into the fixture. The +/// correct offline representation of "no answer" is a cache miss. +fn is_null_success_result(resp: &alloy_json_rpc::Response) -> bool { + resp.payload.as_success().is_some_and(|raw| raw.get().trim() == "null") +} + /// Transport wrapper that records all JSON-RPC responses into a /// [`TransportCache`]. Used in capture mode: cache hits are served locally, /// misses are forwarded to the inner transport and the response is cached. @@ -174,11 +183,16 @@ where return fut; } - // Cache miss: forward to inner, cache successful responses only. - // JSON-RPC error bodies (e.g. transient rate-limit errors that the - // endpoint surfaces via `error` instead of an HTTP status) must not - // be baked into the fixture — otherwise replay would replay the - // error forever. + // Cache miss: forward to inner, cache non-null successful responses + // only. + // + // - JSON-RPC error bodies (e.g. transient rate-limit errors that the endpoint surfaces + // via `error` instead of an HTTP status) must not be baked into the fixture — + // otherwise replay would replay the error forever. + // - `"result": null` is likewise skipped: a transient null (e.g. + // eth_getTransactionByHash for a briefly-invisible tx) would otherwise freeze "not + // found" into the fixture with no in-tool recovery (`--rpc.clear-cache` conflicts + // with capture mode). Offline, a cache miss names the exact request instead. let cache = self.cache.clone(); let method = r.method().to_string(); let fut = self.inner.call(req); @@ -190,6 +204,11 @@ where method = %method, "Skipping cache for JSON-RPC error response", ); + } else if is_null_success_result(resp) { + tracing::warn!( + method = %method, + "Skipping cache for JSON-RPC null result", + ); } else if let Ok(serialized) = serde_json::to_string(resp) { cache.put(key, serialized); } @@ -260,18 +279,77 @@ impl tower::Service for ReplayTransport { #[cfg(test)] mod tests { - use std::path::PathBuf; + use std::{ + path::PathBuf, + sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }, + task::{Context, Poll}, + }; + + use alloy_json_rpc::{Id, Response, ResponsePayload}; + use serde_json::value::to_raw_value; + use tower::Service; use super::*; + /// Inner transport that returns a fixed single response for every call. + #[derive(Clone)] + struct FixedResponseTransport { + response: Arc, + calls: Arc, + } + + impl FixedResponseTransport { + fn new(response: Response) -> Self { + Self { response: Arc::new(response), calls: Arc::new(AtomicUsize::new(0)) } + } + } + + impl Service for FixedResponseTransport { + type Response = ResponsePacket; + type Error = TransportError; + type Future = TransportFut<'static>; + + fn poll_ready( + &mut self, + _cx: &mut Context<'_>, + ) -> Poll> { + Poll::Ready(Ok(())) + } + + fn call(&mut self, _req: RequestPacket) -> Self::Future { + self.calls.fetch_add(1, Ordering::SeqCst); + let response = ResponsePacket::Single((*self.response).clone()); + Box::pin(async move { Ok(response) }) + } + } + + fn success_response(result_json: &str) -> Response { + Response { + id: Id::Number(1), + payload: ResponsePayload::Success( + to_raw_value(&serde_json::from_str::(result_json).unwrap()) + .unwrap(), + ), + } + } + + fn single_request(method: &'static str) -> RequestPacket { + RequestPacket::Single( + alloy_json_rpc::Request::new(method, Id::Number(1), ()) + .serialize() + .expect("serialize request"), + ) + } + /// The replay transport is always ready. /// Single-request cache misses return a descriptive Custom error (safe because /// the retry layer is never installed on the replay path). /// Batch misses return `BackendGone`. #[tokio::test] async fn test_replay_transport_cache_miss() { - use tower::Service; - let mut transport = ReplayTransport::new(PathBuf::from("/tmp/test.cache.json"), TransportCache::new()); @@ -292,4 +370,114 @@ mod tests { assert!(msg.contains("eth_blockNumber"), "error should include method: {msg}"); assert!(msg.contains("test.cache.json"), "error should include fixture path: {msg}"); } + + /// A success with a non-null result is cached and served on the next call. + #[tokio::test] + async fn test_caching_transport_caches_non_null_success() { + let inner = FixedResponseTransport::new(success_response(r#""0x42""#)); + let calls = Arc::clone(&inner.calls); + let cache = TransportCache::new(); + let mut transport = CachingTransport::new(inner, cache.clone()); + + let first = transport.call(single_request("eth_blockNumber")).await.expect("first call"); + assert!(matches!(first, ResponsePacket::Single(_))); + assert_eq!(cache.len(), 1, "non-null success must be cached"); + assert_eq!(calls.load(Ordering::SeqCst), 1); + + // Second call must hit the cache and not touch the inner transport. + let second = transport.call(single_request("eth_blockNumber")).await.expect("cache hit"); + assert!(matches!(second, ResponsePacket::Single(_))); + assert_eq!(calls.load(Ordering::SeqCst), 1, "cache hit must not call inner"); + } + + /// A success with `"result": null` is returned to the caller but never put + /// in the cache — absent entry stays absent. + #[tokio::test] + async fn test_caching_transport_skips_null_result() { + let inner = FixedResponseTransport::new(success_response("null")); + let calls = Arc::clone(&inner.calls); + let cache = TransportCache::new(); + let mut transport = CachingTransport::new(inner, cache.clone()); + + let response = transport.call(single_request("eth_getTransactionByHash")).await; + let response = response.expect("null result is still a transport success"); + match response { + ResponsePacket::Single(resp) => { + assert!(is_null_success_result(&resp), "caller must receive the null unchanged"); + } + other => panic!("expected single response, got {other:?}"), + } + assert_eq!(cache.len(), 0, "null result must not be cached"); + assert_eq!(calls.load(Ordering::SeqCst), 1); + + // A second call is still a miss and re-forwards — never a cached null. + let _ = transport.call(single_request("eth_getTransactionByHash")).await; + assert_eq!(cache.len(), 0, "repeated nulls must still leave the cache empty"); + assert_eq!(calls.load(Ordering::SeqCst), 2, "null must not create a cache hit"); + } + + /// A null response must never reach `put`, so an existing non-null entry + /// for the same key is preserved even if the live endpoint later returns null. + /// + /// The production path usually serves the existing entry as a hit before the + /// network is consulted; this test forces the miss→null path after a prior + /// put by clearing nothing and re-issuing through a transport whose inner + /// now returns null, after first caching a non-null under the same key via + /// a separate `CachingTransport` sharing the cache. + #[tokio::test] + async fn test_caching_transport_null_does_not_overwrite_existing_entry() { + let cache = TransportCache::new(); + + // Seed a non-null entry under eth_blockNumber. + let seed_inner = FixedResponseTransport::new(success_response(r#""0x42""#)); + let mut seeder = CachingTransport::new(seed_inner, cache.clone()); + seeder.call(single_request("eth_blockNumber")).await.expect("seed"); + assert_eq!(cache.len(), 1); + + // A second CachingTransport sharing the same cache: on a hit the null + // never reaches put. Serve the hit and assert the entry is unchanged. + let null_inner = FixedResponseTransport::new(success_response("null")); + let null_calls = Arc::clone(&null_inner.calls); + let mut transport = CachingTransport::new(null_inner, cache.clone()); + let served = transport.call(single_request("eth_blockNumber")).await.expect("hit"); + match served { + ResponsePacket::Single(resp) => { + assert!( + !is_null_success_result(&resp), + "existing non-null entry must be served, not overwritten by a live null", + ); + } + other => panic!("expected single response, got {other:?}"), + } + assert_eq!(null_calls.load(Ordering::SeqCst), 0, "cache hit must not call inner"); + assert_eq!(cache.len(), 1, "null path must never drop the existing entry"); + + // Inspect the cached payload is still the non-null success. + let key = transport_cache_key("eth_blockNumber", None); + let cached = cache.get(&key).expect("entry present"); + let cached_resp: Response = serde_json::from_str(&cached).expect("cached JSON"); + assert!(!is_null_success_result(&cached_resp)); + } + + /// JSON-RPC error responses are still skipped (regression pin). + #[tokio::test] + async fn test_caching_transport_skips_error_response() { + let error = Response { id: Id::Number(1), payload: ResponsePayload::internal_error() }; + let inner = FixedResponseTransport::new(error); + let cache = TransportCache::new(); + let mut transport = CachingTransport::new(inner, cache.clone()); + + let _ = transport.call(single_request("eth_blockNumber")).await; + assert_eq!(cache.len(), 0, "error responses must not be cached"); + } + + #[test] + fn test_is_null_success_result_detects_null_only() { + assert!(is_null_success_result(&success_response("null"))); + assert!(!is_null_success_result(&success_response(r#""0x1""#))); + assert!(!is_null_success_result(&success_response("0"))); + assert!(!is_null_success_result(&success_response("{}"))); + let error = Response { id: Id::Number(1), payload: ResponsePayload::internal_error() }; + assert!(!is_null_success_result(&error)); + } } diff --git a/bin/mega-evme/src/common/state.rs b/bin/mega-evme/src/common/state.rs index bbafccdc..6460e780 100644 --- a/bin/mega-evme/src/common/state.rs +++ b/bin/mega-evme/src/common/state.rs @@ -454,6 +454,25 @@ where Forked(Box>>>), } +/// Normalizes an account fetched over RPC, mapping the all-zero answer to "does not exist". +/// +/// JSON-RPC cannot express account non-existence: `eth_getBalance`, `eth_getTransactionCount`, +/// and `eth_getCode` all answer `0`/`0`/empty for an account that was never created, so the RPC +/// backend materializes it as an *existing* empty account. That flips every consumer of +/// existence, most visibly the EIP-7702 per-authorization refund: a brand-new authority is +/// judged "already in the trie" and the replay refunds 12,500 gas per authorization that the +/// chain did not. +/// +/// Mapping all-zero back to `None` is safe because an existing-but-empty account cannot occur +/// on `MegaETH`: EIP-161 (Spurious Dragon) removes empty accounts on touch and forbids creating +/// them, every chain this tool replays activated it from genesis, and the genesis allocs carry +/// balance or code. The one shape this cannot distinguish — an account with zero +/// balance/nonce/code that still holds storage — also cannot exist post-EIP-161, since storage +/// is only reachable through code and contracts have a non-empty code hash or nonce. +fn normalize_rpc_account(account: Option) -> Option { + account.filter(|info| !info.is_empty()) +} + /// State database that can be backed by either [`EmptyDB`] or [`AlloyDB`] (forked from RPC) #[derive(Debug)] pub struct EvmeState @@ -645,9 +664,9 @@ where Ok(account) } EvmeBackend::Forked(db) => { - let account = db.basic(address).map_err(|e| { + let account = normalize_rpc_account(db.basic(address).map_err(|e| { EvmeError::RpcError(format!("Failed to fetch account {}: {:?}", address, e)) - })?; + })?); trace!(address = %address, account = ?account, "Loaded account basic from forked state"); Ok(account) } @@ -766,9 +785,9 @@ where Ok(account) } EvmeBackend::Forked(db) => { - let account = db.basic_ref(address).map_err(|e| { + let account = normalize_rpc_account(db.basic_ref(address).map_err(|e| { EvmeError::RpcError(format!("Failed to fetch account {}: {:?}", address, e)) - })?; + })?); trace!(address = %address, account = ?account, "Loaded account basic from forked state"); Ok(account) } diff --git a/bin/mega-evme/src/common/tx.rs b/bin/mega-evme/src/common/tx.rs index 596b62ea..4f12e9b0 100644 --- a/bin/mega-evme/src/common/tx.rs +++ b/bin/mega-evme/src/common/tx.rs @@ -4,14 +4,14 @@ use alloy_primitives::{address, Address, Bytes, Signature, B256, U256}; use clap::Args; use mega_evm::{ alloy_consensus::{ - transaction::SignerRecoverable, Sealed, Signed, Transaction as _, TxEip1559, TxEip2930, - TxEip7702, TxLegacy, + transaction::SignerRecoverable, Sealed, Signed, TxEip1559, TxEip2930, TxEip7702, TxLegacy, }, alloy_eips::{ eip2930::{AccessList, AccessListItem}, eip7702::{Authorization, RecoveredAuthority, RecoveredAuthorization, SignedAuthorization}, - Decodable2718, Encodable2718, Typed2718 as _, + Decodable2718, Encodable2718, }, + alloy_evm::FromTxWithEncoded, op_alloy_consensus::{OpTxEnvelope, TxDeposit}, op_revm::transaction::deposit::DepositTransactionParts, revm::{context::tx::TxEnv, primitives::TxKind}, @@ -367,20 +367,18 @@ impl TxArgs { /// Result of decoding a raw EIP-2718 transaction. #[derive(Debug)] pub struct DecodedRawTx { - /// The decoded transaction environment. - pub tx_env: TxEnv, - /// The original raw EIP-2718 encoded bytes. - pub raw_bytes: Bytes, - /// Deposit-specific fields, if this is a deposit transaction. - /// `(source_hash, mint, is_system_transaction)` - pub deposit: Option<(B256, Option, bool)>, + /// The decoded transaction. `enveloped_tx` carries the original raw bytes (used in L1 fee + /// calculation), and the deposit fields are filled for type-126 transactions. + pub tx: MegaTransaction, } impl DecodedRawTx { - /// Decodes raw EIP-2718 encoded transaction bytes into a [`TxEnv`]. + /// Decodes raw EIP-2718 encoded transaction bytes into a [`MegaTransaction`]. /// - /// Recovers the signer from the signature (or uses the `from` field for deposits) - /// and extracts all transaction fields. No CLI overrides are applied. + /// Recovers the signer from the signature (or uses the `from` field for deposits). The + /// per-variant field mapping is derived through [`FromTxWithEncoded`], so new transaction + /// types are picked up from the upstream impl instead of a hand-written mapping here. + /// No CLI overrides are applied. pub fn from_raw(raw_bytes: impl Into) -> Result { let raw_bytes = raw_bytes.into(); let envelope = OpTxEnvelope::decode_2718(&mut &raw_bytes[..]).map_err(|e| { @@ -391,114 +389,73 @@ impl DecodedRawTx { .recover_signer() .map_err(|e| EvmeError::InvalidInput(format!("Failed to recover signer: {e}")))?; - let deposit = envelope.as_deposit().map(|d| { - let mint = if d.mint == 0 { None } else { Some(d.mint) }; - (d.source_hash, mint, d.is_system_transaction) - }); - - let decoded_chain_id = envelope.chain_id(); - let (gas_price, gas_priority_fee) = match &envelope { - OpTxEnvelope::Legacy(_) | OpTxEnvelope::Eip2930(_) => { - (envelope.gas_price().unwrap_or(0), None) - } - OpTxEnvelope::Eip1559(_) | OpTxEnvelope::Eip7702(_) => { - (envelope.max_fee_per_gas(), envelope.max_priority_fee_per_gas()) - } - OpTxEnvelope::Deposit(_) | OpTxEnvelope::PostExec(_) => (0, None), - }; - - let authorization_list = envelope - .authorization_list() - .map(|list| list.iter().map(|sa| Either::Right(sa.clone().into_recovered())).collect()) - .unwrap_or_default(); - - let tx_env = TxEnv { - caller, - gas_price, - gas_priority_fee, - blob_hashes: Vec::new(), - max_fee_per_blob_gas: 0, - tx_type: envelope.ty(), - gas_limit: envelope.gas_limit(), - data: envelope.input().clone(), - nonce: envelope.nonce(), - value: envelope.value(), - access_list: envelope.access_list().cloned().unwrap_or_default(), - authorization_list, - kind: envelope.kind(), - chain_id: decoded_chain_id, - }; - - Ok(Self { tx_env, raw_bytes, deposit }) + Ok(Self { tx: MegaTransaction::from_encoded_tx(&envelope, caller, raw_bytes) }) } - /// Applies explicitly-set [`TxArgs`] fields as overrides to the decoded [`TxEnv`]. + /// Applies explicitly-set [`TxArgs`] fields as overrides to the decoded transaction. /// /// Only fields that were explicitly provided via CLI flags are overridden; /// `None` / empty fields in `tx_args` leave the base value unchanged. pub fn override_tx_env(mut self, tx_args: &TxArgs) -> Result { + let was_deposit = self.tx.base.tx_type == MegaTxType::Deposit as u8; + if let Some(tx_type) = tx_args.tx_type { - self.tx_env.tx_type = tx_type; + self.tx.base.tx_type = tx_type; } if let Some(gas) = tx_args.gas { - self.tx_env.gas_limit = gas; + self.tx.base.gas_limit = gas; } if let Some(basefee) = tx_args.basefee { - self.tx_env.gas_price = basefee as u128; + self.tx.base.gas_price = basefee as u128; } if let Some(priority_fee) = tx_args.priority_fee { - self.tx_env.gas_priority_fee = Some(priority_fee as u128); + self.tx.base.gas_priority_fee = Some(priority_fee as u128); } if let Some(sender) = tx_args.sender { - self.tx_env.caller = sender; + self.tx.base.caller = sender; } if let Some(ref value) = tx_args.value { - self.tx_env.value = parse_ether_value(value)?; + self.tx.base.value = parse_ether_value(value)?; } if let Some(nonce) = tx_args.nonce { - self.tx_env.nonce = nonce; + self.tx.base.nonce = nonce; } if tx_args.input.is_some() || tx_args.inputfile.is_some() { - self.tx_env.data = + self.tx.base.data = load_hex(tx_args.input.clone(), tx_args.inputfile.clone())?.unwrap_or_default(); } if tx_args.create.unwrap_or(false) { - self.tx_env.kind = TxKind::Create; + self.tx.base.kind = TxKind::Create; } else if let Some(receiver) = tx_args.receiver { - self.tx_env.kind = TxKind::Call(receiver); + self.tx.base.kind = TxKind::Call(receiver); } if !tx_args.access.is_empty() { - self.tx_env.access_list = tx_args.parse_access_list()?; + self.tx.base.access_list = tx_args.parse_access_list()?; } if !tx_args.auth.is_empty() { - let chain_id = self.tx_env.chain_id.unwrap_or(0); - self.tx_env.authorization_list = tx_args + let chain_id = self.tx.base.chain_id.unwrap_or(0); + self.tx.base.authorization_list = tx_args .parse_authorization_list(chain_id)? .into_iter() .map(Either::Right) .collect(); } - if let Some((ref mut source_hash, ref mut mint, _)) = self.deposit { + // Deposit overrides apply only to a transaction decoded as a deposit — for any + // other type the deposit parts are defaults that must not be given meaning. + if was_deposit { if let Some(sh) = tx_args.source_hash { - *source_hash = sh; + self.tx.deposit.source_hash = sh; } if tx_args.mint.is_some() { - *mint = tx_args.mint; + self.tx.deposit.mint = tx_args.mint; } } Ok(self) } /// Converts the decoded raw transaction into a [`MegaTransaction`]. - /// - /// Uses the stored raw bytes for `enveloped_tx` (used in L1 fee calculation). pub fn into_tx(self) -> MegaTransaction { - let mut tx = MegaTransaction::new(self.tx_env); - tx.enveloped_tx = Some(self.raw_bytes); - if let Some((source_hash, mint, is_system_transaction)) = self.deposit { - tx.deposit = DepositTransactionParts { source_hash, mint, is_system_transaction }; - } - tx + self.tx } } @@ -608,3 +565,292 @@ fn create_fake_envelope(tx_env: &TxEnv) -> Result { MegaTxType::PostExec => Err(EvmeError::UnsupportedTxType(tx_env.tx_type)), } } + +#[cfg(test)] +mod tests { + use super::*; + use alloy_primitives::b256; + use mega_evm::alloy_consensus::{crypto::secp256k1, SignableTransaction}; + + /// The EIP-155 appendix example: a chain-1 legacy transaction with a known + /// signer, exercising signature recovery on a real signed payload. + const EIP155_RAW: &str = "0xf86c098504a817c800825208943535353535353535353535353535353535353535880de0b6b3a76400008025a028ef61340bd939bc2195fe537567866003e1a15d3c71ff63e1590620aa636276a067cbe9d8997f761aecb703304b3800ccf555c9f3dc64214b297fb1966a3b6d83"; + const EIP155_SIGNER: Address = address!("9d8A62f656a8d1615C1294fd71e9CFb3E4855A4F"); + + /// Hardhat account #0 private key; recovered address is [`DEFAULT_SENDER`]. + const TEST_SECRET: B256 = + b256!("ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"); + + /// Fixed chain and receiver used by the typed-envelope signing vectors. + const TYPED_CHAIN_ID: u64 = 1; + const TYPED_TO: Address = address!("3535353535353535353535353535353535353535"); + const ACCESS_ADDR: Address = address!("1111111111111111111111111111111111111111"); + const ACCESS_KEY: B256 = + b256!("2222222222222222222222222222222222222222222222222222222222222222"); + const AUTH_DELEGATION: Address = address!("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"); + + fn eip155_raw_bytes() -> Bytes { + load_hex(Some(EIP155_RAW.to_string()), None).expect("valid hex").expect("non-empty") + } + + /// Signs a signable transaction body with [`TEST_SECRET`] and returns the + /// EIP-2718 envelope bytes for the given `MegaTxEnvelope` constructor. + fn sign_and_encode_envelope( + envelope: impl FnOnce(Signature) -> MegaTxEnvelope, + signature_hash: B256, + ) -> Bytes { + let sig = secp256k1::sign_message(TEST_SECRET, signature_hash).expect("sign must succeed"); + Bytes::from(envelope(sig).encoded_2718()) + } + + fn sample_access_list() -> AccessList { + AccessList(vec![AccessListItem { address: ACCESS_ADDR, storage_keys: vec![ACCESS_KEY] }]) + } + + /// Builds a genuinely signed EIP-7702 authorization for the fixed fields. + fn sample_signed_authorization() -> SignedAuthorization { + let auth = Authorization { + chain_id: U256::from(TYPED_CHAIN_ID), + address: AUTH_DELEGATION, + nonce: 3, + }; + let sig = secp256k1::sign_message(TEST_SECRET, auth.signature_hash()) + .expect("auth sign must succeed"); + auth.into_signed(sig) + } + + /// A `TxArgs` with no flag set, the base for override tests. + fn empty_tx_args() -> TxArgs { + TxArgs { + tx_type: None, + gas: None, + basefee: None, + priority_fee: None, + sender: None, + receiver: None, + nonce: None, + create: None, + value: None, + input: None, + inputfile: None, + source_hash: None, + mint: None, + auth: Vec::new(), + access: Vec::new(), + } + } + + #[test] + fn test_from_raw_legacy_recovers_signer_and_maps_fields() { + let raw = eip155_raw_bytes(); + let decoded = DecodedRawTx::from_raw(raw.clone()).expect("decode"); + + let base = &decoded.tx.base; + assert_eq!(base.caller, EIP155_SIGNER); + assert_eq!(base.tx_type, 0); + assert_eq!(base.nonce, 9); + assert_eq!(base.gas_limit, 21_000); + assert_eq!(base.gas_price, 20_000_000_000); + assert_eq!(base.kind, TxKind::Call(address!("3535353535353535353535353535353535353535"))); + assert_eq!(base.value, U256::from(10u64).pow(U256::from(18u64))); + assert_eq!(base.chain_id, Some(1)); + assert_eq!( + decoded.tx.enveloped_tx.as_ref(), + Some(&raw), + "the original raw bytes must back the L1 fee calculation", + ); + } + + #[test] + fn test_from_raw_eip2930_recovers_signer_and_preserves_access_list() { + let access_list = sample_access_list(); + let tx = TxEip2930 { + chain_id: TYPED_CHAIN_ID, + nonce: 4, + gas_price: 30_000_000_000, + gas_limit: 50_000, + to: TxKind::Call(TYPED_TO), + value: U256::from(1), + access_list: access_list.clone(), + input: Bytes::from_static(b"\xca\xfe"), + }; + let signature_hash = tx.signature_hash(); + let raw = sign_and_encode_envelope( + |sig| MegaTxEnvelope::Eip2930(tx.into_signed(sig)), + signature_hash, + ); + + let decoded = DecodedRawTx::from_raw(raw.clone()).expect("decode"); + let base = &decoded.tx.base; + + assert_eq!(base.caller, DEFAULT_SENDER, "signer recovery must match the test key"); + assert_eq!(base.tx_type, MegaTxType::Eip2930 as u8); + assert_eq!(base.nonce, 4); + assert_eq!(base.gas_limit, 50_000); + assert_eq!(base.gas_price, 30_000_000_000, "gas_price must survive typed decode"); + assert_eq!(base.kind, TxKind::Call(TYPED_TO), "to must survive typed decode"); + assert_eq!(base.value, U256::from(1), "value must survive typed decode"); + assert_eq!(base.data, Bytes::from_static(b"\xca\xfe"), "input must survive typed decode"); + assert_eq!(base.chain_id, Some(TYPED_CHAIN_ID)); + assert_eq!(base.access_list, access_list, "access list addresses and keys must survive"); + assert_eq!( + decoded.tx.enveloped_tx.as_ref(), + Some(&raw), + "the original raw bytes must back the L1 fee calculation", + ); + } + + #[test] + fn test_from_raw_eip1559_recovers_signer_and_maps_fee_fields() { + let max_fee_per_gas = 40_000_000_000u128; + let max_priority_fee_per_gas = 2_000_000_000u128; + let tx = TxEip1559 { + chain_id: TYPED_CHAIN_ID, + nonce: 7, + gas_limit: 80_000, + max_fee_per_gas, + max_priority_fee_per_gas, + to: TxKind::Call(TYPED_TO), + value: U256::from(2), + access_list: AccessList::default(), + input: Bytes::new(), + }; + let signature_hash = tx.signature_hash(); + let raw = sign_and_encode_envelope( + |sig| MegaTxEnvelope::Eip1559(tx.into_signed(sig)), + signature_hash, + ); + + let decoded = DecodedRawTx::from_raw(raw.clone()).expect("decode"); + let base = &decoded.tx.base; + + assert_eq!(base.caller, DEFAULT_SENDER, "signer recovery must match the test key"); + assert_eq!(base.tx_type, MegaTxType::Eip1559 as u8); + assert_eq!(base.nonce, 7); + assert_eq!(base.gas_limit, 80_000); + assert_eq!(base.chain_id, Some(TYPED_CHAIN_ID)); + assert_eq!(base.kind, TxKind::Call(TYPED_TO), "to must survive typed decode"); + assert_eq!(base.value, U256::from(2), "value must survive typed decode"); + assert_eq!(base.gas_price, max_fee_per_gas, "gas_price must map from max_fee_per_gas"); + assert_eq!( + base.gas_priority_fee, + Some(max_priority_fee_per_gas), + "gas_priority_fee must map from max_priority_fee_per_gas", + ); + assert_eq!( + decoded.tx.enveloped_tx.as_ref(), + Some(&raw), + "the original raw bytes must back the L1 fee calculation", + ); + } + + #[test] + fn test_from_raw_eip7702_recovers_signer_and_preserves_authorization_list() { + let signed_auth = sample_signed_authorization(); + let expected_inner = signed_auth.inner().clone(); + let max_fee_per_gas = 50_000_000_000u128; + let max_priority_fee_per_gas = 1_000_000_000u128; + let tx = TxEip7702 { + chain_id: TYPED_CHAIN_ID, + nonce: 11, + gas_limit: 120_000, + max_fee_per_gas, + max_priority_fee_per_gas, + to: TYPED_TO, + value: U256::ZERO, + access_list: AccessList::default(), + authorization_list: vec![signed_auth], + input: Bytes::new(), + }; + let signature_hash = tx.signature_hash(); + let raw = sign_and_encode_envelope( + |sig| MegaTxEnvelope::Eip7702(tx.into_signed(sig)), + signature_hash, + ); + + let decoded = DecodedRawTx::from_raw(raw.clone()).expect("decode"); + let base = &decoded.tx.base; + + assert_eq!(base.caller, DEFAULT_SENDER, "signer recovery must match the test key"); + assert_eq!(base.tx_type, MegaTxType::Eip7702 as u8); + assert_eq!(base.nonce, 11); + assert_eq!(base.gas_limit, 120_000); + assert_eq!(base.chain_id, Some(TYPED_CHAIN_ID)); + assert_eq!(base.gas_price, max_fee_per_gas, "gas_price must map from max_fee_per_gas"); + assert_eq!( + base.gas_priority_fee, + Some(max_priority_fee_per_gas), + "gas_priority_fee must map from max_priority_fee_per_gas", + ); + assert_eq!(base.kind, TxKind::Call(TYPED_TO), "to must survive typed decode"); + assert_eq!(base.authorization_list.len(), 1, "authorization list length must survive"); + match &base.authorization_list[0] { + Either::Right(recovered) => { + assert_eq!(*recovered.chain_id(), expected_inner.chain_id); + assert_eq!(*recovered.address(), expected_inner.address); + assert_eq!(recovered.nonce(), expected_inner.nonce); + // Authority recovery is independent of field survival: a + // recovery regression that yields RecoveredAuthority::Invalid + // must not pass this test. + assert_eq!( + recovered.authority(), + Some(DEFAULT_SENDER), + "authorization authority must recover to the test signer", + ); + } + Either::Left(signed) => { + panic!( + "from_raw must recover the authorization authority, got unrecovered signed auth: {signed:?}" + ); + } + } + assert_eq!( + decoded.tx.enveloped_tx.as_ref(), + Some(&raw), + "the original raw bytes must back the L1 fee calculation", + ); + } + + #[test] + fn test_from_raw_deposit_fills_deposit_parts() { + let deposit = TxDeposit { + source_hash: b256!("1111111111111111111111111111111111111111111111111111111111111111"), + from: address!("00000000000000000000000000000000000000aa"), + to: TxKind::Call(address!("00000000000000000000000000000000000000bb")), + mint: 5, + value: U256::from(7), + gas_limit: 100_000, + is_system_transaction: false, + input: Bytes::from_static(b"\x01\x02"), + }; + let envelope = MegaTxEnvelope::Deposit(Sealed::new_unchecked(deposit.clone(), B256::ZERO)); + let raw = Bytes::from(envelope.encoded_2718()); + + let decoded = DecodedRawTx::from_raw(raw).expect("decode"); + let tx = decoded.into_tx(); + + assert_eq!(tx.base.tx_type, MegaTxType::Deposit as u8); + assert_eq!(tx.base.caller, deposit.from); + assert_eq!(tx.base.value, deposit.value); + assert_eq!(tx.deposit.source_hash, deposit.source_hash); + assert_eq!(tx.deposit.mint, Some(5)); + assert!(!tx.deposit.is_system_transaction); + } + + #[test] + fn test_override_tx_env_applies_explicit_flags_only() { + let overrides = + TxArgs { gas: Some(300_000), value: Some("2ether".to_string()), ..empty_tx_args() }; + + let decoded = DecodedRawTx::from_raw(eip155_raw_bytes()) + .expect("decode") + .override_tx_env(&overrides) + .expect("override"); + + let base = &decoded.tx.base; + assert_eq!(base.gas_limit, 300_000, "explicit --gas must override"); + assert_eq!(base.value, U256::from(2) * U256::from(10u64).pow(U256::from(18u64))); + assert_eq!(base.caller, EIP155_SIGNER, "unset flags must keep decoded values"); + assert_eq!(base.nonce, 9, "unset flags must keep decoded values"); + } +} diff --git a/bin/mega-evme/src/lib.rs b/bin/mega-evme/src/lib.rs index 02c41cb5..2c4e5687 100644 --- a/bin/mega-evme/src/lib.rs +++ b/bin/mega-evme/src/lib.rs @@ -6,6 +6,8 @@ //! the library directly and exercise the public API the same way an external //! consumer would. +/// Offline RPC cache utilities (`cache merge`, …). +pub mod cache; /// Top-level CLI command parser and dispatch (`MainCmd`, `Commands`, `Error`). pub mod cmd; /// Shared building blocks: RPC provider/session, state, env, error, output @@ -26,18 +28,80 @@ pub use common::*; /// Install a thread panic hook that prints a custom backtrace and exits with a /// non-zero status. Lets failing tests and CLI runs surface a useful trace /// without relying on `RUST_BACKTRACE`. +/// +/// When the raw process argv contains `--json`, the hook also prints the +/// standard structured error object on stdout before exiting so a machine- +/// readable run never ends with empty stdout on panic. +/// +/// Every write the hook performs is fallible: a closed stdout or stderr must +/// not re-panic inside the hook, or the runtime aborts (SIGABRT) before the +/// documented `exit(1)`. Consumers that close the pipe early +/// (`… --json | head`) therefore still see exit class 1 rather than an +/// undefined signal death. pub fn set_thread_panic_hook() { use std::{ backtrace::Backtrace, + io::{self, Write}, panic::{set_hook, take_hook}, process::exit, }; let orig_hook = take_hook(); set_hook(Box::new(move |panic_info| { // Raw stderr rather than `tracing`: the subscriber may not be - // installed yet when a panic fires during CLI startup. - eprintln!("Custom backtrace: {}", Backtrace::capture()); + // installed yet when a panic fires during CLI startup. Discard write + // errors so a closed stderr cannot abort the process from here. + let _ = writeln!(io::stderr(), "Custom backtrace: {}", Backtrace::capture()); orig_hook(panic_info); + if raw_argv_wants_json() { + // Keep the panic text on stderr (via `orig_hook`); the structured + // object is the machine-readable final stdout line. + // `print_json_error` itself is non-panicking on a closed stdout — + // the broken pipe that often triggered this panic must not cause a + // second panic before `exit(1)`. + let message = format!("panic: {panic_info}"); + print_json_error(ExitCode::ExecutionError, &message); + } exit(1); })); } + +/// Whether the raw process argv contains `--json`. +/// +/// Used by the panic hook when the parsed command is not available (and kept +/// public so unit tests can document the same decision as production). +pub fn raw_argv_wants_json() -> bool { + std::env::args_os().any(|arg| arg == "--json") +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The panic-hook JSON decision is driven by raw argv, not the parsed CLI. + #[test] + fn test_raw_argv_wants_json_detects_flag() { + // Unit-test the predicate shape by scanning a synthetic argv slice + // (the production helper reads process args; this mirrors its logic). + fn wants_json(args: &[&str]) -> bool { + args.contains(&"--json") + } + assert!(!wants_json(&["mega-evme", "replay", "0xabc"])); + assert!(wants_json(&["mega-evme", "replay", "--json", "0xabc"])); + assert!(wants_json(&["mega-evme", "--json"])); + // Only the exact flag; a value containing the substring is not enough. + assert!(!wants_json(&["mega-evme", "--json-pretty"])); + } + + /// The structured panic object uses the standard error envelope shape. + #[test] + fn test_panic_json_error_object_shape() { + let code = ExitCode::ExecutionError; + assert_eq!(code.code(), 1); + assert_eq!(code.kind(), "execution-error"); + // Message prefix matches the hook's `panic: …` form; printing itself is + // covered by `print_json_error` and cannot be unit-tested without + // capturing stdout, so a deterministic binary panic trigger is not used. + let message = format!("panic: {}", "explicit test panic"); + assert!(message.starts_with("panic: ")); + } +} diff --git a/bin/mega-evme/src/main.rs b/bin/mega-evme/src/main.rs index da36f3de..8defa229 100644 --- a/bin/mega-evme/src/main.rs +++ b/bin/mega-evme/src/main.rs @@ -3,15 +3,81 @@ //! All business logic lives in the `mega_evme` library crate (`src/lib.rs`). //! This binary is intentionally minimal: parse CLI arguments, install the panic //! hook, dispatch to the parsed command, and exit. +//! +//! This is the only place a finished command becomes a process status; the +//! taxonomy of statuses lives in `common::exit`. + +use std::process::ExitCode; use clap::Parser; -use mega_evme::{ - cmd::{Error, MainCmd}, - set_thread_panic_hook, -}; +use mega_evme::{cmd::MainCmd, print_json_error, report_command_result, set_thread_panic_hook}; #[tokio::main] -async fn main() -> std::result::Result<(), Error> { +async fn main() -> ExitCode { set_thread_panic_hook(); - MainCmd::parse().run().await.inspect_err(|e| println!("{e:?}")) + + // Test-only injection: force a panic after the process-wide hook is + // installed so integration tests can pin the structured JSON envelope on + // an open stdout. Same gate as the fixture pre-state inject + // (`test-utils`, enabled for the test-profile binary via the self + // dev-dependency). Production builds never carry this branch. + // `manual_assert` is allowed: this must be a plain `panic!` payload so the + // hook message stays `panic: …`, not an assertion-failure rewrite. + #[cfg(feature = "test-utils")] + #[allow(clippy::manual_assert)] + if std::env::var_os("MEGA_EVME_INJECT_PANIC").is_some() { + panic!("injected panic for panic-hook JSON envelope test"); + } + + let cmd = match MainCmd::try_parse() { + Ok(cmd) => cmd, + Err(err) => { + // `--help` / `--version` are not failures: clap writes them to + // stdout and the run exits 0. A usage error is bad input, which the + // taxonomy classifies with the other input errors. + let _ = err.print(); + if !err.use_stderr() { + return ExitCode::SUCCESS; + } + let code = mega_evme::ExitCode::ExecutionError; + // The parsed command does not exist yet, so the output mode is read + // off the raw arguments: a `--json` run must end its stdout with the + // structured error object even when it never got as far as running. + if wants_json_output() { + print_json_error(code, &parse_error_summary(&err)); + } + return ExitCode::from(code); + } + }; + + // Read the output mode before `run` consumes the command. + let json = cmd.json_output(); + let result = cmd.run().await; + ExitCode::from(report_command_result(result, json)) +} + +/// Whether the raw arguments ask for machine-readable output. +fn wants_json_output() -> bool { + std::env::args_os().any(|arg| arg == "--json") +} + +/// One-line summary of an argument parsing failure. +/// +/// `clap` renders a multi-line report — the message, the usage block, and the +/// help hint — of which only the leading message paragraph describes what went +/// wrong; it is joined into the single line the structured object carries. +fn parse_error_summary(err: &clap::Error) -> String { + let rendered = err.to_string(); + let summary = rendered + .lines() + .map(str::trim) + .take_while(|line| !line.is_empty()) + .collect::>() + .join(" "); + let summary = summary.strip_prefix("error: ").unwrap_or(&summary).trim(); + if summary.is_empty() { + "invalid command-line arguments".to_string() + } else { + summary.to_string() + } } diff --git a/bin/mega-evme/src/replay/batch.rs b/bin/mega-evme/src/replay/batch.rs new file mode 100644 index 00000000..74f8f066 --- /dev/null +++ b/bin/mega-evme/src/replay/batch.rs @@ -0,0 +1,2238 @@ +//! Batch replay driver: replay many transactions inside a single process. +//! +//! The single-transaction path ([`super::cmd`]) builds a provider, forks state at +//! the parent block, and executes one block per process. Verifying a large corpus +//! that way pays the provider/cache setup once per transaction, which dominates +//! the actual EVM work. This module reuses one provider and one RPC cache for the +//! whole run, groups the requested transactions by their containing block, and +//! executes each block exactly once while recording the result of every target it +//! passes through. +//! +//! Every RPC call issued by a plain batch replay has the same shape as the +//! single-transaction path (`eth_getTransactionByHash`, `eth_getBlockByNumber` +//! with hash-only bodies, and the state reads behind [`EvmeState::new_forked`]), +//! so an offline envelope captured by single-transaction replays serves batch +//! runs without a miss. `--verify-receipt` and `--dump-fixture-dir` are the +//! exception: both fetch `eth_getTransactionReceipt` for *every* target of a +//! block, including the non-targets a single-transaction capture never asked +//! about. Offline, those come back as `rpc` entries and the run exits `3`. + +use std::{ + collections::{BTreeMap, HashMap, HashSet}, + path::{Path, PathBuf}, + str::FromStr, + time::{Duration, Instant}, +}; + +use alloy_consensus::{BlockHeader, Transaction as _}; +use alloy_network::ReceiptResponse; +use alloy_primitives::{Address, B256}; +use alloy_provider::Provider; +use alloy_rpc_types_eth::Block; +use mega_evm::{ + alloy_evm::{ + block::{BlockExecutionError, BlockExecutor, BlockValidationError}, + Evm, EvmEnv, + }, + alloy_op_evm::block::OpAlloyReceiptBuilder, + revm::{ + context::{result::ExecutionResult, ContextTr}, + database::{states::bundle_state::BundleRetention, StateBuilder}, + DatabaseRef, + }, + BlockLimits, MegaBlockExecutionCtx, MegaBlockExecutorFactory, MegaEvmFactory, MegaHaltReason, + MegaHardforks, MegaSpecId, +}; +use op_alloy_rpc_types::Transaction; +use serde::Serialize; +use state_test::types::MegaEnv; +use tracing::{debug, info, warn}; + +use crate::{ + common::{ + op_receipt_to_tx_receipt, print_execution_summary, print_receipt, BatchExitFloor, + BatchFailureCounts, EvmeExternalEnvs, ExecutionSummary, ExitCode, OpTxReceipt, + }, + replay::get_hardfork_config, + ChainArgs, EvmeState, +}; + +use super::{ + cmd::retrieve_block_env, + fixture, + verify::{self, ReceiptFacts, VerificationOutcome}, + ReplayError, Result, +}; + +/// How a batch run reports its targets. +#[derive(Debug, Clone)] +pub(super) struct ReportArgs { + /// Emit one NDJSON line per target instead of the human-readable summary. + pub json: bool, + /// Verify every target against its on-chain receipt. + pub verify_receipt: bool, + /// When set, dump a self-validating fixture for every successful target into + /// this directory as `/.json`. + pub dump_fixture_dir: Option, + /// Replace existing fixture files under [`Self::dump_fixture_dir`]. + pub overwrite: bool, +} + +/// Per-target fixture dump outcome reported on the NDJSON / human result line. +/// +/// Exactly one field is set: the fixture was written, expectedly skipped +/// (fidelity mismatch, BLOCKHASH, unsupported shape), or could not be written. +/// A write failure — and an unanswered receipt question for the fidelity gate — +/// is reported here rather than replacing the target's result, so a target that +/// did replay keeps its result — including its receipt verification verdict — +/// and still fails the run. +#[derive(Debug, Clone, Serialize)] +struct FixtureReport { + /// Absolute or as-written path of a successfully written fixture. + #[serde(skip_serializing_if = "Option::is_none")] + path: Option, + /// Why the fixture was not written for this target. + #[serde(skip_serializing_if = "Option::is_none")] + skipped: Option, + /// Why writing the fixture failed. + #[serde(skip_serializing_if = "Option::is_none")] + error: Option, + /// Batch tally class for [`Self::error`]. Construction and write failures + /// are execution-class; an unanswered on-chain receipt (transport, pruned, + /// divergent inclusion, missing from the offline envelope) is rpc-class; a + /// draft discarded because the block aborted inherits the abort's class so + /// a transient RPC abort does not become exit 1. + /// Not serialized: the wire shape stays `path` / `skipped` / `error`. + #[serde(skip)] + error_kind: BatchErrorKind, +} + +impl FixtureReport { + fn written(path: &Path) -> Self { + Self { + path: Some(path.display().to_string()), + skipped: None, + error: None, + error_kind: BatchErrorKind::Execution, + } + } + + fn skipped(reason: impl Into) -> Self { + Self { + path: None, + skipped: Some(reason.into()), + error: None, + error_kind: BatchErrorKind::Execution, + } + } + + /// Construction or write failure of this target's fixture (execution-class). + fn error(message: impl Into) -> Self { + Self { + path: None, + skipped: None, + error: Some(message.into()), + error_kind: BatchErrorKind::Execution, + } + } + + /// Fidelity gate could not run because the on-chain receipt question went + /// unanswered (transport, null, reorg, or offline envelope missing it). + /// + /// Distinct from a genuine skip (BLOCKHASH, unsupported shape, fidelity + /// mismatch): the dump was requested and the receipt call failed, so the + /// run exits non-zero as rpc-class. + fn rpc_error(message: impl Into) -> Self { + Self { + path: None, + skipped: None, + error: Some(message.into()), + error_kind: BatchErrorKind::Rpc, + } + } + + /// Fixture discarded because the block aborted after the draft was built. + /// + /// The target keeps its execution result; only the fixture field fails, and + /// the failure class matches the abort so the run exit reflects the cause. + fn abort_error(message: impl Into, kind: BatchErrorKind) -> Self { + Self { path: None, skipped: None, error: Some(message.into()), error_kind: kind } + } + + /// Whether the fixture the run was asked to write could not be written. + const fn is_error(&self) -> bool { + self.error.is_some() + } + + /// One-line human summary printed under the transaction header. + fn human_line(&self) -> String { + if let Some(path) = &self.path { + format!("fixture: written to {path}") + } else if let Some(reason) = &self.skipped { + format!("fixture: skipped ({reason})") + } else if let Some(message) = &self.error { + format!("fixture: FAILED ({message})") + } else { + "fixture: (no report)".to_string() + } + } +} + +/// What a batch run was asked to replay. +#[derive(Debug)] +pub(super) enum BatchMode { + /// Transaction hashes read from `--tx-file`, in file order. + TxList(Vec), + /// Every transaction of the block given by `--block`. + Block(u64), +} + +/// Why a target transaction produced no execution result. +/// +/// Execution outcomes (success, revert, halt) are normal results and never map +/// to one of these kinds. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum BatchErrorKind { + /// The transaction hash is unknown to the endpoint. + NotFound, + /// The transaction exists but is not mined yet (no block number). + Pending, + /// An RPC call failed or returned nothing. + Rpc, + /// The block executor rejected the transaction or the block setup failed. + Execution, +} + +impl BatchErrorKind { + /// Wire name used in the NDJSON error line and the human-readable output. + const fn as_str(self) -> &'static str { + match self { + Self::NotFound => "not_found", + Self::Pending => "pending", + Self::Rpc => "rpc", + Self::Execution => "execution", + } + } +} + +/// Outcome of a single target transaction. +enum BatchEntry { + /// The transaction executed and produced a result (success, revert, or halt). + Executed(Box), + /// The transaction could not be executed. + Failed(FailedTx), +} + +impl BatchEntry { + /// Hash of the target this entry reports on. + const fn tx_hash(&self) -> B256 { + match self { + Self::Executed(tx) => tx.tx_hash, + Self::Failed(tx) => tx.tx_hash, + } + } +} + +/// A target transaction that ran to completion. +struct ExecutedTx { + tx_hash: B256, + block_number: u64, + tx_index: u64, + exec_result: ExecutionResult, + contract_address: Option
, + exec_time: Duration, + receipt: OpTxReceipt, + /// On-chain receipt verdict, present iff `--verify-receipt` was given. + verification: Option, + /// Fixture dump outcome, present iff `--dump-fixture-dir` was given. + fixture: Option, +} + +/// A target transaction that hit an infrastructure failure. +struct FailedTx { + tx_hash: B256, + kind: BatchErrorKind, + message: String, +} + +/// Running tally of a batch run's per-target outcomes. +/// +/// A batch reports each target as it goes and fails once at the end, so the +/// outcome classes are counted here rather than recovered from the emitted +/// lines. +/// +/// Per-target counters ([`Self::counts`], [`Self::reported`]) stay strictly +/// about emitted target entries. A non-target abort's class is carried only as +/// [`Self::exit_floor`] so the human "N of M" totals stay truthful while the +/// run exit still reflects the root cause. +#[derive(Debug, Default)] +struct BatchTally { + /// Targets the run reported on, one per emitted entry. + reported: usize, + /// Targets that produced an execution result. + replayed: usize, + /// Targets compared against an on-chain receipt. + verified: usize, + /// Failed and mismatched targets, by class (reported targets only). + counts: BatchFailureCounts, + /// Run-level exit floor from a non-target abort not carried by any target. + exit_floor: BatchExitFloor, +} + +impl BatchTally { + /// Count one reported target. + fn record(&mut self, entry: &BatchEntry) { + match entry { + BatchEntry::Executed(tx) => { + self.record_executed(tx.verification.as_ref(), tx.fixture.as_ref()); + } + // A transaction the endpoint does not know, or that is not mined + // yet, is a definitive answer about the target rather than an + // unanswered question, so it counts as an execution failure. + BatchEntry::Failed(tx) => { + self.reported += 1; + match tx.kind { + BatchErrorKind::Rpc => self.counts.rpc += 1, + BatchErrorKind::NotFound | + BatchErrorKind::Pending | + BatchErrorKind::Execution => self.counts.execution += 1, + } + } + } + } + + /// Count the findings of one target that produced an execution result. + /// + /// A verdict and a fixture failure are independent findings about the same + /// target: a replay that diverged from its receipt is counted as a mismatch + /// whether or not its fixture could be written. An unanswered receipt + /// question (verification unavailable, or dump-dir fidelity gate starved of + /// a receipt) is rpc-class and does not count as verified. + /// + /// When both `--verify-receipt` and `--dump-fixture-dir` fail on the same + /// unanswered receipt, both result fields stay on the line but the shared + /// rpc failure is counted once. + fn record_executed( + &mut self, + verification: Option<&VerificationOutcome>, + fixture: Option<&FixtureReport>, + ) { + self.reported += 1; + self.replayed += 1; + let mut receipt_rpc_counted = false; + if let Some(verification) = verification { + if verification.is_unavailable() { + // Compared path never ran: the receipt question went unanswered. + self.counts.rpc += 1; + receipt_rpc_counted = true; + } else { + self.verified += 1; + if !verification.matched { + self.counts.mismatched += 1; + } + } + } + // A fixture the run was asked to write and could not is a failure of + // that target, even though its replay produced a result. Construction + // and write failures are execution-class; an unanswered receipt for the + // fidelity gate is rpc-class; an abort-inherited discard uses the + // abort's class (see [`FixtureReport::abort_error`]). + if let Some(fixture) = fixture.filter(|f| f.is_error()) { + match fixture.error_kind { + BatchErrorKind::Rpc => { + // Same missing receipt as verification.error: one target, + // one rpc count. Independent fixture rpc failures (none + // today share the gate without verification) still count. + if !receipt_rpc_counted { + self.counts.rpc += 1; + } + } + BatchErrorKind::NotFound | BatchErrorKind::Pending | BatchErrorKind::Execution => { + self.counts.execution += 1 + } + } + } + } + + /// Record a mid-block abort whose root-cause class is not already carried by + /// a per-target failure entry. + /// + /// Swept targets always stay `rpc` ("unanswered"). When the aborting + /// transaction is not itself a reported target, that class would otherwise + /// be lost and a deterministic executor abort would exit 3. The abort is + /// recorded as an exit floor only: it does not emit an NDJSON line, does + /// not increment `reported`, and does not inflate the per-target counters. + fn record_uncounted_abort(&mut self, kind: BatchErrorKind) { + let floor = match kind { + BatchErrorKind::Rpc => BatchExitFloor::Rpc, + BatchErrorKind::NotFound | BatchErrorKind::Pending | BatchErrorKind::Execution => { + BatchExitFloor::Execution + } + }; + // Multiple blocks can each contribute a floor; keep the more severe. + self.exit_floor = match (self.exit_floor, floor) { + (BatchExitFloor::Execution, _) | (_, BatchExitFloor::Execution) => { + BatchExitFloor::Execution + } + (BatchExitFloor::Rpc, _) | (_, BatchExitFloor::Rpc) => BatchExitFloor::Rpc, + (BatchExitFloor::None, BatchExitFloor::None) => BatchExitFloor::None, + }; + } + + /// Targets that failed, by any class other than a receipt mismatch. + const fn failed(&self) -> usize { + self.counts.execution + self.counts.rpc + } + + /// The run's terminal error, or `None` when every target came out clean. + /// + /// Infrastructure failures are reported with their counts by class so the + /// exit-code mapping resolves the precedence between them and a mismatch; a + /// run whose only finding is divergence fails as the mismatch it is. + /// Fixture skips never count as failures; a fixture that could not be + /// written does, as an execution-class failure of its target. + /// + /// A non-target abort floor alone also fails the run (with empty target + /// failure counters) so the exit still reflects the root cause. + fn into_error(self) -> Option { + if self.failed() > 0 || self.exit_floor != BatchExitFloor::None { + return Some(ReplayError::BatchFailed(BatchFailureCounts { + total: self.reported, + exit_floor: self.exit_floor, + ..self.counts + })); + } + if self.counts.mismatched > 0 { + return Some(ReplayError::VerificationMismatch { + mismatched: self.counts.mismatched, + total: self.verified, + }); + } + None + } +} + +/// One target of a [`BlockJob`], carrying the inclusion hash it resolved with. +/// +/// Per-target inclusion (rather than a job-level first-seen anchor) keeps +/// outcomes order-independent: two same-height targets that report different +/// hashes each validate against the fetched block on their own. +struct JobTarget { + hash: B256, + /// Inclusion block hash from `eth_getTransactionByHash` (`--tx-file`). + /// `None` for `--block` targets, which come from the body itself. + inclusion_hash: Option, +} + +/// One block's worth of work. +struct BlockJob { + /// Number of the block holding the targets. + number: u64, + /// Block body, present when planning already fetched it (`--block`). + block: Option>, + /// Targets whose results are reported for this block. + targets: Vec, +} + +/// Fixture work for one target, held until `finish()` succeeds. +/// +/// Skips and construction failures are decided against the pre-commit state and +/// carried as a final report. A successfully built draft is written only after +/// the transaction commits and the block finishes — a commit-time rejection or +/// finish failure must not leave a fixture file on disk (and must not clobber a +/// pre-existing file under `--overwrite`). +enum DeferredFixture { + /// Already decided (skip, construction error, or refused overwrite). + Report(FixtureReport), + /// Draft built against pre-commit state; write after `finish()` succeeds. + /// Boxed so the enum is not dominated by the draft's size on the skip path. + /// `overwrite` is enforced at materialization via noclobber persist — the + /// prep-time existence check is only a fast path. + Ready { draft: Box, path: PathBuf, overwrite: bool }, +} + +/// A target that executed, awaiting the receipt harvested by `finish()`. +struct PendingTarget { + tx_hash: B256, + tx_index: u64, + /// Position of this transaction among the block's committed transactions. + commit_index: usize, + exec_result: ExecutionResult, + exec_time: Duration, + gas_used: u64, + pre_execution_nonce: u64, + from: Address, + to: Option
, + effective_gas_price: u128, + /// Fixture dump work, present iff `--dump-fixture-dir` was given. + fixture: Option, +} + +/// NDJSON line for a target that produced an execution result. +#[derive(Serialize)] +struct BatchResultLine<'a> { + tx_hash: B256, + block_number: u64, + tx_index: u64, + #[serde(flatten)] + summary: &'a ExecutionSummary, + #[serde(skip_serializing_if = "Option::is_none")] + fixture: Option<&'a FixtureReport>, +} + +/// NDJSON line for a target that produced an infrastructure error. +#[derive(Serialize)] +struct BatchErrorLine<'a> { + tx_hash: B256, + error: BatchErrorBody<'a>, +} + +/// Error payload of a [`BatchErrorLine`]. +#[derive(Serialize)] +struct BatchErrorBody<'a> { + kind: &'static str, + message: &'a str, +} + +/// Replay every requested transaction, reporting one entry per target. +/// +/// Returns an error when at least one target produced an infrastructure error +/// entry, so the process exits non-zero; execution outcomes never fail the run. +/// Fixture skips (fidelity gate, BLOCKHASH readers, unsupported tx shapes) do +/// not fail the run. The failure carries the counts by class +/// ([`ReplayError::BatchFailed`]), which decide the exit code. With +/// `--verify-receipt`, a run in which every target replayed but some diverged +/// from its on-chain receipt fails with [`ReplayError::VerificationMismatch`] +/// instead — a distinct variant, so a divergence is never confused with a +/// target that could not be replayed. +pub(super) async fn run

( + provider: &P, + chain_id: u64, + mode: &BatchMode, + external_envs: EvmeExternalEnvs, + report: ReportArgs, +) -> Result<()> +where + P: Provider + Clone + std::fmt::Debug, +{ + let start = Instant::now(); + let mut tally = BatchTally::default(); + let mut fixtures_written = 0usize; + let mut fixtures_skipped = 0usize; + let mut fixtures_failed = 0usize; + + if let Some(dir) = &report.dump_fixture_dir { + std::fs::create_dir_all(dir).map_err(|e| { + ReplayError::Other(format!( + "failed to create --dump-fixture-dir '{}': {e}", + dir.display() + )) + })?; + } + + let jobs = match mode { + BatchMode::Block(number) => { + let block = fetch_block(provider, *number).await?; + let targets: Vec = block.transactions.hashes().collect(); + info!(block = number, tx_count = targets.len(), "Batch replay of a whole block"); + if targets.is_empty() { + // Nothing failed, so this is a clean exit — but a silent one is + // indistinguishable from a run that produced no output for a bad + // reason, so say why stdout is empty. No job is queued: with no + // targets to report, forking the parent state would buy nothing. + eprintln!("Block {number} contains no transactions; nothing to replay"); + vec![] + } else { + vec![BlockJob { + number: *number, + block: Some(block), + // Whole-block mode takes its targets from the body, so there + // is no separate inclusion claim to reconcile later. + targets: targets + .into_iter() + .map(|hash| JobTarget { hash, inclusion_hash: None }) + .collect(), + }] + } + } + BatchMode::TxList(hashes) => { + let (jobs, failures) = resolve_targets(provider, hashes).await; + info!( + requested = hashes.len(), + blocks = jobs.len(), + unresolved = failures.len(), + "Batch replay of a transaction list", + ); + for failure in failures { + let entry = BatchEntry::Failed(failure); + tally.record(&entry); + emit(&entry, report.json); + } + jobs + } + }; + + for job in jobs { + let outcome = replay_block(provider, chain_id, job, external_envs.clone(), &report).await; + for entry in outcome.entries { + if let BatchEntry::Executed(tx) = &entry { + match &tx.fixture { + Some(fixture) if fixture.path.is_some() => fixtures_written += 1, + Some(fixture) if fixture.is_error() => fixtures_failed += 1, + Some(_) => fixtures_skipped += 1, + None => {} + } + } + tally.record(&entry); + emit(&entry, report.json); + } + // Root-cause class of a non-target abort is not on any per-target line. + if let Some(kind) = outcome.uncounted_abort { + tally.record_uncounted_abort(kind); + } + } + + info!( + replayed = tally.replayed, + failed = tally.failed(), + elapsed = ?start.elapsed(), + "Batch replay finished", + ); + if report.verify_receipt { + info!( + verified = tally.verified, + mismatched = tally.counts.mismatched, + "On-chain receipt verification finished", + ); + } + if report.dump_fixture_dir.is_some() { + info!( + written = fixtures_written, + skipped = fixtures_skipped, + failed = fixtures_failed, + "Fixture dump finished" + ); + } + + tally.into_error().map_or(Ok(()), Err) +} + +/// Resolve each requested hash to its containing block. +/// +/// Returns the per-block jobs in ascending block order, plus the failures for +/// hashes that could not be resolved (in the order they were requested). +/// +/// Grouping is by block number only. Each target keeps the inclusion hash its +/// own lookup reported; agreement with the fetched block is checked later in +/// [`replay_block`], so two same-height targets that disagree with each other +/// still get independent outcomes instead of a first-seen race. +async fn resolve_targets

(provider: &P, hashes: &[B256]) -> (Vec, Vec) +where + P: Provider, +{ + let mut grouped: BTreeMap> = BTreeMap::new(); + let mut failures = Vec::new(); + + for hash in hashes { + match provider.get_transaction_by_hash(*hash).await { + Err(e) => failures.push(FailedTx { + tx_hash: *hash, + kind: BatchErrorKind::Rpc, + message: format!("Failed to fetch transaction: {e}"), + }), + Ok(None) => failures.push(FailedTx { + tx_hash: *hash, + kind: BatchErrorKind::NotFound, + message: "Transaction not found".to_string(), + }), + // Every (block_number, block_hash) shape the endpoint can return is + // handled explicitly so a contradictory row cannot fall through a + // wildcard into the pending arm. + Ok(Some(tx)) => match (tx.block_number, tx.block_hash) { + (Some(number), Some(theirs)) => { + grouped + .entry(number) + .or_default() + .push(JobTarget { hash: *hash, inclusion_hash: Some(theirs) }); + } + // A mined transaction without an inclusion hash is an unanchored + // view: the number alone cannot prove which block body to replay + // against, so the target is unanswered rather than queued without + // an inclusion claim. + (Some(number), None) => failures.push(FailedTx { + tx_hash: *hash, + kind: BatchErrorKind::Rpc, + message: format!( + "endpoint reported a mined transaction in block {number} \ + without an inclusion hash: unanchored view" + ), + }), + // A hash proves inclusion; a null number denies it. That pair is + // self-contradictory metadata, not a pending transaction. + (None, Some(hash_value)) => failures.push(FailedTx { + tx_hash: *hash, + kind: BatchErrorKind::Rpc, + message: format!( + "endpoint reported inclusion hash {hash_value} without a block \ + number: contradictory metadata" + ), + }), + (None, None) => failures.push(FailedTx { + tx_hash: *hash, + kind: BatchErrorKind::Pending, + message: "Transaction is pending (no block number)".to_string(), + }), + }, + } + } + + let jobs = grouped + .into_iter() + .map(|(number, targets)| BlockJob { number, block: None, targets }) + .collect(); + (jobs, failures) +} + +/// Outcome of replaying one block's targets. +struct BlockReplayOutcome { + /// One entry per target of the job (executed or failed). + entries: Vec, + /// Root-cause class of a mid-block abort that no reported entry carries. + /// + /// Present when the aborting transaction is not a target: swept targets stay + /// `rpc`, and this class is tallied so the run exit reflects the abort. + uncounted_abort: Option, +} + +impl BlockReplayOutcome { + /// Order entries into documented stream order before returning. + /// + /// Pre-execution inclusion/membership failures are collected before the + /// execute loop, while canonical results are appended after `finish()`. + /// Without a final reorder, a later same-block target's inclusion failure + /// would precede an earlier target's execution result. + fn ordered( + entries: Vec, + job_targets: &[JobTarget], + block_tx_order: Option<&[B256]>, + uncounted_abort: Option, + ) -> Self { + Self { entries: order_block_entries(entries, job_targets, block_tx_order), uncounted_abort } + } +} + +/// Order a block's entries: targets present in the body by ascending transaction +/// index, then targets the block cannot place (inclusion/membership failures) +/// last, in job input order. +fn order_block_entries( + entries: Vec, + job_targets: &[JobTarget], + block_tx_order: Option<&[B256]>, +) -> Vec { + if entries.len() <= 1 { + return entries; + } + let mut by_hash: HashMap = HashMap::with_capacity(entries.len()); + for entry in entries { + by_hash.insert(entry.tx_hash(), entry); + } + let mut ordered = Vec::with_capacity(by_hash.len()); + if let Some(block_txs) = block_tx_order { + for hash in block_txs { + if let Some(entry) = by_hash.remove(hash) { + ordered.push(entry); + } + } + } + // Residual targets (absent from the body, or no body order available) keep + // the job's input order — the documented absent-last placement. + for target in job_targets { + if let Some(entry) = by_hash.remove(&target.hash) { + ordered.push(entry); + } + } + // Defensive: anything not listed on the job (should not happen). + ordered.extend(by_hash.into_values()); + ordered +} + +/// Replay one block, reporting an entry for every target it was asked about. +/// +/// The block is executed exactly once: every transaction runs in order, and each +/// target's result is recorded before the transaction is committed. Receipts are +/// harvested from the finished block, which is why the block's entries are only +/// produced once the block is done. +/// +/// When `--dump-fixture-dir` is set, each target's fixture draft is built from +/// the pre-commit state (same moment as the single-transaction dump), gated per +/// target for fidelity and BLOCKHASH, and written only after the block +/// `finish()` succeeds — so a commit-time rejection or finish failure cannot +/// leave a fixture file on disk. +async fn replay_block

( + provider: &P, + chain_id: u64, + job: BlockJob, + external_envs: EvmeExternalEnvs, + report: &ReportArgs, +) -> BlockReplayOutcome +where + P: Provider + Clone + std::fmt::Debug, +{ + let BlockJob { number, block, targets: job_targets } = job; + let verify_receipt = report.verify_receipt; + let dump_dir = report.dump_fixture_dir.as_deref(); + let overwrite = report.overwrite; + let target_hashes = || job_targets.iter().map(|t| t.hash); + + if number == 0 { + // Distinct from `--block 0` (invalid request, exit 1): an endpoint that + // resolves a hash into block 0 is contradictory endpoint data — the + // same unanswered class as unanchored / contradictory metadata. + return BlockReplayOutcome::ordered( + fail_all( + target_hashes(), + BatchErrorKind::Rpc, + "endpoint resolved the target into block 0, which has no parent block \ + to fork from: contradictory endpoint data", + ), + &job_targets, + None, + None, + ); + } + + let block = match block { + Some(block) => block, + None => match fetch_block(provider, number).await { + Ok(block) => block, + Err(e) => { + return BlockReplayOutcome::ordered( + fail_all(target_hashes(), BatchErrorKind::Rpc, &e.to_string()), + &job_targets, + None, + None, + ); + } + }, + }; + // Body order for the documented ascending `(block, tx_index)` stream. + let block_tx_order: Vec = block.transactions.hashes().collect(); + + // Per-target inclusion and membership guards. `--tx-file` resolved each + // target through `eth_getTransactionByHash`, which reported the block it + // belongs to. Agreement is checked against the fetched body, not against + // a first-seen peer, so two same-height targets that report different + // hashes get independent outcomes. A target whose reported hash matches + // the body but is missing from it is an endpoint self-contradiction (`rpc`), + // not a definitive "unknown hash". + // + // When none of the job's targets appear in the body, every target already + // has its definitive answer here — skip parent fetch, state forking, and + // the execute loop entirely. Otherwise `last_target_index` would be `None` + // and the foreign block would be walked for nothing. + // + // Pre-execution failures are buffered into `entries` and reordered with + // executed results at return time so a later-index inclusion failure cannot + // precede an earlier target's result line. + let fetched = block.hash(); + let body_txs: HashSet = block_tx_order.iter().copied().collect(); + let mut entries = Vec::with_capacity(job_targets.len()); + let mut active: Vec = Vec::new(); + for target in &job_targets { + if let Some(reported) = target.inclusion_hash { + if reported != fetched { + entries.push(failure( + target.hash, + BatchErrorKind::Rpc, + format!( + "block {number} has hash {fetched}, but the target transaction was \ + resolved as included in {reported}: the endpoint served divergent \ + views of this block (reorg in progress, or a load-balanced \ + endpoint); retry once the chain settles" + ), + )); + continue; + } + if !body_txs.contains(&target.hash) { + entries.push(failure( + target.hash, + BatchErrorKind::Rpc, + format!( + "block {number} ({fetched}) does not list target transaction {}, which \ + the endpoint resolved as included in it: the endpoint served \ + divergent views of this block (reorg in progress, or a \ + load-balanced endpoint); retry once the chain settles", + target.hash, + ), + )); + continue; + } + } else if !body_txs.contains(&target.hash) { + // `--block` targets come from the body, so this arm is defensive. + // A residual not-in-body without an inclusion claim is still an + // unanswered view of this height, not a definitive not-found. + entries.push(failure( + target.hash, + BatchErrorKind::Rpc, + format!( + "block {number} ({fetched}) does not list target transaction {}, which \ + was queued against it: the endpoint served divergent views of this \ + block (reorg in progress, or a load-balanced endpoint); retry once \ + the chain settles", + target.hash, + ), + )); + continue; + } + active.push(target.hash); + } + // Every target either failed an inclusion/membership check or was a + // `--block` target already taken from the body. Nothing left to execute. + if active.is_empty() { + return BlockReplayOutcome::ordered(entries, &job_targets, Some(&block_tx_order), None); + } + let targets = active; + + // Both guards below check the *headers* the endpoint served. The state + // reads behind the fork are still addressed by block number, so an endpoint + // that serves headers and state from different backends can still hand back + // state for a different block at this height. Anchoring state reads to the + // validated hash would need the fork to take a block hash rather than a + // number, and would change every cached RPC key (alloy hashes the block id + // into the cache key), invalidating every committed offline capture. + // + // Parent/block linkage guard: across a reorg or a load-balanced endpoint + // serving divergent views, `eth_getBlockByNumber(N-1)` can return a block + // that is not the parent of the block being replayed. Forking from that + // state would silently execute against the wrong pre-state. + let parent_block = match fetch_block(provider, number - 1).await { + Ok(block) => block, + Err(e) => { + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Rpc, &e.to_string()), + &job_targets, + Some(&block_tx_order), + None, + ); + } + }; + let parent_hash = parent_block.hash(); + let expected_parent = block.header.parent_hash(); + if parent_hash != expected_parent { + let message = format!( + "parent block hash {parent_hash} != block parent_hash {expected_parent}: the parent \ + block describes a different chain than the block being replayed (reorg in progress, \ + or a load-balanced endpoint serving divergent views); retry once the chain settles" + ); + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Rpc, &message), + &job_targets, + Some(&block_tx_order), + None, + ); + } + + // Fetch the on-chain receipts before the block runs. Needed for + // `--verify-receipt` (mismatch vs unverified) and for `--dump-fixture-dir` + // (fidelity gate). A receipt that cannot be fetched, or that describes a + // different inclusion than this block, is recorded here and interpreted by + // each feature below. + let need_receipts = verify_receipt || dump_dir.is_some(); + let onchain_receipts = if need_receipts { + fetch_target_receipts(provider, &targets, block.hash()).await + } else { + BTreeMap::new() + }; + + // Sorted once so every fixture of this block is byte-reproducible for the + // same megaEnv (hash-map iteration order is otherwise non-deterministic). + let mega_env = dump_dir.map(|_| { + let mut bucket_capacities = external_envs.bucket_capacities(); + bucket_capacities.sort_unstable(); + let mut oracle_storage = external_envs.oracle_storage(); + oracle_storage.sort_unstable(); + MegaEnv { bucket_capacities, oracle_storage } + }); + + let hardforks = get_hardfork_config(chain_id); + let timestamp = block.header.timestamp(); + let spec = hardforks.spec_id(timestamp); + let chain_args = ChainArgs { chain_id, spec: spec.to_string() }; + debug!(block = number, chain_id, spec = %spec, "Block configuration"); + + let cfg_env = match chain_args.create_cfg_env() { + Ok(cfg) => cfg, + Err(e) => { + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Execution, &e.to_string()), + &job_targets, + Some(&block_tx_order), + None, + ); + } + }; + let block_env = match retrieve_block_env(&block) { + Ok(env) => env, + Err(e) => { + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Execution, &e.to_string()), + &job_targets, + Some(&block_tx_order), + None, + ); + } + }; + let executed_spec = cfg_env.spec; + let evm_env = EvmEnv::new(cfg_env, block_env); + + let Some(hardfork) = hardforks.hardfork(timestamp) else { + let message = format!("No `MegaHardfork` active at block timestamp: {timestamp}"); + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Execution, &message), + &job_targets, + Some(&block_tx_order), + None, + ); + }; + let block_limits = + BlockLimits::from_hardfork_and_block_gas_limit(hardfork, block.header.gas_limit()); + let block_ctx = MegaBlockExecutionCtx::new( + parent_block.hash(), + block.header.parent_beacon_block_root(), + block.header.extra_data().clone(), + block_limits, + ); + + info!(block = number, fork_block = parent_block.header.number(), "Forking state for block"); + let mut database = match EvmeState::new_forked( + provider.clone(), + Some(parent_block.header.number()), + Default::default(), + Default::default(), + ) + .await + { + Ok(database) => database, + Err(e) => { + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, BatchErrorKind::Rpc, &e.to_string()), + &job_targets, + Some(&block_tx_order), + None, + ); + } + }; + + let evm_factory = MegaEvmFactory::new().with_external_env_factory(external_envs); + let block_executor_factory = + MegaBlockExecutorFactory::new(&hardforks, evm_factory, OpAlloyReceiptBuilder::default()); + let mut state = StateBuilder::new().with_database(&mut database).with_bundle_update().build(); + let mut block_executor = block_executor_factory.create_executor(&mut state, block_ctx, evm_env); + + if let Err(e) = block_executor.apply_pre_execution_changes() { + let error = ReplayError::BlockExecutionError(e); + return BlockReplayOutcome::ordered( + fail_remaining(&targets, entries, classify(&error), &error.to_string()), + &job_targets, + Some(&block_tx_order), + None, + ); + } + + let target_set: HashSet = targets.iter().copied().collect(); + // Prefer the already-collected body order so stream ordering and the loop + // walk the same sequence. + let tx_hashes = block_tx_order.clone(); + // Highest block index among this job's targets: once that transaction has + // committed we can stop — later non-targets are not needed for receipts or + // fixtures, and requiring them would force incomplete offline captures to + // abort after a successful dump target. + let last_target_index = tx_hashes + .iter() + .enumerate() + .filter(|(_, hash)| target_set.contains(*hash)) + .map(|(i, _)| i) + .max(); + let mut pending: Vec = Vec::new(); + let mut committed = 0usize; + + // Run the block's transactions in order. Any failure aborts the block: the + // executor state no longer matches the chain, so the remaining targets + // cannot be replayed faithfully. + // + // `in_flight` names the transaction whose iteration raised the abort. It is + // the attribution ground truth: some rejections raised *about* a + // transaction do not embed its hash in the error (the block-gas admission + // check, for one), and attributing from error introspection alone would + // sweep the aborter itself as an unanswered peer. + let mut in_flight: Option = None; + let loop_result: Result<()> = async { + for (tx_index, tx_hash) in tx_hashes.iter().enumerate() { + in_flight = Some(*tx_hash); + // Isolate BLOCKHASH reads per transaction so a fixture dump sees only + // the target's own accesses (mirrors the single-tx clear after + // preceding transactions). + block_executor.clear_accessed_block_hashes(); + + // Every hash here came from the block body this endpoint already + // served. `Ok(None)` therefore means the endpoint is inconsistent + // (reorg or load-balanced divergent views), not that the hash is + // unknown — that definitive answer only applies to a user-supplied + // target lookup on the single-transaction path. + let tx = provider + .get_transaction_by_hash(*tx_hash) + .await + .map_err(|e| ReplayError::BlockBodyTransactionFetch { + tx_hash: *tx_hash, + message: e.to_string(), + })? + .ok_or(ReplayError::BlockBodyTransactionNull(*tx_hash))?; + // A served object that fails authentication is the same class as a + // null answer on a body-listed hash: the endpoint failed to deliver + // a transaction it claimed to include. Executing it instead would + // advance the block state on the wrong transaction, or report + // another transaction's outcome under a target hash. + verify::authenticate_transaction(&tx, *tx_hash).map_err(|message| { + ReplayError::BlockBodyTransactionFetch { tx_hash: *tx_hash, message } + })?; + + let is_target = target_set.contains(tx_hash); + let start = Instant::now(); + let pre_execution_nonce = if is_target { + block_executor + .evm() + .db_ref() + .basic_ref(tx.inner.inner.signer())? + .map(|acc| acc.nonce) + .unwrap_or(0) + } else { + 0 + }; + + let outcome = block_executor + .run_transaction(tx.as_recovered()) + .map_err(ReplayError::BlockExecutionError)?; + + // Fixture draft must be built before commit: the pre-state closure + // is the database after preceding txs, with the target's result + // state still uncommitted — same moment as the single-tx dump. + // The draft is only written after `finish()` succeeds (see harvest). + let fixture = if is_target { + if let (Some(dir), Some(mega_env)) = (dump_dir, mega_env.as_ref()) { + let accessed_block_hashes = block_executor.get_accessed_block_hashes(); + Some(prepare_target_fixture( + block_executor.evm().db_ref(), + DumpFixtureArgs { + accessed_block_hash_count: accessed_block_hashes.len(), + exec_result: &outcome.inner.result, + evm_state: &outcome.inner.state, + chain_id, + executed_spec, + block: &block, + target_tx: &tx, + mega_env: mega_env.clone(), + onchain: onchain_receipts.get(tx_hash), + dir, + overwrite, + }, + )) + } else { + None + } + } else { + None + }; + + // Record the target's result before committing, mirroring the + // single-transaction path. + let exec_result = is_target.then(|| outcome.inner.result.clone()); + let gas_used = block_executor + .commit_transaction_outcome(outcome) + .map_err(ReplayError::BlockExecutionError)?; + let commit_index = committed; + committed += 1; + + if let Some(exec_result) = exec_result { + pending.push(PendingTarget { + tx_hash: *tx_hash, + tx_index: tx_index as u64, + commit_index, + exec_result, + exec_time: start.elapsed(), + gas_used, + pre_execution_nonce, + from: tx.inner.inner.signer(), + to: tx.inner.inner.to(), + effective_gas_price: tx.inner.effective_gas_price.unwrap_or(0), + fixture, + }); + } + + // Stop once every requested target that can run has committed: trailing + // non-targets are irrelevant to this job's receipts and fixtures. + if Some(tx_index) == last_target_index { + break; + } + } + Ok(()) + } + .await; + + // Finish the block even when it aborted midway: targets that already ran + // still have a receipt worth reporting. `entries` already holds any + // inclusion/membership failures recorded before the block started. + match block_executor.finish() { + Ok((evm, block_result)) => { + let (db, _) = evm.finish(); + db.merge_transitions(BundleRetention::Reverts); + let receipts = block_result.receipts; + // Receipts are pushed one per committed transaction; index from the + // end so any receipt produced before the first transaction (now or + // later) cannot shift the mapping. + let offset = receipts.len().saturating_sub(committed); + let block_hash = block.hash(); + // A fixture that could not be written stays on the target's own + // result line below: the target did replay, so its receipt and its + // verification verdict are still what the run was asked for. The + // failed dump fails the run through the tally, not by replacing the + // result with an error entry. + // + // Finalize+write runs only here, after finish succeeded: a + // commit-time rejection never reaches pending, and a finish failure + // drops ready drafts unwritten (see the Err arm). + for target in pending { + let Some(envelope) = receipts.get(offset + target.commit_index) else { + entries.push(failure( + target.tx_hash, + BatchErrorKind::Execution, + format!("No receipt produced for transaction index {}", target.tx_index), + )); + continue; + }; + let contract_address = (target.to.is_none() && envelope.is_success()) + .then(|| target.from.create(target.pre_execution_nonce)); + // Block-global log index: cumulative log count of all committed + // receipts that precede this target in the block. + let first_log_index: u64 = receipts[offset..offset + target.commit_index] + .iter() + .map(|r| r.logs().len() as u64) + .sum(); + let receipt = op_receipt_to_tx_receipt( + envelope, + number, + timestamp, + target.from, + target.to, + contract_address, + target.effective_gas_price, + target.gas_used, + Some(target.tx_hash), + Some(block_hash), + target.tx_index, + first_log_index, + ); + // Keep the execution result even when the receipt question went + // unanswered: the target did replay, so its summary, local + // receipt, and timing stay on the result line. The verification + // field carries the failure; the tally counts it as rpc. + let verification = if verify_receipt { + match onchain_receipts.get(&target.tx_hash) { + Some(Ok(onchain)) => { + Some(verify::compare(onchain, &ReceiptFacts::from_receipt(&receipt))) + } + Some(Err(message)) => { + Some(VerificationOutcome::unavailable(message.clone())) + } + None => Some(VerificationOutcome::unavailable( + "No on-chain receipt was fetched for this transaction", + )), + } + } else { + None + }; + // Materialize only when the block loop completed cleanly: a + // mid-block abort after this target built a Ready draft must + // not publish (or clobber) a fixture for a block that failed. + // Keep the execution result; only the fixture field fails, and + // it inherits the abort's class so a transient RPC abort exits 3. + let fixture = target.fixture.map(|deferred| match &loop_result { + Ok(()) => materialize_deferred_fixture(deferred), + Err(abort) => match deferred { + DeferredFixture::Report(report) => report, + DeferredFixture::Ready { path, .. } => FixtureReport::abort_error( + format!( + "fixture not written: block aborted before a clean finish \ + (draft for {} was discarded): {abort}", + path.display() + ), + classify(abort), + ), + }, + }); + entries.push(BatchEntry::Executed(Box::new(ExecutedTx { + tx_hash: target.tx_hash, + block_number: number, + tx_index: target.tx_index, + exec_result: target.exec_result, + contract_address, + exec_time: target.exec_time, + receipt, + verification, + fixture, + }))); + } + } + // The block itself failed to finish, so no target of it has a receipt + // and no deferred fixture is written or replaced. + Err(e) => { + let error = ReplayError::BlockExecutionError(e); + let kind = classify(&error); + let message = error.to_string(); + for target in pending { + entries.push(failure(target.tx_hash, kind, message.clone())); + } + } + } + + // Any active target that produced no entry sat behind an abort (or is a + // residual not-in-body case for `--block`, which has no inclusion claim). + // They are appended in block transaction-index order, keeping the run's + // ascending (block, index) order; a target the block does not contain has + // no index and keeps its input position among the active set. + let reported: HashSet = entries.iter().map(BatchEntry::tx_hash).collect(); + let block_txs: HashSet = tx_hashes.iter().copied().collect(); + let unreported = tx_hashes + .iter() + .filter(|hash| target_set.contains(*hash)) + .chain(targets.iter().filter(|hash| !block_txs.contains(*hash))) + .filter(|hash| !reported.contains(*hash)); + + let mut uncounted_abort = None; + match &loop_result { + Ok(()) => { + // Active targets are already filtered for inclusion agreement; a + // remaining absence from the body is still an endpoint + // inconsistency (the target was queued against this block), not a + // definitive not-found. + for tx_hash in unreported { + entries.push(failure( + *tx_hash, + BatchErrorKind::Rpc, + format!( + "block {number} ({fetched}) does not list target transaction {tx_hash}, \ + which the endpoint resolved as included in it: the endpoint served \ + divergent views of this block (reorg in progress, or a load-balanced \ + endpoint); retry once the chain settles" + ), + )); + } + } + Err(e) => { + warn!(block = number, error = %e, "Aborted block replay; skipping its remaining targets"); + // The iteration that raised the abort is authoritative; error + // introspection only covers errors raised outside the loop (a + // failed `finish`, for one), which can still name a transaction. + let aborting = in_flight.or_else(|| aborting_tx_hash(e)); + let root_kind = classify(e); + let mut root_on_target = false; + for tx_hash in unreported { + if aborting == Some(*tx_hash) { + // The abort is this target's own answer. + root_on_target = true; + entries.push(failure(*tx_hash, root_kind, e.to_string())); + } else { + // The abort belongs to another transaction of the block, so + // nothing was established about this target: it went + // unanswered rather than being unknown or invalid. + entries.push(failure( + *tx_hash, + swept_kind(e), + format!("Block replay aborted before this transaction: {e}"), + )); + } + } + // When the aborter is not a reported target, no failure entry carries + // the abort's own class. Tallied separately so the run exit reflects + // the root cause (e.g. exit 1 for a deterministic non-target abort). + // Fixture abort-errors on executed targets may also carry the class; + // double-counting the same class still yields the correct exit. + if !root_on_target { + // If finish failed for pending targets that already include the + // aborter as a Failed entry, the class is already counted. + let already_counted = aborting.is_some_and(|hash| { + entries.iter().any(|entry| match entry { + BatchEntry::Failed(tx) => tx.tx_hash == hash && tx.kind == root_kind, + BatchEntry::Executed(_) => false, + }) + }); + // Abort-inherited fixture failures on executed targets already + // contribute the abort class to the tally. + let fixture_carries_class = entries.iter().any(|entry| match entry { + BatchEntry::Executed(tx) => tx + .fixture + .as_ref() + .is_some_and(|f| f.is_error() && f.error_kind == root_kind), + BatchEntry::Failed(_) => false, + }); + if !already_counted && !fixture_carries_class { + uncounted_abort = Some(root_kind); + } + } + } + } + + BlockReplayOutcome::ordered(entries, &job_targets, Some(&block_tx_order), uncounted_abort) +} + +/// Inputs for [`prepare_target_fixture`], grouped so the dump path stays a single +/// call site without a long positional argument list. +struct DumpFixtureArgs<'a> { + accessed_block_hash_count: usize, + exec_result: &'a ExecutionResult, + evm_state: &'a mega_evm::revm::state::EvmState, + chain_id: u64, + executed_spec: MegaSpecId, + block: &'a Block, + target_tx: &'a Transaction, + mega_env: MegaEnv, + onchain: Option<&'a std::result::Result>, + dir: &'a Path, + overwrite: bool, +} + +/// Prepare a fixture for one successfully executed target against pre-commit state. +/// +/// Genuine skips (fidelity mismatch, BLOCKHASH, unsupported transaction shapes) +/// become a final [`FixtureReport::skipped`] and never fail the run. +/// An unanswered on-chain receipt (transport, pruned/null, divergent inclusion, +/// or offline envelope lacking it) becomes a rpc-class fixture error so the run +/// exits 3 while the target keeps its execution result line. +/// Database and other construction failures become a fixture error (execution-class). +/// A successfully built draft is carried as [`DeferredFixture::Ready`] and only +/// written by [`materialize_deferred_fixture`] after `finish()` succeeds. +/// +/// `db` must reflect the pre-target-commit state (preceding txs committed, target +/// not yet), matching the single-transaction dump. +fn prepare_target_fixture(db: &DB, args: DumpFixtureArgs<'_>) -> DeferredFixture +where + DB: DatabaseRef, + DB::Error: core::fmt::Display, +{ + let DumpFixtureArgs { + accessed_block_hash_count, + exec_result, + evm_state, + chain_id, + executed_spec, + block, + target_tx, + mega_env, + onchain, + dir, + overwrite, + } = args; + + // Fidelity gate needs the on-chain receipt. When the receipt question went + // unanswered the dump fails as rpc (not a fidelity-gate skip): the run was + // asked to write a fixture and could not obtain the receipt it needs. Genuine + // gate skips (BLOCKHASH, unsupported shape, fidelity mismatch) stay skips. + let facts = match onchain { + Some(Ok(facts)) => facts, + Some(Err(message)) => { + return DeferredFixture::Report(FixtureReport::rpc_error(message.clone())); + } + None => { + return DeferredFixture::Report(FixtureReport::rpc_error( + "no on-chain receipt was fetched for this transaction", + )); + } + }; + + if accessed_block_hash_count > 0 { + return DeferredFixture::Report(FixtureReport::skipped(format!( + "transaction reads block hashes (BLOCKHASH): {accessed_block_hash_count} block \ + hash(es) were accessed and the fixture cannot faithfully reproduce them" + ))); + } + + let anchor = fixture::anchor_from_receipt_facts(facts); + if let Err(reason) = fixture::check_fidelity(exec_result, &anchor, chain_id) { + return DeferredFixture::Report(FixtureReport::skipped(format!( + "fidelity gate failed: {reason}" + ))); + } + + let draft = match fixture::build_draft( + db, + evm_state, + chain_id, + executed_spec, + block, + target_tx, + fixture::FixtureInputs { mega_env, result: exec_result, anchor }, + ) { + Ok(draft) => draft, + Err(e) => return DeferredFixture::Report(fixture_report_from_build_err(e)), + }; + + let tx_hash = target_tx.inner.inner.tx_hash(); + let path = dir.join(format!("{tx_hash:#x}.json")); + // Fast-path courtesy: refuse overwrite before carrying a ready draft so the + // harvest path never confuses a finish failure with an overwrite refusal. + // Correctness against a concurrent creator is still enforced at materialize + // time via noclobber persist. + if path.exists() && !overwrite { + return DeferredFixture::Report(FixtureReport::error(format!( + "fixture already exists at {} (pass --overwrite to replace)", + path.display() + ))); + } + + DeferredFixture::Ready { draft: Box::new(draft), path, overwrite } +} + +/// Finalize a deferred fixture after the block `finish()` succeeded. +/// +/// Ready drafts are self-validated and written here. Pre-decided reports pass +/// through unchanged. On finish failure the caller drops the deferred value +/// without calling this, so no file is written or replaced. +fn materialize_deferred_fixture(deferred: DeferredFixture) -> FixtureReport { + match deferred { + DeferredFixture::Report(report) => report, + DeferredFixture::Ready { draft, path, overwrite } => { + match fixture::finalize_and_write(*draft, &path, overwrite) { + Ok(()) => { + info!(path = %path.display(), "Wrote self-validating fixture"); + FixtureReport::written(&path) + } + Err(e) => { + let message = e.to_string(); + // Noclobber / prep-time refusal already carry the full + // "already exists … --overwrite" text; do not wrap them. + if message.contains("already exists") { + FixtureReport::error(message) + } else { + FixtureReport::error(format!("fixture write failed: {message}")) + } + } + } + } + } +} + +/// Classify a [`fixture::build_draft`] error as a skip (unsupported shape) or a +/// fixture construction error (database / other failures). +/// +/// Unsupported shapes are expected in whole-block sweeps and must not fail the +/// run. Endpoint/DB failures during construction mean the requested artifact +/// could not be produced and fail the run as an execution-class fixture error. +/// The builder decides which is which at the point it knows, so rewording any of +/// its messages cannot silently reclassify a sweep. +fn fixture_report_from_build_err(err: fixture::FixtureBuildError) -> FixtureReport { + match err { + fixture::FixtureBuildError::Unsupported(reason) => FixtureReport::skipped(reason), + fixture::FixtureBuildError::Construction(err) => { + FixtureReport::error(format!("fixture construction failed: {err}")) + } + } +} + +/// Fetch the on-chain receipt of every target of a block. +/// +/// Each target maps either to the consensus facts its receipt reports, or to the +/// message explaining why it could not be verified (the endpoint failed the +/// call or pruned the receipt, or the receipt describes a different inclusion +/// than the block being replayed). +async fn fetch_target_receipts

( + provider: &P, + targets: &[B256], + block_hash: B256, +) -> BTreeMap> +where + P: Provider, +{ + let mut receipts = BTreeMap::new(); + for tx_hash in targets { + let fetched = match verify::fetch_receipt(provider, *tx_hash).await { + Ok(receipt) => match verify::check_inclusion(receipt.block_hash(), block_hash) { + Ok(()) => Ok(ReceiptFacts::from_receipt(&receipt.inner)), + Err(message) => Err(message), + }, + // The reported entry already carries the `rpc` kind, so the error's + // own "RPC error" prefix would only repeat it. + Err(ReplayError::RpcError(message)) => Err(message), + Err(e) => Err(e.to_string()), + }; + if let Err(message) = &fetched { + warn!(tx_hash = %tx_hash, %message, "Could not fetch the on-chain receipt"); + } + receipts.insert(*tx_hash, fetched); + } + receipts +} + +/// Fetch a block by number, using the same call shape as the single-transaction path. +async fn fetch_block

(provider: &P, number: u64) -> Result> +where + P: Provider, +{ + provider + .get_block_by_number(number.into()) + .await + .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? + .ok_or(ReplayError::BlockNotFound(number)) +} + +/// Map an error raised while replaying a block onto the kind reported for the +/// target the error is about. +fn classify(err: &ReplayError) -> BatchErrorKind { + match err { + ReplayError::TransactionNotFound(_) => BatchErrorKind::NotFound, + ReplayError::RpcError(_) | + ReplayError::RpcTransportError(_) | + ReplayError::BlockBodyTransactionNull(_) | + ReplayError::BlockBodyTransactionFetch { .. } => BatchErrorKind::Rpc, + // A block error the EVM raised because a state read failed is that + // read's failure: the same classification the run-level exit code uses. + ReplayError::BlockExecutionError(_) + if ExitCode::from_evme_error(err) == ExitCode::RpcFailure => + { + BatchErrorKind::Rpc + } + _ => BatchErrorKind::Execution, + } +} + +/// The kind reported for a target swept up by an abort caused elsewhere. +/// +/// The abort says nothing about this target: whatever class caused the block to +/// stop (unknown hash, RPC failure, executor/setup error on another +/// transaction), a non-aborting swept target is unanswered (`rpc`). Only the +/// transaction that caused the abort keeps its own classified kind (when it is +/// a reported target); otherwise the run tallies the abort class separately so +/// the exit code still reflects the root cause. +/// +/// The error is taken and ignored on purpose: the signature keeps the decision +/// visible at the call site, so a future change that wants to classify by cause +/// has to argue against this rule rather than silently add a parameter. +fn swept_kind(_err: &ReplayError) -> BatchErrorKind { + BatchErrorKind::Rpc +} + +/// The transaction an aborting error is about, when it names one. +fn aborting_tx_hash(err: &ReplayError) -> Option { + match err { + ReplayError::TransactionNotFound(hash) | ReplayError::BlockBodyTransactionNull(hash) => { + Some(*hash) + } + ReplayError::BlockBodyTransactionFetch { tx_hash, .. } => Some(*tx_hash), + ReplayError::BlockExecutionError(err) => block_error_tx_hash(err), + _ => None, + } +} + +/// The transaction a block execution error names, when it carries one. +fn block_error_tx_hash(err: &BlockExecutionError) -> Option { + if let Some(validation) = err.as_validation() { + return match validation { + BlockValidationError::InvalidTx { hash, .. } | + BlockValidationError::EVM { hash, .. } => Some(*hash), + _ => None, + }; + } + err.as_internal()?.as_evm().map(|(hash, _)| *hash) +} + +/// Build a failure entry. +fn failure(tx_hash: B256, kind: BatchErrorKind, message: String) -> BatchEntry { + BatchEntry::Failed(FailedTx { tx_hash, kind, message }) +} + +/// Report the same failure for every target of a block that never started. +fn fail_all( + targets: impl IntoIterator, + kind: BatchErrorKind, + message: &str, +) -> Vec { + targets.into_iter().map(|hash| failure(hash, kind, message.to_string())).collect() +} + +/// Append the same failure for every remaining target, keeping any entries +/// already recorded (for example inclusion mismatches decided earlier). +fn fail_remaining( + targets: &[B256], + mut entries: Vec, + kind: BatchErrorKind, + message: &str, +) -> Vec { + let reported: HashSet = entries.iter().map(BatchEntry::tx_hash).collect(); + for hash in targets { + if !reported.contains(hash) { + entries.push(failure(*hash, kind, message.to_string())); + } + } + entries +} + +/// Write one entry to stdout: a compact NDJSON line, or the human-readable +/// summary used by the single-transaction path. +fn emit(entry: &BatchEntry, json: bool) { + if json { + let line = match entry { + BatchEntry::Executed(tx) => { + let mut summary = + ExecutionSummary::from_result(&tx.exec_result, tx.contract_address); + summary.receipt = + Some(serde_json::to_value(&tx.receipt).expect("failed to serialize receipt")); + summary.verification = tx.verification.as_ref().map(|verification| { + serde_json::to_value(verification).expect("failed to serialize verification") + }); + serde_json::to_string(&BatchResultLine { + tx_hash: tx.tx_hash, + block_number: tx.block_number, + tx_index: tx.tx_index, + summary: &summary, + fixture: tx.fixture.as_ref(), + }) + } + BatchEntry::Failed(tx) => serde_json::to_string(&BatchErrorLine { + tx_hash: tx.tx_hash, + error: BatchErrorBody { kind: tx.kind.as_str(), message: &tx.message }, + }), + }; + println!("{}", line.expect("failed to serialize output")); + return; + } + + match entry { + BatchEntry::Executed(tx) => { + println!(); + println!( + "=== Transaction {} (block {}, index {}) ===", + tx.tx_hash, tx.block_number, tx.tx_index + ); + print_execution_summary(&tx.exec_result, tx.contract_address, tx.exec_time); + print_receipt(&tx.receipt); + if let Some(verification) = &tx.verification { + println!(); + println!("{}", verification.verdict_line()); + } + if let Some(fixture) = &tx.fixture { + println!(); + println!("{}", fixture.human_line()); + } + } + BatchEntry::Failed(tx) => { + println!(); + println!("=== Transaction {} ===", tx.tx_hash); + println!("Error ({}): {}", tx.kind.as_str(), tx.message); + } + } +} + +/// Parse the newline-separated transaction hash list behind `--tx-file`. +/// +/// Blank lines and `#`-prefixed comment lines are ignored. Duplicates are +/// dropped, keeping the first occurrence. +pub(super) fn parse_tx_hash_list(contents: &str) -> Result> { + let mut hashes = Vec::new(); + let mut seen = HashSet::new(); + + for (index, raw_line) in contents.lines().enumerate() { + let line = raw_line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let line_number = index + 1; + let hash = B256::from_str(line).map_err(|e| { + ReplayError::InvalidInput(format!( + "invalid transaction hash on line {line_number}: '{line}' ({e})" + )) + })?; + if seen.insert(hash) { + hashes.push(hash); + } else { + warn!( + tx_hash = %hash, + line = line_number, + "Duplicate transaction hash in --tx-file; replaying it once", + ); + } + } + + Ok(hashes) +} + +/// `clap` value parser for `--block`, accepting decimal or `0x`-prefixed hex. +pub(super) fn parse_block_number(value: &str) -> std::result::Result { + let trimmed = value.trim(); + match trimmed.strip_prefix("0x").or_else(|| trimmed.strip_prefix("0X")) { + Some(hex) => u64::from_str_radix(hex, 16) + .map_err(|e| format!("invalid hex block number '{value}': {e}")), + None => trimmed.parse::().map_err(|e| format!("invalid block number '{value}': {e}")), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::common::ExitCode; + + const HASH_A: &str = "0xde3d56dc739484166b8af1bea757bf7e3e9a4b9a0fb62d722703345570dfc1d6"; + const HASH_B: &str = "0x323ddc8e67dfc134284d78c65f3c1dc7ff45ba1db02eeaf62e211ae3253478ef"; + + /// A verification verdict as a run would have reported it. + fn verdict(matched: bool) -> VerificationOutcome { + VerificationOutcome::compared(matched, None) + } + + /// Build a tally from the outcomes a run would have reported: `failures` + /// error entries, plus `replayed` verified result lines of which + /// `mismatched` diverged from their receipt. + fn tally(failures: &[BatchErrorKind], replayed: usize, mismatched: usize) -> BatchTally { + let mut tally = BatchTally::default(); + for kind in failures { + tally.record(&failure(B256::ZERO, *kind, String::new())); + } + for index in 0..replayed { + tally.record_executed(Some(&verdict(index >= mismatched)), None); + } + tally + } + + /// The exit code a batch run with these outcomes ends with. + fn exit_code(failures: &[BatchErrorKind], replayed: usize, mismatched: usize) -> ExitCode { + tally(failures, replayed, mismatched) + .into_error() + .map_or(ExitCode::Success, |err| ExitCode::from_evme_error(&err)) + } + + /// A clean run has nothing to report and exits 0. + #[test] + fn test_batch_tally_clean_run_has_no_error() { + assert!(tally(&[], 3, 0).into_error().is_none()); + assert_eq!(exit_code(&[], 3, 0), ExitCode::Success); + } + + /// Mixed failures are ranked by class: an execution failure outranks the + /// rest, an RPC failure outranks a mismatch. + #[test] + fn test_batch_tally_failure_precedence() { + use BatchErrorKind::{Execution, NotFound, Pending, Rpc}; + + assert_eq!(exit_code(&[Execution, Rpc], 1, 1), ExitCode::ExecutionError); + assert_eq!(exit_code(&[Rpc, Rpc], 1, 1), ExitCode::RpcFailure); + assert_eq!(exit_code(&[], 2, 1), ExitCode::VerificationMismatch); + // A definitive answer about a target is an execution-class failure. + assert_eq!(exit_code(&[NotFound], 1, 0), ExitCode::ExecutionError); + assert_eq!(exit_code(&[Pending], 1, 0), ExitCode::ExecutionError); + } + + /// The aggregate error carries the counts by class, not a formatted string + /// the exit mapping would have to parse. + #[test] + fn test_batch_tally_aggregate_carries_counts() { + use BatchErrorKind::{Execution, NotFound, Rpc}; + + let err = tally(&[Execution, NotFound, Rpc], 2, 1).into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("infrastructure failures must aggregate: {err:?}"); + }; + assert_eq!( + counts, + BatchFailureCounts { + execution: 2, + rpc: 1, + mismatched: 1, + total: 5, + ..Default::default() + } + ); + assert!( + counts.to_string().contains("3 of 5 target transaction(s) failed"), + "unexpected message: {counts}" + ); + } + + /// A target whose fixture could not be written keeps its result line and + /// its verdict, and still fails the run as an execution-class failure — + /// including when that same target diverged from its on-chain receipt. + #[test] + fn test_batch_tally_counts_a_fixture_failure_and_its_mismatch() { + let mut tally = BatchTally::default(); + tally.record_executed(Some(&verdict(false)), Some(&FixtureReport::error("disk full"))); + + assert_eq!(tally.replayed, 1, "the target replayed"); + assert_eq!(tally.verified, 1, "the target was verified"); + assert_eq!(tally.counts.mismatched, 1, "its divergence is counted"); + assert_eq!(tally.counts.execution, 1, "its failed fixture is counted"); + + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("a failed fixture must fail the run: {err:?}"); + }; + assert_eq!( + counts, + BatchFailureCounts { + execution: 1, + rpc: 0, + mismatched: 1, + total: 1, + ..Default::default() + } + ); + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::ExecutionError); + } + + /// A written or skipped fixture is not a failure. + #[test] + fn test_batch_tally_ignores_written_and_skipped_fixtures() { + let mut tally = BatchTally::default(); + tally.record_executed(None, Some(&FixtureReport::written(Path::new("/tmp/tx.json")))); + tally.record_executed(None, Some(&FixtureReport::skipped("fidelity gate failed"))); + + assert_eq!(tally.counts.execution, 0); + assert!(tally.into_error().is_none(), "fixture skips never fail the run"); + } + + /// Every non-aborting swept target is unanswered (`rpc`), even when the + /// abort itself is an execution-class failure of another transaction. + /// + /// The abort's own class is tallied separately when the aborter is not a + /// target (`record_uncounted_abort`); swept entries stay `rpc`. + #[test] + fn test_swept_kind_always_rpc_regardless_of_abort_class() { + // Unknown hash: already unanswered for the cause, and for swept peers. + assert_eq!(swept_kind(&ReplayError::TransactionNotFound(B256::ZERO)), BatchErrorKind::Rpc); + // Transport/RPC failure. + assert_eq!(swept_kind(&ReplayError::RpcError("endpoint down".into())), BatchErrorKind::Rpc); + // Block-body null is rpc-class for the aborting target and for sweeps. + assert_eq!( + swept_kind(&ReplayError::BlockBodyTransactionNull(B256::ZERO)), + BatchErrorKind::Rpc + ); + // Execution-class aborts (other, setup, internal) must not blame swept targets. + assert_eq!( + swept_kind(&ReplayError::Other("executor setup failed".into())), + BatchErrorKind::Rpc + ); + assert_eq!( + swept_kind(&ReplayError::InvalidInput("bad hardfork schedule".into())), + BatchErrorKind::Rpc + ); + // classify itself still distinguishes execution for the aborting target. + assert_eq!( + classify(&ReplayError::Other("executor setup failed".into())), + BatchErrorKind::Execution + ); + } + + /// A block-body hash resolving to null is an RPC inconsistency that still + /// names the vanished transaction for the abort sweep. + #[test] + fn test_block_body_transaction_null_classifies_as_rpc_and_names_the_hash() { + let hash = B256::repeat_byte(0xab); + let err = ReplayError::BlockBodyTransactionNull(hash); + assert_eq!(classify(&err), BatchErrorKind::Rpc); + assert_eq!(aborting_tx_hash(&err), Some(hash)); + // Contrasts with the user-supplied unknown-hash definitive answer. + assert_eq!(classify(&ReplayError::TransactionNotFound(hash)), BatchErrorKind::NotFound); + assert_eq!(aborting_tx_hash(&ReplayError::TransactionNotFound(hash)), Some(hash)); + } + + /// A block-body fetch failure (transport / cache miss) is rpc-class and + /// names the hash, matching the null-answer pattern. + #[test] + fn test_block_body_transaction_fetch_classifies_as_rpc_and_names_the_hash() { + let hash = B256::repeat_byte(0xcd); + let err = ReplayError::BlockBodyTransactionFetch { + tx_hash: hash, + message: "cache miss in offline replay file".into(), + }; + assert_eq!(classify(&err), BatchErrorKind::Rpc); + assert_eq!(aborting_tx_hash(&err), Some(hash)); + let message = err.to_string(); + assert!(message.contains(&hash.to_string()) || message.contains(&format!("{hash:#x}"))); + assert!(message.contains("fetching it failed"), "unexpected message: {message}"); + assert!(message.contains("cache miss"), "unexpected message: {message}"); + } + + /// An uncounted non-target abort floors the exit class without a synthetic + /// reported entry and without inflating per-target failure totals. + #[test] + fn test_batch_tally_uncounted_abort_drives_exit_class() { + let mut tally = BatchTally::default(); + // Two targets swept as unanswered behind a non-target execution abort. + tally.record(&failure(B256::repeat_byte(0x01), BatchErrorKind::Rpc, "swept".into())); + tally.record(&failure(B256::repeat_byte(0x02), BatchErrorKind::Rpc, "swept".into())); + tally.record_uncounted_abort(BatchErrorKind::Execution); + + assert_eq!(tally.reported, 2, "uncounted abort is not a reported target"); + assert_eq!(tally.counts.rpc, 2, "target counters stay per-target"); + assert_eq!(tally.counts.execution, 0, "abort must not inflate execution count"); + assert_eq!(tally.exit_floor, BatchExitFloor::Execution); + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("expected batch failure: {err:?}"); + }; + assert_eq!( + counts.to_string(), + "2 of 2 target transaction(s) failed (0 execution, 2 rpc)", + "aggregate message must stay truthful about targets" + ); + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::ExecutionError); + } + + /// One unanswered receipt with both `--verify-receipt` and + /// `--dump-fixture-dir` counts as a single rpc failure, while both result + /// fields remain present on the executed entry. + #[test] + fn test_batch_tally_shared_receipt_failure_counted_once() { + let mut tally = BatchTally::default(); + tally.record_executed( + Some(&VerificationOutcome::unavailable("receipt pruned")), + Some(&FixtureReport::rpc_error("no on-chain receipt was fetched for this transaction")), + ); + + assert_eq!(tally.reported, 1); + assert_eq!(tally.replayed, 1); + assert_eq!(tally.verified, 0); + assert_eq!(tally.counts.rpc, 1, "shared receipt failure is one rpc count"); + assert_eq!(tally.counts.execution, 0); + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("expected batch failure: {err:?}"); + }; + assert_eq!(counts.to_string(), "1 of 1 target transaction(s) failed (0 execution, 1 rpc)"); + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::RpcFailure); + } + + /// Independent findings on the same target still both count: a receipt + /// mismatch plus a fixture write failure is not a shared root cause. + #[test] + fn test_batch_tally_mismatch_and_fixture_error_are_independent() { + let mut tally = BatchTally::default(); + tally.record_executed(Some(&verdict(false)), Some(&FixtureReport::error("disk full"))); + + assert_eq!(tally.counts.mismatched, 1); + assert_eq!(tally.counts.execution, 1); + assert_eq!(tally.counts.rpc, 0); + } + + /// Documented stream order: body-index first, then absent targets last in + /// job input order — independent of the order entries were collected. + #[test] + fn test_order_block_entries_body_index_before_absent_last() { + let early = B256::repeat_byte(0x11); + let mid = B256::repeat_byte(0x22); + let late_absent = B256::repeat_byte(0x33); + let job_targets = vec![ + JobTarget { hash: late_absent, inclusion_hash: Some(B256::ZERO) }, + JobTarget { hash: early, inclusion_hash: Some(B256::ZERO) }, + JobTarget { hash: mid, inclusion_hash: Some(B256::ZERO) }, + ]; + // Collected in the buggy pre-execution-first order: absent then results. + let entries = vec![ + failure(late_absent, BatchErrorKind::Rpc, "inclusion".into()), + failure(mid, BatchErrorKind::Rpc, "swept".into()), + failure(early, BatchErrorKind::Rpc, "swept".into()), + ]; + let body = [early, mid, B256::repeat_byte(0x99)]; + let ordered = order_block_entries(entries, &job_targets, Some(&body)); + let hashes: Vec = ordered.iter().map(BatchEntry::tx_hash).collect(); + assert_eq!( + hashes, + vec![early, mid, late_absent], + "body order first, absent last: {hashes:?}" + ); + } + + /// A fixture discarded after a transport abort counts as rpc, not execution, + /// so the run exit matches the abort class. + #[test] + fn test_batch_tally_abort_inherited_fixture_error_is_rpc_class() { + let mut tally = BatchTally::default(); + tally.record_executed( + None, + Some(&FixtureReport::abort_error( + "fixture not written: block aborted: transport", + BatchErrorKind::Rpc, + )), + ); + + assert_eq!(tally.replayed, 1); + assert_eq!(tally.counts.rpc, 1); + assert_eq!(tally.counts.execution, 0); + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("expected batch failure: {err:?}"); + }; + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::RpcFailure); + } + + /// An unanswered receipt for `--verify-receipt` keeps the target as + /// replayed, counts as rpc (not mismatched), and is not "verified". + #[test] + fn test_batch_tally_verification_unavailable_is_rpc_and_still_replayed() { + let mut tally = BatchTally::default(); + tally.record_executed(Some(&VerificationOutcome::unavailable("receipt pruned")), None); + + assert_eq!(tally.replayed, 1, "the target still replayed"); + assert_eq!(tally.verified, 0, "no comparison ran"); + assert_eq!(tally.counts.mismatched, 0, "unverified is not a mismatch"); + assert_eq!(tally.counts.rpc, 1); + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("expected batch failure: {err:?}"); + }; + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::RpcFailure); + } + + /// An unanswered receipt for `--dump-fixture-dir` is a rpc-class fixture + /// error, not a skip that exits 0. + #[test] + fn test_batch_tally_fixture_receipt_unavailable_is_rpc_class() { + let mut tally = BatchTally::default(); + tally.record_executed( + None, + Some(&FixtureReport::rpc_error("no on-chain receipt was fetched for this transaction")), + ); + + assert_eq!(tally.replayed, 1); + assert_eq!(tally.counts.rpc, 1); + assert_eq!(tally.counts.execution, 0); + let err = tally.into_error().expect("run failed"); + let ReplayError::BatchFailed(counts) = err else { + panic!("expected batch failure: {err:?}"); + }; + assert_eq!(ExitCode::from_batch_failures(&counts), ExitCode::RpcFailure); + } + + /// A run whose only finding is divergence fails as the mismatch it is. + #[test] + fn test_batch_tally_mismatch_only_reports_the_verification_error() { + let err = tally(&[], 4, 2).into_error().expect("run failed"); + assert!( + matches!(err, ReplayError::VerificationMismatch { mismatched: 2, total: 4 }), + "unexpected error: {err:?}" + ); + } + + #[test] + fn test_parse_tx_hash_list_skips_blanks_and_comments() { + let contents = + format!("# leading comment\n\n{HASH_A}\n \n # indented comment\n\t{HASH_B} \n\n"); + + let hashes = parse_tx_hash_list(&contents).expect("should parse"); + + assert_eq!(hashes, vec![B256::from_str(HASH_A).unwrap(), B256::from_str(HASH_B).unwrap()]); + } + + #[test] + fn test_parse_tx_hash_list_deduplicates_preserving_order() { + let contents = format!("{HASH_B}\n{HASH_A}\n{HASH_B}\n"); + + let hashes = parse_tx_hash_list(&contents).expect("should parse"); + + assert_eq!(hashes, vec![B256::from_str(HASH_B).unwrap(), B256::from_str(HASH_A).unwrap()]); + } + + #[test] + fn test_parse_tx_hash_list_reports_offending_line_number() { + let contents = format!("# comment\n\n{HASH_A}\nnot-a-hash\n"); + + let err = parse_tx_hash_list(&contents).expect_err("should reject the invalid hash"); + + let message = err.to_string(); + assert!(message.contains("line 4"), "error should name the line, got: {message}"); + assert!(message.contains("not-a-hash"), "error should quote the line, got: {message}"); + } + + #[test] + fn test_parse_tx_hash_list_accepts_empty_input() { + let hashes = parse_tx_hash_list("# only a comment\n\n").expect("should parse"); + assert!(hashes.is_empty()); + } + + #[test] + fn test_parse_block_number_decimal() { + assert_eq!(parse_block_number("22945844"), Ok(22_945_844)); + assert_eq!(parse_block_number(" 22945844 "), Ok(22_945_844)); + assert_eq!(parse_block_number("0"), Ok(0)); + } + + #[test] + fn test_parse_block_number_hex() { + assert_eq!(parse_block_number("0x15e2034"), Ok(22_945_844)); + assert_eq!(parse_block_number("0X15E2034"), Ok(22_945_844)); + } + + #[test] + fn test_parse_block_number_rejects_garbage() { + assert!(parse_block_number("").is_err()); + assert!(parse_block_number("0x").is_err()); + assert!(parse_block_number("0xzz").is_err()); + assert!(parse_block_number("-1").is_err()); + assert!(parse_block_number("12.5").is_err()); + } + + /// Unsupported shapes remain skips; construction failures become fixture + /// errors so the run exits non-zero. + /// + /// Which builder rejection lands in which variant is decided inside + /// `build_draft` and pinned by the integration tests that drive the real + /// builder (deposit skip, injected pre-state failure); this test only pins + /// the variant-to-report mapping, which no rewording can move. + #[test] + fn test_fixture_build_err_classifies_skips_vs_construction_errors() { + let unsupported = fixture_report_from_build_err(fixture::FixtureBuildError::Unsupported( + "--dump-fixture does not support deposit transactions".into(), + )); + assert!(unsupported.skipped.is_some(), "unsupported shape is a skip: {unsupported:?}"); + assert!(unsupported.error.is_none()); + assert_eq!( + unsupported.skipped.as_deref(), + Some("--dump-fixture does not support deposit transactions"), + "the builder's reason is reported verbatim" + ); + + let construction = fixture_report_from_build_err(fixture::FixtureBuildError::Construction( + ReplayError::Other( + "pre-state read for 0x00000000000000000000000000000000000000aa: \ + database unavailable" + .into(), + ), + )); + assert!( + construction.error.as_ref().is_some_and(|m| m.contains("construction failed")), + "construction failure is a fixture error: {construction:?}" + ); + assert!(construction.skipped.is_none()); + } + + /// A pre-decided fixture report is never rewritten by materialization, so a + /// finish failure that drops a Ready draft (without calling materialize) + /// cannot leave a file and a Report never touches the filesystem. + #[test] + fn test_materialize_deferred_fixture_passes_reports_through() { + let skipped = materialize_deferred_fixture(DeferredFixture::Report( + FixtureReport::skipped("fidelity gate failed: gas_used"), + )); + assert_eq!(skipped.skipped.as_deref(), Some("fidelity gate failed: gas_used")); + assert!(skipped.path.is_none()); + assert!(skipped.error.is_none()); + + let err = materialize_deferred_fixture(DeferredFixture::Report(FixtureReport::error( + "fixture construction failed: code fetch failed", + ))); + assert!(err.error.as_ref().is_some_and(|m| m.contains("construction failed"))); + assert!(err.path.is_none()); + + let rpc = materialize_deferred_fixture(DeferredFixture::Report(FixtureReport::rpc_error( + "no on-chain receipt was fetched for this transaction", + ))); + assert!(rpc.is_error()); + assert_eq!(rpc.error_kind, BatchErrorKind::Rpc); + assert!(rpc.skipped.is_none()); + } +} diff --git a/bin/mega-evme/src/replay/cmd.rs b/bin/mega-evme/src/replay/cmd.rs index 571d7714..95c7b572 100644 --- a/bin/mega-evme/src/replay/cmd.rs +++ b/bin/mega-evme/src/replay/cmd.rs @@ -1,10 +1,10 @@ -use std::{str::FromStr, time::Instant}; +use std::{path::PathBuf, str::FromStr, time::Instant}; use alloy_consensus::{BlockHeader, Transaction as _}; use alloy_primitives::{B256, U256}; use alloy_provider::Provider; use alloy_rpc_types_eth::Block; -use clap::Parser; +use clap::{ArgGroup, Parser}; use mega_evm::{ alloy_evm::{block::BlockExecutor, Evm, EvmEnv}, alloy_op_evm::block::OpAlloyReceiptBuilder, @@ -14,10 +14,10 @@ use mega_evm::{ primitives::eip4844, DatabaseRef, }, - BlockLimits, EvmTxRuntimeLimits, MegaBlockExecutionCtx, MegaBlockExecutorFactory, - MegaEvmFactory, MegaHardforks, MegaSpecId, + BlockLimits, MegaBlockExecutionCtx, MegaBlockExecutorFactory, MegaEvmFactory, MegaHardforks, + MegaSpecId, }; -use tracing::{debug, info, trace, warn}; +use tracing::{debug, error, info, trace, warn}; use alloy_network::ReceiptResponse; use op_alloy_rpc_types::Transaction; @@ -26,20 +26,44 @@ use crate::{ common::{ op_receipt_to_tx_receipt, parse_bucket_capacity, print_execution_summary, print_execution_trace, print_receipt, BuildProviderOutput, EvmeExternalEnvs, EvmeOutcome, - ExecutionSummary, ExternalEnvSnapshot, OpTxReceipt, RpcCacheStore, TxOverrideArgs, + ExecutionSummary, ExternalEnvSnapshot, OpTxReceipt, RpcArgs, RpcCacheStore, TracerType, + TxOverrideArgs, }, - replay::get_hardfork_config, + replay::{get_hardfork_config, ReplayHardforks}, run, ChainArgs, EvmeState, }; -use super::{ReplayError, Result}; +use super::{ + batch, + verify::{self, VerificationOutcome}, + ReplayError, Result, +}; /// Replay a transaction from RPC #[derive(Parser, Debug)] +#[command(group( + ArgGroup::new("replay_target").required(true).args(["tx_hash", "tx_file", "block"]) +))] pub struct Cmd { /// Transaction hash to replay #[arg(value_name = "TX_HASH")] - pub tx_hash: B256, + pub tx_hash: Option, + + /// Replay every transaction hash listed in the given file, one per line. + /// + /// Blank lines and `#`-prefixed comment lines are ignored, and duplicates are + /// replayed once. All hashes are replayed in a single process: transactions + /// are grouped by their containing block and each block is executed once. + /// Batch mode does not support `--dump-fixture` (use `--dump-fixture-dir`), + /// transaction overrides, `--override.spec`, tracing, or state dumps. + #[arg(long = "tx-file", value_name = "PATH")] + pub tx_file: Option, + + /// Replay every transaction of the given block (decimal or `0x`-prefixed hex). + /// + /// Same batch semantics and restrictions as `--tx-file`. + #[arg(long = "block", value_name = "N", value_parser = batch::parse_block_number)] + pub block: Option, /// RPC configuration #[command(flatten)] @@ -79,8 +103,40 @@ pub struct Cmd { /// replay, and `state-test --bench` benchmarks it. The dump is rejected /// unless the local replay reproduces the on-chain receipt's gas and success /// status. Incompatible with transaction overrides and `--override.spec`. + /// Single-transaction only; for batch mode use `--dump-fixture-dir`. #[arg(long = "dump-fixture", value_name = "FILE")] pub dump_fixture: Option, + + /// Dump a self-validating EEST state-test fixture for every successfully + /// replayed target into `

/.json`. + /// + /// Batch mode only (`--tx-file` / `--block`). Per-target gating mirrors the + /// single-transaction dump: fidelity-gate failures and BLOCKHASH readers are + /// skipped with a recorded reason instead of failing the run; pending or + /// unresolvable targets stay error entries. Existing files are refused unless + /// `--overwrite` is set. Registration into `bench/replay/manifest.json` is + /// not performed — corpus curation stays manual. + #[arg(long = "dump-fixture-dir", value_name = "DIR")] + pub dump_fixture_dir: Option, + + /// Replace existing files when writing fixtures with `--dump-fixture-dir`. + /// + /// Without this flag, a target whose `/.json` already exists is + /// reported as an infrastructure error for that target. + #[arg(long = "overwrite")] + pub overwrite: bool, + + /// Verify every replayed transaction against its on-chain receipt. + /// + /// Fetches the receipt of each target and compares the success status, the + /// gas used, and the emitted logs (count plus each log's address, topics, + /// and data). The verdict is reported per transaction, and a mismatch makes + /// the run exit non-zero. A target whose receipt cannot be fetched, or whose + /// receipt describes a different inclusion than the replayed block, is + /// reported as an infrastructure failure rather than a mismatch. Supported + /// in both single-transaction and batch mode. + #[arg(long = "verify-receipt")] + pub verify_receipt: bool, } /// Resolved provider and associated metadata from `--rpc` / `--rpc.capture-file` / @@ -100,10 +156,13 @@ pub(super) struct ReplayOutcome { pub receipt: OpTxReceipt, /// Self-validating fixture draft, present iff `--dump-fixture` was given. pub fixture: Option, + /// On-chain receipt verdict, present iff `--verify-receipt` was given. + pub verification: Option, } /// Intermediate context fetched from RPC before execution. struct ReplayContext { + tx_hash: B256, target_tx: Transaction, parent_block: Block, block: Block, @@ -111,13 +170,63 @@ struct ReplayContext { preceding_tx_hashes: Vec, } +/// What the command was asked to replay, resolved from the target argument group. +enum ReplayMode { + /// The single transaction named by the positional `TX_HASH`. + Single(B256), + /// Many transactions replayed in one process (`--tx-file` / `--block`). + Batch(batch::BatchMode), +} + impl Cmd { - /// Replay a historical transaction. + /// Replay one or more historical transactions. pub async fn run(&self) -> Result<()> { - // Pure input validation — reject before any network/state work. A dumped - // fixture must represent the on-chain transaction, so it can neither apply - // transaction overrides nor force a spec: both would make the recorded - // execution a what-if, not the on-chain one. + self.validate()?; + let mode = self.resolve_mode()?; + + let mut pctx = self.resolve_provider().await?; + + // Execute, report, and (for --dump-fixture) finalize/write — but defer + // error propagation until the cache store has persisted: in capture mode + // an execution or dump-gate failure is exactly the case you'd want to + // debug offline, so the captured RPC responses must not be discarded. + let run_result = match &mode { + ReplayMode::Single(tx_hash) => self.run_single(&mut pctx, *tx_hash).await, + ReplayMode::Batch(batch_mode) => self.run_batch(&mut pctx, batch_mode).await, + }; + + let persist_result = pctx.cache_store.persist(); + match run_result { + Ok(()) => Ok(persist_result?), + Err(run_err) => { + // The run error is the root cause and keeps the exit code, so + // the persist failure is not propagated. It is still reported on + // stderr the way the central reporter reports a failure — a + // capture file that never reached disk must not be silent just + // because the run it captured also failed. + if let Err(persist_err) = persist_result { + error!( + error = %persist_err, + "Failed to persist RPC cache while handling an earlier error", + ); + eprintln!("error: {persist_err}"); + } + Err(run_err) + } + } + } + + /// Pure input validation — reject before any network/state work. + fn validate(&self) -> Result<()> { + if self.is_batch() { + self.validate_batch_args()?; + } else { + self.validate_single_args()?; + } + + // A dumped fixture must represent the on-chain transaction, so it can + // neither apply transaction overrides nor force a spec: both would make + // the recorded execution a what-if, not the on-chain one. if self.dump_fixture.is_some() { if self.tx_override_args.has_overrides() { return Err(ReplayError::Other( @@ -136,38 +245,178 @@ impl Cmd { } } - let mut pctx = self.resolve_provider().await?; - let rctx = self.fetch_replay_context(&pctx.provider, pctx.chain_id).await?; - let (external_envs, env_snapshot) = self.resolve_external_envs(&pctx)?; + if self.dump_fixture.is_some() && self.dump_fixture_dir.is_some() { + return Err(ReplayError::Other( + "--dump-fixture and --dump-fixture-dir are mutually exclusive: dump one \ + transaction with --dump-fixture, or a batch with --dump-fixture-dir" + .to_string(), + )); + } - // Execute, report, and (for --dump-fixture) finalize/write — but defer - // error propagation until the cache store has persisted: in capture mode - // an execution or dump-gate failure is exactly the case you'd want to - // debug offline, so the captured RPC responses must not be discarded. - let run_result = self.execute_and_report(&pctx.provider, &rctx, external_envs).await; + Ok(()) + } - // Hand the effective external-env snapshot to the store before the final - // persist; no-op unless this is a fixture-capture store. - if let Some(snapshot) = env_snapshot { - pctx.cache_store.set_external_env(snapshot); + /// Reject batch-only flags in single-transaction mode. + fn validate_single_args(&self) -> Result<()> { + if self.dump_fixture_dir.is_some() { + return Err(ReplayError::Other( + "--dump-fixture-dir is only supported by batch replay (--tx-file / --block); \ + dump a single transaction with --dump-fixture " + .to_string(), + )); } - let persist_result = pctx.cache_store.persist(); - match run_result { - Ok(()) => Ok(persist_result?), - Err(run_err) => { - // Surface the original error; a persist failure on top of it is - // logged, not propagated, so it cannot mask the root cause. - if let Err(persist_err) = persist_result { - warn!( - error = %persist_err, - "Failed to persist RPC cache while handling an earlier error", - ); - } - Err(run_err) + Ok(()) + } + + /// The spec forced by `--override.spec`, parsed. + fn resolve_spec_override(&self) -> Result> { + self.spec_override + .as_deref() + .map(|spec| { + MegaSpecId::from_str(spec) + .map_err(|e| ReplayError::Other(format!("Invalid spec: {e:?}"))) + }) + .transpose() + } + + /// Whether this invocation selects a batch of transactions. + fn is_batch(&self) -> bool { + self.tx_file.is_some() || self.block.is_some() + } + + /// Reject the single-transaction-only flags in batch mode. + /// + /// Batch mode reports one summary per transaction; single-file fixture dumps, + /// tracing, state dumps, and what-if knobs (overrides, forced spec) have no + /// meaningful batch semantics, so they are rejected up front rather than + /// silently ignored. Per-target fixture sedimentation uses + /// `--dump-fixture-dir` instead of `--dump-fixture`. + fn validate_batch_args(&self) -> Result<()> { + const MODE: &str = "batch replay (--tx-file / --block)"; + + // Genesis has no transactions and no parent to fork from. Without this + // the run would collect zero targets, emit nothing, and exit 0 — the + // block-0 rejection raised during replay never reaching the user. + if self.block == Some(0) { + return Err(ReplayError::Other( + "--block 0 cannot be replayed: the genesis block has no transactions \ + and no parent block to fork from" + .to_string(), + )); + } + if self.dump_fixture.is_some() { + return Err(ReplayError::Other(format!( + "--dump-fixture is not supported by {MODE}; dump fixtures for a batch \ + with --dump-fixture-dir " + ))); + } + if self.tx_override_args.has_overrides() { + return Err(ReplayError::Other(format!( + "transaction overrides (--override.gas-limit / --override.value / \ + --override.input / --override.input-file) are not supported by {MODE}" + ))); + } + if self.spec_override.is_some() { + return Err(ReplayError::Other(format!( + "--override.spec is not supported by {MODE}; each block's spec is \ + auto-detected from its timestamp" + ))); + } + if has_trace_args(&self.trace_args) { + return Err(ReplayError::Other(format!( + "trace options (--trace / --trace.output / --tracer / --trace.*) are not \ + supported by {MODE}" + ))); + } + if has_dump_args(&self.dump_args) { + return Err(ReplayError::Other(format!( + "state dump options (--dump / --dump.output) are not supported by {MODE}" + ))); + } + + Ok(()) + } + + /// Resolve the target argument group into an execution mode. + /// + /// Reads `--tx-file` from disk here so a malformed list fails before any + /// provider is built. + fn resolve_mode(&self) -> Result { + if let Some(path) = &self.tx_file { + let contents = std::fs::read_to_string(path).map_err(|e| { + ReplayError::InvalidInput(format!( + "Failed to read --tx-file '{}': {e}", + path.display() + )) + })?; + let hashes = batch::parse_tx_hash_list(&contents)?; + if hashes.is_empty() { + return Err(ReplayError::InvalidInput(format!( + "--tx-file '{}' contains no transaction hashes", + path.display() + ))); } + return Ok(ReplayMode::Batch(batch::BatchMode::TxList(hashes))); + } + if let Some(number) = self.block { + return Ok(ReplayMode::Batch(batch::BatchMode::Block(number))); + } + match self.tx_hash { + Some(tx_hash) => Ok(ReplayMode::Single(tx_hash)), + // Unreachable through clap (the target group is required), but the + // library API can construct `Cmd` directly. + None => Err(ReplayError::InvalidInput( + "'mega-evme replay' requires a TX_HASH, '--tx-file ', or '--block '" + .to_string(), + )), } } + /// Replay the single transaction named by the positional argument. + async fn run_single(&self, pctx: &mut ProviderContext, tx_hash: B256) -> Result<()> { + let rctx = self.fetch_replay_context(&pctx.provider, tx_hash, pctx.chain_id).await?; + // A pending transaction has no receipt to verify against; fail clearly + // instead of replaying it and then surfacing the receipt lookup's + // confusing "transaction is unknown to the endpoint". + if self.verify_receipt && rctx.target_tx.block_number.is_none() { + return Err(ReplayError::Other( + "--verify-receipt does not support pending transactions: the comparison needs \ + the on-chain receipt, which does not exist yet" + .to_string(), + )); + } + let external_envs = self.apply_external_envs(pctx)?; + self.execute_and_report(&pctx.provider, &rctx, external_envs).await + } + + /// Replay a batch of transactions through the shared per-block driver. + async fn run_batch(&self, pctx: &mut ProviderContext, mode: &batch::BatchMode) -> Result<()> { + let external_envs = self.apply_external_envs(pctx)?; + batch::run( + &pctx.provider, + pctx.chain_id, + mode, + external_envs, + batch::ReportArgs { + json: self.output_args.json, + verify_receipt: self.verify_receipt, + dump_fixture_dir: self.dump_fixture_dir.clone(), + overwrite: self.overwrite, + }, + ) + .await + } + + /// Resolve the external environment and hand the capture snapshot to the + /// cache store, which persists it on exit. + fn apply_external_envs(&self, pctx: &mut ProviderContext) -> Result { + let (external_envs, env_snapshot) = self.resolve_external_envs(pctx)?; + if let Some(snapshot) = env_snapshot { + pctx.cache_store.set_external_env(snapshot); + } + Ok(external_envs) + } + /// Execute the replay, print the results, and (for `--dump-fixture`) /// finalize and write the fixture. /// @@ -183,13 +432,22 @@ impl Cmd { P: Provider + Clone + std::fmt::Debug, { let result = self.execute(provider, rctx, external_envs).await?; + // Read the verdict before `result.fixture` is moved below; the mismatch + // is reported after every artifact has been written, so a failing + // verification never costs the user the output it was derived from. + let mismatched = result.verification.as_ref().is_some_and(|v| !v.matched); self.output_results(&result)?; // Write the self-validating fixture (re-executes the isolated unit through // state-test and cross-checks it against the replay before writing). if let (Some(path), Some(draft)) = (&self.dump_fixture, result.fixture) { - super::fixture::finalize_and_write(draft, path)?; + // Single-file `--dump-fixture` always replaces the destination path + // (there is no `--overwrite` gate on this form). + super::fixture::finalize_and_write(draft, path, true)?; info!(path = %path.display(), "Wrote self-validating fixture"); } + if mismatched { + return Err(ReplayError::VerificationMismatch { mismatched: 1, total: 1 }); + } Ok(()) } @@ -211,7 +469,11 @@ impl Cmd { self.rpc_args.build_replay_provider().await? } else if let Some(rpc) = &self.rpc_args.rpc_url { info!(rpc = %rpc, "Provider mode: online RPC"); - self.rpc_args.build_provider().await? + if self.is_batch() { + self.batch_rpc_args().build_provider().await? + } else { + self.rpc_args.build_provider().await? + } } else { return Err(ReplayError::Other( "'mega-evme replay' requires '--rpc ', '--rpc.capture-file ', \ @@ -224,27 +486,98 @@ impl Cmd { Ok(ProviderContext { provider, cache_store, external_env, chain_id }) } + /// The RPC args a batch run actually uses: on-disk cache persistence is opt-in for batch. + /// + /// A batch scan walks linear history — its request keys are block-scoped and essentially + /// never repeat across runs, so a shared cache file buys almost no hits, while its + /// clean-exit persist re-reads, merges, and rewrites the whole file under a cross-process + /// lock. That exit tail grows linearly with the file and serializes across concurrent + /// batch processes sharing the default cache directory, so batch mode keeps the disk + /// cache off unless the invocation says otherwise. The in-memory LRU (and its + /// intra-run reuse across a block's transactions) is unaffected. + /// + /// Two flags say otherwise. `--rpc.cache-dir` names the file to use, and `--rpc.clear-cache` + /// asks for that file to be deleted — a request that only means something while the disk + /// cache is engaged, so forcing the cache off would silently swallow it and leave the + /// polluted file in place for the next run. Either flag therefore opts the batch run back + /// into the disk cache. An explicit `--rpc.no-cache-file` still wins over both, keeping the + /// same meaning it has outside batch mode. + fn batch_rpc_args(&self) -> RpcArgs { + let mut args = self.rpc_args.clone(); + if args.cache_dir.is_none() && !args.clear_cache && !args.no_cache_file { + info!( + "Batch replay leaves the on-disk RPC cache disabled; \ + pass --rpc.cache-dir to enable it" + ); + args.no_cache_file = true; + } + args + } + /// Fetch the transaction, its block, and preceding transaction hashes from the provider. - async fn fetch_replay_context

(&self, provider: &P, chain_id: u64) -> Result + async fn fetch_replay_context

( + &self, + provider: &P, + tx_hash: B256, + chain_id: u64, + ) -> Result where P: Provider, { - info!(tx_hash = %self.tx_hash, "Fetching transaction"); + info!(tx_hash = %tx_hash, "Fetching transaction"); + // The user supplied this hash and nothing the endpoint served so far + // claims it exists, so `Ok(None)` is a definitive "unknown transaction" + // rather than an inconsistency — unlike the block-body-derived lookups + // further down, which the endpoint has already vouched for. let target_tx = provider - .get_transaction_by_hash(self.tx_hash) + .get_transaction_by_hash(tx_hash) .await .map_err(|e| ReplayError::RpcError(format!("Failed to fetch transaction: {e}")))? - .ok_or_else(|| ReplayError::TransactionNotFound(self.tx_hash))?; + .ok_or_else(|| ReplayError::TransactionNotFound(tx_hash))?; + // Authenticate before trusting anything in the answer: the inclusion + // metadata read next, and the execution below, must describe the + // requested transaction rather than whatever the endpoint served. + verify::authenticate_transaction(&target_tx, tx_hash).map_err(ReplayError::RpcError)?; debug!(block_number = ?target_tx.block_number, "Transaction found"); - let (state_base_block, block_number, is_pending) = if let Some(n) = target_tx.block_number { - (n - 1, n, false) + // Classify the target from its `(block_number, block_hash)` pair before + // anything else is fetched. Every shape the endpoint can return is + // handled explicitly, so a contradictory row cannot fall through into the + // pending arm, and a shape that can never be replayed is answered from the + // metadata alone — no block fetch precedes the verdict, and no fetch + // failure can mask it. A mined target keeps its inclusion hash here, which + // is what the block fetched below is anchored against. + let mined: Option<(u64, B256)> = match (target_tx.block_number, target_tx.block_hash) { + (Some(number), Some(inclusion)) => Some((number, inclusion)), + // A mined transaction without an inclusion hash is an unanchored + // view: the block number alone cannot prove which block body the + // target belongs to, so there is nothing to anchor the replay to. + (Some(number), None) => { + return Err(ReplayError::RpcError(format!( + "endpoint reported a mined transaction in block {number} without an \ + inclusion hash: unanchored view" + ))) + } + // A hash proves inclusion; a null number denies it. That pair is + // self-contradictory metadata, not a pending transaction. + (None, Some(inclusion)) => { + return Err(ReplayError::RpcError(format!( + "endpoint reported inclusion hash {inclusion} without a block \ + number: contradictory metadata" + ))) + } + (None, None) => None, + }; + let is_pending = mined.is_none(); + + let (state_base_block, block_number) = if let Some((n, _)) = mined { + (n - 1, n) } else { let latest = provider .get_block_number() .await .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))?; - (latest, latest, true) + (latest, latest) }; debug!( state_base_block = state_base_block, @@ -253,30 +586,120 @@ impl Cmd { "Block numbers determined", ); - let parent_block = provider - .get_block_by_number(state_base_block.into()) - .await - .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? - .ok_or(ReplayError::BlockNotFound(state_base_block))?; + // A mined target forks from its parent block, which is a different block + // than the one it is replayed in. A pending target forks from the latest + // block, which *is* the block it is replayed in: fetching that one height + // twice lets a reorg or a load-balanced endpoint answer the two roles + // from different blocks, and the replay would then run a pre-state from + // one view under a block environment from another. The pending path + // therefore fetches once and uses the same block for both roles, so the + // two roles cannot disagree at all. + // Both heights below come from the endpoint's own answers — the target's + // inclusion metadata for a mined transaction, the reported latest height + // for a pending one — so a null block is the endpoint contradicting + // itself (a reorg in progress, or a load-balanced endpoint serving + // divergent views), not a definitive "unknown block". `BlockNotFound` + // (exit 1) stays reserved for user-supplied heights, where the null is + // the answer; here the same context is an infrastructure failure + // (exit 3), matching the batch path's classification. + let missing_resolved_block = |number: u64| { + ReplayError::RpcError(format!( + "endpoint did not serve block {number}, which it itself resolved (the target's \ + inclusion metadata, or its reported latest height): the endpoint served \ + divergent views (reorg in progress, or a load-balanced endpoint); retry once \ + the chain settles" + )) + }; + let parent_block = if is_pending { + None + } else { + Some( + provider + .get_block_by_number(state_base_block.into()) + .await + .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? + .ok_or_else(|| missing_resolved_block(state_base_block))?, + ) + }; let block = provider .get_block_by_number(block_number.into()) .await .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? - .ok_or(ReplayError::BlockNotFound(block_number))?; + .ok_or_else(|| missing_resolved_block(block_number))?; + let parent_block = parent_block.unwrap_or_else(|| block.clone()); + + // Parent/block linkage guard: the two blocks above were fetched by + // number in separate calls, so across a reorg or a load-balanced + // endpoint serving divergent views `eth_getBlockByNumber(N-1)` can + // return a block that is not the parent of the block being replayed. + // Forking from that state would silently execute against the wrong + // pre-state, and the divergence would surface later as a receipt + // mismatch rather than as the infrastructure failure it is. + // + // A pending transaction has no such pair: its state base *is* the + // latest block, so one fetch fills both roles and there is no linkage, + // no inclusion, and no membership to check. + if let Some((_, reported)) = mined { + let parent_hash = parent_block.hash(); + let expected_parent = block.header.parent_hash(); + if parent_hash != expected_parent { + return Err(ReplayError::RpcError(format!( + "parent block hash {parent_hash} != block parent_hash {expected_parent}: the \ + parent block describes a different chain than the block being replayed (reorg \ + in progress, or a load-balanced endpoint serving divergent views); retry once \ + the chain settles" + ))); + } + + // Inclusion guard: the linkage above only proves the two fetched + // blocks belong to one chain, not that the target belongs to them. + // The lookup that resolved the target reported which block includes + // it, in a separate call, so a reorg or a load-balanced endpoint can + // answer both numbered fetches from a replacement block the target is + // not part of. Replaying that block anyway executes the target + // against a body it never ran in. The reported hash is the one the + // metadata classification kept, so a mined target always has one to + // anchor against. + let fetched = block.hash(); + if reported != fetched { + return Err(ReplayError::RpcError(format!( + "block {block_number} has hash {fetched}, but the target transaction was \ + resolved as included in {reported}: the endpoint served divergent views of \ + this block (reorg in progress, or a load-balanced endpoint); retry once the \ + chain settles" + ))); + } + } + // The preceding transactions are the block-body entries ahead of the + // target, so the target's own position in that body defines the set and + // the body must contain it. A body that does not list the target would + // silently make every transaction of the block count as preceding and + // execute the target after the whole block. let mut preceding_tx_hashes = vec![]; if !is_pending { + let mut found = false; for hash in block.transactions.hashes() { - if hash == self.tx_hash { + if hash == tx_hash { + found = true; break; } preceding_tx_hashes.push(hash); } + if !found { + return Err(ReplayError::RpcError(format!( + "block {block_number} ({}) does not list target transaction {tx_hash}, which \ + the endpoint resolved as included in it: the endpoint served divergent views \ + of this block (reorg in progress, or a load-balanced endpoint); retry once \ + the chain settles", + block.hash(), + ))); + } } debug!(chain_id, preceding_count = preceding_tx_hashes.len(), "Replay context ready"); - Ok(ReplayContext { target_tx, parent_block, block, chain_id, preceding_tx_hashes }) + Ok(ReplayContext { tx_hash, target_tx, parent_block, block, chain_id, preceding_tx_hashes }) } /// Build the external environment and (for capture mode) the envelope snapshot. @@ -350,7 +773,16 @@ impl Cmd { where P: Provider + Clone + std::fmt::Debug, { - let hardforks = get_hardfork_config(ctx.chain_id); + // `--override.spec` replaces the whole execution world, not just the EVM semantics: the + // synthesized schedule drives the pre-block predeploys and the block-level limits too, so + // the replay is a coherent what-if rather than a mix of the historical setup with forced + // semantics. + let chain_hardforks = get_hardfork_config(ctx.chain_id); + let spec_override = self.resolve_spec_override()?; + if let Some(spec_override) = spec_override { + info!(spec_override = %spec_override, "Overriding EVM spec"); + } + let hardforks = ReplayHardforks::resolve(&chain_hardforks, spec_override); let spec = hardforks.spec_id(ctx.block.header.timestamp()); let chain_args = ChainArgs { chain_id: ctx.chain_id, spec: spec.to_string() }; debug!(chain_id = ctx.chain_id, spec = %spec, "Chain configuration"); @@ -366,7 +798,7 @@ impl Cmd { let block_env = retrieve_block_env(&ctx.block)?; trace!(?block_env, "Block environment built"); - let mut evm_env = EvmEnv::new(chain_args.create_cfg_env()?, block_env); + let evm_env = EvmEnv::new(chain_args.create_cfg_env()?, block_env); // For `--dump-fixture`, snapshot the two inputs a fixture // needs before the external env is moved into the factory: the effective @@ -381,8 +813,9 @@ impl Cmd { // self-validation alone cannot catch. let fixture_inputs = if self.dump_fixture.is_some() { // A pending transaction has no receipt yet, so the fidelity gate cannot - // run; fail clearly instead of surfacing the receipt lookup's confusing - // `TransactionNotFound`. + // run; fail clearly here, where the missing receipt is a definitive + // property of the target, instead of surfacing the receipt lookup's + // infrastructure-class failure, which would invite a pointless retry. if ctx.target_tx.block_number.is_none() { return Err(ReplayError::Other( "--dump-fixture does not support pending transactions: the fidelity \ @@ -399,28 +832,22 @@ impl Cmd { let mut oracle_storage = external_envs.oracle_storage(); oracle_storage.sort_unstable(); let mega_env = state_test::types::MegaEnv { bucket_capacities, oracle_storage }; - let receipt = provider - .get_transaction_receipt(self.tx_hash) - .await - .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? - .ok_or(ReplayError::TransactionNotFound(self.tx_hash))?; + // Fetched through the same helper `--verify-receipt` uses, so both + // modes classify an unserved receipt identically. A null answer for a + // target this run has already resolved as mined is the endpoint + // failing to answer — a pruned receipt, or a backend serving a + // divergent view — not a definitive statement that the transaction + // does not exist, so it is a retryable infrastructure failure rather + // than an execution verdict. + let receipt = verify::fetch_receipt(provider, ctx.tx_hash).await?; // Anchor the receipt to the replayed block: across a reorg or a // load-balanced endpoint serving divergent views, the receipt can - // describe a different inclusion than the block fetched earlier, - // and the fidelity gate would then compare the replay against the - // wrong on-chain execution. - if let Some(receipt_block_hash) = receipt.block_hash() { - let replayed_block_hash = ctx.block.hash(); - if receipt_block_hash != replayed_block_hash { - return Err(ReplayError::Other(format!( - "receipt block hash {receipt_block_hash} != replayed block hash \ - {replayed_block_hash}: the receipt describes a different inclusion \ - than the fetched block (reorg in progress, or a load-balanced \ - endpoint serving divergent views); retry the dump once the chain \ - settles" - ))); - } - } + // describe a different inclusion than the block fetched earlier + // (including a receipt with `blockHash: null`). Same check and same + // failure class as `--verify-receipt`, so dump and verify agree on + // unanchored receipts. + verify::check_inclusion(receipt.block_hash(), ctx.block.hash()) + .map_err(ReplayError::RpcError)?; // RLP-hash the receipt's logs with the same helper the state-test // runner uses for `logsRoot`, so the dump can check the replay's logs // against the chain (the rich RPC logs' `inner` is the consensus log). @@ -436,13 +863,31 @@ impl Cmd { None }; + // For `--verify-receipt`, fetch the on-chain receipt here — before the + // executor borrows the database, and with the same call shape the + // fixture path uses — so `--rpc.capture-file` records it and a later + // offline run verifies without network access. It is compared against + // the replay's own receipt once the block is finished. + let onchain_receipt = if self.verify_receipt { + let receipt = verify::fetch_receipt(provider, ctx.tx_hash).await?; + // A receipt describing a different inclusion than the replayed block + // would compare the replay against the wrong on-chain execution: + // that is an infrastructure failure, not a verification mismatch. + verify::check_inclusion(receipt.block_hash(), ctx.block.hash()) + .map_err(ReplayError::RpcError)?; + Some(receipt) + } else { + None + }; + let evm_factory = MegaEvmFactory::new().with_external_env_factory(external_envs); - let block_executor_factory = MegaBlockExecutorFactory::new( - &hardforks, - evm_factory, - OpAlloyReceiptBuilder::default(), - ); - let mut block_limits = BlockLimits::from_hardfork_and_block_gas_limit( + let block_executor_factory = + MegaBlockExecutorFactory::new(hardforks, evm_factory, OpAlloyReceiptBuilder::default()); + // Both the per-transaction and the block-level dimensions come from the fork resolved out + // of the schedule above, so a spec override moves all of them at once. The no-override + // path keeps the "no fork active" failure: a block older than the chain's first hardfork + // has no limits to execute under. A synthesized schedule always has one active. + let block_limits = BlockLimits::from_hardfork_and_block_gas_limit( hardforks.hardfork(ctx.block.header.timestamp()).ok_or(ReplayError::Other(format!( "No `MegaHardfork` active at block timestamp: {}", ctx.block.header.timestamp() @@ -450,14 +895,6 @@ impl Cmd { ctx.block.header.gas_limit(), ); - if let Some(spec_override) = &self.spec_override { - info!(spec_override = %spec_override, "Overriding EVM spec"); - let spec = MegaSpecId::from_str(spec_override) - .map_err(|e| ReplayError::Other(format!("Invalid spec: {e:?}")))?; - evm_env.cfg_env.set_spec_and_mainnet_gas_params(spec); - block_limits = block_limits.with_tx_runtime_limits(EvmTxRuntimeLimits::from_spec(spec)); - } - // The spec the target transaction will execute under (after any override), // captured before `evm_env` is moved into the executor. let executed_spec = evm_env.cfg_env.spec; @@ -480,26 +917,37 @@ impl Cmd { &mut inspector, ); - block_executor - .apply_pre_execution_changes() - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + block_executor.apply_pre_execution_changes().map_err(ReplayError::BlockExecutionError)?; // Execute preceding transactions info!(preceding_count = ctx.preceding_tx_hashes.len(), "Executing preceding transactions",); for tx_hash in &ctx.preceding_tx_hashes { debug!(tx_hash = %tx_hash, "Executing preceding transaction"); + // These hashes were read out of the block body this endpoint already + // served, so `Ok(None)` contradicts an answer it gave itself: the + // endpoint is inconsistent (reorg, or a load-balanced backend serving + // divergent views), not definitively denying the hash. Only the + // user-supplied target lookup keeps `TransactionNotFound`, because + // there the null is a definitive answer about a hash the caller asked + // about. let tx = provider .get_transaction_by_hash(*tx_hash) .await .map_err(|e| ReplayError::RpcError(format!("RPC transport error: {e}")))? - .ok_or(ReplayError::TransactionNotFound(*tx_hash))?; + .ok_or(ReplayError::BlockBodyTransactionNull(*tx_hash))?; + // A served object that fails authentication is the same class as a + // null answer on a body-listed hash: the endpoint failed to deliver + // a transaction it claimed to include. + verify::authenticate_transaction(&tx, *tx_hash).map_err(|message| { + ReplayError::BlockBodyTransactionFetch { tx_hash: *tx_hash, message } + })?; let outcome = block_executor .run_transaction(tx.as_recovered()) - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + .map_err(ReplayError::BlockExecutionError)?; trace!(tx_hash = %tx_hash, ?outcome, "Preceding transaction executed"); block_executor .commit_transaction_outcome(outcome) - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + .map_err(ReplayError::BlockExecutionError)?; } // Clear block hash reads accumulated by the preceding transactions so the @@ -521,9 +969,8 @@ impl Cmd { .unwrap_or(0); block_executor.inspector_mut().fuse(); - let outcome = block_executor - .run_transaction(wrapped_tx) - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + let outcome = + block_executor.run_transaction(wrapped_tx).map_err(ReplayError::BlockExecutionError)?; trace!(tx_hash = %ctx.target_tx.inner.inner.tx_hash(), ?outcome, "Target transaction executed"); let exec_result = outcome.inner.result.clone(); let evm_state = outcome.inner.state.clone(); @@ -588,17 +1035,26 @@ impl Cmd { let gas_used = block_executor .commit_transaction_outcome(outcome) - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + .map_err(ReplayError::BlockExecutionError)?; let duration = start.elapsed(); - let (evm, block_result) = block_executor - .finish() - .map_err(|e| ReplayError::Other(format!("Block execution error: {e}")))?; + let (evm, block_result) = + block_executor.finish().map_err(ReplayError::BlockExecutionError)?; let (db, _) = evm.finish(); db.merge_transitions(BundleRetention::Reverts); let receipt_envelope = block_result.receipts.last().unwrap().clone(); trace!(?receipt_envelope, "Receipt envelope obtained"); + // Block-global log index: cumulative log count of every preceding + // receipt in this block (same data `finish()` harvested). + let first_log_index: u64 = block_result + .receipts + .iter() + .rev() + .skip(1) + .map(|envelope| envelope.logs().len() as u64) + .sum(); + let from = ctx.target_tx.inner.inner.signer(); let to = ctx.target_tx.inner.inner.to(); let contract_address = (to.is_none() && receipt_envelope.is_success()) @@ -615,8 +1071,19 @@ impl Cmd { Some(ctx.target_tx.inner.inner.tx_hash()), Some(ctx.block.hash()), ctx.preceding_tx_hashes.len() as u64, + first_log_index, ); + let verification = onchain_receipt.as_ref().map(|onchain| { + verify::compare( + &verify::ReceiptFacts::from_receipt(&onchain.inner), + &verify::ReceiptFacts::from_receipt(&receipt), + ) + }); + if let Some(verification) = &verification { + debug!(matched = verification.matched, "On-chain receipt verified"); + } + Ok(ReplayOutcome { outcome: EvmeOutcome { pre_execution_nonce, @@ -627,6 +1094,7 @@ impl Cmd { }, receipt, fixture, + verification, }) } @@ -641,6 +1109,9 @@ impl Cmd { summary.fill_trace_and_dump(&result.outcome, &self.trace_args, &self.dump_args)?; summary.receipt = Some(serde_json::to_value(&result.receipt).expect("failed to serialize receipt")); + summary.verification = result.verification.as_ref().map(|verification| { + serde_json::to_value(verification).expect("failed to serialize verification") + }); println!( "{}", serde_json::to_string_pretty(&summary).expect("failed to serialize output") @@ -652,6 +1123,10 @@ impl Cmd { result.outcome.exec_time, ); print_receipt(&result.receipt); + if let Some(verification) = &result.verification { + println!(); + println!("{}", verification.verdict_line()); + } print_execution_trace( result.outcome.trace_data.as_deref(), self.trace_args.trace_output_file.as_deref(), @@ -664,12 +1139,36 @@ impl Cmd { } } +/// Whether any trace option was set on the command line. +/// +/// `--tracer` carries a default, so it counts as set only when it names a +/// non-default tracer. +fn has_trace_args(args: &run::TraceArgs) -> bool { + args.trace || + args.trace_output_file.is_some() || + !matches!(args.tracer, TracerType::Opcode) || + args.trace_opcode_disable_memory || + args.trace_opcode_disable_stack || + args.trace_opcode_disable_storage || + args.trace_opcode_enable_return_data || + args.trace_call_only_top_call || + args.trace_call_with_log || + args.trace_prestate_diff_mode || + args.trace_prestate_disable_code || + args.trace_prestate_disable_storage +} + +/// Whether any state dump option was set on the command line. +fn has_dump_args(args: &run::StateDumpArgs) -> bool { + args.dump || args.dump_output_file.is_some() +} + /// Build a [`BlockEnv`] from the RPC block header. /// /// Reads `excess_blob_gas` directly from the header rather than using a /// hardcoded default, so blob-fee-sensitive opcodes (e.g. `BLOBBASEFEE`) /// match on-chain semantics during replay. -fn retrieve_block_env(block: &Block) -> Result { +pub(super) fn retrieve_block_env(block: &Block) -> Result { let mut block_env = BlockEnv { number: U256::from(block.number()), beneficiary: block.header.beneficiary(), @@ -704,6 +1203,266 @@ mod tests { use alloy_rpc_types_eth::Header as RpcHeader; use mega_evm::revm::context_interface::block::BlobExcessGasAndPrice; + const TX: &str = "0x323ddc8e67dfc134284d78c65f3c1dc7ff45ba1db02eeaf62e211ae3253478ef"; + const RPC: [&str; 2] = ["--rpc.replay-file", "/tmp/envelope.json"]; + + /// Parse a `replay` invocation, prefixing the shared offline RPC flags. + fn parse(extra: &[&str]) -> std::result::Result { + let mut argv = vec!["replay"]; + argv.extend_from_slice(&RPC); + argv.extend_from_slice(extra); + Cmd::try_parse_from(argv) + } + + /// Parse a batch invocation carrying one extra flag, and return the + /// validation error message it must be rejected with. + fn batch_rejection(extra: &[&str]) -> String { + let mut argv = vec!["--block", "22945844"]; + argv.extend_from_slice(extra); + let cmd = parse(&argv).expect("flags should parse"); + cmd.validate().expect_err("batch mode must reject the flag").to_string() + } + + /// Parse an online `replay` invocation (no offline RPC flags). + fn parse_online(extra: &[&str]) -> Cmd { + let mut argv = vec!["replay", "--rpc", "http://localhost:1"]; + argv.extend_from_slice(extra); + Cmd::try_parse_from(argv).expect("online flags should parse") + } + + #[test] + fn test_batch_rpc_args_disables_disk_cache_by_default() { + let cmd = parse_online(&["--block", "1"]); + assert!(!cmd.rpc_args.no_cache_file, "the flag itself must default off"); + assert!( + cmd.batch_rpc_args().no_cache_file, + "a batch run without --rpc.cache-dir must not touch the disk cache", + ); + } + + #[test] + fn test_batch_rpc_args_explicit_cache_dir_opts_back_in() { + let cmd = parse_online(&["--block", "1", "--rpc.cache-dir", "/tmp/evme-cache"]); + let args = cmd.batch_rpc_args(); + assert!(!args.no_cache_file, "an explicit --rpc.cache-dir must keep persistence on"); + assert_eq!(args.cache_dir.as_deref(), Some(std::path::Path::new("/tmp/evme-cache"))); + } + + #[test] + fn test_batch_rpc_args_explicit_clear_cache_opts_back_in() { + let cmd = parse_online(&["--block", "1", "--rpc.clear-cache"]); + let args = cmd.batch_rpc_args(); + assert!( + !args.no_cache_file, + "--rpc.clear-cache asks for the disk cache file to be deleted, which only \ + happens while the disk cache is engaged", + ); + assert!(args.clear_cache, "the clear request itself must survive"); + assert_eq!(args.cache_dir, None, "the default cache path is the one being cleared"); + } + + #[test] + fn test_batch_rpc_args_keeps_explicit_no_cache_file() { + let cmd = parse_online(&["--block", "1", "--rpc.no-cache-file"]); + assert!(cmd.batch_rpc_args().no_cache_file, "--rpc.no-cache-file must stay honored"); + } + + /// `--rpc.no-cache-file` and `--rpc.clear-cache` are not mutually exclusive at the + /// parser level. Batch mode must not reinterpret the pair: it passes both through so + /// the combination behaves exactly as it does in single-transaction mode. + #[test] + fn test_batch_rpc_args_passes_no_cache_file_with_clear_cache_through() { + let cmd = parse_online(&["--block", "1", "--rpc.no-cache-file", "--rpc.clear-cache"]); + let args = cmd.batch_rpc_args(); + assert!(args.no_cache_file, "--rpc.no-cache-file must stay honored alongside clear"); + assert!(args.clear_cache, "the clear flag must be passed through unmodified"); + } + + /// `--rpc.cache-dir` and `--rpc.clear-cache` together name the file to clear. + #[test] + fn test_batch_rpc_args_cache_dir_with_clear_cache_opts_back_in() { + let cmd = parse_online(&[ + "--block", + "1", + "--rpc.cache-dir", + "/tmp/evme-cache", + "--rpc.clear-cache", + ]); + let args = cmd.batch_rpc_args(); + assert!(!args.no_cache_file, "both flags opt the batch run into the disk cache"); + assert!(args.clear_cache); + assert_eq!(args.cache_dir.as_deref(), Some(std::path::Path::new("/tmp/evme-cache"))); + } + + #[test] + fn test_replay_target_group_accepts_each_form() { + assert!(matches!( + parse(&[TX]).expect("positional").resolve_mode().expect("mode"), + ReplayMode::Single(_) + )); + assert!(matches!( + parse(&["--block", "0x15e2034"]).expect("block").resolve_mode().expect("mode"), + ReplayMode::Batch(batch::BatchMode::Block(22_945_844)), + )); + // `--tx-file` reads the file, so only the parse is checked here. + assert_eq!( + parse(&["--tx-file", "/tmp/list.txt"]).expect("tx-file").tx_file, + Some(PathBuf::from("/tmp/list.txt")), + ); + } + + #[test] + fn test_replay_target_group_is_required() { + let err = parse(&[]).expect_err("a replay target is required"); + assert_eq!(err.kind(), clap::error::ErrorKind::MissingRequiredArgument); + } + + #[test] + fn test_replay_target_group_is_mutually_exclusive() { + for extra in [ + vec![TX, "--block", "1"], + vec![TX, "--tx-file", "/tmp/list.txt"], + vec!["--block", "1", "--tx-file", "/tmp/list.txt"], + ] { + let err = parse(&extra).expect_err("targets must be mutually exclusive"); + assert_eq!( + err.kind(), + clap::error::ErrorKind::ArgumentConflict, + "unexpected error for {extra:?}: {err}" + ); + } + } + + #[test] + fn test_batch_rejects_single_transaction_only_flags() { + for (extra, expected) in [ + (vec!["--dump-fixture", "/tmp/f.json"], "--dump-fixture"), + (vec!["--override.gas-limit", "50000"], "transaction overrides"), + (vec!["--override.value", "1ether"], "transaction overrides"), + (vec!["--override.input", "0xdeadbeef"], "transaction overrides"), + (vec!["--override.input-file", "/tmp/in.hex"], "transaction overrides"), + (vec!["--override.spec", "Rex4"], "--override.spec"), + (vec!["--trace"], "trace options"), + (vec!["--trace.output", "/tmp/t.json"], "trace options"), + (vec!["--tracer", "call"], "trace options"), + (vec!["--trace.call.with-log"], "trace options"), + (vec!["--trace.prestate.diff-mode"], "trace options"), + (vec!["--dump"], "state dump options"), + (vec!["--dump.output", "/tmp/s.json"], "state dump options"), + ] { + let message = batch_rejection(&extra); + assert!( + message.contains(expected) && message.contains("batch replay"), + "unexpected rejection for {extra:?}: {message}" + ); + } + } + + /// `--verify-receipt` is a whole-corpus flag: both replay modes take it. + #[test] + fn test_verify_receipt_is_accepted_in_both_modes() { + for extra in [vec![TX], vec!["--block", "1"], vec!["--tx-file", "/tmp/list.txt"]] { + let mut argv = extra.clone(); + argv.push("--verify-receipt"); + let cmd = parse(&argv).expect("--verify-receipt should parse"); + assert!(cmd.verify_receipt, "the flag must be recorded for {extra:?}"); + cmd.validate() + .unwrap_or_else(|e| panic!("--verify-receipt must be accepted for {extra:?}: {e}")); + } + } + + /// The flag defaults to off, so a replay without it does no receipt fetch. + #[test] + fn test_verify_receipt_defaults_to_off() { + assert!(!parse(&[TX]).expect("parse").verify_receipt); + } + + /// `--dump-fixture-dir` is batch-only; single-transaction mode keeps + /// `--dump-fixture` and must reject the dir flag. + #[test] + fn test_dump_fixture_dir_rejected_in_single_transaction_mode() { + let cmd = parse(&["--dump-fixture-dir", "/tmp/fixtures", TX]).expect("parse"); + let message = + cmd.validate().expect_err("single-tx must reject --dump-fixture-dir").to_string(); + assert!( + message.contains("--dump-fixture-dir") && message.contains("batch"), + "unexpected rejection: {message}" + ); + } + + /// Batch mode accepts `--dump-fixture-dir` and still rejects the single-file + /// dump flag (use the dir form for sedimentation sweeps). + #[test] + fn test_dump_fixture_dir_accepted_in_batch_mode() { + let cmd = parse(&["--block", "1", "--dump-fixture-dir", "/tmp/fixtures"]).expect("parse"); + cmd.validate().expect("--dump-fixture-dir must be accepted in batch mode"); + assert_eq!(cmd.dump_fixture_dir, Some(PathBuf::from("/tmp/fixtures"))); + + let with_overwrite = + parse(&["--tx-file", "/tmp/list.txt", "--dump-fixture-dir", "/tmp/f", "--overwrite"]) + .expect("parse"); + with_overwrite.validate().expect("--overwrite is allowed with --dump-fixture-dir"); + assert!(with_overwrite.overwrite); + } + + /// The two dump forms are mutually exclusive even when one would otherwise + /// be valid for the selected mode. + #[test] + fn test_dump_fixture_forms_are_mutually_exclusive() { + let cmd = parse(&[ + "--block", + "1", + "--dump-fixture", + "/tmp/f.json", + "--dump-fixture-dir", + "/tmp/fixtures", + ]) + .expect("parse"); + // Batch validation rejects --dump-fixture first; either way both must not run. + let message = cmd.validate().expect_err("both dump forms must be rejected").to_string(); + assert!(message.contains("--dump-fixture"), "unexpected rejection: {message}"); + } + + /// `--block 0` is rejected as input rather than replayed into an empty, + /// silent, exit-0 run. + #[test] + fn test_batch_rejects_block_zero() { + let err = parse(&["--block", "0"]) + .expect("parse") + .validate() + .expect_err("genesis cannot be replayed"); + let message = err.to_string(); + assert!(message.contains("--block 0"), "message={message}"); + assert!(message.contains("genesis"), "message={message}"); + } + + #[test] + fn test_batch_accepts_the_flags_it_supports() { + parse(&["--block", "1", "--json"]).expect("parse").validate().expect("--json is allowed"); + parse(&["--tx-file", "/tmp/list.txt"]) + .expect("parse") + .validate() + .expect("plain batch replay is allowed"); + } + + /// The single-transaction path keeps accepting every flag batch mode rejects. + #[test] + fn test_single_transaction_path_keeps_all_flags() { + for extra in [ + vec!["--trace", "--tracer", "call"], + vec!["--dump", "--dump.output", "/tmp/s.json"], + vec!["--override.spec", "Rex4"], + vec!["--override.gas-limit", "50000"], + ] { + let mut argv = vec![TX]; + argv.extend_from_slice(&extra); + parse(&argv) + .expect("parse") + .validate() + .unwrap_or_else(|e| panic!("single-transaction replay must accept {extra:?}: {e}")); + } + } + fn make_block(excess_blob_gas: Option) -> Block { let inner = ConsensusHeader { excess_blob_gas, ..Default::default() }; Block::empty(RpcHeader::new(inner)) diff --git a/bin/mega-evme/src/replay/fixture.rs b/bin/mega-evme/src/replay/fixture.rs index ce203422..b66e3448 100644 --- a/bin/mega-evme/src/replay/fixture.rs +++ b/bin/mega-evme/src/replay/fixture.rs @@ -39,6 +39,40 @@ use state_test::{ use super::{ReplayError, Result}; +/// Why [`build_draft`] refused to produce a fixture. +/// +/// The two variants carry different consequences, so the distinction is typed +/// rather than recovered from the message: a whole-block sweep always meets some +/// transactions the fixture format cannot express, and those must not fail the +/// run, whereas a failure to construct a draft the caller asked for must. +pub(crate) enum FixtureBuildError { + /// The transaction, spec, or replay is outside what a fixture can express + /// (deposit and set-code transactions, specs with no fixture mapping, a + /// replay that does not reproduce the chain). Reported as a skip. + Unsupported(String), + /// The draft could not be built (pre-state or code read failed). The + /// requested artifact was not produced; reported as an error. + Construction(ReplayError), +} + +impl Display for FixtureBuildError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Unsupported(reason) => f.write_str(reason), + Self::Construction(err) => write!(f, "{err}"), + } + } +} + +impl From for ReplayError { + fn from(err: FixtureBuildError) -> Self { + match err { + FixtureBuildError::Unsupported(reason) => Self::Other(reason), + FixtureBuildError::Construction(err) => err, + } + } +} + /// The on-chain receipt values a dumped fixture is anchored to: a replay that /// does not reproduce all of these did not reproduce the on-chain transaction. pub(crate) struct OnchainAnchor { @@ -79,6 +113,63 @@ const DEPOSIT_TX_TYPE: u8 = 0x7e; /// than emit a fixture whose isolated run diverges from the chain. const EIP7702_TX_TYPE: u8 = 0x04; +/// Check that a local replay reproduces the on-chain receipt's gas, success +/// status, and logs root. +/// +/// A mismatch means the replay executed under the wrong spec / hardfork config +/// for this chain and block; self-validation cannot catch this, because the +/// fixture is validated under the same spec it was dumped with. +/// +/// Logs are checked, not just inferred from gas: LOG gas depends on topic count +/// and data length, never content, so two executions can burn identical gas yet +/// emit different log payloads (e.g. a preceding-tx divergence that changes a +/// value the target re-emits). +/// +/// Returns the explanatory reason on failure so batch dump can record a skip +/// without treating it as an infrastructure error. +pub(crate) fn check_fidelity( + result: &ExecutionResult, + anchor: &OnchainAnchor, + chain_id: u64, +) -> std::result::Result<(), String> { + let actual_gas = result.tx_gas_used(); + if actual_gas != anchor.gas_used { + return Err(format!( + "replay gas {actual_gas} != on-chain receipt gas {}: the local replay does \ + not reproduce on-chain execution (likely a wrong spec or hardfork config \ + for chain {chain_id} at this block)", + anchor.gas_used + )); + } + if result.is_success() != anchor.success { + return Err(format!( + "replay status (success={}) != on-chain receipt status (success={}): the \ + local replay does not reproduce on-chain execution for chain {chain_id}", + result.is_success(), + anchor.success + )); + } + let actual_logs_root = state_test::utils::log_rlp_hash(result.logs()); + if actual_logs_root != anchor.logs_root { + return Err(format!( + "replay logs root {actual_logs_root} != on-chain receipt logs root {}: the \ + local replay emits different logs than the chain for chain {chain_id} \ + (same gas/status, different log contents)", + anchor.logs_root + )); + } + Ok(()) +} + +/// Build an [`OnchainAnchor`] from the consensus facts of an on-chain receipt. +pub(crate) fn anchor_from_receipt_facts(facts: &super::verify::ReceiptFacts) -> OnchainAnchor { + OnchainAnchor { + gas_used: facts.gas_used, + success: facts.status, + logs_root: state_test::utils::log_rlp_hash(&facts.logs), + } +} + /// A fixture built from a replay, awaiting its `post` expectation. /// /// The `post` map is filled by [`finalize_and_write`] after re-executing the @@ -119,19 +210,19 @@ pub(crate) fn build_draft( block: &Block, target_tx: &Transaction, inputs: FixtureInputs<'_>, -) -> Result +) -> std::result::Result where DB: DatabaseRef, DB::Error: Display, { let envelope: &OpTxEnvelope = &target_tx.inner.inner; if envelope.ty() == DEPOSIT_TX_TYPE { - return Err(ReplayError::Other( + return Err(FixtureBuildError::Unsupported( "--dump-fixture does not support deposit transactions".to_string(), )); } if envelope.ty() == EIP7702_TX_TYPE { - return Err(ReplayError::Other( + return Err(FixtureBuildError::Unsupported( "--dump-fixture does not support EIP-7702 (set-code) transactions: the \ fixture builder does not serialize the authorization list" .to_string(), @@ -144,55 +235,19 @@ where let actual_output = inputs.result.output().cloned(); let actual_logs_root = state_test::utils::log_rlp_hash(inputs.result.logs()); - // Fidelity gate: the local replay must reproduce the on-chain receipt's gas, - // success status, and logs. A mismatch means the replay executed under the - // wrong spec / hardfork config for this chain and block; self-validation - // cannot catch this, because the fixture is validated under the same spec it - // was dumped with. Refuse to build a fixture that does not match the chain. - // - // Logs are checked, not just inferred from gas: LOG gas depends on topic count - // and data length, never content, so two executions can burn identical gas yet - // emit different log payloads (e.g. a preceding-tx divergence that changes a - // value the target re-emits). The receipt's logs are already fetched, so the - // comparison is a single root equality. `finalize_and_write` then re-checks the - // isolated run's logs root against this same value, so any gas-, output-, or - // log-visible divergence from the zeroed L1 data fee aborts the dump. One - // channel stays open by construction: the isolated run's sender balance is - // shifted by the zeroed fee, so a contract that stores a balance-derived value - // bakes that shifted value into `post` (gas, status, output, and logs all - // still match). The fixture still self-validates and reproduces gas exactly. - let anchor = &inputs.anchor; - if actual_gas != anchor.gas_used { - return Err(ReplayError::Other(format!( - "replay gas {actual_gas} != on-chain receipt gas {}: the local replay does \ - not reproduce on-chain execution (likely a wrong spec or hardfork config \ - for chain {chain_id} at this block)", - anchor.gas_used - ))); - } - if inputs.result.is_success() != anchor.success { - return Err(ReplayError::Other(format!( - "replay status (success={}) != on-chain receipt status (success={}): the \ - local replay does not reproduce on-chain execution for chain {chain_id}", - inputs.result.is_success(), - anchor.success - ))); - } - if actual_logs_root != anchor.logs_root { - return Err(ReplayError::Other(format!( - "replay logs root {actual_logs_root} != on-chain receipt logs root {}: the \ - local replay emits different logs than the chain for chain {chain_id} \ - (same gas/status, different log contents)", - anchor.logs_root - ))); - } + // Fidelity gate: refuse to dump a fixture that does not match the chain. + // See [`check_fidelity`] for the rationale and the dimensions checked. + // Every rejection it can return is an unsupported replay, not a construction + // failure, so the classification does not depend on which one fired. + check_fidelity(inputs.result, &inputs.anchor, chain_id) + .map_err(FixtureBuildError::Unsupported)?; - let pre = build_pre_state(db, evm_state)?; + let pre = build_pre_state(db, evm_state).map_err(FixtureBuildError::Construction)?; let env = build_env(chain_id, block); let transaction = build_transaction(target_tx)?; let spec_name = SpecName::from_mega_spec(spec); if spec_name == SpecName::Unknown { - return Err(ReplayError::Other(format!( + return Err(FixtureBuildError::Unsupported(format!( "--dump-fixture: spec {spec:?} has no fixture mapping" ))); } @@ -224,7 +279,16 @@ where /// Re-execute the isolated unit through `state-test`, cross-check it against the /// observed replay outcome, fill the `post` expectation, and write the fixture. -pub(crate) fn finalize_and_write(draft: FixtureDraft, path: &std::path::Path) -> Result<()> { +/// +/// `overwrite` controls the final publish step: when false, the write refuses to +/// replace an existing file (`persist_noclobber`); when true, it replaces via +/// `persist`. Existence checks earlier in the dump pipeline are a fast-path only +/// — correctness against a concurrent creator comes from the noclobber publish. +pub(crate) fn finalize_and_write( + draft: FixtureDraft, + path: &std::path::Path, + overwrite: bool, +) -> Result<()> { let executed = execute_unit_collect(&draft.unit, &draft.spec) .map_err(|e| ReplayError::Other(format!("fixture self-execution failed: {e}")))?; @@ -279,20 +343,52 @@ pub(crate) fn finalize_and_write(draft: FixtureDraft, path: &std::path::Path) -> let suite = TestSuite(BTreeMap::from([(draft.name, unit)])); let json = serde_json::to_string_pretty(&suite) .map_err(|e| ReplayError::Other(format!("failed to serialize fixture: {e}")))?; - // Write to a sibling temp file and rename so an interrupted write cannot - // truncate an existing fixture at `path` (e.g. a committed corpus entry - // being refreshed in place). - let tmp = path.with_extension("json.tmp"); - std::fs::write(&tmp, json).map_err(|e| { - ReplayError::Other(format!("failed to write fixture {}: {e}", tmp.display())) + + // Unique temp file in the target directory, then persist (or noclobber-persist) + // into `path`. A fixed sibling name would race two concurrent dumps; a unique + // name plus noclobber makes `--overwrite=false` safe at materialization time. + let dir = path.parent().unwrap_or_else(|| std::path::Path::new(".")); + let mut tmp = tempfile::NamedTempFile::new_in(dir).map_err(|e| { + ReplayError::Other(format!("failed to create temp fixture file in {}: {e}", dir.display())) })?; - std::fs::rename(&tmp, path).map_err(|e| { - ReplayError::Other(format!( - "failed to rename fixture {} -> {}: {e}", - tmp.display(), - path.display() - )) - }) + use std::io::Write; + tmp.write_all(json.as_bytes()) + .map_err(|e| ReplayError::Other(format!("failed to write fixture temp file: {e}")))?; + tmp.flush() + .map_err(|e| ReplayError::Other(format!("failed to flush fixture temp file: {e}")))?; + // flush() only clears the userspace buffer; the rename below is atomic but + // the contents are not. A benchmark corpus that a crash left holding a + // truncated fixture would fail in a way that looks like a replay bug. + tmp.as_file() + .sync_all() + .map_err(|e| ReplayError::Other(format!("failed to sync fixture temp file: {e}")))?; + if overwrite { + tmp.persist(path).map_err(|e| { + ReplayError::Other(format!( + "failed to persist fixture to {}: {}", + path.display(), + e.error + )) + })?; + } else { + tmp.persist_noclobber(path).map_err(|e| { + // Target already present (or appeared between prep and publish): same + // refused-overwrite path the prep-time existence check uses. + if path.exists() { + ReplayError::Other(format!( + "fixture already exists at {} (pass --overwrite to replace)", + path.display() + )) + } else { + ReplayError::Other(format!( + "failed to persist fixture to {}: {}", + path.display(), + e.error + )) + } + })?; + } + Ok(()) } /// Read the pre-execution values of every account in the target transaction's @@ -306,18 +402,34 @@ where DB: DatabaseRef, DB::Error: Display, { + // Test-only injection: the offline State cache reuses account basics already + // loaded during execution, so doctoring the capture cannot force a draft-only + // pre-state failure. Integration tests set this env var to exercise the + // construction-error path after a successful execution. Compiled out of + // production builds: only the test profile and the `test-utils` feature + // (enabled for the binary via the self dev-dependency) carry the hook. + #[cfg(any(test, feature = "test-utils"))] + if std::env::var_os("MEGA_EVME_INJECT_FIXTURE_PRE_STATE_ERROR").is_some() { + return Err(ReplayError::Other( + "pre-state read for 0x0000000000000000000000000000000000000001: \ + injected draft-time database failure" + .to_string(), + )); + } + let mut pre = BTreeMap::new(); for (address, account) in evm_state { let Some(info) = db .basic_ref(*address) .map_err(|e| ReplayError::Other(format!("pre-state read for {address}: {e}")))? else { - // The database reports no account. RPC-backed databases (AlloyDB) - // always materialize an account (possibly all-empty), so on a forked - // replay this branch never fires and accounts created by the target - // transaction enter `pre` as explicit empty accounts — equivalent - // under EIP-161 state clearing. A database that can signal - // nonexistence omits the account here. + // The database reports no account. On a forked replay the RPC + // backend normalizes an all-zero (balance, nonce, code) answer to + // `None` (see `normalize_rpc_account` in `common/state.rs`), so + // this branch fires for every pre-transaction nonexistent account + // — including accounts the target transaction itself creates. + // Omitting them is correct state-test semantics: absence in `pre` + // means the account did not exist. continue; }; @@ -397,7 +509,9 @@ fn build_env(chain_id: u64, block: &Block) -> Env { } /// Build the EEST `transaction` (single-element index arrays) from the target tx. -fn build_transaction(target_tx: &Transaction) -> Result { +fn build_transaction( + target_tx: &Transaction, +) -> std::result::Result { let sender = target_tx.inner.inner.signer(); let tx: &OpTxEnvelope = &target_tx.inner.inner; let tx_type = tx.ty(); @@ -409,7 +523,7 @@ fn build_transaction(target_tx: &Transaction) -> Result { let (gas_price, max_fee_per_gas) = match tx_type { 0 | 1 => { let gas_price = tx.gas_price().ok_or_else(|| { - ReplayError::Other(format!( + FixtureBuildError::Unsupported(format!( "--dump-fixture: transaction type {tx_type} reports no gas price; \ refusing to record a guessed price in the fixture" )) @@ -438,3 +552,296 @@ fn build_transaction(target_tx: &Transaction) -> Result { max_fee_per_blob_gas: tx.max_fee_per_blob_gas().map(U256::from), }) } + +#[cfg(test)] +mod tests { + use alloy_consensus::transaction::Recovered; + use alloy_primitives::Sealed; + use mega_evm::revm::{ + context::result::{Output, ResultGas, SuccessReason}, + primitives::{StorageKey, StorageValue}, + state::{AccountInfo as RevmAccountInfo, Bytecode}, + }; + use op_alloy_consensus::TxDeposit; + + use super::*; + + fn success_result(gas_used: u64) -> ExecutionResult { + ExecutionResult::Success { + reason: SuccessReason::Stop, + gas: ResultGas::default().with_total_gas_spent(gas_used), + logs: Vec::new(), + output: Output::Call(Bytes::new()), + } + } + + /// A database that fails every read. + /// + /// Any `build_draft` rejection that fires *before* the pre-state closure is + /// read must be reachable with this: if a rejection ever moved behind a + /// database read, the test would surface a `Construction` error instead. + struct UnreadableDb; + + impl DatabaseRef for UnreadableDb { + type Error = crate::common::EvmeError; + + fn basic_ref( + &self, + _: Address, + ) -> std::result::Result, Self::Error> { + Err(unavailable()) + } + + fn code_by_hash_ref(&self, _: B256) -> std::result::Result { + Err(unavailable()) + } + + fn storage_ref( + &self, + _: Address, + _: StorageKey, + ) -> std::result::Result { + Err(unavailable()) + } + + fn block_hash_ref(&self, _: u64) -> std::result::Result { + Err(unavailable()) + } + } + + fn unavailable() -> crate::common::EvmeError { + crate::common::EvmeError::InvalidInput("database unavailable".to_string()) + } + + /// A database whose `basic_ref` answers are supplied per address. + /// + /// Used to pin the `build_pre_state` shape for both existence outcomes: + /// a touched address that returns `None` is omitted from `pre`, and a + /// touched address that returns `Some` is recorded with its fields. + struct MapDb { + accounts: std::collections::HashMap>, + } + + impl DatabaseRef for MapDb { + type Error = crate::common::EvmeError; + + fn basic_ref( + &self, + address: Address, + ) -> std::result::Result, Self::Error> { + Ok(self.accounts.get(&address).cloned().unwrap_or(None)) + } + + fn code_by_hash_ref(&self, _: B256) -> std::result::Result { + Err(unavailable()) + } + + fn storage_ref( + &self, + _: Address, + _: StorageKey, + ) -> std::result::Result { + Err(unavailable()) + } + + fn block_hash_ref(&self, _: u64) -> std::result::Result { + Err(unavailable()) + } + } + + /// Touched addresses with no pre-transaction account are omitted from `pre`; + /// touched addresses that exist are recorded with their fields — including + /// an explicitly present-but-empty account (`Some(AccountInfo::default())`). + /// + /// Absence-means-nonexistence is the state-test fixture shape: a forked + /// backend returns `None` for all-zero RPC answers, so accounts created by + /// the target transaction must not appear as explicit empty entries. + /// Presence of an empty account is a different DB answer and must still be + /// recorded; `build_pre_state` does not filter empties with `is_empty()`. + #[test] + fn test_build_pre_state_omits_nonexistent_and_records_existing() { + let missing = Address::repeat_byte(0xaa); + let present = Address::repeat_byte(0xbb); + let empty_present = Address::repeat_byte(0xcc); + let balance = U256::from(42u64); + let nonce = 7u64; + + let mut accounts = std::collections::HashMap::new(); + accounts.insert(missing, None); + accounts.insert( + present, + Some(RevmAccountInfo { + balance, + nonce, + code_hash: KECCAK256_EMPTY, + code: Some(Bytecode::default()), + ..Default::default() + }), + ); + // Present-but-empty: the DB returns Some with zero fields. This must + // stay in `pre` so a future `is_empty()` filter cannot creep in. + accounts.insert(empty_present, Some(RevmAccountInfo::default())); + let db = MapDb { accounts }; + + let mut evm_state = EvmState::default(); + evm_state.insert(missing, Default::default()); + evm_state.insert(present, Default::default()); + evm_state.insert(empty_present, Default::default()); + + let pre = build_pre_state(&db, &evm_state).expect("pre-state construction succeeds"); + + assert!( + !pre.contains_key(&missing), + "touched + basic_ref=None must be absent from pre (nonexistence)" + ); + let recorded = pre.get(&present).expect("touched + basic_ref=Some must appear in pre"); + assert_eq!(recorded.balance, balance); + assert_eq!(recorded.nonce, nonce); + assert!(recorded.code.is_empty()); + assert!(recorded.storage.is_empty()); + + let empty_recorded = pre + .get(&empty_present) + .expect("touched + basic_ref=Some(default) must appear in pre (empty is not absent)"); + assert_eq!(empty_recorded.balance, U256::ZERO); + assert_eq!(empty_recorded.nonce, 0); + assert!(empty_recorded.code.is_empty()); + assert!(empty_recorded.storage.is_empty()); + } + + fn deposit_transaction() -> Transaction { + let envelope = OpTxEnvelope::Deposit(Sealed::new(TxDeposit::default())); + let inner = alloy_rpc_types_eth::Transaction { + inner: Recovered::new_unchecked(envelope, Address::ZERO), + block_hash: None, + block_number: None, + block_timestamp: None, + transaction_index: None, + effective_gas_price: None, + }; + Transaction { inner, deposit_nonce: None, deposit_receipt_version: None } + } + + /// A deposit transaction is an unsupported shape, not a construction failure. + /// + /// Every OP-stack block opens with one, so misclassifying this would make + /// `--block N --dump-fixture-dir` exit non-zero on every block instead of + /// skipping the transaction the fixture format cannot express. The database + /// here fails every read, which proves the rejection is reached without + /// touching state — a `Construction` verdict would mean the check moved. + #[test] + fn test_build_draft_rejects_a_deposit_as_unsupported() { + let result = success_result(21_000); + let anchor = OnchainAnchor { + gas_used: 21_000, + success: true, + logs_root: state_test::utils::log_rlp_hash(&[]), + }; + let err = build_draft( + &UnreadableDb, + &EvmState::default(), + 4326, + MegaSpecId::REX6, + &Block::default(), + &deposit_transaction(), + FixtureInputs { mega_env: MegaEnv::default(), result: &result, anchor }, + ) + .err() + .expect("a deposit cannot be dumped"); + match err { + FixtureBuildError::Unsupported(reason) => { + assert!(reason.contains("deposit"), "reason={reason}"); + } + FixtureBuildError::Construction(err) => { + panic!("a deposit is an unsupported shape, not a construction failure: {err}") + } + } + } + + /// A failing pre-state read is a construction error, not a skip. + /// + /// The counterpart to the deposit case: this rejection means the artifact + /// the caller asked for was not produced, so the run must fail rather than + /// report a skip and exit 0. + #[test] + fn test_build_draft_reports_a_failed_pre_state_read_as_construction() { + let result = success_result(21_000); + let anchor = OnchainAnchor { + gas_used: 21_000, + success: true, + logs_root: state_test::utils::log_rlp_hash(&[]), + }; + // One touched account is enough to force a `basic_ref` during the + // pre-state closure; the transaction itself is a plain legacy call. + let mut evm_state = EvmState::default(); + evm_state.insert(Address::repeat_byte(0x11), Default::default()); + let envelope = OpTxEnvelope::Eip1559(alloy_consensus::Signed::new_unchecked( + alloy_consensus::TxEip1559::default(), + alloy_primitives::Signature::new(U256::ONE, U256::ONE, false), + B256::ZERO, + )); + let inner = alloy_rpc_types_eth::Transaction { + inner: Recovered::new_unchecked(envelope, Address::ZERO), + block_hash: None, + block_number: None, + block_timestamp: None, + transaction_index: None, + effective_gas_price: None, + }; + let tx = Transaction { inner, deposit_nonce: None, deposit_receipt_version: None }; + let err = build_draft( + &UnreadableDb, + &evm_state, + 4326, + MegaSpecId::REX6, + &Block::default(), + &tx, + FixtureInputs { mega_env: MegaEnv::default(), result: &result, anchor }, + ) + .err() + .expect("the pre-state read fails"); + match err { + FixtureBuildError::Construction(err) => { + let message = err.to_string(); + assert!(message.contains("pre-state read"), "message={message}"); + } + FixtureBuildError::Unsupported(reason) => { + panic!("a failed database read is not an unsupported shape: {reason}") + } + } + } + + /// Each of the three fidelity dimensions rejects on its own. + /// + /// [`build_draft`] wraps every rejection at one `map_err` site, so all three + /// are reported as [`FixtureBuildError::Unsupported`] — a whole-block sweep + /// skips a diverging replay rather than failing the run, whichever dimension + /// diverged. Only the gas and status messages share a phrase; a classifier + /// keyed on message text would have had to enumerate the third separately. + #[test] + fn test_check_fidelity_rejects_each_dimension() { + let logs_root = state_test::utils::log_rlp_hash(&[]); + let matching = OnchainAnchor { gas_used: 21_000, success: true, logs_root }; + let result = success_result(21_000); + check_fidelity(&result, &matching, 4326).expect("a faithful replay must pass the gate"); + + let cases = [ + ("gas", OnchainAnchor { gas_used: 42_000, ..matching }), + ("status", OnchainAnchor { success: false, ..matching }), + ("logs root", OnchainAnchor { logs_root: B256::repeat_byte(0xab), ..matching }), + ]; + let mut reasons = Vec::new(); + for (dimension, anchor) in cases { + let Err(reason) = check_fidelity(&result, &anchor, 4326) else { + panic!("a {dimension} divergence must be rejected"); + }; + assert!(!reason.is_empty(), "{dimension} rejection must explain itself"); + reasons.push(reason); + } + assert_eq!( + reasons.iter().collect::>().len(), + 3, + "each dimension explains its own divergence: {reasons:?}" + ); + } +} diff --git a/bin/mega-evme/src/replay/hardforks.rs b/bin/mega-evme/src/replay/hardforks.rs index 011514f2..a2b207c9 100644 --- a/bin/mega-evme/src/replay/hardforks.rs +++ b/bin/mega-evme/src/replay/hardforks.rs @@ -1,4 +1,12 @@ -use mega_evm::MegaHardforkConfig; +use core::any::Any; + +use mega_evm::{ + alloy_hardforks::{EthereumHardfork, ForkCondition}, + alloy_op_hardforks::{EthereumHardforks, OpHardfork, OpHardforks}, + MegaHardfork, MegaHardforkConfig, MegaHardforks, MegaSpecId, +}; + +use crate::common::FixedHardfork; /// Returns the hardfork configuration for a given chain ID. /// @@ -8,3 +16,241 @@ use mega_evm::MegaHardforkConfig; pub fn get_hardfork_config(chain_id: u64) -> MegaHardforkConfig { mega_evm::hardfork_schedule(chain_id) } + +/// The hardfork schedule a replay executes under. +/// +/// Without a spec override this is the chain's real schedule, so the replay reproduces the block +/// as it happened. With `--override.spec` it is a schedule synthesized from the forced spec, which +/// makes the override a coherent what-if: the pre-block predeploys, the EIP-2935 / EIP-4788 +/// gating, the block-level resource limits and the EVM semantics all come from the same spec, +/// instead of mixing the historical setup with forced semantics into a world that never existed. +/// +/// The synthesized schedule takes activation from the forced spec but keeps the chain's per-fork +/// parameters, which are chain data rather than spec data (the Rex5+ `SequencerRegistry` seeds). +#[derive(Debug, Clone, Copy)] +pub enum ReplayHardforks<'a> { + /// The chain's published activation schedule. + Chain(&'a MegaHardforkConfig), + /// A schedule synthesized from a forced spec, with parameters from the chain. + Forced(FixedHardfork<'a>), +} + +impl<'a> ReplayHardforks<'a> { + /// Selects the schedule for a replay: the chain's own, or one synthesized from + /// `spec_override`. + pub fn resolve(chain: &'a MegaHardforkConfig, spec_override: Option) -> Self { + match spec_override { + Some(spec) => Self::Forced(FixedHardfork::new(spec).with_params_from(chain)), + None => Self::Chain(chain), + } + } +} + +impl EthereumHardforks for ReplayHardforks<'_> { + fn ethereum_fork_activation(&self, fork: EthereumHardfork) -> ForkCondition { + match self { + Self::Chain(chain) => chain.ethereum_fork_activation(fork), + Self::Forced(forced) => forced.ethereum_fork_activation(fork), + } + } +} + +impl OpHardforks for ReplayHardforks<'_> { + fn op_fork_activation(&self, fork: OpHardfork) -> ForkCondition { + match self { + Self::Chain(chain) => chain.op_fork_activation(fork), + Self::Forced(forced) => forced.op_fork_activation(fork), + } + } +} + +impl MegaHardforks for ReplayHardforks<'_> { + fn mega_fork_activation(&self, fork: MegaHardfork) -> ForkCondition { + match self { + Self::Chain(chain) => chain.mega_fork_activation(fork), + Self::Forced(forced) => forced.mega_fork_activation(fork), + } + } + + fn fork_params_any(&self, fork: MegaHardfork) -> Option<&(dyn Any + Send + Sync)> { + match self { + Self::Chain(chain) => chain.fork_params_any(fork), + Self::Forced(forced) => forced.fork_params_any(fork), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use mega_evm::{ + flat_system_contract_specs, BlockLimits, EvmTxRuntimeLimits, SequencerRegistryConfig, + SequencerRegistryRex6Config, MAINNET_CHAIN_ID, TESTNET_CHAIN_ID, + }; + + /// A mainnet timestamp inside the Rex4 window: Rex4 is active, Rex5 is not. + const REX4_TIMESTAMP: u64 = 1_776_700_000; + + /// A mainnet timestamp inside the `MiniRex` window, before any Rex fork. + const MINI_REX_TIMESTAMP: u64 = 1_764_000_000; + + /// Without an override the replay world is the chain's schedule, unchanged. + #[test] + fn test_without_override_the_chain_schedule_is_used() { + let chain = get_hardfork_config(MAINNET_CHAIN_ID); + let world = ReplayHardforks::resolve(&chain, None); + + for timestamp in [0, MINI_REX_TIMESTAMP, REX4_TIMESTAMP, u64::MAX] { + assert_eq!(world.spec_id(timestamp), chain.spec_id(timestamp), "at {timestamp}"); + assert_eq!(world.hardfork(timestamp), chain.hardfork(timestamp), "at {timestamp}"); + } + for fork in MegaHardfork::VARIANTS { + assert_eq!( + world.mega_fork_activation(*fork), + chain.mega_fork_activation(*fork), + "{fork:?}", + ); + } + assert_eq!( + world.fork_params::(), + chain.fork_params::(), + ); + } + + /// A block before the first `MegaHardfork` has no active fork, and the replay must report that + /// rather than silently pick one. Testnet's `MiniRex` activates at timestamp 0, so this is + /// checked on a config whose first fork activates later. + #[test] + fn test_without_override_a_block_before_any_fork_has_no_hardfork() { + let chain = + MegaHardforkConfig::new().with(MegaHardfork::Rex, ForkCondition::Timestamp(100)); + let world = ReplayHardforks::resolve(&chain, None); + + assert_eq!(world.hardfork(99), None); + assert_eq!(world.hardfork(100), Some(MegaHardfork::Rex)); + } + + /// With an override, the whole schedule follows the forced spec: it resolves to that spec at + /// the block's timestamp (and at any other), so every consumer that reads the schedule — + /// predeploys, block limits, EVM semantics — sees the same world. + #[test] + fn test_override_makes_the_schedule_follow_the_forced_spec() { + let chain = get_hardfork_config(MAINNET_CHAIN_ID); + let world = ReplayHardforks::resolve(&chain, Some(MegaSpecId::REX5)); + + assert_eq!(world.spec_id(MINI_REX_TIMESTAMP), MegaSpecId::REX5); + assert_eq!(world.spec_id(REX4_TIMESTAMP), MegaSpecId::REX5); + assert_eq!(world.hardfork(MINI_REX_TIMESTAMP), Some(MegaHardfork::Rex5)); + assert!(world.is_rex_5_active_at_timestamp(MINI_REX_TIMESTAMP)); + assert!(!world.is_rex_6_active_at_timestamp(MINI_REX_TIMESTAMP)); + } + + /// The forced schedule keeps the chain's per-fork parameters. Without this the pre-block + /// `SequencerRegistry` deploy fails closed on every Rex5+ override. + #[test] + fn test_override_keeps_the_chain_fork_params() { + for chain_id in [MAINNET_CHAIN_ID, TESTNET_CHAIN_ID] { + let chain = get_hardfork_config(chain_id); + let world = ReplayHardforks::resolve(&chain, Some(MegaSpecId::REX5)); + + assert_eq!( + world.fork_params::(), + chain.fork_params::(), + "chain {chain_id}", + ); + assert!( + world.fork_params::().is_some(), + "chain {chain_id} must carry the Rex5 registry parameters", + ); + } + } + + /// Every parameter type the chain carries is delegated, not just the one the Rex5 deploy path + /// needs today. + #[test] + fn test_override_delegates_every_params_type() { + // The unknown-chain fallback carries both registry parameter types. + let chain = get_hardfork_config(0xdead_beef); + let world = ReplayHardforks::resolve(&chain, Some(MegaSpecId::REX6)); + + assert_eq!( + world.fork_params::(), + chain.fork_params::(), + ); + assert!(world.fork_params::().is_some()); + } + + /// The predeploy set follows the override, in both directions: forcing a newer spec on an old + /// block installs contracts that did not exist at that block, and forcing an older spec on a + /// recent block withholds contracts that did. + #[test] + fn test_override_switches_the_predeploy_set() { + let chain = get_hardfork_config(MAINNET_CHAIN_ID); + + let historical = ReplayHardforks::resolve(&chain, None); + let upgraded = ReplayHardforks::resolve(&chain, Some(MegaSpecId::REX5)); + let downgraded = ReplayHardforks::resolve(&chain, Some(MegaSpecId::MINI_REX)); + + let at = |world: &ReplayHardforks<'_>, timestamp| { + flat_system_contract_specs(world, timestamp) + .into_iter() + .map(|spec| spec.address) + .collect::>() + }; + + // MegaLimitControl arrives with Rex4, so a MiniRex-era block gains it under a Rex5 + // override and a Rex4-era block loses it under a MiniRex override. + let historical_mini_rex = at(&historical, MINI_REX_TIMESTAMP); + let forced_rex5 = at(&upgraded, MINI_REX_TIMESTAMP); + assert!(forced_rex5.len() > historical_mini_rex.len()); + assert!(forced_rex5.contains(&mega_evm::LIMIT_CONTROL_ADDRESS)); + assert!(!historical_mini_rex.contains(&mega_evm::LIMIT_CONTROL_ADDRESS)); + + let historical_rex4 = at(&historical, REX4_TIMESTAMP); + let forced_mini_rex = at(&downgraded, REX4_TIMESTAMP); + assert!(historical_rex4.contains(&mega_evm::LIMIT_CONTROL_ADDRESS)); + assert!(!forced_mini_rex.contains(&mega_evm::LIMIT_CONTROL_ADDRESS)); + + // The registry is deployed separately from the flat predeploys; its gate reads the same + // schedule, so a Rex5 override activates it on a MiniRex-era block. + assert!(upgraded.is_rex_5_active_at_timestamp(MINI_REX_TIMESTAMP)); + assert!(!historical.is_rex_5_active_at_timestamp(MINI_REX_TIMESTAMP)); + } + + /// Block-level limits follow the override too, not only the per-transaction ones. The + /// block-level dimensions are the ones a per-transaction patch cannot reach: they come from + /// the hardfork resolved out of the schedule. + #[test] + fn test_override_switches_block_level_limits() { + let chain = get_hardfork_config(MAINNET_CHAIN_ID); + let gas_limit = 10_000_000_000; + + let historical = ReplayHardforks::resolve(&chain, None); + let forced = ReplayHardforks::resolve(&chain, Some(MegaSpecId::REX5)); + + let historical_limits = BlockLimits::from_hardfork_and_block_gas_limit( + historical.hardfork(MINI_REX_TIMESTAMP).expect("MiniRex is active"), + gas_limit, + ); + let forced_limits = BlockLimits::from_hardfork_and_block_gas_limit( + forced.hardfork(MINI_REX_TIMESTAMP).expect("the forced spec is always active"), + gas_limit, + ); + + // State growth metering arrives with Rex: the block-level budget is unlimited under + // MiniRex and bounded under the forced Rex5 world. + assert_eq!(historical_limits.block_state_growth_limit, u64::MAX); + assert_ne!(forced_limits.block_state_growth_limit, u64::MAX); + assert_eq!( + forced_limits, + BlockLimits::from_hardfork_and_block_gas_limit(MegaHardfork::Rex5, gas_limit), + ); + + // The per-transaction dimensions follow as well, which is what makes the previous + // per-transaction patch redundant rather than merely subsumed. + assert_eq!( + forced_limits.to_evm_tx_runtime_limits(), + EvmTxRuntimeLimits::from_spec(MegaSpecId::REX5), + ); + } +} diff --git a/bin/mega-evme/src/replay/mod.rs b/bin/mega-evme/src/replay/mod.rs index 8424e9fa..fdcc03f0 100644 --- a/bin/mega-evme/src/replay/mod.rs +++ b/bin/mega-evme/src/replay/mod.rs @@ -3,9 +3,11 @@ //! This module provides functionality to replay historical transactions //! by fetching them from an RPC endpoint and re-executing them. +mod batch; mod cmd; mod fixture; mod hardforks; +mod verify; pub use cmd::Cmd; pub use hardforks::*; diff --git a/bin/mega-evme/src/replay/verify.rs b/bin/mega-evme/src/replay/verify.rs new file mode 100644 index 00000000..569e29b3 --- /dev/null +++ b/bin/mega-evme/src/replay/verify.rs @@ -0,0 +1,806 @@ +//! Compare a local replay against the transaction's on-chain receipt. +//! +//! `mega-evme replay --verify-receipt` fetches the on-chain receipt of every +//! replayed target and checks that the local execution reproduces it. The +//! comparison is a pure function over [`ReceiptFacts`] — the consensus facts +//! both sides carry — so it is independent of how either receipt was obtained +//! and testable without a provider. +//! +//! Anything that prevents the comparison from running at all (a receipt the +//! endpoint cannot serve, a receipt describing a different transaction than the +//! one requested, or a receipt describing a different inclusion than the +//! replayed block) is an infrastructure failure, never a mismatch: a target that +//! could not be verified must not be reported as a divergence. + +use core::fmt; + +use alloy_consensus::TxReceipt; +use alloy_primitives::{keccak256, Address, Bytes, Log, B256}; +use alloy_provider::Provider; +use alloy_rpc_types_eth::{Log as RpcLog, TransactionReceipt}; +use mega_evm::{alloy_consensus::transaction::SignerRecoverable, alloy_eips::Encodable2718}; +use op_alloy_rpc_types::{OpTransactionReceipt, Transaction}; +use serde::Serialize; + +use super::{ReplayError, Result}; + +/// The consensus facts compared between the on-chain receipt and the receipt +/// the local replay produced. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct ReceiptFacts { + /// Whether the transaction succeeded. + pub status: bool, + /// Gas the transaction used. + pub gas_used: u64, + /// The consensus logs the transaction emitted, in order. + pub logs: Vec, +} + +impl ReceiptFacts { + /// Extract the compared facts from a receipt envelope. + /// + /// Both sides go through this one accessor set — the on-chain side is the + /// RPC receipt's inner envelope, the local side the envelope the replay + /// built — so neither side can be read with different semantics. + pub(super) fn from_receipt(receipt: &TransactionReceipt) -> Self + where + T: TxReceipt, + { + Self { + status: receipt.inner.status(), + gas_used: receipt.gas_used, + logs: receipt.logs().iter().map(|log| log.inner.clone()).collect(), + } + } +} + +/// The verdict for one verified transaction. +/// +/// Three shapes on the wire: +/// - compared and equal: `{"match": true}` +/// - compared and diverged: `{"match": false, "diff": …}` +/// - receipt question unanswered: `{"error": "…"}` — the target still replayed; only the comparison +/// could not run (transport, pruned, reorg). +/// +/// Serialize is hand-written so an unavailable outcome never emits a false +/// `match` that a consumer would read as a divergence. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct VerificationOutcome { + /// Whether the local replay reproduced the on-chain receipt. + /// + /// Meaningless when [`Self::error`] is set (kept for a simple bool check + /// on the compared path); the wire shape omits `match` in that case. + pub matched: bool, + /// The mismatched dimensions; absent when the replay matched or when the + /// comparison never ran. + pub diff: Option, + /// Why the on-chain receipt could not be compared, when the target still + /// produced a local result. Mutually exclusive with a real match/diff. + pub error: Option, +} + +impl VerificationOutcome { + /// A completed comparison against an on-chain receipt. + pub(super) fn compared(matched: bool, diff: Option) -> Self { + Self { matched, diff, error: None } + } + + /// The target replayed, but the on-chain receipt question went unanswered. + pub(super) fn unavailable(message: impl Into) -> Self { + Self { matched: false, diff: None, error: Some(message.into()) } + } + + /// Whether this outcome is an unanswered receipt fetch, not a comparison. + pub(super) const fn is_unavailable(&self) -> bool { + self.error.is_some() + } + + /// The one-line human verdict printed for a verified transaction. + pub(super) fn verdict_line(&self) -> String { + if let Some(error) = &self.error { + format!("verification: FAILED ({error})") + } else if let Some(diff) = &self.diff { + format!("verification: MISMATCH ({})", diff.describe()) + } else { + "verification: MATCH".to_string() + } + } +} + +impl Serialize for VerificationOutcome { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: serde::Serializer, + { + use serde::ser::SerializeMap; + if let Some(error) = &self.error { + let mut map = serializer.serialize_map(Some(1))?; + map.serialize_entry("error", error)?; + return map.end(); + } + let fields = 1 + usize::from(self.diff.is_some()); + let mut map = serializer.serialize_map(Some(fields))?; + map.serialize_entry("match", &self.matched)?; + if let Some(diff) = &self.diff { + map.serialize_entry("diff", diff)?; + } + map.end() + } +} + +/// The mismatched dimensions of a verification. Dimensions that agree are +/// absent, so a diff never has to be scanned for "everything equal" entries. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +pub(super) struct VerificationDiff { + /// Present when the success flags differ. + #[serde(skip_serializing_if = "Option::is_none")] + pub status: Option>, + /// Present when the gas used differs. + #[serde(skip_serializing_if = "Option::is_none")] + pub gas_used: Option>, + /// Present when the emitted logs differ. + #[serde(skip_serializing_if = "Option::is_none")] + pub logs: Option, +} + +impl VerificationDiff { + /// Whether every compared dimension agreed. + fn is_empty(&self) -> bool { + self.status.is_none() && self.gas_used.is_none() && self.logs.is_none() + } + + /// Render every mismatched dimension as one comma-separated line. + fn describe(&self) -> String { + let mut parts = Vec::new(); + if let Some(m) = &self.status { + parts.push(format!("status: onchain {} vs replay {}", m.onchain, m.replay)); + } + if let Some(m) = &self.gas_used { + parts.push(format!("gas_used: onchain {} vs replay {}", m.onchain, m.replay)); + } + if let Some(logs) = &self.logs { + if let Some(m) = &logs.count { + parts.push(format!("logs_count: onchain {} vs replay {}", m.onchain, m.replay)); + } + if let Some(m) = &logs.first_mismatch { + parts.push(format!( + "logs[{}].{}: onchain {} vs replay {}", + m.index, + m.field.as_str(), + m.onchain, + m.replay, + )); + } + } + parts.join(", ") + } +} + +/// One dimension's two values. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(super) struct Mismatch { + /// The value the on-chain receipt reports. + pub onchain: T, + /// The value the local replay produced. + pub replay: T, +} + +/// How the emitted logs differ. +/// +/// A differing log count and a differing log field are independent findings: +/// both are reported when both apply, so truncated logs and rewritten logs are +/// distinguishable. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +pub(super) struct LogsDiff { + /// Present when the two sides emitted a different number of logs. + #[serde(skip_serializing_if = "Option::is_none")] + pub count: Option>, + /// The first log both sides emitted whose contents differ, if any. + #[serde(skip_serializing_if = "Option::is_none")] + pub first_mismatch: Option, +} + +impl LogsDiff { + /// Whether the logs agreed. + fn is_empty(&self) -> bool { + self.count.is_none() && self.first_mismatch.is_none() + } +} + +/// The first differing field of the first differing log. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(super) struct LogFieldMismatch { + /// Position of the log in the transaction's log list. + pub index: usize, + /// Which field of the log differs. + pub field: LogField, + /// That field's value in the on-chain receipt. + pub onchain: LogFieldValue, + /// That field's value in the local replay. + pub replay: LogFieldValue, +} + +/// The log field a [`LogFieldMismatch`] reports on. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +pub(super) enum LogField { + /// The emitting contract's address. + Address, + /// The indexed topics. + Topics, + /// The unindexed data payload. + Data, +} + +impl LogField { + /// Wire name, shared by the JSON diff and the human verdict line. + const fn as_str(self) -> &'static str { + match self { + Self::Address => "address", + Self::Topics => "topics", + Self::Data => "data", + } + } +} + +/// The value of the log field named by a [`LogFieldMismatch`]. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(untagged)] +pub(super) enum LogFieldValue { + /// An emitting contract address. + Address(Address), + /// A topic list. + Topics(Vec), + /// A data payload. + Data(Bytes), +} + +impl fmt::Display for LogFieldValue { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Address(address) => write!(f, "{address}"), + Self::Topics(topics) => { + write!(f, "[")?; + for (index, topic) in topics.iter().enumerate() { + if index > 0 { + write!(f, ", ")?; + } + write!(f, "{topic}")?; + } + write!(f, "]") + } + Self::Data(data) => write!(f, "{data}"), + } + } +} + +/// Compare the on-chain receipt against the local replay's receipt. +pub(super) fn compare(onchain: &ReceiptFacts, replay: &ReceiptFacts) -> VerificationOutcome { + let mut diff = VerificationDiff::default(); + + if onchain.status != replay.status { + diff.status = Some(Mismatch { onchain: onchain.status, replay: replay.status }); + } + if onchain.gas_used != replay.gas_used { + diff.gas_used = Some(Mismatch { onchain: onchain.gas_used, replay: replay.gas_used }); + } + let logs = compare_logs(&onchain.logs, &replay.logs); + if !logs.is_empty() { + diff.logs = Some(logs); + } + + if diff.is_empty() { + VerificationOutcome::compared(true, None) + } else { + VerificationOutcome::compared(false, Some(diff)) + } +} + +/// Compare two log lists: their length, and the contents of the logs both sides +/// emitted. +fn compare_logs(onchain: &[Log], replay: &[Log]) -> LogsDiff { + let count = (onchain.len() != replay.len()) + .then_some(Mismatch { onchain: onchain.len(), replay: replay.len() }); + // Only the logs both sides emitted can be compared field by field; a length + // difference is already reported by `count`. + let first_mismatch = onchain + .iter() + .zip(replay) + .enumerate() + .find_map(|(index, (onchain, replay))| compare_log(index, onchain, replay)); + LogsDiff { count, first_mismatch } +} + +/// Report the first differing field of one log, if any. +fn compare_log(index: usize, onchain: &Log, replay: &Log) -> Option { + if onchain.address != replay.address { + return Some(LogFieldMismatch { + index, + field: LogField::Address, + onchain: LogFieldValue::Address(onchain.address), + replay: LogFieldValue::Address(replay.address), + }); + } + if onchain.topics() != replay.topics() { + return Some(LogFieldMismatch { + index, + field: LogField::Topics, + onchain: LogFieldValue::Topics(onchain.topics().to_vec()), + replay: LogFieldValue::Topics(replay.topics().to_vec()), + }); + } + if onchain.data.data != replay.data.data { + return Some(LogFieldMismatch { + index, + field: LogField::Data, + onchain: LogFieldValue::Data(onchain.data.data.clone()), + replay: LogFieldValue::Data(replay.data.data.clone()), + }); + } + None +} + +/// Fetch a transaction's on-chain receipt. +/// +/// Uses the same call shape as the `--dump-fixture` path, so a run with +/// `--rpc.capture-file` records the receipt and a later offline run verifies +/// without network access. +/// +/// A receipt the endpoint cannot serve — a transport failure, or a receipt +/// pruned below the endpoint's retention height — is an [`ReplayError::RpcError`] +/// so the target is reported as unverified rather than as a mismatch. So is a +/// receipt that describes a different transaction than the one requested: the +/// identity check runs here, at the one seam every mode fetches through, so no +/// caller can compare against or anchor to a receipt it never asked for. +pub(super) async fn fetch_receipt

(provider: &P, tx_hash: B256) -> Result +where + P: Provider, +{ + let receipt = provider + .get_transaction_receipt(tx_hash) + .await + .map_err(|e| ReplayError::RpcError(format!("Failed to fetch receipt: {e}")))? + .ok_or_else(|| { + ReplayError::RpcError(format!( + "No on-chain receipt for transaction {tx_hash}: the transaction is unknown to \ + the endpoint, or the endpoint has pruned its receipt" + )) + })?; + check_transaction_identity(receipt.inner.transaction_hash, tx_hash) + .map_err(ReplayError::RpcError)?; + Ok(receipt) +} + +/// Check that a fetched receipt describes the transaction it was requested for. +/// +/// `eth_getTransactionReceipt` is asked by transaction hash, but nothing in the +/// answer forces the endpoint to honour it: an inconsistent backend, or a +/// tampered offline capture, can serve another transaction's receipt. Comparing +/// against it would report a verdict about the wrong transaction — a mismatch +/// blamed on the replay, or a spurious match when the two transactions happen to +/// share their consensus facts — and the dump path would anchor a fixture to it. +/// Returns the explanatory message so each mode can wrap it in the error shape it +/// reports. +pub(super) fn check_transaction_identity( + receipt_tx_hash: B256, + requested_tx_hash: B256, +) -> std::result::Result<(), String> { + if receipt_tx_hash == requested_tx_hash { + return Ok(()); + } + Err(format!( + "receipt is for transaction {receipt_tx_hash}, but transaction {requested_tx_hash} was \ + requested: the endpoint served the receipt of a different transaction (an inconsistent \ + backend, or a tampered capture); the transaction is unverified" + )) +} + +/// Check that a fetched transaction is the one it was requested for. +/// +/// `eth_getTransactionByHash` is asked by transaction hash, but nothing in the +/// answer forces the endpoint to honour it: an inconsistent backend, or a +/// tampered offline capture, can serve another transaction under the requested +/// hash — and the replay would execute it, advancing the block state on the +/// wrong transaction or reporting another transaction's outcome under the +/// target's name. The served envelope is authenticated against the request +/// rather than trusted: the transaction hash is recomputed from the served +/// consensus encoding (the response's own `hash` field is as unauthenticated as +/// the rest of it), and the sender is re-derived from the signature, since the +/// served `from` field is not covered by the hash of a signed transaction (a +/// deposit's `from` is part of its encoding, so the hash already covers it). +/// Returns the explanatory message so each call site can wrap it in the error +/// shape it reports. +pub(super) fn authenticate_transaction( + tx: &Transaction, + requested_tx_hash: B256, +) -> std::result::Result<(), String> { + let envelope = tx.inner.inner.inner(); + // Hash the consensus encoding directly: `trie_hash()`/`tx_hash()` return + // the envelope's *cached* hash, which an RPC deserialization seeds from the + // response's own `hash` field — the very value being authenticated. + let computed = keccak256(envelope.encoded_2718()); + if computed != requested_tx_hash { + return Err(format!( + "the served transaction hashes to {computed}, but transaction {requested_tx_hash} \ + was requested: the endpoint served a different transaction (an inconsistent \ + backend, or a tampered capture)" + )); + } + let recovered = envelope.recover_signer().map_err(|e| { + format!( + "transaction {requested_tx_hash}: the served transaction's signature does not \ + recover a signer ({e}): the endpoint served a corrupted transaction (an \ + inconsistent backend, or a tampered capture)" + ) + })?; + let served = tx.inner.inner.signer(); + if recovered != served { + return Err(format!( + "transaction {requested_tx_hash}: the served `from` address {served} does not match \ + the signer {recovered} recovered from the signature: the endpoint served an \ + inconsistent transaction (a corrupted backend, or a tampered capture)" + )); + } + Ok(()) +} + +/// Check that a fetched receipt describes the block the replay executed. +/// +/// Across a reorg, or against a load-balanced endpoint serving divergent views, +/// the receipt can describe a different inclusion than the block the replay ran, +/// which would compare the replay against the wrong on-chain execution. Returns +/// the explanatory message so each mode can wrap it in the error shape it +/// reports — a hard error in single-transaction mode, an `rpc` error entry in +/// batch mode. +pub(super) fn check_inclusion( + receipt_block_hash: Option, + replayed_block_hash: B256, +) -> std::result::Result<(), String> { + match receipt_block_hash { + Some(hash) if hash == replayed_block_hash => Ok(()), + Some(hash) => Err(format!( + "receipt block hash {hash} != replayed block hash {replayed_block_hash}: the receipt \ + describes a different inclusion than the replayed block (reorg in progress, or a \ + load-balanced endpoint serving divergent views); the transaction is unverified, \ + retry once the chain settles" + )), + // A receipt with no inclusion hash cannot be anchored to the replayed + // block, so it is the same class of failure as a mismatched hash. + None => Err(format!( + "receipt has no block hash: cannot anchor the receipt to the replayed block \ + {replayed_block_hash} (reorg in progress, or a load-balanced endpoint serving \ + divergent views); the transaction is unverified, retry once the chain settles" + )), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy_primitives::{address, b256, LogData}; + + const ADDR_A: Address = address!("0x00000000000000000000000000000000000000aa"); + const ADDR_B: Address = address!("0x00000000000000000000000000000000000000bb"); + const TOPIC_A: B256 = + b256!("0x000000000000000000000000000000000000000000000000000000000000000a"); + const TOPIC_B: B256 = + b256!("0x000000000000000000000000000000000000000000000000000000000000000b"); + + /// Parse a log's hex-encoded data payload. + fn data(hex: &str) -> Bytes { + hex.parse().expect("valid hex payload") + } + + /// Build a log from its three compared fields. + fn log(address: Address, topics: &[B256], data: Bytes) -> Log { + Log { + address, + data: LogData::new(topics.to_vec(), data).expect("topic count within bounds"), + } + } + + /// A successful 21,000-gas receipt emitting the given logs. + fn facts(logs: Vec) -> ReceiptFacts { + ReceiptFacts { status: true, gas_used: 21_000, logs } + } + + /// The `diff` of an outcome that must be a mismatch. + fn diff_of(outcome: &VerificationOutcome) -> &VerificationDiff { + assert!(!outcome.matched, "expected a mismatch, got {outcome:?}"); + outcome.diff.as_ref().expect("a mismatch always carries a diff") + } + + /// Serialize an outcome the way the JSON output does. + fn json(outcome: &VerificationOutcome) -> serde_json::Value { + serde_json::to_value(outcome).expect("outcome is serializable") + } + + #[test] + fn test_compare_equal_receipts_match() { + let onchain = facts(vec![log(ADDR_A, &[TOPIC_A], data("0xdeadbeef"))]); + let replay = onchain.clone(); + + let outcome = compare(&onchain, &replay); + + assert!(outcome.matched); + assert!(outcome.diff.is_none(), "a match carries no diff"); + assert_eq!(json(&outcome), serde_json::json!({ "match": true })); + assert_eq!(outcome.verdict_line(), "verification: MATCH"); + } + + /// An unanswered receipt serializes as `{"error": …}` with no `match` field, + /// so consumers never read it as a false mismatch. + #[test] + fn test_unavailable_outcome_serializes_as_error_only() { + let outcome = VerificationOutcome::unavailable("receipt pruned below retention"); + assert!(outcome.is_unavailable()); + assert_eq!( + json(&outcome), + serde_json::json!({ "error": "receipt pruned below retention" }) + ); + assert_eq!(outcome.verdict_line(), "verification: FAILED (receipt pruned below retention)"); + } + + #[test] + fn test_compare_empty_logs_on_both_sides_match() { + let outcome = compare(&facts(vec![]), &facts(vec![])); + + assert!(outcome.matched); + assert_eq!(json(&outcome), serde_json::json!({ "match": true })); + } + + #[test] + fn test_compare_reports_status_flip() { + let onchain = facts(vec![]); + let replay = ReceiptFacts { status: false, ..facts(vec![]) }; + + let outcome = compare(&onchain, &replay); + + let diff = diff_of(&outcome); + assert_eq!(diff.status, Some(Mismatch { onchain: true, replay: false })); + assert!(diff.gas_used.is_none(), "gas agreed, so it must be absent: {diff:?}"); + assert!(diff.logs.is_none(), "logs agreed, so they must be absent: {diff:?}"); + assert_eq!( + json(&outcome), + serde_json::json!({ + "match": false, + "diff": { "status": { "onchain": true, "replay": false } }, + }) + ); + assert_eq!( + outcome.verdict_line(), + "verification: MISMATCH (status: onchain true vs replay false)" + ); + } + + #[test] + fn test_compare_reports_gas_delta() { + let onchain = facts(vec![]); + let replay = ReceiptFacts { gas_used: 22_000, ..facts(vec![]) }; + + let outcome = compare(&onchain, &replay); + + let diff = diff_of(&outcome); + assert_eq!(diff.gas_used, Some(Mismatch { onchain: 21_000, replay: 22_000 })); + assert!(diff.status.is_none(), "status agreed, so it must be absent: {diff:?}"); + assert_eq!( + json(&outcome), + serde_json::json!({ + "match": false, + "diff": { "gas_used": { "onchain": 21000, "replay": 22000 } }, + }) + ); + assert_eq!( + outcome.verdict_line(), + "verification: MISMATCH (gas_used: onchain 21000 vs replay 22000)" + ); + } + + #[test] + fn test_compare_reports_log_count_delta() { + let entry = log(ADDR_A, &[TOPIC_A], data("0x")); + let onchain = facts(vec![entry.clone(), entry.clone()]); + let replay = facts(vec![entry]); + + let outcome = compare(&onchain, &replay); + + let logs = diff_of(&outcome).logs.as_ref().expect("logs differ"); + assert_eq!(logs.count, Some(Mismatch { onchain: 2, replay: 1 })); + assert!( + logs.first_mismatch.is_none(), + "the shared prefix is identical, so no field mismatch: {logs:?}" + ); + assert_eq!( + json(&outcome), + serde_json::json!({ + "match": false, + "diff": { "logs": { "count": { "onchain": 2, "replay": 1 } } }, + }) + ); + } + + #[test] + fn test_compare_reports_log_address_delta() { + let onchain = facts(vec![log(ADDR_A, &[TOPIC_A], data("0x"))]); + let replay = facts(vec![log(ADDR_B, &[TOPIC_A], data("0x"))]); + + let outcome = compare(&onchain, &replay); + + let logs = diff_of(&outcome).logs.as_ref().expect("logs differ"); + assert!(logs.count.is_none(), "both sides emitted one log: {logs:?}"); + assert_eq!( + logs.first_mismatch, + Some(LogFieldMismatch { + index: 0, + field: LogField::Address, + onchain: LogFieldValue::Address(ADDR_A), + replay: LogFieldValue::Address(ADDR_B), + }) + ); + assert_eq!( + json(&outcome)["diff"]["logs"]["first_mismatch"], + serde_json::json!({ + "index": 0, + "field": "address", + "onchain": "0x00000000000000000000000000000000000000aa", + "replay": "0x00000000000000000000000000000000000000bb", + }) + ); + } + + #[test] + fn test_compare_reports_log_topics_delta() { + let onchain = facts(vec![log(ADDR_A, &[TOPIC_A], data("0x"))]); + let replay = facts(vec![log(ADDR_A, &[TOPIC_A, TOPIC_B], data("0x"))]); + + let outcome = compare(&onchain, &replay); + + let first = diff_of(&outcome).logs.as_ref().and_then(|l| l.first_mismatch.clone()); + assert_eq!( + first, + Some(LogFieldMismatch { + index: 0, + field: LogField::Topics, + onchain: LogFieldValue::Topics(vec![TOPIC_A]), + replay: LogFieldValue::Topics(vec![TOPIC_A, TOPIC_B]), + }) + ); + assert_eq!(json(&outcome)["diff"]["logs"]["first_mismatch"]["field"], "topics"); + } + + #[test] + fn test_compare_reports_log_data_delta() { + let onchain = facts(vec![log(ADDR_A, &[TOPIC_A], data("0xdeadbeef"))]); + let replay = facts(vec![log(ADDR_A, &[TOPIC_A], data("0xfeedface"))]); + + let outcome = compare(&onchain, &replay); + + let first = diff_of(&outcome).logs.as_ref().and_then(|l| l.first_mismatch.clone()); + assert_eq!( + first, + Some(LogFieldMismatch { + index: 0, + field: LogField::Data, + onchain: LogFieldValue::Data(data("0xdeadbeef")), + replay: LogFieldValue::Data(data("0xfeedface")), + }) + ); + assert_eq!( + json(&outcome)["diff"]["logs"]["first_mismatch"], + serde_json::json!({ + "index": 0, + "field": "data", + "onchain": "0xdeadbeef", + "replay": "0xfeedface", + }) + ); + } + + /// The reported log mismatch is the first differing one, and a later + /// difference does not displace it. + #[test] + fn test_compare_reports_the_first_differing_log() { + let same = log(ADDR_A, &[TOPIC_A], data("0x")); + let onchain = facts(vec![same.clone(), same.clone(), same.clone()]); + let replay = facts(vec![ + same, + log(ADDR_B, &[TOPIC_A], data("0x")), + log(ADDR_A, &[TOPIC_A], data("0xff")), + ]); + + let outcome = compare(&onchain, &replay); + + let first = diff_of(&outcome).logs.as_ref().and_then(|l| l.first_mismatch.clone()); + assert_eq!(first.map(|m| (m.index, m.field)), Some((1, LogField::Address))); + } + + /// Every mismatched dimension is reported at once — a status flip does not + /// hide the gas delta or the log difference behind it. + #[test] + fn test_compare_reports_all_mismatched_dimensions() { + let onchain = facts(vec![log(ADDR_A, &[TOPIC_A], data("0x"))]); + let replay = ReceiptFacts { + status: false, + gas_used: 30_000, + logs: vec![log(ADDR_B, &[TOPIC_A], data("0x")), log(ADDR_A, &[], data("0x"))], + }; + + let outcome = compare(&onchain, &replay); + + let diff = diff_of(&outcome); + assert!(diff.status.is_some() && diff.gas_used.is_some()); + let logs = diff.logs.as_ref().expect("logs differ"); + assert_eq!(logs.count, Some(Mismatch { onchain: 1, replay: 2 })); + assert_eq!(logs.first_mismatch.as_ref().map(|m| m.field), Some(LogField::Address)); + assert_eq!( + outcome.verdict_line(), + format!( + "verification: MISMATCH (status: onchain true vs replay false, \ + gas_used: onchain 21000 vs replay 30000, logs_count: onchain 1 vs replay 2, \ + logs[0].address: onchain {ADDR_A} vs replay {ADDR_B})" + ) + ); + } + + #[test] + fn test_check_inclusion_accepts_the_replayed_block() { + let hash = b256!("0x1111111111111111111111111111111111111111111111111111111111111111"); + + assert!(check_inclusion(Some(hash), hash).is_ok()); + } + + #[test] + fn test_check_inclusion_rejects_a_different_inclusion() { + let message = check_inclusion( + Some(b256!("0x1111111111111111111111111111111111111111111111111111111111111111")), + b256!("0x2222222222222222222222222222222222222222222222222222222222222222"), + ) + .expect_err("a receipt from another block must be rejected"); + + assert!( + message.contains("different inclusion") && message.contains("unverified"), + "message must explain the reorg and that the target is unverified: {message}" + ); + } + + #[test] + fn test_check_transaction_identity_accepts_the_requested_transaction() { + let hash = b256!("0x3333333333333333333333333333333333333333333333333333333333333333"); + + assert!(check_transaction_identity(hash, hash).is_ok()); + } + + /// A receipt for another transaction is rejected, and the message names both + /// hashes so the served/requested confusion is diagnosable from the error + /// alone. + #[test] + fn test_check_transaction_identity_rejects_another_transactions_receipt() { + let served = b256!("0x3333333333333333333333333333333333333333333333333333333333333333"); + let requested = b256!("0x4444444444444444444444444444444444444444444444444444444444444444"); + + let message = check_transaction_identity(served, requested) + .expect_err("a receipt for another transaction must be rejected"); + + assert!( + message.contains(&format!("{served}")) && + message.contains(&format!("{requested}")) && + message.contains("different transaction") && + message.contains("unverified"), + "message must name both hashes and explain the target is unverified: {message}" + ); + } + + #[test] + fn test_check_inclusion_rejects_a_missing_block_hash() { + let replayed = b256!("0x2222222222222222222222222222222222222222222222222222222222222222"); + let message = check_inclusion(None, replayed) + .expect_err("a receipt without a block hash must be rejected"); + + assert!( + message.contains("no block hash") && + message.contains("unverified") && + message.contains(&format!("{replayed}")), + "message must explain the missing anchor and name the replayed block: {message}" + ); + } +} diff --git a/bin/mega-evme/src/tx/cmd.rs b/bin/mega-evme/src/tx/cmd.rs index 732adbb7..e9321df0 100644 --- a/bin/mega-evme/src/tx/cmd.rs +++ b/bin/mega-evme/src/tx/cmd.rs @@ -66,10 +66,10 @@ impl Cmd { let tx = if let Some(ref raw) = self.raw { let raw_bytes = load_hex(Some(raw.clone()), None)?.unwrap_or_default(); let decoded = DecodedRawTx::from_raw(raw_bytes)?.override_tx_env(&self.tx_args)?; - if decoded.tx_env.chain_id != Some(chain_id) { + if decoded.tx.base.chain_id != Some(chain_id) { warn!( chain_id, - decoded_chain_id = decoded.tx_env.chain_id, + decoded_chain_id = decoded.tx.base.chain_id, "Raw transaction chain_id does not match the configured chain_id" ); } @@ -168,6 +168,7 @@ impl Cmd { None, None, 0, + 0, ); if self.output_args.json { diff --git a/bin/mega-evme/tests/account_existence.rs b/bin/mega-evme/tests/account_existence.rs new file mode 100644 index 00000000..d0b64048 --- /dev/null +++ b/bin/mega-evme/tests/account_existence.rs @@ -0,0 +1,209 @@ +//! Integration tests for the forked backend's account-existence normalization. +//! +//! JSON-RPC cannot express "this account was never created": `eth_getBalance`, +//! `eth_getTransactionCount`, and `eth_getCode` all answer `0`/`0`/empty for +//! it, and the RPC backend would otherwise materialize that answer as an +//! *existing* empty account. `EvmeState` maps the all-zero answer back to +//! `None` (safe post-EIP-161, where existing-but-empty accounts cannot occur). +//! +//! The DB-level tests pin the normalization boundary: only the fully-zero +//! account maps to `None`; any single non-zero dimension keeps it existing. +//! The execution-level tests pin the consumer that made the bug observable: +//! EIP-7702 refunds 12,500 gas per authorization only when the authority +//! already exists in the trie, so a brand-new authority materialized as an +//! existing empty account made every replayed type-4 transaction with fresh +//! authorities under-report `gasUsed` by 12,500 per authorization. + +use alloy_eips::{ + eip2930::{AccessList, AccessListItem}, + eip7702::{Authorization, RecoveredAuthority, RecoveredAuthorization}, +}; +use alloy_primitives::{address, Address, Bytes, B256, U256}; +use clap::Parser; +use mega_evm::{ + revm::{ + context::{result::ExecutionResult, tx::TxEnvBuilder}, + state::EvmState, + DatabaseRef, ExecuteEvm, + }, + MegaContext, MegaEvm, MegaSpecId, MegaTransaction, MegaTransactionNew as _, +}; +use mega_evme::common::{EvmeExternalEnvs, EvmeState, OpProvider, PreStateArgs}; +use op_alloy_network::Optimism; +use rstest::rstest; + +mod common; +use common::{test_rpc_args, MockRpcServer}; + +/// The transaction sender; funded through a prestate override, so its account +/// never reaches the RPC mock. +const CALLER: Address = address!("00000000000000000000000000000000c0ffee01"); + +/// The call target. Has no code on the mock chain. +const CALLEE: Address = address!("00000000000000000000000000000000c0ffee02"); + +/// The EIP-7702 authority whose existence the scenarios vary. +const AUTHORITY: Address = address!("00000000000000000000000000000000c0ffee03"); + +/// The delegation designator target. Never loaded (only written). +const DELEGATE: Address = address!("00000000000000000000000000000000c0ffee04"); + +/// EIP-7702 `PER_EMPTY_ACCOUNT_COST - PER_AUTH_BASE_COST`: the per-authorization +/// refund granted when the authority already exists in the trie. +const EXISTING_AUTHORITY_REFUND: u64 = 12_500; + +/// Build a forked `EvmeState` against a mock whose `eth_getBalance` / +/// `eth_getTransactionCount` / `eth_getCode` answers are fixed for every +/// address. `eth_getStorageAt` answers the zero word so OP L1-fee loading +/// resolves without a live node. +async fn forked_state( + server: &MockRpcServer, + balance: &str, + nonce: &str, + code: &str, +) -> EvmeState { + server.respond_eth_chain_id(4326, 1).await; + server.respond_method_result("eth_getBalance", balance, 2).await; + server.respond_method_result("eth_getTransactionCount", nonce, 2).await; + server.respond_method_result("eth_getCode", code, 2).await; + server + .respond_method_result( + "eth_getStorageAt", + "0x0000000000000000000000000000000000000000000000000000000000000000", + 2, + ) + .await; + + let prestate_args = PreStateArgs::parse_from(["mega-evme", "--fork", "--fork.block", "1"]); + let rpc_args = test_rpc_args(&server.uri(), None); + let (state, _cache_store) = + prestate_args.create_initial_state(&CALLER, &rpc_args).await.expect("create_initial_state"); + state +} + +// ─── DB-level: the normalization boundary ──────────────────────────────────── + +/// An account whose balance, nonce, and code are all zero does not exist. +#[tokio::test(flavor = "multi_thread")] +async fn test_forked_all_zero_account_reads_as_nonexistent() { + let server = MockRpcServer::start().await; + let state = forked_state(&server, "0x0", "0x0", "0x").await; + + let account = state.basic_ref(AUTHORITY).expect("basic_ref"); + assert_eq!(account, None, "an all-zero RPC answer must read as a nonexistent account"); +} + +/// A balance alone keeps the account existing (e.g. a plain EOA that only +/// ever received funds). +#[tokio::test(flavor = "multi_thread")] +async fn test_forked_balance_only_account_exists() { + let server = MockRpcServer::start().await; + let state = forked_state(&server, "0x1", "0x0", "0x").await; + + let account = state.basic_ref(AUTHORITY).expect("basic_ref").expect("account must exist"); + assert_eq!(account.balance, U256::from(1)); + assert_eq!(account.nonce, 0); +} + +/// A nonce alone keeps the account existing (e.g. an EOA that spent its +/// entire balance on fees). +#[tokio::test(flavor = "multi_thread")] +async fn test_forked_nonce_only_account_exists() { + let server = MockRpcServer::start().await; + let state = forked_state(&server, "0x0", "0x1", "0x").await; + + let account = state.basic_ref(AUTHORITY).expect("basic_ref").expect("account must exist"); + assert_eq!(account.balance, U256::ZERO); + assert_eq!(account.nonce, 1); +} + +/// Code alone keeps the account existing (e.g. a contract with neither +/// balance nor nonce is still a contract). +#[tokio::test(flavor = "multi_thread")] +async fn test_forked_code_only_account_exists() { + let server = MockRpcServer::start().await; + let state = forked_state(&server, "0x0", "0x0", "0x6001").await; + + let account = state.basic_ref(AUTHORITY).expect("basic_ref").expect("account must exist"); + assert!(!account.is_empty_code_hash(), "the fetched code must be reflected in the code hash"); +} + +// ─── Execution-level: the EIP-7702 refund consumer ─────────────────────────── + +/// Replay a type-4 transaction whose single authorization names `AUTHORITY`, +/// returning the gas it used. The access list pads execution gas so the +/// EIP-3529 refund cap (`gas_used / 5`) stays above the full 12,500 refund — +/// otherwise the two scenarios' gas would differ by the cap, not the refund. +async fn type4_gas_used(server: &MockRpcServer, spec: MegaSpecId) -> u64 { + let mut state = forked_state(server, "0x0", "0x0", "0x").await; + state.set_account_balance(CALLER, U256::from(10).pow(U256::from(18))); + + let authorization = RecoveredAuthorization::new_unchecked( + // chain_id 0 = valid on any chain, so the context's chain id is irrelevant. + Authorization { chain_id: U256::ZERO, address: DELEGATE, nonce: 0 }, + RecoveredAuthority::Valid(AUTHORITY), + ); + let access_list = AccessList(vec![AccessListItem { + address: CALLEE, + storage_keys: (0u64..40).map(|i| B256::from(U256::from(i))).collect(), + }]); + let tx_env = TxEnvBuilder::default() + .caller(CALLER) + .call(CALLEE) + .gas_limit(1_000_000) + .access_list(access_list) + .authorization_list_recovered(vec![authorization]) + .build_fill(); + + let context = + MegaContext::new(&mut state, spec).with_external_envs(EvmeExternalEnvs::new().into()); + let mut evm = MegaEvm::new(context); + let mut tx = MegaTransaction::new(tx_env); + tx.enveloped_tx = Some(Bytes::new()); + let outcome = evm.transact(tx).expect("type-4 replay must execute"); + + assert!( + matches!(outcome.result, ExecutionResult::Success { .. }), + "type-4 replay must succeed, got {:?}", + outcome.result, + ); + assert_delegated(&outcome.state); + outcome.result.tx_gas_used() +} + +/// The authorization must have actually applied — a skipped authorization +/// would make the gas comparison vacuous. +fn assert_delegated(state: &EvmState) { + let authority = state.get(&AUTHORITY).expect("the authority must be in the post-state"); + assert_eq!(authority.info.nonce, 1, "the applied authorization must bump the authority nonce"); + assert!( + authority.info.code.as_ref().is_some_and(|code| code.is_eip7702()), + "the authority must carry the delegation designator", + ); +} + +/// A brand-new authority (all-zero on RPC) must not earn the existing-account +/// refund: the replayed `gasUsed` is exactly 12,500 above the run whose +/// authority exists. Before normalization both runs were refunded and a replay +/// under-reported `gasUsed` against the on-chain receipt. +#[rstest] +#[case::rex5(MegaSpecId::REX5)] +#[case::rex6(MegaSpecId::REX6)] +#[tokio::test(flavor = "multi_thread")] +async fn test_type4_fresh_authority_is_not_refunded(#[case] spec: MegaSpecId) { + let fresh_server = MockRpcServer::start().await; + let gas_fresh_authority = type4_gas_used(&fresh_server, spec).await; + + // Same transaction, but every RPC account (including the authority) holds + // 1 wei. The caller and its balance come from the prestate override in + // both runs, so the authority's existence is the only difference. + let existing_server = MockRpcServer::start().await; + existing_server.respond_method_result("eth_getBalance", "0x1", 1).await; + let gas_existing_authority = type4_gas_used(&existing_server, spec).await; + + assert_eq!( + gas_fresh_authority, + gas_existing_authority + EXISTING_AUTHORITY_REFUND, + "a fresh authority must not earn the 12,500 existing-account refund", + ); +} diff --git a/bin/mega-evme/tests/batch_cache_default.rs b/bin/mega-evme/tests/batch_cache_default.rs new file mode 100644 index 00000000..14b4df9f --- /dev/null +++ b/bin/mega-evme/tests/batch_cache_default.rs @@ -0,0 +1,291 @@ +//! Integration tests for the batch-mode on-disk cache default. +//! +//! Batch replay (`--tx-file` / `--block`) engages the on-disk RPC cache only +//! when the invocation asks for it explicitly. The clean-exit persist +//! re-reads, merges, and atomically rewrites the whole per-chain cache file +//! under a cross-process lock, so its cost grows with the file and serializes +//! across concurrent processes — while a linear history scan gets almost no +//! cache hits in return. With the default in place a batch exit performs zero +//! disk-cache work, making its cost independent of any cache a machine has +//! accumulated. Single-transaction replay keeps the previous default. +//! +//! `--rpc.clear-cache` opts a batch run back in the same way `--rpc.cache-dir` +//! does: deleting the cache file is a request that only means something while +//! the disk cache is engaged, so forcing it off would make the documented +//! recovery flag a no-op and leave the polluted file for the next run. +//! +//! The tests point the child's platform cache directory into a temp dir via +//! `HOME` / `XDG_CACHE_HOME`, so the real user cache is never touched. + +use std::{ + path::{Path, PathBuf}, + process::Command, +}; + +use tempfile::TempDir; + +mod common; +use common::MockRpcServer; + +/// Any syntactically valid transaction hash; every lookup fails at the mock. +const TX: &str = "0x1111111111111111111111111111111111111111111111111111111111111111"; + +/// A fake home directory the child process resolves its platform cache dir in. +struct FakeHome { + dir: TempDir, +} + +impl FakeHome { + fn new() -> Self { + Self { dir: tempfile::tempdir().expect("tempdir") } + } + + fn path(&self) -> &Path { + self.dir.path() + } + + /// Where the child's default per-chain cache file lands for chain 4326. + /// + /// Mirrors `dirs::cache_dir()` under the overridden environment: macOS + /// resolves `$HOME/Library/Caches`, other unixes `$XDG_CACHE_HOME` (which + /// the tests always set). + fn default_cache_file(&self) -> PathBuf { + let base = if cfg!(target_os = "macos") { + self.path().join("Library/Caches") + } else { + self.path().join("xdg-cache") + }; + base.join("mega-evme/rpc/rpc-cache-4326.json") + } + + /// Every `rpc-cache-*.json` anywhere under the fake home. + fn cache_files(&self) -> Vec { + fn walk(dir: &Path, hits: &mut Vec) { + let Ok(entries) = std::fs::read_dir(dir) else { return }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + walk(&path, hits); + } else if let Some(name) = path.file_name().and_then(|n| n.to_str()) { + if name.starts_with("rpc-cache-") && name.ends_with(".json") { + hits.push(path); + } + } + } + } + let mut hits = Vec::new(); + walk(self.path(), &mut hits); + hits + } +} + +/// Run `mega-evme replay` with the platform cache dir redirected into `home`. +fn replay(home: &FakeHome, args: &[&str]) { + let output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .arg("replay") + .args(args) + .env("HOME", home.path()) + .env("XDG_CACHE_HOME", home.path().join("xdg-cache")) + .output() + .expect("failed to run mega-evme"); + // Every scenario here replays a transaction the mock cannot answer, so the + // run itself fails; the assertions are about the cache file side effects. + assert!(output.status.code().is_some(), "mega-evme must exit, not die on a signal: {output:?}"); +} + +/// A mock whose chain id resolves (4326) and whose every other request fails +/// without triggering the retry layer. +async fn failing_mock() -> MockRpcServer { + let server = MockRpcServer::start().await; + server.respond_eth_chain_id(4326, 1).await; + server.respond_jsonrpc_error(-32601, "no such method", 2).await; + server +} + +/// Write a `--tx-file` under the fake home and return its path. +fn tx_file(home: &FakeHome) -> PathBuf { + let path = home.path().join("targets.txt"); + std::fs::write(&path, format!("{TX}\n")).expect("write tx file"); + path +} + +/// A default-flag batch run must neither read nor write any on-disk cache: +/// a pre-existing default cache file survives byte-identical (even though its +/// content is garbage a load would have rejected), and no new cache file +/// appears anywhere. The exit therefore does no work proportional to the +/// cache a machine has accumulated, no matter how many targets were replayed. +#[tokio::test(flavor = "multi_thread")] +async fn test_batch_default_leaves_disk_cache_untouched() { + let home = FakeHome::new(); + let seeded = home.default_cache_file(); + std::fs::create_dir_all(seeded.parent().expect("cache file has a parent")).expect("mkdir"); + std::fs::write(&seeded, b"not even json").expect("seed cache file"); + + let server = failing_mock().await; + let targets = tx_file(&home); + + replay( + &home, + &[ + "--tx-file", + targets.to_str().expect("utf-8"), + "--rpc", + &server.uri(), + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + "--json", + ], + ); + + let bytes = std::fs::read(&seeded).expect("seeded file must still exist"); + assert_eq!(bytes, b"not even json", "the seeded default cache file must stay byte-identical"); + assert_eq!( + home.cache_files(), + vec![seeded], + "no other cache file may appear anywhere under the fake home", + ); +} + +/// An explicit `--rpc.clear-cache` opts a batch run back into the disk cache at +/// the default path: the seeded file is deleted before the run and a fresh cache +/// file is persisted on exit. Forcing the cache off instead would make the flag +/// parse and do nothing, leaving the polluted file for the next non-batch run. +#[tokio::test(flavor = "multi_thread")] +async fn test_batch_clear_cache_clears_and_repersists_default_path() { + let home = FakeHome::new(); + let seeded = home.default_cache_file(); + std::fs::create_dir_all(seeded.parent().expect("cache file has a parent")).expect("mkdir"); + std::fs::write(&seeded, b"not even json").expect("seed cache file"); + + let server = failing_mock().await; + let targets = tx_file(&home); + + replay( + &home, + &[ + "--tx-file", + targets.to_str().expect("utf-8"), + "--rpc", + &server.uri(), + "--rpc.clear-cache", + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + "--json", + ], + ); + + let bytes = std::fs::read(&seeded).expect("a fresh cache file must exist after the run"); + assert_ne!( + bytes, b"not even json", + "the seeded cache file must have been cleared, not carried forward", + ); + // The clear only happened because the disk cache was engaged, so the exit + // persist must have written a well-formed provider cache in its place. + serde_json::from_slice::(&bytes) + .expect("the persisted cache file must be valid JSON"); + assert_eq!( + home.cache_files(), + vec![seeded], + "the run must not create a cache file anywhere else under the fake home", + ); +} + +/// `--rpc.no-cache-file` wins over `--rpc.clear-cache`: with no cache file in +/// play there is nothing to delete, load, or persist, so a seeded file survives +/// byte-identical. Batch mode passes the pair through unchanged, so both target +/// forms behave the same way. +#[tokio::test(flavor = "multi_thread")] +async fn test_no_cache_file_wins_over_clear_cache_in_both_modes() { + for batch in [false, true] { + let home = FakeHome::new(); + let seeded = home.default_cache_file(); + std::fs::create_dir_all(seeded.parent().expect("cache file has a parent")).expect("mkdir"); + std::fs::write(&seeded, b"not even json").expect("seed cache file"); + + let server = failing_mock().await; + let uri = server.uri(); + let targets = tx_file(&home); + + let mut args = + if batch { vec!["--tx-file", targets.to_str().expect("utf-8")] } else { vec![TX] }; + args.extend_from_slice(&[ + "--rpc", + &uri, + "--rpc.no-cache-file", + "--rpc.clear-cache", + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + "--json", + ]); + replay(&home, &args); + + let bytes = std::fs::read(&seeded).expect("the seeded file must still exist"); + assert_eq!( + bytes, b"not even json", + "--rpc.no-cache-file must keep the disk cache out of play (batch = {batch})", + ); + assert_eq!( + home.cache_files(), + vec![seeded], + "no cache file may be written anywhere (batch = {batch})", + ); + } +} + +/// An explicit `--rpc.cache-dir` opts a batch run back into persistence: the +/// per-chain cache file is written on exit. +#[tokio::test(flavor = "multi_thread")] +async fn test_batch_explicit_cache_dir_still_persists() { + let home = FakeHome::new(); + let cache_dir = home.path().join("explicit-cache"); + + let server = failing_mock().await; + let targets = tx_file(&home); + + replay( + &home, + &[ + "--tx-file", + targets.to_str().expect("utf-8"), + "--rpc", + &server.uri(), + "--rpc.cache-dir", + cache_dir.to_str().expect("utf-8"), + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + "--json", + ], + ); + + assert!( + cache_dir.join("rpc-cache-4326.json").exists(), + "an explicit --rpc.cache-dir must persist the cache file on exit", + ); +} + +/// Single-transaction replay keeps the previous default: the per-chain cache +/// file is persisted into the platform cache directory. +#[tokio::test(flavor = "multi_thread")] +async fn test_single_replay_still_persists_by_default() { + let home = FakeHome::new(); + + let server = failing_mock().await; + + replay( + &home, + &[TX, "--rpc", &server.uri(), "--rpc.max-retries", "0", "--rpc.backoff-ms", "1", "--json"], + ); + + assert!( + home.default_cache_file().exists(), + "single-transaction replay must keep persisting the default cache file", + ); +} diff --git a/bin/mega-evme/tests/cache_clear_lock.rs b/bin/mega-evme/tests/cache_clear_lock.rs new file mode 100644 index 00000000..ee5cdd87 --- /dev/null +++ b/bin/mega-evme/tests/cache_clear_lock.rs @@ -0,0 +1,194 @@ +//! Cross-process serialization for `--rpc.clear-cache` against the sidecar lock. +//! +//! Clear and persist share the exclusive advisory lock on `.lock`. A +//! clear that unlinks without that lock can race a writer mid re-read-merge- +//! rename: the writer's rename lands after the clear (undoing it), or the clear +//! deletes the file the locked writer just re-read. These tests hold the lock +//! in the test process, spawn a real `mega-evme` that takes the clear-cache +//! path, show the clear does not complete while the lock is held, then release +//! so the clear finishes. + +use std::{ + fs::{self, File, OpenOptions}, + path::{Path, PathBuf}, + process::{Child, Command, Stdio}, + time::{Duration, Instant}, +}; + +mod common; +use common::MockRpcServer; + +/// How long the lock is held while the spawned clear must make no progress. +const HOLD: Duration = Duration::from_secs(2); + +/// Upper bound on how long clear may take after the lock is released. +const COMPLETION_DEADLINE: Duration = Duration::from_secs(60); + +/// The advisory lock sidecar the binary locks for `target`. +fn sidecar(target: &Path) -> PathBuf { + let mut os = target.as_os_str().to_owned(); + os.push(".lock"); + PathBuf::from(os) +} + +/// Take the exclusive lock on the cache file's sidecar and keep it until the +/// returned handle is dropped. +fn hold_cache_lock(target: &Path) -> File { + let file = OpenOptions::new() + .create(true) + .read(true) + .write(true) + .truncate(false) + .open(sidecar(target)) + .expect("open the cache sidecar"); + file.lock().expect("hold the cache lock"); + file +} + +/// Spawn a single-tx online `replay` that hits `build_provider` with +/// `--rpc.clear-cache`. After the clear the lookup fails; we only care that +/// clear ran under the lock. +fn spawn_clear_cache(rpc: &str, cache_dir: &Path) -> Child { + Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args([ + "replay", + "--rpc", + rpc, + "--rpc.cache-dir", + cache_dir.to_str().expect("utf-8 cache dir"), + "--rpc.clear-cache", + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + // Dummy hash: chain-id is enough for clear; the later tx fetch fails. + "0x0000000000000000000000000000000000000000000000000000000000000001", + ]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("spawn mega-evme replay --rpc.clear-cache") +} + +/// Assert the spawned process makes no progress for the whole hold window. +fn assert_blocked_while_held(child: &mut Child) { + let deadline = Instant::now() + HOLD; + while Instant::now() < deadline { + if let Some(status) = child.try_wait().expect("poll the clear-cache process") { + panic!("clear-cache completed while the cache lock was held (status {status})",); + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!( + child.try_wait().expect("poll the clear-cache process").is_none(), + "clear-cache must still be waiting for the cache lock", + ); +} + +/// Wait until the child exits (success or failure) after the lock is free. +fn finish(mut child: Child) -> std::process::Output { + let deadline = Instant::now() + COMPLETION_DEADLINE; + loop { + if child.try_wait().expect("poll the clear-cache process").is_some() { + break; + } + assert!( + Instant::now() < deadline, + "clear-cache did not finish within {COMPLETION_DEADLINE:?} of the lock being released", + ); + std::thread::sleep(Duration::from_millis(20)); + } + child.wait_with_output().expect("collect clear-cache output") +} + +/// Clear waits on a held sidecar lock, does not unlink while blocked, and +/// wipes the seeded content only after the lock is released. +/// +/// The file may reappear empty if a later clean-exit-adjacent persist runs +/// after the clear (online replay still installs a disk store); the proof is +/// that the pre-clear seed is gone, not that the path stays unlinked. +#[tokio::test(flavor = "multi_thread")] +async fn test_clear_cache_serializes_with_a_held_sidecar_lock() { + let server = MockRpcServer::start().await; + let chain_id: u64 = 55; + server.respond_eth_chain_id(chain_id, 1).await; + // After clear, eth_getTransactionByHash (and friends) may be called; a + // null result fails the run cleanly without hanging. + server.respond_jsonrpc_null_result(10).await; + + let dir = tempfile::tempdir().expect("tempdir"); + let cache_file = dir.path().join(format!("rpc-cache-{chain_id}.json")); + let seed = r#"[{"key":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","value":"seed-before-clear"}]"#; + fs::write(&cache_file, seed).expect("seed cache file"); + assert!(cache_file.exists()); + + let lock = hold_cache_lock(&cache_file); + let mut child = spawn_clear_cache(&server.uri(), dir.path()); + assert_blocked_while_held(&mut child); + assert_eq!( + fs::read_to_string(&cache_file).expect("read while held"), + seed, + "clear must not unlink while the sidecar lock is held", + ); + + drop(lock); + let out = finish(child); + let after = fs::read_to_string(&cache_file).unwrap_or_default(); + assert!( + !after.contains("seed-before-clear"), + "clear must wipe the seeded content once the lock is free.\n\ + after={after}\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr), + ); +} + +/// A concurrent writer that lands under the lock while clear is queued must +/// still be wiped: clear's critical section is unlink + exists-check + load, +/// so the file written just before clear acquires cannot be reloaded into the +/// clearing session. +/// +/// Inverse of `test_clear_cache_serializes_with_a_held_sidecar_lock`: that test +/// seeds before the hold and proves clear does not act while blocked; this one +/// writes only while clear is blocked (as a clean-exit persist would, under +/// the same lock) and proves the injected entries do not survive the clear. +#[tokio::test(flavor = "multi_thread")] +async fn test_clear_cache_wipes_file_written_while_queued_on_lock() { + let server = MockRpcServer::start().await; + let chain_id: u64 = 56; + server.respond_eth_chain_id(chain_id, 1).await; + server.respond_jsonrpc_null_result(10).await; + + let dir = tempfile::tempdir().expect("tempdir"); + let cache_file = dir.path().join(format!("rpc-cache-{chain_id}.json")); + // No seed before the hold: the only pollution is what the concurrent + // writer leaves under the lock while clear waits. + assert!(!cache_file.exists(), "precondition: cache file must be absent"); + + let lock = hold_cache_lock(&cache_file); + let mut child = spawn_clear_cache(&server.uri(), dir.path()); + assert_blocked_while_held(&mut child); + + // Concurrent persist finishes under the lock clear is waiting for: its + // rename lands, then it releases — clear acquires next and must treat this + // file as the one to wipe, not as content to load after an early unlock. + let injected = r#"[{"key":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","value":"injected-while-clear-queued"}]"#; + fs::write(&cache_file, injected).expect("write while clear is queued"); + assert_eq!( + fs::read_to_string(&cache_file).expect("read injected"), + injected, + "injected file must still be present while clear is blocked", + ); + + drop(lock); + let out = finish(child); + let after = fs::read_to_string(&cache_file).unwrap_or_default(); + assert!( + !after.contains("injected-while-clear-queued"), + "clear must delete the concurrent write and start empty; injected \ + entries must not reappear via load-then-persist.\n\ + after={after}\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr), + ); +} diff --git a/bin/mega-evme/tests/cache_merge_diagnostics.rs b/bin/mega-evme/tests/cache_merge_diagnostics.rs new file mode 100644 index 00000000..5f5ac772 --- /dev/null +++ b/bin/mega-evme/tests/cache_merge_diagnostics.rs @@ -0,0 +1,208 @@ +//! Binary-level tests for the diagnostics `cache merge` owes the user. +//! +//! `cache merge` has safeguards that can only warn: chain identity that cannot +//! be derived from a filename, and an unreadable output file that the merge is +//! about to replace. Both report a result that is silently wrong or lossy, and +//! both are worthless if the user never sees them. +//! +//! The CLI initializes tracing with the filter at `off` unless `-v` flags or +//! `RUST_LOG` raise it, so these cannot be asserted through a tracing capture +//! in-process: doing that would prove the event is emitted while the default +//! command line still shows nothing. These tests therefore run the real binary +//! with no verbosity flags and `RUST_LOG` removed from its environment, and +//! read what an operator would actually see on stderr. + +use std::{ + fs, + path::Path, + process::{Command, Output}, +}; + +use alloy_primitives::B256; +use serde_json::{json, Value}; + +/// One `{key, value}` provider-cache entry, keyed by a repeated byte. +fn kv(byte: u8, value: &str) -> Value { + json!({ "key": B256::repeat_byte(byte), "value": value }) +} + +/// A capture envelope holding `entries`. +fn envelope(entries: Vec) -> Value { + json!({ "version": 1, "chain_id": 4326, "cache": entries, "external_env": null }) +} + +/// Run `mega-evme cache merge` exactly as a default command line would: no `-v` +/// flags, and no inherited `RUST_LOG` that could raise the filter for us. +fn run_merge(inputs: &[&Path], output: &Path) -> Output { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_mega-evme")); + cmd.args(["cache", "merge"]); + for input in inputs { + cmd.arg(input); + } + cmd.arg("--output").arg(output); + cmd.env_remove("RUST_LOG"); + cmd.output().expect("run mega-evme cache merge") +} + +/// Assert the merge succeeded, and return `(stdout, stderr)`. +fn succeeds(out: &Output) -> (String, String) { + let stdout = String::from_utf8_lossy(&out.stdout).into_owned(); + let stderr = String::from_utf8_lossy(&out.stderr).into_owned(); + assert_eq!( + out.status.code(), + Some(0), + "the merge must still succeed.\nstdout: {stdout}\nstderr: {stderr}", + ); + (stdout, stderr) +} + +/// Value stored for the entry keyed by a repeated byte, if present. +fn value_of(entries: &[Value], byte: u8) -> Option { + let key = json!(B256::repeat_byte(byte)); + entries + .iter() + .find(|e| e.get("key") == Some(&key)) + .and_then(|e| e.get("value")) + .and_then(Value::as_str) + .map(str::to_owned) +} + +/// Read the merged provider-cache array at `path`. +fn read_provider(path: &Path) -> Vec { + serde_json::from_str(&fs::read_to_string(path).expect("read merged output")) + .expect("merged output is a provider-cache array") +} + +/// Filenames that carry no chain id leave the cross-chain safeguard unable to +/// run. The merge proceeds — renamed shards are legitimate — but the user is +/// told, on stderr, without asking for verbosity. +#[test] +fn test_cache_merge_warns_on_stderr_when_chain_identity_cannot_be_validated() { + let dir = tempfile::tempdir().expect("tempdir"); + let a = dir.path().join("worker-a.json"); + let b = dir.path().join("worker-b.json"); + let out = dir.path().join("merged.json"); + + fs::write(&a, serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()).expect("write a"); + fs::write(&b, serde_json::to_string(&vec![kv(2, "from-b")]).unwrap()).expect("write b"); + + let output = run_merge(&[&a, &b], &out); + let (stdout, stderr) = succeeds(&output); + + assert!( + stderr.contains("chain identity cannot be validated"), + "the safeguard must announce that it could not run: stderr={stderr}", + ); + assert!( + stderr.contains("worker-a.json") && stderr.contains("worker-b.json"), + "each unvalidatable file must be named: stderr={stderr}", + ); + + // Behavior is otherwise unchanged: the merge still produced the union. + let merged = read_provider(&out); + assert_eq!(value_of(&merged, 1).as_deref(), Some("from-a"), "{merged:?}"); + assert_eq!(value_of(&merged, 2).as_deref(), Some("from-b"), "{merged:?}"); + assert_eq!(merged.len(), 2, "{merged:?}"); + assert!(stdout.contains("Merged"), "the summary still goes to stdout: {stdout}"); +} + +/// The warning is specific to unvalidatable names: filenames that agree on a +/// chain id let the safeguard run, and a quiet merge stays quiet. +#[test] +fn test_cache_merge_is_silent_when_filenames_agree_on_the_chain_id() { + let dir = tempfile::tempdir().expect("tempdir"); + let worker0 = dir.path().join("worker0"); + let worker1 = dir.path().join("worker1"); + let merged_dir = dir.path().join("merged"); + for d in [&worker0, &worker1, &merged_dir] { + fs::create_dir(d).expect("create dir"); + } + let a = worker0.join("rpc-cache-4326.json"); + let b = worker1.join("rpc-cache-4326.json"); + let out = merged_dir.join("rpc-cache-4326.json"); + + fs::write(&a, serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()).expect("write a"); + fs::write(&b, serde_json::to_string(&vec![kv(2, "from-b")]).unwrap()).expect("write b"); + + let output = run_merge(&[&a, &b], &out); + let (_, stderr) = succeeds(&output); + + assert!( + !stderr.contains("chain identity"), + "a validated same-chain merge must not warn: stderr={stderr}", + ); + assert!(stderr.is_empty(), "a clean merge writes nothing to stderr: stderr={stderr}"); + + let merged = read_provider(&out); + assert_eq!(merged.len(), 2, "{merged:?}"); +} + +/// An unreadable provider output is replaced by the merged inputs, dropping +/// whatever it held. That data loss reaches stderr at default verbosity too. +#[test] +fn test_cache_merge_warns_on_stderr_when_replacing_an_unreadable_provider_output() { + let dir = tempfile::tempdir().expect("tempdir"); + let inputs = dir.path().join("inputs"); + let merged_dir = dir.path().join("merged"); + for d in [&inputs, &merged_dir] { + fs::create_dir(d).expect("create dir"); + } + // Convention-following names on both sides, so the only warning that can + // fire here is the one under test. + let a = inputs.join("rpc-cache-4326.json"); + let out = merged_dir.join("rpc-cache-4326.json"); + + fs::write(&a, serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()).expect("write a"); + fs::write(&out, "not-json{{{").expect("write a corrupt output"); + + let output = run_merge(&[&a], &out); + let (_, stderr) = succeeds(&output); + + assert!( + stderr.contains("Replacing the existing merge output"), + "the replacement must be announced: stderr={stderr}", + ); + assert!( + stderr.contains("discarded"), + "the user must be told entries are lost: stderr={stderr}", + ); + assert!( + !stderr.contains("chain identity"), + "no chain-identity warning is due here: stderr={stderr}", + ); + + let merged = read_provider(&out); + assert_eq!(value_of(&merged, 1).as_deref(), Some("from-a"), "{merged:?}"); + assert_eq!(merged.len(), 1, "{merged:?}"); +} + +/// The envelope shape replaces an unreadable output the same way, and warns the +/// same way. +#[test] +fn test_cache_merge_warns_on_stderr_when_replacing_an_unreadable_envelope_output() { + let dir = tempfile::tempdir().expect("tempdir"); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + fs::write(&a, serde_json::to_string_pretty(&envelope(vec![kv(1, "from-a")])).unwrap()) + .expect("write a"); + fs::write(&out, "not-json{{{").expect("write a corrupt output"); + + let output = run_merge(&[&a], &out); + let (_, stderr) = succeeds(&output); + + assert!( + stderr.contains("Replacing the existing merge output"), + "the replacement must be announced: stderr={stderr}", + ); + assert!( + stderr.contains("discarded"), + "the user must be told entries are lost: stderr={stderr}", + ); + + let merged: Value = serde_json::from_str(&fs::read_to_string(&out).expect("read output")) + .expect("merged output is an envelope"); + let entries = merged["cache"].as_array().expect("cache array").clone(); + assert_eq!(value_of(&entries, 1).as_deref(), Some("from-a"), "{entries:?}"); + assert_eq!(entries.len(), 1, "{entries:?}"); +} diff --git a/bin/mega-evme/tests/cache_merge_lock.rs b/bin/mega-evme/tests/cache_merge_lock.rs new file mode 100644 index 00000000..2c75f61f --- /dev/null +++ b/bin/mega-evme/tests/cache_merge_lock.rs @@ -0,0 +1,207 @@ +//! Two-process serialization tests for the `cache merge` output lock. +//! +//! `cache merge` writes a file a live `mega-evme` run may be persisting to at +//! the same time. Both writers take the exclusive advisory lock on the output's +//! sidecar and re-read the file while holding it, so neither side's entries are +//! lost to whichever rename lands last. +//! +//! Mutual exclusion cannot be demonstrated inside one process: a single-process +//! test that "takes the lock" and then calls the merge in-process either +//! deadlocks or proves nothing about a second process. These tests hold the +//! sidecar lock in the test process, spawn the real binary, show it makes no +//! progress while the lock is held, write a concurrent writer's entries under +//! that same lock, and only then release — so the merge's output can contain +//! those entries only by re-reading the file after it acquired the lock. + +use std::{ + fs::{self, File, OpenOptions}, + path::{Path, PathBuf}, + process::{Child, Command, Stdio}, + time::{Duration, Instant}, +}; + +use alloy_primitives::B256; +use serde_json::{json, Value}; + +/// How long the lock is held while the spawned merge must make no progress. +/// +/// Merging two one-entry files takes milliseconds, so staying alive for this +/// long is only explainable by the lock. +const HOLD: Duration = Duration::from_secs(2); + +/// Upper bound on how long the merge may take after the lock is released. +/// Generous on purpose: this bound exists to fail a hung merge with a message +/// instead of hanging the suite, not to measure anything. +const COMPLETION_DEADLINE: Duration = Duration::from_secs(60); + +/// One `{key, value}` cache entry, keyed by a repeated byte. +fn kv(byte: u8, value: &str) -> Value { + json!({ "key": B256::repeat_byte(byte), "value": value }) +} + +/// The advisory lock sidecar the binary locks for `output`. +fn sidecar(output: &Path) -> PathBuf { + let mut os = output.as_os_str().to_owned(); + os.push(".lock"); + PathBuf::from(os) +} + +/// Take the exclusive lock on the output's sidecar and keep it until the +/// returned handle is dropped. +fn hold_output_lock(output: &Path) -> File { + let file = OpenOptions::new() + .create(true) + .read(true) + .write(true) + .truncate(false) + .open(sidecar(output)) + .expect("open the output sidecar"); + file.lock().expect("hold the output lock"); + file +} + +/// Spawn the real `mega-evme cache merge` process. +fn spawn_merge(inputs: &[&Path], output: &Path) -> Child { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_mega-evme")); + cmd.args(["cache", "merge"]); + for input in inputs { + cmd.arg(input); + } + cmd.arg("--output").arg(output).stdout(Stdio::piped()).stderr(Stdio::piped()); + cmd.spawn().expect("spawn mega-evme cache merge") +} + +/// Assert the spawned merge makes no progress for the whole hold window. +fn assert_blocked_while_held(child: &mut Child) { + let deadline = Instant::now() + HOLD; + while Instant::now() < deadline { + if let Some(status) = child.try_wait().expect("poll the merge") { + panic!("the merge completed while the output lock was held (status {status})"); + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!( + child.try_wait().expect("poll the merge").is_none(), + "the merge must still be waiting for the output lock", + ); +} + +/// Wait for the released merge to finish and return its stdout. +fn finish(mut child: Child) -> String { + let deadline = Instant::now() + COMPLETION_DEADLINE; + loop { + if child.try_wait().expect("poll the merge").is_some() { + break; + } + assert!( + Instant::now() < deadline, + "the merge did not finish within {COMPLETION_DEADLINE:?} of the lock being released", + ); + std::thread::sleep(Duration::from_millis(20)); + } + let out = child.wait_with_output().expect("collect the merge output"); + assert!( + out.status.success(), + "the merge must succeed once the lock is free.\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr), + ); + String::from_utf8_lossy(&out.stdout).into_owned() +} + +/// Value stored for the entry keyed by a repeated byte, if present. +fn value_of(entries: &[Value], byte: u8) -> Option { + let key = json!(B256::repeat_byte(byte)); + entries + .iter() + .find(|e| e.get("key") == Some(&key)) + .and_then(|e| e.get("value")) + .and_then(Value::as_str) + .map(str::to_owned) +} + +/// Provider shape: the merge waits for the lock, then folds in what a +/// concurrent writer left in the output while it waited. +#[test] +fn test_cache_merge_serializes_with_a_concurrent_provider_writer() { + let dir = tempfile::tempdir().expect("tempdir"); + let a = dir.path().join("a.json"); + let b = dir.path().join("b.json"); + let out = dir.path().join("out.json"); + + fs::write(&a, serde_json::to_string(&vec![kv(1, "from-a")]).unwrap()).expect("write a"); + fs::write(&b, serde_json::to_string(&vec![kv(2, "from-b")]).unwrap()).expect("write b"); + + let lock = hold_output_lock(&out); + let mut child = spawn_merge(&[&a, &b], &out); + assert_blocked_while_held(&mut child); + + // A concurrent writer lands its entries the way a clean-exit persist does: + // while it holds the same lock the merge is waiting for. + fs::write(&out, serde_json::to_string(&vec![kv(9, "from-concurrent-writer")]).unwrap()) + .expect("concurrent write"); + drop(lock); + + let stdout = finish(child); + + let merged: Vec = + serde_json::from_str(&fs::read_to_string(&out).expect("read merged output")) + .expect("merged output is a provider-cache array"); + assert_eq!( + value_of(&merged, 9).as_deref(), + Some("from-concurrent-writer"), + "the entry written while the merge was blocked must survive: {merged:?}", + ); + assert_eq!(value_of(&merged, 1).as_deref(), Some("from-a"), "input entry lost: {merged:?}"); + assert_eq!(value_of(&merged, 2).as_deref(), Some("from-b"), "input entry lost: {merged:?}"); + assert_eq!(merged.len(), 3, "the union is exactly both sides: {merged:?}"); + assert!( + stdout.contains("already in the output"), + "the summary must report the folded-in entries: {stdout}", + ); +} + +/// Envelope shape: same protocol, same guarantee. +#[test] +fn test_cache_merge_serializes_with_a_concurrent_envelope_writer() { + let dir = tempfile::tempdir().expect("tempdir"); + let a = dir.path().join("a.json"); + let out = dir.path().join("out.json"); + + let envelope = |entries: Vec| { + json!({ + "version": 1, + "chain_id": 4326, + "cache": entries, + "external_env": null, + }) + }; + fs::write(&a, serde_json::to_string_pretty(&envelope(vec![kv(1, "from-a")])).unwrap()) + .expect("write a"); + + let lock = hold_output_lock(&out); + let mut child = spawn_merge(&[&a], &out); + assert_blocked_while_held(&mut child); + + fs::write( + &out, + serde_json::to_string_pretty(&envelope(vec![kv(9, "from-concurrent-writer")])).unwrap(), + ) + .expect("concurrent write"); + drop(lock); + + finish(child); + + let merged: Value = + serde_json::from_str(&fs::read_to_string(&out).expect("read merged output")) + .expect("merged output is an envelope"); + let entries = merged["cache"].as_array().expect("cache array").clone(); + assert_eq!( + value_of(&entries, 9).as_deref(), + Some("from-concurrent-writer"), + "the entry written while the merge was blocked must survive: {entries:?}", + ); + assert_eq!(value_of(&entries, 1).as_deref(), Some("from-a"), "input entry lost: {entries:?}"); + assert_eq!(entries.len(), 2, "the union is exactly both sides: {entries:?}"); + assert_eq!(merged["chain_id"], json!(4326)); +} diff --git a/bin/mega-evme/tests/common/mod.rs b/bin/mega-evme/tests/common/mod.rs index 54dfc716..6d188a91 100644 --- a/bin/mega-evme/tests/common/mod.rs +++ b/bin/mega-evme/tests/common/mod.rs @@ -8,10 +8,68 @@ #![allow(dead_code)] // Each test binary uses a different subset of helpers. +use std::{ + collections::HashSet, + path::{Path, PathBuf}, + process::Command, + sync::{Mutex, OnceLock}, +}; + use clap::Parser; use mega_evme::common::RpcArgs; +use tempfile::TempDir; use wiremock::{matchers, Mock, MockServer, ResponseTemplate}; +/// Resolve a fixture in `tests/fixtures/` by name, extracting it if it is +/// stored compressed. +/// +/// A fixture is either the file itself, or a `.tar.gz` holding exactly +/// that one file. Compression is worth it only where the raw file would bloat a +/// pull-request diff — git already compresses blobs, so it buys little on its +/// own, and a compressed blob cannot delta against its previous revision. +/// +/// Archives are extracted once per test binary into a temporary directory that +/// lives for the whole run. Extraction shells out to `tar` rather than linking a +/// decompressor: every platform that runs these tests has one, and this is the +/// only place that reads an archive. +pub(crate) fn fixture(name: &str) -> PathBuf { + let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures"); + let plain = dir.join(name); + if plain.is_file() { + return plain; + } + + let archive = dir.join(format!("{name}.tar.gz")); + assert!( + archive.is_file(), + "no fixture named {name}: neither {} nor {} exists", + plain.display(), + archive.display(), + ); + + static ROOT: OnceLock = OnceLock::new(); + static EXTRACTED: OnceLock>> = OnceLock::new(); + let root = ROOT.get_or_init(|| { + tempfile::tempdir().expect("failed to create a temp dir for extracted fixtures") + }); + let mut extracted = + EXTRACTED.get_or_init(|| Mutex::new(HashSet::new())).lock().expect("fixture lock"); + + let path = root.path().join(name); + if extracted.insert(name.to_string()) { + let status = Command::new("tar") + .arg("-xzf") + .arg(&archive) + .arg("-C") + .arg(root.path()) + .status() + .expect("failed to run tar"); + assert!(status.success(), "failed to extract {}", archive.display()); + assert!(path.is_file(), "{} does not contain {name}", archive.display()); + } + path +} + /// A mock JSON-RPC server tuned for mega-evme integration tests. /// /// All mounted mocks match `POST` (the JSON-RPC verb) and use priorities so @@ -84,6 +142,130 @@ impl MockRpcServer { .await; } + /// Mount an unbounded mock that always returns a successful JSON-RPC body + /// with `"result": null` (HTTP 200). Models a transient not-found answer + /// (e.g. `eth_getTransactionByHash` for a briefly-invisible transaction) + /// that capture must not bake into the fixture. + pub(crate) async fn respond_jsonrpc_null_result(&self, priority: u8) { + let body = serde_json::json!({ + "jsonrpc": "2.0", + "id": 0, + "result": null, + }); + Mock::given(matchers::method("POST")) + .respond_with(ResponseTemplate::new(200).set_body_json(body)) + .with_priority(priority) + .mount(&self.server) + .await; + } + + /// Mount an unbounded mock that answers every JSON-RPC request for + /// `method` with the given hex `result`, regardless of params. + pub(crate) async fn respond_method_result(&self, method: &str, hex_result: &str, priority: u8) { + let body = serde_json::json!({ + "jsonrpc": "2.0", + "id": 0, + "result": hex_result, + }); + Mock::given(matchers::method("POST")) + .and(matchers::body_partial_json(serde_json::json!({ "method": method }))) + .respond_with(ResponseTemplate::new(200).set_body_json(body)) + .with_priority(priority) + .mount(&self.server) + .await; + } + + /// Mount an unbounded mock that answers every JSON-RPC request for `method` + /// with the given JSON `result`, regardless of params. + /// + /// The result is any JSON value, so this serves structured answers (blocks, + /// transactions) that [`Self::respond_method_result`]'s hex string cannot. + pub(crate) async fn respond_method_json( + &self, + method: &str, + result: serde_json::Value, + priority: u8, + ) { + let body = serde_json::json!({ "jsonrpc": "2.0", "id": 0, "result": result }); + Mock::given(matchers::method("POST")) + .and(matchers::body_partial_json(serde_json::json!({ "method": method }))) + .respond_with(ResponseTemplate::new(200).set_body_json(body)) + .with_priority(priority) + .mount(&self.server) + .await; + } + + /// Mount an unbounded mock that answers `method` calls whose params match + /// `params` with the given JSON `result`. + /// + /// Needed where one method is called with different arguments in the same + /// run and the answers must differ — `eth_getBlockByNumber` for a block and + /// its parent, for instance. + pub(crate) async fn respond_method_params_json( + &self, + method: &str, + params: serde_json::Value, + result: serde_json::Value, + priority: u8, + ) { + let body = serde_json::json!({ "jsonrpc": "2.0", "id": 0, "result": result }); + Mock::given(matchers::method("POST")) + .and(matchers::body_partial_json( + serde_json::json!({ "method": method, "params": params }), + )) + .respond_with(ResponseTemplate::new(200).set_body_json(body)) + .with_priority(priority) + .mount(&self.server) + .await; + } + + /// Mount a mock that answers the first `n` calls of `method` with matching + /// `params` and then stops matching, so a lower-priority mock for the same + /// request serves every later call. + /// + /// Models an endpoint whose answer to one repeated request changes + /// mid-run — a reorg landing between two calls, or a load balancer moving + /// the run to another backend. + pub(crate) async fn respond_method_params_json_n_times( + &self, + method: &str, + params: serde_json::Value, + result: serde_json::Value, + n: u64, + priority: u8, + ) { + let body = serde_json::json!({ "jsonrpc": "2.0", "id": 0, "result": result }); + Mock::given(matchers::method("POST")) + .and(matchers::body_partial_json( + serde_json::json!({ "method": method, "params": params }), + )) + .respond_with(ResponseTemplate::new(200).set_body_json(body)) + .up_to_n_times(n) + .with_priority(priority) + .mount(&self.server) + .await; + } + + /// How many single JSON-RPC requests for `method` the server has received. + /// + /// Batched requests (a JSON array body) are not counted: every call these + /// tests make is a single request. + pub(crate) async fn received_method_count(&self, method: &str) -> usize { + let requests = self.server.received_requests().await.expect("received_requests"); + requests + .iter() + .filter(|request| { + serde_json::from_slice::(&request.body) + .ok() + .and_then(|body| { + body.get("method").and_then(serde_json::Value::as_str).map(str::to_string) + }) + .as_deref() == + Some(method) + }) + .count() + } + /// Mount a mock that returns `eth_chainId` with the given chain id. pub(crate) async fn respond_eth_chain_id(&self, chain_id: u64, priority: u8) { let body = serde_json::json!({ @@ -105,11 +287,47 @@ impl MockRpcServer { pub(crate) async fn received_request_count(&self) -> usize { self.server.received_requests().await.expect("received_requests").len() } + + /// Accept every POST and delay the response far beyond any test timeout. + /// + /// Models a black-hole endpoint that accepts the TCP connection (and the + /// HTTP request) but never answers in time — the failure mode that + /// `--rpc.request-timeout` is meant to bound. The delay is 5 minutes so a + /// client with a 1s timeout fires first while the mock still records the hit. + pub(crate) async fn respond_black_hole(&self) { + use std::time::Duration; + + Mock::given(matchers::method("POST")) + .respond_with(ResponseTemplate::new(200).set_delay(Duration::from_secs(300))) + .mount(&self.server) + .await; + } +} + +/// Parse every top-level JSON value a run printed on stdout. +/// +/// Streaming parse, so it covers both the pretty-printed single-transaction +/// summary and the compact NDJSON of a batch run — in either case followed by +/// the structured error object a failing `--json` run ends with. +pub(crate) fn json_values(stdout: &str) -> Vec { + serde_json::Deserializer::from_str(stdout) + .into_iter::() + .collect::>() + .unwrap_or_else(|e| panic!("stdout is not a JSON stream ({e}):\n{stdout}")) +} + +/// Whether a printed value is the run-level error object of a failing `--json` +/// run (`{"error":{"code":…,"kind":…,"message":…}}`). +/// +/// A per-target NDJSON error line carries its transaction hash alongside the +/// `error` key, so the single-key shape identifies the run-level object. +pub(crate) fn is_run_error(value: &serde_json::Value) -> bool { + value.as_object().is_some_and(|obj| obj.len() == 1 && obj.contains_key("error")) } /// Build [`RpcArgs`] for a test pointed at `url` with the on-disk cache disabled. /// -/// Defaults: `--rpc.cache-size 0` (no cache layer, no disk persistence), +/// Defaults: `--rpc.no-cache-file` (in-memory LRU still applies; no disk persistence), /// 1ms backoff, production rate limit. `build_provider` still calls /// `eth_chainId`, so the caller must mount a mock for it. Pass `Some(n)` to /// override `--rpc.max-retries`; `None` keeps the production default. @@ -118,8 +336,7 @@ pub(crate) fn test_rpc_args(url: &str, max_retries: Option) -> RpcArgs { "mega-evme".into(), "--rpc".into(), url.into(), - "--rpc.cache-size".into(), - "0".into(), + "--rpc.no-cache-file".into(), "--rpc.backoff-ms".into(), "1".into(), "--rpc.rate-limit".into(), @@ -134,7 +351,7 @@ pub(crate) fn test_rpc_args(url: &str, max_retries: Option) -> RpcArgs { /// Build [`RpcArgs`] for a test that exercises the on-disk cache path. /// -/// Sets `--rpc.cache-size 256` and an explicit `--rpc.cache-dir`. The caller +/// Sets `--rpc.cache-max-entries 256` and an explicit `--rpc.cache-dir`. The caller /// must mount a mock `eth_chainId` response on the server so that /// `build_provider`'s `resolve_chain_id` call succeeds — use /// [`MockRpcServer::respond_eth_chain_id`] for this. @@ -147,7 +364,7 @@ pub(crate) fn test_rpc_args_cached( "mega-evme".into(), "--rpc".into(), url.into(), - "--rpc.cache-size".into(), + "--rpc.cache-max-entries".into(), "256".into(), "--rpc.cache-dir".into(), cache_dir.to_str().expect("cache_dir utf-8").to_string(), diff --git a/bin/mega-evme/tests/exit_codes.rs b/bin/mega-evme/tests/exit_codes.rs new file mode 100644 index 00000000..4dfecee9 --- /dev/null +++ b/bin/mega-evme/tests/exit_codes.rs @@ -0,0 +1,842 @@ +//! Integration tests for the CLI's exit-code taxonomy and its failure output. +//! +//! They run fully offline against the committed RPC capture +//! (`fixtures/replay_offline.cache.json`), so they are deterministic: a hash the +//! capture cannot answer models an endpoint that never answers, and the +//! validation paths need no provider at all. The mismatch class (exit 2) is +//! covered by `replay_verify.rs`, which doctors a copy of the same capture. + +use std::process::{Command, Output}; + +mod common; + +/// Offline RPC capture used as the replay file. +/// Name of the committed offline capture, resolved through the shared fixture +/// helper so its location lives in exactly one place. +const CACHE: &str = "replay_offline.cache.json"; + +/// Path of the committed offline capture. +fn cache() -> std::path::PathBuf { + common::fixture(CACHE) +} + +/// The transaction the committed capture can replay. +const TX_OK: &str = "0x41d34e7e13dfe0f85da9d407e2b2c381955d8c7eed428b17dc82327b2616b000"; + +/// Number of the block `TX_OK` was mined in, as the capture reports it. +const BLOCK_NUMBER: u64 = 18_172_461; + +/// A hash the capture holds no response for: the question goes unanswered. +const UNANSWERABLE_TX: &str = "0x0000000000000000000000000000000000000000000000000000000000000001"; + +/// Request fingerprint of a state read `TX_OK` performs while it executes. +/// +/// Entries are keyed by the request, so dropping this one from a copy of the +/// capture models an endpoint that stops answering mid-execution — the read +/// then fails inside the EVM and surfaces as a block execution error. +const IN_EXECUTION_STATE_READ: &str = + "0x0d9aee1b171e0c4a2be0107def891d838cc94d71e4046cb95b00a1c2a61cffed"; + +/// Request fingerprint of the EIP-2935 history-storage slot the pre-block +/// system call writes when replaying `TX_OK`'s block. +/// +/// Dropping this entry makes `apply_pre_execution_changes` fail inside the +/// blockhash contract call. mega-evm stringifies that database failure into +/// `BlockHashContractCall { message }`, so classification must recover the RPC +/// class from the stable `RPC error:` Display prefix rather than from a typed +/// cause chain. +const PRE_BLOCK_HISTORY_STORAGE_READ: &str = + "0x3abed4482ce079cf23c80a8e43bd75f8ac32b8b108e925b39f0a5c93de4aff48"; + +/// Request fingerprint of an EIP-4788 beacon-roots storage slot the pre-block +/// system call touches for the same block — a second stringified validation +/// path (`BeaconRootContractCall`) with the same recovery rule. +const PRE_BLOCK_BEACON_ROOT_STORAGE_READ: &str = + "0x49e3c5174c528b49897a0556c762d4fb88e1ad5e6aa8f8795ddbc37aa6c278f0"; + +/// Outcome of one `mega-evme` invocation. +struct Run { + code: Option, + stdout: String, + stderr: String, +} + +impl Run { + /// The process exit code the run ended with. + fn code(&self) -> i32 { + self.code.expect("mega-evme was killed by a signal") + } + + /// The structured error object a failing `--json` run ends with. + fn error_object(&self) -> serde_json::Value { + let values = common::json_values(&self.stdout); + let last = values + .last() + .unwrap_or_else(|| panic!("a failing --json run must not leave stdout empty")); + assert!( + common::is_run_error(last), + "the last stdout value must be the error object, got: {last}" + ); + last.clone() + } + + /// How many failure reports stderr carries. + /// + /// Counted by the report prefix: a message may itself span lines (an RPC + /// error appends a re-capture hint), and only the report opens one. + fn error_lines(&self) -> usize { + self.stderr.lines().filter(|line| line.starts_with("error: ")).count() + } +} + +fn run(args: &[&str]) -> Run { + let output: Output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args(args) + .output() + .expect("failed to run mega-evme"); + Run { + code: output.status.code(), + stdout: String::from_utf8(output.stdout).expect("stdout is utf-8"), + stderr: String::from_utf8(output.stderr).expect("stderr is utf-8"), + } +} + +/// Run `replay` against the committed offline capture. +fn replay(args: &[&str]) -> Run { + let cache = cache(); + let mut argv = + vec!["replay", "--rpc.replay-file", cache.to_str().expect("fixture path is utf-8")]; + argv.extend_from_slice(args); + run(&argv) +} + +/// Write a copy of the committed capture without the entry `key` answers, and +/// return its path. +fn cache_without_entry(name: &str, key: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + let before = entries.len(); + entries.retain(|entry| entry["key"].as_str() != Some(key)); + assert_eq!(entries.len() + 1, before, "the capture must hold exactly one entry for {key}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_exit_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write pruned cache"); + path +} + +/// Write a copy of the committed capture whose parent-block response reports a +/// hash that does not link to the replayed block, and return its path together +/// with the hash the untouched capture reported. +/// +/// The capture answers `eth_getBlockByNumber` for exactly two heights: the +/// replayed block and its parent. Rewriting the lower-numbered body's own `hash` +/// models an endpoint serving divergent views of the chain, since the two blocks +/// are fetched in separate calls. Only that field changes — state reads are +/// keyed by block number, so every other response still resolves. +fn cache_with_unlinked_parent(name: &str, wrong_hash: &str) -> (std::path::PathBuf, String) { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + + let mut blocks: Vec<(usize, u64)> = vec![]; + for (index, entry) in entries.iter().enumerate() { + let value = entry["value"].as_str().expect("entry value is a string"); + let Ok(response) = serde_json::from_str::(value) else { + continue; + }; + let Some(result) = response.get("result") else { + continue; + }; + // A block body is the only response carrying a parent hash. + if !result.is_object() || result.get("parentHash").is_none() { + continue; + } + let number = result["number"].as_str().expect("block number is a string"); + let number = + u64::from_str_radix(number.trim_start_matches("0x"), 16).expect("block number is hex"); + blocks.push((index, number)); + } + assert_eq!(blocks.len(), 2, "the capture must hold the replayed block and its parent"); + blocks.sort_unstable_by_key(|&(_, number)| number); + let (parent_index, _) = blocks[0]; + + let entry = &mut entries[parent_index]; + let mut response: serde_json::Value = + serde_json::from_str(entry["value"].as_str().expect("entry value is a string")) + .expect("parse parent block response"); + let result = &mut response["result"]; + let original = result["hash"].as_str().expect("parent block hash").to_string(); + result["hash"] = serde_json::Value::String(wrong_hash.to_string()); + entry["value"] = serde_json::Value::String(response.to_string()); + + let path = + std::env::temp_dir().join(format!("mega_evme_exit_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored cache"); + (path, original) +} + +/// Write a copy of the committed capture whose `eth_getTransactionByHash` +/// response for `TX_OK` carries `block_hash` as its inclusion hash, and return +/// its path together with the hash the untouched capture reported. +/// +/// Entries are keyed by the request, so the doctored answer still resolves. The +/// transaction lookup and the block fetch are separate calls, so rewriting only +/// the lookup models an endpoint that describes the target's inclusion +/// differently than the block it serves for that number. Passing +/// [`serde_json::Value::Null`] models a lookup that reports a mined transaction +/// without anchoring it to any block at all. +fn cache_with_inclusion_hash( + name: &str, + block_hash: serde_json::Value, +) -> (std::path::PathBuf, String) { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + // Only the transaction's own response carries it as the `hash` field; the + // block body lists bare hashes and a receipt names it `transactionHash`. + let marker = format!("\"hash\":\"{TX_OK}\""); + let mut original = None; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object for {TX_OK}"); + assert!( + result.get("blockNumber").is_some_and(|n| !n.is_null()), + "the captured transaction must report a block number" + ); + original = Some(result["blockHash"].as_str().expect("inclusion hash").to_string()); + result["blockHash"] = block_hash.clone(); + entry["value"] = serde_json::Value::String(response.to_string()); + } + let original = original.expect("the capture must hold exactly one response for the target"); + + let path = + std::env::temp_dir().join(format!("mega_evme_exit_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored cache"); + (path, original) +} + +/// Write a copy of the committed capture whose replayed block no longer lists +/// `TX_OK` in its body, and return its path together with that block's hash. +/// +/// The block keeps its own hash, so the lookup's inclusion hash still matches +/// what the endpoint serves for that number: only the membership the preceding +/// transaction set is derived from is gone. +fn cache_without_target_in_block_body(name: &str) -> (std::path::PathBuf, String) { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let mut block_hash = None; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + let Ok(mut response) = serde_json::from_str::(value) else { + continue; + }; + // A block body is the only response carrying a transaction list. + let Some(txs) = response + .get_mut("result") + .and_then(|result| result.get_mut("transactions")) + .and_then(|txs| txs.as_array_mut()) + else { + continue; + }; + if !txs.iter().any(|hash| hash.as_str() == Some(TX_OK)) { + continue; + } + txs.retain(|hash| hash.as_str() != Some(TX_OK)); + block_hash = Some(response["result"]["hash"].as_str().expect("block hash").to_string()); + entry["value"] = serde_json::Value::String(response.to_string()); + } + let block_hash = block_hash.expect("exactly one captured block body lists the target"); + + let path = + std::env::temp_dir().join(format!("mega_evme_exit_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored cache"); + (path, block_hash) +} + +/// Write a `--tx-file` holding `contents`, and return its path. +fn tx_file(name: &str, contents: &str) -> std::path::PathBuf { + let path = + std::env::temp_dir().join(format!("mega_evme_exit_{name}_{}.txt", std::process::id())); + std::fs::write(&path, contents).expect("write tx list"); + path +} + +/// Bad input is an execution-class failure: exit 1, with the structured object +/// as the last stdout line. +#[test] +fn test_invalid_input_exits_one_with_a_json_error_object() { + let list = tx_file("bad_hash", "not-a-hash\n"); + + let run = replay(&["--tx-file", list.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 1, "bad input exits 1.\nstderr: {}", run.stderr); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(1)); + assert_eq!(error["error"]["kind"].as_str(), Some("execution-error")); + assert!( + error["error"]["message"].as_str().is_some_and(|m| m.contains("not-a-hash")), + "the message must name the offending input: {error}" + ); +} + +/// A rejected flag combination is bad input too, and still ends `--json` stdout +/// with the error object rather than nothing at all. +#[test] +fn test_rejected_flag_combination_exits_one_with_a_json_error_object() { + let run = replay(&["--dump-fixture-dir", "/tmp/mega-evme-should-not-exist", "--json", TX_OK]); + + assert_eq!(run.code(), 1, "a rejected flag combination exits 1.\nstderr: {}", run.stderr); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("execution-error")); +} + +/// A transaction the offline capture cannot answer is an RPC failure: the +/// question went unanswered, which is distinct from a definitive "no". +#[test] +fn test_offline_cache_miss_exits_rpc_failure_with_a_json_error_object() { + let run = replay(&["--json", UNANSWERABLE_TX]); + + assert_eq!(run.code(), 3, "a cache miss exits 3.\nstderr: {}", run.stderr); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + error["error"]["message"].as_str().is_some_and(|m| m.contains("cache miss")), + "the message must explain the miss: {error}" + ); +} + +/// A state read that fails while the EVM is executing arrives as a block +/// execution error, but it is still an unanswered question: the run exits 3, and +/// a batch reports the target as an `rpc` failure rather than an execution one. +#[test] +fn test_state_read_failure_during_execution_is_an_rpc_failure() { + let path = cache_without_entry("state_read", IN_EXECUTION_STATE_READ); + let cache = path.to_str().unwrap(); + + let single = run(&["replay", "--rpc.replay-file", cache, "--json", TX_OK]); + assert_eq!(single.code(), 3, "an unanswered state read exits 3.\nstderr: {}", single.stderr); + let error = single.error_object(); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + error["error"]["message"] + .as_str() + .is_some_and(|m| m.contains("Block execution error") && m.contains("cache miss")), + "the failure must be the block error carrying the missed read: {error}" + ); + + let list = tx_file("state_read", &format!("{TX_OK}\n")); + let batch = run(&["replay", "--rpc.replay-file", cache, "--tx-file", list.to_str().unwrap()]); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_file(&path); + + assert_eq!(batch.code(), 3, "the batch run exits 3 too.\nstderr: {}", batch.stderr); + assert!( + batch.stdout.contains("Error (rpc):"), + "the target is reported as unanswered:\n{}", + batch.stdout + ); +} + +/// A cache miss during the pre-block EIP-2935 blockhash system call is an +/// unanswered RPC question even though mega-evm stringifies it into +/// `BlockHashContractCall { message }` before the exit classifier sees it. +/// +/// Without prefix recovery this lands as exit 1 / `execution-error`; with it, +/// single-run and batch both report the RPC class (exit 3). +#[test] +fn test_pre_block_blockhash_system_call_cache_miss_is_an_rpc_failure() { + let path = cache_without_entry("pre_block_2935", PRE_BLOCK_HISTORY_STORAGE_READ); + let cache = path.to_str().unwrap(); + + let single = run(&["replay", "--rpc.replay-file", cache, "--json", TX_OK]); + assert_eq!( + single.code(), + 3, + "a pre-block history-storage miss exits 3 (was exit 1 before stringified-RPC recovery).\n\ + stderr: {}", + single.stderr + ); + let error = single.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("blockhash contract call") && + message.contains("RPC error:") && + message.contains("cache miss"), + "the message must name the stringified pre-block path and the miss: {error}" + ); + + let list = tx_file("pre_block_2935", &format!("{TX_OK}\n")); + let batch = run(&["replay", "--rpc.replay-file", cache, "--tx-file", list.to_str().unwrap()]); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_file(&path); + + assert_eq!(batch.code(), 3, "the batch run exits 3 too.\nstderr: {}", batch.stderr); + assert!( + batch.stdout.contains("Error (rpc):"), + "the target is reported as unanswered:\n{}", + batch.stdout + ); +} + +/// Same recovery for the EIP-4788 beacon-root pre-block system call, which uses +/// `BeaconRootContractCall { message }` rather than the blockhash variant. +#[test] +fn test_pre_block_beacon_root_system_call_cache_miss_is_an_rpc_failure() { + let path = cache_without_entry("pre_block_4788", PRE_BLOCK_BEACON_ROOT_STORAGE_READ); + let cache = path.to_str().unwrap(); + + let single = run(&["replay", "--rpc.replay-file", cache, "--json", TX_OK]); + let _ = std::fs::remove_file(&path); + + assert_eq!( + single.code(), + 3, + "a pre-block beacon-root miss exits 3.\nstderr: {}", + single.stderr + ); + let error = single.error_object(); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("beacon root contract call") && + message.contains("RPC error:") && + message.contains("cache miss"), + "the message must name the stringified beacon-root path and the miss: {error}" + ); +} + +/// A parent block that does not link to the replayed block is an unanswered +/// question, not a wrong answer: the single-transaction run exits 3, with or +/// without `--verify-receipt`. +/// +/// The block and its parent are fetched by number in two separate calls, so a +/// reorg (or a load-balanced endpoint serving divergent views) can answer them +/// from different chains. Replaying anyway would fork from a pre-state that does +/// not precede the block, and the divergence would surface later as a receipt +/// mismatch (exit 2) or as a silently wrong replay. +#[test] +fn test_unlinked_parent_block_is_an_rpc_failure() { + const WRONG_PARENT: &str = "0x1111111111111111111111111111111111111111111111111111111111111111"; + + let (path, expected_parent) = cache_with_unlinked_parent("unlinked_parent", WRONG_PARENT); + let cache = path.to_str().unwrap(); + + for extra in [&[][..], &["--verify-receipt"][..]] { + let mut argv = vec!["replay", "--rpc.replay-file", cache, "--json"]; + argv.extend_from_slice(extra); + argv.push(TX_OK); + let outcome = run(&argv); + + assert_eq!( + outcome.code(), + 3, + "a broken parent linkage exits 3 for {extra:?}.\nstderr: {}", + outcome.stderr + ); + let error = outcome.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains(WRONG_PARENT) && message.contains(&expected_parent), + "the message must name both hashes (parent {expected_parent}, served \ + {WRONG_PARENT}): {error}" + ); + assert!(message.contains("divergent views"), "the message must name the cause: {error}"); + assert!( + !outcome.stdout.contains("MISMATCH") && + !outcome.stderr.contains("verification mismatch"), + "an unanswered question must not be reported as a mismatch:\n{}\n{}", + outcome.stdout, + outcome.stderr, + ); + } + + let _ = std::fs::remove_file(&path); +} + +/// A block whose hash is not the one the target was resolved as included in is +/// an unanswered question: the single-transaction run exits 3 rather than +/// replaying the target against a block it never ran in. +/// +/// The parent linkage can hold while both numbered fetches answer from a +/// replacement block, so the linkage guard alone does not anchor the target. +#[test] +fn test_block_that_does_not_match_the_reported_inclusion_is_an_rpc_failure() { + const WRONG_INCLUSION: &str = + "0x2222222222222222222222222222222222222222222222222222222222222222"; + + let (path, served) = + cache_with_inclusion_hash("wrong_inclusion", serde_json::json!(WRONG_INCLUSION)); + let run = run(&["replay", "--rpc.replay-file", path.to_str().unwrap(), "--json", TX_OK]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "a divergent inclusion exits 3.\nstderr: {}", run.stderr); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains(WRONG_INCLUSION) && message.contains(&served), + "the message must name both hashes (served {served}, reported {WRONG_INCLUSION}): {error}" + ); + assert!(message.contains("divergent views"), "the message must name the cause: {error}"); +} + +/// A block body that does not list the target is an unanswered question too: the +/// run exits 3 instead of treating every transaction of the block as preceding +/// and executing the target after the whole block. +#[test] +fn test_target_absent_from_the_block_body_is_an_rpc_failure() { + let (path, block_hash) = cache_without_target_in_block_body("absent_target"); + let run = run(&["replay", "--rpc.replay-file", path.to_str().unwrap(), "--json", TX_OK]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "a target absent from the body exits 3.\nstderr: {}", run.stderr); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains(TX_OK) && message.contains(&block_hash), + "the message must name the target and the block it is missing from: {error}" + ); + assert!( + !run.stdout.contains("\"success\""), + "the run must not produce an execution summary:\n{}", + run.stdout + ); +} + +/// A mined lookup carrying no inclusion hash is an unanchored view: the block +/// number alone cannot prove which body the target belongs to, so the run exits +/// 3 — the same class the batch driver rejects it with. +#[test] +fn test_mined_target_without_an_inclusion_hash_is_an_rpc_failure() { + let (path, _) = cache_with_inclusion_hash("unanchored", serde_json::Value::Null); + let run = run(&["replay", "--rpc.replay-file", path.to_str().unwrap(), "--json", TX_OK]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "an unanchored view exits 3.\nstderr: {}", run.stderr); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("inclusion hash") && message.contains("unanchored"), + "the message must name the unanchored view: {error}" + ); + assert!( + message.contains(&BLOCK_NUMBER.to_string()), + "the message must name the block number the lookup reported: {error}" + ); +} + +/// A batch run's error object follows the per-target lines, so a parser reading +/// the stream sees every target before the run-level verdict. +#[test] +fn test_batch_error_object_follows_the_per_target_lines() { + let list = tx_file("batch_miss", &format!("{UNANSWERABLE_TX}\n")); + + let run = replay(&["--tx-file", list.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 3, "an unanswered target exits 3.\nstderr: {}", run.stderr); + let values = common::json_values(&run.stdout); + assert_eq!(values.len(), 2, "one per-target line plus the error object:\n{}", run.stdout); + assert_eq!( + values[0]["tx_hash"].as_str(), + Some(UNANSWERABLE_TX), + "the per-target line comes first: {}", + values[0] + ); + assert!(common::is_run_error(&values[1]), "the error object comes last: {}", values[1]); +} + +/// Human mode reports the failure once, as `Display` text, and leaves stdout +/// untouched. +#[test] +fn test_human_failure_prints_exactly_one_error_line() { + let list = tx_file("human", "not-a-hash\n"); + + let run = replay(&["--tx-file", list.to_str().unwrap()]); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 1); + assert_eq!(run.stderr.lines().count(), 1, "exactly one line on stderr:\n{}", run.stderr); + assert!(run.stderr.starts_with("error: "), "the report is prefixed:\n{}", run.stderr); + assert!( + !run.stderr.contains("Evme(") && !run.stderr.contains("InvalidInput("), + "the report must be Display-formatted, not Debug:\n{}", + run.stderr + ); + assert!(run.stdout.is_empty(), "human mode prints no failure on stdout:\n{}", run.stdout); +} + +/// A message that carries its own extra lines (the RPC hint) is still reported +/// exactly once, and never in `Debug` form. +#[test] +fn test_human_failure_reports_a_multi_line_message_once() { + let run = replay(&[UNANSWERABLE_TX]); + + assert_eq!(run.code(), 3); + assert_eq!(run.error_lines(), 1, "exactly one report on stderr:\n{}", run.stderr); + assert!( + !run.stderr.contains("Evme(") && !run.stderr.contains("RpcError("), + "the report must be Display-formatted, not Debug:\n{}", + run.stderr + ); + assert!(run.stdout.is_empty(), "human mode prints no failure on stdout:\n{}", run.stdout); +} + +/// A successful run exits 0 and prints no error object: the failure surface +/// leaves the success output untouched. +#[test] +fn test_successful_run_exits_zero_without_an_error_object() { + let run = replay(&["--json", TX_OK]); + + assert_eq!(run.code(), 0, "a faithful replay exits 0.\nstderr: {}", run.stderr); + let values = common::json_values(&run.stdout); + assert_eq!(values.len(), 1, "only the summary is printed:\n{}", run.stdout); + assert!(!common::is_run_error(&values[0]), "a successful run prints no error object"); + assert_eq!(run.error_lines(), 0, "a successful run reports nothing on stderr"); +} + +/// A capture that could not be persisted is reported even when the run it was +/// capturing also failed: the run error keeps the exit code (it is the root +/// cause), and both failures are named on stderr without `-v`, so a stale or +/// missing capture file cannot go unnoticed. +#[tokio::test(flavor = "multi_thread")] +async fn test_capture_persist_failure_is_reported_next_to_the_run_error() { + let server = common::MockRpcServer::start().await; + // Chain id resolves, every other call fails: the replay itself goes + // unanswered while the capture store still has entries to write. + server.respond_eth_chain_id(6342, 1).await; + server.respond_status_always(500).await; + let url = server.uri(); + + // A capture path whose parent is a regular file: persisting cannot succeed. + let blocker = + std::env::temp_dir().join(format!("mega_evme_capture_blocker_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&blocker); + std::fs::write(&blocker, b"not a directory").expect("write blocker file"); + let capture = blocker.join("capture.json"); + + let mut argv = vec![ + "replay", + "--rpc", + &url, + "--rpc.capture-file", + capture.to_str().unwrap(), + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + TX_OK, + ]; + let human = run(&argv); + argv.push("--json"); + let json = run(&argv); + let _ = std::fs::remove_file(&blocker); + + for run in [&human, &json] { + assert_eq!(run.code(), 3, "the run error keeps the exit code.\nstderr: {}", run.stderr); + assert_eq!(run.error_lines(), 2, "both failures are reported:\n{}", run.stderr); + assert!( + run.stderr.contains("Failed to fetch transaction"), + "the run error must be reported:\n{}", + run.stderr + ); + assert!( + run.stderr.contains(blocker.to_str().expect("blocker path is utf-8")), + "the persist failure must name where the capture could not be written:\n{}", + run.stderr + ); + } + + // The structured object still reports the run error, which owns the code. + let error = json.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert!( + error["error"]["message"] + .as_str() + .is_some_and(|m| m.contains("Failed to fetch transaction")), + "the object carries the run error, not the persist failure: {error}" + ); +} + +/// A usage error is bad input, so it joins the execution class instead of +/// colliding with the mismatch code; `--help` stays a successful run. +#[test] +fn test_usage_errors_exit_one_and_help_exits_zero() { + // No replay target: rejected by argument parsing. + let usage = run(&["replay"]); + assert_eq!(usage.code(), 1, "a usage error exits 1.\nstderr: {}", usage.stderr); + assert!(usage.stdout.is_empty(), "a usage error prints nothing on stdout"); + + let help = run(&["--help"]); + assert_eq!(help.code(), 0, "--help exits 0"); + assert!(help.stdout.contains("mega-evme"), "--help prints usage on stdout"); +} + +/// A usage error of a `--json` run still ends stdout with the structured error +/// object: argument parsing fails before the command exists, but a +/// machine-readable run must never end with empty stdout. +#[test] +fn test_usage_error_in_json_mode_ends_stdout_with_the_error_object() { + // No replay target: rejected by argument parsing. + let usage = run(&["replay", "--json"]); + + assert_eq!(usage.code(), 1, "a usage error exits 1.\nstderr: {}", usage.stderr); + let error = usage.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(1)); + assert_eq!(error["error"]["kind"].as_str(), Some("execution-error")); + let message = error["error"]["message"].as_str().expect("the object carries a message"); + assert!(!message.contains('\n'), "the message is a single line: {message}"); + assert!( + message.contains("required arguments"), + "the message must summarize the usage error: {message}" + ); + // clap keeps rendering its own report, including the usage block. + assert!(usage.stderr.contains("Usage:"), "clap still reports on stderr:\n{}", usage.stderr); +} + +/// `--help` in a `--json` run is still not a failure: no error object, exit 0. +#[test] +fn test_help_in_json_mode_prints_no_error_object() { + let help = run(&["--help", "--json"]); + + assert_eq!(help.code(), 0, "--help exits 0"); + assert!( + !help.stdout.lines().any(|line| line.trim_start().starts_with(r#"{"error""#)), + "--help prints no error object:\n{}", + help.stdout + ); +} + +/// Closing stdout mid-batch must not abort the process. +/// +/// Rust ignores SIGPIPE, so the next NDJSON `println!` panics with a broken +/// pipe. The panic hook still has to reach `exit(1)` even when it cannot write +/// the structured error object to the same closed stdout — otherwise the +/// runtime aborts (SIGABRT, shell status 134) and scripts that branch on the +/// documented 0/1/2/3 exit classes see an undefined status. +#[test] +fn test_closed_stdout_during_json_batch_exits_one() { + use std::{ + io::{BufRead, BufReader, Read}, + process::{Command, Stdio}, + thread, + }; + + // Multi-target offline batch: many NDJSON lines, so dropping the pipe after + // the first line still leaves further writes that hit the broken pipe. + let envelope = common::fixture("replay_batch_blocks.cache.json"); + let mut child = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args([ + "replay", + "--rpc.replay-file", + envelope.to_str().expect("fixture path is utf-8"), + "--block", + "22945844", + "--json", + ]) + // Bound panic-hook stderr volume regardless of the ambient env: a full + // backtrace can fill the pipe buffer and deadlock child-vs-`wait()` if + // stderr is never drained. We still drain (below) so a caller that + // exports `RUST_BACKTRACE=full` cannot hang this test either. + .env("RUST_BACKTRACE", "0") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("failed to spawn mega-evme"); + + // Drain stderr on a reader thread before any wait, so a noisy panic hook + // cannot fill the pipe and stall the child forever. + let stderr = child.stderr.take().expect("child stderr was piped"); + let stderr_drain = thread::spawn(move || { + let mut sink = Vec::new(); + let _ = BufReader::new(stderr).read_to_end(&mut sink); + sink + }); + + let stdout = child.stdout.take().expect("child stdout was piped"); + let mut first_line = String::new(); + BufReader::new(stdout) + .read_line(&mut first_line) + .expect("failed to read the first NDJSON line"); + assert!( + !first_line.trim().is_empty(), + "batch --json must print at least one NDJSON line before further writes" + ); + // Dropping the BufReader closes the read end. The child's next stdout write + // then fails with EPIPE and panics into the process-wide hook. + // (Binding ends here; no further use of the pipe.) + + let status = child.wait().expect("failed to wait for mega-evme"); + let _stderr_bytes = stderr_drain.join().expect("stderr drain thread panicked"); + assert_eq!( + status.code(), + Some(1), + "closed stdout must exit 1 (execution-error), not signal death.\nstatus: {status:?}" + ); + assert!( + status.code().is_some(), + "process must not be signal-killed (e.g. SIGABRT from a double panic in the hook)" + ); +} + +/// A panic under `--json` with an open stdout ends the stream with the standard +/// error envelope (`code: 1`, `kind: "execution-error"`). +/// +/// The closed-stdout case only proves `exit(1)` when the hook cannot write. +/// This test pins the machine-readable object the hook prints when stdout is +/// still open. Triggered via the test-only `MEGA_EVME_INJECT_PANIC` hook (same +/// `test-utils` gate as the fixture pre-state inject), not via invalid input. +#[test] +fn test_panic_under_json_prints_execution_error_envelope() { + let output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args(["--json"]) + .env("MEGA_EVME_INJECT_PANIC", "1") + .env("RUST_BACKTRACE", "0") + .output() + .expect("failed to run mega-evme"); + + assert_eq!( + output.status.code(), + Some(1), + "injected panic must exit 1.\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + + let stdout = String::from_utf8(output.stdout).expect("stdout is utf-8"); + let values = common::json_values(&stdout); + let last = values + .last() + .unwrap_or_else(|| panic!("panic under --json must not leave stdout empty:\n{stdout}")); + assert!( + common::is_run_error(last), + "the final stdout line must be the run-level error object, got: {last}" + ); + assert_eq!(last["error"]["code"].as_u64(), Some(1)); + assert_eq!(last["error"]["kind"].as_str(), Some("execution-error")); + let message = last["error"]["message"].as_str().expect("error.message must be a string"); + assert!( + message.starts_with("panic: "), + "panic-hook message must keep the `panic: …` prefix: {message}" + ); +} diff --git a/bin/mega-evme/tests/fixtures/halt_logs_repro.cache.json.tar.gz b/bin/mega-evme/tests/fixtures/halt_logs_repro.cache.json.tar.gz new file mode 100644 index 00000000..6edb488d Binary files /dev/null and b/bin/mega-evme/tests/fixtures/halt_logs_repro.cache.json.tar.gz differ diff --git a/bin/mega-evme/tests/fixtures/replay_batch_blocks.cache.json.tar.gz b/bin/mega-evme/tests/fixtures/replay_batch_blocks.cache.json.tar.gz new file mode 100644 index 00000000..cf87d534 Binary files /dev/null and b/bin/mega-evme/tests/fixtures/replay_batch_blocks.cache.json.tar.gz differ diff --git a/bin/mega-evme/tests/fixtures/test_replay_offline_bucket_cap.json b/bin/mega-evme/tests/fixtures/test_replay_offline_bucket_cap.json index ea805833..1ae2a337 100644 --- a/bin/mega-evme/tests/fixtures/test_replay_offline_bucket_cap.json +++ b/bin/mega-evme/tests/fixtures/test_replay_offline_bucket_cap.json @@ -21,11 +21,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x0000000000000000000000000000000000000000000000000000000005f6080c555344540000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x0", "removed": false }, @@ -35,11 +35,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x0000000000000000000000000000000000000000000000000000000007e7c208585250000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x1", "removed": false }, @@ -49,11 +49,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x00000000000000000000000000000000000000000000000000000000008acf28444f47450000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x2", "removed": false }, @@ -63,11 +63,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x000000000000000000000000000000000000000000000000000006716f482ed2425443000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x3", "removed": false }, @@ -77,11 +77,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x00000000000000000000000000000000000000000000000000000000016c0b50414441000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x4", "removed": false }, @@ -91,11 +91,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x0000000000000000000000000000000000000000000000000000000005f592dd555344430000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x5", "removed": false }, @@ -105,11 +105,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x0000000000000000000000000000000000000000000000000000000deaecfcd1424e42000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x6", "removed": false }, @@ -119,11 +119,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x00000000000000000000000000000000000000000000000000000032f3a28db8455448000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x7", "removed": false }, @@ -133,11 +133,11 @@ "0xffcbf7f11b241556bfaa9228d6d51124009fd1f67009431663b49b6c87da0dd3" ], "data": "0x00000000000000000000000000000000000000000000000000000001e8730400534f4c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064f54ad9fbea900000000000000000000000000000000000000000000000000064f54ad9f28400000000000000000000000000000000000000000000000000000000000000000", - "blockHash": null, + "blockHash": "0x3f77d9845128bfbaa01176be2caf5e2db891e27e44c4503ba0fbd52e0ffb3ac6", "blockNumber": "0xf9b3e2", "blockTimestamp": "0x69dcc0da", - "transactionHash": "0x0000000000000000000000000000000000000000000000000000000000000000", - "transactionIndex": "0x0", + "transactionHash": "0x346c9ecd95ea9502e62534b68d8797f592d5f20bf7ed2d7b88f6ca7970e46919", + "transactionIndex": "0x4", "logIndex": "0x8", "removed": false } diff --git a/bin/mega-evme/tests/fixtures/test_run_rpc_args_parse.json b/bin/mega-evme/tests/fixtures/test_run_rpc_args_parse.json index 0e5fb157..e8d43539 100644 --- a/bin/mega-evme/tests/fixtures/test_run_rpc_args_parse.json +++ b/bin/mega-evme/tests/fixtures/test_run_rpc_args_parse.json @@ -3,7 +3,7 @@ "args": [ "run", "0x604260005260206000f3", - "--rpc.cache-size", "100", + "--rpc.cache-max-entries", "100", "--rpc.max-retries", "3", "--rpc.backoff-ms", "500", "--rpc.rate-limit", "660" diff --git a/bin/mega-evme/tests/provider.rs b/bin/mega-evme/tests/provider.rs index f575964f..80e01bb0 100644 --- a/bin/mega-evme/tests/provider.rs +++ b/bin/mega-evme/tests/provider.rs @@ -14,7 +14,7 @@ use std::path::PathBuf; use alloy_primitives::B256; use alloy_provider::Provider; use clap::Parser; -use mega_evme::common::{BuildProviderOutput, EvmeError, RpcArgs}; +use mega_evme::common::{BuildProviderOutput, EvmeError, ExitCode, RpcArgs}; use tempfile::tempdir; mod common; @@ -24,11 +24,12 @@ use common::{test_rpc_args, test_rpc_args_cached, MockRpcServer}; #[test] fn test_rpc_args_parses_all_new_flags() { + // Keep `--rpc.rate-limit` here so the visible alias stays pin-tested. let args = RpcArgs::parse_from([ "mega-evme", "--rpc", "https://example.test/rpc", - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", "/tmp/example-cache", @@ -40,15 +41,64 @@ fn test_rpc_args_parses_all_new_flags() { "250", "--rpc.rate-limit", "1234", + "--rpc.request-timeout", + "45", ]); assert_eq!(args.rpc_url, Some("https://example.test/rpc".to_string())); - assert_eq!(args.cache_size, 256); + assert_eq!(args.cache_max_entries, 256); assert_eq!(args.cache_dir, Some(PathBuf::from("/tmp/example-cache"))); assert!(args.no_cache_file); assert!(args.clear_cache); assert_eq!(args.max_retries, 7); assert_eq!(args.backoff_ms, 250); assert_eq!(args.compute_units_per_sec, 1234); + assert_eq!(args.request_timeout, 45); +} + +/// Explicit `--rpc.request-timeout` values parse, including the disable sentinel `0`. +#[test] +fn test_rpc_args_parses_request_timeout() { + let defaulted = RpcArgs::parse_from(["mega-evme"]); + assert_eq!(defaulted.request_timeout, 30, "default request timeout is 30s"); + + let explicit = RpcArgs::parse_from(["mega-evme", "--rpc.request-timeout", "12"]); + assert_eq!(explicit.request_timeout, 12); + + let disabled = RpcArgs::parse_from(["mega-evme", "--rpc.request-timeout", "0"]); + assert_eq!(disabled.request_timeout, 0, "0 disables the per-request timeout"); +} + +/// The removed `--rpc.cache-size` flag must fail to parse (pin the deletion). +#[test] +fn test_rpc_args_rejects_removed_cache_size_flag() { + let err = RpcArgs::try_parse_from([ + "mega-evme", + "--rpc", + "https://example.test/rpc", + "--rpc.cache-size", + "100", + ]) + .expect_err("removed --rpc.cache-size must not parse"); + let msg = err.to_string(); + assert!( + msg.contains("unexpected argument") || + msg.contains("unknown") || + msg.contains("cache-size"), + "error must reject the removed flag, got: {msg}", + ); +} + +/// Canonical flag name `--rpc.cu-per-sec` parses into the same field. +#[test] +fn test_rpc_args_parses_cu_per_sec_flag() { + let args = RpcArgs::parse_from([ + "mega-evme", + "--rpc", + "https://example.test/rpc", + "--rpc.cu-per-sec", + "1234", + ]); + assert_eq!(args.compute_units_per_sec, 1234); } /// `--rpc.cache-dir ""` (and whitespace-only) must be rejected at parse @@ -80,27 +130,30 @@ fn test_rpc_args_rejects_empty_cache_dir() { fn test_rpc_args_default_values() { let args = RpcArgs::parse_from(["mega-evme"]); assert_eq!(args.rpc_url, None); - assert_eq!(args.cache_size, 10_000); + assert_eq!(args.cache_max_entries, 0, "default is unlimited (never evict)"); assert_eq!(args.cache_dir, None); assert!(!args.no_cache_file); assert!(!args.clear_cache); assert_eq!(args.max_retries, 5); assert_eq!(args.backoff_ms, 1_000); assert_eq!(args.compute_units_per_sec, 660); + assert_eq!(args.request_timeout, 30); } // ─── build_provider shape variants ─────────────────────────────────────────── -/// `--rpc.cache-size 0`: noop store, but `chain_id` is still resolved. +/// Default `--rpc.cache-max-entries 0` (unlimited) still resolves `chain_id` +/// and installs the in-memory cache layer; with `--rpc.no-cache-file` the +/// disk store is a no-op. #[tokio::test(flavor = "multi_thread")] -async fn test_build_provider_without_cache() { +async fn test_build_provider_default_unlimited_with_no_cache_file() { let server = MockRpcServer::start().await; server.respond_eth_chain_id(4326, 1).await; - let args = RpcArgs::parse_from(["mega-evme", "--rpc", &server.uri(), "--rpc.cache-size", "0"]); + let args = RpcArgs::parse_from(["mega-evme", "--rpc", &server.uri(), "--rpc.no-cache-file"]); let BuildProviderOutput { cache_store, chain_id, .. } = args.build_provider().await.expect("build_provider"); - assert!(cache_store.is_noop(), "cache_size == 0 must produce a no-op store"); - assert_eq!(chain_id, 4326, "chain_id must be resolved even when cache is disabled"); + assert!(cache_store.is_noop(), "--rpc.no-cache-file must produce a no-op store"); + assert_eq!(chain_id, 4326, "chain_id must be resolved with unlimited cache default"); cache_store.persist().expect("persist"); } @@ -114,7 +167,7 @@ async fn test_build_provider_no_cache_file_skips_persistence() { "mega-evme", "--rpc", &server.uri(), - "--rpc.cache-size", + "--rpc.cache-max-entries", "100", "--rpc.no-cache-file", ]); @@ -136,20 +189,31 @@ async fn test_build_provider_with_cache_names_file_from_fetched_chain_id() { let BuildProviderOutput { cache_store, .. } = args.build_provider().await.expect("build_provider"); - assert!(!cache_store.is_noop(), "cache_size > 0 + cache_dir must produce a real store"); + assert!( + !cache_store.is_noop(), + "cache_dir without --rpc.no-cache-file must produce a real store" + ); assert_eq!(cache_store.cache_path(), Some(dir.path().join("rpc-cache-4326.json").as_path())); } +/// Malformed `--rpc` is bad input (exit 1), not an RPC transport failure (exit 3). #[tokio::test(flavor = "multi_thread")] async fn test_build_provider_invalid_url() { - let args = RpcArgs::parse_from(["mega-evme", "--rpc", "not a url", "--rpc.cache-size", "0"]); + let args = RpcArgs::parse_from(["mega-evme", "--rpc", "not a url", "--rpc.no-cache-file"]); let err = args.build_provider().await.expect_err("build_provider should fail"); - match err { - EvmeError::RpcError(msg) => { + match &err { + EvmeError::InvalidInput(msg) => { assert!(msg.contains("not a url"), "error must echo the original input, got: {msg}"); + assert!(msg.contains("Invalid RPC URL"), "msg={msg}"); } - other => panic!("expected EvmeError::RpcError, got {other:?}"), + other => panic!("expected EvmeError::InvalidInput, got {other:?}"), } + // Exit-code taxonomy: InvalidInput → execution-error (code 1), never rpc-failure (3). + assert_eq!( + ExitCode::from_evme_error(&err).code(), + 1, + "malformed --rpc must exit 1 (bad input), not 3 (rpc-failure)", + ); } // ─── Chain-id resolution ───────────────────────────────────────────────────── @@ -169,7 +233,7 @@ async fn test_build_provider_fetches_chain_id_from_rpc() { "mega-evme", "--rpc", &server.uri(), - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -201,7 +265,7 @@ async fn test_build_provider_chain_id_rpc_failure_is_hard_error() { "mega-evme", "--rpc", &server.uri(), - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -306,7 +370,7 @@ async fn test_build_provider_clear_cache_deletes_file_before_load() { "mega-evme", "--rpc", &server.uri(), - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -324,6 +388,60 @@ async fn test_build_provider_clear_cache_deletes_file_before_load() { ); } +/// `--rpc.clear-cache` fails closed when the sidecar lock cannot be acquired: +/// the user asked for a deletion that is not safe to do unlocked, so the file +/// must remain and the build must hard-error rather than unlink without the lock. +#[tokio::test(flavor = "multi_thread")] +async fn test_build_provider_clear_cache_fails_closed_when_lock_unacquirable() { + let server = MockRpcServer::start().await; + server.respond_eth_chain_id(77, 1).await; + + let dir = tempdir().expect("tempdir"); + let cache_file = dir.path().join("rpc-cache-77.json"); + let seed = r#"[{"key":"0x0000000000000000000000000000000000000000000000000000000000000001","value":"seed"}]"#; + std::fs::write(&cache_file, seed).expect("seed cache"); + // A directory in the sidecar's place makes the lock un-acquirable. + std::fs::create_dir(format!("{}.lock", cache_file.display())).expect("occupy sidecar"); + + let args = RpcArgs::parse_from([ + "mega-evme", + "--rpc", + &server.uri(), + "--rpc.cache-max-entries", + "256", + "--rpc.cache-dir", + dir.path().to_str().unwrap(), + "--rpc.clear-cache", + ]); + + let err = args.build_provider().await.expect_err("clear-cache must fail closed on lock"); + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "a local lock failure is not the endpoint's fault: retrying or switching \ + the RPC cannot fix it, so it must not classify as an rpc failure", + ); + match err { + EvmeError::InvalidInput(msg) => { + assert!(msg.contains("lock"), "error must name the lock failure, got: {msg}"); + assert!( + msg.contains("rpc-cache-77.json.lock") || msg.contains(".lock"), + "error must name the sidecar, got: {msg}", + ); + assert!( + msg.contains("Refusing to clear") || msg.contains("clear"), + "error must state the clear was refused, got: {msg}", + ); + } + other => panic!("expected EvmeError::InvalidInput, got {other:?}"), + } + assert_eq!( + std::fs::read_to_string(&cache_file).expect("file still readable"), + seed, + "no unlocked unlink happened", + ); +} + /// `--rpc.clear-cache` must hard-error when the file exists but cannot be /// unlinked, rather than warn-and-continue. Silent fallback would reload /// exactly the content the user asked to wipe, defeating the recovery path. @@ -345,6 +463,10 @@ async fn test_build_provider_clear_cache_hard_errors_on_unlink_failure() { // Seed a "polluted" cache file the user would want to wipe. let cache_file = dir.path().join(format!("rpc-cache-{chain_id}.json")); std::fs::write(&cache_file, r#"{"polluted":"content"}"#).expect("write seed"); + // Pre-create the lock sidecar while the directory is still writable. + // Clear acquires that sidecar before unlinking; without it, a read-only + // parent would fail lock creation first and never exercise the unlink path. + std::fs::write(format!("{}.lock", cache_file.display()), b"").expect("seed sidecar"); // Revoke write permission on the parent dir so `remove_file` fails. // Read/execute stays on so the file is still visible to `path.exists()` @@ -358,7 +480,7 @@ async fn test_build_provider_clear_cache_hard_errors_on_unlink_failure() { "mega-evme", "--rpc", &server.uri(), - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -372,14 +494,20 @@ async fn test_build_provider_clear_cache_hard_errors_on_unlink_failure() { std::fs::set_permissions(dir.path(), orig_perms).expect("chmod restore"); let err = result.expect_err("clear-cache must hard-error on unlink failure"); + assert_eq!( + ExitCode::from_evme_error(&err), + ExitCode::ExecutionError, + "a local unlink failure is not the endpoint's fault: retrying or switching \ + the RPC cannot fix it, so it must not classify as an rpc failure", + ); match err { - EvmeError::RpcError(msg) => { + EvmeError::InvalidInput(msg) => { assert!( msg.contains("Failed to clear RPC cache"), "error must name the failed operation, got: {msg}", ); } - other => panic!("expected EvmeError::RpcError, got {other:?}"), + other => panic!("expected EvmeError::InvalidInput, got {other:?}"), } assert!(cache_file.exists(), "the cache file should still be on disk — unlink failed"); } @@ -528,6 +656,132 @@ async fn test_retry_layer_retries_on_unreachable_endpoint() { ); } +// ─── Request-timeout behavior ──────────────────────────────────────────────── + +/// A hung endpoint (accepts TCP, never responds) with +/// `--rpc.request-timeout 1` and `--rpc.max-retries 0` fails quickly at +/// chain-id resolution as `EvmeError::RpcError` (exit 3), instead of hanging. +#[tokio::test(flavor = "multi_thread")] +async fn test_request_timeout_fails_black_hole_within_bound() { + let server = MockRpcServer::start().await; + server.respond_black_hole().await; + + let args = RpcArgs::parse_from([ + "mega-evme", + "--rpc", + &server.uri(), + "--rpc.no-cache-file", + "--rpc.request-timeout", + "1", + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + ]); + + let started = std::time::Instant::now(); + let err = args.build_provider().await.expect_err("black-hole must time out"); + let elapsed = started.elapsed(); + + assert!( + elapsed < std::time::Duration::from_secs(15), + "must fail within a few seconds (timeout=1s, retries=0), took {elapsed:?}", + ); + // Floor: a real 1s timeout should not return in sub-millisecond time. + assert!( + elapsed >= std::time::Duration::from_millis(500), + "must wait for the request timeout, took {elapsed:?}", + ); + + match &err { + EvmeError::RpcError(msg) => { + assert!( + msg.contains("Failed to fetch chain ID"), + "timeout must surface via chain-id resolution, got: {msg}", + ); + } + other => panic!("expected EvmeError::RpcError, got {other:?}"), + } + assert_eq!( + ExitCode::from_evme_error(&err).code(), + 3, + "exhausted timeout must exit 3 (rpc-failure)", + ); + + // Spawned-binary assertion: the same flags on `replay` must exit 3 within + // the wall-time bound (chain-id fetch is the first networked call). + let started = std::time::Instant::now(); + let output = std::process::Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args([ + "replay", + "--rpc", + &server.uri(), + "--rpc.no-cache-file", + "--rpc.request-timeout", + "1", + "--rpc.max-retries", + "0", + "--rpc.backoff-ms", + "1", + "0x0000000000000000000000000000000000000000000000000000000000000001", + ]) + .output() + .expect("spawn mega-evme"); + let elapsed = started.elapsed(); + assert!( + elapsed < std::time::Duration::from_secs(15), + "spawned binary must fail within a few seconds, took {elapsed:?}", + ); + assert_eq!( + output.status.code(), + Some(3), + "spawned binary must exit 3 on timeout.\nstderr: {}", + String::from_utf8_lossy(&output.stderr), + ); +} + +/// A black-hole endpoint with `--rpc.max-retries 2` makes three attempts +/// (1 initial + 2 retries) before giving up — proving reqwest timeouts are +/// classified as retryable `TransportErrorKind::Custom` errors. +#[tokio::test(flavor = "multi_thread")] +async fn test_request_timeout_is_retried_up_to_max_retries() { + let server = MockRpcServer::start().await; + server.respond_black_hole().await; + + let args = RpcArgs::parse_from([ + "mega-evme", + "--rpc", + &server.uri(), + "--rpc.no-cache-file", + "--rpc.request-timeout", + "1", + "--rpc.max-retries", + "2", + "--rpc.backoff-ms", + "1", + ]); + + let started = std::time::Instant::now(); + let err = args.build_provider().await.expect_err("black-hole must exhaust retries"); + let elapsed = started.elapsed(); + + // 3 attempts × ~1s timeout + small backoffs; keep a generous upper bound. + assert!( + elapsed < std::time::Duration::from_secs(30), + "3×1s timeouts must finish well under 30s, took {elapsed:?}", + ); + assert_eq!( + server.received_request_count().await, + 3, + "max-retries=2 → 1 initial + 2 retries against the black-hole", + ); + assert_eq!( + ExitCode::from_evme_error(&err).code(), + 3, + "exhausted timeout retries must exit 3 (rpc-failure)", + ); +} + // ─── Contract regression guards (fixture-file modes) ─────────────────────── /// The `env = "RPC_URL"` attribute was removed from `--rpc`, so parsing @@ -739,6 +993,70 @@ async fn test_capture_does_not_cache_jsonrpc_error_response() { assert!(cached.get("error").is_none(), "cached entry must not be an error response"); } +/// A success with `"result": null` must be served to the caller but must not +/// be baked into the capture fixture. Offline replay of that fixture then +/// fails with a cache-miss error naming the request, not a silent not-found +/// from a frozen null. +#[tokio::test(flavor = "multi_thread")] +async fn test_capture_does_not_cache_null_result_and_offline_misses() { + let server = MockRpcServer::start().await; + // eth_chainId must succeed so build_capture_provider can complete. + server.respond_eth_chain_id(4326, 1).await; + // Any other call resolves to a null result at HTTP 200. + server.respond_jsonrpc_null_result(2).await; + + let dir = tempdir().expect("tempdir"); + let cache_file = dir.path().join("null.cache.json"); + + let capture_args = RpcArgs::parse_from([ + "mega-evme", + "--rpc", + &server.uri(), + "--rpc.capture-file", + cache_file.to_str().unwrap(), + ]); + let output = capture_args.build_capture_provider().await.expect("capture build"); + + // Null is a transport-level success; the provider fails deserializing it + // into a block number. Capture must still observe the response and skip it. + let _ = output.provider.get_block_number().await; + output.cache_store.persist().expect("persist should still succeed"); + + // The envelope must hold only eth_chainId — no entry for the null call. + let raw = std::fs::read_to_string(&cache_file).expect("read envelope"); + let envelope: serde_json::Value = serde_json::from_str(&raw).expect("parse envelope"); + let entries = envelope["cache"].as_array().expect("cache is a JSON array"); + assert_eq!( + entries.len(), + 1, + "only eth_chainId should be cached; null results must be skipped. entries = {entries:#?}", + ); + let cached: serde_json::Value = + serde_json::from_str(entries[0]["value"].as_str().expect("value is a JSON string")) + .expect("cached response is valid JSON"); + assert!( + cached.get("result").is_some_and(|r| !r.is_null()), + "sole cached entry must be a non-null success, got {cached}", + ); + + // Offline replay: the null was never captured, so the same request is a + // cache miss that names the method — not a silent null / not_found. + let replay_args = + RpcArgs::parse_from(["mega-evme", "--rpc.replay-file", cache_file.to_str().unwrap()]); + let replay = replay_args.build_replay_provider().await.expect("replay build"); + let err = replay + .provider + .get_block_number() + .await + .expect_err("missing null entry must surface as cache miss"); + let msg = format!("{err}"); + assert!(msg.contains("cache miss"), "offline error must say 'cache miss', got: {msg}",); + assert!( + msg.contains("eth_blockNumber"), + "offline error must name the missing method, got: {msg}", + ); +} + /// Cross-chain contamination guard: an existing envelope claiming chain X /// combined with an endpoint returning chain Y must hard-error, not silently /// mix responses from two chains. @@ -831,7 +1149,7 @@ fn test_capture_file_mutex_with_other_cache_flags() { (&["--rpc.cache-dir", "/tmp/cache"], "--rpc.cache-dir"), (&["--rpc.clear-cache"], "--rpc.clear-cache"), (&["--rpc.no-cache-file"], "--rpc.no-cache-file"), - (&["--rpc.cache-size", "256"], "--rpc.cache-size"), + (&["--rpc.cache-max-entries", "256"], "--rpc.cache-max-entries"), ]; for (extra_flags, label) in cases { let mut argv = @@ -856,7 +1174,7 @@ fn test_replay_file_mutex_with_rpc_and_cache_flags() { (&["--rpc.cache-dir", "/tmp/cache"], "--rpc.cache-dir"), (&["--rpc.clear-cache"], "--rpc.clear-cache"), (&["--rpc.no-cache-file"], "--rpc.no-cache-file"), - (&["--rpc.cache-size", "256"], "--rpc.cache-size"), + (&["--rpc.cache-max-entries", "256"], "--rpc.cache-max-entries"), ]; for (extra_flags, label) in cases { let mut argv = vec!["mega-evme", "--rpc.replay-file", "/tmp/replay.json"]; diff --git a/bin/mega-evme/tests/replay_batch.rs b/bin/mega-evme/tests/replay_batch.rs new file mode 100644 index 00000000..2d335e52 --- /dev/null +++ b/bin/mega-evme/tests/replay_batch.rs @@ -0,0 +1,2042 @@ +//! Offline integration tests for `mega-evme replay --block` / `--tx-file`. +//! +//! Replaying whole blocks needs an RPC capture covering every transaction of +//! each block, which is far larger than a single-transaction capture. The +//! envelope is committed as a gzipped archive and extracted into a temporary +//! directory once per test binary, so these run in CI without setup. They are +//! the only tests that exercise the batch driver's multi-target paths: more +//! than one target in a block, whole-block mode, grouping targets across +//! blocks, sweeping targets on both sides of a mid-block abort, and +//! block-global log indexing against real logs. +//! +//! Set `MEGA_EVME_TEST_ENVELOPE` to replay against a different capture instead. +//! +//! To regenerate the archive, capture both blocks into one envelope (the second +//! run merges into the first) and repack it. `--verify-receipt` is what puts the +//! receipts in the capture, which the verification and fixture-dump tests need: +//! +//! ```bash +//! for block in 22945844 22945853; do +//! mega-evme replay --rpc --rpc.capture-file replay_batch_blocks.cache.json \ +//! --block "$block" --verify-receipt --json +//! done +//! tar -czf replay_batch_blocks.cache.json.tar.gz replay_batch_blocks.cache.json +//! ``` +//! +//! The endpoint must serve state at those blocks; a pruning node fails every +//! target with "state at block #N is pruned". + +use std::process::Command; + +mod common; + +/// Block fully covered by the envelope, and its transaction count. +const BLOCK: u64 = 22_945_844; +const BLOCK_TX_COUNT: usize = 23; + +/// Sample transactions of `BLOCK`: the index-0 deposit, a mid-block call, and +/// the last transaction. +const BLOCK_TXS: [(&str, u64); 3] = [ + ("0xde3d56dc739484166b8af1bea757bf7e3e9a4b9a0fb62d722703345570dfc1d6", 0), + ("0x323ddc8e67dfc134284d78c65f3c1dc7ff45ba1db02eeaf62e211ae3253478ef", 3), + ("0xb6a0b7a302c741f64b8e46861a3dcb2d5c1047f6f2cb89a35b5c2183c96296b7", 22), +]; + +/// A mid-block type-0x2 call (not a deposit): zeroing its gas makes the block +/// executor reject the transaction as invalid and abort — unlike deposits, +/// which can still halt as `FailedDeposit` without aborting the block. +const EXEC_ABORT_TX: (&str, u64) = + ("0xa637d68cda9423d67826e008b1c90295193f30f19cd74a6f4acf54022d56cae2", 2); + +/// Non-target body transaction between `BLOCK_TXS[1]` (index 3) and +/// `BLOCK_TXS[2]` (index 22). Used to abort after an early dumpable target +/// without putting the aborter on the reported target list. +const MID_BLOCK_NON_TARGET: (&str, u64) = + ("0x63e032fdff2676824fd6a71df09d88f62146d07effc7e4ed7e246034df2e9b22", 4); + +/// Last transaction of the envelope's second block. +const OTHER_BLOCK: u64 = 22_945_853; +const OTHER_BLOCK_TX: &str = "0x18302160f2395069a44e1654d173fa9eed95ead8f922f12bfe07b6bdcc0a14f2"; +const OTHER_BLOCK_TX_INDEX: u64 = 23; + +/// Path of the offline envelope. +/// +/// `MEGA_EVME_TEST_ENVELOPE` overrides the committed capture with another one. +fn envelope() -> String { + if let Ok(path) = std::env::var("MEGA_EVME_TEST_ENVELOPE") { + return path; + } + common::fixture(ENVELOPE_NAME).display().to_string() +} + +/// Name of the committed capture, stored compressed alongside the other fixtures. +const ENVELOPE_NAME: &str = "replay_batch_blocks.cache.json"; + +fn mega_evme() -> Command { + Command::new(env!("CARGO_BIN_EXE_mega-evme")) +} + +/// Run `replay` offline and return its stdout, asserting the exit status. +fn replay(args: &[&str], expect_success: bool) -> String { + let envelope = envelope(); + let mut cmd = mega_evme(); + cmd.args(["replay", "--rpc.replay-file", &envelope]); + cmd.args(args); + let output = cmd.output().expect("failed to run mega-evme"); + assert_eq!( + output.status.success(), + expect_success, + "unexpected exit status for {args:?}\nstderr: {}", + String::from_utf8_lossy(&output.stderr), + ); + String::from_utf8(output.stdout).expect("stdout is utf-8") +} + +/// Run `replay` offline and return its stdout plus its exit code. +fn replay_with_code(args: &[&str]) -> (String, Option) { + let envelope = envelope(); + let mut cmd = mega_evme(); + cmd.args(["replay", "--rpc.replay-file", &envelope]); + cmd.args(args); + let output = cmd.output().expect("failed to run mega-evme"); + (String::from_utf8(output.stdout).expect("stdout is utf-8"), output.status.code()) +} + +/// Write a copy of the envelope whose `eth_getTransactionByHash` response for +/// `tx_hash` answers "unknown transaction", and return its path. +/// +/// Entries are keyed by the request, not the response, so the doctored answer +/// still resolves. This models the endpoint losing one transaction of a block it +/// still serves — the block body lists the hash, the lookup denies it. +fn envelope_without_transaction(name: &str, tx_hash: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + // Only the transaction's own response carries it as the `hash` field; the + // block body lists bare hashes and a receipt names it `transactionHash`. + let marker = format!("\"hash\":\"{tx_hash}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + response["result"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one response for {tx_hash}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope with the `eth_getTransactionByHash` entry for +/// `tx_hash` removed entirely, and return its path. +/// +/// Distinct from [`envelope_without_transaction`]: there the endpoint answers +/// "no such transaction", here it does not answer at all — an offline cache +/// miss, which is how a transport failure reaches the same call site. +fn envelope_dropping_transaction(name: &str, tx_hash: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let marker = format!("\"hash\":\"{tx_hash}\""); + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + let before = entries.len(); + entries.retain(|entry| { + !entry["value"].as_str().expect("entry value is a string").contains(&marker) + }); + assert_eq!( + before - entries.len(), + 1, + "the envelope must hold exactly one response for {tx_hash}" + ); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope whose `eth_getBalance` answer for `tx_hash`'s +/// sender at its parent block is zero, and return its path. +/// +/// The served transaction stays byte-identical — it still authenticates against +/// the requested hash — but the block executor rejects it (the sender cannot +/// fund its gas) and aborts the block: an execution-class abort raised through +/// served *state*, which carries no proof and cannot be authenticated the way a +/// consensus object can. +fn envelope_with_drained_sender(name: &str, tx_hash: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + // The transaction's own response names its sender and inclusion block. + let marker = format!("\"hash\":\"{tx_hash}\""); + let mut sender_block: Option<(String, u64)> = None; + for entry in envelope["cache"].as_array().expect("cache entries") { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = &response["result"]; + let from = result["from"].as_str().expect("transaction `from`").to_string(); + let number = u64::from_str_radix( + result["blockNumber"].as_str().expect("blockNumber").trim_start_matches("0x"), + 16, + ) + .expect("hex block number"); + assert!( + sender_block.replace((from, number)).is_none(), + "the envelope must hold exactly one response for {tx_hash}" + ); + } + let (from, number) = sender_block.expect("the envelope must hold the transaction"); + // The state fork reads the sender at the parent block. The entry is keyed + // by the same `method\x00params` digest the capturing transport writes, so + // the key is recomputed rather than searched for by value. + let params = format!("[\"{from}\",\"0x{:x}\"]", number - 1); + let key = format!("{}", alloy_primitives::keccak256(format!("eth_getBalance\x00{params}"))); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + if entry["key"].as_str() != Some(key.as_str()) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(entry["value"].as_str().expect("entry value is a string")) + .expect("parse balance response"); + response["result"] = serde_json::Value::String("0x0".into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold the sender's parent-block balance"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope whose `eth_getTransactionByHash` response for +/// `tx_hash` still returns the transaction object, but with `gas` set to `0x0`. +/// +/// The tampered body no longer hashes to the requested hash, so the replay must +/// refuse to execute it: authentication fails before the transaction reaches +/// the block executor. Models a tampered capture (or a corrupted backend) +/// serving a body that does not match the hash it was asked for. +fn envelope_with_zero_gas_transaction(name: &str, tx_hash: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let marker = format!("\"hash\":\"{tx_hash}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object for {tx_hash}"); + result["gas"] = serde_json::Value::String("0x0".into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one response for {tx_hash}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope whose `eth_getTransactionByHash` response for +/// `tx_hash` keeps the signed body byte-identical but reports a different +/// `from` address, and return its path. +/// +/// A signed transaction's `from` is not part of its encoding — it is derived +/// from the signature — so the tampered answer still hashes to the requested +/// hash. Executing it would run the transaction under the wrong sender; +/// authentication must instead re-derive the signer and reject the served +/// `from`. +fn envelope_with_reassigned_sender(name: &str, tx_hash: &str) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let marker = format!("\"hash\":\"{tx_hash}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object for {tx_hash}"); + result["from"] = + serde_json::Value::String("0x000000000000000000000000000000000000dead".into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one response for {tx_hash}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope whose `eth_getBlockByNumber` answer for block +/// `number` is null, and return its path. +/// +/// The height was resolved by the endpoint's own answers (the target's +/// inclusion metadata names the block, whose parent must then exist), so the +/// null models an endpoint contradicting itself across a reorg or divergent +/// load-balanced views — not a user asking about an unknown height. +fn envelope_without_block(name: &str, number: u64) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let number_hex = format!("0x{number:x}"); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains("\"transactions\"") { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse block response"); + if response["result"]["number"].as_str() != Some(number_hex.as_str()) { + continue; + } + response["result"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one header for block {number}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Write a copy of the envelope whose header for block `number` advertises +/// `gas_limit`, and return its path. +/// +/// Shrinking the advertised limit makes the block-gas admission check reject +/// the first transaction whose own gas limit exceeds what remains — a +/// deterministic execution-class rejection whose error names no transaction +/// hash, raised while that transaction is in flight. +fn envelope_with_block_gas_limit(name: &str, number: u64, gas_limit: u64) -> std::path::PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let number_hex = format!("0x{number:x}"); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains("\"transactions\"") { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse block response"); + let result = &mut response["result"]; + if result["number"].as_str() != Some(number_hex.as_str()) { + continue; + } + result["gasLimit"] = serde_json::Value::String(format!("0x{gas_limit:x}")); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one header for block {number}"); + + let path = + std::env::temp_dir().join(format!("mega_evme_batch_{name}_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + path +} + +/// Run `replay` against `envelope_path` and return its stdout plus its exit code. +fn replay_envelope_with_code( + envelope_path: &std::path::Path, + args: &[&str], +) -> (String, Option) { + let (stdout, _stderr, code) = replay_envelope_full(envelope_path, args); + (stdout, code) +} + +/// Run `replay` against `envelope_path` and return stdout, stderr, and exit code. +fn replay_envelope_full( + envelope_path: &std::path::Path, + args: &[&str], +) -> (String, String, Option) { + let mut cmd = mega_evme(); + cmd.args(["replay", "--rpc.replay-file", envelope_path.to_str().expect("path is utf-8")]); + cmd.args(args); + let output = cmd.output().expect("failed to run mega-evme"); + ( + String::from_utf8(output.stdout).expect("stdout is utf-8"), + String::from_utf8(output.stderr).expect("stderr is utf-8"), + output.status.code(), + ) +} + +/// Parse NDJSON stdout into one JSON value per line, dropping the structured +/// error object a failing run ends with. +fn ndjson(stdout: &str) -> Vec { + let mut lines: Vec = stdout + .lines() + .map(|line| { + assert!(!line.trim().is_empty(), "NDJSON output must not contain blank lines"); + serde_json::from_str(line) + .unwrap_or_else(|e| panic!("stdout line is not compact JSON ({e}): {line}")) + }) + .collect(); + if lines.last().is_some_and(common::is_run_error) { + lines.pop(); + } + lines +} + +/// The structured error object a failing `--json` run ends with. +fn run_error(stdout: &str) -> serde_json::Value { + let last = stdout.lines().last().unwrap_or_else(|| panic!("stdout must not be empty")); + let value: serde_json::Value = serde_json::from_str(last) + .unwrap_or_else(|e| panic!("last stdout line is not compact JSON ({e}): {last}")); + assert!(common::is_run_error(&value), "the last line must be the error object: {value}"); + value +} + +/// The structured error object a failing single-transaction `--json` run ends +/// with. +/// +/// Single-transaction output is pretty-printed rather than NDJSON, so the +/// object is recovered by streaming every JSON value on stdout instead of +/// reading the last line. +fn single_run_error(stdout: &str) -> serde_json::Value { + let values = common::json_values(stdout); + let last = + values.last().unwrap_or_else(|| panic!("a failing --json run must not leave stdout empty")); + assert!(common::is_run_error(last), "the last stdout value must be the error object: {last}"); + last.clone() +} + +/// `--block N --json` emits exactly one NDJSON line per transaction of the +/// block, in transaction order, and exits 0. +#[test] +fn test_replay_block_emits_one_ndjson_line_per_transaction() { + let stdout = replay(&["--block", &BLOCK.to_string(), "--json"], true); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "expected one line per transaction of the block"); + for (index, line) in lines.iter().enumerate() { + assert_eq!( + line["block_number"].as_u64(), + Some(BLOCK), + "every line must report the replayed block: {line}" + ); + assert_eq!( + line["tx_index"].as_u64(), + Some(index as u64), + "lines must be ordered by transaction index: {line}" + ); + assert!(line["tx_hash"].is_string(), "line must carry the transaction hash: {line}"); + assert!(line["receipt"].is_object(), "line must carry the receipt: {line}"); + assert!(line.get("error").is_none(), "line must not be an error entry: {line}"); + // Batch mode rejects the trace/dump flags, so those fields never appear. + assert!(line.get("trace").is_none(), "batch output must carry no trace: {line}"); + assert!(line.get("state").is_none(), "batch output must carry no state dump: {line}"); + } +} + +/// A batch line and a single-transaction replay of the same transaction must +/// agree on the execution outcome. +#[test] +fn test_replay_batch_matches_single_transaction_replay() { + let batch = ndjson(&replay(&["--block", &BLOCK.to_string(), "--json"], true)); + + for (tx_hash, tx_index) in BLOCK_TXS { + let single: serde_json::Value = serde_json::from_str(&replay(&["--json", tx_hash], true)) + .expect("single-transaction output is JSON"); + let line = batch + .iter() + .find(|line| line["tx_hash"] == tx_hash) + .unwrap_or_else(|| panic!("batch output is missing {tx_hash}")); + + assert_eq!(line["tx_index"].as_u64(), Some(tx_index), "wrong index for {tx_hash}"); + for field in ["success", "gas_used", "logs_count"] { + assert_eq!( + line[field], single[field], + "batch and single-transaction replay disagree on {field} for {tx_hash}", + ); + } + } +} + +/// `--tx-file` replays transactions from several blocks in one process and +/// reports them ordered by (block, transaction index). +#[test] +fn test_replay_tx_file_spans_blocks_in_order() { + // Deliberately unordered, with a comment, a blank line, and a duplicate. + let list = format!( + "# sample corpus\n{OTHER_BLOCK_TX}\n\n{}\n {}\n{}\n{}\n", + BLOCK_TXS[2].0, BLOCK_TXS[0].0, BLOCK_TXS[1].0, BLOCK_TXS[0].0, + ); + let path = std::env::temp_dir().join(format!("mega_evme_tx_list_{}.txt", std::process::id())); + std::fs::write(&path, list).expect("write tx list"); + + let stdout = replay(&["--tx-file", path.to_str().unwrap(), "--json"], true); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + let observed: Vec<(u64, u64, &str)> = lines + .iter() + .map(|line| { + ( + line["block_number"].as_u64().expect("block number"), + line["tx_index"].as_u64().expect("transaction index"), + line["tx_hash"].as_str().expect("transaction hash"), + ) + }) + .collect(); + let expected: Vec<(u64, u64, &str)> = vec![ + (BLOCK, BLOCK_TXS[0].1, BLOCK_TXS[0].0), + (BLOCK, BLOCK_TXS[1].1, BLOCK_TXS[1].0), + (BLOCK, BLOCK_TXS[2].1, BLOCK_TXS[2].0), + (OTHER_BLOCK, OTHER_BLOCK_TX_INDEX, OTHER_BLOCK_TX), + ]; + + assert_eq!(observed, expected, "results must be ordered by (block, transaction index)"); +} + +/// A hash that cannot be resolved is reported as an error entry, the remaining +/// targets still replay, and the process exits non-zero with the class of the +/// failure — here an unanswered lookup against the offline envelope. +#[test] +fn test_replay_tx_file_reports_unresolved_targets_and_exits_nonzero() { + let unknown = "0x0000000000000000000000000000000000000000000000000000000000000001"; + let path = + std::env::temp_dir().join(format!("mega_evme_tx_list_bad_{}.txt", std::process::id())); + std::fs::write(&path, format!("{unknown}\n{}\n", BLOCK_TXS[1].0)).expect("write tx list"); + + let (stdout, code) = replay_with_code(&["--tx-file", path.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), 2, "every target gets exactly one line, including failures"); + assert_eq!(lines[0]["tx_hash"].as_str(), Some(unknown)); + assert!(lines[0]["error"]["kind"].is_string(), "failure line carries an error kind"); + assert!(lines[0]["error"]["message"].is_string(), "failure line carries a message"); + assert_eq!(lines[1]["tx_hash"].as_str(), Some(BLOCK_TXS[1].0)); + assert_eq!(lines[1]["success"].as_bool(), Some(true), "the resolvable target still replays"); + + // A hash the envelope cannot answer is an RPC-class failure for the run. + assert_eq!(code, Some(3), "an unanswered target exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// `--verify-receipt` against an envelope that carries no receipts: every target +/// becomes an `rpc` error entry (unverified), never a mismatch, and the run +/// exits non-zero. +/// +/// The development envelope is captured by replays, which do not fetch receipts, +/// `--block N --verify-receipt` verifies every transaction of the block against +/// its on-chain receipt and exits 0 when they all reproduce. +/// +/// This is the multi-target verification fan-out: one receipt fetch and one +/// verdict per target, each carried on that target's own result line. +#[test] +fn test_replay_block_verify_receipt_reports_a_verdict_per_target() { + let (stdout, code) = + replay_with_code(&["--block", &BLOCK.to_string(), "--verify-receipt", "--json"]); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is reported exactly once"); + for line in &lines { + assert!(line.get("error").is_none(), "a verified target is not an error entry: {line}"); + assert_eq!( + line["verification"]["match"].as_bool(), + Some(true), + "every target must reproduce its on-chain receipt: {line}" + ); + } + assert_eq!(code, Some(0), "a fully matching run exits 0"); +} + +/// An abort caused by a block-body transaction resolving to null is not a +/// definitive "unknown hash": the hash came from the block the endpoint already +/// served, so the null is an RPC inconsistency. The aborting target is reported +/// as `rpc` (not `not_found`), and every target swept up behind it is also +/// unanswered (`rpc`) with a message naming the transaction that aborted the +/// block. +#[test] +fn test_replay_block_sweeps_targets_behind_an_abort_as_unanswered() { + let (missing, missing_index) = BLOCK_TXS[1]; + let path = envelope_without_transaction("abort_block", missing); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--block", &BLOCK.to_string(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is still reported exactly once"); + for (index, line) in lines.iter().enumerate() { + let index = index as u64; + if index < missing_index { + assert!(line.get("error").is_none(), "targets before the abort replay: {line}"); + continue; + } + if index == missing_index { + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "a block-body hash resolving to null is an RPC inconsistency: {line}" + ); + assert!( + line["error"]["message"].as_str().is_some_and(|m| { + m.contains(missing) && + m.contains("Block body") && + m.contains("resolves it to null") + }), + "the aborting error must name the hash and the inconsistency: {line}" + ); + continue; + } + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "a target swept up behind the abort went unanswered: {line}" + ); + assert!( + line["error"]["message"].as_str().is_some_and(|m| m.contains(missing)), + "the message must name the transaction that aborted the block: {line}" + ); + } + + // Every failure is RPC-class (inconsistency + unanswered sweeps), so the + // run exits 3 rather than the definitive-answer class of a user-supplied + // unknown hash. + assert_eq!(code, Some(3), "a block-body null lookup exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// A preceding transaction of a single-transaction replay resolving to null is +/// an endpoint inconsistency (exit 3), not a definitive unknown transaction. +/// +/// The single-transaction path derives its preceding hashes from the block body +/// the endpoint already served, so the batch driver's reasoning applies +/// unchanged: a null lookup contradicts an answer the endpoint gave itself. +/// Doctors the index-0 transaction's response and replays a mid-block target, +/// which executes that transaction before its own. +#[test] +fn test_replay_single_transaction_preceding_null_is_an_rpc_failure() { + let missing = BLOCK_TXS[0].0; + let (target, target_index) = BLOCK_TXS[1]; + assert!(target_index > 0, "the target must have preceding transactions to execute"); + let path = envelope_without_transaction("single_preceding_null", missing); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "a preceding block-body null exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + error["error"]["message"].as_str().is_some_and(|m| { + m.contains(missing) && m.contains("Block body") && m.contains("resolves it to null") + }), + "the failure must name the hash and the inconsistency: {error}" + ); +} + +/// The user-supplied target of a single-transaction replay resolving to null +/// keeps the definitive-answer class (exit 1). +/// +/// Nothing the endpoint served claims that hash exists, so the null is an +/// answer about the caller's own question rather than a contradiction — the one +/// lookup on this path that stays `TransactionNotFound`. +#[test] +fn test_replay_single_transaction_target_null_is_not_found() { + let target = BLOCK_TXS[1].0; + let path = envelope_without_transaction("single_target_null", target); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(1), "an unknown user-supplied hash exits 1: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["code"].as_u64(), Some(1)); + assert_eq!(error["error"]["kind"].as_str(), Some("execution-error")); + assert!( + error["error"]["message"] + .as_str() + .is_some_and(|m| m.contains("Transaction not found") && m.contains(target)), + "the failure must stay a definitive not-found naming the hash: {error}" + ); +} + +/// A preceding transaction the endpoint never answers is an unanswered +/// question, not a null answer: same exit class (3), different message. +/// +/// Pins that reclassifying the null answer did not swallow the transport +/// failure reaching the same call site. +#[test] +fn test_replay_single_transaction_preceding_transport_error_is_an_rpc_failure() { + let missing = BLOCK_TXS[0].0; + let target = BLOCK_TXS[1].0; + let path = envelope_dropping_transaction("single_preceding_miss", missing); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "an unanswered preceding lookup exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!(message.contains("cache miss"), "the failure must name the missed request: {error}"); + assert!( + !message.contains("resolves it to null"), + "an unanswered lookup is not a null answer: {error}" + ); +} + +/// An executor/setup abort on a mid-block transaction is still an execution +/// failure for that transaction only: every target behind it is unanswered +/// (`rpc`), not blamed as execution. +/// +/// Doctors the sender's parent-block balance to zero — the lookup succeeds and +/// the transaction authenticates, but the block executor rejects it as an +/// invalid transaction (the sender cannot fund its gas) and aborts the block — +/// an execution-class error, not `TransactionNotFound`. +#[test] +fn test_replay_block_sweeps_targets_behind_execution_abort_as_rpc() { + let (aborting, aborting_index) = EXEC_ABORT_TX; + let path = envelope_with_drained_sender("exec_abort_block", aborting); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--block", &BLOCK.to_string(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is still reported exactly once"); + for (index, line) in lines.iter().enumerate() { + let index = index as u64; + if index < aborting_index { + assert!(line.get("error").is_none(), "targets before the abort replay: {line}"); + continue; + } + if index == aborting_index { + assert_eq!( + line["error"]["kind"].as_str(), + Some("execution"), + "the aborting transaction keeps its own execution kind: {line}" + ); + continue; + } + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "a target swept up behind an execution abort went unanswered: {line}" + ); + assert!( + line["error"]["message"].as_str().is_some_and(|m| { + m.contains(aborting) || m.contains("aborted") || m.contains("Block replay") + }), + "the message must name the abort cause: {line}" + ); + } + + // The aborting transaction is an execution-class failure, which outranks + // the unanswered ones. + assert_eq!(code, Some(1), "a definitive execution failure exits 1"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("execution-error")); +} + +/// A non-target deterministic executor abort still exits 1: swept targets stay +/// `rpc` ("unanswered"), but the run floors the exit on the abort's own class so +/// a retryable exit is not reported for a permanent failure. +/// +/// Per-target totals stay truthful ("2 of 2"): the abort is not a synthetic +/// third target failure. +/// +/// `EXEC_ABORT_TX`'s sender is drained and the transaction kept out of the +/// `--tx-file` target list; only later targets of the same block are requested. +#[test] +fn test_replay_tx_file_non_target_execution_abort_exits_execution() { + let (aborting, aborting_index) = EXEC_ABORT_TX; + let (target_a, target_a_index) = BLOCK_TXS[1]; + let (target_b, _) = BLOCK_TXS[2]; + assert!(target_a_index > aborting_index, "targets must sit behind the non-target aborter"); + let path = envelope_with_drained_sender("non_target_exec_abort", aborting); + let list = format!("{target_a}\n{target_b}\n"); + let list_path = std::env::temp_dir() + .join(format!("mega_evme_tx_list_non_target_exec_{}.txt", std::process::id())); + std::fs::write(&list_path, list).expect("write tx list"); + + let (stdout, stderr, code) = + replay_envelope_full(&path, &["--tx-file", list_path.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list_path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), 2, "only requested targets are reported: {stdout}"); + for line in &lines { + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "swept target stays unanswered: {line}" + ); + assert!( + line["error"]["message"].as_str().is_some_and(|m| { + m.contains(aborting) || m.contains("aborted") || m.contains("Block replay") + }), + "the message must name the non-target abort: {line}" + ); + } + + assert_eq!(code, Some(1), "a non-target execution abort exits 1, not 3"); + let err = run_error(&stdout); + assert_eq!(err["error"]["kind"].as_str(), Some("execution-error")); + let message = err["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("2 of 2 target transaction(s) failed"), + "aggregate must stay truthful about targets (not 3 of 2): {message}" + ); + assert!( + stderr.contains("2 of 2 target transaction(s) failed") || + message.contains("2 of 2 target transaction(s) failed"), + "stderr/stdout aggregate must not count the non-target abort as a target: \ + stderr={stderr}\nmessage={message}" + ); +} + +/// A served transaction whose body does not hash to the requested hash must +/// not execute: the batch refuses it as a failed body-listed fetch (`rpc`) and +/// sweeps the targets behind it, instead of advancing the block state on the +/// wrong transaction. +/// +/// Doctors a mid-block transaction's `gas` — any body change breaks the hash. +#[test] +fn test_replay_block_tampered_transaction_body_fails_authentication_as_rpc() { + let (tampered, tampered_index) = EXEC_ABORT_TX; + let path = envelope_with_zero_gas_transaction("tampered_body_block", tampered); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--block", &BLOCK.to_string(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is still reported exactly once"); + for (index, line) in lines.iter().enumerate() { + let index = index as u64; + if index < tampered_index { + assert!( + line.get("error").is_none(), + "targets before the tampered fetch replay: {line}" + ); + continue; + } + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "a tampered or swept target went unanswered — never an execution verdict: {line}" + ); + } + let message = lines[tampered_index as usize]["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("the endpoint served a different transaction"), + "the tampered target must name the authentication failure: {message}" + ); + + assert_eq!(code, Some(3), "an unauthenticated body-listed fetch exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// A served transaction whose signed body authenticates but whose `from` field +/// does not match the signature's signer must be refused: executing it would +/// run the transaction under the wrong sender. +#[test] +fn test_replay_single_transaction_reassigned_sender_fails_authentication() { + let (target, _) = BLOCK_TXS[1]; + let path = envelope_with_reassigned_sender("single_reassigned_from", target); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "an unauthenticated target fetch exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("does not match the signer") && message.contains(target), + "the failure must name the sender mismatch and the transaction: {message}" + ); +} + +/// A tampered preceding transaction is refused before execution on the single +/// path too: the target's pre-state depends on it, so the run fails as a failed +/// body-listed fetch (`rpc`) naming the tampered hash. +#[test] +fn test_replay_single_transaction_tampered_preceding_fails_authentication() { + let (tampered, tampered_index) = EXEC_ABORT_TX; + let (target, target_index) = BLOCK_TXS[1]; + assert!(target_index > tampered_index, "the tampered transaction must precede the target"); + let path = envelope_with_zero_gas_transaction("single_tampered_preceding", tampered); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "an unauthenticated preceding fetch exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("Block body lists transaction") && + message.contains(tampered) && + message.contains("served a different transaction"), + "the failure must name the tampered body-listed fetch: {message}" + ); +} + +/// A parent block the endpoint itself resolved — the target claims inclusion +/// in its child — answering null is an endpoint self-contradiction: a +/// retryable infrastructure failure (exit 3, matching the batch path), not a +/// definitive exit-1 "block not found". +#[test] +fn test_replay_single_transaction_null_resolved_parent_is_an_rpc_failure() { + let (target, _) = BLOCK_TXS[1]; + let path = envelope_without_block("single_null_parent", BLOCK - 1); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "a null resolved parent exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("divergent views") && message.contains(&(BLOCK - 1).to_string()), + "the failure must name the divergent view and the block: {message}" + ); +} + +/// The replayed block itself answering null after the target's metadata named +/// it is the same self-contradiction as a null parent: exit 3, not exit 1. +#[test] +fn test_replay_single_transaction_null_resolved_block_is_an_rpc_failure() { + let (target, _) = BLOCK_TXS[1]; + let path = envelope_without_block("single_null_block", BLOCK); + + let (stdout, code) = replay_envelope_with_code(&path, &["--json", target]); + let _ = std::fs::remove_file(&path); + + assert_eq!(code, Some(3), "a null resolved block exits 3: {stdout}"); + let error = single_run_error(&stdout); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("divergent views") && message.contains(&BLOCK.to_string()), + "the failure must name the divergent view and the block: {message}" + ); +} + +/// A deterministic rejection that does not name its transaction still lands on +/// the transaction it was raised about: the in-flight target keeps the +/// execution-class abort as its own answer, and only the targets behind it are +/// swept as unanswered. +/// +/// Shrinks the block's advertised gas limit so admission rejects the index-1 +/// transaction (`TransactionGasLimitMoreThanAvailableBlockGas` names no hash); +/// the index-0 deposit still fits the shrunken limit. +#[test] +fn test_replay_block_hashless_abort_lands_on_the_in_flight_target() { + let path = envelope_with_block_gas_limit("hashless_abort", BLOCK, 200_000_000); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--block", &BLOCK.to_string(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is still reported exactly once"); + assert!(lines[0].get("error").is_none(), "the deposit fits the shrunken limit: {}", lines[0]); + let aborter = &lines[1]; + assert_eq!( + aborter["error"]["kind"].as_str(), + Some("execution"), + "the in-flight target keeps the hashless rejection as its own answer: {aborter}" + ); + let message = aborter["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("gas") && !message.contains("aborted before this transaction"), + "the aborter's line carries the rejection itself, not a swept notice: {message}" + ); + for line in &lines[2..] { + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "targets behind the abort went unanswered: {line}" + ); + } + + assert_eq!(code, Some(1), "a deterministic rejection exits 1"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("execution-error")); +} + +/// A non-target transport abort (cache miss) exits 3 and names the failing +/// fetch so swept entries are distinguishable from the cause. +#[test] +fn test_replay_tx_file_non_target_transport_abort_names_hash_and_exits_rpc() { + let (aborting, aborting_index) = EXEC_ABORT_TX; + let (target_a, target_a_index) = BLOCK_TXS[1]; + let (target_b, _) = BLOCK_TXS[2]; + assert!(target_a_index > aborting_index, "targets must sit behind the non-target aborter"); + let path = envelope_dropping_transaction("non_target_transport_abort", aborting); + let list = format!("{target_a}\n{target_b}\n"); + let list_path = std::env::temp_dir() + .join(format!("mega_evme_tx_list_non_target_rpc_{}.txt", std::process::id())); + std::fs::write(&list_path, list).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--tx-file", list_path.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list_path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), 2, "only requested targets are reported: {stdout}"); + for line in &lines { + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "swept target stays unanswered: {line}" + ); + assert!( + line["error"]["message"].as_str().is_some_and(|m| m.contains(aborting)), + "the abort message must name the failing fetch: {line}" + ); + } + + assert_eq!(code, Some(3), "a transport abort exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// With `--dump-fixture-dir`, a target that executed and drafted a fixture +/// before a later transport abort keeps its result line; only the fixture field +/// fails, and that failure inherits the abort's `rpc` class so the run exits 3 +/// rather than converting the discard into an execution-class error. +#[test] +fn test_replay_tx_file_dump_fixture_inherits_transport_abort_class() { + let (aborting, aborting_index) = MID_BLOCK_NON_TARGET; + // Early non-deposit target builds a Ready draft; a later target forces the + // job past the non-target aborter between them. + let (early, early_index) = BLOCK_TXS[1]; + let (late, late_index) = BLOCK_TXS[2]; + assert!(early_index < aborting_index && aborting_index < late_index); + + let path = envelope_dropping_transaction("fixture_inherits_rpc_abort", aborting); + let list = format!("{early}\n{late}\n"); + let list_path = std::env::temp_dir() + .join(format!("mega_evme_tx_list_fixture_abort_{}.txt", std::process::id())); + std::fs::write(&list_path, list).expect("write tx list"); + let dir = + std::env::temp_dir().join(format!("mega_evme_fixture_abort_dir_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("create dump dir"); + + let (stdout, code) = replay_envelope_with_code( + &path, + &[ + "--tx-file", + list_path.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list_path); + let _ = std::fs::remove_dir_all(&dir); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), 2, "both targets are reported: {stdout}"); + + let early_line = lines.iter().find(|l| l["tx_hash"] == early).expect("early target line"); + assert!( + early_line.get("error").is_none(), + "the early target keeps its execution result line: {early_line}" + ); + assert!( + early_line["success"].is_boolean(), + "result fields are present on the early target: {early_line}" + ); + let fixture = &early_line["fixture"]; + assert!( + fixture["error"].is_string(), + "the drafted fixture must report the abort (not a skip): {early_line}" + ); + assert!( + fixture["error"].as_str().is_some_and(|m| m.contains("aborted") || m.contains("discarded")), + "fixture error names the abort: {early_line}" + ); + assert!(fixture.get("path").is_none(), "no fixture file is written after abort"); + assert!(fixture.get("skipped").is_none(), "abort discard is an error, not a skip"); + + let late_line = lines.iter().find(|l| l["tx_hash"] == late).expect("late target line"); + assert_eq!( + late_line["error"]["kind"].as_str(), + Some("rpc"), + "the late target is swept as unanswered: {late_line}" + ); + assert!( + late_line["error"]["message"].as_str().is_some_and(|m| m.contains(aborting)), + "swept message names the failing fetch: {late_line}" + ); + + // Abort is transport-class; fixture discard inherits it — not exit 1. + assert_eq!(code, Some(3), "transport abort with fixture discard exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// Targets swept up by an abort are reported in block transaction-index order, +/// whatever order `--tx-file` listed them in. +#[test] +fn test_replay_tx_file_sweeps_targets_in_block_order() { + let missing = BLOCK_TXS[0].0; + let path = envelope_without_transaction("abort_order", missing); + // Deliberately reversed: the last transaction of the block first. + let list = format!("{}\n{}\n", BLOCK_TXS[2].0, BLOCK_TXS[1].0); + let list_path = + std::env::temp_dir().join(format!("mega_evme_tx_list_order_{}.txt", std::process::id())); + std::fs::write(&list_path, list).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--tx-file", list_path.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list_path); + let lines = ndjson(&stdout); + + let observed: Vec<&str> = lines.iter().map(|line| line["tx_hash"].as_str().unwrap()).collect(); + assert_eq!( + observed, + vec![BLOCK_TXS[1].0, BLOCK_TXS[2].0], + "swept targets must follow the block's transaction order, not the input order", + ); + for line in &lines { + assert_eq!(line["error"]["kind"].as_str(), Some("rpc"), "swept target: {line}"); + assert!( + line["error"]["message"].as_str().is_some_and(|m| m.contains(missing)), + "the message must name the transaction that aborted the block: {line}" + ); + } + + // No target was answered definitively, so the run is an RPC failure. + assert_eq!(code, Some(3), "targets that went unanswered exit 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// Batch mode rejects the single-transaction-only flags before doing any work. +#[test] +fn test_replay_batch_rejects_single_transaction_flags() { + let envelope = envelope(); + for (extra, expected) in [ + (vec!["--dump-fixture", "/tmp/should-not-exist.json"], "--dump-fixture"), + (vec!["--override.gas-limit", "50000"], "transaction overrides"), + (vec!["--override.spec", "Rex4"], "--override.spec"), + (vec!["--trace"], "trace options"), + (vec!["--dump"], "state dump options"), + ] { + let mut cmd = mega_evme(); + cmd.args(["replay", "--rpc.replay-file", &envelope, "--block", &BLOCK.to_string()]); + cmd.args(&extra); + let output = cmd.output().expect("failed to run mega-evme"); + + assert!(!output.status.success(), "batch mode must reject {extra:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(expected) && stderr.contains("batch replay"), + "unexpected error for {extra:?}: {stderr}" + ); + assert!(output.stdout.is_empty(), "a rejected batch run must print nothing on stdout"); + } +} + +/// A `--tx-file` target whose reported inclusion block is not the block fetched +/// by that number is unanswered, not replayed. +/// +/// The endpoint answers `eth_getTransactionByHash` and `eth_getBlockByNumber` +/// separately, so a reorg or a load-balanced backend can serve two views. The +/// resolution step records the inclusion hash so the mismatch is caught before +/// the block runs, instead of replaying targets against a block they are not in. +#[test] +fn test_replay_tx_file_rejects_a_block_that_does_not_match_the_resolved_inclusion() { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let wrong_hash = "0x2222222222222222222222222222222222222222222222222222222222222222"; + let (target, _) = BLOCK_TXS[1]; + + // Rewrite only the transaction's own response: it now claims to belong to a + // block whose hash differs from the one `eth_getBlockByNumber` returns. + let marker = format!("\"hash\":\"{target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + result["blockHash"] = serde_json::Value::String(wrong_hash.into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the target transaction"); + + let envelope_path = + std::env::temp_dir().join(format!("mega_evme_batch_inclusion_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_inclusion_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 1, "the single target is reported once: {stdout}"); + assert_eq!( + lines[0]["error"]["kind"].as_str(), + Some("rpc"), + "divergent views are unanswered, not a wrong answer: {}", + lines[0] + ); + let message = lines[0]["error"]["message"].as_str().unwrap_or_default(); + assert!(message.contains("divergent views"), "message names the cause: {message}"); + assert_eq!(code, Some(3), "an unanswered target exits 3"); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// Same-block mix of a successful early target and a later inclusion failure: +/// NDJSON line order follows ascending `(block, tx_index)`, so the earlier +/// result precedes the later inclusion failure even when the failure was +/// decided before the execute loop. +/// +/// Looks up by hash alone cannot catch this; the test asserts line positions. +#[test] +fn test_replay_tx_file_same_block_mixed_inclusion_preserves_line_order() { + let (early_target, early_index) = BLOCK_TXS[1]; + let (late_target, late_index) = BLOCK_TXS[2]; + assert!(early_index < late_index); + let wrong_hash = "0x2222222222222222222222222222222222222222222222222222222222222222"; + + // Doctor only the later target's inclusion hash so it fails the membership + // guard while the earlier target still replays. + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let marker = format!("\"hash\":\"{late_target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + result["blockHash"] = serde_json::Value::String(wrong_hash.into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the late target"); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_mixed_order_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + + // List the later (failing) target first so a pre-execution-first emit would + // put its failure line before the earlier result. + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_mixed_order_{}.txt", std::process::id())); + std::fs::write(&list, format!("{late_target}\n{early_target}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + // End-to-end line order: earlier body index first, absent/divergent last. + assert_eq!( + lines[0]["tx_hash"].as_str(), + Some(early_target), + "line 0 must be the earlier target's result, got: {}", + lines[0] + ); + assert!(lines[0].get("error").is_none(), "earlier target still replays: {}", lines[0]); + assert_eq!(lines[0]["tx_index"].as_u64(), Some(early_index)); + assert_eq!(lines[0]["success"].as_bool(), Some(true)); + + assert_eq!( + lines[1]["tx_hash"].as_str(), + Some(late_target), + "line 1 must be the later target's inclusion failure, got: {}", + lines[1] + ); + assert_eq!(lines[1]["error"]["kind"].as_str(), Some("rpc"), "inclusion failure: {}", lines[1]); + + assert_eq!(code, Some(3), "an unanswered target exits 3"); +} + +/// Two same-height targets that report different inclusion hashes get +/// independent outcomes: the one that matches the fetched block replays, the +/// one that does not fails as `rpc`. Outcomes must not depend on file order. +/// +/// The resolution step used to keep a first-seen job-level anchor and reject +/// later peers that disagreed, so a stale target listed first could poison the +/// canonical peer. Both orders are exercised against the same doctored capture. +#[test] +fn test_replay_tx_file_inclusion_mismatch_is_order_independent() { + let (stale_target, _) = BLOCK_TXS[1]; + let (canonical_target, canonical_index) = BLOCK_TXS[2]; + let wrong_hash = "0x2222222222222222222222222222222222222222222222222222222222222222"; + + // Doctor only the stale target's inclusion hash; the canonical target and + // the block body keep their original agreement. + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let marker = format!("\"hash\":\"{stale_target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + result["blockHash"] = serde_json::Value::String(wrong_hash.into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the stale target"); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_inclusion_order_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + + for (label, first, second) in [ + ("stale_first", stale_target, canonical_target), + ("canonical_first", canonical_target, stale_target), + ] { + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_inclusion_order_{label}_{}.txt", std::process::id())); + std::fs::write(&list, format!("{first}\n{second}\n")).expect("write tx list"); + + let (stdout, code) = replay_envelope_with_code( + &envelope_path, + &["--tx-file", list.to_str().unwrap(), "--json"], + ); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "{label}: every target is reported once: {stdout}"); + + let stale = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(stale_target)) + .unwrap_or_else(|| panic!("{label}: stale target must be reported")); + assert_eq!( + stale["error"]["kind"].as_str(), + Some("rpc"), + "{label}: mismatched inclusion is unanswered: {stale}" + ); + let message = stale["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("divergent views") && + message.contains(wrong_hash) && + message.contains("resolved as included"), + "{label}: message names both views: {message}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(canonical_target)) + .unwrap_or_else(|| panic!("{label}: canonical target must be reported")); + assert!(ok.get("error").is_none(), "{label}: matching inclusion still replays: {ok}"); + assert_eq!(ok["block_number"].as_u64(), Some(BLOCK)); + assert_eq!(ok["tx_index"].as_u64(), Some(canonical_index)); + assert_eq!(ok["success"].as_bool(), Some(true)); + + assert_eq!(code, Some(3), "{label}: an unanswered target exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); + + let _ = std::fs::remove_file(&list); + } + + let _ = std::fs::remove_file(&envelope_path); +} + +/// A target whose reported inclusion hash matches the fetched block, but which +/// the block body does not list, is an endpoint self-contradiction (`rpc`), not +/// a definitive `not_found`. +/// +/// The lookup said "in block B"; B's body lacks it. That is the same class as +/// the single-transaction membership guard, not an answer that the hash is +/// unknown. +#[test] +fn test_replay_tx_file_anchored_but_absent_target_is_rpc() { + let (target, _) = BLOCK_TXS[1]; + + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + + // Keep the transaction lookup intact (correct number + hash) but drop the + // hash from the block body it claims to belong to. + let mut body_doctored = 0; + let mut block_hash = None; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + let Ok(mut response) = serde_json::from_str::(value) else { + continue; + }; + let Some(result) = response.get_mut("result") else { + continue; + }; + if !result.is_object() { + continue; + } + let number = result.get("number").and_then(|n| { + n.as_str().and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + }); + if number != Some(BLOCK) { + continue; + } + let Some(txs) = result.get_mut("transactions").and_then(|t| t.as_array_mut()) else { + continue; + }; + let before = txs.len(); + txs.retain(|tx| tx.as_str() != Some(target)); + if txs.len() != before { + block_hash = result.get("hash").and_then(|h| h.as_str()).map(str::to_string); + entry["value"] = serde_json::Value::String(response.to_string()); + body_doctored += 1; + } + } + assert_eq!(body_doctored, 1, "exactly one block body for {BLOCK} must list the target"); + let block_hash = block_hash.expect("block hash"); + + // Pair with another-block target so a clean job still runs alongside the + // inconsistency: only the absent target fails. + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_anchored_absent_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_anchored_absent_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n{OTHER_BLOCK_TX}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .expect("absent target must be reported"); + assert_eq!( + failed["error"]["kind"].as_str(), + Some("rpc"), + "anchored-but-absent is an RPC inconsistency, not not_found: {failed}" + ); + let message = failed["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains(target) && + message.contains(&block_hash) && + message.contains("does not list") && + message.contains("divergent views"), + "message names the target, the block, and the cause: {message}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(OTHER_BLOCK_TX)) + .expect("other-block target must be reported"); + assert!(ok.get("error").is_none(), "targets in other blocks still replay: {ok}"); + assert_eq!(ok["block_number"].as_u64(), Some(OTHER_BLOCK)); + assert_eq!(ok["success"].as_bool(), Some(true)); + + assert_eq!(code, Some(3), "an unanswered target exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// When every target of a job is absent from the fetched block body, each gets +/// its definitive `rpc` answer pre-loop and the block is never executed. +/// +/// After doctoring the body, the envelope is stripped of every response that is +/// not a target transaction lookup or the doctored block body (parent header, +/// state, receipts, body-tx objects). A clean early return needs only those two +/// shapes; forking state or walking the body would miss and fail differently. +#[test] +fn test_replay_tx_file_all_targets_absent_from_body_skips_block_execution() { + let targets: Vec<&str> = BLOCK_TXS.iter().map(|(h, _)| *h).collect(); + + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + + let mut body_doctored = 0; + let mut block_hash = None; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + let Ok(mut response) = serde_json::from_str::(value) else { + continue; + }; + let Some(result) = response.get_mut("result") else { + continue; + }; + if !result.is_object() { + continue; + } + let number = result.get("number").and_then(|n| { + n.as_str().and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + }); + if number != Some(BLOCK) { + continue; + } + let Some(txs) = result.get_mut("transactions").and_then(|t| t.as_array_mut()) else { + continue; + }; + let before = txs.len(); + txs.retain(|tx| { + let hash = tx.as_str().unwrap_or(""); + !targets.contains(&hash) + }); + if txs.len() != before { + block_hash = result.get("hash").and_then(|h| h.as_str()).map(str::to_string); + entry["value"] = serde_json::Value::String(response.to_string()); + body_doctored += 1; + } + } + assert_eq!(body_doctored, 1, "exactly one block body for {BLOCK} must list the targets"); + let block_hash = block_hash.expect("block hash"); + + // Cache keys are request hashes, so filter by response shape: keep only the + // target transaction lookups and the doctored block-at-height body. + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + entries.retain(|entry| { + let value = entry["value"].as_str().unwrap_or(""); + let Ok(response) = serde_json::from_str::(value) else { + return false; + }; + let Some(result) = response.get("result") else { + return false; + }; + if !result.is_object() { + return false; + } + // Transaction lookup for one of our targets. + if let Some(hash) = result.get("hash").and_then(|h| h.as_str()) { + if targets.contains(&hash) && result.get("blockNumber").is_some() { + return true; + } + } + // Doctored block body at the job height. + let number = result.get("number").and_then(|n| { + n.as_str().and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + }); + number == Some(BLOCK) && result.get("transactions").is_some() + }); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_all_absent_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_all_absent_{}.txt", std::process::id())); + std::fs::write(&list, format!("{}\n", targets.join("\n"))).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), targets.len(), "every target is reported once: {stdout}"); + + for target in &targets { + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .unwrap_or_else(|| panic!("target {target} must be reported")); + assert_eq!(failed["error"]["kind"].as_str(), Some("rpc"), "all-absent is rpc: {failed}"); + let message = failed["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("does not list") && message.contains(&block_hash), + "message names absence and block: {message}" + ); + assert!( + failed.get("success").is_none() && failed.get("receipt").is_none(), + "no execution result for an unexecuted target: {failed}" + ); + } + + assert_eq!(code, Some(3), "all-absent exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); + // If the driver had forked state or walked body transactions, the stripped + // envelope would have produced a cache-miss error naming parent/state — not + // a clean per-target membership rpc answer for every hash. + assert!( + !stdout.contains("cache miss") && + !stdout.contains("not found in the offline") && + !stdout.contains("not present in the offline"), + "early return must not touch parent/state/body-tx paths:\n{stdout}" + ); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// A hash whose resolution answers `null` keeps the definitive `not_found` +/// class: the endpoint denied the hash, rather than claiming inclusion and then +/// contradicting itself. +#[test] +fn test_replay_tx_file_null_resolution_stays_not_found() { + let (target, _) = BLOCK_TXS[1]; + let path = envelope_without_transaction("tx_file_null_resolution", target); + + // Pair with a clean other-block target so the run still produces a success + // line next to the definitive not-found. + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_null_resolution_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n{OTHER_BLOCK_TX}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .expect("null-resolution target must be reported"); + assert_eq!( + failed["error"]["kind"].as_str(), + Some("not_found"), + "a null lookup is a definitive not_found: {failed}" + ); + assert_eq!( + failed["error"]["message"].as_str(), + Some("Transaction not found"), + "not_found message is unchanged: {failed}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(OTHER_BLOCK_TX)) + .expect("other-block target must be reported"); + assert!(ok.get("error").is_none(), "targets in other blocks still replay: {ok}"); + assert_eq!(ok["success"].as_bool(), Some(true)); + + // Definitive not_found is an execution-class failure (exit 1), not rpc. + assert_eq!(code, Some(1), "a definitive not_found exits 1"); + + let _ = std::fs::remove_file(&list); +} + +/// A mined `--tx-file` target whose `eth_getTransactionByHash` answer carries a +/// block number but no inclusion hash is unanswered: the endpoint served an +/// unanchored view, so the target is not queued and other blocks still replay. +#[test] +fn test_replay_tx_file_rejects_mined_target_without_inclusion_hash() { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let (target, _) = BLOCK_TXS[1]; + + // Keep the block number so the response still looks mined, but drop the + // inclusion hash. Cache entries are keyed by the request, so the doctored + // answer still resolves. + let marker = format!("\"hash\":\"{target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + assert!( + result.get("blockNumber").is_some_and(|n| !n.is_null()), + "fixture transaction must report a block number" + ); + result["blockHash"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the target transaction"); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_null_inclusion_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + // Pair the unanchored target with one from another block so a clean job + // still runs when resolution fails for only one hash. + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_null_inclusion_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n{OTHER_BLOCK_TX}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .expect("doctored target must be reported"); + assert_eq!( + failed["error"]["kind"].as_str(), + Some("rpc"), + "a mined target without an inclusion hash is unanswered: {failed}" + ); + let message = failed["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("inclusion hash") && message.contains("unanchored"), + "message names the unanchored view: {message}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(OTHER_BLOCK_TX)) + .expect("other-block target must be reported"); + assert!(ok.get("error").is_none(), "targets in other blocks still replay: {ok}"); + assert_eq!(ok["block_number"].as_u64(), Some(OTHER_BLOCK)); + assert_eq!(ok["success"].as_bool(), Some(true)); + + assert_eq!(code, Some(3), "an unanswered target exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// A `--tx-file` target whose `eth_getTransactionByHash` answer carries an +/// inclusion hash but no block number is unanswered: the endpoint served +/// contradictory metadata, so the target is not treated as pending. +#[test] +fn test_replay_tx_file_rejects_inclusion_hash_without_block_number() { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let (target, _) = BLOCK_TXS[1]; + + // Keep the inclusion hash so the response still claims a mined block, but + // drop the block number. Cache entries are keyed by the request, so the + // doctored answer still resolves. + let marker = format!("\"hash\":\"{target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + assert!( + result.get("blockHash").is_some_and(|h| !h.is_null()), + "fixture transaction must report an inclusion hash" + ); + assert!( + result.get("blockNumber").is_some_and(|n| !n.is_null()), + "fixture transaction must report a block number before doctoring" + ); + result["blockNumber"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the target transaction"); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_null_number_with_hash_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + // Pair the contradictory target with one from another block so a clean job + // still runs when resolution fails for only one hash. + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_null_number_with_hash_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n{OTHER_BLOCK_TX}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .expect("doctored target must be reported"); + assert_eq!( + failed["error"]["kind"].as_str(), + Some("rpc"), + "contradictory metadata is unanswered as rpc, not pending: {failed}" + ); + let message = failed["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("contradictory") && + (message.contains("without a block number") || message.contains("block number")), + "message names the contradiction: {message}" + ); + assert!( + !message.contains("pending"), + "contradictory metadata must not be classified as pending: {message}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(OTHER_BLOCK_TX)) + .expect("other-block target must be reported"); + assert!(ok.get("error").is_none(), "targets in other blocks still replay: {ok}"); + assert_eq!(ok["block_number"].as_u64(), Some(OTHER_BLOCK)); + assert_eq!(ok["success"].as_bool(), Some(true)); + + assert_eq!(code, Some(3), "an unanswered target exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// A genuinely pending `--tx-file` target (`blockNumber` and `blockHash` both +/// null) keeps the pending classification and is not queued for replay. +#[test] +fn test_replay_tx_file_classifies_null_number_and_hash_as_pending() { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let (target, _) = BLOCK_TXS[1]; + + let marker = format!("\"hash\":\"{target}\""); + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&marker) { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + let result = response.get_mut("result").expect("transaction result"); + assert!(result.is_object(), "expected a transaction object"); + result["blockNumber"] = serde_json::Value::Null; + result["blockHash"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "exactly one response describes the target transaction"); + + let envelope_path = std::env::temp_dir() + .join(format!("mega_evme_batch_pending_nulls_{}.json", std::process::id())); + std::fs::write(&envelope_path, envelope.to_string()).expect("write doctored envelope"); + let list = std::env::temp_dir() + .join(format!("mega_evme_tx_list_pending_nulls_{}.txt", std::process::id())); + std::fs::write(&list, format!("{target}\n{OTHER_BLOCK_TX}\n")).expect("write tx list"); + + let (stdout, code) = + replay_envelope_with_code(&envelope_path, &["--tx-file", list.to_str().unwrap(), "--json"]); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), 2, "every target is reported once: {stdout}"); + + let failed = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(target)) + .expect("doctored target must be reported"); + assert_eq!( + failed["error"]["kind"].as_str(), + Some("pending"), + "null number and null hash is pending: {failed}" + ); + let message = failed["error"]["message"].as_str().unwrap_or_default(); + assert_eq!( + message, "Transaction is pending (no block number)", + "pending message is unchanged: {message}" + ); + + let ok = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(OTHER_BLOCK_TX)) + .expect("other-block target must be reported"); + assert!(ok.get("error").is_none(), "targets in other blocks still replay: {ok}"); + assert_eq!(ok["block_number"].as_u64(), Some(OTHER_BLOCK)); + assert_eq!(ok["success"].as_bool(), Some(true)); + + // Pending counts as an execution-class failure (exit 1), not rpc (exit 3). + assert_eq!(code, Some(1), "a pending target exits 1"); + + let _ = std::fs::remove_file(&envelope_path); + let _ = std::fs::remove_file(&list); +} + +/// infrastructure failure for every target of that block (reorg / divergent views). +#[test] +fn test_replay_block_rejects_mismatched_parent_hash() { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(envelope()).expect("read envelope")) + .expect("parse envelope"); + let wrong_parent = "0x1111111111111111111111111111111111111111111111111111111111111111"; + let mut expected_parent = None; + let mut doctored = 0; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + let Ok(mut response) = serde_json::from_str::(value) else { + continue; + }; + let Some(result) = response.get_mut("result") else { + continue; + }; + if !result.is_object() { + continue; + } + // Doctor the parent block (number == BLOCK - 1), not the target block. + let number = result.get("number").and_then(|n| { + n.as_str().and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + }); + if number != Some(BLOCK - 1) { + continue; + } + let original = result.get("hash").and_then(|h| h.as_str()).map(str::to_string); + assert!(original.is_some(), "parent block must report a hash"); + expected_parent = original; + result["hash"] = serde_json::Value::String(wrong_parent.into()); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored += 1; + } + assert_eq!(doctored, 1, "the envelope must hold exactly one parent-block body for {BLOCK}"); + let expected_parent = expected_parent.expect("parent hash"); + + let path = std::env::temp_dir() + .join(format!("mega_evme_batch_parent_mismatch_{}.json", std::process::id())); + std::fs::write(&path, envelope.to_string()).expect("write doctored envelope"); + + let (stdout, code) = + replay_envelope_with_code(&path, &["--block", &BLOCK.to_string(), "--json"]); + let _ = std::fs::remove_file(&path); + let lines = ndjson(&stdout); + + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is still reported exactly once"); + for line in &lines { + assert_eq!( + line["error"]["kind"].as_str(), + Some("rpc"), + "a parent/block linkage failure is an infrastructure error: {line}" + ); + let message = line["error"]["message"].as_str().unwrap_or(""); + assert!( + message.contains(wrong_parent) && message.contains(&expected_parent), + "the message must name both hashes (got parent {expected_parent}, wrong {wrong_parent}): {line}" + ); + } + assert_eq!(code, Some(3), "an infrastructure failure exits 3"); + assert_eq!(run_error(&stdout)["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// Receipts from a late multi-log transaction stamp every inner log with the +/// outer block/tx identity and a block-global `logIndex` that starts above zero +/// (preceding receipts already emitted logs). +#[test] +fn test_replay_receipt_inner_log_metadata_nonzero_preceding_offset() { + // Last transaction of BLOCK: multi-log, with many preceding logs in-block. + const LATE_TX: &str = "0xb6a0b7a302c741f64b8e46861a3dcb2d5c1047f6f2cb89a35b5c2183c96296b7"; + + let stdout = replay(&["--json", LATE_TX], true); + let summary = common::json_values(&stdout) + .into_iter() + .find(|v| v.get("receipt").is_some()) + .expect("replay summary with receipt"); + let receipt = &summary["receipt"]; + let block_hash = receipt["blockHash"].as_str().expect("blockHash"); + let tx_hash = receipt["transactionHash"].as_str().expect("transactionHash"); + let logs = receipt["logs"].as_array().expect("logs"); + assert!(!logs.is_empty(), "late tx must emit logs"); + let first = u64::from_str_radix( + logs[0]["logIndex"].as_str().expect("logIndex").trim_start_matches("0x"), + 16, + ) + .expect("parse logIndex"); + assert!(first > 0, "expected non-zero preceding-log offset, got {first}"); + for (i, log) in logs.iter().enumerate() { + assert_eq!(log["blockHash"].as_str(), Some(block_hash), "log {i}"); + assert_eq!(log["transactionHash"].as_str(), Some(tx_hash), "log {i}"); + assert!(log["logIndex"].is_string(), "log {i} logIndex"); + } +} + +/// Sweeping a block with `--dump-fixture-dir` writes a fixture for every +/// transaction it can express and skips genuine unsupported shapes (deposit) +/// without failing the run. +/// +/// Every OP-stack block opens with a deposit, which the fixture format cannot +/// represent. Reporting that as an error rather than a skip would make a +/// whole-block sweep exit non-zero on every block, so this pins the +/// classification end to end: 22 files written, the deposit skipped with its +/// reason, nothing reported as an error, and exit 0. +/// +/// (An unanswered on-chain receipt is a separate rpc-class fixture error and is +/// covered by the doctored dump-dir tests in `replay_verify`.) +#[test] +fn test_replay_block_dump_fixture_dir_writes_all_but_the_deposit() { + let dir = std::env::temp_dir() + .join(format!("mega_evme_dump_dir_sweep_{}_{BLOCK}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let stdout = replay( + &["--block", &BLOCK.to_string(), "--dump-fixture-dir", dir.to_str().unwrap(), "--json"], + true, + ); + let lines = ndjson(&stdout); + assert_eq!(lines.len(), BLOCK_TX_COUNT, "every target is reported exactly once"); + + let mut written = 0; + let mut skipped = Vec::new(); + for line in &lines { + assert!(line.get("error").is_none(), "a sweep must not produce error entries: {line}"); + if line["fixture"]["path"].is_string() { + written += 1; + } else { + skipped.push( + line["fixture"]["skipped"] + .as_str() + .unwrap_or_else(|| panic!("a line reported neither path nor skip: {line}")) + .to_string(), + ); + } + } + + assert_eq!(skipped.len(), 1, "only the index-0 deposit is unsupported: {skipped:?}"); + assert!( + skipped[0].contains("does not support deposit"), + "the skip must name the reason: {}", + skipped[0] + ); + assert_eq!(written, BLOCK_TX_COUNT - 1, "every other transaction is dumped"); + + let on_disk = std::fs::read_dir(&dir).expect("read dump dir").count(); + assert_eq!(on_disk, written, "each reported path is a file on disk"); + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/bin/mega-evme/tests/replay_dump.rs b/bin/mega-evme/tests/replay_dump.rs index 516e0661..a3a24f83 100644 --- a/bin/mega-evme/tests/replay_dump.rs +++ b/bin/mega-evme/tests/replay_dump.rs @@ -9,14 +9,23 @@ //! `state-test --bench`; see `bench/replay/`.) use std::{ - process::Command, + path::{Path, PathBuf}, + process::{Command, Output}, sync::{Arc, Mutex}, time::Duration, }; -/// Offline RPC capture (includes the on-chain receipt needed by the fidelity gate). -const CACHE: &str = - concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/replay_offline.cache.json"); +mod common; + +/// Offline RPC capture (includes the on-chain receipt needed by the fidelity +/// gate). Name of the committed offline capture, resolved through the shared fixture +/// helper so its location lives in exactly one place. +const CACHE: &str = "replay_offline.cache.json"; + +/// Path of the committed offline capture. +fn cache() -> PathBuf { + common::fixture(CACHE) +} /// The transaction captured in `CACHE` (a 75,514-gas Rex5 mainnet call). const TX: &str = "0x41d34e7e13dfe0f85da9d407e2b2c381955d8c7eed428b17dc82327b2616b000"; @@ -25,6 +34,122 @@ fn mega_evme() -> Command { Command::new(env!("CARGO_BIN_EXE_mega-evme")) } +/// A temp path unique to this process and this test. +fn temp_path(name: &str) -> PathBuf { + std::env::temp_dir().join(format!("mega_evme_dump_{name}_{}.json", std::process::id())) +} + +/// Run `replay --dump-fixture` offline against `cache`, writing to `out`. +fn dump(cache: &Path, out: &Path) -> Output { + mega_evme() + .args([ + "replay", + "--rpc.replay-file", + cache.to_str().expect("cache path is utf-8"), + "--dump-fixture", + out.to_str().expect("fixture path is utf-8"), + "--json", + TX, + ]) + .output() + .expect("failed to run mega-evme") +} + +/// Run `replay --verify-receipt` offline against `cache`, the mode whose +/// classification the dump path reuses. +fn verify(cache: &Path) -> Output { + mega_evme() + .args([ + "replay", + "--rpc.replay-file", + cache.to_str().expect("cache path is utf-8"), + "--verify-receipt", + "--json", + TX, + ]) + .output() + .expect("failed to run mega-evme") +} + +/// The structured error object a failing `--json` run ends its stdout with. +fn error_object(stdout: &str) -> serde_json::Value { + let values = common::json_values(stdout); + let last = values + .last() + .unwrap_or_else(|| panic!("a failing --json run must not leave stdout empty:\n{stdout}")); + assert!(common::is_run_error(last), "the last stdout value must be the error object: {last}"); + last["error"].clone() +} + +/// Write a copy of the committed capture in which the `result` of every cached +/// response `selects` accepts is rewritten by `doctor`, and return its path. +/// +/// Cache entries are keyed by the request, not the response, so a doctored entry +/// still resolves and the run meets the doctored answer where it would meet the +/// real one. +fn rewrite_cache( + name: &str, + selects: impl Fn(&serde_json::Value) -> bool, + doctor: impl Fn(&mut serde_json::Value), +) -> PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let mut doctored = false; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse cached response"); + if !selects(&response["result"]) { + continue; + } + doctor(&mut response["result"]); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored = true; + } + assert!(doctored, "offline capture should contain the entry being doctored"); + + let path = temp_path(name); + std::fs::write(&path, envelope.to_string()).expect("write doctored cache"); + path +} + +/// A copy of the committed capture whose on-chain receipt response is rewritten +/// by `doctor`. The receipt is the only cached response carrying +/// `cumulativeGasUsed`. +fn cache_with_doctored_receipt(name: &str, doctor: impl Fn(&mut serde_json::Value)) -> PathBuf { + rewrite_cache(name, |result| result.get("cumulativeGasUsed").is_some(), doctor) +} + +/// A copy of the committed capture with the on-chain receipt dropped entirely. +/// Offline, the absent entry surfaces as a cache miss — the transport failing to +/// answer the receipt request at all. +fn cache_without_receipt(name: &str) -> PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + let before = entries.len(); + entries.retain(|entry| { + !entry["value"].as_str().expect("entry value is a string").contains("cumulativeGasUsed") + }); + assert!(entries.len() < before, "offline capture should contain the receipt entry"); + + let path = temp_path(name); + std::fs::write(&path, envelope.to_string()).expect("write pruned cache"); + path +} + +/// A copy of the committed capture whose target-transaction lookup answers null, +/// modelling an endpoint that does not know the hash the caller asked about. +fn cache_with_null_target_transaction(name: &str) -> PathBuf { + rewrite_cache( + name, + |result| result.get("hash").and_then(serde_json::Value::as_str) == Some(TX), + |result| *result = serde_json::Value::Null, + ) +} + /// `--dump-fixture` is incompatible with transaction overrides (the isolated /// execution would not represent the on-chain transaction), and must be /// rejected before any execution, writing nothing. @@ -33,11 +158,12 @@ fn test_replay_dump_rejects_transaction_overrides() { let out = std::env::temp_dir().join(format!("mega_evme_dump_ovr_{}.json", std::process::id())); let _ = std::fs::remove_file(&out); + let cache = cache(); let output = mega_evme() .args([ "replay", "--rpc.replay-file", - CACHE, + cache.to_str().unwrap(), "--dump-fixture", out.to_str().unwrap(), "--override.gas-limit", @@ -64,8 +190,16 @@ fn test_replay_dump_fixture_writes_validatable_file() { let out = std::env::temp_dir().join(format!("mega_evme_dump_{}.json", std::process::id())); let _ = std::fs::remove_file(&out); + let cache = cache(); let output = mega_evme() - .args(["replay", "--rpc.replay-file", CACHE, "--dump-fixture", out.to_str().unwrap(), TX]) + .args([ + "replay", + "--rpc.replay-file", + cache.to_str().unwrap(), + "--dump-fixture", + out.to_str().unwrap(), + TX, + ]) .output() .expect("failed to run mega-evme"); @@ -93,11 +227,12 @@ fn test_replay_dump_is_byte_reproducible() { let out = std::env::temp_dir() .join(format!("mega_evme_repro_{}_{suffix}.json", std::process::id())); let _ = std::fs::remove_file(&out); + let cache = cache(); let output = mega_evme() .args([ "replay", "--rpc.replay-file", - CACHE, + cache.to_str().unwrap(), "--dump-fixture", out.to_str().unwrap(), TX, @@ -121,21 +256,28 @@ fn test_replay_dump_is_byte_reproducible() { ); } -/// Dumping over an existing fixture must go through a sibling temp file + -/// rename: on success the target holds the new (valid) content and no -/// `.json.tmp` residue is left behind, so an interrupt mid-write can no longer -/// truncate a committed corpus fixture. +/// Dumping over an existing fixture must go through a unique temp file + +/// persist: on success the target holds the new (valid) content and no +/// leftover temp files remain in the destination directory, so an interrupt +/// mid-write can no longer truncate a committed corpus fixture. #[test] fn test_replay_dump_overwrites_atomically_without_tmp_residue() { let out = std::env::temp_dir().join(format!("mega_evme_dump_atomic_{}.json", std::process::id())); - let tmp = out.with_extension("json.tmp"); - let _ = std::fs::remove_file(&tmp); + let _ = std::fs::remove_file(&out); // Seed a pre-existing "committed" fixture that the dump overwrites in place. std::fs::write(&out, br#"{"pre-existing":"corpus fixture"}"#).expect("seed existing fixture"); + let cache = cache(); let output = mega_evme() - .args(["replay", "--rpc.replay-file", CACHE, "--dump-fixture", out.to_str().unwrap(), TX]) + .args([ + "replay", + "--rpc.replay-file", + cache.to_str().unwrap(), + "--dump-fixture", + out.to_str().unwrap(), + TX, + ]) .output() .expect("failed to run mega-evme"); @@ -144,7 +286,20 @@ fn test_replay_dump_overwrites_atomically_without_tmp_residue() { "dump over an existing fixture failed.\nstderr: {}", String::from_utf8_lossy(&output.stderr) ); - assert!(!tmp.exists(), "dump must not leave a .json.tmp file behind"); + // NamedTempFile uses a random name; ensure only the destination remains. + let parent = out.parent().expect("temp dir"); + let stem = out.file_stem().and_then(|s| s.to_str()).expect("utf-8 stem"); + let leftovers: Vec<_> = std::fs::read_dir(parent) + .expect("list temp dir") + .filter_map(|e| e.ok()) + .filter(|e| { + let name = e.file_name(); + let name = name.to_string_lossy(); + name.starts_with(stem) && name != out.file_name().unwrap().to_string_lossy() + }) + .map(|e| e.path()) + .collect(); + assert!(leftovers.is_empty(), "dump must not leave temp residue: {leftovers:?}"); let content = std::fs::read_to_string(&out).expect("read dumped fixture"); let _ = std::fs::remove_file(&out); @@ -161,61 +316,211 @@ fn test_replay_dump_overwrites_atomically_without_tmp_residue() { /// The fidelity gate must reject a receipt that describes a different inclusion /// than the replayed block (a reorg in progress, or a load-balanced endpoint -/// serving divergent views). Doctor the captured receipt's `blockHash` and -/// expect a clear error with no fixture written. +/// serving divergent views). The endpoint served a view the run cannot use, so +/// this is a retryable infrastructure failure (exit 3) — the same class +/// `--verify-receipt` gives the identical condition — and no fixture is written. #[test] fn test_replay_dump_rejects_receipt_from_different_block() { - // Doctor the capture: flip the receipt's blockHash. Cache entries are keyed - // by the request, not the response, so the doctored entry still resolves. - let mut envelope: serde_json::Value = - serde_json::from_str(&std::fs::read_to_string(CACHE).expect("read offline cache")) - .expect("parse offline cache"); - let mut doctored = false; - for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { - let value = entry["value"].as_str().expect("entry value is a string"); - // The receipt is the only cached response carrying cumulativeGasUsed. - if !value.contains("cumulativeGasUsed") { - continue; - } - let mut response: serde_json::Value = - serde_json::from_str(value).expect("parse receipt response"); - response["result"]["blockHash"] = serde_json::Value::String( - "0x1111111111111111111111111111111111111111111111111111111111111111".into(), - ); - entry["value"] = serde_json::Value::String(response.to_string()); - doctored = true; - } - assert!(doctored, "offline cache should contain the receipt entry"); - - let doctored_cache = - std::env::temp_dir().join(format!("mega_evme_reorg_cache_{}.json", std::process::id())); - std::fs::write(&doctored_cache, envelope.to_string()).expect("write doctored cache"); - let out = - std::env::temp_dir().join(format!("mega_evme_dump_reorg_{}.json", std::process::id())); + let doctored_cache = cache_with_doctored_receipt("reorg_cache", |receipt| { + receipt["blockHash"] = + "0x1111111111111111111111111111111111111111111111111111111111111111".into(); + }); + let out = temp_path("reorg"); let _ = std::fs::remove_file(&out); - let output = mega_evme() - .args([ - "replay", - "--rpc.replay-file", - doctored_cache.to_str().unwrap(), - "--dump-fixture", - out.to_str().unwrap(), - TX, - ]) - .output() - .expect("failed to run mega-evme"); + let output = dump(&doctored_cache, &out); let _ = std::fs::remove_file(&doctored_cache); - assert!(!output.status.success(), "a receipt from a different block must abort the dump"); let stderr = String::from_utf8_lossy(&output.stderr); + assert_eq!( + output.status.code(), + Some(3), + "a receipt from a different block is an infrastructure failure.\nstderr: {stderr}", + ); assert!( stderr.contains("different inclusion"), "expected reorg/divergent-endpoint hint, got stderr:\n{stderr}" ); + let error = error_object(&String::from_utf8_lossy(&output.stdout)); + assert_eq!(error["kind"].as_str(), Some("rpc-failure"), "got: {error}"); assert!(!out.exists(), "must not write a fixture when the receipt anchor mismatches"); } +/// The fidelity gate must reject a receipt that describes a different +/// transaction than the one requested: anchoring a fixture to another +/// transaction's gas, status and logs would bake a wrong expectation into the +/// artifact. The endpoint answered a question that was never asked, so this is an +/// infrastructure failure (exit 3) and no fixture is written. +#[test] +fn test_replay_dump_rejects_receipt_for_another_transaction() { + const OTHER_TX: &str = "0x00000000000000000000000000000000000000000000000000000000feed0001"; + + let doctored_cache = cache_with_doctored_receipt("wrong_tx_cache", |receipt| { + receipt["transactionHash"] = OTHER_TX.into(); + }); + let out = temp_path("wrong_tx"); + let _ = std::fs::remove_file(&out); + + let output = dump(&doctored_cache, &out); + let _ = std::fs::remove_file(&doctored_cache); + + let stderr = String::from_utf8_lossy(&output.stderr); + assert_eq!( + output.status.code(), + Some(3), + "a receipt for another transaction is an infrastructure failure.\nstderr: {stderr}", + ); + let error = error_object(&String::from_utf8_lossy(&output.stdout)); + assert_eq!(error["kind"].as_str(), Some("rpc-failure"), "got: {error}"); + let message = error["message"].as_str().expect("the error object carries a message"); + assert!( + message.contains(OTHER_TX) && message.contains(TX), + "the message must name both the served and the requested transaction: {message}" + ); + assert!(!out.exists(), "must not write a fixture when the receipt is for another transaction"); +} + +/// A receipt the endpoint answers with `null` leaves the fidelity gate's +/// question unanswered: the run resolved this very transaction as mined moments +/// earlier, so the null is a pruned receipt or a divergent backend, not a +/// definitive "unknown transaction". It must exit 3 as an infrastructure +/// failure, name the receipt, and write no fixture. +#[test] +fn test_replay_dump_null_receipt_is_an_rpc_failure() { + let doctored_cache = cache_with_doctored_receipt("null_receipt_cache", |receipt| { + *receipt = serde_json::Value::Null; + }); + let out = temp_path("null_receipt"); + let _ = std::fs::remove_file(&out); + + let output = dump(&doctored_cache, &out); + let _ = std::fs::remove_file(&doctored_cache); + + let stderr = String::from_utf8_lossy(&output.stderr); + assert_eq!( + output.status.code(), + Some(3), + "an unanswered receipt is an infrastructure failure.\nstderr: {stderr}", + ); + let error = error_object(&String::from_utf8_lossy(&output.stdout)); + assert_eq!(error["kind"].as_str(), Some("rpc-failure"), "got: {error}"); + let message = error["message"].as_str().expect("the error object carries a message"); + assert!( + message.contains("No on-chain receipt for transaction") && message.contains(TX), + "the message must name the unanswered receipt and its transaction: {message}" + ); + assert!( + !message.contains("Transaction not found"), + "a replayed transaction must not be reported as unknown: {message}" + ); + assert!(!out.exists(), "must not write a fixture when the receipt is unanswered"); +} + +/// A receipt request the transport never answers (offline: the capture holds no +/// receipt entry) is the third receipt-fetch anomaly, and stays an +/// infrastructure failure with no fixture written. +#[test] +fn test_replay_dump_unanswered_receipt_request_is_an_rpc_failure() { + let pruned_cache = cache_without_receipt("pruned_receipt_cache"); + let out = temp_path("pruned_receipt"); + let _ = std::fs::remove_file(&out); + + let output = dump(&pruned_cache, &out); + let _ = std::fs::remove_file(&pruned_cache); + + let stderr = String::from_utf8_lossy(&output.stderr); + assert_eq!( + output.status.code(), + Some(3), + "an unanswered receipt request is an infrastructure failure.\nstderr: {stderr}", + ); + let error = error_object(&String::from_utf8_lossy(&output.stdout)); + assert_eq!(error["kind"].as_str(), Some("rpc-failure"), "got: {error}"); + assert!( + error["message"].as_str().is_some_and(|m| m.contains("eth_getTransactionReceipt")), + "the message must name the unanswered request: {error}" + ); + assert!(!out.exists(), "must not write a fixture when the receipt request goes unanswered"); +} + +/// Negative control for the classification above: a null answer for the *target +/// transaction* is the endpoint answering the caller's own question with a +/// definitive "unknown transaction", and keeps its execution class (exit 1). +/// Only the fidelity gate's receipt question moved to the infrastructure class, +/// not every null the run can meet. +#[test] +fn test_replay_dump_null_target_transaction_stays_an_execution_error() { + let doctored_cache = cache_with_null_target_transaction("null_target_cache"); + let out = temp_path("null_target"); + let _ = std::fs::remove_file(&out); + + let output = dump(&doctored_cache, &out); + let _ = std::fs::remove_file(&doctored_cache); + + let stderr = String::from_utf8_lossy(&output.stderr); + assert_eq!( + output.status.code(), + Some(1), + "an unknown target transaction is a definitive negative answer.\nstderr: {stderr}", + ); + let error = error_object(&String::from_utf8_lossy(&output.stdout)); + assert_eq!(error["kind"].as_str(), Some("execution-error"), "got: {error}"); + assert!( + error["message"].as_str().is_some_and(|m| m.contains("Transaction not found")), + "got: {error}" + ); + assert!(!out.exists(), "must not write a fixture when the target is unknown"); +} + +/// The dump path and the verify path fetch the same receipt for the same +/// transaction, so an endpoint anomaly must produce the identical failure: same +/// exit code, same class, same message. A pipeline can then branch on the exit +/// code without knowing which mode produced it. +#[test] +fn test_dump_and_verify_classify_receipt_anomalies_identically() { + // A receipt the endpoint does not serve at all. + assert_dump_and_verify_agree("agree_null_receipt", |receipt| { + *receipt = serde_json::Value::Null; + }); + // A receipt describing a different inclusion than the replayed block. + assert_dump_and_verify_agree("agree_reorg", |receipt| { + receipt["blockHash"] = + "0x1111111111111111111111111111111111111111111111111111111111111111".into(); + }); + // A receipt describing a different transaction than the one requested. + assert_dump_and_verify_agree("agree_wrong_tx", |receipt| { + receipt["transactionHash"] = + "0x00000000000000000000000000000000000000000000000000000000feed0001".into(); + }); +} + +/// Run both modes against a capture whose receipt response is rewritten by +/// `doctor`, and assert the two runs fail with the same exit code and the same +/// error object. +fn assert_dump_and_verify_agree(name: &str, doctor: impl Fn(&mut serde_json::Value)) { + let doctored_cache = cache_with_doctored_receipt(&format!("{name}_cache"), doctor); + let out = temp_path(name); + let _ = std::fs::remove_file(&out); + + let dumped = dump(&doctored_cache, &out); + let verified = verify(&doctored_cache); + let _ = std::fs::remove_file(&doctored_cache); + let _ = std::fs::remove_file(&out); + + assert_eq!( + dumped.status.code(), + verified.status.code(), + "{name}: dump and verify must exit with the same code.\ndump stderr: {}\nverify stderr: {}", + String::from_utf8_lossy(&dumped.stderr), + String::from_utf8_lossy(&verified.stderr), + ); + assert_eq!( + error_object(&String::from_utf8_lossy(&dumped.stdout)), + error_object(&String::from_utf8_lossy(&verified.stdout)), + "{name}: dump and verify must report the same failure object", + ); +} + /// `--dump-fixture` must reject `--override.spec` (a forced spec would make the /// fixture a what-if, not the on-chain transaction) and write nothing. #[test] @@ -223,11 +528,12 @@ fn test_replay_dump_rejects_spec_override() { let out = std::env::temp_dir().join(format!("mega_evme_dump_spec_{}.json", std::process::id())); let _ = std::fs::remove_file(&out); + let cache = cache(); let output = mega_evme() .args([ "replay", "--rpc.replay-file", - CACHE, + cache.to_str().unwrap(), "--dump-fixture", out.to_str().unwrap(), "--override.spec", diff --git a/bin/mega-evme/tests/replay_halt_logs.rs b/bin/mega-evme/tests/replay_halt_logs.rs new file mode 100644 index 00000000..c4df5bf7 --- /dev/null +++ b/bin/mega-evme/tests/replay_halt_logs.rs @@ -0,0 +1,94 @@ +//! Mainnet regression: a halted transaction's receipt must carry no logs. +//! +//! revm 27 gave `ExecutionResult::Halt` no `logs` field, so "a failed transaction's receipt has no +//! logs" was guaranteed by the type. revm 40 puts a log list on every variant and fills it from +//! `journal.take_logs()`. `MegaETH` rewrites an already-committed frame result into a failure — +//! pre-REX5 a CREATE's code-deposit compute gas is recorded once the constructor's checkpoint is +//! committed — so the committed logs reached the receipt and changed its logs root. +//! +//! A full-history replay of the pre-REX4 range caught three mainnet transactions doing exactly +//! that. They are captured here with their on-chain receipts, so the regression is pinned against +//! the chain rather than against a hand-written expectation: `--verify-receipt` compares status, +//! gas and logs, and fails the run on any difference. +//! +//! Runs fully offline — `--rpc.replay-file` never falls back to the network, and a cache miss is a +//! hard error. The unit-level coverage of the same defect lives in the `mega-evm` crate's +//! per-spec test suites; this file is the end-to-end half. + +use std::{path::PathBuf, process::Command}; + +mod common; + +/// Offline RPC capture: the three transactions, their on-chain receipts, the state their blocks +/// need, and the external-env snapshot. Stored compressed; resolved through the shared helper. +const CACHE: &str = "halt_logs_repro.cache.json"; + +/// The captured transactions. All three are large mainnet CREATEs on the `Rex` spec whose +/// constructor emitted a log before the post-commit code-deposit charge halted the transaction; +/// each on-chain receipt records zero logs. +const TXS: [&str; 3] = [ + "0x002ecbc328e5259b3756b69a221fc7ff7956dd616a9d872eda1701914bb6f3cc", + "0x0a85678457f7b5db647f6ecd05f1ccaf17c5ef2df771d02126a73fa8b41865bb", + "0xac0ae5fc76d7939fc55015d8865799412235387926dcf1444084c63e07ddf565", +]; + +fn cache() -> PathBuf { + common::fixture(CACHE) +} + +fn replay(tx: &str, args: &[&str]) -> (bool, String, String) { + let output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args(["replay", "--rpc.replay-file", cache().to_str().expect("cache path is utf-8")]) + .args(args) + .arg(tx) + .output() + .expect("failed to run mega-evme"); + ( + output.status.success(), + String::from_utf8(output.stdout).expect("stdout is utf-8"), + String::from_utf8(output.stderr).expect("stderr is utf-8"), + ) +} + +/// Each captured transaction replays to its on-chain receipt exactly. Before the log strip these +/// exited 2 with `logs_count: onchain 0 vs replay 1`. +#[test] +fn test_halted_mainnet_creates_reproduce_their_onchain_receipts() { + for tx in TXS { + let (success, stdout, stderr) = replay(tx, &["--verify-receipt", "--json"]); + + assert!(success, "{tx} must verify against its on-chain receipt.\nstderr: {stderr}"); + let result = common::json_values(&stdout) + .pop() + .unwrap_or_else(|| panic!("{tx} produced no JSON result")); + assert_eq!( + result["verification"], + serde_json::json!({ "match": true }), + "{tx} must report a receipt match, got: {result}", + ); + } +} + +/// The receipt each of them replays to reports failure and carries no logs — the window this +/// regression is about. Asserted separately from the match above so a capture that somehow lost +/// its on-chain receipts cannot let the previous test pass vacuously. +/// +/// The assertion reads the emitted receipt, not the summary's `logs_count`: that field is only +/// populated on the success arm of the outcome builder and reports zero for every failed result, +/// so it cannot distinguish a leaking replay from a clean one. +#[test] +fn test_halted_mainnet_creates_report_failure_with_no_logs() { + for tx in TXS { + let (success, stdout, stderr) = replay(tx, &["--json"]); + + assert!(success, "{tx} must replay.\nstderr: {stderr}"); + let result = common::json_values(&stdout) + .pop() + .unwrap_or_else(|| panic!("{tx} produced no JSON result")); + assert_eq!(result["success"], serde_json::json!(false), "{tx} halted on-chain"); + let logs = result["receipt"]["logs"] + .as_array() + .unwrap_or_else(|| panic!("{tx} produced no receipt logs array: {result}")); + assert!(logs.is_empty(), "{tx} must replay with an empty receipt log list, got: {logs:?}"); + } +} diff --git a/bin/mega-evme/tests/replay_override_spec.rs b/bin/mega-evme/tests/replay_override_spec.rs new file mode 100644 index 00000000..61d29a01 --- /dev/null +++ b/bin/mega-evme/tests/replay_override_spec.rs @@ -0,0 +1,450 @@ +//! Integration tests for `mega-evme replay --override.spec`. +//! +//! The override is a coherent what-if: the whole execution world switches to the +//! forced spec, as if the block had run on a chain at that spec. These tests pin +//! the consequences that are visible from outside the process — the pre-block +//! predeploys (their presence and their version) and the block-level resource +//! limits follow the override rather than the block's position in the chain's +//! schedule, and a fork whose parameters the chain never published cannot be +//! forced at all. +//! +//! They run against a mock JSON-RPC endpoint rather than a recorded capture: a +//! higher-spec override reads state the historical replay never touched (the +//! `SequencerRegistry` account, for one), and an offline capture answers a miss +//! with a hard error rather than the state the forced world needs. + +use std::process::Command; + +use serde_json::{json, Value}; + +mod common; +use common::MockRpcServer; + +/// `MegaETH` mainnet: the chain whose published schedule carries the Rex5 +/// `SequencerRegistry` parameters that a Rex5+ override needs. +const CHAIN_ID: u64 = 4326; + +/// The replayed block. Its parent is `BLOCK_NUMBER - 1`. +const BLOCK_NUMBER: u64 = 18_172_461; + +/// A mainnet timestamp inside the `MiniRex` window, well before Rex4 (the first +/// fork that deploys `MegaLimitControl`) and Rex5 (the first that deploys the +/// `SequencerRegistry`). Forcing a newer spec on this block is what makes the +/// two worlds — historical and forced — visibly different. +const MINI_REX_TIMESTAMP: u64 = 1_764_000_000; + +/// Signature of the replayed transaction: a fixed, well-formed secp256k1 pair. +/// +/// The replay authenticates every served transaction — its hash is recomputed +/// from the encoding and its sender re-derived from the signature — so the mock +/// cannot serve invented `hash`/`from` constants. The authentic identity is +/// computed by [`tx_identity`] from the transaction being built; the sender is +/// whatever address this signature recovers to for it, funded like every other +/// account by the mock's blanket balance. +const SIG_R: &str = "0xa19f0f1f52e2951452711b4f4aa5d177442c9a56abeb609b803fe2412ed24946"; +const SIG_S: &str = "0x7af21777b2e7d91c745d0077ba2726ee1bb75ccf00039a6218d64fdced768491"; + +/// Hash of the replayed block. +const BLOCK_HASH: &str = "0x2801837c261826beb8047e46139dfc4eb93ab5b3196ce23f312d3c7658262a62"; + +/// Hash of the parent block, which the replay forks its state from. +const PARENT_HASH: &str = "0xd482d481e9d11dd116ef6c41bf95ca608f159206c8f07900b1b53936d196ccb3"; + +/// Hash of the grandparent, so the parent block is a well-formed header. +const GRANDPARENT_HASH: &str = "0x152b00e0c659a9ea0827f7d3b7666951c100bb6a6761a90e20ed7f79099a82e1"; + +/// `SequencerRegistry`, deployed pre-block from Rex5 on. +const SEQUENCER_REGISTRY: &str = "0x6342000000000000000000000000000000000006"; + +/// `MegaLimitControl`, deployed pre-block from Rex4 on. +const LIMIT_CONTROL: &str = "0x6342000000000000000000000000000000000005"; + +/// `version()` — declared by `ISemver`, implemented by the registry bytecode, +/// and answering with the deployed version string. +const VERSION_SELECTOR: &str = "0x54fd4d50"; + +/// ABI-encoded `"1.0.0"`: the version the pre-Rex6 `SequencerRegistry` reports. +const VERSION_1_0_0: &str = concat!( + "0x", + "0000000000000000000000000000000000000000000000000000000000000020", + "0000000000000000000000000000000000000000000000000000000000000005", + "312e302e30000000000000000000000000000000000000000000000000000000", +); + +/// ABI-encoded `"2.0.0"`: the version the Rex6 `SequencerRegistry` reports. +const VERSION_2_0_0: &str = concat!( + "0x", + "0000000000000000000000000000000000000000000000000000000000000020", + "0000000000000000000000000000000000000000000000000000000000000005", + "322e302e30000000000000000000000000000000000000000000000000000000", +); + +/// A chain id with no published schedule. `mega-evm` answers those with an +/// all-activated schedule that carries every registry parameter type, so it is +/// the counterpart to mainnet for the parameter-availability cases below. +const UNKNOWN_CHAIN_ID: u64 = 0xdead_beef; + +/// Outcome of one `mega-evme replay` invocation. +struct Run { + code: Option, + stdout: String, + stderr: String, +} + +impl Run { + /// The single `--json` summary the run printed. + fn summary(&self) -> Value { + let mut values = common::json_values(&self.stdout); + if values.last().is_some_and(common::is_run_error) { + values.pop(); + } + assert_eq!( + values.len(), + 1, + "expected one summary on stdout:\n{}\nstderr:\n{}", + self.stdout, + self.stderr, + ); + values.pop().expect("checked above") + } + + /// The hex return data of a successful run, or `None` when the call + /// returned nothing (an account with no code answers empty). + fn output(&self) -> Option { + let summary = self.summary(); + assert_eq!( + summary["success"], + json!(true), + "the replay must succeed:\n{}\nstderr:\n{}", + self.stdout, + self.stderr, + ); + summary["output"].as_str().map(str::to_string) + } +} + +/// A block header the RPC backend and the replay accept, carrying only the +/// fields either of them reads. +fn block_json(number: u64, hash: &str, parent_hash: &str, timestamp: u64, txs: Vec<&str>) -> Value { + json!({ + "hash": hash, + "parentHash": parent_hash, + "number": format!("0x{number:x}"), + "timestamp": format!("0x{timestamp:x}"), + "gasLimit": "0x2540be400", + "gasUsed": "0x0", + "baseFeePerGas": "0xf4240", + "blobGasUsed": "0x0", + "excessBlobGas": "0x0", + "difficulty": "0x0", + "extraData": "0x00000000fa00000001", + "logsBloom": format!("0x{}", "0".repeat(512)), + "miner": "0x4200000000000000000000000000000000000011", + "mixHash": "0x5cd8791a477b467456670744425e11d5bd91fd54575d6d3bf80d761ab39d957f", + "nonce": "0x0000000000000000", + "parentBeaconBlockRoot": + "0x67123956bf748ccfcfa68f03531dd12c1c647f9f31cc91935ce4271fa7399e24", + "receiptsRoot": "0x16fe124682128dd43a5da7f2cee0a3bf076deaf12682d19c656914bbea4615e3", + "requestsHash": "0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347", + "size": "0x43e7", + "stateRoot": "0xa342aba318978654abcf7f09f9494ed271e2136040b628edacb6d384e9074416", + "transactionsRoot": "0x2f3c5d0b0c4c8d34dd4e1c8bb4b4a4b6d6a2a3d3b8f6a9a2c1d0e9f8a7b6c5d4", + "withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421", + "uncles": [], + "withdrawals": [], + "transactions": txs, + }) +} + +/// The authentic identity of the replayed transaction: `(hash, from)`. +/// +/// Builds the same consensus object the replay will deserialize from +/// [`tx_json`], hashes its encoding, and recovers its signer — the two values +/// the replay authenticates the served answer against. +fn tx_identity(chain_id: u64, to: &str, input: &str) -> (String, String) { + use mega_evm::{ + alloy_consensus::{transaction::SignerRecoverable, SignableTransaction, TxEip1559}, + op_alloy_consensus::OpTxEnvelope, + }; + + let tx = TxEip1559 { + chain_id, + nonce: 0, + gas_limit: 0x249f0, + max_fee_per_gas: 0x200b20, + max_priority_fee_per_gas: 0x186a0, + to: alloy_primitives::TxKind::Call(to.parse().expect("`to` is an address")), + value: alloy_primitives::U256::ZERO, + access_list: Default::default(), + input: input.parse::().expect("calldata is hex"), + }; + let signature = alloy_primitives::Signature::new( + SIG_R.parse().expect("r is a hex word"), + SIG_S.parse().expect("s is a hex word"), + false, + ); + let signed = tx.into_signed(signature); + let hash = format!("{:#x}", signed.hash()); + let from = OpTxEnvelope::Eip1559(signed).recover_signer().expect("signature recovers"); + (hash, format!("{from:#x}")) +} + +/// The replayed transaction: an EIP-1559 call to `to` with `input` as calldata. +fn tx_json(chain_id: u64, to: &str, input: &str) -> Value { + let (hash, from) = tx_identity(chain_id, to, input); + json!({ + "type": "0x2", + "chainId": format!("0x{chain_id:x}"), + "nonce": "0x0", + "gas": "0x249f0", + "maxFeePerGas": "0x200b20", + "maxPriorityFeePerGas": "0x186a0", + "gasPrice": "0x10c8e0", + "to": to, + "value": "0x0", + "accessList": [], + "input": input, + "r": SIG_R, + "s": SIG_S, + "yParity": "0x0", + "v": "0x0", + "hash": hash, + "from": from, + "blockHash": BLOCK_HASH, + "blockNumber": format!("0x{BLOCK_NUMBER:x}"), + "transactionIndex": "0x0", + }) +} + +/// A mock chain and the hash of the one transaction it serves, which is what +/// the replay is pointed at. +struct MockChain { + server: MockRpcServer, + tx_hash: String, +} + +/// A mock endpoint serving a one-transaction mainnet block at `timestamp`, +/// whose transaction calls `to` with `input`. +/// +/// Account reads are answered blanket: every account holds 1 ETH, has nonce 0, +/// no code, and zero storage. That leaves the pre-block deploys as the only +/// source of code on the forked state, which is what makes "did this spec's +/// predeploys land" observable from the transaction's own return data. +async fn mock_chain(to: &str, input: &str, timestamp: u64) -> MockChain { + mock_chain_with_id(CHAIN_ID, to, input, timestamp).await +} + +/// [`mock_chain`], on the chain id of the caller's choosing. +async fn mock_chain_with_id(chain_id: u64, to: &str, input: &str, timestamp: u64) -> MockChain { + let (tx_hash, _) = tx_identity(chain_id, to, input); + let server = MockRpcServer::start().await; + server.respond_eth_chain_id(chain_id, 1).await; + server + .respond_method_params_json( + "eth_getBlockByNumber", + json!([format!("0x{BLOCK_NUMBER:x}"), false]), + block_json(BLOCK_NUMBER, BLOCK_HASH, PARENT_HASH, timestamp, vec![&tx_hash]), + 2, + ) + .await; + server + .respond_method_params_json( + "eth_getBlockByNumber", + json!([format!("0x{:x}", BLOCK_NUMBER - 1), false]), + block_json(BLOCK_NUMBER - 1, PARENT_HASH, GRANDPARENT_HASH, timestamp - 1, vec![]), + 2, + ) + .await; + server.respond_method_json("eth_getTransactionByHash", tx_json(chain_id, to, input), 3).await; + server.respond_method_result("eth_getBalance", "0xde0b6b3a7640000", 4).await; + server.respond_method_result("eth_getTransactionCount", "0x0", 4).await; + server.respond_method_result("eth_getCode", "0x", 4).await; + server + .respond_method_result( + "eth_getStorageAt", + "0x0000000000000000000000000000000000000000000000000000000000000000", + 4, + ) + .await; + MockChain { server, tx_hash } +} + +/// Replay the mock's transaction, optionally with extra flags. +fn replay(chain: &MockChain, args: &[&str]) -> Run { + let output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args(["replay", &chain.tx_hash, "--rpc", &chain.server.uri()]) + .args(["--rpc.no-cache-file", "--rpc.max-retries", "0", "--rpc.backoff-ms", "1", "--json"]) + .args(args) + .output() + .expect("failed to run mega-evme"); + Run { + code: output.status.code(), + stdout: String::from_utf8(output.stdout).expect("stdout is utf-8"), + stderr: String::from_utf8(output.stderr).expect("stderr is utf-8"), + } +} + +/// Without an override the block executes where it sits in the chain's +/// schedule: on a MiniRex-era block the Rex5 `SequencerRegistry` was never +/// deployed, so the call reaches an account with no code. +#[tokio::test(flavor = "multi_thread")] +async fn test_without_override_predeploys_follow_the_block_timestamp() { + let server = mock_chain(SEQUENCER_REGISTRY, VERSION_SELECTOR, MINI_REX_TIMESTAMP).await; + let run = replay(&server, &[]); + + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + assert_eq!(run.output(), None, "a MiniRex-era block must not carry the Rex5 SequencerRegistry",); +} + +/// Forcing Rex5 on a MiniRex-era block installs the Rex5 predeploys, including +/// the `SequencerRegistry` — whose pre-block deploy needs the chain's Rex5 +/// parameters. A synthesized schedule that dropped them would fail the run +/// before any transaction executed. +#[tokio::test(flavor = "multi_thread")] +async fn test_override_installs_the_forced_spec_predeploys() { + let server = mock_chain(SEQUENCER_REGISTRY, VERSION_SELECTOR, MINI_REX_TIMESTAMP).await; + let run = replay(&server, &["--override.spec", "Rex5"]); + + assert!( + !run.stderr.contains("SequencerRegistryConfig not configured"), + "the forced world must keep the chain's Rex5 registry parameters:\n{}", + run.stderr, + ); + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + assert_eq!( + run.output().as_deref(), + Some(VERSION_1_0_0), + "the forced spec's registry version must answer the call", + ); +} + +/// Forcing an older spec withholds predeploys the block did have: +/// `MegaLimitControl` arrives with Rex4, and neither its bytecode nor its +/// interception is present in a forced `MiniRex` world. +#[tokio::test(flavor = "multi_thread")] +async fn test_override_to_an_older_spec_withholds_later_predeploys() { + let selector = remaining_compute_gas_selector(); + let server = mock_chain(LIMIT_CONTROL, &selector, MINI_REX_TIMESTAMP).await; + let run = replay(&server, &["--override.spec", "MiniRex"]); + + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + assert_eq!(run.output(), None, "MegaLimitControl must not answer in a forced MiniRex world"); +} + +/// The block-level resource limits follow the override too. `MegaLimitControl` +/// reports the compute gas left in the current call, which is the forced spec's +/// per-transaction compute budget minus what the transaction has spent — the +/// `MiniRex` budget the block's own schedule carries is five times larger, so the +/// two are never confusable. +#[tokio::test(flavor = "multi_thread")] +async fn test_override_switches_the_block_limits() { + let selector = remaining_compute_gas_selector(); + let server = mock_chain(LIMIT_CONTROL, &selector, MINI_REX_TIMESTAMP).await; + let run = replay(&server, &["--override.spec", "Rex5"]); + + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + let output = run.output().expect("MegaLimitControl must answer under the forced Rex5 world"); + let remaining = decode_remaining_compute_gas(&output); + + let forced_budget = compute_gas_budget(mega_evm::MegaSpecId::REX5); + let historical_budget = compute_gas_budget(mega_evm::MegaSpecId::MINI_REX); + assert!( + historical_budget > forced_budget, + "the scenario needs the two budgets to differ to tell the worlds apart", + ); + assert!( + remaining <= forced_budget && remaining > forced_budget - 1_000_000, + "remaining compute gas {remaining} must come from the forced Rex5 budget \ + {forced_budget}, not the block's MiniRex budget {historical_budget}", + ); +} + +/// An override below the chain's own spec moves the predeploys back to that +/// spec's versions, not just their presence: on a chain running Rex6 the +/// registry is v2.0.0, and forcing Rex5 deploys v1.0.0 instead. +#[tokio::test(flavor = "multi_thread")] +async fn test_override_downgrade_switches_the_predeploy_version() { + let server = mock_chain_with_id( + UNKNOWN_CHAIN_ID, + SEQUENCER_REGISTRY, + VERSION_SELECTOR, + MINI_REX_TIMESTAMP, + ) + .await; + + let historical = replay(&server, &[]); + assert_eq!( + historical.output().as_deref(), + Some(VERSION_2_0_0), + "a chain with no published schedule runs the latest spec", + ); + + let forced = replay(&server, &["--override.spec", "Rex5"]); + assert_eq!(forced.code, Some(0), "stdout:\n{}\nstderr:\n{}", forced.stdout, forced.stderr); + assert_eq!( + forced.output().as_deref(), + Some(VERSION_1_0_0), + "the forced Rex5 world must deploy the Rex5 registry version", + ); +} + +/// A forced spec needs the chain's parameters for every fork it activates. Where +/// the chain's schedule carries them the what-if runs; where it does not, the +/// pre-block deploy fails closed with the missing-parameter error rather than +/// inventing a value the chain never published. Mainnet carries the Rex5 +/// parameters but not the Rex6 ones, so which arm applies is read from the +/// chain configuration instead of being assumed here. +#[tokio::test(flavor = "multi_thread")] +async fn test_override_needs_the_chain_params_of_every_fork_it_activates() { + use mega_evm::{MegaHardforks, SequencerRegistryRex6Config}; + + let configured = mega_evm::hardfork_schedule(CHAIN_ID) + .fork_params::() + .is_some(); + + let server = mock_chain(SEQUENCER_REGISTRY, VERSION_SELECTOR, MINI_REX_TIMESTAMP).await; + let run = replay(&server, &["--override.spec", "Rex6"]); + + if configured { + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + assert_eq!(run.output().as_deref(), Some(VERSION_2_0_0)); + } else { + assert_eq!( + run.code, + Some(1), + "a fork whose parameters the chain does not carry must not be forced silently:\n{}", + run.stdout, + ); + assert!( + run.stderr.contains("SequencerRegistryRex6Config not configured"), + "the refusal must name the missing parameters:\n{}", + run.stderr, + ); + } +} + +/// A spec's per-transaction compute gas budget, which the block limits carry +/// into execution. +fn compute_gas_budget(spec: mega_evm::MegaSpecId) -> u64 { + mega_evm::EvmTxRuntimeLimits::from_spec(spec).tx_compute_gas_limit +} + +/// Calldata for `IMegaLimitControl.remainingComputeGas()`. +fn remaining_compute_gas_selector() -> String { + use mega_evm::{alloy_sol_types::SolCall, IMegaLimitControl}; + + format!( + "0x{}", + alloy_primitives::hex::encode(IMegaLimitControl::remainingComputeGasCall {}.abi_encode()) + ) +} + +/// Decode the `uint64` `MegaLimitControl` returns. +fn decode_remaining_compute_gas(output: &str) -> u64 { + use mega_evm::{alloy_sol_types::SolCall, IMegaLimitControl}; + + let bytes = alloy_primitives::hex::decode(output).expect("output is hex"); + IMegaLimitControl::remainingComputeGasCall::abi_decode_returns(&bytes) + .expect("output decodes as uint64") +} diff --git a/bin/mega-evme/tests/replay_pending.rs b/bin/mega-evme/tests/replay_pending.rs new file mode 100644 index 00000000..7474a857 --- /dev/null +++ b/bin/mega-evme/tests/replay_pending.rs @@ -0,0 +1,401 @@ +//! Integration tests for the pending-transaction single-transaction replay path +//! and for the target-metadata classification that decides who enters it. +//! +//! A pending target has no parent/block pair: its state base *is* the latest +//! block, which is also the block it is replayed in. The two roles must +//! therefore be filled by one and the same block, and these tests pin that from +//! outside the process — an endpoint that changes its answer between two calls +//! at the same height must not be able to produce a mixed-view replay. +//! +//! Only a target reporting neither a block number nor an inclusion hash is +//! pending. The other `(block_number, block_hash)` shapes are classified from the +//! metadata alone, before any block is fetched, and these tests pin that too by +//! counting the requests the endpoint receives. +//! +//! They run against a mock JSON-RPC endpoint rather than a recorded capture. An +//! offline capture cannot represent this case at all: identical requests are +//! served from the same keyed entry, so one fetch and two fetches are +//! indistinguishable offline, and a capture recorded for a mined replay answers +//! its state reads at the parent height while a pending replay reads them at the +//! latest one. + +use std::process::Command; + +use serde_json::{json, Value}; + +mod common; +use common::MockRpcServer; + +/// `MegaETH` mainnet, whose published schedule the replayed block runs under. +const CHAIN_ID: u64 = 4326; + +/// Height the endpoint reports as `latest`, and the only block it serves. +const LATEST: u64 = 18_172_461; + +/// A mainnet timestamp inside the `MiniRex` window. +const TIMESTAMP: u64 = 1_764_000_000; + +/// Hash of the block the endpoint serves first for `LATEST`. +const LATEST_HASH: &str = "0x2801837c261826beb8047e46139dfc4eb93ab5b3196ce23f312d3c7658262a62"; + +/// Hash of the replacement block the endpoint serves for `LATEST` from the +/// second call on — the divergent view a second fetch would pick up. +const REPLACEMENT_HASH: &str = "0x3333333333333333333333333333333333333333333333333333333333333333"; + +/// Parent of `LATEST_HASH`, so its header is well formed. +const PARENT_HASH: &str = "0xd482d481e9d11dd116ef6c41bf95ca608f159206c8f07900b1b53936d196ccb3"; + +/// Parent of the replacement block: a different chain, as a reorg would leave it. +const REPLACEMENT_PARENT_HASH: &str = + "0x4444444444444444444444444444444444444444444444444444444444444444"; + +/// Signature of the pending transaction: a fixed, well-formed secp256k1 pair. +/// +/// The replay authenticates every served transaction — its hash is recomputed +/// from the encoding and its sender re-derived from the signature — so the mock +/// cannot serve invented `hash`/`from` constants; [`tx_identity`] computes the +/// authentic pair. The sender is whatever address this signature recovers to, +/// funded like every other account by the mock's blanket balance. +const SIG_R: &str = "0xa19f0f1f52e2951452711b4f4aa5d177442c9a56abeb609b803fe2412ed24946"; +const SIG_S: &str = "0x7af21777b2e7d91c745d0077ba2726ee1bb75ccf00039a6218d64fdced768491"; + +/// Recipient of the pending transaction: an account with no code, so the call +/// succeeds without depending on any contract the mock does not serve. +const RECIPIENT: &str = "0x681e908b8ab57c49c74d770f369754ccc3e1ae09"; + +/// The authentic identity of the pending transaction: `(hash, from)`. +/// +/// Builds the same consensus object the replay will deserialize from +/// [`tx_json`], hashes its encoding, and recovers its signer — the two values +/// the replay authenticates the served answer against. +fn tx_identity() -> (String, String) { + use mega_evm::{ + alloy_consensus::{transaction::SignerRecoverable, SignableTransaction, TxEip1559}, + op_alloy_consensus::OpTxEnvelope, + }; + + let tx = TxEip1559 { + chain_id: CHAIN_ID, + nonce: 0, + gas_limit: 0x249f0, + max_fee_per_gas: 0x200b20, + max_priority_fee_per_gas: 0x186a0, + to: alloy_primitives::TxKind::Call(RECIPIENT.parse().expect("`to` is an address")), + value: alloy_primitives::U256::ZERO, + access_list: Default::default(), + input: alloy_primitives::Bytes::new(), + }; + let signature = alloy_primitives::Signature::new( + SIG_R.parse().expect("r is a hex word"), + SIG_S.parse().expect("s is a hex word"), + false, + ); + let signed = tx.into_signed(signature); + let hash = format!("{:#x}", signed.hash()); + let from = OpTxEnvelope::Eip1559(signed).recover_signer().expect("signature recovers"); + (hash, format!("{from:#x}")) +} + +/// A block header the RPC backend and the replay accept, carrying only the +/// fields either of them reads. +fn block_json(hash: &str, parent_hash: &str) -> Value { + json!({ + "hash": hash, + "parentHash": parent_hash, + "number": format!("0x{LATEST:x}"), + "timestamp": format!("0x{TIMESTAMP:x}"), + "gasLimit": "0x2540be400", + "gasUsed": "0x0", + "baseFeePerGas": "0xf4240", + "blobGasUsed": "0x0", + "excessBlobGas": "0x0", + "difficulty": "0x0", + "extraData": "0x00000000fa00000001", + "logsBloom": format!("0x{}", "0".repeat(512)), + "miner": "0x4200000000000000000000000000000000000011", + "mixHash": "0x5cd8791a477b467456670744425e11d5bd91fd54575d6d3bf80d761ab39d957f", + "nonce": "0x0000000000000000", + "parentBeaconBlockRoot": + "0x67123956bf748ccfcfa68f03531dd12c1c647f9f31cc91935ce4271fa7399e24", + "receiptsRoot": "0x16fe124682128dd43a5da7f2cee0a3bf076deaf12682d19c656914bbea4615e3", + "requestsHash": "0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347", + "size": "0x43e7", + "stateRoot": "0xa342aba318978654abcf7f09f9494ed271e2136040b628edacb6d384e9074416", + "transactionsRoot": "0x2f3c5d0b0c4c8d34dd4e1c8bb4b4a4b6d6a2a3d3b8f6a9a2c1d0e9f8a7b6c5d4", + "withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421", + "uncles": [], + "withdrawals": [], + "transactions": [], + }) +} + +/// The replayed transaction, carrying the `(blockNumber, blockHash)` pair the +/// endpoint reports for it. Everything else is the same transaction, so a test +/// varies only the metadata the classification reads. +fn tx_json(block_number: Value, block_hash: Value) -> Value { + let (hash, from) = tx_identity(); + json!({ + "type": "0x2", + "chainId": format!("0x{CHAIN_ID:x}"), + "nonce": "0x0", + "gas": "0x249f0", + "maxFeePerGas": "0x200b20", + "maxPriorityFeePerGas": "0x186a0", + "gasPrice": "0x10c8e0", + "to": RECIPIENT, + "value": "0x0", + "accessList": [], + "input": "0x", + "r": SIG_R, + "s": SIG_S, + "yParity": "0x0", + "v": "0x0", + "hash": hash, + "from": from, + "blockHash": block_hash, + "blockNumber": block_number, + "transactionIndex": Value::Null, + }) +} + +/// The replayed transaction, reported as pending: no block number and no +/// inclusion hash. +fn pending_tx_json() -> Value { + tx_json(Value::Null, Value::Null) +} + +/// A mock endpoint holding one pending transaction, whose `latest` height is +/// answered with [`LATEST_HASH`] once and with [`REPLACEMENT_HASH`] from the +/// second call on. +async fn mock_chain() -> MockRpcServer { + mock_chain_serving(pending_tx_json()).await +} + +/// A mock endpoint that resolves the target to `tx`, and otherwise behaves like +/// [`mock_chain`]: the `latest` height is answered with [`LATEST_HASH`] once and +/// with [`REPLACEMENT_HASH`] from the second call on. +/// +/// Account reads are answered blanket: every account holds 1 ETH, has nonce 0, +/// no code, and zero storage. +async fn mock_chain_serving(tx: Value) -> MockRpcServer { + let server = MockRpcServer::start().await; + server.respond_eth_chain_id(CHAIN_ID, 1).await; + server.respond_method_result("eth_blockNumber", &format!("0x{LATEST:x}"), 2).await; + server + .respond_method_params_json_n_times( + "eth_getBlockByNumber", + json!([format!("0x{LATEST:x}"), false]), + block_json(LATEST_HASH, PARENT_HASH), + 1, + 2, + ) + .await; + server + .respond_method_json( + "eth_getBlockByNumber", + block_json(REPLACEMENT_HASH, REPLACEMENT_PARENT_HASH), + 3, + ) + .await; + server.respond_method_json("eth_getTransactionByHash", tx, 3).await; + server.respond_method_result("eth_getBalance", "0xde0b6b3a7640000", 4).await; + server.respond_method_result("eth_getTransactionCount", "0x0", 4).await; + server.respond_method_result("eth_getCode", "0x", 4).await; + server + .respond_method_result( + "eth_getStorageAt", + "0x0000000000000000000000000000000000000000000000000000000000000000", + 4, + ) + .await; + server +} + +/// Outcome of one `mega-evme replay` invocation. +struct Run { + code: Option, + stdout: String, + stderr: String, +} + +impl Run { + /// The single `--json` summary the run printed. + fn summary(&self) -> Value { + let mut values = common::json_values(&self.stdout); + if values.last().is_some_and(common::is_run_error) { + values.pop(); + } + assert_eq!( + values.len(), + 1, + "expected one summary on stdout:\n{}\nstderr:\n{}", + self.stdout, + self.stderr, + ); + values.pop().expect("checked above") + } + + /// The structured error object a failing `--json` run ends with. + fn error_object(&self) -> Value { + let values = common::json_values(&self.stdout); + let last = values.last().unwrap_or_else(|| { + panic!("a failing --json run must not leave stdout empty:\nstderr:\n{}", self.stderr) + }); + assert!( + common::is_run_error(last), + "the last stdout value must be the error object, got: {last}" + ); + last.clone() + } +} + +/// Replay the mock's pending transaction. +fn replay(server: &MockRpcServer) -> Run { + let (tx_hash, _) = tx_identity(); + let output = Command::new(env!("CARGO_BIN_EXE_mega-evme")) + .args(["replay", &tx_hash, "--rpc", &server.uri()]) + .args(["--rpc.no-cache-file", "--rpc.max-retries", "0", "--rpc.backoff-ms", "1", "--json"]) + .output() + .expect("failed to run mega-evme"); + Run { + code: output.status.code(), + stdout: String::from_utf8(output.stdout).expect("stdout is utf-8"), + stderr: String::from_utf8(output.stderr).expect("stderr is utf-8"), + } +} + +/// A pending replay fetches the latest block exactly once and fills both the +/// state-base and the replayed-block role from that one answer. +/// +/// The endpoint changes its answer for the same height after the first call, so +/// a second fetch would hand the run a replacement block: the pre-state would +/// come from one view and the block environment from the other, and the run +/// would still exit 0 while reporting a receipt anchored to a block it never +/// forked from. One fetch removes that possibility structurally rather than +/// detecting it afterwards. +#[tokio::test(flavor = "multi_thread")] +async fn test_pending_replay_fetches_the_latest_block_once() { + let server = mock_chain().await; + + let run = replay(&server); + + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + assert_eq!( + server.received_method_count("eth_getBlockByNumber").await, + 1, + "the two roles must be filled by a single fetch:\n{}", + run.stdout, + ); + let receipt = &run.summary()["receipt"]; + assert_eq!( + receipt["blockHash"].as_str(), + Some(LATEST_HASH), + "the replay must report the block it forked from, not the replacement: {receipt}", + ); + assert_eq!(receipt["blockNumber"].as_str(), Some(format!("0x{LATEST:x}").as_str())); +} + +/// A pending target still replays against a coherent endpoint: the reused block +/// fills both roles, so the transaction executes on top of the latest block and +/// reports its result there. +#[tokio::test(flavor = "multi_thread")] +async fn test_pending_replay_executes_against_the_latest_block() { + let server = mock_chain().await; + + let run = replay(&server); + + assert_eq!(run.code, Some(0), "stdout:\n{}\nstderr:\n{}", run.stdout, run.stderr); + let summary = run.summary(); + assert_eq!(summary["success"], json!(true), "the pending transaction must execute: {summary}"); + assert_eq!( + summary["receipt"]["transactionHash"].as_str(), + Some(tx_identity().0.as_str()), + "the receipt must describe the replayed transaction: {summary}", + ); +} + +/// An inclusion hash paired with a null block number is contradictory metadata, +/// not a pending transaction: the hash proves inclusion while the null number +/// denies it. The run answers that from the metadata alone — exit 3 without a +/// single block fetch — rather than reading the null number as "pending", +/// skipping every inclusion and body guard, and replaying the target against +/// latest with exit 0. +#[tokio::test(flavor = "multi_thread")] +async fn test_inclusion_hash_without_a_block_number_is_rejected_before_any_fetch() { + const INCLUSION: &str = "0x5555555555555555555555555555555555555555555555555555555555555555"; + + let server = mock_chain_serving(tx_json(Value::Null, json!(INCLUSION))).await; + + let run = replay(&server); + + assert_eq!( + run.code, + Some(3), + "contradictory metadata exits 3.\nstdout:\n{}\nstderr:\n{}", + run.stdout, + run.stderr, + ); + assert_eq!( + server.received_method_count("eth_getBlockByNumber").await, + 0, + "the verdict must precede every block fetch:\n{}", + run.stdout, + ); + assert_eq!( + server.received_method_count("eth_blockNumber").await, + 0, + "the verdict must precede the latest-height lookup too:\n{}", + run.stdout, + ); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains(INCLUSION) && message.contains("contradictory metadata"), + "the message must name the hash and the contradiction: {error}" + ); + assert!( + !run.stdout.contains("\"success\""), + "the run must not produce an execution summary:\n{}", + run.stdout, + ); +} + +/// A block number paired with a null inclusion hash is an unanchored view: the +/// number alone cannot prove which block body the target belongs to. The run +/// answers that from the metadata alone — exit 3 without a single block fetch — +/// so neither a missing block nor a broken parent linkage can mask it. +#[tokio::test(flavor = "multi_thread")] +async fn test_mined_target_without_an_inclusion_hash_is_rejected_before_any_fetch() { + let server = mock_chain_serving(tx_json(json!(format!("0x{LATEST:x}")), Value::Null)).await; + + let run = replay(&server); + + assert_eq!( + run.code, + Some(3), + "an unanchored view exits 3.\nstdout:\n{}\nstderr:\n{}", + run.stdout, + run.stderr, + ); + assert_eq!( + server.received_method_count("eth_getBlockByNumber").await, + 0, + "the verdict must precede every block fetch:\n{}", + run.stdout, + ); + let error = run.error_object(); + assert_eq!(error["error"]["code"].as_u64(), Some(3)); + assert_eq!(error["error"]["kind"].as_str(), Some("rpc-failure")); + let message = error["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("inclusion hash") && message.contains("unanchored"), + "the message must name the unanchored view: {error}" + ); + assert!( + message.contains(&LATEST.to_string()), + "the message must name the block number the lookup reported: {error}" + ); +} diff --git a/bin/mega-evme/tests/replay_verify.rs b/bin/mega-evme/tests/replay_verify.rs new file mode 100644 index 00000000..9c912e09 --- /dev/null +++ b/bin/mega-evme/tests/replay_verify.rs @@ -0,0 +1,1183 @@ +//! Integration tests for `mega-evme replay --verify-receipt`: the end-to-end +//! comparison against the on-chain receipt in single-transaction and batch mode. +//! +//! They run fully offline against the committed RPC capture that carries the +//! on-chain receipt (`fixtures/replay_offline.cache.json`), so they are +//! deterministic. Mismatch and infrastructure cases are produced by doctoring a +//! copy of that capture: its entries are keyed by the request, not the response, +//! so a doctored response still resolves. + +use std::{ + path::{Path, PathBuf}, + process::Command, +}; + +mod common; + +/// Offline RPC capture, including the transaction's on-chain receipt. +/// Name of the committed offline capture, resolved through the shared fixture +/// helper so its location lives in exactly one place. +const CACHE: &str = "replay_offline.cache.json"; + +/// The transaction captured in `CACHE` (a 75,514-gas Rex5 mainnet call). +const TX: &str = "0x41d34e7e13dfe0f85da9d407e2b2c381955d8c7eed428b17dc82327b2616b000"; + +/// Gas the transaction used on-chain, which a faithful replay reproduces. +const GAS_USED: u64 = 75_514; + +/// A transaction hash that is not the replayed target, used to model an endpoint +/// answering a receipt request with another transaction's receipt. +const OTHER_TX: &str = "0x00000000000000000000000000000000000000000000000000000000feed0001"; + +/// Outcome of one `mega-evme replay` invocation. +struct Run { + success: bool, + code: Option, + stdout: String, + stderr: String, +} + +impl Run { + /// The process exit code the run ended with. + fn code(&self) -> i32 { + self.code.expect("mega-evme was killed by a signal") + } + + /// The results printed on stdout, without the structured error object a + /// failing `--json` run ends with. + fn results(&self) -> Vec { + let mut values = common::json_values(&self.stdout); + if values.last().is_some_and(common::is_run_error) { + values.pop(); + } + values + } + + /// Parse the stdout of a `--json` single-transaction run. + fn json(&self) -> serde_json::Value { + let mut results = self.results(); + assert_eq!(results.len(), 1, "expected one summary on stdout:\n{}", self.stdout); + results.pop().expect("checked above") + } + + /// Parse the stdout of a `--json` batch run as one value per NDJSON line. + fn ndjson(&self) -> Vec { + self.results() + } + + /// The structured error object a failing `--json` run ends with. + fn error_object(&self) -> serde_json::Value { + let values = common::json_values(&self.stdout); + let last = values + .last() + .unwrap_or_else(|| panic!("a failing --json run must not leave stdout empty")); + assert!( + common::is_run_error(last), + "the last stdout value must be the error object, got: {last}" + ); + last.clone() + } +} + +/// Run `replay` offline against `cache`. +fn replay(cache: &Path, args: &[&str]) -> Run { + replay_with_env(cache, args, &[]) +} + +/// Run `replay` offline against `cache` with additional process environment. +fn replay_with_env(cache: &Path, args: &[&str], envs: &[(&str, &str)]) -> Run { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_mega-evme")); + cmd.args(["replay", "--rpc.replay-file", cache.to_str().expect("cache path is utf-8")]) + .args(args); + for (key, value) in envs { + cmd.env(key, value); + } + let output = cmd.output().expect("failed to run mega-evme"); + Run { + success: output.status.success(), + code: output.status.code(), + stdout: String::from_utf8(output.stdout).expect("stdout is utf-8"), + stderr: String::from_utf8(output.stderr).expect("stderr is utf-8"), + } +} + +/// The committed capture, unmodified. +fn cache() -> PathBuf { + common::fixture(CACHE) +} + +/// A temp path unique to this process and this test. +fn temp_path(name: &str) -> PathBuf { + std::env::temp_dir().join(format!("mega_evme_verify_{name}_{}.json", std::process::id())) +} + +/// Write a copy of the committed capture whose receipt response is rewritten by +/// `doctor`, and return its path. +fn doctored_cache(name: &str, doctor: impl Fn(&mut serde_json::Value)) -> PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let mut doctored = false; + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + // The receipt is the only cached response carrying cumulativeGasUsed. + if !value.contains("cumulativeGasUsed") { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse receipt response"); + doctor(&mut response["result"]); + entry["value"] = serde_json::Value::String(response.to_string()); + doctored = true; + } + assert!(doctored, "offline cache should contain the receipt entry"); + + let path = temp_path(name); + std::fs::write(&path, envelope.to_string()).expect("write doctored cache"); + path +} + +/// Write a copy of the committed capture with the receipt dropped entirely, +/// modelling an endpoint that has pruned it. +fn cache_without_receipt(name: &str) -> PathBuf { + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + let entries = envelope["cache"].as_array_mut().expect("cache entries"); + let before = entries.len(); + entries.retain(|entry| { + !entry["value"].as_str().expect("entry value is a string").contains("cumulativeGasUsed") + }); + assert!(entries.len() < before, "offline cache should contain the receipt entry"); + + let path = temp_path(name); + std::fs::write(&path, envelope.to_string()).expect("write pruned cache"); + path +} + +/// Write a `--tx-file` holding the single captured transaction. +fn tx_file(name: &str) -> PathBuf { + let path = + std::env::temp_dir().join(format!("mega_evme_verify_{name}_{}.txt", std::process::id())); + std::fs::write(&path, format!("{TX}\n")).expect("write tx list"); + path +} + +/// A faithful replay reproduces the on-chain receipt, reports a match, and exits 0. +#[test] +fn test_verify_receipt_reports_a_match() { + let run = replay(&cache(), &["--verify-receipt", "--json", TX]); + + assert!(run.success, "a matching verification must exit 0.\nstderr: {}", run.stderr); + assert_eq!(run.json()["verification"], serde_json::json!({ "match": true })); +} + +/// Human-readable output carries one verdict line per transaction. +#[test] +fn test_verify_receipt_prints_a_human_verdict_line() { + let run = replay(&cache(), &["--verify-receipt", TX]); + + assert!(run.success, "a matching verification must exit 0.\nstderr: {}", run.stderr); + assert!( + run.stdout.contains("verification: MATCH"), + "expected a verdict line, got stdout:\n{}", + run.stdout + ); +} + +/// Without the flag the single-transaction JSON is unchanged: no `verification` +/// key, and the flag adds that key and nothing else. +#[test] +fn test_single_transaction_json_is_unchanged_without_the_flag() { + let plain = replay(&cache(), &["--json", TX]); + let verified = replay(&cache(), &["--verify-receipt", "--json", TX]); + + assert!(plain.success && verified.success, "both runs must exit 0"); + assert!( + !plain.stdout.contains("verification"), + "output without the flag must not mention verification:\n{}", + plain.stdout + ); + assert!(plain.json().get("verification").is_none(), "the key must be absent without the flag"); + + let mut stripped = verified.json(); + stripped.as_object_mut().expect("summary is an object").remove("verification"); + assert_eq!(stripped, plain.json(), "--verify-receipt must add the verdict and nothing else"); +} + +/// A gas divergence is reported as a `gas_used` diff and fails the run with the +/// dedicated mismatch exit code. +#[test] +fn test_verify_receipt_reports_a_gas_mismatch() { + let path = doctored_cache("gas", |receipt| receipt["gasUsed"] = "0x1".into()); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 2, "a mismatch exits 2.\nstderr: {}", run.stderr); + assert_eq!( + run.json()["verification"], + serde_json::json!({ + "match": false, + "diff": { "gas_used": { "onchain": 1, "replay": GAS_USED } }, + }) + ); + assert_eq!(run.error_object()["error"]["code"].as_u64(), Some(2)); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("verification-mismatch")); + assert!( + run.stderr.contains("Receipt verification mismatch"), + "expected the mismatch error, got stderr:\n{}", + run.stderr + ); +} + +/// A status divergence is reported on its own, without dragging in the +/// dimensions that agreed. +#[test] +fn test_verify_receipt_reports_a_status_mismatch() { + let path = doctored_cache("status", |receipt| receipt["status"] = "0x0".into()); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 2, "a mismatch exits 2.\nstderr: {}", run.stderr); + assert_eq!( + run.json()["verification"], + serde_json::json!({ + "match": false, + "diff": { "status": { "onchain": false, "replay": true } }, + }) + ); +} + +/// A log divergence is reported under `logs`. +#[test] +fn test_verify_receipt_reports_a_log_mismatch() { + let path = doctored_cache("logs", |receipt| { + receipt["logs"] = serde_json::json!([{ + "address": "0x00000000000000000000000000000000000000aa", + "topics": ["0x000000000000000000000000000000000000000000000000000000000000000a"], + "data": "0xdeadbeef", + "blockHash": receipt["blockHash"], + "blockNumber": receipt["blockNumber"], + "transactionHash": receipt["transactionHash"], + "transactionIndex": receipt["transactionIndex"], + "logIndex": "0x0", + "removed": false, + }]); + }); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 2, "a mismatch exits 2.\nstderr: {}", run.stderr); + assert_eq!( + run.json()["verification"], + serde_json::json!({ + "match": false, + "diff": { "logs": { "count": { "onchain": 1, "replay": 0 } } }, + }) + ); +} + +/// A receipt describing a different inclusion than the replayed block is an +/// infrastructure failure: the transaction is unverified, never mismatched. +#[test] +fn test_verify_receipt_reorg_is_an_infrastructure_error() { + let path = doctored_cache("reorg", |receipt| { + receipt["blockHash"] = + "0x1111111111111111111111111111111111111111111111111111111111111111".into(); + }); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + // The comparison never ran, so the run fails as an RPC-class failure, not + // as a mismatch. + assert_eq!(run.code(), 3, "an unverifiable target exits 3.\nstderr: {}", run.stderr); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + run.stderr.contains("different inclusion"), + "expected the reorg/divergent-endpoint hint, got stderr:\n{}", + run.stderr + ); + assert!( + !run.stderr.contains("verification mismatch") && !run.stdout.contains("MISMATCH"), + "an unverifiable transaction must not be reported as a mismatch:\n{}\n{}", + run.stdout, + run.stderr, + ); +} + +/// A receipt describing a different transaction than the one requested is an +/// infrastructure failure: comparing against it would report a verdict about the +/// wrong transaction, so the target is unverified and the message names both the +/// requested and the served hash. +#[test] +fn test_verify_receipt_for_another_transaction_is_an_infrastructure_error() { + let path = doctored_cache("wrong_tx", |receipt| { + receipt["transactionHash"] = OTHER_TX.into(); + }); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "a receipt for another transaction exits 3.\nstderr: {}", run.stderr); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); + let message = run.error_object()["error"]["message"] + .as_str() + .expect("the error object carries a message") + .to_string(); + assert!( + message.contains(OTHER_TX) && message.contains(TX), + "the message must name both the served and the requested transaction: {message}" + ); + assert!( + !run.stderr.contains("verification mismatch") && !run.stdout.contains("MISMATCH"), + "a receipt for another transaction must not be reported as a mismatch:\n{}\n{}", + run.stdout, + run.stderr, + ); +} + +/// A receipt the endpoint cannot serve (e.g. pruned below its retention height) +/// is an infrastructure failure, not a mismatch. +#[test] +fn test_verify_receipt_missing_receipt_is_an_infrastructure_error() { + let path = cache_without_receipt("pruned"); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "an unavailable receipt exits 3.\nstderr: {}", run.stderr); + assert!( + run.stderr.contains("receipt"), + "expected an error naming the receipt, got stderr:\n{}", + run.stderr + ); + assert!( + !run.stderr.contains("verification mismatch") && !run.stdout.contains("MISMATCH"), + "an unverifiable transaction must not be reported as a mismatch:\n{}\n{}", + run.stdout, + run.stderr, + ); +} + +/// Batch mode carries the verdict on the transaction's NDJSON line. +#[test] +fn test_batch_verify_receipt_reports_a_match() { + let list = tx_file("batch_match"); + + let run = + replay(&cache(), &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&list); + + assert!(run.success, "a matching verification must exit 0.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + assert_eq!(lines[0]["tx_hash"].as_str(), Some(TX)); + assert_eq!(lines[0]["verification"], serde_json::json!({ "match": true })); +} + +/// A batch mismatch keeps the result line (with its diff) and fails the run +/// through the dedicated verification error. +#[test] +fn test_batch_verify_receipt_reports_a_mismatch_and_exits_nonzero() { + let path = doctored_cache("batch_gas", |receipt| receipt["gasUsed"] = "0x1".into()); + let list = tx_file("batch_gas"); + + let run = replay(&path, &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 2, "a mismatch exits 2.\nstderr: {}", run.stderr); + assert_eq!(run.error_object()["error"]["code"].as_u64(), Some(2)); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "a mismatch is still a result line, not an error entry"); + assert!(lines[0].get("error").is_none(), "a mismatch is not an infrastructure error"); + assert_eq!( + lines[0]["verification"], + serde_json::json!({ + "match": false, + "diff": { "gas_used": { "onchain": 1, "replay": GAS_USED } }, + }) + ); + assert!( + run.stderr.contains("Receipt verification mismatch"), + "expected the mismatch error, got stderr:\n{}", + run.stderr + ); +} + +/// In batch mode an unavailable receipt keeps the replayed result line and +/// reports the failure on `verification.error` — never as a mismatch, and never +/// by discarding the execution summary. +#[test] +fn test_batch_verify_receipt_missing_receipt_keeps_result_and_is_rpc() { + let path = cache_without_receipt("batch_pruned"); + let list = tx_file("batch_pruned"); + + let run = replay(&path, &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 3, "an unverified target exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + assert!( + lines[0].get("error").is_none(), + "a replayed target keeps its result line, not a bare error entry: {}", + lines[0] + ); + assert!(lines[0]["receipt"].is_object(), "local receipt is kept: {}", lines[0]); + assert_eq!(lines[0]["success"].as_bool(), Some(true)); + assert_eq!(lines[0]["gas_used"].as_u64(), Some(GAS_USED)); + assert!( + lines[0]["verification"]["error"].is_string(), + "verification carries the unanswered receipt: {}", + lines[0] + ); + assert!( + lines[0]["verification"].get("match").is_none(), + "unavailable is not a match/mismatch verdict: {}", + lines[0] + ); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + !run.stderr.contains("verification mismatch"), + "an unverifiable target must not fail as a mismatch:\n{}", + run.stderr + ); +} + +/// The reorg guard applies in batch mode too: the result is kept and the +/// divergent-inclusion failure is reported on `verification.error`. +#[test] +fn test_batch_verify_receipt_reorg_keeps_result_and_is_rpc() { + let path = doctored_cache("batch_reorg", |receipt| { + receipt["blockHash"] = + "0x1111111111111111111111111111111111111111111111111111111111111111".into(); + }); + let list = tx_file("batch_reorg"); + + let run = replay(&path, &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 3, "an unverified target exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert!(lines[0].get("error").is_none(), "result line is kept: {}", lines[0]); + assert!( + lines[0]["verification"]["error"] + .as_str() + .is_some_and(|message| message.contains("different inclusion")), + "expected the reorg/divergent-endpoint hint: {}", + lines[0] + ); +} + +/// The identity guard applies in batch mode too: the target replayed, so it +/// keeps its result line, and the failure to answer its receipt question is +/// reported on `verification.error` naming both hashes — tallied rpc once. +#[test] +fn test_batch_verify_receipt_for_another_transaction_keeps_result_and_is_rpc() { + let path = doctored_cache("batch_wrong_tx", |receipt| { + receipt["transactionHash"] = OTHER_TX.into(); + }); + let list = tx_file("batch_wrong_tx"); + + let run = replay(&path, &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 3, "an unverified target exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + assert!( + lines[0].get("error").is_none(), + "a replayed target keeps its result line, not a bare error entry: {}", + lines[0] + ); + assert!(lines[0]["receipt"].is_object(), "local receipt is kept: {}", lines[0]); + assert_eq!(lines[0]["success"].as_bool(), Some(true)); + assert_eq!(lines[0]["gas_used"].as_u64(), Some(GAS_USED)); + assert!( + lines[0]["verification"]["error"] + .as_str() + .is_some_and(|message| message.contains(OTHER_TX) && message.contains(TX)), + "expected both the served and the requested transaction: {}", + lines[0] + ); + assert!( + lines[0]["verification"].get("match").is_none(), + "a receipt served for another transaction is not a match/mismatch verdict: {}", + lines[0] + ); + let err = run.error_object(); + assert_eq!(err["error"]["kind"].as_str(), Some("rpc-failure")); + let message = err["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("1 of 1 target transaction(s) failed"), + "the one unverified target is the one failure: {message}" + ); + assert!( + message.contains("(0 execution, 1 rpc)"), + "an unanswered receipt question is tallied rpc, not execution: {message}" + ); + assert!( + !run.stderr.contains("verification mismatch"), + "an unverifiable target must not fail as a mismatch:\n{}", + run.stderr + ); +} + +/// A receipt with a null `blockHash` cannot be anchored to the replayed block: +/// infrastructure failure, never a match/mismatch verdict. +#[test] +fn test_verify_receipt_null_block_hash_is_an_infrastructure_error() { + let path = doctored_cache("null_block_hash", |receipt| { + receipt["blockHash"] = serde_json::Value::Null; + }); + + let run = replay(&path, &["--verify-receipt", "--json", TX]); + let _ = std::fs::remove_file(&path); + + assert_eq!(run.code(), 3, "an unverifiable target exits 3.\nstderr: {}", run.stderr); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); + assert!( + run.stderr.contains("no block hash") || run.stderr.contains("block hash"), + "expected a missing-inclusion-hash hint, got stderr:\n{}", + run.stderr + ); + assert!( + !run.stderr.contains("verification mismatch") && !run.stdout.contains("MISMATCH"), + "an unanchorable receipt must not be reported as a mismatch:\n{}\n{}", + run.stdout, + run.stderr, + ); +} + +/// Batch mode reports a null `blockHash` on the kept result line as +/// `verification.error` (rpc), not as a bare error entry. +#[test] +fn test_batch_verify_receipt_null_block_hash_keeps_result_and_is_rpc() { + let path = doctored_cache("batch_null_block_hash", |receipt| { + receipt["blockHash"] = serde_json::Value::Null; + }); + let list = tx_file("batch_null_block_hash"); + + let run = replay(&path, &["--tx-file", list.to_str().unwrap(), "--verify-receipt", "--json"]); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + + assert_eq!(run.code(), 3, "an unverified target exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert!(lines[0].get("error").is_none(), "result line is kept: {}", lines[0]); + assert!( + lines[0]["verification"]["error"] + .as_str() + .is_some_and(|message| message.contains("no block hash")), + "expected a missing-inclusion-hash hint: {}", + lines[0] + ); + assert!( + lines[0]["verification"].get("match").is_none(), + "unavailable is not a match/mismatch verdict: {}", + lines[0] + ); +} + +/// Dump-dir with a nulled receipt response fails the fidelity gate as rpc on +/// the kept result line (not a silent `fidelity-gate-unavailable` skip). +#[test] +fn test_batch_dump_fixture_dir_null_receipt_is_rpc_fixture_error() { + let path = doctored_cache("batch_dump_null_receipt", |receipt| { + // Doctor the whole result to null by replacing the entry value below. + let _ = receipt; + }); + // Null the receipt response entirely (result: null), modelling a pruned + // or unanswered eth_getTransactionReceipt. + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&path).expect("read doctored cache")) + .expect("parse"); + for entry in envelope["cache"].as_array_mut().expect("cache entries").iter_mut() { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains("cumulativeGasUsed") { + continue; + } + let mut response: serde_json::Value = + serde_json::from_str(value).expect("parse receipt response"); + response["result"] = serde_json::Value::Null; + entry["value"] = serde_json::Value::String(response.to_string()); + } + std::fs::write(&path, envelope.to_string()).expect("rewrite null-receipt cache"); + + let list = tx_file("batch_dump_null_receipt"); + let dir = std::env::temp_dir() + .join(format!("mega_evme_batch_dump_null_receipt_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay( + &path, + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert_eq!(run.code(), 3, "unanswered receipt for dump exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1); + assert!(lines[0].get("error").is_none(), "result line is kept: {}", lines[0]); + assert!(lines[0]["receipt"].is_object(), "local receipt is kept: {}", lines[0]); + assert_eq!(lines[0]["success"].as_bool(), Some(true)); + assert!( + lines[0]["fixture"]["error"].is_string(), + "fixture reports the unanswered receipt: {}", + lines[0] + ); + assert!( + lines[0]["fixture"].get("skipped").is_none(), + "receipt fetch failure is not a fidelity-gate skip: {}", + lines[0] + ); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// Dump-dir against an envelope that never captured the receipt is the same +/// unanswered class: rpc fixture error, result kept, exit 3. +#[test] +fn test_batch_dump_fixture_dir_missing_receipt_is_rpc_fixture_error() { + let path = cache_without_receipt("batch_dump_no_receipt"); + let list = tx_file("batch_dump_no_receipt"); + let dir = std::env::temp_dir() + .join(format!("mega_evme_batch_dump_no_receipt_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay( + &path, + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert_eq!(run.code(), 3, "missing receipt for dump exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1); + assert!(lines[0].get("error").is_none(), "result line is kept: {}", lines[0]); + assert!(lines[0]["receipt"].is_object(), "local receipt is kept: {}", lines[0]); + assert!( + lines[0]["fixture"]["error"].as_str().is_some_and(|m| m.contains("receipt") || + m.contains("not found") || + m.contains("cache")), + "fixture.error names the unanswered receipt: {}", + lines[0] + ); + assert!( + lines[0]["fixture"].get("skipped").is_none(), + "missing receipt is not a silent skip: {}", + lines[0] + ); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// Combined `--verify-receipt --dump-fixture-dir` with a missing receipt: both +/// result fields report the failure, the shared receipt failure is counted once +/// ("1 of 1"), and the run exits 3. +#[test] +fn test_batch_verify_and_dump_missing_receipt_counted_once() { + let path = cache_without_receipt("batch_verify_dump_no_receipt"); + let list = tx_file("batch_verify_dump_no_receipt"); + let dir = std::env::temp_dir() + .join(format!("mega_evme_batch_verify_dump_no_receipt_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay( + &path, + &[ + "--tx-file", + list.to_str().unwrap(), + "--verify-receipt", + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert_eq!(run.code(), 3, "shared missing receipt exits 3.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one target: {}", run.stdout); + assert!(lines[0].get("error").is_none(), "result line is kept: {}", lines[0]); + assert!(lines[0]["receipt"].is_object(), "local receipt is kept: {}", lines[0]); + assert!( + lines[0]["verification"]["error"].is_string(), + "verification.error carries the unanswered receipt: {}", + lines[0] + ); + assert!( + lines[0]["fixture"]["error"].is_string(), + "fixture.error also carries the unanswered receipt: {}", + lines[0] + ); + let err = run.error_object(); + let message = err["error"]["message"].as_str().unwrap_or_default(); + assert!( + message.contains("1 of 1 target transaction(s) failed"), + "shared receipt failure must not double-count: {message}" + ); + assert!( + message.contains("1 rpc") || message.contains("(0 execution, 1 rpc)"), + "exactly one rpc failure in the aggregate: {message}" + ); + assert_eq!(err["error"]["kind"].as_str(), Some("rpc-failure")); +} + +/// A genuine fidelity-gate skip (local gas disagrees with on-chain receipt) +/// still exits 0: the receipt question was answered, the dump was correctly +/// refused, and skips never fail the run. +#[test] +fn test_batch_dump_fixture_dir_fidelity_mismatch_stays_skip() { + let path = doctored_cache("batch_dump_fidelity_skip", |receipt| { + receipt["gasUsed"] = "0x1".into(); + }); + let list = tx_file("batch_dump_fidelity_skip"); + let dir = std::env::temp_dir() + .join(format!("mega_evme_batch_dump_fidelity_skip_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay( + &path, + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert_eq!(run.code(), 0, "a fidelity skip exits 0.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1); + assert!( + lines[0]["fixture"]["skipped"].as_str().is_some_and(|m| m.contains("fidelity gate failed")), + "expected fidelity-gate skip: {}", + lines[0] + ); + assert!(lines[0]["fixture"].get("error").is_none()); +} + +/// Batch `--dump-fixture-dir` writes a self-validating fixture for a target +/// whose capture includes the on-chain receipt, and exits 0. +#[test] +fn test_batch_dump_fixture_dir_writes_validatable_file() { + let list = tx_file("batch_dump"); + let dir = std::env::temp_dir().join(format!("mega_evme_batch_dump_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&list); + + assert!(run.success, "a successful dump must exit 0.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + assert_eq!(lines[0]["tx_hash"].as_str(), Some(TX)); + let path = lines[0]["fixture"]["path"] + .as_str() + .unwrap_or_else(|| panic!("expected a written fixture path: {}", lines[0])); + assert!( + path.ends_with(&format!("{TX}.json")), + "fixture path should be

/.json, got: {path}" + ); + assert!(std::path::Path::new(path).exists(), "fixture file must exist at {path}"); + + let elapsed = std::sync::Arc::new(std::sync::Mutex::new(std::time::Duration::ZERO)); + let result = state_test::runner::execute_test_suite(Path::new(path), &elapsed, false, false); + let _ = std::fs::remove_dir_all(&dir); + result.unwrap_or_else(|e| panic!("dumped fixture failed to validate: {e}")); +} + +/// A target whose fixture could not be written is still verified against its +/// on-chain receipt: the failed dump is reported on the target's result line +/// alongside the verdict, and fails the run as an execution-class failure. +#[test] +fn test_batch_fixture_write_failure_keeps_the_receipt_verification() { + let list = tx_file("batch_dump_verify"); + let dir = + std::env::temp_dir().join(format!("mega_evme_batch_dump_verify_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + // Pre-create the fixture path so the dump is refused without --overwrite. + std::fs::create_dir_all(&dir).expect("create dump dir"); + std::fs::write(dir.join(format!("{TX}.json")), "{}").expect("pre-create fixture file"); + + let run = replay( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--verify-receipt", + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert_eq!(run.code(), 1, "a failed dump exits 1.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + assert!(lines[0].get("error").is_none(), "a failed dump is not an error entry: {}", lines[0]); + assert_eq!( + lines[0]["verification"], + serde_json::json!({ "match": true }), + "the verification still ran: {}", + lines[0] + ); + assert!( + lines[0]["fixture"]["error"].as_str().is_some_and(|m| m.contains("already exists")), + "the failed dump is reported on the result line: {}", + lines[0] + ); + assert_eq!(run.error_object()["error"]["kind"].as_str(), Some("execution-error")); +} + +/// Without `--overwrite`, a second dump into a directory that already holds the +/// fixture fails that target as an infrastructure error. +#[test] +fn test_batch_dump_fixture_dir_refuses_overwrite_without_flag() { + let list = tx_file("batch_dump_ow"); + let dir = std::env::temp_dir().join(format!("mega_evme_batch_dump_ow_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let first = replay( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + assert!(first.success, "first dump must succeed.\nstderr: {}", first.stderr); + + let second = replay( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + ); + // The dump failed for the target, which is an execution-class failure. The + // target still replayed, so it keeps its result line and reports the failed + // dump on it. + assert_eq!(second.code(), 1, "a failed dump exits 1.\nstderr: {}", second.stderr); + let lines = second.ndjson(); + assert_eq!(lines.len(), 1); + assert!(lines[0].get("error").is_none(), "a failed dump is not an error entry: {}", lines[0]); + assert!(lines[0]["receipt"].is_object(), "the replayed target keeps its receipt: {}", lines[0]); + assert!( + lines[0]["fixture"]["error"] + .as_str() + .is_some_and(|m| m.contains("already exists") && m.contains("--overwrite")), + "expected overwrite refusal: {}", + lines[0] + ); + assert_eq!(second.error_object()["error"]["kind"].as_str(), Some("execution-error")); + + // With --overwrite the second dump succeeds and replaces the file. + let third = replay( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--overwrite", + "--json", + ], + ); + let _ = std::fs::remove_file(&list); + assert!(third.success, "dump with --overwrite must succeed.\nstderr: {}", third.stderr); + let lines = third.ndjson(); + assert!(lines[0]["fixture"]["path"].is_string(), "overwrite must report a written path"); + let _ = std::fs::remove_dir_all(&dir); +} + +/// A fixture-construction failure (database / pre-state reads) is reported as +/// `fixture.error` and fails the run — never as a silent skip that exits 0. +/// +/// Classification of construction vs unsupported-shape errors is unit-tested in +/// `batch::tests::test_fixture_build_err_classifies_skips_vs_construction_errors`. +/// +/// The offline State cache reuses account basics already loaded during +/// execution, so doctoring bytecode responses only kills execution. This test +/// injects a draft-time pre-state failure after execution succeeds +/// (`MEGA_EVME_INJECT_FIXTURE_PRE_STATE_ERROR`), proving the construction path +/// itself — a result line with `fixture.error` containing `construction failed`, +/// exit 1, and no skip. +#[test] +fn test_batch_fixture_construction_failure_is_fixture_error_not_skip() { + let list = tx_file("fixture_construction"); + let dir = + std::env::temp_dir().join(format!("mega_evme_fixture_construction_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + + let run = replay_with_env( + &cache(), + &[ + "--tx-file", + list.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--json", + ], + &[("MEGA_EVME_INJECT_FIXTURE_PRE_STATE_ERROR", "1")], + ); + let _ = std::fs::remove_file(&list); + let _ = std::fs::remove_dir_all(&dir); + + assert!(!run.success, "construction failure must not exit 0.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert_eq!(lines.len(), 1, "one line per requested transaction"); + // Sentinel: execution reached the dump target (receipt present); only the + // draft failed. The old code path accepted an execution-only failure here. + assert!( + lines[0].get("error").is_none(), + "execution must succeed so the failure is fixture construction, not an error entry: {}", + lines[0] + ); + assert!( + lines[0]["success"].as_bool() == Some(true), + "target must have executed successfully: {}", + lines[0] + ); + let fixture_error = lines[0]["fixture"]["error"] + .as_str() + .expect("fixture.error must be set for a construction failure"); + assert!( + fixture_error.contains("construction failed"), + "expected construction failed, got: {fixture_error}" + ); + assert!( + fixture_error.contains("pre-state") || fixture_error.contains("injected"), + "expected pre-state/injected failure detail, got: {fixture_error}" + ); + assert!( + lines[0]["fixture"].get("skipped").is_none(), + "construction failure must not be reported as a skip: {}", + lines[0] + ); + assert_eq!(run.code(), 1, "fixture construction failure exits 1"); +} + +/// When the block aborts after a dump target built a Ready draft, no fixture +/// file is written and `--overwrite` does not clobber a pre-existing file. +/// +/// Seeds a zero-gas object for the index-2 hash so resolve succeeds but the +/// block loop aborts on that transaction *after* the dump target (index 1) +/// has executed and built a deferred draft. Materialize runs only on a clean +/// loop, so the Ready draft is discarded. Happy-path writes are covered by +/// [`test_batch_dump_fixture_dir_writes_validatable_file`]. +#[test] +fn test_batch_dump_does_not_write_or_clobber_when_block_aborts_before_finish() { + // Index-2 hash of the captured block. Not present as a full TX object in + // the offline capture; we inject a zero-gas type-2 call so lookup succeeds + // and execution aborts after the dump target has drafted. + const LATER: &str = "0xfc0a0b9d76b13125ac1e36e524f6df3a72c25720c023b960b23c6f5891be05bc"; + // `keccak256("eth_getTransactionByHash\0[\"\"]")` — same formula as + // `transport_cache_key` in `common/provider/transport.rs`. + const LATER_CACHE_KEY: &str = + "0x91bbb37d27a588e217e5be6aeab0fb377ffea0ad3a2714d1f54ceb69852124f2"; + + let mut envelope: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cache()).expect("read offline cache")) + .expect("parse offline cache"); + // Clone the dump target's TX response shape and rewrite hash + gas so the + // later index is fetchable but rejected at execution (intrinsic gas). + let mut template: Option = None; + let target_marker = format!("\"hash\":\"{TX}\""); + for entry in envelope["cache"].as_array().expect("cache entries") { + let value = entry["value"].as_str().expect("entry value is a string"); + if !value.contains(&target_marker) { + continue; + } + let response: serde_json::Value = + serde_json::from_str(value).expect("parse transaction response"); + if response["result"].get("hash").and_then(|h| h.as_str()) == Some(TX) { + template = Some(response); + break; + } + } + let mut response = template.expect("offline cache must hold the dump target TX object"); + let result = response["result"].as_object_mut().expect("tx result object"); + result.insert("hash".into(), serde_json::Value::String(LATER.to_string())); + result.insert("transactionIndex".into(), serde_json::Value::String("0x2".into())); + result.insert("gas".into(), serde_json::Value::String("0x0".into())); + envelope["cache"].as_array_mut().expect("cache").push(serde_json::json!({ + "key": LATER_CACHE_KEY, + "value": response.to_string(), + })); + + let cache_path = temp_path("dump_abort_after"); + std::fs::write(&cache_path, envelope.to_string()).expect("write doctored cache"); + + let list_path = std::env::temp_dir() + .join(format!("mega_evme_verify_dump_abort_after_{}.txt", std::process::id())); + std::fs::write(&list_path, format!("{TX}\n{LATER}\n")).expect("write tx list"); + + let dir = + std::env::temp_dir().join(format!("mega_evme_batch_dump_abort_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("create dump dir"); + let fixture_path = dir.join(format!("{TX}.json")); + let sentinel = br#"{"pre-existing":"must-not-be-clobbered"}"#; + std::fs::write(&fixture_path, sentinel).expect("seed pre-existing fixture"); + + let run = replay( + &cache_path, + &[ + "--tx-file", + list_path.to_str().unwrap(), + "--dump-fixture-dir", + dir.to_str().unwrap(), + "--overwrite", + "--json", + ], + ); + let _ = std::fs::remove_file(&cache_path); + let _ = std::fs::remove_file(&list_path); + + assert!(!run.success, "an aborted block must exit non-zero.\nstderr: {}", run.stderr); + let lines = run.ndjson(); + assert!(lines.len() >= 2, "expected lines for both targets, got: {lines:?}"); + + let dump_line = lines + .iter() + .find(|line| line["tx_hash"].as_str() == Some(TX)) + .expect("dump target must appear in the output"); + // Sentinel: the target ran and built a Ready draft that was then discarded. + // The old test aborted before the target, so there was never a fixture report. + assert!( + dump_line.get("error").is_none(), + "dump target must execute before the abort: {dump_line}" + ); + assert!(dump_line["success"].as_bool() == Some(true), "dump target must succeed: {dump_line}"); + let fixture_error = dump_line["fixture"]["error"] + .as_str() + .expect("discarded Ready draft must surface as fixture.error"); + assert!( + fixture_error.contains("discarded") || fixture_error.contains("aborted"), + "expected draft-discarded message, got: {fixture_error}" + ); + assert!( + dump_line["fixture"].get("path").is_none(), + "discarded draft must not report a written path: {dump_line}" + ); + + let kept = std::fs::read(&fixture_path).expect("pre-existing fixture must still exist"); + assert_eq!(kept, sentinel, "--overwrite must not clobber when the dump never finalizes"); + let _ = std::fs::remove_dir_all(&dir); +} + +/// Replay receipts stamp each inner log with the outer receipt's block/tx +/// identity and a block-global `logIndex` that starts above zero when earlier +/// receipts in the block already emitted logs. +/// +/// The single-transaction capture this file otherwise uses is log-less, so this +/// one reads the whole-block capture, whose late transactions emit logs and sit +/// behind other log-emitting receipts. `MEGA_EVME_TEST_ENVELOPE` overrides it. +#[test] +fn test_replay_receipt_inner_log_metadata_matches_outer_receipt() { + let envelope = match std::env::var("MEGA_EVME_TEST_ENVELOPE") { + Ok(path) if !path.is_empty() => PathBuf::from(path), + _ => common::fixture("replay_batch_blocks.cache.json"), + }; + + // Last transaction of block 22945844: multi-log, with many preceding logs. + const LATE_TX: &str = "0xb6a0b7a302c741f64b8e46861a3dcb2d5c1047f6f2cb89a35b5c2183c96296b7"; + + let run = replay(&envelope, &["--json", LATE_TX]); + assert!(run.success, "envelope replay must exit 0.\nstderr: {}", run.stderr); + let summary = run.json(); + let receipt = &summary["receipt"]; + let block_hash = receipt["blockHash"].as_str().expect("receipt blockHash"); + let tx_hash = receipt["transactionHash"].as_str().expect("receipt transactionHash"); + let tx_index = receipt["transactionIndex"].as_str().expect("receipt transactionIndex"); + assert_eq!(tx_hash, LATE_TX); + let logs = receipt["logs"].as_array().expect("receipt logs array"); + assert!(!logs.is_empty(), "late envelope tx must emit logs (got empty); envelope may be stale"); + // Sentinel: preceding receipts emitted logs, so the first log_index is > 0. + let first_log_index = u64::from_str_radix( + logs[0]["logIndex"].as_str().expect("logIndex string").trim_start_matches("0x"), + 16, + ) + .expect("parse logIndex"); + assert!( + first_log_index > 0, + "expected non-zero preceding-log offset, got logIndex={first_log_index}" + ); + for (i, log) in logs.iter().enumerate() { + assert_eq!(log["blockHash"].as_str(), Some(block_hash), "log {i} blockHash"); + assert_eq!(log["transactionHash"].as_str(), Some(tx_hash), "log {i} transactionHash"); + assert_eq!(log["transactionIndex"].as_str(), Some(tx_index), "log {i} transactionIndex"); + assert!(log["logIndex"].is_string(), "log {i} must carry logIndex: {log}"); + } + + // Batch path stamps the same fields. + let list_path = std::env::temp_dir() + .join(format!("mega_evme_verify_batch_log_meta_{}.txt", std::process::id())); + std::fs::write(&list_path, format!("{LATE_TX}\n")).expect("write tx list"); + let batch = replay(&envelope, &["--tx-file", list_path.to_str().unwrap(), "--json"]); + let _ = std::fs::remove_file(&list_path); + assert!(batch.success, "batch replay must exit 0.\nstderr: {}", batch.stderr); + let line = &batch.ndjson()[0]; + let batch_receipt = &line["receipt"]; + assert_eq!(batch_receipt["blockHash"].as_str(), Some(block_hash)); + assert_eq!(batch_receipt["transactionHash"].as_str(), Some(tx_hash)); + let batch_logs = batch_receipt["logs"].as_array().expect("batch receipt logs"); + assert!(!batch_logs.is_empty(), "batch path must also carry non-empty logs"); + let batch_first = u64::from_str_radix( + batch_logs[0]["logIndex"].as_str().expect("logIndex").trim_start_matches("0x"), + 16, + ) + .expect("parse batch logIndex"); + assert!(batch_first > 0, "batch logIndex must start above zero, got {batch_first}"); + for (i, log) in batch_logs.iter().enumerate() { + assert_eq!(log["blockHash"].as_str(), Some(block_hash), "batch log {i} blockHash"); + assert_eq!(log["transactionHash"].as_str(), Some(tx_hash), "batch log {i} transactionHash"); + assert!(log["logIndex"].is_string(), "batch log {i} must carry logIndex: {log}"); + } +} diff --git a/bin/mega-evme/tests/state.rs b/bin/mega-evme/tests/state.rs index bb811a91..3a88a614 100644 --- a/bin/mega-evme/tests/state.rs +++ b/bin/mega-evme/tests/state.rs @@ -200,7 +200,7 @@ async fn test_create_initial_state_fork_real_rpc_smoke() { "mega-evme", "--rpc", &rpc_url, - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -272,7 +272,7 @@ async fn test_create_initial_state_fork_real_rpc_storage_cache_hit() { "mega-evme", "--rpc", &rpc_url, - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), @@ -304,7 +304,7 @@ async fn test_create_initial_state_fork_real_rpc_storage_cache_hit() { "mega-evme", "--rpc", &rpc_url, - "--rpc.cache-size", + "--rpc.cache-max-entries", "256", "--rpc.cache-dir", dir.path().to_str().unwrap(), diff --git a/docs/mega-evme/SUMMARY.md b/docs/mega-evme/SUMMARY.md index 7c817797..995a2c09 100644 --- a/docs/mega-evme/SUMMARY.md +++ b/docs/mega-evme/SUMMARY.md @@ -8,6 +8,7 @@ - [run](commands/run.md) - [tx](commands/tx.md) - [replay](commands/replay.md) +- [cache](commands/cache.md) ## Configuration diff --git a/docs/mega-evme/commands/cache.md b/docs/mega-evme/commands/cache.md new file mode 100644 index 00000000..320bc547 --- /dev/null +++ b/docs/mega-evme/commands/cache.md @@ -0,0 +1,110 @@ +--- +description: Merge provider-cache files or capture envelopes offline. +--- + +# cache + +Offline utilities for RPC cache files produced by `mega-evme`. + +Today the only subcommand is `merge`, which consolidates multiple cache files into one without contacting a network. + +## Usage + +``` +mega-evme cache merge ... --output +``` + +## `cache merge` + +Union one or more input files into a single output file. + +Inputs are auto-detected by JSON shape: + +| Shape | On-disk form | Produced by | +| ---------------- | ------------------------------------------- | --------------------------------------- | +| Provider cache | JSON array of `{key, value}` | `--rpc.cache-dir` per-chain files | +| Capture envelope | `{version, chain_id, cache, external_env?}` | `--rpc.capture-file` / offline fixtures | + +All inputs in one invocation must share the same shape. +Mixing a provider-cache file with a capture envelope is a hard error that names the offending path. + +### Output locking + +`--output` may be a file a live `mega-evme` run is persisting to. +The merge therefore uses the same protocol that clean-exit persist uses (see [State Management](../configuration/state-management.md#concurrent-cache-dir-sharing)): + +1. Take the exclusive advisory lock on the output's sidecar (`.lock`), blocking until it is free. +2. Under that lock, read whatever the output file holds now and fold it into the union as one more input. +3. Write via temp file + atomic rename, then release the lock. + +Folding the current output in is what makes the lock worth taking: entries a concurrent process wrote while the merge waited are carried into the merged result instead of being overwritten. +The merge's own inputs win where their keys collide with the output's prior entries. + +If the lock cannot be acquired at all (for example the sidecar path is not writable), the merge fails with an error and writes nothing. +An unlocked write would silently drop a concurrent process's entries, which is the failure the merge exists to prevent. + +An existing output that cannot be parsed at all (corrupt JSON) is replaced by the merged inputs, with a warning. +An existing output that parses but cannot be folded — the other cache shape, an unrecognized JSON shape, a different `chain_id`, a different envelope `version` — is a hard error that names the output path and leaves the file untouched. +Both shapes classify it the same way: a mistyped `--output` should not destroy a file the merge cannot read as its own. + +Warnings about a merge that may be silently wrong or lossy — an output being replaced, or chain identity that cannot be validated — are printed on stderr regardless of verbosity. +They do not depend on `-v` flags or `RUST_LOG`, which only add the structured log event alongside them. +Stdout carries the summary line only, so it stays parseable. + +### Provider-cache merge + +- Union entries by `key`. +- Later inputs win on collision; the inputs win over entries already in `--output`. +- Output is a provider-cache-shaped JSON array, written atomically (temp file + rename) under the output lock. +- Chain identity is taken only from the standard filename `rpc-cache-{chain_id}.json` (provider-cache bodies have no chain field). + Every input path and `--output` that matches that pattern must name the same chain id; a mismatch is a hard error that names the conflicting files. + Paths that do not match the pattern emit a warning that chain identity cannot be validated for them, and the merge proceeds for those paths without a filename-based check. + +### Envelope merge + +- Every input must use the current envelope `version` and the same `chain_id` (else hard error naming the mismatch). + An envelope already at `--output` must agree with them too. +- Union the `cache` arrays by key; later inputs win on collision, and the inputs win over entries already in `--output`. +- `external_env`: if two inputs carry non-identical snapshots, hard error; otherwise propagate the non-null snapshot. + A snapshot already at `--output` is held to the same rule. +- Output is a pretty-printed envelope, written atomically under the output lock. + +### Summary + +On success, `cache merge` prints one line and exits 0: + +``` +Merged 3 inputs (120 entries in) → 95 unique entries out +``` + +When the output file already held entries, they are counted separately, so the arithmetic still adds up: + +``` +Merged 3 inputs (120 entries in + 12 already in the output) → 101 unique entries out +``` + +### Examples + +Merge sharded worker provider caches after a multi-process campaign: + +```bash +mega-evme cache merge \ + worker0/rpc-cache-4326.json \ + worker1/rpc-cache-4326.json \ + worker2/rpc-cache-4326.json \ + --output ./rpc-cache-4326.json +``` + +Merge two capture envelopes for the same chain: + +```bash +mega-evme cache merge \ + capture-a.json \ + capture-b.json \ + -o merged-capture.json +``` + +## See also + +- [State Management](../configuration/state-management.md#rpc-cache-and-retry) — live `--rpc.cache-dir` behavior and concurrent sharing +- [replay](replay.md#rpc-cache-file) — capture and offline replay fixtures diff --git a/docs/mega-evme/commands/replay.md b/docs/mega-evme/commands/replay.md index dd56ea11..db0bc2c0 100644 --- a/docs/mega-evme/commands/replay.md +++ b/docs/mega-evme/commands/replay.md @@ -1,28 +1,47 @@ --- -description: Fetch and re-execute an on-chain transaction with optional overrides and tracing. +description: Fetch and re-execute one or many on-chain transactions with optional overrides, tracing, and on-chain receipt verification. --- # replay -Re-execute a historical transaction locally using an RPC endpoint or a previously captured fixture file. +Re-execute historical transactions locally using an RPC endpoint or a previously captured fixture file. In online mode, `mega-evme` fetches the transaction, block environment, and pre-state from the RPC and re-executes locally. In offline mode (`--rpc.replay-file`), all data is served from a local fixture captured by an earlier run — no network access is required. +`replay` has two modes. +The single-transaction mode replays the transaction named by the positional `TX_HASH` and supports the full option set (overrides, tracing, state dumps, fixture dumps). +[Batch mode](#batch-replay) (`--tx-file` / `--block`) replays many transactions in one process and reports one summary per transaction. + ## Usage ``` -mega-evme replay [OPTIONS] +mega-evme replay [OPTIONS] |--block > ``` +Exactly one replay target is required: the positional `TX_HASH`, `--tx-file`, or `--block`. +The three are mutually exclusive. + ## Arguments ### `TX_HASH` -The transaction hash to replay (32-byte hex, required). +The transaction hash to replay (32-byte hex). `mega-evme` re-executes the transaction locally using state and block context sourced from either an RPC endpoint or a local fixture file. This gives you a fully reproducible execution without needing a local archive node. +The transaction lookup's own block number and inclusion hash are classified first, before any block is fetched. +Both present is a mined target; neither present is a pending one. +The two mixed shapes cannot be replayed at all and are reported as RPC failures (exit `3`) from the metadata alone, so no fetch precedes the verdict and no later failure can mask it: +a block number without an inclusion hash is an unanchored view, since the number alone cannot anchor the replay to a block body, and an inclusion hash without a block number is contradictory metadata, since the hash proves inclusion while the missing number denies it. + +Resolving a mined transaction then takes two more calls — the block the lookup reports and that block's parent — which a reorg in progress or a load-balanced endpoint can answer from different views of the chain. +Replaying a mixed view yields a plausible but wrong result, so the answers are checked against each other and a disagreement is reported as an RPC failure (exit `3`) instead of being replayed: +the parent block must be the replayed block's parent, the fetched block must be the one the transaction was resolved as included in, and that block's body must list the transaction — its position there is what defines the preceding transactions replayed ahead of it. + +A pending transaction has no such pair, since its state base is the latest block, which is also the block it is replayed in. +That block is fetched once and fills both roles, so the two cannot disagree. + ### `--rpc ` Aliases: `--rpc-url` @@ -35,6 +54,264 @@ Required for online replay and capture mode; omit when using `--rpc.replay-file` mega-evme replay --rpc https://mainnet.megaeth.com/rpc ``` +## Batch Replay + +Replaying a corpus of transactions one process at a time pays for provider construction, chain-id resolution, and RPC cache parsing once per transaction — work that dominates the actual EVM execution. +Batch mode does all of it once. + +A batch run builds a single provider and a single RPC cache, groups the requested transactions by their containing block, and processes the blocks in ascending order. +Each block is executed exactly once: state is forked at the parent block, pre-execution changes are applied, and every transaction of the block runs in order, with each requested transaction's result recorded before it is committed. +A capture file (`--rpc.capture-file`) is persisted once, on exit, even if some transactions failed — the captured responses are the artifact you need to debug the failure offline. +The per-chain on-disk RPC cache is opt-in for batch runs: it is loaded and persisted only when `--rpc.cache-dir` names a directory explicitly, or when `--rpc.clear-cache` asks for the cache file to be deleted. +A batch scan walks linear history whose request keys essentially never repeat across runs, so a shared cache file buys almost no hits, while its clean-exit re-read-merge-rewrite grows with the file and serializes concurrent processes on the persist lock. +The in-memory cache still serves every repeated request within the run. + +`--rpc.clear-cache` counts as an explicit opt-in because deleting the cache file only means something while the disk cache is engaged: a batch run that forced the cache off would parse the flag, do nothing, and leave the polluted file in place for the next run. +With it, the cache file (at the default path, or under `--rpc.cache-dir`) is deleted under the sidecar lock, the run starts from an empty cache, and the cache is persisted on exit. +An explicit `--rpc.no-cache-file` still wins over both flags and keeps the disk cache off, exactly as in single-transaction mode. + +A plain batch replay issues the same RPC calls as single-transaction replay, so an offline envelope captured by single-transaction runs serves a batch run without a cache miss. +`--verify-receipt` and `--dump-fixture-dir` are the exception: both fetch the receipt of every target in the block, including transactions a single-transaction capture never asked about, so an older envelope will miss them and the run exits `3`. + +### `--tx-file ` + +Replay every transaction hash listed in ``, one per line. + +Blank lines and lines whose first non-whitespace character is `#` are ignored. +A hash listed more than once is replayed once. +A line that is not a valid 32-byte hex hash aborts the run before any network access, naming the offending line number. + +### `--block ` + +Replay every transaction of block `N`, given in decimal or `0x`-prefixed hex. + +### Restrictions + +Batch mode reports one summary per transaction and has no meaningful semantics for single-file fixture dumps, tracing, state dumps, or what-if knobs, so the following are rejected up front with an explanatory error rather than silently ignored: + +- `--dump-fixture` — use [`--dump-fixture-dir`](#--dump-fixture-dir-dir) for batch sedimentation +- Transaction overrides (`--override.gas-limit`, `--override.value`, `--override.input`, `--override.input-file`) +- `--override.spec` — each block's spec is auto-detected from its timestamp +- All trace options (`--trace`, `--trace.output`, `--tracer`, `--trace.*`) +- All state dump options (`--dump`, `--dump.output`) + +Single-transaction replay keeps accepting all of them. +Batch mode additionally accepts [`--dump-fixture-dir`](#--dump-fixture-dir-dir) for per-target fixture sedimentation. + +### Output + +With `--json`, batch mode writes NDJSON: exactly one compact, single-line JSON object per requested transaction, in processing order (ascending block, then transaction index). + +A transaction that executed is reported as its `tx_hash`, `block_number`, and `tx_index`, followed by the same fields the single-transaction JSON output carries (`success`, `gas_used`, `logs_count`, and the optional `output` / `contract_address` / `revert_reason` / `halt_reason`) and its `receipt`. +Both shapes below are expanded for readability; on the wire each object occupies exactly one line. + +```json +{ + "tx_hash": "0x…", + "block_number": 22945844, + "tx_index": 3, + "success": true, + "gas_used": 81740, + "logs_count": 0, + "receipt": { "…": "…" } +} +``` + +A transaction that could not be executed is reported as an error entry instead: + +```json +{ + "tx_hash": "0x…", + "error": { "kind": "not_found", "message": "Transaction not found" } +} +``` + +`kind` is one of `not_found` (unknown hash), `pending` (mined into no block yet), `rpc` (an RPC call failed), or `execution` (block setup or the block executor rejected the transaction). +Execution outcomes are not errors: a reverted or halted transaction is a normal result line with `success: false`. + +A failure while running the block aborts it, because the executor state no longer matches the chain. +The transaction the failure is about — the hash the endpoint denied, or the one the executor rejected — is reported with that failure's own kind. +Every target behind it is reported as `rpc` with a message naming the aborting cause: nothing was established about those transactions, so they went unanswered rather than being unknown. +Targets that never ran are still emitted in the block's transaction-index order, keeping the whole stream in ascending `(block, tx_index)` order; a hash the endpoint claimed for this block but that the body does not list is reported last within its block, in input order, as `rpc` (an unanswered, divergent view — not a definitive unknown hash). +Hashes that could not be resolved to a block at all (unknown as `not_found`, pending, or an endpoint failure during resolution) are emitted before every block result, since the run cannot place them in the stream's order. + +Without `--json`, each transaction is printed with a header naming its hash, block, and index, followed by the same summary and receipt the single-transaction mode prints. +A final one-line summary (transactions replayed, transactions failed, elapsed time) is logged at `INFO` level, so pass `-vvv` to see it. + +With [`--verify-receipt`](#receipt-verification), each result line additionally carries a `verification` object. +With [`--dump-fixture-dir`](#--dump-fixture-dir-dir), each result line additionally carries a `fixture` object (`path`, `skipped`, or `error`). + +### Exit Status + +A batch run exits `0` when every requested transaction produced an execution result and nothing the run was asked to do failed, and non-zero otherwise — see [Exit codes](../overview.md#exit-codes) for how the failure classes are ranked. +Fixture skips (fidelity mismatch, BLOCKHASH readers, unsupported shapes) are not failures and do not fail the run; a fixture construction or write failure is an execution-class failure of its target; an unanswered on-chain receipt for the fidelity gate is an rpc-class failure of its target. +When a mid-block abort discards a drafted fixture, that fixture error inherits the abort's class (so a transport abort still exits `3`). +The NDJSON stream is written to stdout in both cases; diagnostics go to stderr. + +Swept targets behind an abort always report as `rpc` ("unanswered"). +When the aborting transaction is not itself a target, the run still tallies the abort's own class so the process exit reflects the root cause — a non-target executor abort exits `1`, a transport abort exits `3`. +`--block 0` is rejected as invalid input (exit `1`): the user asked for a genesis block that cannot be replayed. +An endpoint that resolves a transaction hash into block 0 is contradictory endpoint data instead — each such target is reported as `rpc` and the run exits `3`, in the same family as unanchored views and contradictory metadata. +A block that genuinely holds no transactions produces no stdout lines, exits `0`, and says so on stderr. + +### Examples + +Replay a whole block offline and stream the results as NDJSON: + +```bash +mega-evme replay --rpc.replay-file ./fixtures/blocks.json --block 22945844 --json +``` + +Replay a corpus of transactions against a live RPC, one process for the lot: + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc --tx-file ./corpus.txt --json > results.ndjson +``` + +Where `corpus.txt` looks like: + +``` +# regression corpus, refreshed 2026-08-03 +0xde3d56dc739484166b8af1bea757bf7e3e9a4b9a0fb62d722703345570dfc1d6 +0x323ddc8e67dfc134284d78c65f3c1dc7ff45ba1db02eeaf62e211ae3253478ef +``` + +Count the transactions that did not succeed: + +```bash +jq -c 'select(.tx_hash and (.error != null or .success == false))' results.ndjson | wc -l +``` + +A failed run ends its stdout with a run-level `{"error": …}` object (see [Exit codes](../overview.md#exit-codes)), which carries no `tx_hash`, so selecting on `.tx_hash` keeps the count to per-transaction lines. + +## Receipt Verification + +Replaying a transaction only proves that the local EVM produced _some_ result; equivalence verification needs that result checked against what the chain recorded. +`--verify-receipt` builds that check into the tool: it fetches the on-chain receipt of every replayed transaction and compares it against the receipt the replay produced, so verifying an upgrade is one command over one transaction list instead of a replay run plus a separate receipt-diffing pipeline. + +### `--verify-receipt` + +Verify every replayed transaction against its on-chain receipt. +Supported in both single-transaction and [batch](#batch-replay) mode. + +Three dimensions are compared: + +- **Status** — the success flag. +- **Gas used** — the transaction's gas, not the block's cumulative gas. +- **Logs** — the number of logs, and each log's `address`, `topics`, and `data`. + +Logs are compared explicitly rather than inferred from gas: `LOG` gas depends on topic count and data length, never on content, so two executions can burn identical gas yet emit different log payloads. + +The receipt is fetched with the same call the [fixture dump](#self-validating-fixture-dump) uses, so a run with `--rpc.capture-file` records it and a later `--rpc.replay-file` run verifies the same transaction offline. +An envelope captured without `--verify-receipt` (or by any earlier run that never needed a receipt) holds no receipts, so verifying against it fails the receipt fetch — capture once online with the flag, then re-verify offline as often as you like. + +### Verified, Unverified, and Mismatched + +A transaction is only reported as mismatched when both receipts were compared and disagreed. +Anything that prevents the comparison from running is an infrastructure failure — the transaction is _unverified_, which is a different finding from a divergence: + +- The endpoint fails the receipt call, or has pruned the receipt below its retention height (common on non-archive endpoints): reported as an `rpc` failure. +- The receipt describes a different inclusion than the replayed block (its `blockHash` differs from the replayed block, or is null — a reorg in progress, or a load-balanced endpoint serving divergent views): reported as an `rpc` failure, because comparing against it would compare the replay to the wrong on-chain execution, and a receipt with no inclusion hash cannot be anchored at all. +- The receipt describes a different transaction than the one requested (its `transactionHash` is not the hash the receipt was asked for — an inconsistent endpoint, or a tampered capture): reported as an `rpc` failure, because the verdict would describe the wrong transaction, and two transactions sharing their consensus facts would even yield a spurious match. +- The target is a pending transaction, which has no receipt yet: rejected up front in single-transaction mode, and reported as a `pending` error entry in batch mode. + +In batch mode, when a target already produced an execution result and only the receipt fetch failed, the target keeps its full result line (execution summary, local receipt, timing) and reports the failure on that line as `"verification": {"error": "…"}`. +The target still counts as `replayed`; the unanswered receipt is tallied as `rpc` and the run exits `3`. +A target that never reached execution (pending, not-found, block setup failure) remains a bare error entry, exactly like any other infrastructure failure before replay. + +Transaction overrides and `--override.spec` are still accepted with `--verify-receipt`, but they make the replay a what-if that the chain never executed, so the comparison will normally report a mismatch. + +### Output + +With `--json`, the verdict is a `verification` object — added to the single-transaction summary, and to each batch result line. +The field is absent entirely without the flag. + +A match carries nothing else: + +```json +{ "match": true } +``` + +An unanswered receipt (fetch failed, pruned, reorg / divergent inclusion) carries only the error — no `match` field, so it is never confused with a divergence: + +```json +{ "error": "No on-chain receipt was fetched for this transaction" } +``` + +A mismatch carries a `diff` holding only the dimensions that disagreed, each as `{"onchain": …, "replay": …}`: + +```json +{ + "match": false, + "diff": { + "status": { "onchain": true, "replay": false }, + "gas_used": { "onchain": 75514, "replay": 75500 }, + "logs": { + "count": { "onchain": 2, "replay": 1 }, + "first_mismatch": { + "index": 0, + "field": "address", + "onchain": "0x00000000000000000000000000000000000000aa", + "replay": "0x00000000000000000000000000000000000000bb" + } + } + } +} +``` + +Under `logs`, `count` is present when the two sides emitted a different number of logs, and `first_mismatch` names the first log both sides emitted whose contents differ — its `field` is `address`, `topics`, or `data`, and the two values are that field's contents on each side. +Both can appear at once, which distinguishes truncated logs from rewritten ones. + +Without `--json`, each transaction gets one verdict line after its usual output: + +``` +verification: MATCH +verification: MISMATCH (gas_used: onchain 75514 vs replay 75500) +verification: FAILED (No on-chain receipt was fetched for this transaction) +``` + +The mismatch line names every dimension that disagreed, comma-separated. +The failed line is used when the comparison never ran. + +### Exit Status + +A run in which every target replayed and every verification matched exits `0`. +A verification mismatch exits `2` through a dedicated error (`Receipt verification mismatch: N of M verified transaction(s) did not reproduce the on-chain receipt`), reported after every result line has been written. +Infrastructure failures keep their own exit code and take precedence in a batch run: a target that never replayed was also never verified, so reporting it as a mismatch would overstate what the run found. +An execution or input failure exits `1`, an RPC failure (including a receipt the endpoint cannot serve) exits `3`. +See [Exit codes](../overview.md#exit-codes) for the full taxonomy and the batch precedence rule. + +### Examples + +Verify one transaction against a live RPC: + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc --verify-receipt 0xabc123... +``` + +Verify a whole corpus in one process and collect the divergences: + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc \ + --tx-file ./corpus.txt --verify-receipt --json > results.ndjson + +jq -c 'select(.tx_hash and .verification.match == false)' results.ndjson # mismatched +jq -c 'select(.tx_hash and (.error != null or .verification.error != null))' results.ndjson # unverified +``` + +Receipt-fetch failures on a target that still replayed live under `.verification.error` on the result line (the line keeps `receipt` / `success`); infrastructure failures that prevented execution live under `.error`. +Both selectors require `.tx_hash` so that the run-level `{"error": …}` object a failed run appends to stdout is not counted as an unverified transaction. + +Capture once online, then re-verify the same corpus offline: + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc \ + --rpc.capture-file ./corpus.cache.json --tx-file ./corpus.txt --verify-receipt --json + +mega-evme replay --rpc.replay-file ./corpus.cache.json \ + --tx-file ./corpus.txt --verify-receipt --json +``` + ## RPC Cache File `mega-evme replay` supports a transport-level JSON-RPC fixture mechanism that records every request/response pair to a single file and serves them back on later runs without touching the network. @@ -53,9 +330,12 @@ On subsequent runs the existing file is loaded, its entries are merged into the The updated set of entries is persisted back to the same file on clean exit. The file also embeds an external-environment snapshot — currently the set of `--bucket-capacity` values in effect — so the captured fixture is self-contained. -If `--bucket-capacity` is not passed on a subsequent run, the previous envelope's values are reused; passing `--bucket-capacity` overrides them. +If `--bucket-capacity` is not passed on a subsequent run, the previous envelope's values are reused; passing `--bucket-capacity` overrides them (an intentional A→B refresh of an existing capture is accepted at persist when no concurrent writer changed the on-disk snapshot, and a run that reused the previous values yields to a concurrent refresh rather than conflicting with it; only two writers changing the same snapshot differently hard-errors, naming the load-time, caller, and on-disk values — see [state management](../configuration/state-management.md#rpc-cache-and-retry)). + +The capture is written even when the replay itself failed — an execution or verification failure is exactly the case you want to debug offline. +If the write fails, it is reported on stderr like any other failure, next to the run's own error; the run error keeps the exit code, since it is the root cause. -`--rpc.capture-file` is mutually exclusive with `--rpc.replay-file`, `--rpc.cache-dir`, `--rpc.clear-cache`, `--rpc.no-cache-file`, and `--rpc.cache-size`. +`--rpc.capture-file` is mutually exclusive with `--rpc.replay-file`, `--rpc.cache-dir`, `--rpc.clear-cache`, `--rpc.no-cache-file`, and `--rpc.cache-max-entries`. ### `--rpc.replay-file ` @@ -67,7 +347,7 @@ Any request that is not present in the fixture aborts the run with a hard error Bucket-capacity data is read from the fixture envelope, so `--bucket-capacity` is neither required nor accepted with `--rpc.replay-file`. Passing `--bucket-capacity` together with `--rpc.replay-file` is rejected; to regenerate a fixture with new capacities, re-run in capture mode. -`--rpc.replay-file` is mutually exclusive with `--rpc`, `--rpc.capture-file`, `--rpc.cache-dir`, `--rpc.clear-cache`, `--rpc.no-cache-file`, and `--rpc.cache-size`. +`--rpc.replay-file` is mutually exclusive with `--rpc`, `--rpc.capture-file`, `--rpc.cache-dir`, `--rpc.clear-cache`, `--rpc.no-cache-file`, and `--rpc.cache-max-entries`. ### Examples @@ -116,6 +396,7 @@ The fixture still self-validates and reproduces gas exactly; only such balance-d A target transaction that reads a block hash via `BLOCKHASH` is also rejected: fixtures carry no historical block hashes, so the isolated re-execution could not reproduce the values the replay observed. Block hash reads by preceding transactions in the same block do not matter — only the target transaction's reads are checked. Because the fidelity gate reads the receipt, an offline dump (`--rpc.replay-file`) requires the receipt to be present in the capture — so capture and dump together in the online run, then re-dump offline reproducibly. +A receipt the endpoint does not serve — no receipt at all, one describing a different inclusion than the replayed block, or one describing a different transaction than the one requested — is classified exactly as under [`--verify-receipt`](#--verify-receipt): an RPC failure (exit `3`), because the question went unanswered rather than answered no. When combined with `--rpc.capture-file`, the capture file is written even if execution or the fidelity gate fails, so the captured RPC responses remain available for debugging the failure offline. ```bash @@ -130,6 +411,50 @@ mega-evme replay --rpc.replay-file ./cap.json --dump-fixture ./fixtures/0xabc123 state-test ./fixtures/0xabc123.json ``` +### `--dump-fixture-dir ` + +Batch-only. +Dump a self-validating fixture for every successfully replayed target into `/.json`. +The fixture content and format match the single-transaction [`--dump-fixture`](#--dump-fixture-file) path (same EEST schema, same sorted `megaEnv`, same self-validation via `state-test`). +The directory is created if it does not exist. +Existing files are refused unless `--overwrite` is also set — a refused overwrite is a failed dump for that target, not a skip. + +Per-target gating mirrors the single-transaction rules, but records a skip instead of failing the run. +The fixture draft is built against the pre-commit state (same moment as the single-transaction dump) and only written after the block finishes successfully — a commit-time rejection or finish failure never creates or replaces a fixture file. + +| Gate | Outcome | +| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| On-chain receipt unavailable (not in capture, pruned, reorg/divergent inclusion, receipt for another transaction) | `fixture.error` with the reason; rpc-class failure | +| Fidelity mismatch (gas / status / logs root) | `fixture.skipped` with `fidelity gate failed: …` | +| Target reads `BLOCKHASH` | `fixture.skipped` (fixtures carry no historical block hashes) | +| Unsupported shape (deposit, EIP-7702, unknown spec mapping) | `fixture.skipped` | +| Fixture construction failure (database / pre-state reads) | `fixture.error` with the reason; execution-class failure | +| Finalize / write / self-validation failure, refused overwrite | `fixture.error` with the reason; execution-class failure | +| Pending / unresolvable target | already an error entry; no fixture report | + +`BLOCKHASH` access is isolated per transaction: the access record is cleared before each transaction of the block, so preceding readers do not poison a later target's dump. + +NDJSON result lines gain `"fixture": {"path": "…"}`, `"fixture": {"skipped": ""}`, or `"fixture": {"error": ""}`. +Human mode prints one fixture line per target. +An end-of-run `INFO` summary reports written / skipped / failed counts. + +A failed dump is reported on the target's own result line rather than replacing it: the transaction did replay, so its receipt — and, with [`--verify-receipt`](#receipt-verification), its verdict — is still what the run was asked for, and a divergence found on such a target is still counted as a mismatch. +Fixture skips do not fail the run; a failed dump does — as an execution-class failure for construction/write failures, or as an rpc-class failure when the on-chain receipt question went unanswered. + +Registration into `bench/replay/manifest.json` is not performed — corpus curation stays manual. +`--dump-fixture-dir` cannot be combined with `--dump-fixture`, and is rejected in single-transaction mode. + +```bash +# Sweep a whole block offline into per-tx fixtures (targets whose capture lacks +# a receipt fail as rpc on the result line and exit 3): +mega-evme replay --rpc.replay-file ./fixtures/blocks.json \ + --block 22945844 --dump-fixture-dir ./fixtures/out --json + +# Sediment a curated list, replacing any previously written files: +mega-evme replay --rpc https://mainnet.megaeth.com/rpc \ + --tx-file ./corpus.txt --dump-fixture-dir ./fixtures/out --overwrite +``` + ## Throughput Benchmark To benchmark a replayed transaction, dump it to a fixture and time the fixture with the `state-test` runner — there is no `replay`-side benchmark flag: @@ -161,6 +486,17 @@ Useful when you want to test how the transaction would behave under a different mega-evme replay --override.spec Rex2 ``` +The override replaces the entire execution world, not just the EVM semantics. +The block is executed as if it had run on a chain whose schedule activates the forced spec at genesis: the pre-block system contract deploys, the EIP-2935 and EIP-4788 pre-block calls, the block-level resource limits, and the EVM semantics all come from the forced spec. +This keeps a forced replay coherent — mixing the historical setup with forced semantics would execute a world that never existed on any chain. + +A consequence worth stating explicitly: replaying an old block under a newer spec installs predeploys that did not exist at that block (for example, forcing `Rex5` on a pre-`Rex5` block deploys the `SequencerRegistry`), and forcing an older spec withholds predeploys the block did have, or installs an earlier version of them. +That is intentional — it is what "how would this transaction behave under spec X" means. +The replayed state therefore diverges from the chain's historical state by construction, so `--verify-receipt` will normally report a mismatch and `--dump-fixture` is rejected outright. + +The forced spec does not synthesize chain configuration. +Per-fork parameters (currently the `SequencerRegistry` seeds a chain publishes for `Rex5` and `Rex6`) are taken from the chain's own configuration, so a fork the chain has not configured cannot be forced: the run fails before executing, naming the missing parameters, rather than proceeding with an invented value. + ## Transaction Overrides Override flags let you modify the transaction before re-executing it. @@ -181,20 +517,25 @@ All of that context comes from the RPC. `replay` supports the following shared option groups. See the linked pages for full details. +Options marked _(single transaction only)_ are rejected in [batch mode](#batch-replay). +- **Batch replay** — Replay many transactions in one process via `--tx-file` / `--block`. + See [Batch Replay](#batch-replay) above. +- **Receipt verification** — Check every replayed transaction against its on-chain receipt via `--verify-receipt`. + See [Receipt Verification](#receipt-verification) above. - **SALT buckets** — Configure SALT bucket capacity for dynamic storage gas pricing. See [SALT Buckets](../configuration/salt-buckets.md). -- **State dump** — Dump or load pre/post-state snapshots. +- **State dump** _(single transaction only)_ — Dump or load pre/post-state snapshots. See [State Management](../configuration/state-management.md). - **RPC cache file** — Single-file JSON-RPC capture and offline replay via `--rpc.capture-file` / `--rpc.replay-file`. See [RPC Cache File](#rpc-cache-file) above. - **RPC cache / retry** — Per-chain response cache, retry, and rate-limit settings. See [RPC Cache and Retry](../configuration/state-management.md#rpc-cache-and-retry). -- **Tracing** — Emit execution traces (call traces, opcode traces, gas profiles, etc.). +- **Tracing** _(single transaction only)_ — Emit execution traces (call traces, opcode traces, gas profiles, etc.). See [Tracing Overview](../tracing/overview.md). -- **Fixture dump** — Write a self-validating EEST state-test fixture via `--dump-fixture`. +- **Fixture dump** — Write a self-validating EEST state-test fixture via `--dump-fixture` (single transaction) or `--dump-fixture-dir` (batch). See [Self-Validating Fixture Dump](#self-validating-fixture-dump) above. -- **Throughput benchmark** — Dump a fixture (`--dump-fixture`) and time it with `state-test --bench`. +- **Throughput benchmark** — Dump a fixture (`--dump-fixture` / `--dump-fixture-dir`) and time it with `state-test --bench`. See [Throughput Benchmark](#throughput-benchmark) above. ## Examples @@ -232,6 +573,18 @@ mega-evme replay --rpc https://mainnet.megaeth.com/rpc --override.input 0xdeadbe mega-evme replay --rpc https://mainnet.megaeth.com/rpc --override.spec Rex2 0xabc123... ``` +**Replay a whole block as NDJSON** + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc --block 22945844 --json +``` + +**Verify a whole block against its on-chain receipts** + +```bash +mega-evme replay --rpc https://mainnet.megaeth.com/rpc --block 22945844 --verify-receipt --json +``` + ## See Also - [`run`](./run.md) — Execute raw EVM bytecode locally without fetching from RPC diff --git a/docs/mega-evme/commands/run.md b/docs/mega-evme/commands/run.md index 3204eb69..8a873a91 100644 --- a/docs/mega-evme/commands/run.md +++ b/docs/mega-evme/commands/run.md @@ -54,7 +54,7 @@ Each group is documented on its own page. | Chain and spec | `--spec`, `--chain-id` | [Chain and Spec](../configuration/chain-and-spec.md) | | Block environment | `--block.number`, `--block.coinbase`, `--block.timestamp`, `--block.gaslimit`, `--block.basefee`, `--block.difficulty`, `--block.prevrandao`, `--block.blobexcessgas` | [Block Environment](../configuration/block-environment.md) | | SALT buckets | `--bucket-capacity` | [SALT Buckets](../configuration/salt-buckets.md) | -| RPC cache / retry | `--rpc.cache-size`, `--rpc.cache-dir`, `--rpc.no-cache-file`, `--rpc.clear-cache`, `--rpc.max-retries`, `--rpc.backoff-ms`, `--rpc.rate-limit` | [RPC Cache and Retry](../configuration/state-management.md#rpc-cache-and-retry) | +| RPC cache / retry | `--rpc.cache-max-entries`, `--rpc.cache-dir`, `--rpc.no-cache-file`, `--rpc.clear-cache`, `--rpc.max-retries`, `--rpc.backoff-ms`, `--rpc.cu-per-sec`, `--rpc.request-timeout` | [RPC Cache and Retry](../configuration/state-management.md#rpc-cache-and-retry) | | Tracing | `--trace`, `--tracer`, `--trace.output`, and tracer-specific flags | [Tracing Overview](../tracing/overview.md) | | Output | `--json` | See [JSON output](#json-output) below | @@ -304,8 +304,8 @@ RPC Options: --rpc.replay-file (replay command only) Serve JSON-RPC from a captured fixture; not usable as a run/tx offline-fork path - --rpc.cache-size - Max items in the in-memory RPC LRU cache; 0 disables it [default: 10000] + --rpc.cache-max-entries + Max items in the in-memory RPC LRU cache (and therefore the cache file); 0 = effectively unlimited (caps at 1,048,576 entries; the cache index is preallocated proportional to the cap) [default: 0] --rpc.cache-dir Directory for per-chain RPC cache files (default: platform cache dir) @@ -317,13 +317,21 @@ RPC Options: Delete the current chain's cache file before loading it --rpc.max-retries - Max transport retries on 429/503, rate-limit, and transport failures; 0 disables [default: 5] + Max transport retries on 429/503, rate-limit, transport failures, and request timeouts; 0 disables [default: 5] --rpc.backoff-ms Fixed sleep (ms) between retries; no exponential backoff [default: 1000] - --rpc.rate-limit - Compute-units-per-second budget for the retry layer [default: 660] + --rpc.cu-per-sec + Compute-unit budget (CU/s) for the retry layer's rate-limit accounting. This is NOT requests per second: each RPC method costs multiple compute units. A single-digit value will heavily self-throttle. Default (660) matches typical public-endpoint budgets + + [default: 660] + [alias: --rpc.rate-limit] + + --rpc.request-timeout + Total per-HTTP-request timeout in seconds (connect + response). `0` disables the timeout (previous behavior: a hung endpoint can block forever). A non-zero timeout surfaces a hung endpoint as a retryable transport error + + [default: 30] Chain Options: --spec diff --git a/docs/mega-evme/commands/tx.md b/docs/mega-evme/commands/tx.md index ce45e33b..b7f06942 100644 --- a/docs/mega-evme/commands/tx.md +++ b/docs/mega-evme/commands/tx.md @@ -204,13 +204,16 @@ RPC Options: [aliases: --rpc-url] [compat alias: --fork.rpc] --rpc.capture-file (replay command only) capture to fixture; not usable as a run/tx offline path --rpc.replay-file (replay command only) serve from fixture; not usable as a run/tx offline path - --rpc.cache-size In-memory RPC LRU cache size; 0 disables [default: 10000] + --rpc.cache-max-entries In-memory RPC LRU max entries; 0 = effectively unlimited (caps at 1,048,576 entries; the cache index is preallocated proportional to the cap) [default: 0] --rpc.cache-dir Per-chain RPC cache directory (default: platform cache dir) --rpc.no-cache-file Disable on-disk cache persistence --rpc.clear-cache Delete the current chain's cache file before loading --rpc.max-retries Max transport retries; 0 disables [default: 5] --rpc.backoff-ms Fixed retry sleep in ms [default: 1000] - --rpc.rate-limit Retry-layer compute-units-per-second budget [default: 660] + --rpc.cu-per-sec + Compute-unit budget (CU/s) for the retry layer's rate-limit accounting (NOT requests/s) [default: 660] + [alias: --rpc.rate-limit] + --rpc.request-timeout Total per-HTTP-request timeout (connect + response); 0 disables [default: 30] Chain Options: --spec Spec [default: Rex7] diff --git a/docs/mega-evme/configuration/chain-and-spec.md b/docs/mega-evme/configuration/chain-and-spec.md index 6befb4d3..f180fd28 100644 --- a/docs/mega-evme/configuration/chain-and-spec.md +++ b/docs/mega-evme/configuration/chain-and-spec.md @@ -8,6 +8,12 @@ These options control which MegaETH spec and chain ID the EVM uses during execut They are available in the `run` and `tx` commands. The `replay` command auto-detects the spec from the chain ID and block timestamp (see [replay](../commands/replay.md#spec-auto-detection)). +In all three commands a chosen spec defines the whole execution world, not only the opcode and gas rules: the system contracts predeployed before execution, the block-level resource limits, and the EVM semantics all come from that one spec. +For `run` and `tx` there is nothing else it could mean — there is no historical block to contradict it. +For `replay`, [`--override.spec`](../commands/replay.md#--overridespec-spec) makes the same choice explicitly: the block is replayed as if it had run on a chain at the forced spec, so replaying an old block under a newer spec installs predeploys that never existed at that block, and forcing an older spec withholds or downgrades the ones that did. +That is intentional, and it is what makes the answer a coherent what-if rather than a mixture of two worlds. +Chain-specific configuration is not synthesized along with it: a fork whose parameters the chain has not published (the `SequencerRegistry` seeds, today) cannot be forced, and the run fails up front naming what is missing. + ## Options | Flag | Default | Aliases | Description | diff --git a/docs/mega-evme/configuration/state-management.md b/docs/mega-evme/configuration/state-management.md index 86584df8..a542c257 100644 --- a/docs/mega-evme/configuration/state-management.md +++ b/docs/mega-evme/configuration/state-management.md @@ -217,22 +217,80 @@ The default cache directory is the platform cache directory: - **Linux**: `$XDG_CACHE_HOME/mega-evme/rpc` - **macOS**: `~/Library/Caches/mega-evme/rpc` +Batch replay (`--tx-file` / `--block`) is the exception: it engages the on-disk cache only when the invocation asks for it explicitly, and otherwise behaves as if `--rpc.no-cache-file` were set. +A batch scan walks linear history whose request keys essentially never repeat across runs, so the file buys almost no hits, while its clean-exit persist re-reads, merges, and rewrites the whole file under the cross-process lock — a cost that grows with the file and serializes concurrent batch processes. + +Two flags ask for it: `--rpc.cache-dir`, which names the file to use, and `--rpc.clear-cache`, which asks for that file to be deleted. +Clearing only means something while the disk cache is engaged, so a batch run that forced the cache off would parse the recovery flag, do nothing, and leave the polluted file in place for the next run. +With `--rpc.clear-cache`, a batch run deletes the cache file under the sidecar lock, starts from an empty cache, and persists on exit — the same sequence as single-transaction mode. +A clear that fails locally — the sidecar lock cannot be acquired, or the file cannot be unlinked — is an execution-class failure (exit `1`), not an RPC failure: retrying or switching the endpoint cannot fix the local filesystem. +An explicit `--rpc.no-cache-file` still wins over both flags. + +### Concurrent cache-dir sharing + +Multiple `mega-evme` processes may share the same `--rpc.cache-dir` safely. +On clean-exit persist, each process: + +1. Takes an exclusive advisory lock on a sidecar file next to the cache (`rpc-cache-{chain_id}.json.lock`). +2. Re-reads the on-disk cache (a sibling process may have written since this process loaded). +3. Merges its in-memory entries over the on-disk ones (same key → this process's value wins). +4. Writes the result via a temp file and atomic rename, then releases the lock. + +The lock sidecar is left in place after the process exits; only the flock is released when the handle closes. +Lock contention blocks for a short critical section rather than failing the finished run. +If the lock cannot be acquired at all (for example the directory is not writable), persist fails closed and writes nothing. +For the provider cache that means the file is left untouched and a warning names it — the cache is a best-effort artifact, so the cost is a re-fetch on the next run. +An unlocked write is not offered as a fallback: it is exactly the lost-update race the lock exists to prevent, and it would delete a sibling process's entries silently. +A missing or corrupt on-disk file during the re-read degrades to writing this process's entries only (also warned). + +Capture envelopes (`--rpc.capture-file`) use the same lock + re-read-merge path, with additional hard-error checks before writing. +A capture that cannot take the lock fails the run rather than writing unlocked: the envelope is the primary output of capture mode, so losing a concurrent writer's entries is worse than reporting the failure. +The checks are: + +- The on-disk envelope `version` and `chain_id` must match this process's capture. +- `external_env` uses optimistic concurrency against the snapshot observed when this process opened the capture file: + - If the locked on-disk snapshot is absent, or still equals the load-time snapshot, the caller's intentional update wins (so a sequential refresh with `--bucket-capacity` on an existing capture is accepted). + - A run whose snapshot still equals the load-time one has not changed anything, whether it omitted `--bucket-capacity` or passed the values already in force: if another writer refreshed the on-disk snapshot meanwhile, that refresh is kept and this run's cache entries still merge. + Re-asserting the current values is therefore not a way to defend them against a concurrent refresh. + - Persist hard-errors only when this process changed the snapshot **and** the on-disk snapshot also changed since load, to a different value (true concurrent conflict). + The error names all three values: loaded, ours, and on-disk. + - Snapshots are canonicalized before comparison and write (deduplicate by bucket id with last-wins, then sort by id), so two workers with the same effective capacities in different CLI order do not conflict. + - One-sided snapshots still merge: this process's snapshot is kept when set, otherwise the on-disk snapshot is propagated. +- Offline [`cache merge`](../commands/cache.md) still rejects non-identical non-null `external_env` snapshots across inputs (no load-time baseline to compare against). + +A corrupt or unreadable on-disk envelope during re-read degrades to writing this process's entries only (warned), while identity/schema failures remain hard errors. + +To consolidate historical per-worker cache directories offline, use [`cache merge`](../commands/cache.md). +Provider-cache merge also rejects inputs (and `--output`) whose `rpc-cache-{chain_id}.json` filenames disagree on chain id. +`cache merge` follows the same lock protocol for its `--output`: it takes the output's sidecar lock, folds whatever the file holds at that moment into the union, writes, and releases — so merging into a file a live run is still persisting to loses neither side's entries. + ### Cache Flags -| Flag | Type | Default | Description | -| ------------------------ | ----- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ | -| `--rpc.cache-size ` | `u32` | `10000` | Maximum number of items in the in-memory RPC LRU cache. Set to `0` to disable the cache layer entirely. | -| `--rpc.cache-dir ` | path | Platform cache dir | Directory for per-chain cache files. Each chain's cache is stored as `{cache_dir}/rpc-cache-{chain_id}.json`. | -| `--rpc.no-cache-file` | flag | `false` | Disable on-disk cache persistence. The in-memory LRU cache still applies — use `--rpc.cache-size 0` to disable that too. | -| `--rpc.clear-cache` | flag | `false` | Delete the current chain's cache file before loading it. Recovery path for a polluted or corrupt cache. | +| Flag | Type | Default | Description | +| ----------------------------- | ----- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--rpc.cache-max-entries ` | `u32` | `0` | Maximum number of items in the in-memory RPC LRU cache (and therefore what is persisted to the cache file). `0` = effectively unlimited (caps at 1,048,576 entries; the cache index is preallocated proportional to the cap). Default. | +| `--rpc.cache-dir ` | path | Platform cache dir | Directory for per-chain cache files. Each chain's cache is stored as `{cache_dir}/rpc-cache-{chain_id}.json`. Batch replay uses the on-disk cache only when this flag or `--rpc.clear-cache` is passed explicitly. | +| `--rpc.no-cache-file` | flag | `false` | Disable on-disk cache persistence. The in-memory LRU cache still applies. Wins over `--rpc.clear-cache`. Already the default for batch replay unless `--rpc.cache-dir` or `--rpc.clear-cache` is passed. | +| `--rpc.clear-cache` | flag | `false` | Delete the current chain's cache file before loading it. Recovery path for a polluted or corrupt cache. Engages the on-disk cache, including in batch replay. No effect alongside `--rpc.no-cache-file`. | + +The in-memory cache layer is always installed on a forked or online run and cannot be turned off; `--rpc.no-cache-file` disables only on-disk persistence. +At the default cap the cache index is preallocated to tens of MiB regardless of how many entries a run actually stores, which is the trade for never re-fetching during a long verification sweep. +Set `--rpc.cache-max-entries` to a smaller value to reduce that footprint. + +#### Removed Flags + +| Removed | Replacement | Note | +| ---------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--rpc.cache-size ` | `--rpc.cache-max-entries ` | `N > 0` carries over unchanged. `0` inverted meaning — it used to disable the cache, and now means "effectively unlimited" — so the old flag is rejected rather than aliased, and a script passing it fails loudly instead of silently doing the opposite of what it asked. | ### Retry Flags -| Flag | Type | Default | Description | -| ------------------------- | ----- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ | -| `--rpc.max-retries ` | `u32` | `5` | Maximum retry attempts for failing RPC requests. Retries on HTTP 429/503, rate-limit errors, and transport failures. `0` to disable. | -| `--rpc.backoff-ms ` | `u64` | `1000` | Fixed sleep duration in milliseconds between retry attempts (no exponential backoff). | -| `--rpc.rate-limit ` | `u64` | `660` | Compute units per second budget for the retry layer's rate-limit accounting. | +| Flag | Type | Default | Description | +| --------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--rpc.max-retries ` | `u32` | `5` | Maximum retry attempts for failing RPC requests. Retries on HTTP 429/503, rate-limit errors, and transport failures. `0` to disable. | +| `--rpc.backoff-ms ` | `u64` | `1000` | Fixed sleep duration in milliseconds between retry attempts (no exponential backoff). | +| `--rpc.cu-per-sec ` | `u64` | `660` | Compute-unit budget (CU/s) for the retry layer's rate-limit accounting — not requests per second. Alias: `--rpc.rate-limit`. Values below 100 with retries enabled emit a warning. | +| `--rpc.request-timeout ` | `u64` | `30` | Total per-HTTP-request timeout in seconds (connect + response). `0` disables. A hung endpoint then surfaces as a retryable transport error instead of hanging the process. | ### Examples diff --git a/docs/mega-evme/overview.md b/docs/mega-evme/overview.md index 51207926..f37f0460 100644 --- a/docs/mega-evme/overview.md +++ b/docs/mega-evme/overview.md @@ -21,6 +21,7 @@ cargo build --release -p mega-evme | [`run`](commands/run.md) | Execute arbitrary EVM bytecode directly | | [`tx`](commands/tx.md) | Run a transaction with full transaction context and optional RPC state forking | | [`replay`](commands/replay.md) | Replay an existing on-chain transaction from RPC | +| [`cache`](commands/cache.md) | Offline RPC cache utilities (merge provider caches or capture envelopes) | ## Quick Start @@ -66,6 +67,41 @@ These flags apply to all commands. | `--log.file ` | stderr | `--log-file` | Write logs to a file instead of stderr | | `--log.no-color` | `false` | `--log-no-color` | Disable colored console output | +## Exit codes + +Every command reports its outcome through the same set of exit codes, so a pipeline can branch on the process status without parsing output. + +| Code | Class | Meaning | +| ---- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `0` | success | The command completed; with [`--verify-receipt`](commands/replay.md#receipt-verification), every verification matched. | +| `1` | `execution-error` | Execution or internal error: an EVM or setup failure, bad input (including a usage error), or a definitive negative answer such as an unknown transaction. | +| `2` | `verification-mismatch` | The run completed, but at least one replay did not reproduce its on-chain receipt. | +| `3` | `rpc-failure` | An RPC or transport call failed — endpoint unreachable, transport error, or an offline replay file that holds no response for a request the run had to make. | + +Codes `1` and `3` separate the two ways a question can go wrong: `1` means the tool answered, and the answer is negative; `3` means the question went unanswered, so retrying against a healthy endpoint may still produce a result. +A state read that fails while the EVM is executing — an offline replay file without the response, or an endpoint that dies mid-transaction — belongs to `3` as well, even though it surfaces as a block execution error. +A hash the endpoint itself listed in a block body but then resolves to null belongs to `3` too: the null contradicts an answer the endpoint already gave, so it describes an inconsistent endpoint (a reorg, or a load-balanced backend serving divergent views) rather than an unknown transaction. +A served transaction that fails authentication belongs to `3` for the same reason: replay recomputes each fetched transaction's hash from its encoding and re-derives its sender from its signature, and an answer whose body does not match the requested hash — or whose `from` does not match its own signature — is the endpoint serving an inconsistent object, not an answer about the requested transaction. +A block the endpoint itself resolved — the inclusion block of a mined target, its parent, or the reported latest height for a pending one — that then comes back null is the same self-contradiction and belongs to `3`. +Only a hash or block height the caller supplied directly stays in `1` when it resolves to null, since nothing the endpoint served claimed it existed. +Two paths cannot be classified that way: a read that fails inside the pre-block system calls (EIP-4788 beacon root, EIP-2935 block hashes) or inside the sandboxed execution of the keyless-deploy system contract has its cause rendered into a message by the layer that raises it, so `mega-evme` cannot tell it from an execution failure and reports `1`. + +A batch run (`--tx-file` / `--block`) reports every target on its own line and then exits once for the run as a whole, ranking the failure classes it saw: any execution or internal failure exits `1`, otherwise any RPC failure exits `3`, otherwise any verification mismatch exits `2`. +A target that never replayed was also never verified, which is why an infrastructure failure outranks a mismatch. + +On failure the run also prints a report: one `error: ` line per failure on stderr, plus — with `--json` — a structured object as the last line of stdout, so a machine-readable run never ends with empty output. +A run reports more than one line when a secondary failure must not go unnoticed but does not own the exit code — an unwritable [`--rpc.capture-file`](commands/replay.md#--rpccapture-file-path) behind an earlier replay failure, for instance. +The structured object always carries the failure the exit code came from. + +```json +{ "error": { "code": 3, "kind": "rpc-failure", "message": "RPC error: …" } } +``` + +In batch mode that object follows the per-target lines, whose own `error.kind` (`not_found`, `pending`, `rpc`, `execution`) describes why one target failed and is independent of the run-level class above. +A usage error is reported the same way: the argument parser prints its own report and usage block on stderr, and a `--json` run still ends its stdout with the object, whose `message` is a one-line summary of the parse failure. + +New failure classes are added as new codes; the meaning of an existing code does not change. + ## Read more - **[Cookbook](cookbook.md)** — Real-world recipes and worked examples.