Feature/tmosley/scanner upgrades

[tmosleyIII]

Scanner reliability and readability

Surfaces silent scanner failures and makes scan results interpretable by the operators who act on
them. Covers both scanners: the network scanner (why a validator could not be reached) and the
history scanner (why an archive was flagged).

Both halves come from the same failure mode, found in production twice: Radar knew something was
wrong but could not say what, so the diagnosis fell to a human, and in one case the operator was
told to fix something that was not broken.

Motivation: two incidents

A validator became invisible after changing IPs. An operator moved their nodes to new addresses
and asked what Radar needed in order to pick them up. The nodes were healthy and visibly
participating in consensus via SCP, but Radar would not update the address, because it only accepts a
new endpoint after a completed overlay handshake — and the handshake was failing. Radar reported only
that the peer did not connect.

The actual cause was a TLS-terminating reverse proxy in front of port 11625: connecting to the
overlay port produced a TLS handshake instead of stellar-core. Establishing that took a day of manual
work — reproducing from independent validators, capturing terminal output, and writing a diagnostic
report for the operator — to reach a conclusion the scanner already had the bytes to determine.
Nothing in Radar surfaced it.

Healthy archives were reported as corrupt. Separately, Radar was telling archive operators to run
repairs on archives that were provably fine, because the scanner's own hasher could not parse newer
transaction entries and treated its failure as evidence against the archive (§3).

In both cases the information needed to explain the failure existed at the point of failure and was
discarded. This PR keeps it.

---

1. Network scanner: connection failure diagnostics

Connection attempts previously collapsed into a generic failure. A validator unreachable because of
DNS, a closed port, a TLS listener on the overlay port, or a network-passphrase mismatch all looked
identical in the logs, so operators had nothing actionable — the incident above is exactly this.

• Typed outcome taxonomy (packages/crawler/src/connection-attempt.ts) — 14 distinct outcomes
  (dns_failed, tcp_refused, tcp_timeout, tls_detected, http_detected,
  invalid_overlay_frame, overlay_auth_failed, protocol_incompatible, wrong_network,
  unexpected_public_key, …) attributed to a failure stage (dns, tcp, overlay_hello,
  overlay_auth, identity).
• Overlay protocol error classification
  (packages/node-connector/src/connection/connection-protocol-error.ts) — inspects the frame
  header to distinguish OVERLAY_TLS_DETECTED, OVERLAY_HTTP_DETECTED, OVERLAY_WRONG_NETWORK and
  OVERLAY_PROTOCOL_INCOMPATIBLE from a generic parse failure. OVERLAY_TLS_DETECTED is the
  incident above: a reverse proxy answering the overlay port with a TLS handshake is now named at
  the point of failure, rather than requiring a day of manual reproduction to identify.
• Aggregated failure summary (packages/crawler/src/crawl-logger.ts) — each crawl emits one
  Connection failure summary line with counts per outcome, instead of leaving the pattern to be
  reconstructed from scattered lines.
• Overlay probe (packages/crawler/src/overlay-probe.ts) — a single endpoint can be probed
  directly and its outcome classified, so reachability can be checked on demand rather than only as
  a side effect of a full crawl. This is the automated form of the manual reproduction the incident
  required.

2. Network scanner: validator endpoint discovery

Addresses the other half of that incident: a validator whose address changes should not have to wait
on a successful handshake at the old address to become visible again. Adds the machinery behind
"known peers" — candidate endpoints for a validator, how they are discovered, probed, and promoted.

• ValidatorEndpointCandidate domain model with DNS resolution, address policy, probe observations,
  and a candidate manager (apps/backend/src/network-scan/domain/node/endpoint/)
• Persistence plus a schema migration (1785360000000-validator-endpoint-candidates)
• Admin API for inspecting and managing candidates (EndpointCandidateAdminRouter)

3. History scanner: a silent failure that misreported healthy archives

The second incident. Radar was telling archive operators to run repairs on archives that were
provably fine — SDF Validator 1 (GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH) showed
144 × TRANSACTION_SET_HASH errors and a "start repair at ledger …" prompt.

Root cause

The deployed WASM hasher was 0.9.1, built against stellar-xdr 26, which cannot decode
protocol-27 transaction entries. The failure path:

1. Hasher throws on a p27 entry (.unwrap() → wasm panic → opaque RuntimeError: unreachable)
2. The worker error is swallowed into processingErrors; no transaction-set hash is ever calculated
3. verifyTransactions finds the hash missing and silently substitutes the empty-transaction-set
   hash
4. That does not match a ledger which actually had transactions → TRANSACTION_SET_HASH mismatch
   reported against a healthy archive

A scanner-side parse failure was laundered into an accusation of archive corruption. Any future
protocol bump would have done the same thing to every archive Radar scans.

Evidence the archives were never at fault

• rs-stellar-archivist --verify (independent Rust implementation) over the affected range:
  364 files, 0 missing or corrupt
• Differential test linking stellar-xdr 26, 27 and 28 in one binary, over 2,048 real transaction
  entries across 16 checkpoints spanning ledgers 20M–64.08M: hashes identical and correct in
  all three; only v26 fails, and only to parse p27 entries
• Published 0.9.1 and 0.9.2 WASM artifacts tested directly against real archive data

Fixes

Hasher upgraded 0.9.1 → 0.11.0 (^0.11.0 in both apps/history-scanner and apps/backend;
the caret bump is required because ^0.9.x cannot cross a 0.x minor). 0.11.0 also returns a
catchable Error instead of panicking across the wasm boundary, and fixes a latent
Uint8Array::view use-after-free that aliased wasm memory owned by a freed temporary.

Scanner no longer attributes its own faults to the archive:

• New SCANNER_ERROR category (ScanError.ts)
• processingErrors now carry the ledger, recovered from the entry header even when the worker
  fails (CategoryXDRProcessor.ts)
• CategoryVerificationService skips ledgers it could not process rather than comparing a hash
  it never computed — including the knock-on case where an unprocessed ledger breaks the next
  ledger's header-chain check
• Removed a duplicate mapping in CategoryScanner that re-reported every processing error a second
  time at a hardcoded ledger: 0 — the source of both double-counted errors and the nonsensical
  firstLedger: 0 in stored scan records

UI stops prompting repairs for scanner faults: HistoryArchiveScan gains archiveErrors /
scannerErrors accessors; the amber "start repair at ledger N" banner now requires a genuine archive
error, and scanner faults render as a neutral "Archive partially unverified — this is a limitation
of the scanner, not a problem with the archive; no action is needed."

Measured effect

Same ledger range (64081500–64082367) that produced the production errors:

Configuration	Result
hasher 0.9.1 (what production ran)	8 × TRANSACTION_SET_HASH, ledger 0 → 64082311
0.9.1 + scanner fixes	4 × SCANNER_ERROR (correct ledgers), 0 archive errors
0.11.0 + all fixes	errors: []

The middle row is the durable part: even with a hasher a protocol behind, healthy archives are no
longer reported as corrupt.

4. History scanner: throughput and correctness fixes found along the way

• Cross-range bucket de-duplication was broken. RangeScannerAdapter dropped
  scannedBucketHashes when mapping to the IRangeScanner result, so alreadyScannedBucketHashes
  was always empty and every 1M-ledger range re-downloaded and re-hashed buckets it shared with
  earlier ranges. On a full pubnet scan (~64 ranges) the large upper-level snap buckets were
  re-fetched dozens of times.
• Configurable concurrency ceiling. ArchivePerformanceTester had a hardcoded
  [50, 35, 25, 20, 15, 10] ladder. Now derived from HISTORY_MAX_CONCURRENCY (default 100) using
  fixed ratios, so the ladder keeps the same six rungs — no extra benchmark cost — and reproduces the
  historical list exactly at a ceiling of 50.
• libuv thread pool sizing (uv-thread-pool.ts) — bucket verification runs
  gunzip → sha256 on the libuv pool, which defaults to 4 threads. Now sized from the concurrency
  ceiling, applied before anything touches the pool. Measured honestly: this made no difference to
  end-to-end scan time, which is network-bound at ~41 MiB/s. It removes a ceiling rather than
  delivering a speedup.
• verify-single-archive CLI was unrunnable — missing @injectable() and
  @inject(TYPES.ExceptionLogger) meant it crashed on startup before any scanning. Fixed; this is
  the entry point used to reproduce and diagnose archive issues.
• Startup log of resolved uvThreadPoolSize / maxConcurrency, and HISTORY_MAX_CONCURRENCY /
  UV_THREADPOOL_SIZE documented in the history-scanner README.

5. Operator documentation

• network-scan/ENDPOINT_DISCOVERY_OPERATIONS.md (238 lines) — purpose, current implementation,
  configuration, operator workflow, acceptance results, regression coverage, remaining work
• network-scan/VALIDATOR_DISCOVERY_IMPROVEMENT_PLAN.md (743 lines) — objective, success criteria,
  non-goals, design principles, target scan flow, domain model
• network-scan/README.md — component overview and pointers to the operations docs
• history-scanner/README.md — concurrency and thread-pool tuning, and why the two must move
  together

6. Infrastructure

• Terraform / App Platform: CONTACT_RECIPIENT_EMAIL plumbed through the production environment and
  app module, enabling an outbound contact path from the backend.

---

Impact: before and after

Both incidents cost an ecosystem participant real time, and in both cases the scanner already held
the information needed to explain itself.

A validator that changed IP

Before	Operator reports their nodes are not showing the new address. Radar reports only that the peer did not connect — no stage, no cause. Diagnosis requires reproducing the connection from independent validators, capturing raw terminal output, and writing a diagnostic report by hand. ≈1 day, one engineer, one operator blocked. The finding — a TLS-terminating proxy on port 11625 — was determinable from the first bytes on the wire.
After	The frame header is classified at the point of failure as OVERLAY_TLS_DETECTED / outcome tls_detected, attributed to the overlay_hello stage. Every crawl emits a Connection failure summary with counts per outcome, so the same misconfiguration across several validators is visible as a pattern. Any endpoint can be probed on demand via probeOverlayEndpoint instead of waiting for a full crawl.

An archive reported as corrupt

Before	A healthy archive shows 144 × TRANSACTION_SET_HASH and "Archive verification errors detected. Start repair at ledger 64051774." The operator's rational response is a multi-hour repair and cache purge against an archive with nothing wrong with it. The real cause — the scanner's hasher being one protocol behind — surfaced only as an opaque RuntimeError: unreachable, swallowed internally and never shown.
After	A parse failure is reported as SCANNER_ERROR against the ledgers actually affected, and the UI states plainly that this is a scanner limitation and no action is needed. No repair is ever prompted for a fault on Radar's side. With the hasher upgraded the failures do not occur at all.

Measured on the range that produced the production errors (ledgers 64081500–64082367): 8 phantom
archive errors → 0. With the old hasher still in place, 8 → 4 correctly-labelled scanner errors
and 0 archive errors — the guidance to operators is fixed independently of the version bump.

What this changes for ecosystem participants

• Validator operators get a named cause for an unreachable node instead of a silent absence from the
  network view.
• Archive operators are no longer asked to repair archives that are not broken.
• Radar's own faults are labelled as Radar's, so its output can be trusted as a signal about the
  network rather than about the scanner.

---

Testing

• 211 tests pass across history-scanner and shared; tsc --noEmit clean;
  vue-tsc --noEmit clean across the frontend
• New regression tests, each verified to fail against the unfixed code:
  • bucket hashes carried between ranges (Scanner.test.ts)
  • processing errors reported as scanner faults, genuine mismatches still reported, unattributable
    errors ignored, header chain not blamed on an unprocessed predecessor
    (CategoryVerificationService.test.ts)
  • concurrency ladder reproduces the historical values at the old ceiling and never grows past six
    rungs (ArchivePerformanceTester.test.ts)
• End-to-end scans against history.stellar.org/prd/core-live/core_live_001 at each stage, with the
  published 0.11.0 artifact pulled from npm and verified on 256 real entries

Deployment notes

1. Verify the deployed version after shipping. main declared ^0.9.2 from 2026-07-09, yet
   production was still serving 0.9.1 six weeks later. The version bump alone is not sufficient if
   the same thing swallows this deploy.
2. Rescan affected archives. Errors are persisted and will keep displaying until a clean scan
   overwrites them.
3. Hasher 0.11.0 is published to npm and crates.io and tagged v0.11.0.

Follow-up (not in this PR)

• Protocol-lag CI check — fail the build when the hasher's stellar-xdr major trails the
  ledgerVersion seen in recent ledgers. Every safeguard here makes the next occurrence legible;
  this is the one that prevents it. Worth landing before protocol 28 activates.
• Deploy staleness — the six-week gap between a merged version bump and production is a delivery
  problem, not a code problem, and will affect something else.
• StellarArchivistVerifier targets the Go stellar-archivist in both argument order and output
  parsing; the Rust binary exits 2 for every failure, so it cannot be swapped in without also
  reworking the output parsing (its --report JSON would be the robust integration).