Skip to content

fix: comprehensive audit remediation across runtime, snapshot, render, and MCP - #226

Merged
kuny0707 merged 52 commits into
tronprotocol:developfrom
warku123:fix/p0-audit-findings
Aug 31, 2026
Merged

kuny0707 merged 52 commits into
tronprotocol:developfrom
warku123:fix/p0-audit-findings

Conversation

@warku123

@warku123 warku123 commented Aug 24, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes correctness and safety issues found by a full codebase audit, and converges duplicated helpers into single sources of truth.

Correctness

  • Actually swap the node artifact on upgrade/rollback; rollback restores the preserved backup
  • Verify every node during rolling network upgrades; auto-roll-back nodes that fail
  • Stage snapshot downloads and publish atomically; refuse downloads into a running node's data dir
  • Advance the replay cursor only after a block fully replays
  • Probe jar/SSH nodes through the target tunnel; endpoints report the real host

Safety & security

  • Make state saves atomic and surface persistence errors
  • Fail loudly on unparseable memory values; cap JVM heap at half of container memory below 8 GB
  • Fix port wiring and in-container port conflicts; strict YAML parsing; compose env escaping
  • Write JAR configs with private permissions; redact witness keys from MCP resources

Contract alignment

  • Align plan/apply behavior across CLI and MCP
  • Return a structured not-found error instead of panicking on empty inspect selection
  • Pass the node's recorded network into diagnose/heal checks
  • Hash the full rendered config for apply, with intent defaults applied first
  • Split network-upgrade env signaling so backup preserve/restore closes end-to-end

Deduplication

  • Converge endpoint/port helpers, state loading, target resolution, env-var resolution, config diffing, path helpers, and live-config reading — each now has one implementation instead of four to six copies

Why are these changes required?

The audit found defects that break the upgrade/rollback lifecycle, snapshot downloads that could clobber live chain data, silently lost state writes, witness keys leaking into MCP resources, and behavioral drift between CLI and MCP. The copied helper families meant fixes had to be applied in several places and were already drifting apart.

This PR has been tested by:

  • Unit Tests
  • Manual Testing

Build and vet are clean; the full test suite passes. Trial merges against the open shadow-fork PR and current develop show no conflicts.

Follow up

Lower-priority audit items are scheduled for a later wave, plus documentation of the jar upgrade contract and rollback jar-url semantics.

Extra details

Every fix round was independently reviewed before commit.


Correction-chain note (#225 → this batch)

network destroy's own runtime error message used to claim "state cleaned up regardless" while the implementation actually keeps failed entries in tracked state (retryable). The docs corrections in upstream PR #225 were written on top of that lying message and inherited its false claims ("do not retry", "a second run returns NETWORK_NOT_FOUND", "network create stops at the first failed node"). This batch fixes the root cause — the runtime message itself (7f2fcf4) — and realigns AGENTS.md/CHANGELOG to actual behavior (verified by code archaeology back to the initial commit).

Documentation refresh included

First wave of a full-repo documentation audit (council + oracle reviewed, zero behavior changes):

  • AGENTS.md: truthful partial-state semantics for network destroy / network create; wait example fixed to real --json-path/--json-gt flags; versioning policy aligned with internal/schema/embed.go (additive = patch)
  • Output schemas: declare status.monitoring and inspect.intent_hash as already emitted (additive, SchemaVersion 1.16.1) + synthetic contract assertions
  • MCP serverInstructions: exactly the 23 registered tools; prompt chains match their bodies
  • Broken-by-copy fixes: knowledge/cloud-deployment phantom --target flag → intent target block (mirror synced), examples/dev-local.yaml prune flags, README verify usage

Note: a develop merge (#222#225) was cross-checked against this batch — no semantic conflicts remain.

Scope and completion status

This PR now contains the completed audit-remediation scope for the current branch:

  • correctness and safety fixes across lifecycle, target, runtime, state, monitoring, and private-network paths
  • convergence of duplicated CLI/MCP helpers and shared sources of truth
  • machine-contract corrections for emitted output, including versioned intent hashes and monitoring status
  • truthful partial-failure behavior for network destroy and network create
  • documentation and operator guidance corrections where the documented commands or behavior diverged from the implementation
  • regression coverage for the corrected output-schema contracts and agent-facing command guidance

The branch was also checked against the changes merged from develop (#222#225). No unresolved semantic conflicts remain. The final pushed head is 38fcad6 and the PR currently has 14 successful checks, including unit/coverage, e2e, equivalence, schema validation, cross-compilation, vulnerability scanning, lint, and release-artifact validation.

Deliberately deferred follow-up

The broader documentation-maintenance work was audited but is intentionally not part of this PR: historical annotations for specs/, a full AGENTS.md restructuring, a standalone docs/ARCHITECTURE.md, additional README/schema/recipe drift linting, and TODO/OpenSpec governance cleanup. Those items are documentation-governance follow-ups, not prerequisites for the correctness and contract fixes delivered here.

The tiered heap table returned -Xmx2g for any resources.memory < 8GB,
which equals the container limit verbatim (compose.go) — with the image's
forced ZGC (Xms=Xmx) there is zero headroom and the JVM crash-loops with
'Failed to commit memory'. Below 8GB now cap -Xmx at floor(total/2) in MB
(2GB->1024m, 3GB->1536m, 4GB->2g). jvm.heap_max still bypasses verbatim.
Also fix the 1GB edge where -Xmn equaled -Xmx leaving zero old gen
(512m heap now gets 128m new). Found in TX-167 functional testing.
…BFT)

ports.http: 8091 rendered fullNodePort = solidityPort = 8091 because
solidityPort defaulted to 8091 independently of ports.http — the shipped
examples/private-network.yaml crash-looped its fullnode on bind conflict
(HttpApiOnSolidityService, then HttpApiOnPBFTService once that was moved).

RenderHOCON now bumps solidityPort/PBFTPort by +2 when they equal
fullNodePort, with a >65535 overflow guard. Explicitness is judged by
config_overrides only: ApplyDefaults pre-fills ports.solidity_http before
render, so checking the intent field made the avoidance dead code in the
real pipeline (caught in review, with an intent.Parse end-to-end
regression test to pin it). Bumps only de-conflict the three node.http
keys; cross-family clashes with grpc/p2p/jsonrpc/metrics are out of scope
(documented in the function comment).

The example also sets ports.solidity_http: 8093 explicitly. Found in
TX-167 functional testing.
…endpoints

verify/health/diagnose/wait and MCP health probed ssh nodes either via
t.Exec('curl') — rejected by the ssh command allowlist — or by dialing
127.0.0.1 from the trond host, bypassing ssh entirely. On a firewalled
target (22 only) every liveness/sync probe failed while the node was
healthy; status/apply/inspect also reported endpoints as 127.0.0.1.

internal/target now offers a target-aware transport: an optional Dialer
interface (SSHTarget.DialContext = direct-tcpip over the existing ssh
client, restricted to loopback addrs; LocalTarget = net.Dialer), plus
HTTPClient/Get/Post/DialContext helpers with an enforceable timeout
(preemptable even when the underlying dial ignores ctx). Probes migrated:
verify, health, diagnose sync/peers/version/ports, apply LiveStatus jar
branch, wait --port/--http, MCP health. Endpoint reporting (status, apply
created/updated/no_change, inspect, network add/create, MCP status/
endpoints/monitoring, healthTool display) now uses EndpointHost so ssh
rigs report the target host. preflight checkPorts now dials through the
target too — it previously checked the local machine for ssh targets.

Found in TX-167 functional testing.
upgrade/rollback previously only rewrote state.version and restarted the
node, so the 'upgraded' node kept running the old image/JAR.

- runtime: add optional ArtifactUpgrader interface + UpgradeOpts
- docker: pull new image, rewrite compose image tag, up -d (recreate);
  pull/recreate failure leaves old compose untouched for recovery
- jar: download + SHA256-verify to temp file, atomic install, restart;
  requires --jar-url (jar rollback needs the OLD version's URL)
- cmd: --jar-url/--jar-sha256 flags; state version committed only after
  artifact switch + start succeed; failure restores old state
- jar: fix panic when download SHA256 output is empty
AUD-010: snapshot stop no longer signals a recorded PID blindly. On Linux
it reads /proc/<pid>/cmdline, on macOS it falls back to ps, and only
sends SIGTERM/SIGKILL when the process is a trond binary running
'snapshot download'. Mismatches fail closed with
PROCESS_IDENTITY_MISMATCH; dead PIDs keep the existing stale manifest
cleanup. Closes the PID-reuse accidental-kill risk.
AUD-012: new untracked files in the source tree (new .go files, patch
inputs) previously did not affect the dirty hash, so the cache key could
match while build inputs differed. Dirty hashing now lists untracked
files via 'git status --porcelain=v1 -z --untracked-files=all', sorts
them by path, and streams each file's index, size, path and content
into the SHA-256. Ignored files are excluded and the no-untracked case
is byte-identical to the previous behavior.
AUD-014: ParseMemoryGB now returns (int, error) instead of silently
returning 0 for MB forms, decimals and malformed input (which callers
then fell back to 16GB). It accepts integer GB ('8GB'/'8g'/'8'), MB
('1024m'/'1024MB') and fractional GB ('2.5GB'), rounding up to whole
GB, and rejects empty, malformed, negative, zero and unknown-unit
values. All callers (apply, preflight memory checks, config render,
explain-intent, network add, MCP config render) now surface a
validation error instead of silently deploying with 16GB.
… data

AUD-011: snapshot download could extract ~50GB into the live chain DB
directory of a running (or error-state) managed node.

Guard before source resolution/preflight/detach:
- record storage_root in state at apply / network add time
  (bind-mount aware: absolute, ./-relative anchored at the compose
  project dir, named volumes unresolvable)
- candidates: storage root, root/database, root/output-directory{,/database},
  plus parent dir for the conventional output-directory layout;
  symlink-canonicalized comparison with missing-path ancestor fallback
- jar nodes: install_path root; docker nodes skip the /opt/tron default
- refuse for nodes in running or error status (container may still hold
  the LevelDB lock after a failed restart)
- legacy states without storage_root: no_change apply backfills the
  field; guard falls back to parsing the node's docker-compose.yaml
  bind source (/ and ./ prefixed) with an actionable stderr warning
- docs: NODE_RUNNING semantics in --help and AGENTS.md
- AUD-019: jar download on ssh targets fetches locally then uploads
  via PutFile + atomic remote mv (whitelist no longer blocks curl);
  10min client, Content-Length + 512MiB hard limit, SHA256 verified
  before upload, remote temp cleaned on failure
- AUD-020: HTTPClient transports are reused — package-shared for
  LocalTarget, per-instance cache for ssh with Close() deregistration
  and a global CloseIdleConnections
- AUD-021: ssh Connect honors context via DialContext + NewClientConn;
  cancelled handshakes drain and close late clients
AUD-016: SaveState failures were silently swallowed by lifecycle
commands and MCP auto-heal; they now return STATE_ERROR (or mark the
heal item failed) so callers learn the state was not persisted.

AUD-017: Save writes to a unique temp file, fsyncs, and renames
atomically (0600), so an interrupted write cannot leave truncated
JSON behind. Concurrent writers without the state lock may still
lost-update (tracked under AUD-015); the on-disk file itself always
stays a complete, valid document.
# Conflicts:
#	cmd/preflight.go
#	cmd/preflight_test.go
#	cmd/start.go
#	internal/apply/apply.go
#	internal/mcp/conf_helpers.go
#	internal/mcp/resources.go
#	internal/render/jvm_test.go
#	internal/state/store.go
#	internal/target/ssh.go
#	internal/target/target.go
#	internal/target/target_test.go
…v maps

AUD-029: extract into an in-destination staging dir, verify digests before
publish, and swap via rename with a same-filesystem backup so a failed
publish restores the previous chain data (restore failure retains the
backup and reports both errors).

AUD-030: merge host/docker build env through explicit maps so intent env
wins deterministically over host defaults regardless of iteration order.

AUD-037: stop creating destination directories during preflight; checks
stay read-only and creation happens in the download itself.
… port persistence

AUD-031: network create aggregates per-node failures into DEPLOY_ERROR
AUD-032: add skips per-node monitoring (SkipMonitoring) and reloads
  network monitoring from the correct target host
AUD-033: txgen broadcast producer honors ctx cancellation while blocked
AUD-035: network upgrade child stdout/stderr captured (1MiB bound),
  envelope shape validated, no --auto-approve passthrough
AUD-036: config render writes per-node dirs for multi-node intents
AUD-038: destroy resolves runtime per node (jar/docker), cleans up per
  distinct target identity, dedupes target-level failures
AUD-043: intent hash v2 (domain-separated) with legacy raw-hash
  migration gate; recorded monitoring state participates in legacy
  equivalence both directions; auto_ports restores all seven persisted
  ports from state
follow-up: rollback no longer passes target-version --jar-url
AUD-039/040/041/044 plus duplication cleanup:
- internal/apply: EffectiveIntentHash, LegacyIntentHashMatches,
  RestoreAutoPorts, Plan, FindTemplatesDir become the single source
  for intent hash + plan decisions (CLI plan/apply and MCP plan/apply
  all call them; local algorithm copies removed)
- MCP plan returns the promised contract fields with the three-state
  current_state domain and config/version downtime rules
- MCP apply preserves the recorded legacy hash after bypassing
  HUMAN_REQUIRED and passes the resolved template dir to both Plan
  and Apply
- heal routes by recorded node runtime (docker vs jar) and
  diagnose/heal checkers receive the recorded network
- snapshot download and render tools return VALIDATION_ERROR
  envelopes for missing dest / negative node index
- tests exercise real handlers (no self-asserting tautologies)
…dening

AUD-018: logs -f streams over target.StreamExec (local/ssh/docker/jar) with
merged stdout/stderr, exit-error propagation, and Exec fallback when the
stream cannot start. Cancellation, proactive Close, and natural EOF are all
race-free: a background goroutine owns the single session.Wait, a streamDone
signal distinguishes natural EOF from proactive Close, and termination is
shared via sync.Once (SIGTERM then force-close; local kills the process).

AUD-042: bootstrap enforces the private-network gate before resolving the
target. AUD-045: recipe runner no longer panics on nil ProcessState when a
host step fails to start. AUD-046: chaos commands propagate state load
failures as STATE_ERROR instead of proceeding on a nil node. AUD-047: recipe
step IDs must match ^[A-Za-z_][A-Za-z0-9_]*$ in both steps and rollback
sections; built-in recipes and the template doc switched to underscores.

auto-heal now reports result=no_action when every check passes and nothing
was changed, instead of implying success.
… escaping, overlay field merge (AUD-022/023/024/026/027/028)

- hocon: solidity_port conflict detection incl. solidity<->pbft; validate via real render path (AUD-022/023)
- hocon: replaceSolidityGRPCPort block-scoped key match, activate commented template lines (AUD-024)
- compose: escape $ in extra_env values against Compose interpolation (AUD-027)
- intent: yaml KnownFields(true) rejects unknown fields (AUD-026); overlay target field-level merge incl. auto_ports (AUD-028)
- compose: document solidity/pbft ports intentionally unpublished on host
- cli-contract.md: implementation contracts section — single intent-hash source
  (internal/apply EffectiveIntentHash/LegacyIntentHashMatches/RestoreAutoPorts),
  atomic state persistence, artifact upgrade transaction, target transport
  lifecycle, read/write state lock split, witness redaction contract
- spec.md: correct upgrade wording (no built-in health verify; recipe provides
  it) and scope the check-then-act recovery promise to apply only
- plan.md: same scoping note on the apply step
…test hash expectation

Consolidation pass A (post-audit hardening): fold the six isomorphic
persist*State blocks into a single persistNodeState helper, label legacy
compatibility paths (v1 hash fallback, compose bind parse) with removal
notes, and harden test diagnostics (nil-safe extractText, pipeline-derived
ConfigHash expectation with template marker assertion).
Domain 1 of the pre-existing-duplication consolidation: gather the
scattered port fallback (8090/50051), endpoint URL, and probe URL
assembly into internal/apply/endpoints.go (PortOrDefault, HTTPURL,
GRPCAddr, ProbeURL) and rewire CLI, diagnosis, and MCP call sites to
it. Pure refactor: no behavior change, no existing test modified
(oracle-reviewed).
…ronprotocol#5/tronprotocol#6)

Add state.Load/state.LoadNode and target.FromIntent/target.FromManagedNode
as the single implementations; MCP tools and cmd/resolve.go drop their
inlined NewStore/Load/GetNode and mcpResolveTargetFromNode copies.
resources.go and doctorStateFile keep hand-written two-stage error
handling to preserve the external error contract. No behavior change;
existing tests unchanged.
…rs (tronprotocol#11)

- render.DiffLines/DiffText: single positional HOCON diff core with
  whole-slice witness redaction; replaces simpleDiff, simpleHOCONDiff,
  lineDiff, mcpLineDiff (output contracts preserved per caller).
- apply.ResolveEnvVars: single legacy witness-key env resolver;
  replaces duplicated copies in cmd/apply.go and mcp tools_lifecycle.
- Drop thin shells: cmd findTemplatesDir (call apply.FindTemplatesDir
  directly) and mcp redactConfText (call render.RedactWitnessLines).
- Tests updated mechanically at call sites only; assertions unchanged.
…ainer-path copies (tronprotocol#8/tronprotocol#9)

- Delete findTemplatesDir (cmd/network) and findTemplateDir (cmd/config)
  duplicates; all callers use apply.FindTemplatesDir (config now requires
  main_net_config.conf before accepting a template dir, falling back to
  embedded templates otherwise).
- state.NewStore("")/security.NewAuditLog("") resolve defaults via
  internal/paths (honors TROND_STATE_DIR/SetBaseDir; .trond fallback
  when HOME is unavailable).
- Export render.ContainerWorkdir/ContainerDataDir/ContainerConfDir/
  ContainerLogPath and converge the /java-tron literal copies in
  snapshot clone, verify-config, mcp conf helpers, runtime docker logs,
  and apply probe.
- Add paths.DeploymentConfig(name) and use it for the identical
  <deployments>/<name>/<name>.conf joins in plan and config diff.
No test changes.
…fig (tronprotocol#13/tronprotocol#14)

Single jar/docker live-conf reader in internal/target; CLI verify-config
and MCP conf helpers drop their local copies. Error text contracts
preserved per caller (read <path>: / read jar conf: / docker exec cat:).
No test changes.
- C1: gate apply hash on full rendered config, not intent bytes
- C2: refuse snapshot download into running/error node data dirs
- C3: split network-upgrade env signals (TROND_PRESERVE_BACKUP for jar
  upgrade child, TROND_NETWORK_UPGRADE for jar rollback child) so the
  backup preserve/restore lifecycle closes end-to-end
- C4: default unset optional intent fields before hashing/planning
- C5: resolve nodes by name across runtimes in start/rollback paths

Oracle-reviewed (B1/B2 + R1 rounds); targeted tests and gofmt pass.
…UD-044)

CLI diagnose and auto-heal built diagnosis.CheckOpts without Network,
so PeersChecker (private >=1 peer vs >=3) and DiskChecker (mainnet
thresholds) applied wrong thresholds on the CLI paths. MCP paths
already passed node.Network.

Plumb nc.Node.Network through both constructors, add package-var
checker injection seams, and cover both command paths with
recorded-network propagation tests. Oracle reviewed: APPROVE.
- goimports grouping, errcheck/unconvert/unused/gocritic/ineffassign
  cleanup across cmd/internal/tools
- check nodeIntent error in network upgrade verifyNode
- fix vacuous hash-stability assertion in apply_hash_test (SA4000)
- sync apply/events output schemas with versioned intent_hash output;
  allow legacy bare-hex during migration; bump schema to 1.15.1
- render recipe step IDs as hyphenated dry-run labels; keep closeTargets
  after docker-network teardown in network destroy
- add make verify (lint+test+e2e) and verify-fast (lint+test) gates
- move hermetic tests (recipe, agentsmd, schema manifest/conformance)
  out of the e2e build tag into the default suite
- split docker-only helpers into e2e_docker_helpers_test.go
- add TestOutputSchemaContracts: validate apply/events output against
  embedded schemas, incl. v2: and legacy bare-hex intent hashes
…ntime

A node with a jar: block and no explicit image had the default
tronprotocol/java-tron image injected by ApplyDefaults, then got
rejected by apply's mutual-exclusion check while config validate
(pre-defaults) passed. Skip the injection when Jar is set, and
reject jar + effective docker runtime up front in Parse,
LoadWithOverlay, and apply (defense in depth) instead of silently
deploying the official image.
…e empty

Redeploying a jar node from non-empty to empty EnvVars left the old
/etc/systemd/system/<unit>.service.d/env.conf in place, so previous
environment (potentially including witness key material) kept applying
to the service. Delete the drop-in when EnvVars is empty, mark the
change so the service restarts, and cover redeploy/remove cleanup with
regression tests.

Also: document the jar-intent default-image fix (fde40dd) and AUD-044
recorded-network gating in CHANGELOG, tighten the runtime enum test
(explicit empty accepted, podman/systemd rejected), and gitignore the
local .regression-evidence/ archive directory.
EffectiveIntentHash returns v2:<hex> since the versioned-hash series, but
status.schema.json still pinned intent_hash to ^[0-9a-f]{64}$, so real
status output failed schema validation (council R2 finding; status was
uncovered by both the contract test and the docker-gated conformance e2e).

- Loosen pattern to ^(v2:)?[0-9a-f]{64}$ (bare hex stays valid)
- Add status cases to the output-schema contract test (v2 pass, bare
  pass, malformed v2 rejected)
- Bump SchemaVersion 1.15.1 -> 1.16.0 (additive) and refresh baseline
- CHANGELOG entry; TODOS records council R2 leftover items
CI golangci-lint (gofmt) flagged the missing blank comment line between
the 1.15.1 and 1.16.0 version-history blocks.
Five files moved out of the e2e build tag in 3d8943b still carried
misleading *_e2e_test.go names despite running in the default suite:
agentsmd, recipe, recipe_matrix, schema_conformance, schema_manifest.

Rename them (drop the _e2e infix) and add a short header note stating
why they are untagged, pointing to agentsmd_test.go for the full
rationale and hermeticity constraints.
…semantics

The runtime error claimed "state cleaned up regardless" while the
implementation keeps failed entries in tracked state and a re-run
retries exactly those nodes. That lie misled PR tronprotocol#225's docs correction
into encoding it in AGENTS.md; fixing the message at the source closes
the chain (root cause of tronprotocol#226 doc findings).
…tions

First wave of the full-repo documentation audit (council + oracle
reviewed); zero behavior changes.

- AGENTS.md: truthful partial-state semantics for network destroy and
  create per code archaeology (create aggregates failures after trying
  every node; upgrade is genuinely first-failure-stop), wait example
  fixed to the real --json-path/--json-gt flags, versioning policy
  aligned with internal/schema/embed.go (additive = patch)
- schemas/output + embed copies: declare status.monitoring and
  inspect.intent_hash as already emitted (additive, SchemaVersion
  1.16.1); version baseline regenerated; synthetic contract assertions
  added in cmd/output_schema_contract_test.go
- internal/mcp/server.go: serverInstructions now list exactly the 23
  registered tools and prompt chains match their bodies
- knowledge/cloud-deployment.md (+ embedded mirror): drop the
  nonexistent --target flag, SSH setup via the intent target block
- examples/dev-local.yaml / README.md: real build prune flags, verify
  usage with required --intent
- CHANGELOG: user-visible fixes plus tronprotocol#225->tronprotocol#226 correction-chain
  provenance
@SeriousCoding789

Copy link
Copy Markdown

Reviewed the full delta against develop (merge base 0c7451c). The audit-remediation direction looks right, and the big pieces read correctly: transactional artifact swap on upgrade/rollback, snapshot staging with atomic publish, atomic state saves with propagated errors, ParseMemoryGB failing loudly instead of falling back to 16GB, and the txgen
producer/consumer cancellation fix.

Two things I checked specifically and can confirm are fine:

  • verifyNode's yaml.Marshal(projected) → temp file → child verifyintent.Load round trip survives KnownFields(true). I ran it; no unknown-field breakage.
  • Dropping --auto-approve from the upgrade child is a real fix, not a regression —upgrade never registered that flag, so the old call always failed with "unknown flag".

Three things I think need resolving before merge; details inline:

  1. SetProvisioning is deleted but still relied on by cmd/bootstrap.go — AUD-005 regresses, and bootstrap over SSH now hard-fails rather than silently no-opping.
  2. cleanupNetworkBackup shells out via sh, which is not on the SSH allowlist — a fully successful multi-node jar upgrade over SSH reports UPGRADE_ERROR.
  3. replay now aborts on any single failed broadcast, not just an all-failed block.

Non-blocking observations I can file separately if useful: untracked-file hashing ininternal/build/source.go hard-fails on unreadable entries and untracked directories (nested repos hit io.Copy on a directory FD); the directory fsync at the end of state.Store.Save fails on Windows, which is now a fatal STATE_ERROR; the HTTP transport pool keys on pointer identity and reads sharedLocalTransport outside the sync.Once; LocalTarget.StreamExec terminates unconditionally on Close where the SSH version carefully distinguishes natural EOF; network destroy under-reports failures when a target can't be resolved (siblings appear in neither removed nor failures); and the CHANGELOG has no Breaking section for KnownFields(true), the extra_env $ escaping, or the --monitor semantics flip.

- target/ssh: restore SetProvisioning lost in the 43d4482 conflict
  resolution (AUD-005 regression: provisioning silently disabled while
  bootstrap hard-fails useradd under the tronprotocol#221 allowlist); add a
  compile-time interface assertion so the method cannot vanish silently
- bootstrap: resolve targets through a test seam and cover provisioning
  wiring with a fake-target test (asserts SetProvisioning(true) and
  allowlisted provisioning commands are exercised)
- network upgrade: cleanupNetworkBackup removes the backup via
  tgt.Exec("rm","-f","--",path) on the recorded target instead of a
  'trond exec -- sh -c' string concatenation — sh is not allowlisted for
  jar-over-SSH, and the old form carried quoting/injection risk plus a
  printf-pseudo-JSON envelope; cleanup failures now degrade to warnings
  in the result instead of failing an otherwise successful upgrade
- TODOS: register the nine non-blocking review findings with file:line
  pointers; CHANGELOG: Fixed entries for both blocker fixes
- docs: AGENTS.md documents the new upgrade warnings field; restore the
  SetProvisioning safety-rationale comment
Schema version resolved to 1.16.1: develop's shadow-fork-keygen new
schema (MINOR, tronprotocol#220) and this branch's additive status/inspect fields
(PATCH) both landed at 1.16.0 independently; baseline regenerated as
the union of both sides.
…nLoop)

CI lint flagged a defer-in-loop in cleanupNetworkBackup. The loop now
only locates the matching jar node; target resolution, Close, and the
rm exec happen once, after the loop. Semantics unchanged.
Nine non-blocking findings from Sunny's PR review, all verified TRUE:

- build: tolerate untracked directories, dangling symlinks and
  unreadable files in dirty-hash folding (fold markers, never fail)
- state: directory fsync after rename is best-effort (Windows emits
  errors for dir Sync; rename already persisted)
- target: replace sync.Once with a mutex so CloseIdleConnections no
  longer races sharedLocalTransport initialization
- target: LocalTarget.StreamExec gains streamDone so natural EOF does
  not kill an already-exited process (matches SSH behavior)
- rollback: restore the safety rationale for hard-failing on Stop
- upgrade/rollback: a failed artifact SHA256 probe returns an error
  instead of retaining the stale hash (audit trail + skew comments)
- network destroy: every node on an unresolvable target is reported in
  failures (was: only the first, siblings vanished)
- intent overlay: explicit auto_ports: false now overrides base (raw
  YAML key detection, pointer-free)
- changelog: Breaking Changes section for KnownFields, ParseMemoryGB,
  --monitor precedence (intent_hash v2 documented under Fixed)

Also pins the replay block-atomicity semantics with a regression test
(single partial block failure aborts the block, keeps the cursor and
replays the block on retry - intentional design, not a bug).
@warku123

Copy link
Copy Markdown
Author

@Sunny6889 thanks for the thorough review — every finding is now addressed:

Fixed in a96f0a7 (the two blockers):

  1. SetProvisioning restored on SSHTarget (byte-identical semantics to the develop version), plus a compile-time assertion (var _ interface{ SetProvisioning(bool) } = (*SSHTarget)(nil)) so it can never silently vanish again. Covered by a fake-target test asserting provisioning is enabled and the provisioning allowlist path (apt-get/useradd) succeeds.
  2. cleanupNetworkBackup no longer shells out via sh — it goes through the runtime layer (tgt.Exec with rm -f --, which is on the allowlist, argv shell-quoted so paths with spaces are safe), and cleanup failures now downgrade to warnings in the result instead of failing an otherwise-successful upgrade.

#3 replay — intentional design, kept. The blockFail > 0 condition is the AUD-009 fix itself (7862f5b): the bug it closed was cursors advancing past partially-failed blocks, silently dropping transactions. A per-block configurable threshold would partially reintroduce exactly that loss, and replaying mainnet history against a private chain makes per-tx failures expected — so abort-and-replay-the-block is the safer contract. The semantics have been in place since the tool landed (resume-from-explicit---start documented in the replay README from day one, reset-to-zero refinement landed a day later, CHANGELOG documents "cursors advance only after complete block replay"), and 390a4e1 now pins them with TestRunBroadcastFailureDoesNotAdvanceCursor so block atomicity can't be eroded by accident.

All nine non-blocking observations fixed in 390a4e1: dirty-hash folding tolerates untracked directories/dangling symlinks/unreadable files (folds a marker instead of failing the build), directory fsync after rename is best-effort (Windows), the transport pool uses a mutex instead of sync.Once (race fixed under -race), LocalTarget.StreamExec aligns with the SSH natural-EOF handling, the rollback Stop hard-failure keeps its safety rationale plus a CHANGELOG entry, the artifact SHA256 probe fails fast instead of retaining a stale hash (audit trail updated), network destroy reports every node on an unresolvable target, overlays accept an explicit auto_ports: false, and the CHANGELOG gained a Breaking Changes section. The matching TODO entries are cleared; the only open items tracked now are the AUD-020 transport-pool leak and Windows platform support.

A child upgrade that failed after tx.Activate (artifact sha probe or
state persist) exited non-zero without rolling back, and the parent
only admitted successful children to the rollback set - leaving a
node on the new artifact with state recording the old version.

- child: persist the true running version (Version/PreviousVersion/
  Status) on post-activation failure and report the new optional
  top-level artifact_swapped fact in the error envelope (schema
  1.16.1 -> 1.16.2)
- parent: parse the envelope tolerantly (last JSON line with
  error_code), admit swapped-failed nodes to the rollback set, and
  pass pre-attempt version metadata to the rollback child via env
- rollback: network restore mode proceeds with pre-attempt truth and
  preserves the pre-series PreviousVersion; standalone rollback is
  unchanged
- guard the child env against inherited TROND_NETWORK_UPGRADE_*
  shadowing; warn when the best-effort truth persist fails
- document the envelope field in AGENTS.md and CHANGELOG
Aborting the replay when any transaction broadcast in a block fails is
by design: the cursor must never advance past a partially-failed
block, or silently dropped txs diverge the shadow chain from mainnet.
Document the rationale in TODOS.md, note the behavior in CHANGELOG,
and point the re-broadcast test assertion at the TODOS entry. A
configurable --block-fail-threshold (default 0 = strict) is explicitly
deferred to a follow-up PR.
@warku123

Copy link
Copy Markdown
Author

Both points are addressed — thanks for flagging them.

On ③ replay blockFail > 0: the gap was that the rationale never made it into the repo, not the behavior itself. Aborting on any broadcast failure in a block is intentional — the cursor must never advance past a partially-failed block, or the dropped txs silently diverge the shadow chain from mainnet and the replay loses its point. It's now documented where it belongs:

  • TODOS.md: the rationale, the retry semantics (retry resumes at the failed block boundary; already-landed txs get re-broadcast — that's what the want 4 assertion in replayer_test.go pins), and an explicitly deferred --block-fail-threshold N flag (default 0 = current strict abort)
  • a CHANGELOG entry
  • a comment on the test assertion pointing at the TODOS entry

If a configurable threshold turns out to be needed, it can be a small follow-up PR — this one is big enough already.

On the auto-rollback gap: confirmed, real bug — and 390a4e1 turning the sha probe into a hard failure genuinely widened the window; that one's on us. Fixed end-to-end in 28979fe:

  • the child now persists the true running version (Version/PreviousVersion/Status) when it fails after tx.Activate (sha probe or state persist), and reports a new optional top-level artifact_swapped field in the error envelope (schema 1.16.1 → 1.16.2)
  • the parent parses the child envelope tolerantly (last stderr line containing error_code, so cleanup warnings / OTLP noise can't hide it) and admits swapped-failed nodes to the rollback set before the failure path finishes
  • rollback gained a network-restore mode: it restores the pre-attempt artifact and writes the pre-attempt version truth back to state, preserving PreviousVersion so repeated rollbacks stay honest; standalone manual rollback is unchanged
  • regression tests pin it: swapped failure → node enters the rollback set with pre-attempt env (TARGET=A, PREVIOUS=A0); the first-upgrade scenario ends at (A, ""); full suite green with -count=1

CI is green (14/14) on the pushed head; this should be ready for re-review.

@kuny0707
kuny0707 merged commit 0a1bf8c into tronprotocol:develop Aug 31, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants