Skip to content

fix(platform-wallet): close the asset-lock resume broadcast race - #4636

Open
shumkov wants to merge 4 commits into
v4.2-devfrom
fix/asset-lock-resume-broadcast-race
Open

fix(platform-wallet): close the asset-lock resume broadcast race#4636
shumkov wants to merge 4 commits into
v4.2-devfrom
fix/asset-lock-resume-broadcast-race

Conversation

@shumkov

@shumkov shumkov commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Issue being fixed or feature implemented

Two defects on the asset-lock resume and create paths, both reachable today on v4.2-dev.

The create path could downgrade a concurrently finalized row. After its broadcast, broadcast_funded_asset_lock advanced the row to Broadcast unconditionally. If another flow carried the same lock to InstantSendLocked or ChainLocked while that broadcast was in flight, the advance overwrote the stronger status and persisted Broadcast with no proof.

A contested Broadcast row reported the wrong verdict. When a re-broadcast was rejected and an input conflict had been sighted, the Broadcast arm returned TransactionBroadcastUnconfirmed immediately, so the caller never saw AssetLockInputContested — the Built arm already had the opposite behaviour.

Note on scope. This PR opened as a re-derivation of #4016 ("close the asset-lock resume broadcast race") and originally carried a promote_built_to_broadcast CAS that advanced the row before broadcasting. That turned out to be unnecessary: #4355 landed claim_resume_dispatch on v4.2-dev, whose ResumeDispatchClaim — taken under the read guard at snapshot and read under the write guard in untrack_asset_lock — already prevents a create-path rejection from untracking a row and releasing its reservation while a resume's broadcast is in flight. Running the original regression against the merge-base, every safety assertion passes; only the assertions pinning the old promote-before-broadcast ordering fail. The CAS was therefore dropped and #4355's resume arm left untouched. The two fixes below are what genuinely remained.

What was done?

  • build.rs — the create path's post-broadcast advance now goes through advance_asset_lock_status_if(|s| s == Built, Broadcast, None). A row a concurrent flow already carried past Built is left alone, and the call still returns Ok, since the follow-up proof wait resolves from the stronger status.
  • sync/recovery.rs, Broadcast arm — on a rejected re-broadcast with no local proof, an input_conflict sighting now falls through to the existing bounded wait and input_conflict_verdict rather than returning early, mirroring the Built arm. The no-sighting path is unchanged.
  • Doc pass — comments that described Broadcast as proof the transaction reached the network now say a broadcast was attempted, across rs-platform-wallet, rs-platform-wallet-ffi and both mobile SDKs.

sync/recovery.rs's Built arm, await_broadcast_ready, resume_when_transport_ready, the ResumeDispatchClaim lifecycle and absorbing Consumed are byte-identical to the merge-base.

Tests

Test would have caught this in CI: ✖ before the fix, ✔ after — each proven by reverting one production hunk in isolation.

  • create_broadcast_does_not_downgrade_a_concurrently_finalized_row — revert only the build.rs hunk and it fails with left: Broadcast, right: ChainLocked.
  • a_rejected_rebroadcast_of_a_conflicted_built_lock_reports_the_contested_verdict — extended to a second resume with the row already at Broadcast. Revert only the recovery.rs hunk and it fails in that second-resume match with TransactionBroadcastUnconfirmed; the failure is past the whole first half, so the base's Built-arm assertions still pass and it is precisely the Broadcast-arm extension the base cannot satisfy.

Two tests from the original CAS design were removed because they can no longer fail, not because they now fail: rejected_create_while_resume_broadcasts_keeps_row_and_reservation uniquely pinned the abandoned promote-before-broadcast ordering, and every safety assertion it carried is asserted by the base's a_rejection_cleanup_cannot_release_inputs_under_a_parked_resume; stale_built_resume_does_not_downgrade_a_concurrently_finalized_row depended on a test-only hook that only existed alongside the CAS.

One behaviour worth stating plainly: under #4355's design a Built resume that snapshotted before a concurrent finalization still makes a benign redundant re-send. The network answers already-known, advance_if(Built) leaves the row alone, and the wait resolves from the row's proof. That is accepted base behaviour, not introduced here, and it is why the deleted test's "must not broadcast" half is gone.

cargo test -p platform-wallet and -p platform-wallet-ffi pass; clippy -D warnings and fmt --check clean.

Supersedes #4016, which is closed.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added clearer persistence error categories for loading, storing, and restoring wallet data.
    • Added retryability information for transient failures.
    • Swift users can now access detailed persistence failure reasons.
  • Bug Fixes

    • Improved asset-lock recovery when broadcasts are rejected, transport is unavailable, or finality changes during recovery.
    • Prevented active recovery operations from being cleaned up prematurely, preserving reservations and proofs during concurrent activity.
  • Documentation

    • Clarified asset-lock broadcast and conflict-recovery behavior, including cases where an earlier broadcast may have been attempted.

@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: a4c4d4de-0f52-4035-a304-3c48c1ac77d0

📥 Commits

Reviewing files that changed from the base of the PR and between 7907237 and 751cd3b.

📒 Files selected for processing (6)
  • packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/errors/DashSdkError.kt
  • packages/rs-platform-wallet-ffi/ERROR_CODE_REGISTRY.md
  • packages/rs-platform-wallet-ffi/src/error.rs
  • packages/rs-platform-wallet/src/error.rs
  • packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs
  • packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletResult.swift

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The changes update asset-lock dispatch and recovery handling for concurrent and transport-dependent outcomes. They also add typed persistence errors across Rust, FFI, Kotlin, and Swift, with retry classification, user-facing messages, and preserved diagnostics.

Changes

Asset-lock dispatch and recovery

Layer / File(s) Summary
Resume dispatch claim tracking
packages/rs-platform-wallet/src/wallet/asset_lock/sync/tracking.rs
Adds RAII dispatch claims, conditional status advancement, and cleanup guards for Built rows.
Create broadcast state preservation
packages/rs-platform-wallet/src/wallet/asset_lock/build.rs
Advances rows only when they remain Built and preserves concurrent status and proof updates.
Broadcast resume recovery
packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs, packages/rs-platform-wallet/src/wallet/asset_lock/tracked.rs, packages/rs-platform-wallet/src/error.rs, packages/rs-platform-wallet-ffi/src/error.rs, packages/rs-platform-wallet-ffi/ERROR_CODE_REGISTRY.md, packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/errors/DashSdkError.kt, packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletResult.swift
Separates ready and unavailable transport handling, refreshes conflict and finality state after rejected sends, and updates related asset-lock contracts and tests.

Typed persistence error contracts

Layer / File(s) Summary
Persistence error mapping and SDK presentation
packages/rs-platform-wallet/src/error.rs, packages/rs-platform-wallet-ffi/src/error.rs, packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/errors/DashSdkError.kt, packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletResult.swift
Adds operation-specific persistence errors and FFI codes 49–54. Kotlin and Swift map these codes to retryable or non-retryable errors, user-facing text, and raw failure diagnostics. Tests verify the mappings and code slots.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ResumeAssetLock
  participant TransactionBroadcaster
  participant WalletManager
  ResumeAssetLock->>TransactionBroadcaster: defensive re-broadcast
  TransactionBroadcaster-->>ResumeAssetLock: readiness or rejection result
  ResumeAssetLock->>WalletManager: refresh conflict and local finality
  WalletManager-->>ResumeAssetLock: contested verdict, proof, or lookup error
Loading

Merge Risk: ⚪ Minimal · up to 751cd

The asset-lock recovery and typed persistence error changes have no remaining concrete merge-blocking risk.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 85.71% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 11 files. (1 skipped: 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: fixing the asset-lock resume broadcast race.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/asset-lock-resume-broadcast-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thepastaclaw

thepastaclaw commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

✅ Final review complete — no blockers (commit 751cd3b) · triage: normal · Phase 2 only (queue backlog)

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.00%. Comparing base (5f1e0cc) to head (751cd3b).
⚠️ Report is 22 commits behind head on v4.2-dev.

Additional details and impacted files
@@             Coverage Diff              @@
##           v4.2-dev    #4636      +/-   ##
============================================
- Coverage     86.36%   86.00%   -0.37%     
============================================
  Files          2766     2766              
  Lines        366105   367711    +1606     
============================================
+ Hits         316191   316250      +59     
- Misses        49914    51461    +1547     
Components Coverage Δ
dpp 85.96% <ø> (-1.33%) ⬇️
drive 84.17% <ø> (-0.08%) ⬇️
drive-abci 89.66% <ø> (ø)
sdk ∅ <ø> (∅)
dapi-client ∅ <ø> (∅)
platform-version ∅ <ø> (∅)
platform-value 92.92% <ø> (ø)
platform-wallet ∅ <ø> (∅)
drive-proof-verifier 49.78% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 2 only (queue backlog)

Verified the changes at head 7907237; no actionable in-scope defects were found. The conditional create-path status update preserves concurrent finality, and rejected defensive re-broadcasts retain bounded conflict resolution without releasing reservations. Independent validation passed all 98 targeted asset-lock tests, the full platform-wallet and platform-wallet-ffi suites (1,361 passed, 5 ignored), and git diff --check; the worktree remains clean.

Source: reviewer 1: gpt-6-astra (agent: phase2-reviewer, role: general); reviewer 2: gpt-6-astra (agent: phase2-reviewer, role: ffi-engineer); reviewer 3: gpt-6-astra (agent: phase2-reviewer, role: rust-quality); reviewer 4: gpt-6-astra (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6-astra (agent: astra-verifier, role: final-verifier)

Review provenance

  • Triage: critical by gpt-6-astra (effort low) — The change modifies concurrent asset-lock state transitions, broadcast recovery, and funding-reservation protection, where incorrect ordering or status handling could release committed inputs, create conflicting transactions, or compromise wallet fund recovery.
  • Phase 1 reviewers: not run (skipped for throughput: 21 PRs queued, above the 10 limit)
  • Fresh verifier: gpt-6-astra — final-verifier; agent astra-verifier
  • Phase 2 reviewers: gpt-6-astra — general (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — ffi-engineer (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — rust-quality (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — security-auditor (completed, effort xhigh); agent phase2-reviewer

@llbartekll

Copy link
Copy Markdown
Contributor

Went through the diff against the merge-base (299d662). The code changes look right to me, a few things before I approve:

1. The description no longer matches the diff. It describes a promote_built_to_broadcast CAS returning AlreadyAdvanced, a Built arm that promotes before broadcasting, and a test rejected_create_while_resume_broadcasts_keeps_row_and_reservation verified RED on the parent. None of these are in the branch: the Built arm in recovery.rs is byte-identical to base (still sends first, advances only when undispatched.is_none()), and the updated test asserts the opposite ("a send rejected before dispatch must leave the row at Built"). The race in the title is already closed on base by claim_resume_dispatch (#4355). What the diff actually adds is (a) the create path's post-broadcast advance going through advance_asset_lock_status_if(Built), (b) the Broadcast arm keeping a sighted conflict through the bounded wait instead of returning TransactionBroadcastUnconfirmed early, and (c) the doc pass. Since the body becomes the squash commit message, could you rewrite it to describe those?

2. 180 s wait on an offline launch (recovery.rs Broadcast arm, the new if input_conflict.is_none() fall-through). A Broadcast row with a sighted conflict now waits the full UNCONFIRMED_BROADCAST_PROOF_TIMEOUT before reporting AssetLockInputContested, where it used to return right after await_broadcast_ready. When the re-broadcast was rejected because the client is unstarted, nothing can wake that wait, so it is dead time per contested row (Swift catch-up runs at most 4 at once). Is that intended? An alternative is to return the contested verdict straight away when transport_missed is set and leave the bounded wait to the deferred retry. Either way worth a line in the description, since the test only exercises a 10 ms bound.

3. Rebase please. The merge commit picked up a rustfmt-only hunk in rs-drive-abci/.../deletion.rs that is in neither parent; v4.2-dev already has the identical fix (#4586), so it disappears on rebase.

Minor, no action needed unless you are already in there: the code-48 docs (error.rs in both crates, Swift, Kotlin, ERROR_CODE_REGISTRY.md) still say "the lock was (re-)broadcast and waited on", which is no longer true on the new Rejected-with-conflict path.

llbartekll
llbartekll previously approved these changes Sep 10, 2026
Promote Built rows before resume broadcasts through a shared compare-and-set.
Preserve concurrently advanced status and proof in both resume and create paths,
and keep rejected attempts tracked at Broadcast without releasing their inputs.

Test would have caught this in CI:
- rejected_create_while_resume_broadcasts_keeps_row_and_reservation: ✖ before
  the fix the rejected create removed the row and released its reservation;
  ✔ after the fix the row remains Broadcast and a rebuild cannot select its inputs.
- stale_built_resume_does_not_downgrade_a_concurrently_finalized_row: ✖ before
  the fix the stale resume timed out after replacing ChainLocked with Broadcast;
  ✔ after the fix it re-dispatches from the attached ChainLock proof.
- create_broadcast_does_not_downgrade_a_concurrently_finalized_row: ✖ before
  the fix the create completion replaced ChainLocked with Broadcast; ✔ after the
  fix it preserves the finalized status and proof.
- Built-resume rejection assertions: ✖ before the fix the row stayed Built;
  ✔ after the fix it stays tracked at Broadcast for defensive resume.
… send

A Broadcast row now means a broadcast was attempted, not that one reached the
network: two pre-dispatch rejections can leave a row at Broadcast having sent
nothing. The contested-verdict docs in the Rust error type and both mobile SDKs
still asserted an earlier call had sent the transaction.

Docs only; no behaviour change, so no test accompanies it.
Return the contested verdict immediately when transport readiness was missed and the defensive re-broadcast was rejected before dispatch. The readiness-deferred retry owns the next proof wait, while ready transports retain the bounded wait.

This intermediate fast path returns the conflict snapshot taken before readiness. That bounds offline latency but introduces a stale-verdict hazard if finality lands during the post-rejection probe; the follow-up commit refreshes finality and the conflict before code 48.

Test would have caught this in CI:
✖ on a160f11: offline_broadcast_resume_with_a_conflict_skips_the_dead_proof_wait failed "the offline foreground resume must not add the default proof wait" with left: 195s, right: 15s
✔ here: the same test returns AssetLockInputContested after exactly the 15s readiness wait
Refresh local finality and the input conflict immediately before returning code 48 from the offline Broadcast fast path. A recoverable proof completes the resume; finalized evidence without a proof suppresses the contested verdict, while only a genuine FinalityTimeout becomes code 20 and lookup errors such as WalletNotFound propagate unchanged.

Document the refreshed snapshot as the verdict linearization point: code 48 remains provisional, and a proof arriving afterwards is reported by the next resume. Settlement now explicitly requires a recoverable proof.

Tests would have caught this in CI:
✖ on commit A: offline_broadcast_resume_refreshes_finality_before_reporting_a_conflict failed "fresh local finality must outrank the stale conflict snapshot: AssetLockInputContested { ... }"
✔ here: the same test returns the ChainLock proof after exactly the 15s readiness wait
✖ on commit A: offline_broadcast_resume_preserves_wallet_removal_during_refresh failed "a removed wallet must report WalletNotFound, got AssetLockInputContested { ... }"
✔ here: the same test returns WalletNotFound after exactly the 15s readiness wait
@shumkov
shumkov force-pushed the fix/asset-lock-resume-broadcast-race branch from 7907237 to 751cd3b Compare September 11, 2026 02:21
@shumkov

shumkov commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — all three landed, plus one defect your second point led me into.

The 180s offline dead wait is gone. When transport readiness is missed and the
defensive re-broadcast is rejected before dispatch, the verdict returns immediately and the
readiness-deferred retry owns the next proof wait. Ready transports keep their bounded
wait, and the conflicted Built arm is unchanged. Pinned by
offline_broadcast_resume_with_a_conflict_skips_the_dead_proof_wait, which asserts a
duration under paused time rather than just a verdict: 195s before, exactly 15s now.

That fast exit introduced a worse bug than it fixed, so it is now two commits. Returning
the pre-readiness conflict snapshot meant a proof landing during the post-rejection probe
was still reported as AssetLockInputContested — trading a slow answer for a wrong one.
The follow-up commit re-reads local finality and the conflict immediately before emitting
code 48. I split the change so each half carries its own red→green rather than hiding the
intermediate: the first commit states the hazard it introduces, the second closes it.

A removed wallet was being reported as still tracked. Chasing the above, the refresh
fallback was classifying every failure as TransactionBroadcastUnconfirmed — code 20,
whose contract says the reservation is held and hosts must not auto-retry — including
WalletNotFound when the wallet was removed mid-resume. Only a genuine FinalityTimeout
becomes code 20 now; lookup errors propagate unchanged.

Rebased, so the stray rustfmt-only rs-drive-abci hunk is gone; the branch is nine
files. Code-48 docs across Rust, FFI, Swift, Kotlin and the registry now qualify the
immediate return as Broadcast-arm-only — the conflicted Built arm still waits, so the
earlier blanket wording was wrong. Settlement is also qualified as requiring a
recoverable proof: finalized evidence alone suppresses the contested verdict without
completing the resume.

Two things I deliberately did not do, both recorded rather than silently skipped:

  • A proof can still arrive between the refreshed snapshot and the return. That window is
    inherent to any snapshot verdict, so instead of adding another refresh that merely moves
    it, the resume docs now state the contract: code 48 is provisional as of its refreshed
    snapshot, and a proof arriving afterwards is reported by the next resume.
  • The same swallow-the-error shape exists on the arm where there was no initial conflict.
    That one is pre-existing — on the base the Err(probe_err) arm is unconditional — so it
    is out of scope here and deserves its own PR and release note.

Worth flagging for mobile hosts: WalletNotFound propagates in Rust but maps to FFI code
99 (ErrorUnknown), and asset_lock_manager_catch_up_blocking maps everything except
conflict variants to code 6. Neither asymmetry is introduced here.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 2 only (queue backlog)

The reviewed changes correctly address both in-scope defects: the create path now advances only rows that remain in Built, preventing a concurrent finalized status from being downgraded, and the Broadcast recovery path preserves contested evidence while refreshing finality before returning a verdict. The targeted tests and documentation updates cover the changed behavior, and no additional in-scope correctness issues were identified.

Review provenance

Source: reviewer 1: gpt-6-astra (agent: phase2-reviewer, role: general); reviewer 2: gpt-6-astra (agent: phase2-reviewer, role: ffi-engineer); reviewer 3: gpt-6-astra (agent: phase2-reviewer, role: rust-quality); reviewer 4: gpt-6-astra (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6-astra (agent: astra-verifier, role: final-verifier)

  • Triage: normal by gpt-6-astra (effort low) — The diff makes substantial, intricate changes across asset-lock creation and recovery state-machine logic, but it does not itself alter consensus rules, funds movement, cryptography, key handling, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: not run (skipped for throughput: 15 PRs queued, above the 10 limit)
  • Fresh verifier: gpt-6-astra — final-verifier; agent astra-verifier
  • Phase 2 reviewers: gpt-6-astra — general (completed, effort high); agent phase2-reviewer, gpt-6-astra — ffi-engineer (completed, effort high); agent phase2-reviewer, gpt-6-astra — rust-quality (completed, effort high); agent phase2-reviewer, gpt-6-astra — security-auditor (completed, effort high); agent phase2-reviewer
Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

  • Review pre-existing WalletNotFound error mapping across mobile bindings — The Rust wallet can propagate WalletNotFound, while the existing FFI and mobile catch-up wrappers normalize it to generic error codes. This mapping asymmetry is outside the production changes in this PR, which modify recovery behavior and documentation rather than the executable error mapping.
    • Follow-up: Track the WalletNotFound mapping contract separately and add cross-language tests if mobile callers need to distinguish a removed wallet from generic errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants