fix(kotlin-sdk): reconcile the TXO store against the engine and repair restored address pools - #4439
HashEngineering wants to merge 3 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🕓 Review not started yet because this PR is a draft.
Commit 80d61e8. Normal review starts when eligible; priority review starts as soon as a slot is available. |
HashEngineering
changed the title
590ee97 to
ac7c9b2
Compare
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The reconciliation and restore repair target real persistence divergence, but four correctness defects can durably mark unconfirmed coins spent, repeatedly report or credit an insertion that never occurred, corrupt an already-correct transaction amount, and leave DashPay receiving pools sparse. Three additional in-scope issues weaken contact-row classification, make the periodic inventory scan quadratic in account count, and bypass the canonical outpoint conversion.
Source: Codex reviewer lanes codex-general, codex-rust-quality, and codex-ffi-engineer (exact backend model IDs were not supplied in the evidence); final verifier backend grok-4.5; orchestration-only openclaw-agent/cliproxy/gpt-5.6-sol is not reviewer evidence.
Validated blockers were found in the Codex precheck. Opus is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— rust-quality (completed),gpt-5.6-sol— ffi-engineer (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 4 blocking | 🟡 3 suggestion(s)
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt`:
- [BLOCKING] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1345-1352: Do not persist unconfirmed inputs as spent
The exact pinned key-wallet revision inserts every input of a recorded transaction into `spent_outpoints`, regardless of whether its context is mempool or in-block. Treating membership in that set as a confirmed spend contradicts this handler's established rule at lines 928-934 and 3073-3080: mempool-linked inputs remain unspent in Room so they can be restored and reclassified after restart. A reconciliation while a payment is unconfirmed therefore makes the coin durably spent; if the transaction is later abandoned and its release update is lost or interrupted, the deliberate never-unmark policy prevents every later reconciliation from recovering it. Export spender context/finality with each outpoint, or only flip rows for spends proven confirmed by another authoritative source.
- [BLOCKING] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1288-1301: Only account for TXOs that were actually inserted
`upsertUtxoRow` returns early when the parent transaction is still marked `isGloballySwept`, but it returns `Unit`, so this caller cannot distinguish that refusal from a successful insert. The reconciler then increments `inserted`, adds to `insertedDuffs`, and may update `netAmount` even though the TXO remains absent. A missed reinstatement record can leave exactly this stale tombstone while the engine authoritatively holds the output; every periodic pass then repeats the false heal and can repeatedly add the same amount. Make the helper report whether it materialized the row and perform all counters and amount repair only after a successful insert, or explicitly reconcile the stale swept state first.
- [BLOCKING] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1298-1301: Do not infer netAmount from TXO presence
A missing `txos` projection does not prove that the independently persisted transaction record omitted the output from `netAmount`. For example, a corrective transaction callback can already store the engine's recomputed net amount while delivery of the corresponding UTXO projection is omitted, or a TXO can disappear later without changing its parent transaction. In either case this unconditional delta overstates the transaction, and the newly inserted row makes the corruption permanent because later passes become no-ops. The inventory does not include an authoritative expected net amount, so repair must compare against one or recompute the amount from authoritative ownership data rather than infer it from row absence.
- [SUGGESTION] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1321-1340: Resolve contact ownership through coreAddressId
The exclusion only checks `row.accountId`, but production changeset writes leave that field null and route TXO ownership through `coreAddressId -> core_addresses.accountId`, as documented by `buildUtxoRestoreData` at lines 3065-3069. The regression test manually fills `accountId`, so it does not represent production contact rows. Resolve the effective account through `coreAddressId` when the direct FK is null in both reverse-pass loops; otherwise normal DIP-15 contact outputs are repeatedly reported as engine-unknown and defeat the intended regression-tripwire signal.
In `packages/rs-platform-wallet-ffi/src/persistence.rs`:
- [BLOCKING] packages/rs-platform-wallet-ffi/src/persistence.rs:4085-4113: Repair DashPay pools with their concrete account xpub
Both DashPay variants reach this resolver, but the exact pinned implementation of `Wallet::key_source_for_account_type` explicitly returns `NoKeySource` for `DashpayReceivingFunds` and `DashpayExternalAccount`. The guard at line 4131 therefore always skips their hole repair. This is especially harmful for `DashpayReceivingFunds`: it is wallet-owned and funds-bearing, and its concrete account already carries the xpub needed to reconstruct missing addresses. Resolve the full `AccountType` through `wallet.accounts.account_of_type(account_type)` and use that account's xpub as `KeySource::Public`, retaining the existing helper as the fallback for special account types such as the BLS provider account. Add a restore test with a sparse DashPay receiving pool and a real signing wallet; the current test passes `None` and cannot exercise this path.
In `packages/rs-unified-sdk-jni/src/wallet_manager.rs`:
- [SUGGESTION] packages/rs-unified-sdk-jni/src/wallet_manager.rs:3205-3207: Snapshot all account inventories in one pass
The JNI method enumerates all N accounts and then invokes separate per-account UTXO and spent-outpoint FFI accessors. Each accessor reacquires `wallet_manager.blocking_read()`, allocates `all_accounts()`, and linearly searches that N-element collection, making one full reconciliation O(N²) with 2N+1 lock acquisitions. DashPay creates accounts per contact, so this periodic scan scales with wallet history and runs both at sync completion and every 30 minutes. Add a manager-level snapshot that gathers account identity, UTXOs, and spent outpoints under one read lock, then expose it through one FFI operation.
In `packages/rs-platform-wallet-ffi/src/manager_diagnostics.rs`:
- [SUGGESTION] packages/rs-platform-wallet-ffi/src/manager_diagnostics.rs:655-661: Use the canonical OutPointFFI conversion
`OutPointFFI` already implements `From<&dashcore::OutPoint>` and documents that implementation as the single authority for preserving the persistence join key and txid byte order. The new export manually duplicates the conversion. It is equivalent today but can silently diverge if the canonical representation changes, so route this export through the existing conversion boundary.
| key in engineSpentKeys -> { | ||
| // Lost spend update (#4425): the engine knows this | ||
| // coin was spent; the row missed the flip. Flip in | ||
| // place — spendingTxid stays as-is (usually null; | ||
| // the spender's row, if it ever arrives, relinks | ||
| // via the deferred-input drain). | ||
| database.txoDao().upsert(row.copy(isSpent = true)) | ||
| flippedSpent++ |
There was a problem hiding this comment.
🔴 Blocking: Do not persist unconfirmed inputs as spent
The exact pinned key-wallet revision inserts every input of a recorded transaction into spent_outpoints, regardless of whether its context is mempool or in-block. Treating membership in that set as a confirmed spend contradicts this handler's established rule at lines 928-934 and 3073-3080: mempool-linked inputs remain unspent in Room so they can be restored and reclassified after restart. A reconciliation while a payment is unconfirmed therefore makes the coin durably spent; if the transaction is later abandoned and its release update is lost or interrupted, the deliberate never-unmark policy prevents every later reconciliation from recovering it. Export spender context/finality with each outpoint, or only flip rows for spends proven confirmed by another authoritative source.
source: ['codex']
There was a problem hiding this comment.
Resolved in 2195b18 — Do not persist unconfirmed inputs as spent no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
Resolved in 2195b18 and still true after the rebase: the reverse pass only classifies and logs, nothing is flipped.
| upsertUtxoRow( | ||
| database, walletId, txid, vout, amount, address, scriptPubKey, | ||
| height, | ||
| isCoinbase = false, | ||
| isConfirmed = true, | ||
| isInstantLocked = false, | ||
| isLocked = isLocked, | ||
| ) | ||
| inserted++ | ||
| insertedDuffs += amount | ||
| if (priorTx != null && priorTx.transactionData.isNotEmpty()) { | ||
| if (database.transactionDao().addToNetAmount(txid, amount) > 0) { | ||
| netAmountRepairs++ | ||
| } |
There was a problem hiding this comment.
🔴 Blocking: Only account for TXOs that were actually inserted
upsertUtxoRow returns early when the parent transaction is still marked isGloballySwept, but it returns Unit, so this caller cannot distinguish that refusal from a successful insert. The reconciler then increments inserted, adds to insertedDuffs, and may update netAmount even though the TXO remains absent. A missed reinstatement record can leave exactly this stale tombstone while the engine authoritatively holds the output; every periodic pass then repeats the false heal and can repeatedly add the same amount. Make the helper report whether it materialized the row and perform all counters and amount repair only after a successful insert, or explicitly reconcile the stale swept state first.
source: ['codex']
There was a problem hiding this comment.
Resolved in 2195b18 — Only account for TXOs that were actually inserted no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
The refusal path no longer exists on v4.2-dev (no isGloballySwept flag). upsertUtxoRow is v4.2-dev's own insert discipline extracted into one helper and always writes, so the counter and the skippedSwept report field are gone.
| if (priorTx != null && priorTx.transactionData.isNotEmpty()) { | ||
| if (database.transactionDao().addToNetAmount(txid, amount) > 0) { | ||
| netAmountRepairs++ | ||
| } |
There was a problem hiding this comment.
🔴 Blocking: Do not infer netAmount from TXO presence
A missing txos projection does not prove that the independently persisted transaction record omitted the output from netAmount. For example, a corrective transaction callback can already store the engine's recomputed net amount while delivery of the corresponding UTXO projection is omitted, or a TXO can disappear later without changing its parent transaction. In either case this unconditional delta overstates the transaction, and the newly inserted row makes the corruption permanent because later passes become no-ops. The inventory does not include an authoritative expected net amount, so repair must compare against one or recompute the amount from authoritative ownership data rather than infer it from row absence.
source: ['codex']
There was a problem hiding this comment.
Resolved in 2195b18 — Do not infer netAmount from TXO presence no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
Done in 2195b18: netAmount is never mutated; a healed TXO with a pre-existing record is only reported as a netAmountSuspect.
| let key_source = signing_wallet | ||
| .and_then(|wallet| { | ||
| key_wallet::transaction_checking::transaction_router::AccountTypeToCheck::try_from( | ||
| &*managed_type, | ||
| ) | ||
| .ok() | ||
| .map(|check_type| { | ||
| let account_index = match &account_type { | ||
| AccountType::Standard { | ||
| index, .. | ||
| } | ||
| | AccountType::CoinJoin { | ||
| index, | ||
| } | ||
| | AccountType::DashpayReceivingFunds { | ||
| index, .. | ||
| } | ||
| | AccountType::DashpayExternalAccount { | ||
| index, .. | ||
| } => Some(*index), | ||
| AccountType::IdentityTopUp { | ||
| registration_index, | ||
| } => Some(*registration_index), | ||
| _ => None, | ||
| }; | ||
| wallet.key_source_for_account_type(&check_type, account_index) | ||
| }) | ||
| }) | ||
| .unwrap_or(key_wallet::KeySource::NoKeySource); |
There was a problem hiding this comment.
🔴 Blocking: Repair DashPay pools with their concrete account xpub
Both DashPay variants reach this resolver, but the exact pinned implementation of Wallet::key_source_for_account_type explicitly returns NoKeySource for DashpayReceivingFunds and DashpayExternalAccount. The guard at line 4131 therefore always skips their hole repair. This is especially harmful for DashpayReceivingFunds: it is wallet-owned and funds-bearing, and its concrete account already carries the xpub needed to reconstruct missing addresses. Resolve the full AccountType through wallet.accounts.account_of_type(account_type) and use that account's xpub as KeySource::Public, retaining the existing helper as the fallback for special account types such as the BLS provider account. Add a restore test with a sparse DashPay receiving pool and a real signing wallet; the current test passes None and cannot exercise this path.
source: ['codex']
There was a problem hiding this comment.
Resolved in 2195b18 — Repair DashPay pools with their concrete account xpub no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
Done in 2195b18 (key source resolved from the signing wallet). DashPay contact pools are re-derived by contact sync at runtime; that skip is now logged at debug.
There was a problem hiding this comment.
Fixed in 80d61e8 as you describe: the resolver consults accounts.account_of_type(account_type) first and uses that account's xpub as KeySource::Public, keeping key_source_for_account_type as the fallback for the types account_of_type does not return (the BLS provider operator and Ed25519 platform-node accounts). New test dashpay_receiving_pool_holes_are_repaired_from_the_account_xpub restores a sparse DashPay receiving pool with a real signing wallet; without the fix it fails with 30 holes left at indices 20–49, with it every index up to the watermark is derivable. Leaving the thread for you to resolve.
| for (row in storeRows) { | ||
| if (!row.isSpent) continue | ||
| if (row.accountId != null && row.accountId in foreignAccountIds) continue | ||
| val key = "${row.txid?.toHex() ?: continue}:${row.vout}" | ||
| if (key in engineUnspentKeys) { | ||
| stuckSpent++ | ||
| stuckSpentDuffs += row.amount | ||
| Log.w( | ||
| TAG, | ||
| "txos reconcile: store row spent but engine lists it " + | ||
| "unspent outpoint=$key amount=${row.amount} — LOG-ONLY " + | ||
| "(lost release, or a live spend racing the engine)", | ||
| ) | ||
| } | ||
| } | ||
| val storeUnspent = storeRows.filter { !it.isSpent } | ||
| for (row in storeUnspent) { | ||
| if (row.accountId != null && row.accountId in foreignAccountIds) { | ||
| skippedForeign++ | ||
| continue |
There was a problem hiding this comment.
🟡 Suggestion: Resolve contact ownership through coreAddressId
The exclusion only checks row.accountId, but production changeset writes leave that field null and route TXO ownership through coreAddressId -> core_addresses.accountId, as documented by buildUtxoRestoreData at lines 3065-3069. The regression test manually fills accountId, so it does not represent production contact rows. Resolve the effective account through coreAddressId when the direct FK is null in both reverse-pass loops; otherwise normal DIP-15 contact outputs are repeatedly reported as engine-unknown and defeat the intended regression-tripwire signal.
source: ['codex']
There was a problem hiding this comment.
Resolved in 2195b18 — Resolve contact ownership through coreAddressId no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
Done in 2195b18: rowIsForeign resolves both accountId and coreAddressId → core_addresses.accountId, and the insert pass uses the inventory's account tag first.
|
@bfoss765 — this is the store-reconcile series from the job-flower investigation, retargeted off the integration branch per review: it now stacks on #4406 (whose swept-tombstone semantics the shared 🤖 Generated with Claude Code |
|
Heads-up on the base, since this is stacked on Two changes are landing on that branch that touch files you also touch, so it is worth knowing before you build further on them. Already landed ( Coming next, and this one does overlap you: the swept-tombstone lifetime rule is being reworked. The current version stamps a tombstone with the height at which the sweep was observed and collects it after a fixed margin; that is unsound for an InstantSend-locked winner that stays unmined, so it is being replaced with the winner's actual mined height, carried on the event by dashpay/rust-dashcore#975. Concretely, on this branch that will change:
It also needs a repin once #975 merges. Room schema v13 ( No action needed from you — the base moves under you and your PR keeps rebasing — but if you are about to write anything in the tombstone or |
…ot prove thepastaclaw review round on dashpay#4439, all four blockers: - The spent-flip is demoted to LOG-ONLY (wouldFlipSpent): the engine's spent set records every input of every recorded transaction including MEMPOOL spends, with no context — persisting the flip would settle an unconfirmed spend, contradicting this handler's own in-block gating. Re-arm as a mutation only when the engine exports spends with context. - upsertUtxoRow reports whether it wrote: a globally-swept-parent refusal is now visible to the reconcile (skippedSwept), which no longer counts phantom heals nor flags netAmounts for rows that were never inserted. - The netAmount repair is demoted to LOG-ONLY (netAmountSuspects): a corrective record callback can land while its TXO delivery races this sweep, and blind addition double-credits. The event pipeline owns net correctness; the reconcile reports the suspicion. - The restore-time pool repair announces every pool it cannot repair (DashPay contact pools have no public key source by design and re-derive through DashPay sync; hardened pools cannot be publicly derived). Plus the review suggestions: contact-row exclusion now resolves ownership through coreAddressId -> core_addresses.accountId (production rows leave txos.accountId null, so the accountId-only check was ineffective), the neither-inventory log names the finalized-drop ambiguity, and the JNI spent-outpoint export uses the canonical OutPointFFI conversion. The reconcile is now fully observe-and-heal-forward: its only mutation is inserting provably-owned engine UTXOs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Review round addressed in 2195b18 — all four blockers plus the suggestions:
Suggestions: contact-row exclusion now resolves ownership through 117 handler tests green (all reconcile semantics re-pinned to log-only behavior), FFI 277 green. @romchornyi thanks for the base heads-up — will take the tombstone-lifetime rework and the #975 repin as a merge round here when it lands, and expect the v13 schema renumber. 🤖 Generated with Claude Code |
|
Device validation of this branch exactly as pushed (f0f632e + the rust-dashcore#979 pin): full uninstall → seed restore → rescan → kill+relaunch on the CoinJoin-heavy testnet wallet. Reconcile reported One composition note for reviewers: transaction-history net sums only converge with #4438 landed too. This branch heals coins and balances on its own, but multi-account history rows persist one account's slice until the record fold (#4438) is in — observed directly: the same wallet's history sum converges on a base that includes the fold and doesn't here. The two PRs are independent to merge but complementary in effect. Also observed, expected but worth documenting: contact-payment coins land only after a DashPay backfill pass that needs an unlocked wallet session (deferred contact-crypto builds gate it) — a freshly restored wallet that is never unlocked shows a balance short by its incoming contact payments until first unlock. 🤖 Generated with Claude Code |
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The latest review round correctly demotes ambiguous spend and net-amount changes to logging, counts only successful inserts, resolves contact ownership through core addresses, and uses the canonical outpoint conversion. Three blockers remain: restored DashPay pools still skip repair despite carrying usable account xpubs, healed TXOs can lose their exported account ownership and disappear again on restart, and the periodic JNI inventory fully materializes an unbounded chain-inflatable snapshot in a mobile process. Source: Codex general, security-auditor, and FFI-engineer reviewer lanes (exact backend model IDs were not supplied); final verifier: Claude Agent SDK (exact backend model ID was not supplied); openclaw-agent/cliproxy/gpt-5.6-sol is orchestration-only and not reviewer evidence.
Validated blockers were found in the Codex precheck. Opus is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed),gpt-5.6-sol— ffi-engineer (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 3 blocking | 🟡 1 suggestion(s)
2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt`:
- [BLOCKING] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1299-1306: Preserve exported account ownership on healed TXOs
The native inventory identifies the account that owns each UTXO, but this insertion ignores those tags. `upsertUtxoRow` preserves an existing `accountId` and sets `coreAddressId` only when the corresponding `core_addresses` row already exists. If persistence lost both the TXO and its address row—the two divergence classes this PR is intended to repair—the reconciled row is inserted with neither ownership link. On the next launch, `buildUtxoRestoreData` can resolve ownership only through `txo.accountId` or the address relationship, so it skips the healed output and recreates the visible fund loss. Emit the complete account tuple already present in `AccountBalanceEntryFFI` (`typeTag`, `standardTag`, index, registration index, key class, and DashPay identity IDs), resolve the Room account during reconciliation, and persist its ID on the healed TXO even when the address projection is absent.
- [SUGGESTION] packages/kotlin-sdk/sdk/src/main/kotlin/org/dashfoundation/dashsdk/persistence/PlatformWalletPersistenceHandler.kt:1327-1330: Interpolate values in the healed-TXO diagnostic
Each dollar sign is escaped with `${'$'}`, so the diagnostic emits literal placeholders such as `${txid.toHex()}:$vout`, `$amount`, and `${priorTx.netAmount}`. That removes the outpoint and values needed to investigate the nonzero reconciliation heal this warning is intended to identify.
In `packages/rs-unified-sdk-jni/src/wallet_manager.rs`:
- [BLOCKING] packages/rs-unified-sdk-jni/src/wallet_manager.rs:3202-3348: Do not materialize an unbounded chain-controlled inventory
This periodic operation stores one formatted `String` per UTXO and spent outpoint, duplicates them through `join`, builds another complete JSON string with `format!`, copies that value across JNI, and then Kotlin parses it into a full JSON DOM and constructs additional full-inventory hash sets. Inventory cardinality is unbounded and remotely inflatable because anyone who knows a watched address can repeatedly send dust outputs to it. On a memory-constrained mobile process, an attacker-inflated wallet can therefore make every SYNCED transition and 30-minute reconciliation allocate several simultaneous copies of the inventory, causing repeated allocation failure or process termination. Expose a bounded, cursor-based snapshot or stream/iterate records through a native callback or compact binary representation so neither Rust nor Kotlin must hold the complete serialized inventory at once.
- [SUGGESTION] packages/rs-unified-sdk-jni/src/wallet_manager.rs:3205-3207: Snapshot all account inventories in one pass
(existing thread: https://github.com/dashpay/platform/pull/4439#discussion_r3826530759)
The JNI method first enumerates all N accounts and then invokes separate per-account UTXO and spent-outpoint accessors. Each accessor reacquires `wallet_manager.blocking_read()`, rebuilds `all_accounts()`, and linearly searches that N-element collection. One reconciliation therefore performs O(N²) account traversal and 2N+1 lock acquisitions. DashPay adds accounts per contact, and this path runs at every SYNCED transition and every 30 minutes. Add a manager-level inventory operation that gathers each account's identity, UTXOs, and spent outpoints under one read lock before exposing the snapshot through FFI.
In `packages/rs-platform-wallet-ffi/src/persistence.rs`:
- [BLOCKING] packages/rs-platform-wallet-ffi/src/persistence.rs:4085-4113: Repair DashPay pools with their concrete account xpub
(existing thread: https://github.com/dashpay/platform/pull/4439#discussion_r3826530747)
The resolver still routes both DashPay variants through `Wallet::key_source_for_account_type`, which the pinned key-wallet revision explicitly maps to `NoKeySource`. The subsequent `repairable` guard therefore skips every DashPay pool. This is not a cryptographic limitation: `build_wallet_start_state` reconstructs these ECDSA accounts with their persisted `account_xpub`, `AccountCollection::account_of_type` supports both full DashPay account variants, and the normal DashPay registration paths construct the same `Absent` address pools from `KeySource::Public(account.account_xpub)`. A sparse `DashpayReceivingFunds` pool consequently remains unable to recognize payments to omitted indices after cold restore until an unlocked contact-sync happens, so the restore-time repair promised by this PR is incomplete. Resolve the concrete full `AccountType` through `wallet.accounts.account_of_type(account_type)` and use its xpub as the public key source, retaining the existing helper as a fallback for special key accounts. Add a sparse DashPay receiving-pool restore test using a real wallet; the current test passes `None` and cannot exercise derivation.
| val wrote = upsertUtxoRow( | ||
| database, walletId, txid, vout, amount, address, scriptPubKey, | ||
| height, | ||
| isCoinbase = false, | ||
| isConfirmed = true, | ||
| isInstantLocked = false, | ||
| isLocked = isLocked, | ||
| ) |
There was a problem hiding this comment.
🔴 Blocking: Preserve exported account ownership on healed TXOs
The native inventory identifies the account that owns each UTXO, but this insertion ignores those tags. upsertUtxoRow preserves an existing accountId and sets coreAddressId only when the corresponding core_addresses row already exists. If persistence lost both the TXO and its address row—the two divergence classes this PR is intended to repair—the reconciled row is inserted with neither ownership link. On the next launch, buildUtxoRestoreData can resolve ownership only through txo.accountId or the address relationship, so it skips the healed output and recreates the visible fund loss. Emit the complete account tuple already present in AccountBalanceEntryFFI (typeTag, standardTag, index, registration index, key class, and DashPay identity IDs), resolve the Room account during reconciliation, and persist its ID on the healed TXO even when the address projection is absent.
source: ['codex']
There was a problem hiding this comment.
Done in 4c6c6ec and kept: every row carries the account tuple, the reconcile resolves the Room account and stamps resolvedAccountId on the healed row (healedUnowned counts the unresolvable ones).
| "txos reconcile: healed TXO ${'$'}{txid.toHex()}:${'$'}vout " + | ||
| "(${'$'}amount duffs) has a pre-existing record whose " + | ||
| "netAmount may be short by that amount — LOG-ONLY, " + | ||
| "storedNet=${'$'}{priorTx.netAmount}", |
There was a problem hiding this comment.
🟡 Suggestion: Interpolate values in the healed-TXO diagnostic
Each dollar sign is escaped with ${'$'}, so the diagnostic emits literal placeholders such as ${txid.toHex()}:$vout, $amount, and ${priorTx.netAmount}. That removes the outpoint and values needed to investigate the nonzero reconciliation heal this warning is intended to identify.
| "txos reconcile: healed TXO ${'$'}{txid.toHex()}:${'$'}vout " + | |
| "(${'$'}amount duffs) has a pre-existing record whose " + | |
| "netAmount may be short by that amount — LOG-ONLY, " + | |
| "storedNet=${'$'}{priorTx.netAmount}", | |
| "txos reconcile: healed TXO ${txid.toHex()}:$vout " + | |
| "($amount duffs) has a pre-existing record whose " + | |
| "netAmount may be short by that amount — LOG-ONLY, " + | |
| "storedNet=${priorTx.netAmount}", |
source: ['codex']
There was a problem hiding this comment.
Fixed: the diagnostic now interpolates txid:vout, amount and storedNet.
f0f632e to
7c1abcb
Compare
…ot prove thepastaclaw review round on dashpay#4439, all four blockers: - The spent-flip is demoted to LOG-ONLY (wouldFlipSpent): the engine's spent set records every input of every recorded transaction including MEMPOOL spends, with no context — persisting the flip would settle an unconfirmed spend, contradicting this handler's own in-block gating. Re-arm as a mutation only when the engine exports spends with context. - upsertUtxoRow reports whether it wrote: a globally-swept-parent refusal is now visible to the reconcile (skippedSwept), which no longer counts phantom heals nor flags netAmounts for rows that were never inserted. - The netAmount repair is demoted to LOG-ONLY (netAmountSuspects): a corrective record callback can land while its TXO delivery races this sweep, and blind addition double-credits. The event pipeline owns net correctness; the reconcile reports the suspicion. - The restore-time pool repair announces every pool it cannot repair (DashPay contact pools have no public key source by design and re-derive through DashPay sync; hardened pools cannot be publicly derived). Plus the review suggestions: contact-row exclusion now resolves ownership through coreAddressId -> core_addresses.accountId (production rows leave txos.accountId null, so the accountId-only check was ineffective), the neither-inventory log names the finalized-drop ambiguity, and the JNI spent-outpoint export uses the canonical OutPointFFI conversion. The reconcile is now fully observe-and-heal-forward: its only mutation is inserting provably-owned engine UTXOs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| minConfirmations = minConfirmations, | ||
| pageSize = pageSize, | ||
| engineUtxoPage = { cursor, limit -> | ||
| if (cursor == null) firstPage else page(cursor, limit) |
There was a problem hiding this comment.
The first-page prefetch is correct only under an invariant nothing enforces.
firstPage is fetched eagerly and smuggled back through a stateful closure keyed on cursor == null, purely to preserve the "return null on dead transport" signature. The comment states the load-bearing assumption — "The handler asks for the null cursor exactly once" — but reconcileTxos is a public method on another class with no such contract in its own docs.
Any future change there (a retry after a transport failure, a second sweep, restarting the insert pass) would silently replay a stale page and re-walk rows already applied, with no test covering it.
Cheaper and self-enforcing: pass ::page straight through and have reconcileTxos return null itself when the first page comes back null — the null-cursor call is already distinguishable inside the loop.
🤖 Posted autonomously by Claude on behalf of pasta.
There was a problem hiding this comment.
Gone (c5cbc44): reconcileTxoStore passes the page lambda straight through, and reconcileTxos returns null itself when the first page is null (test reconcileReturnsNoReportWhenTheFirstPageIsUnavailable).
| * Must NOT be called from the handler's own [dispatcher] (it takes | ||
| * [callbackExclusion] and runs Room transactions). | ||
| */ | ||
| suspend fun reconcileTxos( |
There was a problem hiding this comment.
reconcileTxos is ~400 lines holding two unrelated passes and 17 mutable counters in one scope.
The insert pass and the classification pass share exactly one piece of state — foreignAccountIds — yet every counter is live across both: skippedImmature/healedUnowned/netAmountSuspects/skippedSwept belong only to the first, wouldFlipSpent/stuckSpent/wouldRemove only to the second. The insert loop nests five deep (while → withLock → withTransaction → for → the skip chain), and both passes declare local suspend funs inside it.
Any edit to either half currently requires reading the whole body to know which counters are in play. Two private functions — healMissingTxos(...) and classifyStoreRows(...), each returning its own half of the report — would leave reconcileTxos as: resolve foreignAccountIds, call both, merge, log.
🤖 Posted autonomously by Claude on behalf of pasta.
There was a problem hiding this comment.
Split (c5cbc44): healMissingTxos (+ healEngineRow per row) and classifyStoreRows each return their half (HealPass / ClassifyPass); reconcileTxos resolves the foreign account set, calls both, merges and logs.
There was a problem hiding this comment.
Done in c5cbc44 and carried forward: reconcileTxos is now ~80 lines that resolve the shared exclusion set, call healMissingTxos and classifyStoreRows, merge the two halves and log. Leaving the thread for you to resolve.
| mapNativeErrors { | ||
| WalletManagerNative.walletManagerUtxosPageJson( | ||
| managerHandle, walletId, network.ffiValue, cursor, limit, | ||
| ) |
There was a problem hiding this comment.
[correctness — confirmed mechanism] The null-transport contract is unreachable: mapNativeErrors rethrows, it never returns null.
mapNativeErrors (DashSdkError.kt:610) catches DashSDKException and throws DashSdkError.fromNative(e) — and the JNI side (guard + take_pwffi_error) throws an SDK exception whenever the native call fails. So page(...) propagates an exception on transport failure rather than returning null:
val firstPage = page(null, pageSize) ?: return null— the documented "returns null when the first page fails" path never fires; callers get an exception instead.- In
reconcileTxos,if (pageJson == null) { transportFailures++; break }(and the classify equivalent) never runs for real JNI failures — a mid-sweep transport error aborts the whole pass by exception, discarding the report the truncate-and-report design promises.maybeReconcileTxoStoresswallows it viarunCatching, so the partial-sweep telemetry is silently lost.
Either wrap the two lambdas in runCatching { ... }.getOrNull() so the handler's null-handling actually engages, or update the contract docs to exception semantics.
🤖 Posted autonomously by Claude on behalf of pasta.
There was a problem hiding this comment.
Fixed (c5cbc44): both transport lambdas wrap mapNativeErrors in runCatching { … }.getOrNull() and log the fault, so a mid-sweep native failure yields the partial report the truncate-and-report design promises.
| if (!synced) return | ||
| val now = System.currentTimeMillis() | ||
| if (!transitioned && now - lastTxoReconcileAtMs < TXO_RECONCILE_INTERVAL_MS) return | ||
| val tipHeight = (progress.filters?.currentHeight ?: 0L).toInt() |
There was a problem hiding this comment.
[correctness — plausible] tipHeight comes solely from progress.filters?.currentHeight; when the filters sub-phase is absent (has_filters == false maps to filters = null in SpvSyncProgressData.fromNative) or reports 0, maybeReconcileTxoStores returns before scheduling anything — silently, on every tick.
If dash-spv ever reports SYNCED without a filters sub-progress (filter sync disabled, or the phase dropped after completion), the entire heal this PR exists to deliver never runs, with no log line to say so. Consider falling back to headers?.currentHeight (or the wallet's synced height accessor) and logging when the sweep is skipped for want of a tip height.
🤖 Posted autonomously by Claude on behalf of pasta.
There was a problem hiding this comment.
Fixed (c5cbc44): falls back to headers.currentHeight and logs when neither height is available instead of returning silently.
| /// page — the reconciler must still see every account that DID read, so | ||
| /// one faulted account cannot mask the others' repair. | ||
| #[no_mangle] | ||
| pub extern "system" fn Java_org_dashfoundation_dashsdk_ffi_WalletManagerNative_walletManagerUtxosPageJson( |
There was a problem hiding this comment.
[conventions — packages/kotlin-sdk/CLAUDE.md] walletManagerUtxosPageJson violates the stated rule: "No JNI functions that stitch together existing Rust calls — add the composite to Rust instead."
This export stitches platform_wallet_manager_get_account_balances + per-account platform_wallet_account_utxos_page, and layers on its own cross-account ordering (account_sort_key), an ad-hoc <accountKeyHex>:<txidHex>:<vout> cursor format, hand-rolled hex and JSON encoders, and dashcore address encoding — all in the JNI shim, where the account-key layout can drift from AccountBalanceEntryFFI with no compile error. A wallet-wide paged accessor beside account_utxos_page_blocking in rs-platform-wallet would own the invariant once, and the Swift host would get it for free.
🤖 Posted autonomously by Claude on behalf of pasta.
There was a problem hiding this comment.
Fixed (ca3c739): the composite lives in rs-platform-wallet (wallet_utxos_page_blocking) and the FFI (platform_wallet_wallet_utxos_page); the JNI is a serializer over one call, and the Swift host gets the same walk.
There was a problem hiding this comment.
The composite is gone. The export is a serde shim over one call, upstream's platform_wallet_wallet_utxos_page from #4638; account ordering, cursor semantics, the page bound and the address rendering all live in platform-wallet now, and the dashcore dependency is out of the JNI crate. Leaving the thread for you to resolve. #4439 (comment)
…pay#4474) The restore scan's ownership set did not include the CoinJoin derivation chains unless the host's account-creation options (or the host itself, pre-scan) supplied them. Change from a CoinJoin-funded send then went to an unwatched CoinJoin-chain address and was attributed foreign: the record was born with net = -(sum of inputs) instead of -(payment + fee), and unspent CoinJoin change was missing from the balance. register_wallet now mirrors every BIP44 account index into a CoinJoin account — while the wallet is still seed-bearing (before the external-signable downgrade) and before ManagedWalletInfo::from_wallet, so the chains ride the initial birth-floor scan checkpoint, the address-pool snapshot, and the persisted account registrations. A deliberate no-op for WalletAccountCreationOptions::None (no funds accounts, nothing to misattribute). Born-wrong records in stores restored before this fix heal through the rescan a CoinJoin account registration triggers: the corrected records arrive via the normal correction callbacks (rust-dashcore#979) and compose with the Android host walker's durable class-level corrections — no store-side net arithmetic (demoted to log-only in the external-signable at load, so the SDK cannot derive the account itself for such stores; load_from_persistor now logs the gap so a host that neither creates nor registers the chain is visible. Acceptance item 2 (unspent CoinJoin change in balance on the live processing path) was already covered by dashpay#4439 + rust-dashcore#979; the new nets-to-payment-plus-fee test pins it here for the restore shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest testing of restoring large coinjoin wallets (testnet) are showing that this PR may not be fixing any problems that once were though to be problems. More testing will be done to decide if this should be closed or kept open. |
bc0d859 to
c5cbc44
Compare
|
Rebased and retargeted to Series now: one squashed rebase commit (
Every thread has an individual reply. Pin: temporarily Tests: platform-wallet / platform-wallet-ffi / rs-unified-sdk-jni green except |
|
we are updating this PR and will retest it. |
A persisted address pool has been observed arriving sparse (a 2026-08-19 field wallet was missing BIP44 change rows 875..=890 between surviving ones). Ingesting the sparse list as-is makes every output paying a missing address permanently unrecognizable — no rescan repairs it, because the row-derived `highest_generated` suppresses the gap-limit re-derivation that otherwise would. Derivation is pure key arithmetic, so re-derive every missing index up to the persisted watermark at load. The key source comes from the pool's own account (`accounts.account_of_type`), not from `key_source_for_account_type`. That helper routes through `extended_public_key_for_account_type`, which has no arm for either DashPay variant and so answers `NoKeySource` for both — skipping the repair on exactly the account type that needs it most. `DashpayReceivingFunds` is wallet-owned and funds-bearing, and its account already carries the xpub the missing addresses derive from; an unrepaired hole there makes a contact's payments to us invisible. The helper stays as the fallback for the types `account_of_type` deliberately does not return: the BLS provider operator account and the Ed25519 platform-node account. Never fatal: a failed repair restores exactly what the rows carried. Requires `AddressPool::ensure_contiguous_to` (rust-dashcore dashpay#979). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lassifier dashpay#4638 landed the wallet-wide paged UTXO inventory and the outpoint classifier in platform-wallet, and stopped at the C boundary — the Swift host reaches them, Kotlin does not. These two exports carry them across JNI and no further. Each is a serialization shim over exactly ONE platform-wallet call. The account ordering, the cursor semantics, the page bound, the address rendering and every classification rule stay in `platform-wallet`, so both hosts walk the identical inventory and get identical verdicts. Nothing here decides anything: serde shapes the JSON against the same field names the Kotlin side decodes, so a renamed key fails that decode loudly instead of healing a defaulted row. Paged rather than swept whole because inventory size is chain-controlled: anyone who knows a watched address can keep sending dust to it, and a periodic full-inventory read would let them decide how much a phone allocates on every pass. Classification queries carry the account tuple the STORE files each coin under and the script it recorded. The engine checks that claim against its own pools rather than trusting it, which is what lets it answer `not-owned` — so a bare outpoint cannot ask the question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Room `txos` mirror is write-behind with no feedback loop: a changeset
that fails to deliver an owned output leaves a permanent hole. Because
the engine is REBUILT from this mirror on restart, that hole graduates to
a fund loss on the next launch — observed in the field as the job-flower
106.43 -> 86.33 restart drop, where a rescan nondeterministically drops
the change outputs of sends funded from CoinJoin-account outputs.
Two bounded passes, sharing only the watch-only contact accounts they
both exclude:
* the insert pass walks the ENGINE's paged inventory and inserts the
rows the store lacks, one Room transaction per page. Insert-only and
idempotent, so a sweep is many small commits rather than one large
one. Only outputs at least `minConfirmations` deep are healed;
immature holes age into the next sweep. The engine's own isConfirmed
/ isInstantLocked / isCoinbase travel with each row and are stamped
as given — a coinbase output filed as an ordinary one would misstate
maturity in the mirror the engine reloads from.
* the classification pass is inverted: it pages the STORE and asks the
engine about one page at a time, so neither side builds a set over a
whole inventory. Every verdict is log-only. `known-uncredited` is the
one class that is positive evidence (the owning account knows the
funding txid, owns the script, does not hold the coin, and a funds
account holds a MINED record spending it), and even that is only
counted: flipping spent state is the one direction where a
reconciler bug spends a coin the user still owns. Acting on it is a
separate, reviewable change.
Watch-only DIP-15 contact coins are excluded on both sides — they are the
contact's money. The engine's inventory omits those accounts outright;
the store-side check resolves ownership through both the explicit account
link and the `core_addresses` projection production writes actually use,
because an accountId-only check silently classifies every contact row.
Triggered inside the SDK on the SPV SYNCED transition and every 30
minutes while synced, rather than left to each host: the hole it repairs
becomes a fund loss on the next engine reload if any host forgets to call
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reworked — the Rust half is upstream now#4638 merged to Dropped (~1,070 lines) — upstream provides all of it:
Kept — #4638 touched zero Kotlin files, so the reconcile itself is still Rewritten:
Upstream's classifier also wants the account tuple and script the STORE filed One behaviour improvement fell out for free: upstream's row carries the real Review threadsOf the 25 open, 22 are marked outdated and most pointed at code this rework
The two outdated TestsOn this branch ( The same three commits also pass on a One thing to know before mergingThe address-pool repair calls |
c5cbc44 to
80d61e8
Compare
5 participants
Issue being fixed
The Room store (
txos/transactions/core_addresses) diverged from the engine under rescan, interruption, and delivery loss — and because the engine is rebuilt FROM the store at launch (buildUtxoRestoreData), divergence graduated into visible fund loss on relaunch (testnet field case: 106.43 → 86.33 after restart; root causes fixed engine-side in dashpay/rust-dashcore#979).Stacked on #4406 (this branch's base): the reconcile's shared TXO-insert path absorbs #4406's swept-tombstone semantics, so both writers honor the same rules. Depends on dashpay/rust-dashcore#979: the first commit pins rust-dashcore to that PR's head so CI builds; re-pin to the
devmerge commit once it lands.What was done
reconcileTxoStore— post-sync audit of the store against the engine's full inventory (newwalletManagerAllUtxosJsonJNI export): heals missing TXOs (insert-only, 100-conf gate), repairs the netAmounts those holes falsified, logs every action. Runs on the SPV SYNCED transition and every 30 min. Nonzero heals after fix: same block core chain lock height #979 = regression tripwire.restore_core_address_poolsresolves each pool's key source from the signing wallet and re-derives indices missing from the persisted rows (holes observed in the field made funds rescan-proof invisible; this also closes the rescan-vs-fresh-restore divergence).platform_wallet_account_spent_outpoints). Store rows the engine proves spent are flipped; rows the engine has never seen are LOGGED, never deleted; spent rows the engine disputes are LOGGED, never un-marked (a live spend racing the engine is indistinguishable from lost-release residue, and un-marking could double-spend). Watch-only DIP-15 contact rows are excluded from classification.derive_new_utxosoverupdatedrecords is included for consistency, with a reviewer note thatfrom_changesetre-derives from records and ignorescs.new_utxos— that field may be vestigial and worth a follow-up decision.How this was tested
Merge order: rust-dashcore#979 → re-pin here → #4406 → this.
🤖 Generated with Claude Code