wallet/main: break the replay -> witness-rebuild -> kill loop on large wallets

A v1.3.0 node with a ~13.6k-note wallet (-dbcache=4084) looked hung for hours:
every start replayed ~62k blocks from a chainstate 26 days stale, then ran a
47-60 min all-core witness rebuild holding LOCK2(cs_main, cs_wallet) (RPC answered
HTTP 500 "Work queue depth exceeded", peers timed out), the GUI killed it before
anything was flushed, and the next start did it all again. Even after escaping,
each new block cost ~30s under LOCK2, so the node fell behind a 36s chain.

D1  BuildWitnessCache: the post-rebuild latch computed witnesses.front().root()
    (~2.4ms) for every pending note on every block, only to feed a log line that
    is printed for bulk rebuilds alone. The anchor check now runs on the Phase-2
    workers and only for a bulk rebuild; the per-block path just sets the latch.
    Phase 2 also runs in parallel whenever the job is big (notes x commitments
    >= 2000), not only for a bulk range, so busy blocks no longer extend every
    note serially. If the OS refuses a worker thread, the remaining shards run
    inline instead of unwinding past joinable threads (std::terminate).

D2  FlushStateToDisk: the coins cache was only flushed on shutdown, gettxoutsetinfo
    or after 24h; on this chain it stays a few MiB, so the size triggers never
    fire and a node that never reached 24h uptime or a clean stop kept its
    chainstate (and block index) weeks behind. Flush at least hourly and every
    DATABASE_FLUSH_BLOCKS (5000) connected blocks -- the latter also inside one
    long ActivateBestChain -- and right after the startup ActivateBestChain when
    it moved the tip.

D3  UpdatedNoteData: a replayed or rescanned tx arrives with FindMySaplingNotes
    data (no nullifier). Witnesses and witnessHeight were carried over but the
    nullifier was dropped, so VerifyAndSetInitialWitness cleared and reseeded the
    witnesses on the next block -- the replay above rewrote 3,750 txs and forced
    the full rebuild. Carry the nullifier with the witnesses, unless the tx was
    re-mined in a different block (its notes moved; reseed as before). When the
    merged data is identical, don't rewrite the tx; just clear the in-memory
    witnessRootValidated so the next block re-checks the kept roots once (a
    rescan still finds and reseeds a witness that went bad, without rebuilding
    the healthy ones). The carry loop no longer deep-copies every note.

    SetBestChain: with hourly coin flushes the wallet locator must not run ahead
    of the async notifier, or a crash permanently skips the blocks it had not yet
    delivered. Stamp the block the wallet's state reflects (pindexWalletTip,
    seeded in init before ThreadNotifyWallets starts -- at the startup rescan's
    checkpoint if that rescan was interrupted -- and advanced contiguously by
    ChainTip), like zcashd's ChainTipAdded. ThreadNotifyWallets no longer hands
    the wallet an empty block after a fatal block-read failure.

Tests: new gtest test_wallet_notedata.cpp (5 tests); 23/23 gtests pass, and the
two core D3 tests fail with D3 reverted. Regtest A/B against master with a
1,008-note wallet: notify-thread CPU per block 1,780ms -> 3ms; worst LOCK2 RPC
stall 1.6-2.9s -> 10-15ms on ordinary blocks and 4.6-7.1s -> ~1.6s on a block
carrying a 40-output self-send; a z_importkey rescan went from 35 reseeds plus a
bulk rebuild (10.6s) to 0 reseeds, same balance; after kill -9 at 150 blocks a
build with a 100-block flush threshold restarted at 100, master at 0. Two
adversarial review rounds; every confirmed finding is addressed here.

Not covered: the hourly SetBestChain/flush paths were reviewed, not exercised
(GetTimeMicros ignores mocktime); no run yet against a copy of a real large wallet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-30 19:05:45 -05:00
co-authored by Claude Opus 5.5
parent da191e447d
commit 69bb035e7e
8 changed files with 353 additions and 40 deletions
+7 -1
View File
@@ -965,8 +965,12 @@ private:
template <class T>
void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>);
// Last block ThreadNotifyWallets delivered to ChainTip, i.e. the block the wallet's state
// reflects (guarded by cs_wallet). SetBestChain stamps this instead of the chain tip.
const CBlockIndex* pindexWalletTip = nullptr;
protected:
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx);
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock);
void MarkAffectedTransactionsDirty(const CTransaction& tx);
/* the hd chain data model (chain counters) */
@@ -1294,6 +1298,8 @@ public:
bool CommitAutomatedTx(const CTransaction& tx);
/** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */
void SetBestChain(const CBlockLocator& loc);
// Seed the block the wallet's state reflects, before ThreadNotifyWallets starts delivering.
void SetWalletTip(const CBlockIndex* pindex);
/** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false
* so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment
* at CWallet::SetBestChain), and reports whether the write actually committed. */