perf(history): persist the sent-sweep watermark (windowed relaunch)

z_listsentbyaddress re-walks the whole wallet when there's no usable watermark, which the daemon does under cs_main for every send -- hundreds of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived only in memory, so every relaunch, reconnect or daemon restart paid for a full sweep. This persists it so those cases run a bounded windowed sweep ([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1). - The watermark (W = last swept tip), its anchor (A = W - margin) and anchor hash (H_A = getblockhash(A)), plus the enrichment backlog and the full- sweep backoff, ride in the encrypted history snapshot as an optional `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as "no watermark" = today's behaviour, so older builds and payloads are safe). - Before the first windowed sweep after a load, the anchor is verified (getblockhash(A) == H_A). A match means any fork is inside the window; a mismatch (reorg deeper than the margin, different chain/datadir) or an inability to check drops the watermark and sweeps the whole wallet. - The first sweep after a load uses a bigger margin (absorbs notifier lag), and a watermark older than 2400 blocks uses the long timeout (callLong). - The backlog is persisted WITH W: a pure z->z send's txid lives only in a sweep result and would be lost if the window moved past an un-enriched one. - Watermark changes reach disk (folded into the store signature), and every hard case still forces a full sweep: reorg (anchor verify), GUI rescan / zap / import / address change, restoreOriginalWallet, and disconnect. Deferred (follow-ups, each only turns a correct-but-slow case faster, never a correctness risk): the banded full sweep for seeding on a pre-note-cache daemon, the external-import tripwire, and the weekly safety-net re-sweep. Part 2 (the daemon's decrypt cost) is already on the daemon's perf/sapling-note-cache branch. Pure helpers + cache round-trip + anchor-verify/large-window covered by tests; the getblockhash-after-sweep updates the pinned RPC sequences. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-16 14:47:48 -05:00
parent 064c9d1b61
commit 3cda8c9e31
12 changed files with 807 additions and 46 deletions
@@ -1314,32 +1314,61 @@ NetworkRefreshService::TransactionRefreshResult NetworkRefreshService::collectTr
    const bool sweepIsFull = sentSweepIsFull(currentBlockHeight, snapshot.sentScanFromHeight);
    if (!snapshot.shieldedAddresses.empty() && !snapshot.skipSentSweep &&
        (result.shieldedScanComplete || (snapshot.allowMidPassSentSweep && !sweepIsFull))) {
-        result.sentSweepRan = true;
-        result.sentSweepFull = sweepIsFull;
-        const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight,
-                                            snapshot.sentScanFromHeight);
-        const auto sweepStart = std::chrono::steady_clock::now();
-        try {
-            json sent = result.sentSweepFull
-                ? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec)
-                : rpc.call("z_listsentbyaddress", params);
-            if (sent.is_array()) {
-                for (const auto& sentTx : sent) {
-                    auto txid = readOptional<std::string>(sentTx, "txid");
-                    if (!txid || txid->empty()) continue;
-                    appendSentTransparentOutputs(result.transactions, *txid, sentTx);
-                    knownTxids.insert(*txid);  // shielded (z->z) legs enriched via z_viewtransaction
-                }
-                result.sentSweepOk = true;
-                result.sentSweepHeight = currentBlockHeight;
-            }
-        } catch (const std::exception& e) {
-            transactionRpcError = true;
-            result.sentSweepErrKind = rpc::rpcFailureKindOf(e);
-            result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout;
-            DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what());
+        int effectiveFromHeight = snapshot.sentScanFromHeight;
+        bool effectiveFull = sweepIsFull;
+        // Anchor verify: a watermark restored from disk is trusted only if the block at A still matches the
+        // stored H_A (no reorg deeper than the margin, same chain/datadir). A mismatch, or an inability to
+        // check, drops the watermark and forces a full sweep. Checked only for a windowed sweep.
+        if (snapshot.sentVerifyPending && !sweepIsFull && snapshot.sentVerifyAnchorHeight > 0) {
+            bool anchorOk = false;
+            try {
+                json h = rpc.call("getblockhash", json::array({snapshot.sentVerifyAnchorHeight}));
+                if (h.is_string()) anchorOk = (h.get<std::string>() == snapshot.sentVerifyAnchorHash);
+            } catch (...) { anchorOk = false; }
+            if (!anchorOk) { effectiveFromHeight = -1; effectiveFull = true; }
+        }
+        // A verify that turned a windowed mid-pass sweep into a full one must wait for a completed pass,
+        // rather than run a whole-wallet sweep in the middle of one.
+        if (!(effectiveFull && !result.shieldedScanComplete)) {
+            result.sentSweepRan = true;
+            result.sentSweepFull = effectiveFull;
+            // Bigger margin on the first windowed sweep after a load (absorbs wallet-notifier lag at save).
+            const int margin = sentSweepMargin(snapshot.sentVerifyPending && !effectiveFull);
+            const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight,
+                                                effectiveFromHeight, margin);
+            // A watermark saved long ago makes a huge "windowed" call; give it the long timeout too.
+            const bool longCall = effectiveFull || sentSweepNeedsLongCall(currentBlockHeight, effectiveFromHeight);
+            const auto sweepStart = std::chrono::steady_clock::now();
+            try {
+                json sent = longCall
+                    ? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec)
+                    : rpc.call("z_listsentbyaddress", params);
+                if (sent.is_array()) {
+                    for (const auto& sentTx : sent) {
+                        auto txid = readOptional<std::string>(sentTx, "txid");
+                        if (!txid || txid->empty()) continue;
+                        appendSentTransparentOutputs(result.transactions, *txid, sentTx);
+                        knownTxids.insert(*txid);  // shielded (z->z) legs enriched via z_viewtransaction
+                    }
+                    result.sentSweepOk = true;
+                    result.sentSweepHeight = currentBlockHeight;
+                    // Anchor for the NEXT load's verify: A = W - steady margin, H_A = getblockhash(A). An
+                    // empty H_A (fetch failed) just means the next load can't verify and does a full sweep.
+                    const int newAnchor = sentSweepAnchor(currentBlockHeight, false);
+                    result.sentSweepAnchorHeight = newAnchor;
+                    try {
+                        json ha = rpc.call("getblockhash", json::array({newAnchor}));
+                        if (ha.is_string()) result.sentSweepAnchorHash = ha.get<std::string>();
+                    } catch (...) {}
+                }
+            } catch (const std::exception& e) {
+                transactionRpcError = true;
+                result.sentSweepErrKind = rpc::rpcFailureKindOf(e);
+                result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout;
+                DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what());
+            }
+            result.sentSweepMs = msSince(sweepStart);
        }
-        result.sentSweepMs = msSince(sweepStart);
    }

    for (const auto& txid : snapshot.sendTxids) {