perf(history): persist the sent-sweep watermark (windowed relaunch)

z_listsentbyaddress re-walks the whole wallet when there's no usable
watermark, which the daemon does under cs_main for every send -- hundreds
of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived
only in memory, so every relaunch, reconnect or daemon restart paid for a
full sweep. This persists it so those cases run a bounded windowed sweep
([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1).

- The watermark (W = last swept tip), its anchor (A = W - margin) and anchor
  hash (H_A = getblockhash(A)), plus the enrichment backlog and the full-
  sweep backoff, ride in the encrypted history snapshot as an optional
  `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as
  "no watermark" = today's behaviour, so older builds and payloads are safe).
- Before the first windowed sweep after a load, the anchor is verified
  (getblockhash(A) == H_A). A match means any fork is inside the window; a
  mismatch (reorg deeper than the margin, different chain/datadir) or an
  inability to check drops the watermark and sweeps the whole wallet.
- The first sweep after a load uses a bigger margin (absorbs notifier lag),
  and a watermark older than 2400 blocks uses the long timeout (callLong).
- The backlog is persisted WITH W: a pure z->z send's txid lives only in a
  sweep result and would be lost if the window moved past an un-enriched one.
- Watermark changes reach disk (folded into the store signature), and every
  hard case still forces a full sweep: reorg (anchor verify), GUI rescan /
  zap / import / address change, restoreOriginalWallet, and disconnect.

Deferred (follow-ups, each only turns a correct-but-slow case faster, never a
correctness risk): the banded full sweep for seeding on a pre-note-cache
daemon, the external-import tripwire, and the weekly safety-net re-sweep.
Part 2 (the daemon's decrypt cost) is already on the daemon's
perf/sapling-note-cache branch.

Pure helpers + cache round-trip + anchor-verify/large-window covered by
tests; the getblockhash-after-sweep updates the pinned RPC sequences.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 14:47:48 -05:00
co-authored by Claude Opus 4.8
parent 064c9d1b61
commit 3cda8c9e31
12 changed files with 807 additions and 46 deletions
+14 -2
View File
@@ -28,6 +28,9 @@ constexpr double kHistoryScanDuty = 0.10; // the same 10% as the othe
constexpr int kHistoryMinStaleBlocks = 8; // re-scan an address at most every 8 blocks...
constexpr int kHistoryMaxStaleBlocks = 500; // ...and at least every 500 (a safety net)
constexpr int kSentSweepReorgMarginBlocks = 100; // > MAX_REORG_LENGTH (99), plus tip drift
constexpr int kSentSweepFirstLoadExtraBlocks = 1000; // E: extra margin on the first sweep after a load,
// to absorb wallet-notifier lag at the last save
constexpr int kSentSweepLongCallBlocks = 2400; // tip - W beyond this: use callLong + count failures
constexpr long kFullSentSweepTimeoutSec = 120; // a whole-wallet sweep must be able to finish
constexpr int kTransparentPageMin = 1000;
constexpr int kTransparentPageMax = 5000;
@@ -50,8 +53,17 @@ int transparentPageSize(std::size_t previousRowCount);
// tip after a reorg), else only txs at most (tip - fromHeight) + kSentSweepReorgMarginBlocks deep.
bool sentSweepIsFull(int tip, int fromHeight);
// [z, 0] for the full sweep, [z, 0, 2, maxDepth] for the window: JSON integers, and never 3 params (the
// daemon rejects that count).
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight);
// daemon rejects that count). `margin` widens the window (the first sweep after a load uses a bigger one).
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight,
int margin = kSentSweepReorgMarginBlocks);
// Window margin: steady-state kSentSweepReorgMarginBlocks, bigger on the first sweep after a load.
int sentSweepMargin(bool firstAfterLoad);
// A: the bottom edge the window covers (W - margin), clamped to >= 1. The anchor whose block hash is stored
// (H_A) and re-verified on the next load.
int sentSweepAnchor(int fromHeight, bool firstAfterLoad);
// A watermark saved long ago makes a huge "windowed" call: past this gap, use callLong and count failures
// toward the backoff, so it doesn't time out on every block while the daemon keeps working.
bool sentSweepNeedsLongCall(int tip, int fromHeight);
// Backoff after full sweeps that failed in a way that cost the daemon the work: 0, 10 min, 1 h, then 6 h.
long fullSentSweepRetryDelaySec(int consecutiveFailures);
// Whether a failed full sweep counts toward that backoff: only when the daemon did the work (reply timeout,