DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(history): persist the sent-sweep watermark (windowed relaunch)

Browse Source 
z_listsentbyaddress re-walks the whole wallet when there's no usable
watermark, which the daemon does under cs_main for every send -- hundreds
of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived
only in memory, so every relaunch, reconnect or daemon restart paid for a
full sweep. This persists it so those cases run a bounded windowed sweep
([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1).

- The watermark (W = last swept tip), its anchor (A = W - margin) and anchor
  hash (H_A = getblockhash(A)), plus the enrichment backlog and the full-
  sweep backoff, ride in the encrypted history snapshot as an optional
  `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as
  "no watermark" = today's behaviour, so older builds and payloads are safe).
- Before the first windowed sweep after a load, the anchor is verified
  (getblockhash(A) == H_A). A match means any fork is inside the window; a
  mismatch (reorg deeper than the margin, different chain/datadir) or an
  inability to check drops the watermark and sweeps the whole wallet.
- The first sweep after a load uses a bigger margin (absorbs notifier lag),
  and a watermark older than 2400 blocks uses the long timeout (callLong).
- The backlog is persisted WITH W: a pure z->z send's txid lives only in a
  sweep result and would be lost if the window moved past an un-enriched one.
- Watermark changes reach disk (folded into the store signature), and every
  hard case still forces a full sweep: reorg (anchor verify), GUI rescan /
  zap / import / address change, restoreOriginalWallet, and disconnect.

Deferred (follow-ups, each only turns a correct-but-slow case faster, never a
correctness risk): the banded full sweep for seeding on a pre-note-cache
daemon, the external-import tripwire, and the weekly safety-net re-sweep.
Part 2 (the daemon's decrypt cost) is already on the daemon's
perf/sapling-note-cache branch.

Pure helpers + cache round-trip + anchor-verify/large-window covered by
tests; the getblockhash-after-sweep updates the pinned RPC sequences.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-16 14:47:48 -05:00
co-authored by Claude Opus 4.8
parent 064c9d1b61
commit 3cda8c9e31
12 changed files with 807 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/history_scan_policy.cpp
+18 -2
View File
@@ -37,18 +37,34 @@ bool sentSweepIsFull(int tip, int fromHeight)
return tip <= 0 || fromHeight < 0 || fromHeight > tip;
}
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight)
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight, int margin)
{
nlohmann::json params = nlohmann::json::array();
params.push_back(anyZAddress);
params.push_back(0);
if (sentSweepIsFull(tip, fromHeight)) return params;
const int maxDepth = (tip - fromHeight) + kSentSweepReorgMarginBlocks;
const int maxDepth = (tip - fromHeight) + std::max(0, margin);
params.push_back(2); // filter type 2: depth
params.push_back(maxDepth); // at most this deep (mempool txs, depth 0, always pass)
return params;
}
int sentSweepMargin(bool firstAfterLoad)
{
return firstAfterLoad ? (kSentSweepReorgMarginBlocks + kSentSweepFirstLoadExtraBlocks)
: kSentSweepReorgMarginBlocks;
}
int sentSweepAnchor(int fromHeight, bool firstAfterLoad)
{
return std::max(1, fromHeight - sentSweepMargin(firstAfterLoad));
}
bool sentSweepNeedsLongCall(int tip, int fromHeight)
{
return tip > 0 && fromHeight >= 0 && (tip - fromHeight) > kSentSweepLongCallBlocks;
}
long fullSentSweepRetryDelaySec(int consecutiveFailures)
{
if (consecutiveFailures <= 0) return 0;
src/services/history_scan_policy.h
+14 -2
View File
@@ -28,6 +28,9 @@ constexpr double kHistoryScanDuty = 0.10; // the same 10% as the othe
constexpr int kHistoryMinStaleBlocks = 8; // re-scan an address at most every 8 blocks...
constexpr int kHistoryMaxStaleBlocks = 500; // ...and at least every 500 (a safety net)
constexpr int kSentSweepReorgMarginBlocks = 100; // > MAX_REORG_LENGTH (99), plus tip drift
constexpr int kSentSweepFirstLoadExtraBlocks = 1000; // E: extra margin on the first sweep after a load,
// to absorb wallet-notifier lag at the last save
constexpr int kSentSweepLongCallBlocks = 2400; // tip - W beyond this: use callLong + count failures
constexpr long kFullSentSweepTimeoutSec = 120; // a whole-wallet sweep must be able to finish
constexpr int kTransparentPageMin = 1000;
constexpr int kTransparentPageMax = 5000;
@@ -50,8 +53,17 @@ int transparentPageSize(std::size_t previousRowCount);
// tip after a reorg), else only txs at most (tip - fromHeight) + kSentSweepReorgMarginBlocks deep.
bool sentSweepIsFull(int tip, int fromHeight);
// [z, 0] for the full sweep, [z, 0, 2, maxDepth] for the window: JSON integers, and never 3 params (the
// daemon rejects that count).
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight);
// daemon rejects that count). `margin` widens the window (the first sweep after a load uses a bigger one).
nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight,
int margin = kSentSweepReorgMarginBlocks);
// Window margin: steady-state kSentSweepReorgMarginBlocks, bigger on the first sweep after a load.
int sentSweepMargin(bool firstAfterLoad);
// A: the bottom edge the window covers (W - margin), clamped to >= 1. The anchor whose block hash is stored
// (H_A) and re-verified on the next load.
int sentSweepAnchor(int fromHeight, bool firstAfterLoad);
// A watermark saved long ago makes a huge "windowed" call: past this gap, use callLong and count failures
// toward the backoff, so it doesn't time out on every block while the daemon keeps working.
bool sentSweepNeedsLongCall(int tip, int fromHeight);
// Backoff after full sweeps that failed in a way that cost the daemon the work: 0, 10 min, 1 h, then 6 h.
long fullSentSweepRetryDelaySec(int consecutiveFailures);
// Whether a failed full sweep counts toward that backoff: only when the daemon did the work (reply timeout,
src/services/network_refresh_service.cpp
+54 -25
View File
@@ -1314,32 +1314,61 @@ NetworkRefreshService::TransactionRefreshResult NetworkRefreshService::collectTr
const bool sweepIsFull = sentSweepIsFull(currentBlockHeight, snapshot.sentScanFromHeight);
if (!snapshot.shieldedAddresses.empty() && !snapshot.skipSentSweep &&
(result.shieldedScanComplete || (snapshot.allowMidPassSentSweep && !sweepIsFull))) {
result.sentSweepRan = true;
result.sentSweepFull = sweepIsFull;
const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight,
snapshot.sentScanFromHeight);
const auto sweepStart = std::chrono::steady_clock::now();
try {
json sent = result.sentSweepFull
? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec)
: rpc.call("z_listsentbyaddress", params);
if (sent.is_array()) {
for (const auto& sentTx : sent) {
auto txid = readOptional<std::string>(sentTx, "txid");
if (!txid || txid->empty()) continue;
appendSentTransparentOutputs(result.transactions, *txid, sentTx);
knownTxids.insert(*txid); // shielded (z->z) legs enriched via z_viewtransaction
}
result.sentSweepOk = true;
result.sentSweepHeight = currentBlockHeight;
}
} catch (const std::exception& e) {
transactionRpcError = true;
result.sentSweepErrKind = rpc::rpcFailureKindOf(e);
result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout;
DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what());
int effectiveFromHeight = snapshot.sentScanFromHeight;
bool effectiveFull = sweepIsFull;
// Anchor verify: a watermark restored from disk is trusted only if the block at A still matches the
// stored H_A (no reorg deeper than the margin, same chain/datadir). A mismatch, or an inability to
// check, drops the watermark and forces a full sweep. Checked only for a windowed sweep.
if (snapshot.sentVerifyPending && !sweepIsFull && snapshot.sentVerifyAnchorHeight > 0) {
bool anchorOk = false;
try {
json h = rpc.call("getblockhash", json::array({snapshot.sentVerifyAnchorHeight}));
if (h.is_string()) anchorOk = (h.get<std::string>() == snapshot.sentVerifyAnchorHash);
} catch (...) { anchorOk = false; }
if (!anchorOk) { effectiveFromHeight = -1; effectiveFull = true; }
}
// A verify that turned a windowed mid-pass sweep into a full one must wait for a completed pass,
// rather than run a whole-wallet sweep in the middle of one.
if (!(effectiveFull && !result.shieldedScanComplete)) {
result.sentSweepRan = true;
result.sentSweepFull = effectiveFull;
// Bigger margin on the first windowed sweep after a load (absorbs wallet-notifier lag at save).
const int margin = sentSweepMargin(snapshot.sentVerifyPending && !effectiveFull);
const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight,
effectiveFromHeight, margin);
// A watermark saved long ago makes a huge "windowed" call; give it the long timeout too.
const bool longCall = effectiveFull || sentSweepNeedsLongCall(currentBlockHeight, effectiveFromHeight);
const auto sweepStart = std::chrono::steady_clock::now();
try {
json sent = longCall
? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec)
: rpc.call("z_listsentbyaddress", params);
if (sent.is_array()) {
for (const auto& sentTx : sent) {
auto txid = readOptional<std::string>(sentTx, "txid");
if (!txid || txid->empty()) continue;
appendSentTransparentOutputs(result.transactions, *txid, sentTx);
knownTxids.insert(*txid); // shielded (z->z) legs enriched via z_viewtransaction
}
result.sentSweepOk = true;
result.sentSweepHeight = currentBlockHeight;
// Anchor for the NEXT load's verify: A = W - steady margin, H_A = getblockhash(A). An
// empty H_A (fetch failed) just means the next load can't verify and does a full sweep.
const int newAnchor = sentSweepAnchor(currentBlockHeight, false);
result.sentSweepAnchorHeight = newAnchor;
try {
json ha = rpc.call("getblockhash", json::array({newAnchor}));
if (ha.is_string()) result.sentSweepAnchorHash = ha.get<std::string>();
} catch (...) {}
}
} catch (const std::exception& e) {
transactionRpcError = true;
result.sentSweepErrKind = rpc::rpcFailureKindOf(e);
result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout;
DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what());
}
result.sentSweepMs = msSince(sweepStart);
}
result.sentSweepMs = msSince(sweepStart);
}
for (const auto& txid : snapshot.sendTxids) {
src/services/network_refresh_service.h
+10 -2
View File
@@ -254,7 +254,13 @@ public:
std::unordered_map<std::string, int> priorityScanFloors;
bool skipShieldedScans = false; // a Light cycle (or a probe-less recent poll): no receive scans
bool skipSentSweep = false;
int sentScanFromHeight = -1; // the sweep's watermark; -1 = sweep the whole wallet
int sentScanFromHeight = -1; // the sweep's watermark (W); -1 = sweep the whole wallet
// A loaded watermark owes an anchor verify before its first windowed sweep: getblockhash(A) must
// still equal H_A (no reorg deeper than the margin). A mismatch forces a full sweep. While pending,
// the window also uses the bigger first-after-load margin.
bool sentVerifyPending = false;
int sentVerifyAnchorHeight = -1; // A
std::string sentVerifyAnchorHash; // H_A
std::size_t transparentRowHint = 0; // listtransactions rows last time: sizes the first page
// Same tip and nothing changed since the rows on screen were fetched: reuse their transparent rows
// instead of paging listtransactions again.
@@ -291,7 +297,9 @@ public:
bool sentSweepRan = false;
bool sentSweepOk = false;
bool sentSweepFull = false;
int sentSweepHeight = -1; // the tip a successful sweep covered
int sentSweepHeight = -1; // W: the tip a successful sweep covered
int sentSweepAnchorHeight = -1; // A = W - margin, the bottom edge the window covered
std::string sentSweepAnchorHash; // H_A = getblockhash(A), fetched after a successful sweep
double sentSweepMs = 0.0; // time in the sweep (part of scanMs)
rpc::RpcFailureKind sentSweepErrKind = rpc::RpcFailureKind::None;
// Txids found but not enriched (over the per-cycle cap, or the view failed transiently): the caller