DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(history): persist the sent-sweep watermark (windowed relaunch)

Browse Source 
z_listsentbyaddress re-walks the whole wallet when there's no usable
watermark, which the daemon does under cs_main for every send -- hundreds
of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived
only in memory, so every relaunch, reconnect or daemon restart paid for a
full sweep. This persists it so those cases run a bounded windowed sweep
([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1).

- The watermark (W = last swept tip), its anchor (A = W - margin) and anchor
  hash (H_A = getblockhash(A)), plus the enrichment backlog and the full-
  sweep backoff, ride in the encrypted history snapshot as an optional
  `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as
  "no watermark" = today's behaviour, so older builds and payloads are safe).
- Before the first windowed sweep after a load, the anchor is verified
  (getblockhash(A) == H_A). A match means any fork is inside the window; a
  mismatch (reorg deeper than the margin, different chain/datadir) or an
  inability to check drops the watermark and sweeps the whole wallet.
- The first sweep after a load uses a bigger margin (absorbs notifier lag),
  and a watermark older than 2400 blocks uses the long timeout (callLong).
- The backlog is persisted WITH W: a pure z->z send's txid lives only in a
  sweep result and would be lost if the window moved past an un-enriched one.
- Watermark changes reach disk (folded into the store signature), and every
  hard case still forces a full sweep: reorg (anchor verify), GUI rescan /
  zap / import / address change, restoreOriginalWallet, and disconnect.

Deferred (follow-ups, each only turns a correct-but-slow case faster, never a
correctness risk): the banded full sweep for seeding on a pre-note-cache
daemon, the external-import tripwire, and the weekly safety-net re-sweep.
Part 2 (the daemon's decrypt cost) is already on the daemon's
perf/sapling-note-cache branch.

Pure helpers + cache round-trip + anchor-verify/large-window covered by
tests; the getblockhash-after-sweep updates the pinned RPC sequences.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-16 14:47:48 -05:00
co-authored by Claude Opus 4.8
parent 064c9d1b61
commit 3cda8c9e31
12 changed files with 807 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/app_network.cpp
+46 -2
View File
@@ -1888,6 +1888,9 @@ void App::invalidateShieldedHistoryScanProgress(bool persistCache, bool keepSent
recent_probe_rotation_.reset();
if (!keepSentScan) {
history_sent_scan_height_ = -1; // the next sweep covers the whole wallet again
history_sent_anchor_height_ = -1;
history_sent_anchor_hash_.clear();
history_sent_verify_pending_ = false;
history_enrich_backlog_.clear();
history_full_sweep_failures_ = 0;
history_full_sweep_retry_at_ = 0;
@@ -2007,11 +2010,26 @@ void App::loadTransactionHistoryCacheIfAvailable()
}
}
confirmed_cache_block_ = loaded.tipHeight;
// Restore the sent-sweep watermark so the first sweep is windowed, not a 400 s full sweep. A verify is
// owed before the window is trusted (the next sweep checks getblockhash(A) == H_A), and the enrichment
// backlog must come back too: a pure z->z send's txid lives only in a sweep result, and would be lost if
// the window moved past an un-enriched one.
history_sent_scan_height_ = loaded.sentSweep.height;
history_sent_anchor_height_ = loaded.sentSweep.anchorHeight;
history_sent_anchor_hash_ = loaded.sentSweep.anchorHash;
history_sent_verify_pending_ = loaded.sentSweep.valid();
history_enrich_backlog_.clear();
history_enrich_backlog_.insert(loaded.sentSweep.backlog.begin(), loaded.sentSweep.backlog.end());
history_full_sweep_failures_ = loaded.sentSweep.fullFailures;
history_full_sweep_retry_at_ = loaded.sentSweep.fullRetryDelaySec > 0
? std::time(nullptr) + loaded.sentSweep.fullRetryDelaySec : 0;
++history_sent_epoch_; // the loaded watermark replaces any in-flight sweep's result
transaction_history_cache_loaded_ = true;
// What is on disk is exactly what was just loaded: the first identical refresh must not rewrite it.
history_store_.sig = data::historySnapshotSignature(
state_.transactions, {}, data::TransactionHistoryCache::walletIdentityHash(walletIdentity),
transaction_history_cache_.keyId(), shielded_history_scan_heights_, loaded.tipHeight, loaded.tipHash);
transaction_history_cache_.keyId(), shielded_history_scan_heights_, loaded.tipHeight, loaded.tipHash,
history_sent_scan_height_, history_sent_anchor_height_, history_sent_anchor_hash_, history_enrich_backlog_);
history_store_.have = true;
history_store_.lastSubmit = std::chrono::steady_clock::now();
history_store_pending_ = false;
@@ -2040,7 +2058,9 @@ void App::storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode mode)
state_.transactions, pendingOpids,
data::TransactionHistoryCache::walletIdentityHash(walletIdentity),
transaction_history_cache_.keyId(), shielded_history_scan_heights_,
state_.sync.blocks, state_.sync.best_blockhash);
state_.sync.blocks, state_.sync.best_blockhash,
history_sent_scan_height_, history_sent_anchor_height_, history_sent_anchor_hash_,
history_enrich_backlog_);
if (sig.rows == 0) return;
const auto now = std::chrono::steady_clock::now();
@@ -2077,6 +2097,18 @@ void App::storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode mode)
if (pendingOpids.find(transaction.txid) != pendingOpids.end()) continue;
job->transactions.push_back(transaction);
}
// The sent-sweep watermark + enrichment backlog travel with the rows (writeSnapshot emits them only
// when self-consistent). The backoff is stored as a delay relative to now (retry_at is absolute).
job->sentSweep.height = history_sent_scan_height_;
job->sentSweep.anchorHeight = history_sent_anchor_height_;
job->sentSweep.anchorHash = history_sent_anchor_hash_;
job->sentSweep.backlog.assign(history_enrich_backlog_.begin(), history_enrich_backlog_.end());
job->sentSweep.fullFailures = history_full_sweep_failures_;
{
const std::time_t nowSec = std::time(nullptr);
job->sentSweep.fullRetryDelaySec = history_full_sweep_retry_at_ > nowSec
? static_cast<long>(history_full_sweep_retry_at_ - nowSec) : 0;
}
auto* cache = &transaction_history_cache_; // App member: outlives every job (see app.h)
history_writer_.submit("tx-history", [cache, job]() { return cache->writeSnapshot(std::move(*job)); });
@@ -2592,6 +2624,9 @@ void App::refreshAddressData(services::ScanTicket ticket)
}
if (newUnscannedZ && (!previousZ.empty() || !history_listed_z_.empty())) {
history_sent_scan_height_ = -1;
history_sent_anchor_height_ = -1;
history_sent_anchor_hash_.clear();
history_sent_verify_pending_ = false;
history_full_sweep_failures_ = 0;
history_full_sweep_retry_at_ = 0;
++history_sent_epoch_;
@@ -2777,6 +2812,9 @@ void App::startTransactionCycle(bool full, bool routineAllowed, services::ScanTi
}
}
transactionSnapshot.sentScanFromHeight = history_sent_scan_height_;
transactionSnapshot.sentVerifyPending = history_sent_verify_pending_;
transactionSnapshot.sentVerifyAnchorHeight = history_sent_anchor_height_;
transactionSnapshot.sentVerifyAnchorHash = history_sent_anchor_hash_;
// A windowed sweep mid-pass too, now and then, so an untracked send (the Console, another wallet on
// the same seed) doesn't wait for a whole pass on a large wallet: at most every quarter of the pass
// cadence, and only in cycles the throttle allows.
@@ -2844,6 +2882,9 @@ void App::startTransactionCycle(bool full, bool routineAllowed, services::ScanTi
if (sameSentEpoch) {
if (result.sentSweepOk && result.sentSweepHeight > 0) {
history_sent_scan_height_ = result.sentSweepHeight;
history_sent_anchor_height_ = result.sentSweepAnchorHeight;
history_sent_anchor_hash_ = result.sentSweepAnchorHash;
history_sent_verify_pending_ = false; // this sweep covered the window; nothing owed
history_full_sweep_failures_ = 0;
history_full_sweep_retry_at_ = 0;
} else if (result.sentSweepRan && result.sentSweepFull &&
@@ -6562,6 +6603,9 @@ void App::restoreOriginalWallet()
connection_status_ = TR("sb_restarting_daemon");
abortMaintenanceCalls(); // a long call holds the curl lock disconnect() needs
if (rpc_ && rpc_->isConnected()) rpc_->disconnect();
// Swapping the wallet + restarting with -rescan changes the history: invalidate the sent-sweep watermark
// (and persist that) before the disconnect, as rescanBlockchain() and the zap path already do.
invalidateShieldedHistoryScanProgress(true);
onDisconnected("Restoring original wallet");
ui::Notifications::instance().info(TR("wallet_restore_started"), 12.0f);