perf(history): scan shielded history in bounded passes, not every block

Every block re-ran z_listreceivedbyaddress for every z-address (O(wallet)
each, under cs_main) and a whole-wallet z_listsentbyaddress. On a 40-address
wallet that was ~80% of wall-clock. The history now advances in passes:

- an address is rescanned when it is stale by a number of blocks scaled to
  its measured cost (8 to 500), never-scanned and hinted addresses first
  (a hint: z_listunspent shows a note the history doesn't);
- z_listsentbyaddress covers only the blocks since the last good sweep
  (+100 reorg margin); the whole-wallet sweep runs once per connect, with
  a 10 min / 1 h / 6 h backoff after real failures;
- listtransactions pages are sized from last time, and an unchanged tip
  reuses the transparent rows;
- found-but-unenriched txids carry over as a backlog; only -5 is a
  definitive "not ours" for z_viewtransaction;
- between blocks, a cheap recent poll plus a one-address probe.

Duty model for a 40-address wallet at 10 s per scan: ~9% instead of ~80%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 11:06:34 -05:00
co-authored by Claude Opus 5.5
parent 8b9a498eee
commit 879de37d18
10 changed files with 1645 additions and 233 deletions
+44 -7
View File
@@ -26,6 +26,7 @@
#include "daemon/stop_delivery.h"
#include "daemon/daemon_busy_detect.h"
#include "rpc/rpc_busy_gate.h"
#include "services/history_scan_policy.h"
#include "services/network_refresh_service.h"
#include "services/wallet_security_controller.h"
#include "services/wallet_security_workflow.h"
@@ -807,8 +808,13 @@ private:
void storeTransactionHistoryCacheIfAvailable();
void wipePendingTransactionHistoryCachePassphrase();
void resetTransactionHistoryCacheSession();
// The wallet locked: drop the cache key, keep the history and its scan progress (an unlock used to
// force a full re-scan of every address, every auto-lock cycle).
void lockTransactionHistoryCacheSession();
void pruneShieldedHistoryScanProgress();
void invalidateShieldedHistoryScanProgress(bool persistCache);
// Forget the receive-scan heights so every address is scanned again. keepSentScan leaves the sent
// sweep's watermark (and the enrichment backlog), so a plain refresh doesn't cost a whole-wallet sweep.
void invalidateShieldedHistoryScanProgress(bool persistCache, bool keepSentScan = false);
// Auto-balance pool selection: drive the periodic hashrate refresh and apply a
// freshly-completed snapshot (weighted-random pick + optional miner restart).
@@ -1361,7 +1367,10 @@ private:
// addressRefreshDue() / txRefreshDue(); only the routine periodic poll is throttled — explicit
// refreshes (tab switch, dirty set, in-flight send) call the refresh directly and bypass this.
double last_address_scan_ms_ = 0.0; // measured cost of the last address scan
double last_tx_scan_ms_ = 0.0; // measured cost of the last history scan
double last_tx_scan_ms_ = 0.0; // measured cost of the last wallet-wide history scan
// When that scan ran (wall-clock): the history throttle's clock. Not state_.last_tx_update, which
// every pending-send row edit and probe-less poll also stamps (that pushed the next scan out).
std::int64_t last_tx_scan_at_ = 0;
// Auto-clear for secrets copied to the clipboard. Only a hash of the copied secret is kept.
std::uint64_t clipboard_secret_hash_ = 0;
double clipboard_clear_deadline_ = 0.0;
@@ -1488,12 +1497,32 @@ private:
int last_tx_block_height_ = -1; // block height at last full tx fetch
static constexpr int MAX_VIEWTX_PER_CYCLE = 25; // cap z_viewtransaction calls per refresh
std::size_t shielded_history_scan_cursor_ = 0;
bool shielded_history_scan_pending_ = false;
// False until the first full shielded-history scan finishes. Drives the History tab's
// "Loading older history…" progress so the user knows transactions are still streaming in
// after the first batch appears; goes quiet for the routine per-block re-scans afterward.
bool initial_history_scan_complete_ = false;
std::unordered_map<std::string, int> shielded_history_scan_heights_;
// The receive-scan pass over every z-address (see services::ShieldedHistoryPass).
services::ShieldedHistoryPass history_pass_;
std::uint64_t history_scan_epoch_ = 0; // bumped when the scan heights are invalidated
std::uint64_t history_sent_epoch_ = 0; // bumped when the sent sweep's watermark is reset
int history_sent_scan_height_ = -1; // tip of the last successful z_listsentbyaddress; -1 = none
int history_full_sweep_failures_ = 0; // whole-wallet sweeps failed in a row (backoff)
std::time_t history_full_sweep_retry_at_ = 0;
std::unordered_set<std::string> history_enrich_backlog_; // txids found, not enriched yet
double history_addr_scan_ms_ = 0.0; // per-address z_listreceivedbyaddress cost (estimate)
std::size_t history_transparent_rows_ = 0; // rows the last listtransactions chain returned
int history_transparent_tip_ = -1; // ...and the tip it ran at (same tip: rows can be reused)
std::uint64_t tx_dirty_generation_ = 0; // bumped by markTransactionsDirty()
// Addresses holding notes the history doesn't show yet (from the address scan's z_listunspent): scanned
// first in the next wallet-wide cycle, so a receive or payout shows within one cycle on a large wallet
// instead of when the routine pass reaches its address. Each txid hints once per session after it is
// mined (a mempool hint doesn't use that up, so the row also catches its confirmation).
std::unordered_map<std::string, int> history_hint_floors_;
std::unordered_set<std::string> history_hinted_txids_;
// The z-addresses of the last successful address enumeration this session: a never-scanned address new
// to it (an imported key) resets the sent sweep's watermark, even if the list in between came back blank.
std::unordered_set<std::string> history_listed_z_;
// P4b: z_viewtransaction result cache — avoids re-calling the RPC for
// txids we've already enriched. Keyed by txid.
@@ -1733,8 +1762,17 @@ private:
void refreshData(); // Orchestrator: dispatches per-category refreshes
void refreshCoreData(bool allowBalance = true); // Balance + blockchain info (can use fast_worker_)
void refreshAddressData(); // Address lists + balances
void refreshTransactionData(); // Transaction list + z_viewtransaction enrichment
void refreshRecentTransactionData(); // Lightweight recent/unconfirmed tx poll
// Transaction history: decides the cycle (services::decideTxCycle) and starts it. Safe to call any
// time: it refuses when the node can't be scanned (not connected, locked, warming up, rescanning, busy).
void refreshTransactionData();
void startTransactionCycle(bool full, bool routineAllowed); // Full or Light cycle
void startRecentTransactionPoll(bool probe); // between-blocks poll
void markTransactionsDirty() { transactions_dirty_ = true; ++tx_dirty_generation_; }
std::vector<std::string> shieldedAddressList() const;
services::ShieldedHistoryPass::Work historyScanWork() const;
// From the address scan's z_listunspent: addresses holding received notes the history doesn't show
// yet (no row, or rows still unconfirmed while the note is mined) become priority scans.
void noteHistoryHintsFromUnspent(const std::vector<services::NetworkRefreshService::UnspentNoteLite>& notes);
bool refreshEncryptionState(); // Wallet encryption/lock state
void refreshBalance(); // Legacy: balance-only refresh (used by specific callers)
void refreshAddresses(); // Legacy: standalone address refresh
@@ -1744,13 +1782,12 @@ private:
bool effectivelySyncing() const; // syncing, or within the post-sync settle window (hysteresis)
bool balanceRefreshDue() const; // adaptive: enough time elapsed given the last balance-scan cost?
bool addressRefreshDue() const; // same adaptive back-off for the address scan (z_listunspent)
bool txRefreshDue() const; // same adaptive back-off for the history scan (z_listreceivedbyaddress)
bool txRefreshDue() const; // same adaptive back-off for wallet-wide history scans
// Shared duty-cycle rule: a scan may resume only once (lastScanMs / kScanDutyCycle) has elapsed since
// lastUpdate, so any single O(mapWallet) scan can occupy at most ~kScanDutyCycle of wall-clock.
bool scanRefreshDue(std::int64_t lastUpdate, double lastScanMs) const;
bool currentPageNeedsWalletDataRefresh() const;
bool shouldRunWalletTransactionRefresh() const;
bool shouldRefreshTransactions() const;
bool shouldRefreshRecentTransactions() const;
void checkAutoLock();
void checkIdleMining();