perf(history): scan shielded history in bounded passes, not every block

Every block re-ran z_listreceivedbyaddress for every z-address (O(wallet)
each, under cs_main) and a whole-wallet z_listsentbyaddress. On a 40-address
wallet that was ~80% of wall-clock. The history now advances in passes:

- an address is rescanned when it is stale by a number of blocks scaled to
  its measured cost (8 to 500), never-scanned and hinted addresses first
  (a hint: z_listunspent shows a note the history doesn't);
- z_listsentbyaddress covers only the blocks since the last good sweep
  (+100 reorg margin); the whole-wallet sweep runs once per connect, with
  a 10 min / 1 h / 6 h backoff after real failures;
- listtransactions pages are sized from last time, and an unchanged tip
  reuses the transparent rows;
- found-but-unenriched txids carry over as a backlog; only -5 is a
  definitive "not ours" for z_viewtransaction;
- between blocks, a cheap recent poll plus a one-address probe.

Duty model for a 40-address wallet at 10 s per scan: ~9% instead of ~80%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 11:06:34 -05:00
co-authored by Claude Opus 5.5
parent 8b9a498eee
commit 879de37d18
10 changed files with 1645 additions and 233 deletions
+45 -10
View File
@@ -35,6 +35,13 @@ public:
virtual ~RefreshRpcGateway() = default;
virtual nlohmann::json call(const std::string& method,
const nlohmann::json& params) = 0;
// A call allowed to outlast the default timeout (a whole-wallet sweep on a large wallet).
// Gateways without a separate timeout just call().
virtual nlohmann::json callLong(const std::string& method, const nlohmann::json& params, long timeoutSec)
{
(void)timeoutSec;
return call(method, params);
}
};
enum class Job {
@@ -175,6 +182,9 @@ public:
double amount = 0.0; // note value, DRGX
bool locked = false; // tied up by an in-flight send
int confirmations = 0; // TRUE depth (rawconfirmations when present, else confirmations)
std::string address; // the z-address holding it
std::string txid; // the transaction that created it
bool change = false; // change from the wallet's own spend (not a receive)
};
struct AddressRefreshResult {
@@ -221,14 +231,25 @@ public:
std::set<std::string> miningAddresses;
std::unordered_map<std::string, int> shieldedScanHeights;
std::size_t shieldedScanStartIndex = 0;
std::size_t maxShieldedReceiveScans = 0;
// How many blocks the tip may advance past an address's last scan before it counts as stale
// and needs re-scanning. 0 = strict (must be scanned at the exact current tip). A small
// tolerance lets a multi-cycle pass over many shielded addresses COMPLETE even though new
// blocks arrive mid-pass — otherwise the "scanned at tip" bar moves every block and the scan
// (and the transactions_dirty_ flag it drives) never finishes. It also naturally throttles
// full rescans to roughly once per `tolerance` blocks.
int shieldedScanTipTolerance = 0;
std::size_t maxShieldedReceiveScans = 0; // receive scans this cycle; 0 = no limit
// The pass's floor (services::ShieldedHistoryPass): an address scanned below it is scanned again.
// -1: only never-scanned addresses (and priority ones) are.
int shieldedScanFloorHeight = -1;
// Scanned first, whatever the cursor, when never scanned or scanned below their floor: new
// addresses, the chat reply address, addresses holding notes the history doesn't show yet.
std::unordered_map<std::string, int> priorityScanFloors;
bool skipShieldedScans = false; // a Light cycle (or a probe-less recent poll): no receive scans
bool skipSentSweep = false;
int sentScanFromHeight = -1; // the sweep's watermark; -1 = sweep the whole wallet
std::size_t transparentRowHint = 0; // listtransactions rows last time: sizes the first page
// Same tip and nothing changed since the rows on screen were fetched: reuse their transparent rows
// instead of paging listtransactions again.
bool reuseTransparentRows = false;
std::unordered_set<std::string> enrichBacklogTxids; // discovered earlier, not enriched yet
// Run a windowed sent sweep even though the receive pass isn't complete (the caller rate-limits
// this), so untracked sends (from the Console, or another wallet on the same seed) don't wait for a
// whole pass. The whole-wallet sweep stays tied to a completed pass.
bool allowMidPassSentSweep = false;
};
struct TransactionRefreshResult {
@@ -245,9 +266,23 @@ public:
std::size_t shieldedAddressCount = 0;
std::unordered_map<std::string, int> shieldedScanHeights;
bool shieldedScanComplete = true;
// Wall-clock spent in the history scan (z_listreceivedbyaddress — O(mapWallet), holds cs_main).
// Lets the caller throttle the routine full rescan by its measured cost (txRefreshDue()).
// Wall-clock spent in the whole cycle: listtransactions, the receive scans and the sent sweep each
// hold cs_main. Lets the caller throttle wallet-wide scans by their measured cost (txRefreshDue()).
double scanMs = 0.0;
std::size_t shieldedScansSucceeded = 0; // shieldedAddressesScanned counts attempts
double receiveScanOkMs = 0.0; // time in the receive scans that succeeded
std::size_t transparentRowCount = 0; // rows the listtransactions chain returned
bool transparentChainOk = false;
bool transparentRowsReused = false;
bool sentSweepRan = false;
bool sentSweepOk = false;
bool sentSweepFull = false;
int sentSweepHeight = -1; // the tip a successful sweep covered
double sentSweepMs = 0.0; // time in the sweep (part of scanMs)
rpc::RpcFailureKind sentSweepErrKind = rpc::RpcFailureKind::None;
// Txids found but not enriched (over the per-cycle cap, or the view failed transiently): the caller
// keeps them and feeds them back as enrichBacklogTxids.
std::unordered_set<std::string> unenrichedTxids;
};
struct OperationStatusPollResult {