ObsidianDragon/SENT-HISTORY-SWEEP.md

Sent-history sweep: GUI watermark fix, then daemon follow-up

Audience: ObsidianDragon (GUI) team and the DragonX daemon maintainer. Date: 2026-10-16. Status: proposal.

Code cited

• GUI: ObsidianDragon @ 064c9d1 (unprefixed paths).
• Daemon: dragonx @ e9247f5c6 (perf/history-rpc-copies), paths prefixed dx:. test/v1.3.1-candidate @ ff28e34c8 is the same plus a crash fix; on that branch, dx:wallet.cpp lines past ~1263 shift by +10.

Anything not marked (inference) or (estimate) was verified in the code or measured.

---

1. Problem

The wallet measured: an isolated copy of the owner's mainnet wallet, run in a network namespace with no peers.

• 16,206 Sapling notes, all unspent.
• 5 z-addresses and 11,889 wallet txs.
• Unencrypted; wallet.dat is about 2.1 GB.

Call	v1.3.0	After daemon PR-A+B+C
Balance RPCs	~14 s	~50 ms
Windowed z_listsentbyaddress (filter type 2)	19.5 s	128 ms
Full z_listsentbyaddress (no watermark)	424 s; RSS 8 → 19.4 GB, never released	113 s
Full-sweep reply	26.2 MB (all 11,889 txs)	same

Why every launch pays for the full sweep

• The watermark lives only in memory. history_sent_scan_height_ starts at -1 (src/app.h:1764).
• Every disconnect resets it to -1. That covers daemon restarts, reconnects and wallet switches. The path is onDisconnected → resetTransactionHistoryCacheSession → invalidateShieldedHistoryScanProgress(false) (src/app_network.cpp:936, :1847, :1889-1895).
• Other resets:
  • a locked-wallet connect;
  • imports (:4013);
  • rescans (:7279, :7302; src/app.cpp:6298) and zap (app.cpp:6352);
  • a new, never-scanned z-address (app_network.cpp:2593-2597);
  • the end of a warmup during which a rescan was observed (:2186-2195).
• No watermark means a full sweep. sentSweepIsFull is then true (src/services/history_scan_policy.cpp:35-38). The GUI sends [z,0] via callLong(..., kFullSentSweepTimeoutSec = 120) (src/services/network_refresh_service.cpp:1324-1325, history_scan_policy.h:31).
• The daemon ignores the address argument. It walks every wallet tx under LOCK2(cs_main, cs_wallet) and calls zsWalletTxJSON(wtx, ret, "*", false, 0, …) for each one (dx:src/wallet/rpcwallet.cpp:3719-3746; the call is at :3743).
• The full sweep barely fits, or never fits, the timeout.
  • On the fixed daemon the sweep finishes only 7 s inside the 120 s timeout.
  • On v1.3.0 it never finishes. Failures back off 10 min → 1 h → 6 h (fullSentSweepRetryDelaySec). That backoff is also kept only in memory (app.h:1765-1766).

What the GUI uses from the reply

• Sweep consumer (network_refresh_service.cpp:1328-1332, appendSentTransparentOutputs ~:1076-1123):
  • It reads only txid, confirmations, blocktime/time and sends.transparentSends.
  • It never reads received (20.8 MB of the 26.2 MB) or spends.
• Shielded sends come from enrichment. Every txid goes into knownTxids for z_viewtransaction enrichment, capped at 25 per cycle (app.h:1753). The rest wait in the in-memory history_enrich_backlog_.
• One exception: locateSweepTxid (seed-migration resume) does read sends.saplingSends (app_network.cpp:6394).

---

2. Part 1: GUI fix (persist the watermark)

Goal: a relaunch, reconnect or daemon restart runs only the windowed sweep, [z,0,2,(tip-W)+margin]. A full sweep then happens only in three cases:

• there is no persisted state;
• the persisted state fails verification;
• the periodic safety-net re-sweep is due.

2.1 Where to store it

Use the existing encrypted history snapshot. Store an optional sent_sweep object in its JSON payload, not in settings.json or wallets.json. Why:

• The watermark is only valid together with the rows it produced. Both are written in one INSERT OR REPLACE in WAL mode (src/data/transaction_history_cache.cpp:438-457), so a crash leaves the previous pair intact.
• The snapshot already provides what the watermark needs:
  • per-wallet keying (BLAKE2b of the sorted z+t address set, :157-186);
  • encryption (Argon2id from the passphrase, or salted BLAKE2b when the wallet is unencrypted);
  • a tip check on load (:338-344);
  • one-wallet pruning (pruneOtherWallets, :655-672).
• Keep kSchemaVersion = 1. load() rejects any other version (:351), so a bump would throw away every user's cache.
  • Unknown keys are ignored, so older builds read new payloads safely.
  • A missing sent_sweep means W = -1, which is today's behaviour.

Field	Type	Meaning
height	int	W: the tip used for the sweep's maxDepth. Read in the same worker job, right before the call
anchor_height	int	A = W - margin: the bottom edge of the window the sweep covered
anchor_hash	hex	H_A = getblockhash(A), fetched by the worker after the sweep succeeds
backlog	[txid]	history_enrich_backlog_. Required, see 2.4
full_failures, full_retry_delay_s	int	Full-sweep backoff. The delay is relative to save time, not absolute
daemon_id	string	Daemon version + start time, from getinfo / uptime. If it changes, the restored backoff resets (2.5)
last_full_height, last_full_time	int	Last successful full sweep; drives the safety net (2.6)
txcount, k	int	Optional tripwire (2.3). Both captured in the sweep's worker job

Do not reuse the snapshot's tip_height/tip_hash columns. They record the tip at store time, not at sweep time (app_network.cpp:2069-2070).

2.2 Code changes

File	Change
src/data/transaction_history_cache.h:20-28, :33-48, :79-84	Add a SentSweepState struct to LoadResult, SnapshotJob and replace()
src/data/transaction_history_cache.cpp:412-426 (writeSnapshot)	Emit sent_sweep only if 0 < A <= W <= job.tipHeight
src/data/transaction_history_cache.cpp:351-370 (load)	Parse with defaults (W=-1, empty backlog). Drop the object if it has the wrong type or W is above the stored tip_height
src/data/transaction_history_store_policy.h:35-41 / .cpp:62-113	Add W/A/H_A to the slow signature. They move every block, so this means a write every 600 s and always on disconnect, lock and exit. Add the backlog to content (written within about 20 s). Otherwise an invalidation that only changes the watermark is skipped as "unchanged". Update both callers (app_network.cpp:2012-2014, :2041-2045). The epoch counter is never persisted, so it doesn't belong in a signature
src/app_network.cpp:2066-2079 (storeTransactionHistoryCacheIfAvailable)	Fill job->sentSweep from the history_sent_* and backlog members
src/app_network.cpp:1998-2017 (loadTransactionHistoryCacheIfAvailable)	Restore the members. ++history_sent_epoch_. Set history_sent_verify_pending_ = true. Apply any pending rescan action (2.3)
src/app_network.cpp:2028	The store returns early when there are no rows. On a persistCache=true invalidation with no rows, call clearWallet() so a stale W can't survive on disk
src/services/network_refresh_service.h:256-266, :299	New snapshot/result fields: anchorHeight/anchorHash in, the verify result and H_A out, and the band cursor
src/services/network_refresh_service.cpp:1314-1345	Read the tip immediately before the call. After success, fetch getblockhash(A). Add the verify step (2.3) and the large-window path (2.5)
src/app_network.cpp:2784-2792	Gate both the end-of-pass sweep and the mid-pass windowed sweep (allowMidPassSentSweep) on the pending verify. Extend the retry gate at :2792, which today applies only when W < 0, to large windows
src/app_network.cpp:2843-2854	Count large-window failures toward the backoff; today only sentSweepFull failures count (:2849). Store the band cursor in App and reset it with history_sent_epoch_, like the watermark
src/app.cpp:restoreOriginalWallet	Call invalidateShieldedHistoryScanProgress(true) before onDisconnected, as rescanBlockchain and zap already do (app.cpp:6298, :6352)
src/services/history_scan_policy.{h,cpp}	Pure helpers: sentSweepAnchor(W), sentSweepNeedsLongCall(tip, W), sentSweepMargin(firstAfterLoad), nextSentSweepBand(...)

2.3 Invalidation and safety rules

Verify at the bottom edge of the window, not at its tip.

• The check: before the first windowed sweep after a load, call getblockhash(A) and compare it with H_A.
  • Match: any fork point lies above A, so it is inside the window. Run the windowed sweep.
  • Mismatch: a reorg went deeper than the margin, or this is a different chain or datadir. Drop W and run the banded full sweep (2.5).
• Why not the tip: checking getblockhash(W) would trigger a full sweep on an ordinary one-block reorg of the tip, even though the window already covers that reorg.

Event	Action	Mechanism
First windowed sweep after a load	Anchor verify (above)	getblockhash: O(1), under cs_main
Stored tip_height > current tip at load (replay after an unclean stop, older datadir)	Today: load() deletes the whole snapshot (cache.cpp:338-344, clearWalletByHash at :341). Rows and W are lost, and a full sweep runs	Keep as the default. Optional improvement: have load() keep the rows and sent_sweep, flag "ahead", and defer the sent sweep until tip >= A, then verify H_A. That requires changes in sentSweepIsFull (history_scan_policy.cpp:37: W > tip means full today), the gate at app_network.cpp:2789-2792, and the test at tests/test_phase4.cpp:15831
Wallet switch, or an address added or imported while the GUI was closed	Nothing new	Different address set → different snapshot key → full sweep
GUI-initiated rescan, zap, import, runtime rescan	Already invalidate(true); with 2.2 it now reaches disk	app.cpp:6298, :6352; app_network.cpp:4013, :7279, :7302
New never-scanned z-address during a session	Already resets W (:2593-2597); the reset must be persisted	Signature change in 2.2
Disconnect, daemon restart, same-wallet reconnect	Keep today's in-memory reset	The disk copy is written first (:857) and reloaded after connect
Rescan observed in the daemon log (the daemon's catch-up "Rescanning last N blocks (from block H)", or an external -rescan)	Sticky pending action (see below)	The warmup-end invalidation at :2186-2195 runs before the snapshot loads, so today it can't reach a persisted W
Wallet changed outside the GUI with the same address set (Console z_importviewingkey, external -rescan, same-seed restore)	Optional tripwire: after the first windowed sweep returns R rows, run the banded full sweep if txcount − txcount_saved > |R| − k + slack. k is the rows of the saved sweep within the same 100 + E depth band the first window uses	getwalletinfo.txcount = mapWallet.size() (dx:rpcwallet.cpp:2791). The GUI already fetches it on every connect (network_refresh_service.cpp:251-252, :295-298; app_network.cpp:2140-2143). Measured cost: ~8-20 ms warm, ~180 ms on the first call after a start
Daemon version change	Keep W. Reset the persisted backoff	Same filter semantics since 2020; v1.3.0 and the candidate both report 1.3.0.50

The sticky rescan signal. Why it's needed:

• During warmup no address list is fetched (scan_policy.h:42-44, :64-68; app_network.cpp:2492), and state_.clear() has emptied the rows (wallet_state.h:406-409).
• So when invalidate(true) runs at app_network.cpp:2195, it finds an empty identity and returns (:2028, :2032).
• The cache then loads afterwards (:2608) and would restore the old W unchanged.

The fix:

1. Whenever rescan_confirmed_active_ is set (app.cpp:1193, :1324, :1397), record a pending action that survives resetTransactionHistoryCacheSession:
   • clamp to H, parsed from "Rescanning last N blocks (from block H)" or "Rescanning from height H";
   • full reset if H is unknown or near genesis.
2. loadTransactionHistoryCacheIfAvailable applies it to the loaded W (W = min(W, H), and A follows) and to the receive-scan heights, then clears it.

With daemon item D4 (wallet locator on clean shutdown, 3.2) shipped, the daemon's catch-up rescan line disappears from normal starts.

Reorg overlap.

• Window extent. Filter type 2 drops only txs deeper than N (dx:rpcwallet.cpp:3740), and maxDepth = tip − W + margin. So the window covers heights ≥ W − margin + 1, plus the mempool (depth 0 always passes).
• Margin.
  • MAX_REORG_LENGTH defaults to 99 (dx:src/main.h:79), and a longer reorg shuts the node down (dx:main.cpp:4314-4333).
  • A margin of 100 therefore leaves 1 block of slack, which tip drift between reading the tip and making the call can use up.
  • Use margin ≈ 120 in steady state (recommendation), and read the tip in the same worker job as the call.
• First sweep after a load. Use margin 120 + E with E ≈ 1000 blocks (~10 h).
  • This absorbs wallet-notifier lag at save time: ThreadNotifyWallets applies blocks asynchronously (dx:src/validationinterface.cpp:89-262), and no RPC exposes the wallet's processed tip.
  • E is an inferred bound, not a measurement.
• Large -maxreorg. A -maxreorg set above the margin is caught by the anchor check.
• Conflicted txs. Conflicted txs (depth −1) are dropped by both the full and the windowed sweep (dx:rpcwallet.cpp:3725), so nothing changes there.

Locked-wallet rule.

• The rule: never advance or persist W from a sweep dispatched or completed while the wallet is locked, and never load or store the snapshot while it is locked.
• What already protects this today:
  • History scans don't run while locked (src/services/scan_policy.h:48-51).
  • An encrypted wallet's snapshot opens only after unlock (app_network.cpp:797-798, :1944-1962).
  • Locking keeps W and stores immediately (:1851-1859).
• What breaks if it's ever violated: a locked daemon wallet returns empty saplingReceived and saplingSends, because GetSaplingHistoryKeys needs spending keys (dx:rpchushwallet.cpp:19-39).
  • The fields the main consumer reads (txid, transparentSends) don't depend on the lock.
  • locateSweepTxid reads saplingSends, so it must never run on a locked wallet.

2.4 Backlog rule (required)

• The risk. Enrichment is capped at 25 per cycle, and the txids that miss the cap go to history_enrich_backlog_ (app_network.cpp:2843).
  • A pure z→z send leaves only a change note, which produces no receive row (network_refresh_service.cpp:972).
  • So its txid exists only in a sweep result.
  • If W is persisted while those txids sit in the backlog, the sends are lost for good once the window moves past them.
• The rule: persist the backlog together with W.
• Rejected fallback: "persist W = -1 while the backlog is non-empty" isn't enough on its own. Enrichment retries are unbounded (network_refresh_service.cpp:1521), so one txid that keeps failing would stop W from ever being persisted. If that fallback is wanted as well, cap the retries per txid.

2.5 Timeouts, large windows, banded full sweep

Large windows.

• The windowed sweep uses rpc.call, which has curl's 30 s timeout (src/rpc/rpc_client.cpp:223). Only full-sweep failures count toward the backoff (app_network.cpp:2846-2853).
• So a watermark saved days ago produces a huge "windowed" call that times out on every block, while the daemon keeps working on each abandoned call.
• Rule: if tip − W > 2400 (about 1 day of 36 s blocks), use callLong and count failures the same way as for a full sweep.

Banded full sweep.

• The call: replace the single 113 s (fixed daemon) or 424 s (v1.3.0) call with depth bands, z_listsentbyaddress(z, minconf=a, 2, maxDepth=b). minconf (dx:rpcwallet.cpp:3732) and filter type 2 (:3740) are independent filters, so each call returns exactly the depth band [a, b].
• Order:
  • Go newest band first; the first band uses minconf=0 so it includes the mempool.
  • Overlap bands by at least 10 blocks, plus the blocks that arrive between calls, and dedupe by txid.
  • Size bands adaptively, aiming for about 5 s each.
• Scheduling: run one band per history cycle, using the epoch-guarded band cursor from 2.2. All bands in one job would tie up the shared RPC worker for about 113 s, exactly as today.
• Watermark: take W from the tip at the first band, and set it only after every band has succeeded.
• Don't page with nCount. The iteration order is nOrderPos, not height (:3709, :3746).
• Limits:
  • Banding shortens each LOCK2 hold but not the total CPU.
  • On v1.3.0 every call has a ~19.5 s floor, because every CWalletTx is deep-copied (inference from the windowed measurement). There the bands need callLong.

Backoff persistence.

• Store the failure count and the relative delay (2.1), so a relaunch doesn't immediately re-run a sweep that keeps failing.
• Reset both when daemon_id changes. Otherwise a user who installs the fixed daemon could wait up to 6 h for a full sweep that would now succeed.

2.6 Safety net

• When: about every 7 days, run one banded full re-sweep.
• Conditions: only when synced, unlocked, idle (not on the Mining tab) and admitted by the governor.
• On success, update last_full_*. On failure, back off.
• Why: it catches what the guards can't see, such as a Console z_importviewingkey. That case is also missed within a session today.

2.7 Edge cases

Case	Handling
Stored tip < current tip: load() doesn't check the hash (cache.cpp:338-344)	The anchor verify covers the watermark. Optionally extend load() too
Transparent send rows from appendSentTransparentOutputs aren't reproduced by windowed sweeps	Probably dropped on the next clean cycle (network_refresh_service.cpp:1356-1368, :1528-1539) (inference). Negligible on DragonX, where consensus bans non-notary/burn t-outputs (dx:main.cpp:1606, :1633, :1657). Optional fix: carry over send rows that listtransactions didn't produce
locateSweepTxid runs a full sweep under the 30 s timeout (app_network.cpp:6388)	That times out on this wallet. Switch it to a windowed or banded call
Watch-only addresses aren't part of the snapshot key	Covered by the tripwire and the safety net
Switching between two wallets	pruneOtherWallets keeps one snapshot, so a switch still costs a full sweep (open question 4)
GUI crash mid-sweep	W advances only on sentSweepOk, and the write is atomic
Regtest or testnet	Different z-address prefixes, so a different snapshot key

2.8 Test plan

Unit tests (tests/test_phase4.cpp):

• In testTransactionHistoryCache (:9057):
  • every sent_sweep field round-trips;
  • a missing key reads as −1;
  • a wrong type, W > tip_height or A > W reads as −1;
  • unknown keys are ignored.
• Signatures (~:9244): a change to W, A or H_A changes slow; a change to the backlog changes content.
• Policy (~:15829, :16098-16213):
  • the anchor is W − margin;
  • the first window after a load uses 120 + E;
  • tip − W > 2400 uses callLong and counts failures;
  • band planning: boundaries, overlap, dedupe, and cursor reset on an epoch change;
  • an anchor mismatch means a full sweep;
  • the pending rescan action clamps W;
  • daemon_id changes reset the backoff.

Integration tests (regtest or -testnode):

1. Full sweep, quit, relaunch: the log shows only [z,0,2,N] calls.
2. invalidateblock one block above A while the GUI is closed: the windowed sweep runs and returns the right rows.
3. invalidateblock below A: the verify fails and a banded full sweep runs.
4. More than 25 z→z sends, then quit before enrichment finishes: after relaunch every send appears.
5. A daemon started with -rescan while the GUI was closed: the sticky action clamps or resets W.
6. restoreOriginalWallet: W is invalidated.
7. Locked encrypted wallet: W doesn't advance and the snapshot isn't loaded.

Mainnet copy: relaunch against the isolated owner copy. Expect no [z,0] call and a sent sweep well under 1 s.

2.9 Expected effect

Scenario	Today	With Part 1
Relaunch or reconnect, fixed daemon	113 s full sweep under LOCK2, 7 s from the timeout	getblockhash plus one windowed sweep over 120 + E + the closure gap. Steady-state windows measure ~128 ms; the first window after a load is larger (unmeasured)
Relaunch, v1.3.0 daemon	424 s; never fits in 120 s, so it loops in backoff with RSS at 19.4 GB	~19.5 s+ windowed once a W exists. Seeding the first W needs the banded sweep
Safety net	—	One banded full sweep a week, each LOCK2 hold about 5 s

---

3. Part 2: daemon follow-up

Where the 113 s goes (owner wallet, full sweep)

Component	Cost	Basis
Trial decryption of outputs that aren't wallet notes: ~172,800 outputs × 5 ivks ≈ 864k failed tries × ~0.13 ms	~112 s (~99%)	Output count derived from tx sizes; per-try cost fitted to the full (112.7 s) and windowed (126 ms / 740 tries) measurements (estimate). Code: OutputDecryptions::Get (dx:rpchushwallet.cpp:87-100), loop at :303-346
DecryptSent esk-path decrypts	0 on this wallet	No tx spends a wallet note, so the sends helper is gated off (:586)
UniValue build + copy under LOCK2	~0.28 s	Synthetic benchmark on the repo's UniValue
Reply serialization (after the lock is released)	~0.31-0.34 s	Same benchmark

Where the foreign outputs come from

Sietch padding. z_sendmany pads its z-recipients up to -sietch-min-zouts (default 7, clamped to 3-50; dx:rpcwallet.cpp:5424-5434), with change on top.

• Recipient and dummy outputs take the sender's ovk: the from-address's fvk.ovk, or the seed ovk for t-address sources (dx:asyncrpcoperation_sendmany.cpp:352-368, applied at :437).
• Consolidation (saplingconsolidation.cpp:244), sweep (sweep.cpp:299), shieldcoinbase (:239-254) and autoshield (:435) also add dummies with the sender's ovk.
• So a sending wallet's DecryptSent opens every one of its own dummies. That is the ~0.36 ms esk-path decrypt each, which is D2.
• Every other wallet sees them as foreign outputs and tries them under every ivk, which is D1.

On this wallet most foreign outputs are not sietch dummies. About 158.6k of the ~174k sit in the 9,597 txs with 10 or more outputs, which are mostly multi-recipient payouts (inference from tx sizes). That doesn't change D1.

Items

#	Item	Output	Gain (owner wallet)	Risk	GUI change
D1	Per-output negative ivk cache, plus parallel pre-fill of the txs a call selects	Byte-identical	Warm 113 s → 0.4-1 s. First full sweep after a restart ~20-25 s (pre-fill at the measured 4.75× warm-up scaling). A windowed call stays ~100 ms (estimate)	Low-med	None
D2	DecryptSent cache per (output, ovk)	Byte-identical	0 here; ~0.36 ms per send output on sending wallets	Low	None
D3	getalldata balance loop	Byte-identical	~20-35 s → 10-30 ms (estimate)	Low	None (the GUI never calls it)
D4	Write the wallet locator on clean shutdown	n/a	Start: −49-62 s (post-rescan locator write, inferred from the log gap) and −5-10 s of block rescan; the ~25 s witness-root pass moves to the first live block. Shutdown: +3-60 s (inferred)	Medium	Check the stop policy
D5	JSON trims	Byte-identical	0.2-0.4 s	Low	None
D6	Sends-only returnType (6th parameter)	Changes output; opt-in	113 s → 0.1-0.3 s; 26.2 → ~4.1 MB (estimate)	Low	Probe + opt-in
D7	Honour the address argument	Changes output	None (slower)	Breaks the GUI	Not recommended

• D6 is the single largest win, but it needs a GUI release.
• D1-D5 are invisible to the GUI.
• With Part 1 shipped, full sweeps become rare, so D1 mostly helps older GUIs, other clients and the safety net.

3.1 D1: per-output decryption cache (byte-identical)

Shape

• A mutable std::vector<uint64_t> on CWalletTx, with one mask per vShieldedOutput index.
  • It is sized once, never serialized, and cleared in Init().
  • Precedents: cachedPlaintext (dx:src/wallet/wallet.h:340-348) and the mutable caches at :472-490.
• Bit k means "history key k does not decrypt this output".
• An append-only ivk → index registry on CWallet, under cs_wallet. It is resolved once per RPC into SaplingHistoryKeys::SpendableAddress (dx:rpchushwallet.cpp:19-39).

Logic in OutputDecryptions::Get

• If the bit is set, return null without decrypting.
• After a failed Decrypt, set the bit.
• Never cache a success: wallet notes already carry cachedPlaintext.
• Keys with index ≥ 64 aren't cached.

Invariant: the result is a pure function of (txid, n, ivk). A v4 txid commits to every output byte, so:

• nothing ever goes stale;
• a newly added key has a clear bit and is still tried;
• locking, or losing keys, only means fewer keys are consulted;
• reorgs, deletions and reloads are harmless;
• output stays byte-identical.

Thread safety

• The mask is mutable behind a const CWalletTx&, so the compiler won't enforce locking. Every mask read and write therefore happens on the RPC thread under cs_wallet, with an AssertLockHeld in the accessor.
• Pre-fill workers follow FillSaplingNoteCache's rule that workers must not lock, or even assert a lock (dx:wallet.cpp:5745-5746). They call only decrypt_with_note on pre-resolved (output, ivk) pairs and write private masks, which are merged serially after the join.
• Workers must not call GetSaplingHistoryKeys or IsNullifierOfAddress, because both assert locks.

Fill

• Lazily, as the helpers run.
• In addition, pre-fill in parallel, but only the txs the call has already selected:
  1. In z_listsentbyaddress, first collect the wtx pointers that pass the filters at dx:rpcwallet.cpp:3722-3741, honouring nCount.
  2. Pre-fill their outputs, using the 128-item threshold and -witnessbuildthreads from dx:wallet.cpp:5739-5797.
  3. Then emit.
• Pre-fill time is spent under LOCK2. For a full sweep that's ~20-25 s once; a windowed call stays small.
• Optionally, also record the misses FindMySaplingNotes already computes (:2282-2322).
• A startup precompute is opt-in only: ~112 CPU-s, about 24 s of wall time per start (estimate).

Also benefits: getalldata's tx list (dx:rpcwallet.cpp:3254) and z_listreceivedaddress (:3555).

Tests

• Extend the verbatim oracle dx:src/gtest/test_wallet_history.cpp: cold, warm and pre-filled runs must match byte for byte.
• A key added after the mask was filled, which opens a foreign output, must appear in the output.
• More than 64 keys; locked and encrypted wallets; -witnessbuildthreads=1 vs 24; a DEBUG_LOCKORDER build.
• The mainnet A/B harness: cmp cold and warm replies against the captured 26,233,144-byte reply (/home/d/dragonx-ab/mainnet-ab/out/B/cap/023_z_listsentbyaddress_addr0_full.json).

D2. Cache DecryptSent results per (n, ovk index): the plaintext and address on success, or "no". That's about 650 B per entry. Key the seed ovk by value, so lock state still decides whether the cache is consulted. Use the same oracle tests as D1.

3.2 Smaller items

D3: the getalldata balance loop (dx:rpcwallet.cpp:3108-3151, under LOCK2 at :2998).

• Today it copies each SaplingNoteData by value, witnesses included (:3110). For every unspent note and every address it re-derives the spending key and trial-decrypts (:3117-3121).
• Fix:
  • build SaplingHistoryKeys once, and reuse it at :3233;
  • take note data by const&;
  • use OutputDecryptions(wtx, op.n).Get(a.ivk) over keys.spendable, moving OutputDecryptions out of its anonymous namespace (rpchushwallet.cpp:56-120) into rpchushwallet.h.
• Keep two quirks for byte-identity, and fix them later as a separate change with a release note:
  • The no-op txType == 3; at :3124 means locked notes are never counted as locked. When fixed, use a per-note local, noteType = (txType == 0 && IsLockedNote(op)) ? 3 : txType. txType is per tx (:3043), so assigning to it would mark every later note of the tx as locked.
  • Every address whose ivk opens the note is credited, because the continue doesn't exit the address loop.
• Don't switch to GetFilteredNotes: that changes the output.
• Test: an oracle gtest with a verbatim copy of the old loop (shared-ivk, locked-note, watch-only and encrypted cases).

D4: write the wallet locator on clean shutdown.

• Evidence: the copy, taken from a clean Windows shutdown, was 12 blocks behind its chainstate. So every start rescans those blocks.

• Cause:

  • FlushStateToDisk writes the locator only if DATABASE_WRITE_INTERVAL (3600 s, dx:main.h:145) has passed, even for FLUSH_STATE_ALWAYS (dx:main.cpp:3838-3841).
  • Shutdown() calls it at dx:init.cpp:259.
  • pwalletMain->Flush(true) at :280 only flushes BDB.

• Hook: in Shutdown(), between :259 and :280:

  LOCK2(cs_main, pwalletMain->cs_wallet);
  const CBlockIndex* tip = pwalletMain->GetWalletTip();   // new accessor: pindexWalletTip is private (wallet.h:988-994)
  if (tip && tip != lastStamped)
      pwalletMain->SetBestChainINTERNAL(walletdb, chainActive.GetLocator(tip));
  

• Don't route it through CWallet::SetBestChain. That swaps in pindexWalletTip when set but falls back to the passed locator when it's null (dx:wallet.cpp:829-834). The null case is exactly "stamp the tip before the init rescan ran", which would skip the unscanned blocks for good.

• The failure path needs the lock too. AppInit2's failure path calls Shutdown() without join_all (dx:bitcoind.cpp:232-243), so the notifier can still be running. Reading the tip under LOCK2 keeps that safe.

• Don't drop the interval check globally. gettxoutsetinfo and init also flush with ALWAYS, and each would then rewrite ~2 GB under cs_main.

• Trade-off: one locator write rewrites every note-bearing CWalletTx, ~2 GB here (SetBestChainINTERNAL, dx:wallet.h:939-986, WriteTx loop :947-962). The cost moves to shutdown rather than disappearing; see the D4 row in the table above.

• Ship it together with a parallel witness-root check at init.

• Risk: the GUI's stop policy (src/daemon/daemon_controller.h:360, evaluateStopWait) must allow about 16 s plus 3-60 s. Context: the 2026-09-30 Windows kill loop.

• Tests:

  • Stop within the hour and restart: no "Rescanning last" line, and identical balances.
  • Stop during the startup rescan: the locator does not advance.

D5: JSON trims.

• Use static_cast<const CTransaction&> at dx:rpchushwallet.cpp:553. Today it copies 190.6 MB of tx data per full sweep.
• Drop the getValues / reverse / push_backV copy (dx:rpcwallet.cpp:3749-3755). Collect the selected wtx pointers in the existing reverse loop, which keeps nCount picking the newest txs, then build ret oldest to newest. The same pattern is at getalldata :3259-3265.

Also found; each needs its own design.

• The hourly SetBestChain rewrites ~2 GB while cs_main is held (3-47 s gaps in the live log, inferred from log gaps).
• FindMySaplingNotes tries foreign outputs once per viewing key and again per address (dx:wallet.cpp:2282-2322). De-duplicating ivks would probably halve the per-block tries (inference).

3.3 Options that change RPC output (need a GUI opt-in)

D6: sends-only mode.

• The parameter: add an optional 6th parameter returnType to z_listsentbyaddress, default 0 (unchanged output).
  • The code's meanings are 1 = spends, 2 = received, 3 = sends. The comment at dx:rpchushwallet.cpp:561-566 swaps 1 and 3; fix it in the same change.
• With 3 and no address filter:
  • the spends helper still runs (:569), but its output isn't emitted;
  • received, where all the trial decryption happens, is skipped (:576-581);
  • sends is emitted (:584-592), and every tx is still pushed (:594-598).
  • The GUI gets the same txids and the same sends.transparentSends.
• Daemon changes:
  • Raise the check at dx:rpcwallet.cpp:3606 from > 5 to > 6, keeping the == 3 rejection.
  • Change :3693 to params.size() >= 5. Today it reads nCount only when size() == 5, so every 6-parameter call would silently drop it.
  • Read params[5], reject anything outside 0-3, and document it in the help text. dx:src/rpc/client.cpp:165 already converts index 5.
• GUI side: send 6 parameters. An old daemon throws its help text for more than 5, so fall back to 5 parameters and remember that per connection.
  • Alternative: a new RPC name, probed through "Method not found" the same way as z_autoshieldstatus (app.h:1744-1747).
• Owner decision: a variant that returns only txs with wallet spends would return 0 rows here, but it changes the knownTxids set.

D7: honour the address argument.

• The GUI passes shieldedAddresses.front() (network_refresh_service.cpp:1319) and relies on the daemon using "*" (dx:rpcwallet.cpp:3743). Honouring the argument by default would silently shrink GUI sweeps to one address.
• It isn't faster either: with a filter set, fBool=true runs zsTxSendsToJSON for every tx (:586).
• Recommendation: document "*" as a compatibility contract. If honouring the argument is ever wanted, put it behind a flag.

Additive and safe for the GUI: a wallet-tip / history fingerprint.

• Expose pindexWalletTip, for example in getwalletinfo or a new z_getwallethistorystate(maxheight) returning {count, txid-hash, wallet_tip}.
• The GUI ignores unknown JSON fields.
• This would make Part 1's invalidation exact, replacing the E-margin guess and the txcount tripwire.

---

4. Rollout and compatibility

Step	Ships in	Works with	Notes
Part 1: watermark + anchor verify + backlog + sticky rescan + large-window path + banding + safety net	Next GUI release	Every daemon version; it uses only existing z_listsentbyaddress parameters, getblockhash, getwalletinfo and getinfo	Old GUI with a new snapshot: the key is ignored. New GUI with an old snapshot: one full sweep, which is today's behaviour. On v1.3.0, seeding the first W needs the banded sweep
D1, D2, D3, D5	Daemon v1.3.1 / v1.3.2	Invisible to the GUI	Gate each on the byte-identical oracle and the mainnet cmp
D6 + GUI probe	Daemon first, then GUI	The GUI falls back on old daemons	Makes the safety net and any full sweep cheap
D4	Together with a parallel witness-root check	Invisible apart from shutdown time	Confirm the GUI stop policy first

---

5. Open questions

1. How big is the backlog after the first full sweep on the owner's wallet? Unmeasured; it sizes the persisted backlog field.
2. What is the full-sweep time on the owner's 24-core Windows node? The 113 s was measured on Linux/WSL, 7 s inside the 120 s timeout.
3. How long does the wallet notifier lag after a catch-up on this wallet? This sets E.
4. Should the snapshot store keep more than one wallet, so that switching wallets also benefits?
5. D6 shape: a 6th parameter or a new RPC? Should a "wallet-spends-only" variant exist?
6. Does any client besides ObsidianDragon depend on returnType 0 output? This sets D1's priority once D6 exists.
7. Are the 5 ivks distinct? Assumed. The per-address note counts sum to exactly 16,206, which fits but doesn't prove it.
8. Does the owner want the additive wallet-tip / fingerprint RPC, which makes Part 1's invalidation exact?