ObsidianDragon/SENT-HISTORY-SWEEP.md

# Sent-history sweep: GUI watermark fix, then daemon follow-up

**Audience:** ObsidianDragon (GUI) team and the DragonX daemon maintainer.
**Date:** 2026-10-16. **Status:** proposal.

**Code cited**
- GUI: `ObsidianDragon` @ `064c9d1` (unprefixed paths).
- Daemon: `dragonx` @ `e9247f5c6` (`perf/history-rpc-copies`), paths prefixed `dx:`. `test/v1.3.1-candidate` @ `ff28e34c8` is the same plus a crash fix; on that branch, `dx:wallet.cpp` lines past ~1263 shift by +10.

Anything not marked *(inference)* or *(estimate)* was verified in the code or measured.

---

## 1. Problem

**The wallet measured:** an isolated copy of the owner's mainnet wallet, run in a network namespace with no peers.
- 16,206 Sapling notes, all unspent.
- 5 z-addresses and 11,889 wallet txs.
- Unencrypted; `wallet.dat` is about 2.1 GB.

| Call | v1.3.0 | After daemon PR-A+B+C |
|---|---|---|
| Balance RPCs | ~14 s | ~50 ms |
| Windowed `z_listsentbyaddress` (filter type 2) | 19.5 s | 128 ms |
| **Full `z_listsentbyaddress` (no watermark)** | **424 s**; RSS 8 → 19.4 GB, never released | **113 s** |
| Full-sweep reply | 26.2 MB (all 11,889 txs) | same |

### Why every launch pays for the full sweep

- **The watermark lives only in memory.** `history_sent_scan_height_` starts at `-1` (`src/app.h:1764`).
- **Every disconnect resets it to `-1`.** That covers daemon restarts, reconnects and wallet switches. The path is `onDisconnected` → `resetTransactionHistoryCacheSession` → `invalidateShieldedHistoryScanProgress(false)` (`src/app_network.cpp:936`, `:1847`, `:1889-1895`).
- **Other resets:**
  - a locked-wallet connect;
  - imports (`:4013`);
  - rescans (`:7279`, `:7302`; `src/app.cpp:6298`) and zap (`app.cpp:6352`);
  - a new, never-scanned z-address (`app_network.cpp:2593-2597`);
  - the end of a warmup during which a rescan was observed (`:2186-2195`).
- **No watermark means a full sweep.** `sentSweepIsFull` is then true (`src/services/history_scan_policy.cpp:35-38`). The GUI sends `[z,0]` via `callLong(..., kFullSentSweepTimeoutSec = 120)` (`src/services/network_refresh_service.cpp:1324-1325`, `history_scan_policy.h:31`).
- **The daemon ignores the address argument.** It walks every wallet tx under `LOCK2(cs_main, cs_wallet)` and calls `zsWalletTxJSON(wtx, ret, "*", false, 0, …)` for each one (`dx:src/wallet/rpcwallet.cpp:3719-3746`; the call is at `:3743`).
- **The full sweep barely fits, or never fits, the timeout.**
  - On the fixed daemon the sweep finishes only **7 s inside** the 120 s timeout.
  - On v1.3.0 it never finishes. Failures back off 10 min → 1 h → 6 h (`fullSentSweepRetryDelaySec`). That backoff is also kept only in memory (`app.h:1765-1766`).

### What the GUI uses from the reply

- **Sweep consumer** (`network_refresh_service.cpp:1328-1332`, `appendSentTransparentOutputs` ~`:1076-1123`):
  - It reads only `txid`, `confirmations`, `blocktime`/`time` and `sends.transparentSends`.
  - It never reads `received` (20.8 MB of the 26.2 MB) or `spends`.
- **Shielded sends come from enrichment.** Every txid goes into `knownTxids` for `z_viewtransaction` enrichment, capped at 25 per cycle (`app.h:1753`). The rest wait in the in-memory `history_enrich_backlog_`.
- **One exception:** `locateSweepTxid` (seed-migration resume) does read `sends.saplingSends` (`app_network.cpp:6394`).

---

## 2. Part 1: GUI fix (persist the watermark)

**Goal:** a relaunch, reconnect or daemon restart runs only the windowed sweep, `[z,0,2,(tip-W)+margin]`. A full sweep then happens only in three cases:
- there is no persisted state;
- the persisted state fails verification;
- the periodic safety-net re-sweep is due.

### 2.1 Where to store it

**Use the existing encrypted history snapshot.** Store an optional `sent_sweep` object in its JSON payload, not in `settings.json` or `wallets.json`. Why:
- The watermark is only valid together with the rows it produced. Both are written in one `INSERT OR REPLACE` in WAL mode (`src/data/transaction_history_cache.cpp:438-457`), so a crash leaves the previous pair intact.
- The snapshot already provides what the watermark needs:
  - per-wallet keying (BLAKE2b of the sorted z+t address set, `:157-186`);
  - encryption (Argon2id from the passphrase, or salted BLAKE2b when the wallet is unencrypted);
  - a tip check on load (`:338-344`);
  - one-wallet pruning (`pruneOtherWallets`, `:655-672`).
- **Keep `kSchemaVersion = 1`.** `load()` rejects any other version (`:351`), so a bump would throw away every user's cache.
  - Unknown keys are ignored, so older builds read new payloads safely.
  - A missing `sent_sweep` means `W = -1`, which is today's behaviour.

| Field | Type | Meaning |
|---|---|---|
| `height` | int | `W`: the tip used for the sweep's `maxDepth`. Read in the same worker job, right before the call |
| `anchor_height` | int | `A = W - margin`: the bottom edge of the window the sweep covered |
| `anchor_hash` | hex | `H_A = getblockhash(A)`, fetched by the worker after the sweep succeeds |
| `backlog` | [txid] | `history_enrich_backlog_`. **Required**, see 2.4 |
| `full_failures`, `full_retry_delay_s` | int | Full-sweep backoff. The delay is relative to save time, not absolute |
| `daemon_id` | string | Daemon version + start time, from `getinfo` / uptime. If it changes, the restored backoff resets (2.5) |
| `last_full_height`, `last_full_time` | int | Last successful full sweep; drives the safety net (2.6) |
| `txcount`, `k` | int | Optional tripwire (2.3). Both captured **in the sweep's worker job** |

Do **not** reuse the snapshot's `tip_height`/`tip_hash` columns. They record the tip at store time, not at sweep time (`app_network.cpp:2069-2070`).

### 2.2 Code changes

| File | Change |
|---|---|
| `src/data/transaction_history_cache.h:20-28`, `:33-48`, `:79-84` | Add a `SentSweepState` struct to `LoadResult`, `SnapshotJob` and `replace()` |
| `src/data/transaction_history_cache.cpp:412-426` (`writeSnapshot`) | Emit `sent_sweep` only if `0 < A <= W <= job.tipHeight` |
| `src/data/transaction_history_cache.cpp:351-370` (`load`) | Parse with defaults (`W=-1`, empty backlog). Drop the object if it has the wrong type or `W` is above the stored `tip_height` |
| `src/data/transaction_history_store_policy.h:35-41` / `.cpp:62-113` | Add `W`/`A`/`H_A` to the `slow` signature. They move every block, so this means a write every 600 s and always on disconnect, lock and exit. Add the backlog to `content` (written within about 20 s). Otherwise an invalidation that only changes the watermark is skipped as "unchanged". Update both callers (`app_network.cpp:2012-2014`, `:2041-2045`). The epoch counter is never persisted, so it doesn't belong in a signature |
| `src/app_network.cpp:2066-2079` (`storeTransactionHistoryCacheIfAvailable`) | Fill `job->sentSweep` from the `history_sent_*` and backlog members |
| `src/app_network.cpp:1998-2017` (`loadTransactionHistoryCacheIfAvailable`) | Restore the members. `++history_sent_epoch_`. Set `history_sent_verify_pending_ = true`. Apply any **pending rescan action** (2.3) |
| `src/app_network.cpp:2028` | The store returns early when there are no rows. On a `persistCache=true` invalidation with no rows, call `clearWallet()` so a stale W can't survive on disk |
| `src/services/network_refresh_service.h:256-266`, `:299` | New snapshot/result fields: `anchorHeight`/`anchorHash` in, the verify result and `H_A` out, and the band cursor |
| `src/services/network_refresh_service.cpp:1314-1345` | Read the tip immediately before the call. After success, fetch `getblockhash(A)`. Add the verify step (2.3) and the large-window path (2.5) |
| `src/app_network.cpp:2784-2792` | Gate both the end-of-pass sweep and the **mid-pass** windowed sweep (`allowMidPassSentSweep`) on the pending verify. Extend the retry gate at `:2792`, which today applies only when `W < 0`, to large windows |
| `src/app_network.cpp:2843-2854` | Count large-window failures toward the backoff; today only `sentSweepFull` failures count (`:2849`). Store the band cursor in `App` and reset it with `history_sent_epoch_`, like the watermark |
| `src/app.cpp:restoreOriginalWallet` | Call `invalidateShieldedHistoryScanProgress(true)` before `onDisconnected`, as `rescanBlockchain` and zap already do (`app.cpp:6298`, `:6352`) |
| `src/services/history_scan_policy.{h,cpp}` | Pure helpers: `sentSweepAnchor(W)`, `sentSweepNeedsLongCall(tip, W)`, `sentSweepMargin(firstAfterLoad)`, `nextSentSweepBand(...)` |

### 2.3 Invalidation and safety rules

**Verify at the bottom edge of the window, not at its tip.**
- **The check:** before the first windowed sweep after a load, call `getblockhash(A)` and compare it with `H_A`.
  - **Match:** any fork point lies above `A`, so it is inside the window. Run the windowed sweep.
  - **Mismatch:** a reorg went deeper than the margin, or this is a different chain or datadir. Drop W and run the banded full sweep (2.5).
- **Why not the tip:** checking `getblockhash(W)` would trigger a full sweep on an ordinary one-block reorg of the tip, even though the window already covers that reorg.

| Event | Action | Mechanism |
|---|---|---|
| First windowed sweep after a load | Anchor verify (above) | `getblockhash`: O(1), under `cs_main` |
| Stored `tip_height` > current tip at load (replay after an unclean stop, older datadir) | **Today:** `load()` deletes the whole snapshot (`cache.cpp:338-344`, `clearWalletByHash` at `:341`). Rows and W are lost, and a full sweep runs | Keep as the default. **Optional improvement:** have `load()` keep the rows and `sent_sweep`, flag "ahead", and defer the sent sweep until `tip >= A`, then verify `H_A`. That requires changes in `sentSweepIsFull` (`history_scan_policy.cpp:37`: `W > tip` means full today), the gate at `app_network.cpp:2789-2792`, and the test at `tests/test_phase4.cpp:15831` |
| Wallet switch, or an address added or imported while the GUI was closed | Nothing new | Different address set → different snapshot key → full sweep |
| GUI-initiated rescan, zap, import, runtime rescan | Already `invalidate(true)`; with 2.2 it now reaches disk | `app.cpp:6298`, `:6352`; `app_network.cpp:4013`, `:7279`, `:7302` |
| New never-scanned z-address during a session | Already resets W (`:2593-2597`); the reset must be persisted | Signature change in 2.2 |
| Disconnect, daemon restart, same-wallet reconnect | Keep today's in-memory reset | The disk copy is written first (`:857`) and reloaded after connect |
| **Rescan observed in the daemon log** (the daemon's catch-up "Rescanning last N blocks (from block H)", or an external `-rescan`) | **Sticky pending action** (see below) | The warmup-end invalidation at `:2186-2195` runs *before* the snapshot loads, so today it can't reach a persisted W |
| Wallet changed outside the GUI with the same address set (Console `z_importviewingkey`, external `-rescan`, same-seed restore) | Optional tripwire: after the first windowed sweep returns R rows, run the banded full sweep if `txcount − txcount_saved > \|R\| − k + slack`. `k` is the rows of the saved sweep within the same `100 + E` depth band the first window uses | `getwalletinfo.txcount` = `mapWallet.size()` (`dx:rpcwallet.cpp:2791`). The GUI already fetches it on every connect (`network_refresh_service.cpp:251-252`, `:295-298`; `app_network.cpp:2140-2143`). Measured cost: ~8-20 ms warm, ~180 ms on the first call after a start |
| Daemon version change | Keep W. Reset the persisted backoff | Same filter semantics since 2020; v1.3.0 and the candidate both report 1.3.0.50 |

**The sticky rescan signal.** Why it's needed:
- During warmup no address list is fetched (`scan_policy.h:42-44`, `:64-68`; `app_network.cpp:2492`), and `state_.clear()` has emptied the rows (`wallet_state.h:406-409`).
- So when `invalidate(true)` runs at `app_network.cpp:2195`, it finds an empty identity and returns (`:2028`, `:2032`).
- The cache then loads afterwards (`:2608`) and would restore the old W unchanged.

The fix:
1. Whenever `rescan_confirmed_active_` is set (`app.cpp:1193`, `:1324`, `:1397`), record a pending action that survives `resetTransactionHistoryCacheSession`:
   - **clamp** to `H`, parsed from "Rescanning last N blocks (from block H)" or "Rescanning from height H";
   - **full reset** if `H` is unknown or near genesis.
2. `loadTransactionHistoryCacheIfAvailable` applies it to the loaded W (`W = min(W, H)`, and `A` follows) and to the receive-scan heights, then clears it.

With daemon item D4 (wallet locator on clean shutdown, 3.2) shipped, the daemon's catch-up rescan line disappears from normal starts.

**Reorg overlap.**
- **Window extent.** Filter type 2 drops only txs deeper than N (`dx:rpcwallet.cpp:3740`), and `maxDepth = tip − W + margin`. So the window covers heights ≥ `W − margin + 1`, plus the mempool (depth 0 always passes).
- **Margin.**
  - `MAX_REORG_LENGTH` defaults to 99 (`dx:src/main.h:79`), and a longer reorg shuts the node down (`dx:main.cpp:4314-4333`).
  - A margin of 100 therefore leaves 1 block of slack, which tip drift between reading the tip and making the call can use up.
  - Use **margin ≈ 120** in steady state *(recommendation)*, and read the tip in the same worker job as the call.
- **First sweep after a load.** Use margin `120 + E` with E ≈ 1000 blocks (~10 h).
  - This absorbs wallet-notifier lag at save time: `ThreadNotifyWallets` applies blocks asynchronously (`dx:src/validationinterface.cpp:89-262`), and no RPC exposes the wallet's processed tip.
  - E is an inferred bound, not a measurement.
- **Large `-maxreorg`.** A `-maxreorg` set above the margin is caught by the anchor check.
- **Conflicted txs.** Conflicted txs (depth −1) are dropped by both the full and the windowed sweep (`dx:rpcwallet.cpp:3725`), so nothing changes there.

**Locked-wallet rule.**
- **The rule:** never advance or persist W from a sweep dispatched or completed while the wallet is locked, and never load or store the snapshot while it is locked.
- **What already protects this today:**
  - History scans don't run while locked (`src/services/scan_policy.h:48-51`).
  - An encrypted wallet's snapshot opens only after unlock (`app_network.cpp:797-798`, `:1944-1962`).
  - Locking keeps W and stores immediately (`:1851-1859`).
- **What breaks if it's ever violated:** a locked daemon wallet returns empty `saplingReceived` and `saplingSends`, because `GetSaplingHistoryKeys` needs spending keys (`dx:rpchushwallet.cpp:19-39`).
  - The fields the main consumer reads (txid, `transparentSends`) don't depend on the lock.
  - `locateSweepTxid` reads `saplingSends`, so it must never run on a locked wallet.

### 2.4 Backlog rule (required)

- **The risk.** Enrichment is capped at 25 per cycle, and the txids that miss the cap go to `history_enrich_backlog_` (`app_network.cpp:2843`).
  - A pure z→z send leaves only a change note, which produces no receive row (`network_refresh_service.cpp:972`).
  - So its txid exists **only** in a sweep result.
  - If W is persisted while those txids sit in the backlog, the sends are lost for good once the window moves past them.
- **The rule:** persist the backlog together with W.
- **Rejected fallback:** "persist `W = -1` while the backlog is non-empty" isn't enough on its own. Enrichment retries are unbounded (`network_refresh_service.cpp:1521`), so one txid that keeps failing would stop W from ever being persisted. If that fallback is wanted as well, cap the retries per txid.

### 2.5 Timeouts, large windows, banded full sweep

**Large windows.**
- The windowed sweep uses `rpc.call`, which has curl's 30 s timeout (`src/rpc/rpc_client.cpp:223`). Only full-sweep failures count toward the backoff (`app_network.cpp:2846-2853`).
- So a watermark saved days ago produces a huge "windowed" call that times out on every block, while the daemon keeps working on each abandoned call.
- Rule: if `tip − W > 2400` (about 1 day of 36 s blocks), use `callLong` and count failures the same way as for a full sweep.

**Banded full sweep.**
- **The call:** replace the single 113 s (fixed daemon) or 424 s (v1.3.0) call with depth bands, `z_listsentbyaddress(z, minconf=a, 2, maxDepth=b)`. `minconf` (`dx:rpcwallet.cpp:3732`) and filter type 2 (`:3740`) are independent filters, so each call returns exactly the depth band `[a, b]`.
- **Order:**
  - Go newest band first; the first band uses `minconf=0` so it includes the mempool.
  - Overlap bands by at least 10 blocks, plus the blocks that arrive between calls, and dedupe by txid.
  - Size bands adaptively, aiming for about 5 s each.
- **Scheduling:** run **one band per history cycle**, using the epoch-guarded band cursor from 2.2. All bands in one job would tie up the shared RPC worker for about 113 s, exactly as today.
- **Watermark:** take `W` from the tip at the first band, and set it only after every band has succeeded.
- **Don't page with `nCount`.** The iteration order is `nOrderPos`, not height (`:3709`, `:3746`).
- **Limits:**
  - Banding shortens each `LOCK2` hold but not the total CPU.
  - On v1.3.0 every call has a ~19.5 s floor, because every `CWalletTx` is deep-copied *(inference from the windowed measurement)*. There the bands need `callLong`.

**Backoff persistence.**
- Store the failure count and the relative delay (2.1), so a relaunch doesn't immediately re-run a sweep that keeps failing.
- Reset both when `daemon_id` changes. Otherwise a user who installs the fixed daemon could wait up to 6 h for a full sweep that would now succeed.

### 2.6 Safety net

- **When:** about every 7 days, run one banded full re-sweep.
- **Conditions:** only when synced, unlocked, idle (not on the Mining tab) and admitted by the governor.
- **On success**, update `last_full_*`. On failure, back off.
- **Why:** it catches what the guards can't see, such as a Console `z_importviewingkey`. That case is also missed within a session today.

### 2.7 Edge cases

| Case | Handling |
|---|---|
| Stored tip < current tip: `load()` doesn't check the hash (`cache.cpp:338-344`) | The anchor verify covers the watermark. Optionally extend `load()` too |
| Transparent send rows from `appendSentTransparentOutputs` aren't reproduced by windowed sweeps | Probably dropped on the next clean cycle (`network_refresh_service.cpp:1356-1368`, `:1528-1539`) *(inference)*. Negligible on DragonX, where consensus bans non-notary/burn t-outputs (`dx:main.cpp:1606`, `:1633`, `:1657`). Optional fix: carry over `send` rows that `listtransactions` didn't produce |
| `locateSweepTxid` runs a full sweep under the 30 s timeout (`app_network.cpp:6388`) | That times out on this wallet. Switch it to a windowed or banded call |
| Watch-only addresses aren't part of the snapshot key | Covered by the tripwire and the safety net |
| Switching between two wallets | `pruneOtherWallets` keeps one snapshot, so a switch still costs a full sweep (open question 4) |
| GUI crash mid-sweep | W advances only on `sentSweepOk`, and the write is atomic |
| Regtest or testnet | Different z-address prefixes, so a different snapshot key |

### 2.8 Test plan

**Unit tests** (`tests/test_phase4.cpp`):
- In `testTransactionHistoryCache` (`:9057`):
  - every `sent_sweep` field round-trips;
  - a missing key reads as −1;
  - a wrong type, `W > tip_height` or `A > W` reads as −1;
  - unknown keys are ignored.
- Signatures (~`:9244`): a change to W, A or `H_A` changes `slow`; a change to the backlog changes `content`.
- Policy (~`:15829`, `:16098-16213`):
  - the anchor is `W − margin`;
  - the first window after a load uses `120 + E`;
  - `tip − W > 2400` uses `callLong` and counts failures;
  - band planning: boundaries, overlap, dedupe, and cursor reset on an epoch change;
  - an anchor mismatch means a full sweep;
  - the pending rescan action clamps W;
  - `daemon_id` changes reset the backoff.

**Integration tests** (regtest or `-testnode`):
1. Full sweep, quit, relaunch: the log shows only `[z,0,2,N]` calls.
2. `invalidateblock` one block above A while the GUI is closed: the windowed sweep runs and returns the right rows.
3. `invalidateblock` below A: the verify fails and a banded full sweep runs.
4. More than 25 z→z sends, then quit before enrichment finishes: after relaunch every send appears.
5. A daemon started with `-rescan` while the GUI was closed: the sticky action clamps or resets W.
6. `restoreOriginalWallet`: W is invalidated.
7. Locked encrypted wallet: W doesn't advance and the snapshot isn't loaded.

**Mainnet copy:** relaunch against the isolated owner copy. Expect no `[z,0]` call and a sent sweep well under 1 s.

### 2.9 Expected effect

| Scenario | Today | With Part 1 |
|---|---|---|
| Relaunch or reconnect, fixed daemon | 113 s full sweep under `LOCK2`, 7 s from the timeout | `getblockhash` plus one windowed sweep over `120 + E` + the closure gap. Steady-state windows measure ~128 ms; the first window after a load is larger *(unmeasured)* |
| Relaunch, v1.3.0 daemon | 424 s; never fits in 120 s, so it loops in backoff with RSS at 19.4 GB | ~19.5 s+ windowed once a W exists. Seeding the first W needs the banded sweep |
| Safety net | — | One banded full sweep a week, each `LOCK2` hold about 5 s |

---

## 3. Part 2: daemon follow-up

### Where the 113 s goes (owner wallet, full sweep)

| Component | Cost | Basis |
|---|---|---|
| Trial decryption of outputs that aren't wallet notes: ~172,800 outputs × 5 ivks ≈ 864k failed tries × ~0.13 ms | **~112 s (~99%)** | Output count derived from tx sizes; per-try cost fitted to the full (112.7 s) and windowed (126 ms / 740 tries) measurements *(estimate)*. Code: `OutputDecryptions::Get` (`dx:rpchushwallet.cpp:87-100`), loop at `:303-346` |
| `DecryptSent` esk-path decrypts | **0 on this wallet** | No tx spends a wallet note, so the sends helper is gated off (`:586`) |
| UniValue build + copy under `LOCK2` | ~0.28 s | Synthetic benchmark on the repo's UniValue |
| Reply serialization (after the lock is released) | ~0.31-0.34 s | Same benchmark |

### Where the foreign outputs come from

**Sietch padding.** `z_sendmany` pads its z-recipients up to `-sietch-min-zouts` (default 7, clamped to 3-50; `dx:rpcwallet.cpp:5424-5434`), with change on top.
- Recipient and dummy outputs take the sender's ovk: the from-address's `fvk.ovk`, or the seed ovk for t-address sources (`dx:asyncrpcoperation_sendmany.cpp:352-368`, applied at `:437`).
- Consolidation (`saplingconsolidation.cpp:244`), sweep (`sweep.cpp:299`), shieldcoinbase (`:239-254`) and autoshield (`:435`) also add dummies with the sender's ovk.
- So a **sending** wallet's `DecryptSent` opens every one of its own dummies. That is the ~0.36 ms esk-path decrypt each, which is D2.
- Every other wallet sees them as foreign outputs and tries them under every ivk, which is D1.

**On this wallet** most foreign outputs are **not** sietch dummies. About 158.6k of the ~174k sit in the 9,597 txs with 10 or more outputs, which are mostly multi-recipient payouts *(inference from tx sizes)*. That doesn't change D1.

### Items

| # | Item | Output | Gain (owner wallet) | Risk | GUI change |
|---|---|---|---|---|---|
| D1 | Per-output negative ivk cache, plus parallel pre-fill of the txs a call selects | Byte-identical | Warm 113 s → 0.4-1 s. First full sweep after a restart ~20-25 s (pre-fill at the measured 4.75× warm-up scaling). A windowed call stays ~100 ms *(estimate)* | Low-med | None |
| D2 | `DecryptSent` cache per (output, ovk) | Byte-identical | 0 here; ~0.36 ms per send output on sending wallets | Low | None |
| D3 | `getalldata` balance loop | Byte-identical | ~20-35 s → 10-30 ms *(estimate)* | Low | None (the GUI never calls it) |
| D4 | Write the wallet locator on clean shutdown | n/a | Start: −49-62 s (post-rescan locator write, *inferred from the log gap*) and −5-10 s of block rescan; the ~25 s witness-root pass moves to the first live block. Shutdown: +3-60 s *(inferred)* | Medium | Check the stop policy |
| D5 | JSON trims | Byte-identical | 0.2-0.4 s | Low | None |
| D6 | Sends-only `returnType` (6th parameter) | **Changes output; opt-in** | 113 s → 0.1-0.3 s; 26.2 → ~4.1 MB *(estimate)* | Low | Probe + opt-in |
| D7 | Honour the address argument | **Changes output** | None (slower) | Breaks the GUI | Not recommended |

- **D6 is the single largest win**, but it needs a GUI release.
- **D1-D5 are invisible to the GUI.**
- **With Part 1 shipped, full sweeps become rare**, so D1 mostly helps older GUIs, other clients and the safety net.

### 3.1 D1: per-output decryption cache (byte-identical)

**Shape**
- A `mutable std::vector<uint64_t>` on `CWalletTx`, with one mask per `vShieldedOutput` index.
  - It is sized once, never serialized, and cleared in `Init()`.
  - Precedents: `cachedPlaintext` (`dx:src/wallet/wallet.h:340-348`) and the mutable caches at `:472-490`.
- Bit k means "history key k does not decrypt this output".
- An append-only ivk → index registry on `CWallet`, under `cs_wallet`. It is resolved once per RPC into `SaplingHistoryKeys::SpendableAddress` (`dx:rpchushwallet.cpp:19-39`).

**Logic in `OutputDecryptions::Get`**
- If the bit is set, return null without decrypting.
- After a failed `Decrypt`, set the bit.
- Never cache a success: wallet notes already carry `cachedPlaintext`.
- Keys with index ≥ 64 aren't cached.

**Invariant: the result is a pure function of (txid, n, ivk).** A v4 txid commits to every output byte, so:
- nothing ever goes stale;
- a newly added key has a clear bit and is still tried;
- locking, or losing keys, only means fewer keys are consulted;
- reorgs, deletions and reloads are harmless;
- output stays byte-identical.

**Thread safety**
- The mask is mutable behind a `const CWalletTx&`, so the compiler won't enforce locking. Every mask read and write therefore happens on the RPC thread under `cs_wallet`, with an `AssertLockHeld` in the accessor.
- Pre-fill workers follow `FillSaplingNoteCache`'s rule that workers must not lock, or even assert a lock (`dx:wallet.cpp:5745-5746`). They call only `decrypt_with_note` on pre-resolved (output, ivk) pairs and write private masks, which are merged serially after the join.
- Workers must not call `GetSaplingHistoryKeys` or `IsNullifierOfAddress`, because both assert locks.

**Fill**
- Lazily, as the helpers run.
- In addition, pre-fill in parallel, but **only the txs the call has already selected**:
  1. In `z_listsentbyaddress`, first collect the `wtx` pointers that pass the filters at `dx:rpcwallet.cpp:3722-3741`, honouring `nCount`.
  2. Pre-fill their outputs, using the 128-item threshold and `-witnessbuildthreads` from `dx:wallet.cpp:5739-5797`.
  3. Then emit.
- Pre-fill time is spent under `LOCK2`. For a full sweep that's ~20-25 s once; a windowed call stays small.
- Optionally, also record the misses `FindMySaplingNotes` already computes (`:2282-2322`).
- A startup precompute is opt-in only: ~112 CPU-s, about 24 s of wall time per start *(estimate)*.

**Also benefits:** `getalldata`'s tx list (`dx:rpcwallet.cpp:3254`) and `z_listreceivedaddress` (`:3555`).

**Tests**
- Extend the verbatim oracle `dx:src/gtest/test_wallet_history.cpp`: cold, warm and pre-filled runs must match byte for byte.
- A key added after the mask was filled, which opens a foreign output, must appear in the output.
- More than 64 keys; locked and encrypted wallets; `-witnessbuildthreads=1` vs 24; a `DEBUG_LOCKORDER` build.
- The mainnet A/B harness: `cmp` cold and warm replies against the captured 26,233,144-byte reply (`/home/d/dragonx-ab/mainnet-ab/out/B/cap/023_z_listsentbyaddress_addr0_full.json`).

**D2.** Cache `DecryptSent` results per (n, ovk index): the plaintext and address on success, or "no". That's about 650 B per entry. Key the seed ovk by value, so lock state still decides whether the cache is consulted. Use the same oracle tests as D1.

### 3.2 Smaller items

**D3: the `getalldata` balance loop** (`dx:rpcwallet.cpp:3108-3151`, under `LOCK2` at `:2998`).
- Today it copies each `SaplingNoteData` by value, witnesses included (`:3110`). For every unspent note and every address it re-derives the spending key and trial-decrypts (`:3117-3121`).
- Fix:
  - build `SaplingHistoryKeys` once, and reuse it at `:3233`;
  - take note data by `const&`;
  - use `OutputDecryptions(wtx, op.n).Get(a.ivk)` over `keys.spendable`, moving `OutputDecryptions` out of its anonymous namespace (`rpchushwallet.cpp:56-120`) into `rpchushwallet.h`.
- Keep two quirks for byte-identity, and fix them later as a separate change with a release note:
  - The no-op `txType == 3;` at `:3124` means locked notes are never counted as locked. When fixed, use a per-note local, `noteType = (txType == 0 && IsLockedNote(op)) ? 3 : txType`. `txType` is per tx (`:3043`), so assigning to it would mark every later note of the tx as locked.
  - Every address whose ivk opens the note is credited, because the `continue` doesn't exit the address loop.
- Don't switch to `GetFilteredNotes`: that changes the output.
- Test: an oracle gtest with a verbatim copy of the old loop (shared-ivk, locked-note, watch-only and encrypted cases).

**D4: write the wallet locator on clean shutdown.**
- **Evidence:** the copy, taken from a clean Windows shutdown, was 12 blocks behind its chainstate. So every start rescans those blocks.
- **Cause:**
  - `FlushStateToDisk` writes the locator only if `DATABASE_WRITE_INTERVAL` (3600 s, `dx:main.h:145`) has passed, even for `FLUSH_STATE_ALWAYS` (`dx:main.cpp:3838-3841`).
  - `Shutdown()` calls it at `dx:init.cpp:259`.
  - `pwalletMain->Flush(true)` at `:280` only flushes BDB.
- **Hook:** in `Shutdown()`, between `:259` and `:280`:

  ```cpp
  LOCK2(cs_main, pwalletMain->cs_wallet);
  const CBlockIndex* tip = pwalletMain->GetWalletTip();   // new accessor: pindexWalletTip is private (wallet.h:988-994)
  if (tip && tip != lastStamped)
      pwalletMain->SetBestChainINTERNAL(walletdb, chainActive.GetLocator(tip));
  ```

- **Don't route it through `CWallet::SetBestChain`.** That swaps in `pindexWalletTip` when set but falls back to the passed locator when it's null (`dx:wallet.cpp:829-834`). The null case is exactly "stamp the tip before the init rescan ran", which would skip the unscanned blocks for good.
- **The failure path needs the lock too.** AppInit2's failure path calls `Shutdown()` without `join_all` (`dx:bitcoind.cpp:232-243`), so the notifier can still be running. Reading the tip under `LOCK2` keeps that safe.
- **Don't drop the interval check globally.** `gettxoutsetinfo` and init also flush with ALWAYS, and each would then rewrite ~2 GB under `cs_main`.
- **Trade-off:** one locator write rewrites every note-bearing `CWalletTx`, ~2 GB here (`SetBestChainINTERNAL`, `dx:wallet.h:939-986`, WriteTx loop `:947-962`). The cost moves to shutdown rather than disappearing; see the D4 row in the table above.
- **Ship it together with a parallel witness-root check at init.**
- **Risk:** the GUI's stop policy (`src/daemon/daemon_controller.h:360`, `evaluateStopWait`) must allow about 16 s plus 3-60 s. Context: the 2026-09-30 Windows kill loop.
- **Tests:**
  - Stop within the hour and restart: no "Rescanning last" line, and identical balances.
  - Stop during the startup rescan: the locator does not advance.

**D5: JSON trims.**
- Use `static_cast<const CTransaction&>` at `dx:rpchushwallet.cpp:553`. Today it copies 190.6 MB of tx data per full sweep.
- Drop the `getValues` / reverse / `push_backV` copy (`dx:rpcwallet.cpp:3749-3755`). Collect the selected `wtx` pointers in the existing reverse loop, which keeps `nCount` picking the newest txs, then build `ret` oldest to newest. The same pattern is at `getalldata` `:3259-3265`.

**Also found; each needs its own design.**
- The hourly `SetBestChain` rewrites ~2 GB while `cs_main` is held (3-47 s gaps in the live log, *inferred from log gaps*).
- `FindMySaplingNotes` tries foreign outputs once per viewing key and again per address (`dx:wallet.cpp:2282-2322`). De-duplicating ivks would probably halve the per-block tries *(inference)*.

### 3.3 Options that change RPC output (need a GUI opt-in)

**D6: sends-only mode.**
- **The parameter:** add an optional 6th parameter `returnType` to `z_listsentbyaddress`, default 0 (unchanged output).
  - The code's meanings are 1 = spends, 2 = received, 3 = sends. The comment at `dx:rpchushwallet.cpp:561-566` swaps 1 and 3; fix it in the same change.
- **With `3` and no address filter:**
  - the spends helper still runs (`:569`), but its output isn't emitted;
  - `received`, where all the trial decryption happens, is skipped (`:576-581`);
  - `sends` is emitted (`:584-592`), and every tx is still pushed (`:594-598`).
  - The GUI gets the same txids and the same `sends.transparentSends`.
- **Daemon changes:**
  - Raise the check at `dx:rpcwallet.cpp:3606` from `> 5` to `> 6`, keeping the `== 3` rejection.
  - Change `:3693` to `params.size() >= 5`. Today it reads `nCount` only when `size() == 5`, so every 6-parameter call would silently drop it.
  - Read `params[5]`, reject anything outside 0-3, and document it in the help text. `dx:src/rpc/client.cpp:165` already converts index 5.
- **GUI side:** send 6 parameters. An old daemon throws its help text for more than 5, so fall back to 5 parameters and remember that per connection.
  - Alternative: a new RPC name, probed through "Method not found" the same way as `z_autoshieldstatus` (`app.h:1744-1747`).
- **Owner decision:** a variant that returns only txs with wallet spends would return 0 rows here, but it changes the `knownTxids` set.

**D7: honour the address argument.**
- The GUI passes `shieldedAddresses.front()` (`network_refresh_service.cpp:1319`) and relies on the daemon using `"*"` (`dx:rpcwallet.cpp:3743`). Honouring the argument by default would silently shrink GUI sweeps to one address.
- It isn't faster either: with a filter set, `fBool=true` runs `zsTxSendsToJSON` for every tx (`:586`).
- Recommendation: document `"*"` as a compatibility contract. If honouring the argument is ever wanted, put it behind a flag.

**Additive and safe for the GUI: a wallet-tip / history fingerprint.**
- Expose `pindexWalletTip`, for example in `getwalletinfo` or a new `z_getwallethistorystate(maxheight)` returning `{count, txid-hash, wallet_tip}`.
- The GUI ignores unknown JSON fields.
- This would make Part 1's invalidation exact, replacing the E-margin guess and the `txcount` tripwire.

---

## 4. Rollout and compatibility

| Step | Ships in | Works with | Notes |
|---|---|---|---|
| Part 1: watermark + anchor verify + backlog + sticky rescan + large-window path + banding + safety net | Next GUI release | Every daemon version; it uses only existing `z_listsentbyaddress` parameters, `getblockhash`, `getwalletinfo` and `getinfo` | Old GUI with a new snapshot: the key is ignored. New GUI with an old snapshot: one full sweep, which is today's behaviour. On v1.3.0, seeding the first W needs the banded sweep |
| D1, D2, D3, D5 | Daemon v1.3.1 / v1.3.2 | Invisible to the GUI | Gate each on the byte-identical oracle and the mainnet `cmp` |
| D6 + GUI probe | Daemon first, then GUI | The GUI falls back on old daemons | Makes the safety net and any full sweep cheap |
| D4 | Together with a parallel witness-root check | Invisible apart from shutdown time | Confirm the GUI stop policy first |

---

## 5. Open questions

1. How big is the backlog after the first full sweep on the owner's wallet? Unmeasured; it sizes the persisted `backlog` field.
2. What is the full-sweep time on the owner's 24-core Windows node? The 113 s was measured on Linux/WSL, 7 s inside the 120 s timeout.
3. How long does the wallet notifier lag after a catch-up on this wallet? This sets E.
4. Should the snapshot store keep more than one wallet, so that switching wallets also benefits?
5. D6 shape: a 6th parameter or a new RPC? Should a "wallet-spends-only" variant exist?
6. Does any client besides ObsidianDragon depend on `returnType 0` output? This sets D1's priority once D6 exists.
7. Are the 5 ivks distinct? Assumed. The per-address note counts sum to exactly 16,206, which fits but doesn't prove it.
8. Does the owner want the additive wallet-tip / fingerprint RPC, which makes Part 1's invalidation exact?