DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(history): persist the sent-sweep watermark (windowed relaunch)

Browse Source 
z_listsentbyaddress re-walks the whole wallet when there's no usable
watermark, which the daemon does under cs_main for every send -- hundreds
of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived
only in memory, so every relaunch, reconnect or daemon restart paid for a
full sweep. This persists it so those cases run a bounded windowed sweep
([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1).

- The watermark (W = last swept tip), its anchor (A = W - margin) and anchor
  hash (H_A = getblockhash(A)), plus the enrichment backlog and the full-
  sweep backoff, ride in the encrypted history snapshot as an optional
  `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as
  "no watermark" = today's behaviour, so older builds and payloads are safe).
- Before the first windowed sweep after a load, the anchor is verified
  (getblockhash(A) == H_A). A match means any fork is inside the window; a
  mismatch (reorg deeper than the margin, different chain/datadir) or an
  inability to check drops the watermark and sweeps the whole wallet.
- The first sweep after a load uses a bigger margin (absorbs notifier lag),
  and a watermark older than 2400 blocks uses the long timeout (callLong).
- The backlog is persisted WITH W: a pure z->z send's txid lives only in a
  sweep result and would be lost if the window moved past an un-enriched one.
- Watermark changes reach disk (folded into the store signature), and every
  hard case still forces a full sweep: reorg (anchor verify), GUI rescan /
  zap / import / address change, restoreOriginalWallet, and disconnect.

Deferred (follow-ups, each only turns a correct-but-slow case faster, never a
correctness risk): the banded full sweep for seeding on a pre-note-cache
daemon, the external-import tripwire, and the weekly safety-net re-sweep.
Part 2 (the daemon's decrypt cost) is already on the daemon's
perf/sapling-note-cache branch.

Pure helpers + cache round-trip + anchor-verify/large-window covered by
tests; the getblockhash-after-sweep updates the pinned RPC sequences.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-16 14:47:48 -05:00
co-authored by Claude Opus 4.8
parent 064c9d1b61
commit 3cda8c9e31
12 changed files with 807 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SENT-HISTORY-SWEEP.md
+432
View File
@@ -0,0 +1,432 @@
# Sent-history sweep: GUI watermark fix, then daemon follow-up
**Audience:** ObsidianDragon (GUI) team and the DragonX daemon maintainer.
**Date:** 2026-10-16. **Status:** proposal.
**Code cited**
- GUI: `ObsidianDragon` @ `064c9d1` (unprefixed paths).
- Daemon: `dragonx` @ `e9247f5c6` (`perf/history-rpc-copies`), paths prefixed `dx:`. `test/v1.3.1-candidate` @ `ff28e34c8` is the same plus a crash fix; on that branch, `dx:wallet.cpp` lines past ~1263 shift by +10.
Anything not marked *(inference)* or *(estimate)* was verified in the code or measured.
---
## 1. Problem
**The wallet measured:** an isolated copy of the owner's mainnet wallet, run in a network namespace with no peers.
- 16,206 Sapling notes, all unspent.
- 5 z-addresses and 11,889 wallet txs.
- Unencrypted; `wallet.dat` is about 2.1 GB.
| Call | v1.3.0 | After daemon PR-A+B+C |
|---|---|---|
| Balance RPCs | ~14 s | ~50 ms |
| Windowed `z_listsentbyaddress` (filter type 2) | 19.5 s | 128 ms |
| **Full `z_listsentbyaddress` (no watermark)** | **424 s**; RSS 8 → 19.4 GB, never released | **113 s** |
| Full-sweep reply | 26.2 MB (all 11,889 txs) | same |
### Why every launch pays for the full sweep
- **The watermark lives only in memory.** `history_sent_scan_height_` starts at `-1` (`src/app.h:1764`).
- **Every disconnect resets it to `-1`.** That covers daemon restarts, reconnects and wallet switches. The path is `onDisconnected` → `resetTransactionHistoryCacheSession` → `invalidateShieldedHistoryScanProgress(false)` (`src/app_network.cpp:936`, `:1847`, `:1889-1895`).
- **Other resets:**
- a locked-wallet connect;
- imports (`:4013`);
- rescans (`:7279`, `:7302`; `src/app.cpp:6298`) and zap (`app.cpp:6352`);
- a new, never-scanned z-address (`app_network.cpp:2593-2597`);
- the end of a warmup during which a rescan was observed (`:2186-2195`).
- **No watermark means a full sweep.** `sentSweepIsFull` is then true (`src/services/history_scan_policy.cpp:35-38`). The GUI sends `[z,0]` via `callLong(..., kFullSentSweepTimeoutSec = 120)` (`src/services/network_refresh_service.cpp:1324-1325`, `history_scan_policy.h:31`).
- **The daemon ignores the address argument.** It walks every wallet tx under `LOCK2(cs_main, cs_wallet)` and calls `zsWalletTxJSON(wtx, ret, "*", false, 0, …)` for each one (`dx:src/wallet/rpcwallet.cpp:3719-3746`; the call is at `:3743`).
- **The full sweep barely fits, or never fits, the timeout.**
- On the fixed daemon the sweep finishes only **7 s inside** the 120 s timeout.
- On v1.3.0 it never finishes. Failures back off 10 min → 1 h → 6 h (`fullSentSweepRetryDelaySec`). That backoff is also kept only in memory (`app.h:1765-1766`).
### What the GUI uses from the reply
- **Sweep consumer** (`network_refresh_service.cpp:1328-1332`, `appendSentTransparentOutputs` ~`:1076-1123`):
- It reads only `txid`, `confirmations`, `blocktime`/`time` and `sends.transparentSends`.
- It never reads `received` (20.8 MB of the 26.2 MB) or `spends`.
- **Shielded sends come from enrichment.** Every txid goes into `knownTxids` for `z_viewtransaction` enrichment, capped at 25 per cycle (`app.h:1753`). The rest wait in the in-memory `history_enrich_backlog_`.
- **One exception:** `locateSweepTxid` (seed-migration resume) does read `sends.saplingSends` (`app_network.cpp:6394`).
---
## 2. Part 1: GUI fix (persist the watermark)
**Goal:** a relaunch, reconnect or daemon restart runs only the windowed sweep, `[z,0,2,(tip-W)+margin]`. A full sweep then happens only in three cases:
- there is no persisted state;
- the persisted state fails verification;
- the periodic safety-net re-sweep is due.
### 2.1 Where to store it
**Use the existing encrypted history snapshot.** Store an optional `sent_sweep` object in its JSON payload, not in `settings.json` or `wallets.json`. Why:
- The watermark is only valid together with the rows it produced. Both are written in one `INSERT OR REPLACE` in WAL mode (`src/data/transaction_history_cache.cpp:438-457`), so a crash leaves the previous pair intact.
- The snapshot already provides what the watermark needs:
- per-wallet keying (BLAKE2b of the sorted z+t address set, `:157-186`);
- encryption (Argon2id from the passphrase, or salted BLAKE2b when the wallet is unencrypted);
- a tip check on load (`:338-344`);
- one-wallet pruning (`pruneOtherWallets`, `:655-672`).
- **Keep `kSchemaVersion = 1`.** `load()` rejects any other version (`:351`), so a bump would throw away every user's cache.
- Unknown keys are ignored, so older builds read new payloads safely.
- A missing `sent_sweep` means `W = -1`, which is today's behaviour.
| Field | Type | Meaning |
|---|---|---|
| `height` | int | `W`: the tip used for the sweep's `maxDepth`. Read in the same worker job, right before the call |
| `anchor_height` | int | `A = W - margin`: the bottom edge of the window the sweep covered |
| `anchor_hash` | hex | `H_A = getblockhash(A)`, fetched by the worker after the sweep succeeds |
| `backlog` | [txid] | `history_enrich_backlog_`. **Required**, see 2.4 |
| `full_failures`, `full_retry_delay_s` | int | Full-sweep backoff. The delay is relative to save time, not absolute |
| `daemon_id` | string | Daemon version + start time, from `getinfo` / uptime. If it changes, the restored backoff resets (2.5) |
| `last_full_height`, `last_full_time` | int | Last successful full sweep; drives the safety net (2.6) |
| `txcount`, `k` | int | Optional tripwire (2.3). Both captured **in the sweep's worker job** |
Do **not** reuse the snapshot's `tip_height`/`tip_hash` columns. They record the tip at store time, not at sweep time (`app_network.cpp:2069-2070`).
### 2.2 Code changes
| File | Change |
|---|---|
| `src/data/transaction_history_cache.h:20-28`, `:33-48`, `:79-84` | Add a `SentSweepState` struct to `LoadResult`, `SnapshotJob` and `replace()` |
| `src/data/transaction_history_cache.cpp:412-426` (`writeSnapshot`) | Emit `sent_sweep` only if `0 < A <= W <= job.tipHeight` |
| `src/data/transaction_history_cache.cpp:351-370` (`load`) | Parse with defaults (`W=-1`, empty backlog). Drop the object if it has the wrong type or `W` is above the stored `tip_height` |
| `src/data/transaction_history_store_policy.h:35-41` / `.cpp:62-113` | Add `W`/`A`/`H_A` to the `slow` signature. They move every block, so this means a write every 600 s and always on disconnect, lock and exit. Add the backlog to `content` (written within about 20 s). Otherwise an invalidation that only changes the watermark is skipped as "unchanged". Update both callers (`app_network.cpp:2012-2014`, `:2041-2045`). The epoch counter is never persisted, so it doesn't belong in a signature |
| `src/app_network.cpp:2066-2079` (`storeTransactionHistoryCacheIfAvailable`) | Fill `job->sentSweep` from the `history_sent_*` and backlog members |
| `src/app_network.cpp:1998-2017` (`loadTransactionHistoryCacheIfAvailable`) | Restore the members. `++history_sent_epoch_`. Set `history_sent_verify_pending_ = true`. Apply any **pending rescan action** (2.3) |
| `src/app_network.cpp:2028` | The store returns early when there are no rows. On a `persistCache=true` invalidation with no rows, call `clearWallet()` so a stale W can't survive on disk |
| `src/services/network_refresh_service.h:256-266`, `:299` | New snapshot/result fields: `anchorHeight`/`anchorHash` in, the verify result and `H_A` out, and the band cursor |
| `src/services/network_refresh_service.cpp:1314-1345` | Read the tip immediately before the call. After success, fetch `getblockhash(A)`. Add the verify step (2.3) and the large-window path (2.5) |
| `src/app_network.cpp:2784-2792` | Gate both the end-of-pass sweep and the **mid-pass** windowed sweep (`allowMidPassSentSweep`) on the pending verify. Extend the retry gate at `:2792`, which today applies only when `W < 0`, to large windows |
| `src/app_network.cpp:2843-2854` | Count large-window failures toward the backoff; today only `sentSweepFull` failures count (`:2849`). Store the band cursor in `App` and reset it with `history_sent_epoch_`, like the watermark |
| `src/app.cpp:restoreOriginalWallet` | Call `invalidateShieldedHistoryScanProgress(true)` before `onDisconnected`, as `rescanBlockchain` and zap already do (`app.cpp:6298`, `:6352`) |
| `src/services/history_scan_policy.{h,cpp}` | Pure helpers: `sentSweepAnchor(W)`, `sentSweepNeedsLongCall(tip, W)`, `sentSweepMargin(firstAfterLoad)`, `nextSentSweepBand(...)` |
### 2.3 Invalidation and safety rules
**Verify at the bottom edge of the window, not at its tip.**
- **The check:** before the first windowed sweep after a load, call `getblockhash(A)` and compare it with `H_A`.
- **Match:** any fork point lies above `A`, so it is inside the window. Run the windowed sweep.
- **Mismatch:** a reorg went deeper than the margin, or this is a different chain or datadir. Drop W and run the banded full sweep (2.5).
- **Why not the tip:** checking `getblockhash(W)` would trigger a full sweep on an ordinary one-block reorg of the tip, even though the window already covers that reorg.
| Event | Action | Mechanism |
|---|---|---|
| First windowed sweep after a load | Anchor verify (above) | `getblockhash`: O(1), under `cs_main` |
| Stored `tip_height` > current tip at load (replay after an unclean stop, older datadir) | **Today:** `load()` deletes the whole snapshot (`cache.cpp:338-344`, `clearWalletByHash` at `:341`). Rows and W are lost, and a full sweep runs | Keep as the default. **Optional improvement:** have `load()` keep the rows and `sent_sweep`, flag "ahead", and defer the sent sweep until `tip >= A`, then verify `H_A`. That requires changes in `sentSweepIsFull` (`history_scan_policy.cpp:37`: `W > tip` means full today), the gate at `app_network.cpp:2789-2792`, and the test at `tests/test_phase4.cpp:15831` |
| Wallet switch, or an address added or imported while the GUI was closed | Nothing new | Different address set → different snapshot key → full sweep |
| GUI-initiated rescan, zap, import, runtime rescan | Already `invalidate(true)`; with 2.2 it now reaches disk | `app.cpp:6298`, `:6352`; `app_network.cpp:4013`, `:7279`, `:7302` |
| New never-scanned z-address during a session | Already resets W (`:2593-2597`); the reset must be persisted | Signature change in 2.2 |
| Disconnect, daemon restart, same-wallet reconnect | Keep today's in-memory reset | The disk copy is written first (`:857`) and reloaded after connect |
| **Rescan observed in the daemon log** (the daemon's catch-up "Rescanning last N blocks (from block H)", or an external `-rescan`) | **Sticky pending action** (see below) | The warmup-end invalidation at `:2186-2195` runs *before* the snapshot loads, so today it can't reach a persisted W |
| Wallet changed outside the GUI with the same address set (Console `z_importviewingkey`, external `-rescan`, same-seed restore) | Optional tripwire: after the first windowed sweep returns R rows, run the banded full sweep if `txcount − txcount_saved > \|R\| − k + slack`. `k` is the rows of the saved sweep within the same `100 + E` depth band the first window uses | `getwalletinfo.txcount` = `mapWallet.size()` (`dx:rpcwallet.cpp:2791`). The GUI already fetches it on every connect (`network_refresh_service.cpp:251-252`, `:295-298`; `app_network.cpp:2140-2143`). Measured cost: ~8-20 ms warm, ~180 ms on the first call after a start |
| Daemon version change | Keep W. Reset the persisted backoff | Same filter semantics since 2020; v1.3.0 and the candidate both report 1.3.0.50 |
**The sticky rescan signal.** Why it's needed:
- During warmup no address list is fetched (`scan_policy.h:42-44`, `:64-68`; `app_network.cpp:2492`), and `state_.clear()` has emptied the rows (`wallet_state.h:406-409`).
- So when `invalidate(true)` runs at `app_network.cpp:2195`, it finds an empty identity and returns (`:2028`, `:2032`).
- The cache then loads afterwards (`:2608`) and would restore the old W unchanged.
The fix:
1. Whenever `rescan_confirmed_active_` is set (`app.cpp:1193`, `:1324`, `:1397`), record a pending action that survives `resetTransactionHistoryCacheSession`:
- **clamp** to `H`, parsed from "Rescanning last N blocks (from block H)" or "Rescanning from height H";
- **full reset** if `H` is unknown or near genesis.
2. `loadTransactionHistoryCacheIfAvailable` applies it to the loaded W (`W = min(W, H)`, and `A` follows) and to the receive-scan heights, then clears it.
With daemon item D4 (wallet locator on clean shutdown, 3.2) shipped, the daemon's catch-up rescan line disappears from normal starts.
**Reorg overlap.**
- **Window extent.** Filter type 2 drops only txs deeper than N (`dx:rpcwallet.cpp:3740`), and `maxDepth = tip − W + margin`. So the window covers heights ≥ `W − margin + 1`, plus the mempool (depth 0 always passes).
- **Margin.**
- `MAX_REORG_LENGTH` defaults to 99 (`dx:src/main.h:79`), and a longer reorg shuts the node down (`dx:main.cpp:4314-4333`).
- A margin of 100 therefore leaves 1 block of slack, which tip drift between reading the tip and making the call can use up.
- Use **margin ≈ 120** in steady state *(recommendation)*, and read the tip in the same worker job as the call.
- **First sweep after a load.** Use margin `120 + E` with E ≈ 1000 blocks (~10 h).
- This absorbs wallet-notifier lag at save time: `ThreadNotifyWallets` applies blocks asynchronously (`dx:src/validationinterface.cpp:89-262`), and no RPC exposes the wallet's processed tip.
- E is an inferred bound, not a measurement.
- **Large `-maxreorg`.** A `-maxreorg` set above the margin is caught by the anchor check.
- **Conflicted txs.** Conflicted txs (depth −1) are dropped by both the full and the windowed sweep (`dx:rpcwallet.cpp:3725`), so nothing changes there.
**Locked-wallet rule.**
- **The rule:** never advance or persist W from a sweep dispatched or completed while the wallet is locked, and never load or store the snapshot while it is locked.
- **What already protects this today:**
- History scans don't run while locked (`src/services/scan_policy.h:48-51`).
- An encrypted wallet's snapshot opens only after unlock (`app_network.cpp:797-798`, `:1944-1962`).
- Locking keeps W and stores immediately (`:1851-1859`).
- **What breaks if it's ever violated:** a locked daemon wallet returns empty `saplingReceived` and `saplingSends`, because `GetSaplingHistoryKeys` needs spending keys (`dx:rpchushwallet.cpp:19-39`).
- The fields the main consumer reads (txid, `transparentSends`) don't depend on the lock.
- `locateSweepTxid` reads `saplingSends`, so it must never run on a locked wallet.
### 2.4 Backlog rule (required)
- **The risk.** Enrichment is capped at 25 per cycle, and the txids that miss the cap go to `history_enrich_backlog_` (`app_network.cpp:2843`).
- A pure z→z send leaves only a change note, which produces no receive row (`network_refresh_service.cpp:972`).
- So its txid exists **only** in a sweep result.
- If W is persisted while those txids sit in the backlog, the sends are lost for good once the window moves past them.
- **The rule:** persist the backlog together with W.
- **Rejected fallback:** "persist `W = -1` while the backlog is non-empty" isn't enough on its own. Enrichment retries are unbounded (`network_refresh_service.cpp:1521`), so one txid that keeps failing would stop W from ever being persisted. If that fallback is wanted as well, cap the retries per txid.
### 2.5 Timeouts, large windows, banded full sweep
**Large windows.**
- The windowed sweep uses `rpc.call`, which has curl's 30 s timeout (`src/rpc/rpc_client.cpp:223`). Only full-sweep failures count toward the backoff (`app_network.cpp:2846-2853`).
- So a watermark saved days ago produces a huge "windowed" call that times out on every block, while the daemon keeps working on each abandoned call.
- Rule: if `tip − W > 2400` (about 1 day of 36 s blocks), use `callLong` and count failures the same way as for a full sweep.
**Banded full sweep.**
- **The call:** replace the single 113 s (fixed daemon) or 424 s (v1.3.0) call with depth bands, `z_listsentbyaddress(z, minconf=a, 2, maxDepth=b)`. `minconf` (`dx:rpcwallet.cpp:3732`) and filter type 2 (`:3740`) are independent filters, so each call returns exactly the depth band `[a, b]`.
- **Order:**
- Go newest band first; the first band uses `minconf=0` so it includes the mempool.
- Overlap bands by at least 10 blocks, plus the blocks that arrive between calls, and dedupe by txid.
- Size bands adaptively, aiming for about 5 s each.
- **Scheduling:** run **one band per history cycle**, using the epoch-guarded band cursor from 2.2. All bands in one job would tie up the shared RPC worker for about 113 s, exactly as today.
- **Watermark:** take `W` from the tip at the first band, and set it only after every band has succeeded.
- **Don't page with `nCount`.** The iteration order is `nOrderPos`, not height (`:3709`, `:3746`).
- **Limits:**
- Banding shortens each `LOCK2` hold but not the total CPU.
- On v1.3.0 every call has a ~19.5 s floor, because every `CWalletTx` is deep-copied *(inference from the windowed measurement)*. There the bands need `callLong`.
**Backoff persistence.**
- Store the failure count and the relative delay (2.1), so a relaunch doesn't immediately re-run a sweep that keeps failing.
- Reset both when `daemon_id` changes. Otherwise a user who installs the fixed daemon could wait up to 6 h for a full sweep that would now succeed.
### 2.6 Safety net
- **When:** about every 7 days, run one banded full re-sweep.
- **Conditions:** only when synced, unlocked, idle (not on the Mining tab) and admitted by the governor.
- **On success**, update `last_full_*`. On failure, back off.
- **Why:** it catches what the guards can't see, such as a Console `z_importviewingkey`. That case is also missed within a session today.
### 2.7 Edge cases
| Case | Handling |
|---|---|
| Stored tip < current tip: `load()` doesn't check the hash (`cache.cpp:338-344`) | The anchor verify covers the watermark. Optionally extend `load()` too |
| Transparent send rows from `appendSentTransparentOutputs` aren't reproduced by windowed sweeps | Probably dropped on the next clean cycle (`network_refresh_service.cpp:1356-1368`, `:1528-1539`) *(inference)*. Negligible on DragonX, where consensus bans non-notary/burn t-outputs (`dx:main.cpp:1606`, `:1633`, `:1657`). Optional fix: carry over `send` rows that `listtransactions` didn't produce |
| `locateSweepTxid` runs a full sweep under the 30 s timeout (`app_network.cpp:6388`) | That times out on this wallet. Switch it to a windowed or banded call |
| Watch-only addresses aren't part of the snapshot key | Covered by the tripwire and the safety net |
| Switching between two wallets | `pruneOtherWallets` keeps one snapshot, so a switch still costs a full sweep (open question 4) |
| GUI crash mid-sweep | W advances only on `sentSweepOk`, and the write is atomic |
| Regtest or testnet | Different z-address prefixes, so a different snapshot key |
### 2.8 Test plan
**Unit tests** (`tests/test_phase4.cpp`):
- In `testTransactionHistoryCache` (`:9057`):
- every `sent_sweep` field round-trips;
- a missing key reads as −1;
- a wrong type, `W > tip_height` or `A > W` reads as −1;
- unknown keys are ignored.
- Signatures (~`:9244`): a change to W, A or `H_A` changes `slow`; a change to the backlog changes `content`.
- Policy (~`:15829`, `:16098-16213`):
- the anchor is `W − margin`;
- the first window after a load uses `120 + E`;
- `tip − W > 2400` uses `callLong` and counts failures;
- band planning: boundaries, overlap, dedupe, and cursor reset on an epoch change;
- an anchor mismatch means a full sweep;
- the pending rescan action clamps W;
- `daemon_id` changes reset the backoff.
**Integration tests** (regtest or `-testnode`):
1. Full sweep, quit, relaunch: the log shows only `[z,0,2,N]` calls.
2. `invalidateblock` one block above A while the GUI is closed: the windowed sweep runs and returns the right rows.
3. `invalidateblock` below A: the verify fails and a banded full sweep runs.
4. More than 25 z→z sends, then quit before enrichment finishes: after relaunch every send appears.
5. A daemon started with `-rescan` while the GUI was closed: the sticky action clamps or resets W.
6. `restoreOriginalWallet`: W is invalidated.
7. Locked encrypted wallet: W doesn't advance and the snapshot isn't loaded.
**Mainnet copy:** relaunch against the isolated owner copy. Expect no `[z,0]` call and a sent sweep well under 1 s.
### 2.9 Expected effect
| Scenario | Today | With Part 1 |
|---|---|---|
| Relaunch or reconnect, fixed daemon | 113 s full sweep under `LOCK2`, 7 s from the timeout | `getblockhash` plus one windowed sweep over `120 + E` + the closure gap. Steady-state windows measure ~128 ms; the first window after a load is larger *(unmeasured)* |
| Relaunch, v1.3.0 daemon | 424 s; never fits in 120 s, so it loops in backoff with RSS at 19.4 GB | ~19.5 s+ windowed once a W exists. Seeding the first W needs the banded sweep |
| Safety net | — | One banded full sweep a week, each `LOCK2` hold about 5 s |
---
## 3. Part 2: daemon follow-up
### Where the 113 s goes (owner wallet, full sweep)
| Component | Cost | Basis |
|---|---|---|
| Trial decryption of outputs that aren't wallet notes: ~172,800 outputs × 5 ivks ≈ 864k failed tries × ~0.13 ms | **~112 s (~99%)** | Output count derived from tx sizes; per-try cost fitted to the full (112.7 s) and windowed (126 ms / 740 tries) measurements *(estimate)*. Code: `OutputDecryptions::Get` (`dx:rpchushwallet.cpp:87-100`), loop at `:303-346` |
| `DecryptSent` esk-path decrypts | **0 on this wallet** | No tx spends a wallet note, so the sends helper is gated off (`:586`) |
| UniValue build + copy under `LOCK2` | ~0.28 s | Synthetic benchmark on the repo's UniValue |
| Reply serialization (after the lock is released) | ~0.31-0.34 s | Same benchmark |
### Where the foreign outputs come from
**Sietch padding.** `z_sendmany` pads its z-recipients up to `-sietch-min-zouts` (default 7, clamped to 3-50; `dx:rpcwallet.cpp:5424-5434`), with change on top.
- Recipient and dummy outputs take the sender's ovk: the from-address's `fvk.ovk`, or the seed ovk for t-address sources (`dx:asyncrpcoperation_sendmany.cpp:352-368`, applied at `:437`).
- Consolidation (`saplingconsolidation.cpp:244`), sweep (`sweep.cpp:299`), shieldcoinbase (`:239-254`) and autoshield (`:435`) also add dummies with the sender's ovk.
- So a **sending** wallet's `DecryptSent` opens every one of its own dummies. That is the ~0.36 ms esk-path decrypt each, which is D2.
- Every other wallet sees them as foreign outputs and tries them under every ivk, which is D1.
**On this wallet** most foreign outputs are **not** sietch dummies. About 158.6k of the ~174k sit in the 9,597 txs with 10 or more outputs, which are mostly multi-recipient payouts *(inference from tx sizes)*. That doesn't change D1.
### Items
| # | Item | Output | Gain (owner wallet) | Risk | GUI change |
|---|---|---|---|---|---|
| D1 | Per-output negative ivk cache, plus parallel pre-fill of the txs a call selects | Byte-identical | Warm 113 s → 0.4-1 s. First full sweep after a restart ~20-25 s (pre-fill at the measured 4.75× warm-up scaling). A windowed call stays ~100 ms *(estimate)* | Low-med | None |
| D2 | `DecryptSent` cache per (output, ovk) | Byte-identical | 0 here; ~0.36 ms per send output on sending wallets | Low | None |
| D3 | `getalldata` balance loop | Byte-identical | ~20-35 s → 10-30 ms *(estimate)* | Low | None (the GUI never calls it) |
| D4 | Write the wallet locator on clean shutdown | n/a | Start: −49-62 s (post-rescan locator write, *inferred from the log gap*) and −5-10 s of block rescan; the ~25 s witness-root pass moves to the first live block. Shutdown: +3-60 s *(inferred)* | Medium | Check the stop policy |
| D5 | JSON trims | Byte-identical | 0.2-0.4 s | Low | None |
| D6 | Sends-only `returnType` (6th parameter) | **Changes output; opt-in** | 113 s → 0.1-0.3 s; 26.2 → ~4.1 MB *(estimate)* | Low | Probe + opt-in |
| D7 | Honour the address argument | **Changes output** | None (slower) | Breaks the GUI | Not recommended |
- **D6 is the single largest win**, but it needs a GUI release.
- **D1-D5 are invisible to the GUI.**
- **With Part 1 shipped, full sweeps become rare**, so D1 mostly helps older GUIs, other clients and the safety net.
### 3.1 D1: per-output decryption cache (byte-identical)
**Shape**
- A `mutable std::vector<uint64_t>` on `CWalletTx`, with one mask per `vShieldedOutput` index.
- It is sized once, never serialized, and cleared in `Init()`.
- Precedents: `cachedPlaintext` (`dx:src/wallet/wallet.h:340-348`) and the mutable caches at `:472-490`.
- Bit k means "history key k does not decrypt this output".
- An append-only ivk → index registry on `CWallet`, under `cs_wallet`. It is resolved once per RPC into `SaplingHistoryKeys::SpendableAddress` (`dx:rpchushwallet.cpp:19-39`).
**Logic in `OutputDecryptions::Get`**
- If the bit is set, return null without decrypting.
- After a failed `Decrypt`, set the bit.
- Never cache a success: wallet notes already carry `cachedPlaintext`.
- Keys with index ≥ 64 aren't cached.
**Invariant: the result is a pure function of (txid, n, ivk).** A v4 txid commits to every output byte, so:
- nothing ever goes stale;
- a newly added key has a clear bit and is still tried;
- locking, or losing keys, only means fewer keys are consulted;
- reorgs, deletions and reloads are harmless;
- output stays byte-identical.
**Thread safety**
- The mask is mutable behind a `const CWalletTx&`, so the compiler won't enforce locking. Every mask read and write therefore happens on the RPC thread under `cs_wallet`, with an `AssertLockHeld` in the accessor.
- Pre-fill workers follow `FillSaplingNoteCache`'s rule that workers must not lock, or even assert a lock (`dx:wallet.cpp:5745-5746`). They call only `decrypt_with_note` on pre-resolved (output, ivk) pairs and write private masks, which are merged serially after the join.
- Workers must not call `GetSaplingHistoryKeys` or `IsNullifierOfAddress`, because both assert locks.
**Fill**
- Lazily, as the helpers run.
- In addition, pre-fill in parallel, but **only the txs the call has already selected**:
1. In `z_listsentbyaddress`, first collect the `wtx` pointers that pass the filters at `dx:rpcwallet.cpp:3722-3741`, honouring `nCount`.
2. Pre-fill their outputs, using the 128-item threshold and `-witnessbuildthreads` from `dx:wallet.cpp:5739-5797`.
3. Then emit.
- Pre-fill time is spent under `LOCK2`. For a full sweep that's ~20-25 s once; a windowed call stays small.
- Optionally, also record the misses `FindMySaplingNotes` already computes (`:2282-2322`).
- A startup precompute is opt-in only: ~112 CPU-s, about 24 s of wall time per start *(estimate)*.
**Also benefits:** `getalldata`'s tx list (`dx:rpcwallet.cpp:3254`) and `z_listreceivedaddress` (`:3555`).
**Tests**
- Extend the verbatim oracle `dx:src/gtest/test_wallet_history.cpp`: cold, warm and pre-filled runs must match byte for byte.
- A key added after the mask was filled, which opens a foreign output, must appear in the output.
- More than 64 keys; locked and encrypted wallets; `-witnessbuildthreads=1` vs 24; a `DEBUG_LOCKORDER` build.
- The mainnet A/B harness: `cmp` cold and warm replies against the captured 26,233,144-byte reply (`/home/d/dragonx-ab/mainnet-ab/out/B/cap/023_z_listsentbyaddress_addr0_full.json`).
**D2.** Cache `DecryptSent` results per (n, ovk index): the plaintext and address on success, or "no". That's about 650 B per entry. Key the seed ovk by value, so lock state still decides whether the cache is consulted. Use the same oracle tests as D1.
### 3.2 Smaller items
**D3: the `getalldata` balance loop** (`dx:rpcwallet.cpp:3108-3151`, under `LOCK2` at `:2998`).
- Today it copies each `SaplingNoteData` by value, witnesses included (`:3110`). For every unspent note and every address it re-derives the spending key and trial-decrypts (`:3117-3121`).
- Fix:
- build `SaplingHistoryKeys` once, and reuse it at `:3233`;
- take note data by `const&`;
- use `OutputDecryptions(wtx, op.n).Get(a.ivk)` over `keys.spendable`, moving `OutputDecryptions` out of its anonymous namespace (`rpchushwallet.cpp:56-120`) into `rpchushwallet.h`.
- Keep two quirks for byte-identity, and fix them later as a separate change with a release note:
- The no-op `txType == 3;` at `:3124` means locked notes are never counted as locked. When fixed, use a per-note local, `noteType = (txType == 0 && IsLockedNote(op)) ? 3 : txType`. `txType` is per tx (`:3043`), so assigning to it would mark every later note of the tx as locked.
- Every address whose ivk opens the note is credited, because the `continue` doesn't exit the address loop.
- Don't switch to `GetFilteredNotes`: that changes the output.
- Test: an oracle gtest with a verbatim copy of the old loop (shared-ivk, locked-note, watch-only and encrypted cases).
**D4: write the wallet locator on clean shutdown.**
- **Evidence:** the copy, taken from a clean Windows shutdown, was 12 blocks behind its chainstate. So every start rescans those blocks.
- **Cause:**
- `FlushStateToDisk` writes the locator only if `DATABASE_WRITE_INTERVAL` (3600 s, `dx:main.h:145`) has passed, even for `FLUSH_STATE_ALWAYS` (`dx:main.cpp:3838-3841`).
- `Shutdown()` calls it at `dx:init.cpp:259`.
- `pwalletMain->Flush(true)` at `:280` only flushes BDB.
- **Hook:** in `Shutdown()`, between `:259` and `:280`:
```cpp
LOCK2(cs_main, pwalletMain->cs_wallet);
const CBlockIndex* tip = pwalletMain->GetWalletTip(); // new accessor: pindexWalletTip is private (wallet.h:988-994)
if (tip && tip != lastStamped)
pwalletMain->SetBestChainINTERNAL(walletdb, chainActive.GetLocator(tip));
```
- **Don't route it through `CWallet::SetBestChain`.** That swaps in `pindexWalletTip` when set but falls back to the passed locator when it's null (`dx:wallet.cpp:829-834`). The null case is exactly "stamp the tip before the init rescan ran", which would skip the unscanned blocks for good.
- **The failure path needs the lock too.** AppInit2's failure path calls `Shutdown()` without `join_all` (`dx:bitcoind.cpp:232-243`), so the notifier can still be running. Reading the tip under `LOCK2` keeps that safe.
- **Don't drop the interval check globally.** `gettxoutsetinfo` and init also flush with ALWAYS, and each would then rewrite ~2 GB under `cs_main`.
- **Trade-off:** one locator write rewrites every note-bearing `CWalletTx`, ~2 GB here (`SetBestChainINTERNAL`, `dx:wallet.h:939-986`, WriteTx loop `:947-962`). The cost moves to shutdown rather than disappearing; see the D4 row in the table above.
- **Ship it together with a parallel witness-root check at init.**
- **Risk:** the GUI's stop policy (`src/daemon/daemon_controller.h:360`, `evaluateStopWait`) must allow about 16 s plus 3-60 s. Context: the 2026-09-30 Windows kill loop.
- **Tests:**
- Stop within the hour and restart: no "Rescanning last" line, and identical balances.
- Stop during the startup rescan: the locator does not advance.
**D5: JSON trims.**
- Use `static_cast<const CTransaction&>` at `dx:rpchushwallet.cpp:553`. Today it copies 190.6 MB of tx data per full sweep.
- Drop the `getValues` / reverse / `push_backV` copy (`dx:rpcwallet.cpp:3749-3755`). Collect the selected `wtx` pointers in the existing reverse loop, which keeps `nCount` picking the newest txs, then build `ret` oldest to newest. The same pattern is at `getalldata` `:3259-3265`.
**Also found; each needs its own design.**
- The hourly `SetBestChain` rewrites ~2 GB while `cs_main` is held (3-47 s gaps in the live log, *inferred from log gaps*).
- `FindMySaplingNotes` tries foreign outputs once per viewing key and again per address (`dx:wallet.cpp:2282-2322`). De-duplicating ivks would probably halve the per-block tries *(inference)*.
### 3.3 Options that change RPC output (need a GUI opt-in)
**D6: sends-only mode.**
- **The parameter:** add an optional 6th parameter `returnType` to `z_listsentbyaddress`, default 0 (unchanged output).
- The code's meanings are 1 = spends, 2 = received, 3 = sends. The comment at `dx:rpchushwallet.cpp:561-566` swaps 1 and 3; fix it in the same change.
- **With `3` and no address filter:**
- the spends helper still runs (`:569`), but its output isn't emitted;
- `received`, where all the trial decryption happens, is skipped (`:576-581`);
- `sends` is emitted (`:584-592`), and every tx is still pushed (`:594-598`).
- The GUI gets the same txids and the same `sends.transparentSends`.
- **Daemon changes:**
- Raise the check at `dx:rpcwallet.cpp:3606` from `> 5` to `> 6`, keeping the `== 3` rejection.
- Change `:3693` to `params.size() >= 5`. Today it reads `nCount` only when `size() == 5`, so every 6-parameter call would silently drop it.
- Read `params[5]`, reject anything outside 0-3, and document it in the help text. `dx:src/rpc/client.cpp:165` already converts index 5.
- **GUI side:** send 6 parameters. An old daemon throws its help text for more than 5, so fall back to 5 parameters and remember that per connection.
- Alternative: a new RPC name, probed through "Method not found" the same way as `z_autoshieldstatus` (`app.h:1744-1747`).
- **Owner decision:** a variant that returns only txs with wallet spends would return 0 rows here, but it changes the `knownTxids` set.
**D7: honour the address argument.**
- The GUI passes `shieldedAddresses.front()` (`network_refresh_service.cpp:1319`) and relies on the daemon using `"*"` (`dx:rpcwallet.cpp:3743`). Honouring the argument by default would silently shrink GUI sweeps to one address.
- It isn't faster either: with a filter set, `fBool=true` runs `zsTxSendsToJSON` for every tx (`:586`).
- Recommendation: document `"*"` as a compatibility contract. If honouring the argument is ever wanted, put it behind a flag.
**Additive and safe for the GUI: a wallet-tip / history fingerprint.**
- Expose `pindexWalletTip`, for example in `getwalletinfo` or a new `z_getwallethistorystate(maxheight)` returning `{count, txid-hash, wallet_tip}`.
- The GUI ignores unknown JSON fields.
- This would make Part 1's invalidation exact, replacing the E-margin guess and the `txcount` tripwire.
---
## 4. Rollout and compatibility
| Step | Ships in | Works with | Notes |
|---|---|---|---|
| Part 1: watermark + anchor verify + backlog + sticky rescan + large-window path + banding + safety net | Next GUI release | Every daemon version; it uses only existing `z_listsentbyaddress` parameters, `getblockhash`, `getwalletinfo` and `getinfo` | Old GUI with a new snapshot: the key is ignored. New GUI with an old snapshot: one full sweep, which is today's behaviour. On v1.3.0, seeding the first W needs the banded sweep |
| D1, D2, D3, D5 | Daemon v1.3.1 / v1.3.2 | Invisible to the GUI | Gate each on the byte-identical oracle and the mainnet `cmp` |
| D6 + GUI probe | Daemon first, then GUI | The GUI falls back on old daemons | Makes the safety net and any full sweep cheap |
| D4 | Together with a parallel witness-root check | Invisible apart from shutdown time | Confirm the GUI stop policy first |
---
## 5. Open questions
1. How big is the backlog after the first full sweep on the owner's wallet? Unmeasured; it sizes the persisted `backlog` field.
2. What is the full-sweep time on the owner's 24-core Windows node? The 113 s was measured on Linux/WSL, 7 s inside the 120 s timeout.
3. How long does the wallet notifier lag after a catch-up on this wallet? This sets E.
4. Should the snapshot store keep more than one wallet, so that switching wallets also benefits?
5. D6 shape: a 6th parameter or a new RPC? Should a "wallet-spends-only" variant exist?
6. Does any client besides ObsidianDragon depend on `returnType 0` output? This sets D1's priority once D6 exists.
7. Are the 5 ivks distinct? Assumed. The per-address note counts sum to exactly 16,206, which fits but doesn't prove it.
8. Does the owner want the additive wallet-tip / fingerprint RPC, which makes Part 1's invalidation exact?