From 3cda8c9e3189c6fcf2932c53721ed25ceb49d9a8 Mon Sep 17 00:00:00 2001 From: DanS Date: Fri, 16 Oct 2026 14:47:48 -0500 Subject: [PATCH] perf(history): persist the sent-sweep watermark (windowed relaunch) z_listsentbyaddress re-walks the whole wallet when there's no usable watermark, which the daemon does under cs_main for every send -- hundreds of seconds, 400+ s and ~19 GB RSS on a large wallet. The watermark lived only in memory, so every relaunch, reconnect or daemon restart paid for a full sweep. This persists it so those cases run a bounded windowed sweep ([z, 0, 2, tip - W + margin]) instead (SENT-HISTORY-SWEEP Part 1). - The watermark (W = last swept tip), its anchor (A = W - margin) and anchor hash (H_A = getblockhash(A)), plus the enrichment backlog and the full- sweep backoff, ride in the encrypted history snapshot as an optional `sent_sweep` object (schema stays v1; a missing/inconsistent one reads as "no watermark" = today's behaviour, so older builds and payloads are safe). - Before the first windowed sweep after a load, the anchor is verified (getblockhash(A) == H_A). A match means any fork is inside the window; a mismatch (reorg deeper than the margin, different chain/datadir) or an inability to check drops the watermark and sweeps the whole wallet. - The first sweep after a load uses a bigger margin (absorbs notifier lag), and a watermark older than 2400 blocks uses the long timeout (callLong). - The backlog is persisted WITH W: a pure z->z send's txid lives only in a sweep result and would be lost if the window moved past an un-enriched one. - Watermark changes reach disk (folded into the store signature), and every hard case still forces a full sweep: reorg (anchor verify), GUI rescan / zap / import / address change, restoreOriginalWallet, and disconnect. Deferred (follow-ups, each only turns a correct-but-slow case faster, never a correctness risk): the banded full sweep for seeding on a pre-note-cache daemon, the external-import tripwire, and the weekly safety-net re-sweep. Part 2 (the daemon's decrypt cost) is already on the daemon's perf/sapling-note-cache branch. Pure helpers + cache round-trip + anchor-verify/large-window covered by tests; the getblockhash-after-sweep updates the pinned RPC sequences. Co-Authored-By: Claude Opus 4.8 --- SENT-HISTORY-SWEEP.md | 432 ++++++++++++++++++ src/app.h | 5 +- src/app_network.cpp | 48 +- src/data/transaction_history_cache.cpp | 42 +- src/data/transaction_history_cache.h | 23 +- src/data/transaction_history_store_policy.cpp | 17 +- src/data/transaction_history_store_policy.h | 10 +- src/services/history_scan_policy.cpp | 20 +- src/services/history_scan_policy.h | 16 +- src/services/network_refresh_service.cpp | 79 +++- src/services/network_refresh_service.h | 12 +- tests/test_phase4.cpp | 149 +++++- 12 files changed, 807 insertions(+), 46 deletions(-) create mode 100644 SENT-HISTORY-SWEEP.md diff --git a/SENT-HISTORY-SWEEP.md b/SENT-HISTORY-SWEEP.md new file mode 100644 index 0000000..eda8333 --- /dev/null +++ b/SENT-HISTORY-SWEEP.md @@ -0,0 +1,432 @@ +# Sent-history sweep: GUI watermark fix, then daemon follow-up + +**Audience:** ObsidianDragon (GUI) team and the DragonX daemon maintainer. +**Date:** 2026-10-16. **Status:** proposal. + +**Code cited** +- GUI: `ObsidianDragon` @ `064c9d1` (unprefixed paths). +- Daemon: `dragonx` @ `e9247f5c6` (`perf/history-rpc-copies`), paths prefixed `dx:`. `test/v1.3.1-candidate` @ `ff28e34c8` is the same plus a crash fix; on that branch, `dx:wallet.cpp` lines past ~1263 shift by +10. + +Anything not marked *(inference)* or *(estimate)* was verified in the code or measured. + +--- + +## 1. Problem + +**The wallet measured:** an isolated copy of the owner's mainnet wallet, run in a network namespace with no peers. +- 16,206 Sapling notes, all unspent. +- 5 z-addresses and 11,889 wallet txs. +- Unencrypted; `wallet.dat` is about 2.1 GB. + +| Call | v1.3.0 | After daemon PR-A+B+C | +|---|---|---| +| Balance RPCs | ~14 s | ~50 ms | +| Windowed `z_listsentbyaddress` (filter type 2) | 19.5 s | 128 ms | +| **Full `z_listsentbyaddress` (no watermark)** | **424 s**; RSS 8 → 19.4 GB, never released | **113 s** | +| Full-sweep reply | 26.2 MB (all 11,889 txs) | same | + +### Why every launch pays for the full sweep + +- **The watermark lives only in memory.** `history_sent_scan_height_` starts at `-1` (`src/app.h:1764`). +- **Every disconnect resets it to `-1`.** That covers daemon restarts, reconnects and wallet switches. The path is `onDisconnected` → `resetTransactionHistoryCacheSession` → `invalidateShieldedHistoryScanProgress(false)` (`src/app_network.cpp:936`, `:1847`, `:1889-1895`). +- **Other resets:** + - a locked-wallet connect; + - imports (`:4013`); + - rescans (`:7279`, `:7302`; `src/app.cpp:6298`) and zap (`app.cpp:6352`); + - a new, never-scanned z-address (`app_network.cpp:2593-2597`); + - the end of a warmup during which a rescan was observed (`:2186-2195`). +- **No watermark means a full sweep.** `sentSweepIsFull` is then true (`src/services/history_scan_policy.cpp:35-38`). The GUI sends `[z,0]` via `callLong(..., kFullSentSweepTimeoutSec = 120)` (`src/services/network_refresh_service.cpp:1324-1325`, `history_scan_policy.h:31`). +- **The daemon ignores the address argument.** It walks every wallet tx under `LOCK2(cs_main, cs_wallet)` and calls `zsWalletTxJSON(wtx, ret, "*", false, 0, …)` for each one (`dx:src/wallet/rpcwallet.cpp:3719-3746`; the call is at `:3743`). +- **The full sweep barely fits, or never fits, the timeout.** + - On the fixed daemon the sweep finishes only **7 s inside** the 120 s timeout. + - On v1.3.0 it never finishes. Failures back off 10 min → 1 h → 6 h (`fullSentSweepRetryDelaySec`). That backoff is also kept only in memory (`app.h:1765-1766`). + +### What the GUI uses from the reply + +- **Sweep consumer** (`network_refresh_service.cpp:1328-1332`, `appendSentTransparentOutputs` ~`:1076-1123`): + - It reads only `txid`, `confirmations`, `blocktime`/`time` and `sends.transparentSends`. + - It never reads `received` (20.8 MB of the 26.2 MB) or `spends`. +- **Shielded sends come from enrichment.** Every txid goes into `knownTxids` for `z_viewtransaction` enrichment, capped at 25 per cycle (`app.h:1753`). The rest wait in the in-memory `history_enrich_backlog_`. +- **One exception:** `locateSweepTxid` (seed-migration resume) does read `sends.saplingSends` (`app_network.cpp:6394`). + +--- + +## 2. Part 1: GUI fix (persist the watermark) + +**Goal:** a relaunch, reconnect or daemon restart runs only the windowed sweep, `[z,0,2,(tip-W)+margin]`. A full sweep then happens only in three cases: +- there is no persisted state; +- the persisted state fails verification; +- the periodic safety-net re-sweep is due. + +### 2.1 Where to store it + +**Use the existing encrypted history snapshot.** Store an optional `sent_sweep` object in its JSON payload, not in `settings.json` or `wallets.json`. Why: +- The watermark is only valid together with the rows it produced. Both are written in one `INSERT OR REPLACE` in WAL mode (`src/data/transaction_history_cache.cpp:438-457`), so a crash leaves the previous pair intact. +- The snapshot already provides what the watermark needs: + - per-wallet keying (BLAKE2b of the sorted z+t address set, `:157-186`); + - encryption (Argon2id from the passphrase, or salted BLAKE2b when the wallet is unencrypted); + - a tip check on load (`:338-344`); + - one-wallet pruning (`pruneOtherWallets`, `:655-672`). +- **Keep `kSchemaVersion = 1`.** `load()` rejects any other version (`:351`), so a bump would throw away every user's cache. + - Unknown keys are ignored, so older builds read new payloads safely. + - A missing `sent_sweep` means `W = -1`, which is today's behaviour. + +| Field | Type | Meaning | +|---|---|---| +| `height` | int | `W`: the tip used for the sweep's `maxDepth`. Read in the same worker job, right before the call | +| `anchor_height` | int | `A = W - margin`: the bottom edge of the window the sweep covered | +| `anchor_hash` | hex | `H_A = getblockhash(A)`, fetched by the worker after the sweep succeeds | +| `backlog` | [txid] | `history_enrich_backlog_`. **Required**, see 2.4 | +| `full_failures`, `full_retry_delay_s` | int | Full-sweep backoff. The delay is relative to save time, not absolute | +| `daemon_id` | string | Daemon version + start time, from `getinfo` / uptime. If it changes, the restored backoff resets (2.5) | +| `last_full_height`, `last_full_time` | int | Last successful full sweep; drives the safety net (2.6) | +| `txcount`, `k` | int | Optional tripwire (2.3). Both captured **in the sweep's worker job** | + +Do **not** reuse the snapshot's `tip_height`/`tip_hash` columns. They record the tip at store time, not at sweep time (`app_network.cpp:2069-2070`). + +### 2.2 Code changes + +| File | Change | +|---|---| +| `src/data/transaction_history_cache.h:20-28`, `:33-48`, `:79-84` | Add a `SentSweepState` struct to `LoadResult`, `SnapshotJob` and `replace()` | +| `src/data/transaction_history_cache.cpp:412-426` (`writeSnapshot`) | Emit `sent_sweep` only if `0 < A <= W <= job.tipHeight` | +| `src/data/transaction_history_cache.cpp:351-370` (`load`) | Parse with defaults (`W=-1`, empty backlog). Drop the object if it has the wrong type or `W` is above the stored `tip_height` | +| `src/data/transaction_history_store_policy.h:35-41` / `.cpp:62-113` | Add `W`/`A`/`H_A` to the `slow` signature. They move every block, so this means a write every 600 s and always on disconnect, lock and exit. Add the backlog to `content` (written within about 20 s). Otherwise an invalidation that only changes the watermark is skipped as "unchanged". Update both callers (`app_network.cpp:2012-2014`, `:2041-2045`). The epoch counter is never persisted, so it doesn't belong in a signature | +| `src/app_network.cpp:2066-2079` (`storeTransactionHistoryCacheIfAvailable`) | Fill `job->sentSweep` from the `history_sent_*` and backlog members | +| `src/app_network.cpp:1998-2017` (`loadTransactionHistoryCacheIfAvailable`) | Restore the members. `++history_sent_epoch_`. Set `history_sent_verify_pending_ = true`. Apply any **pending rescan action** (2.3) | +| `src/app_network.cpp:2028` | The store returns early when there are no rows. On a `persistCache=true` invalidation with no rows, call `clearWallet()` so a stale W can't survive on disk | +| `src/services/network_refresh_service.h:256-266`, `:299` | New snapshot/result fields: `anchorHeight`/`anchorHash` in, the verify result and `H_A` out, and the band cursor | +| `src/services/network_refresh_service.cpp:1314-1345` | Read the tip immediately before the call. After success, fetch `getblockhash(A)`. Add the verify step (2.3) and the large-window path (2.5) | +| `src/app_network.cpp:2784-2792` | Gate both the end-of-pass sweep and the **mid-pass** windowed sweep (`allowMidPassSentSweep`) on the pending verify. Extend the retry gate at `:2792`, which today applies only when `W < 0`, to large windows | +| `src/app_network.cpp:2843-2854` | Count large-window failures toward the backoff; today only `sentSweepFull` failures count (`:2849`). Store the band cursor in `App` and reset it with `history_sent_epoch_`, like the watermark | +| `src/app.cpp:restoreOriginalWallet` | Call `invalidateShieldedHistoryScanProgress(true)` before `onDisconnected`, as `rescanBlockchain` and zap already do (`app.cpp:6298`, `:6352`) | +| `src/services/history_scan_policy.{h,cpp}` | Pure helpers: `sentSweepAnchor(W)`, `sentSweepNeedsLongCall(tip, W)`, `sentSweepMargin(firstAfterLoad)`, `nextSentSweepBand(...)` | + +### 2.3 Invalidation and safety rules + +**Verify at the bottom edge of the window, not at its tip.** +- **The check:** before the first windowed sweep after a load, call `getblockhash(A)` and compare it with `H_A`. + - **Match:** any fork point lies above `A`, so it is inside the window. Run the windowed sweep. + - **Mismatch:** a reorg went deeper than the margin, or this is a different chain or datadir. Drop W and run the banded full sweep (2.5). +- **Why not the tip:** checking `getblockhash(W)` would trigger a full sweep on an ordinary one-block reorg of the tip, even though the window already covers that reorg. + +| Event | Action | Mechanism | +|---|---|---| +| First windowed sweep after a load | Anchor verify (above) | `getblockhash`: O(1), under `cs_main` | +| Stored `tip_height` > current tip at load (replay after an unclean stop, older datadir) | **Today:** `load()` deletes the whole snapshot (`cache.cpp:338-344`, `clearWalletByHash` at `:341`). Rows and W are lost, and a full sweep runs | Keep as the default. **Optional improvement:** have `load()` keep the rows and `sent_sweep`, flag "ahead", and defer the sent sweep until `tip >= A`, then verify `H_A`. That requires changes in `sentSweepIsFull` (`history_scan_policy.cpp:37`: `W > tip` means full today), the gate at `app_network.cpp:2789-2792`, and the test at `tests/test_phase4.cpp:15831` | +| Wallet switch, or an address added or imported while the GUI was closed | Nothing new | Different address set → different snapshot key → full sweep | +| GUI-initiated rescan, zap, import, runtime rescan | Already `invalidate(true)`; with 2.2 it now reaches disk | `app.cpp:6298`, `:6352`; `app_network.cpp:4013`, `:7279`, `:7302` | +| New never-scanned z-address during a session | Already resets W (`:2593-2597`); the reset must be persisted | Signature change in 2.2 | +| Disconnect, daemon restart, same-wallet reconnect | Keep today's in-memory reset | The disk copy is written first (`:857`) and reloaded after connect | +| **Rescan observed in the daemon log** (the daemon's catch-up "Rescanning last N blocks (from block H)", or an external `-rescan`) | **Sticky pending action** (see below) | The warmup-end invalidation at `:2186-2195` runs *before* the snapshot loads, so today it can't reach a persisted W | +| Wallet changed outside the GUI with the same address set (Console `z_importviewingkey`, external `-rescan`, same-seed restore) | Optional tripwire: after the first windowed sweep returns R rows, run the banded full sweep if `txcount − txcount_saved > \|R\| − k + slack`. `k` is the rows of the saved sweep within the same `100 + E` depth band the first window uses | `getwalletinfo.txcount` = `mapWallet.size()` (`dx:rpcwallet.cpp:2791`). The GUI already fetches it on every connect (`network_refresh_service.cpp:251-252`, `:295-298`; `app_network.cpp:2140-2143`). Measured cost: ~8-20 ms warm, ~180 ms on the first call after a start | +| Daemon version change | Keep W. Reset the persisted backoff | Same filter semantics since 2020; v1.3.0 and the candidate both report 1.3.0.50 | + +**The sticky rescan signal.** Why it's needed: +- During warmup no address list is fetched (`scan_policy.h:42-44`, `:64-68`; `app_network.cpp:2492`), and `state_.clear()` has emptied the rows (`wallet_state.h:406-409`). +- So when `invalidate(true)` runs at `app_network.cpp:2195`, it finds an empty identity and returns (`:2028`, `:2032`). +- The cache then loads afterwards (`:2608`) and would restore the old W unchanged. + +The fix: +1. Whenever `rescan_confirmed_active_` is set (`app.cpp:1193`, `:1324`, `:1397`), record a pending action that survives `resetTransactionHistoryCacheSession`: + - **clamp** to `H`, parsed from "Rescanning last N blocks (from block H)" or "Rescanning from height H"; + - **full reset** if `H` is unknown or near genesis. +2. `loadTransactionHistoryCacheIfAvailable` applies it to the loaded W (`W = min(W, H)`, and `A` follows) and to the receive-scan heights, then clears it. + +With daemon item D4 (wallet locator on clean shutdown, 3.2) shipped, the daemon's catch-up rescan line disappears from normal starts. + +**Reorg overlap.** +- **Window extent.** Filter type 2 drops only txs deeper than N (`dx:rpcwallet.cpp:3740`), and `maxDepth = tip − W + margin`. So the window covers heights ≥ `W − margin + 1`, plus the mempool (depth 0 always passes). +- **Margin.** + - `MAX_REORG_LENGTH` defaults to 99 (`dx:src/main.h:79`), and a longer reorg shuts the node down (`dx:main.cpp:4314-4333`). + - A margin of 100 therefore leaves 1 block of slack, which tip drift between reading the tip and making the call can use up. + - Use **margin ≈ 120** in steady state *(recommendation)*, and read the tip in the same worker job as the call. +- **First sweep after a load.** Use margin `120 + E` with E ≈ 1000 blocks (~10 h). + - This absorbs wallet-notifier lag at save time: `ThreadNotifyWallets` applies blocks asynchronously (`dx:src/validationinterface.cpp:89-262`), and no RPC exposes the wallet's processed tip. + - E is an inferred bound, not a measurement. +- **Large `-maxreorg`.** A `-maxreorg` set above the margin is caught by the anchor check. +- **Conflicted txs.** Conflicted txs (depth −1) are dropped by both the full and the windowed sweep (`dx:rpcwallet.cpp:3725`), so nothing changes there. + +**Locked-wallet rule.** +- **The rule:** never advance or persist W from a sweep dispatched or completed while the wallet is locked, and never load or store the snapshot while it is locked. +- **What already protects this today:** + - History scans don't run while locked (`src/services/scan_policy.h:48-51`). + - An encrypted wallet's snapshot opens only after unlock (`app_network.cpp:797-798`, `:1944-1962`). + - Locking keeps W and stores immediately (`:1851-1859`). +- **What breaks if it's ever violated:** a locked daemon wallet returns empty `saplingReceived` and `saplingSends`, because `GetSaplingHistoryKeys` needs spending keys (`dx:rpchushwallet.cpp:19-39`). + - The fields the main consumer reads (txid, `transparentSends`) don't depend on the lock. + - `locateSweepTxid` reads `saplingSends`, so it must never run on a locked wallet. + +### 2.4 Backlog rule (required) + +- **The risk.** Enrichment is capped at 25 per cycle, and the txids that miss the cap go to `history_enrich_backlog_` (`app_network.cpp:2843`). + - A pure z→z send leaves only a change note, which produces no receive row (`network_refresh_service.cpp:972`). + - So its txid exists **only** in a sweep result. + - If W is persisted while those txids sit in the backlog, the sends are lost for good once the window moves past them. +- **The rule:** persist the backlog together with W. +- **Rejected fallback:** "persist `W = -1` while the backlog is non-empty" isn't enough on its own. Enrichment retries are unbounded (`network_refresh_service.cpp:1521`), so one txid that keeps failing would stop W from ever being persisted. If that fallback is wanted as well, cap the retries per txid. + +### 2.5 Timeouts, large windows, banded full sweep + +**Large windows.** +- The windowed sweep uses `rpc.call`, which has curl's 30 s timeout (`src/rpc/rpc_client.cpp:223`). Only full-sweep failures count toward the backoff (`app_network.cpp:2846-2853`). +- So a watermark saved days ago produces a huge "windowed" call that times out on every block, while the daemon keeps working on each abandoned call. +- Rule: if `tip − W > 2400` (about 1 day of 36 s blocks), use `callLong` and count failures the same way as for a full sweep. + +**Banded full sweep.** +- **The call:** replace the single 113 s (fixed daemon) or 424 s (v1.3.0) call with depth bands, `z_listsentbyaddress(z, minconf=a, 2, maxDepth=b)`. `minconf` (`dx:rpcwallet.cpp:3732`) and filter type 2 (`:3740`) are independent filters, so each call returns exactly the depth band `[a, b]`. +- **Order:** + - Go newest band first; the first band uses `minconf=0` so it includes the mempool. + - Overlap bands by at least 10 blocks, plus the blocks that arrive between calls, and dedupe by txid. + - Size bands adaptively, aiming for about 5 s each. +- **Scheduling:** run **one band per history cycle**, using the epoch-guarded band cursor from 2.2. All bands in one job would tie up the shared RPC worker for about 113 s, exactly as today. +- **Watermark:** take `W` from the tip at the first band, and set it only after every band has succeeded. +- **Don't page with `nCount`.** The iteration order is `nOrderPos`, not height (`:3709`, `:3746`). +- **Limits:** + - Banding shortens each `LOCK2` hold but not the total CPU. + - On v1.3.0 every call has a ~19.5 s floor, because every `CWalletTx` is deep-copied *(inference from the windowed measurement)*. There the bands need `callLong`. + +**Backoff persistence.** +- Store the failure count and the relative delay (2.1), so a relaunch doesn't immediately re-run a sweep that keeps failing. +- Reset both when `daemon_id` changes. Otherwise a user who installs the fixed daemon could wait up to 6 h for a full sweep that would now succeed. + +### 2.6 Safety net + +- **When:** about every 7 days, run one banded full re-sweep. +- **Conditions:** only when synced, unlocked, idle (not on the Mining tab) and admitted by the governor. +- **On success**, update `last_full_*`. On failure, back off. +- **Why:** it catches what the guards can't see, such as a Console `z_importviewingkey`. That case is also missed within a session today. + +### 2.7 Edge cases + +| Case | Handling | +|---|---| +| Stored tip < current tip: `load()` doesn't check the hash (`cache.cpp:338-344`) | The anchor verify covers the watermark. Optionally extend `load()` too | +| Transparent send rows from `appendSentTransparentOutputs` aren't reproduced by windowed sweeps | Probably dropped on the next clean cycle (`network_refresh_service.cpp:1356-1368`, `:1528-1539`) *(inference)*. Negligible on DragonX, where consensus bans non-notary/burn t-outputs (`dx:main.cpp:1606`, `:1633`, `:1657`). Optional fix: carry over `send` rows that `listtransactions` didn't produce | +| `locateSweepTxid` runs a full sweep under the 30 s timeout (`app_network.cpp:6388`) | That times out on this wallet. Switch it to a windowed or banded call | +| Watch-only addresses aren't part of the snapshot key | Covered by the tripwire and the safety net | +| Switching between two wallets | `pruneOtherWallets` keeps one snapshot, so a switch still costs a full sweep (open question 4) | +| GUI crash mid-sweep | W advances only on `sentSweepOk`, and the write is atomic | +| Regtest or testnet | Different z-address prefixes, so a different snapshot key | + +### 2.8 Test plan + +**Unit tests** (`tests/test_phase4.cpp`): +- In `testTransactionHistoryCache` (`:9057`): + - every `sent_sweep` field round-trips; + - a missing key reads as −1; + - a wrong type, `W > tip_height` or `A > W` reads as −1; + - unknown keys are ignored. +- Signatures (~`:9244`): a change to W, A or `H_A` changes `slow`; a change to the backlog changes `content`. +- Policy (~`:15829`, `:16098-16213`): + - the anchor is `W − margin`; + - the first window after a load uses `120 + E`; + - `tip − W > 2400` uses `callLong` and counts failures; + - band planning: boundaries, overlap, dedupe, and cursor reset on an epoch change; + - an anchor mismatch means a full sweep; + - the pending rescan action clamps W; + - `daemon_id` changes reset the backoff. + +**Integration tests** (regtest or `-testnode`): +1. Full sweep, quit, relaunch: the log shows only `[z,0,2,N]` calls. +2. `invalidateblock` one block above A while the GUI is closed: the windowed sweep runs and returns the right rows. +3. `invalidateblock` below A: the verify fails and a banded full sweep runs. +4. More than 25 z→z sends, then quit before enrichment finishes: after relaunch every send appears. +5. A daemon started with `-rescan` while the GUI was closed: the sticky action clamps or resets W. +6. `restoreOriginalWallet`: W is invalidated. +7. Locked encrypted wallet: W doesn't advance and the snapshot isn't loaded. + +**Mainnet copy:** relaunch against the isolated owner copy. Expect no `[z,0]` call and a sent sweep well under 1 s. + +### 2.9 Expected effect + +| Scenario | Today | With Part 1 | +|---|---|---| +| Relaunch or reconnect, fixed daemon | 113 s full sweep under `LOCK2`, 7 s from the timeout | `getblockhash` plus one windowed sweep over `120 + E` + the closure gap. Steady-state windows measure ~128 ms; the first window after a load is larger *(unmeasured)* | +| Relaunch, v1.3.0 daemon | 424 s; never fits in 120 s, so it loops in backoff with RSS at 19.4 GB | ~19.5 s+ windowed once a W exists. Seeding the first W needs the banded sweep | +| Safety net | — | One banded full sweep a week, each `LOCK2` hold about 5 s | + +--- + +## 3. Part 2: daemon follow-up + +### Where the 113 s goes (owner wallet, full sweep) + +| Component | Cost | Basis | +|---|---|---| +| Trial decryption of outputs that aren't wallet notes: ~172,800 outputs × 5 ivks ≈ 864k failed tries × ~0.13 ms | **~112 s (~99%)** | Output count derived from tx sizes; per-try cost fitted to the full (112.7 s) and windowed (126 ms / 740 tries) measurements *(estimate)*. Code: `OutputDecryptions::Get` (`dx:rpchushwallet.cpp:87-100`), loop at `:303-346` | +| `DecryptSent` esk-path decrypts | **0 on this wallet** | No tx spends a wallet note, so the sends helper is gated off (`:586`) | +| UniValue build + copy under `LOCK2` | ~0.28 s | Synthetic benchmark on the repo's UniValue | +| Reply serialization (after the lock is released) | ~0.31-0.34 s | Same benchmark | + +### Where the foreign outputs come from + +**Sietch padding.** `z_sendmany` pads its z-recipients up to `-sietch-min-zouts` (default 7, clamped to 3-50; `dx:rpcwallet.cpp:5424-5434`), with change on top. +- Recipient and dummy outputs take the sender's ovk: the from-address's `fvk.ovk`, or the seed ovk for t-address sources (`dx:asyncrpcoperation_sendmany.cpp:352-368`, applied at `:437`). +- Consolidation (`saplingconsolidation.cpp:244`), sweep (`sweep.cpp:299`), shieldcoinbase (`:239-254`) and autoshield (`:435`) also add dummies with the sender's ovk. +- So a **sending** wallet's `DecryptSent` opens every one of its own dummies. That is the ~0.36 ms esk-path decrypt each, which is D2. +- Every other wallet sees them as foreign outputs and tries them under every ivk, which is D1. + +**On this wallet** most foreign outputs are **not** sietch dummies. About 158.6k of the ~174k sit in the 9,597 txs with 10 or more outputs, which are mostly multi-recipient payouts *(inference from tx sizes)*. That doesn't change D1. + +### Items + +| # | Item | Output | Gain (owner wallet) | Risk | GUI change | +|---|---|---|---|---|---| +| D1 | Per-output negative ivk cache, plus parallel pre-fill of the txs a call selects | Byte-identical | Warm 113 s → 0.4-1 s. First full sweep after a restart ~20-25 s (pre-fill at the measured 4.75× warm-up scaling). A windowed call stays ~100 ms *(estimate)* | Low-med | None | +| D2 | `DecryptSent` cache per (output, ovk) | Byte-identical | 0 here; ~0.36 ms per send output on sending wallets | Low | None | +| D3 | `getalldata` balance loop | Byte-identical | ~20-35 s → 10-30 ms *(estimate)* | Low | None (the GUI never calls it) | +| D4 | Write the wallet locator on clean shutdown | n/a | Start: −49-62 s (post-rescan locator write, *inferred from the log gap*) and −5-10 s of block rescan; the ~25 s witness-root pass moves to the first live block. Shutdown: +3-60 s *(inferred)* | Medium | Check the stop policy | +| D5 | JSON trims | Byte-identical | 0.2-0.4 s | Low | None | +| D6 | Sends-only `returnType` (6th parameter) | **Changes output; opt-in** | 113 s → 0.1-0.3 s; 26.2 → ~4.1 MB *(estimate)* | Low | Probe + opt-in | +| D7 | Honour the address argument | **Changes output** | None (slower) | Breaks the GUI | Not recommended | + +- **D6 is the single largest win**, but it needs a GUI release. +- **D1-D5 are invisible to the GUI.** +- **With Part 1 shipped, full sweeps become rare**, so D1 mostly helps older GUIs, other clients and the safety net. + +### 3.1 D1: per-output decryption cache (byte-identical) + +**Shape** +- A `mutable std::vector` on `CWalletTx`, with one mask per `vShieldedOutput` index. + - It is sized once, never serialized, and cleared in `Init()`. + - Precedents: `cachedPlaintext` (`dx:src/wallet/wallet.h:340-348`) and the mutable caches at `:472-490`. +- Bit k means "history key k does not decrypt this output". +- An append-only ivk → index registry on `CWallet`, under `cs_wallet`. It is resolved once per RPC into `SaplingHistoryKeys::SpendableAddress` (`dx:rpchushwallet.cpp:19-39`). + +**Logic in `OutputDecryptions::Get`** +- If the bit is set, return null without decrypting. +- After a failed `Decrypt`, set the bit. +- Never cache a success: wallet notes already carry `cachedPlaintext`. +- Keys with index ≥ 64 aren't cached. + +**Invariant: the result is a pure function of (txid, n, ivk).** A v4 txid commits to every output byte, so: +- nothing ever goes stale; +- a newly added key has a clear bit and is still tried; +- locking, or losing keys, only means fewer keys are consulted; +- reorgs, deletions and reloads are harmless; +- output stays byte-identical. + +**Thread safety** +- The mask is mutable behind a `const CWalletTx&`, so the compiler won't enforce locking. Every mask read and write therefore happens on the RPC thread under `cs_wallet`, with an `AssertLockHeld` in the accessor. +- Pre-fill workers follow `FillSaplingNoteCache`'s rule that workers must not lock, or even assert a lock (`dx:wallet.cpp:5745-5746`). They call only `decrypt_with_note` on pre-resolved (output, ivk) pairs and write private masks, which are merged serially after the join. +- Workers must not call `GetSaplingHistoryKeys` or `IsNullifierOfAddress`, because both assert locks. + +**Fill** +- Lazily, as the helpers run. +- In addition, pre-fill in parallel, but **only the txs the call has already selected**: + 1. In `z_listsentbyaddress`, first collect the `wtx` pointers that pass the filters at `dx:rpcwallet.cpp:3722-3741`, honouring `nCount`. + 2. Pre-fill their outputs, using the 128-item threshold and `-witnessbuildthreads` from `dx:wallet.cpp:5739-5797`. + 3. Then emit. +- Pre-fill time is spent under `LOCK2`. For a full sweep that's ~20-25 s once; a windowed call stays small. +- Optionally, also record the misses `FindMySaplingNotes` already computes (`:2282-2322`). +- A startup precompute is opt-in only: ~112 CPU-s, about 24 s of wall time per start *(estimate)*. + +**Also benefits:** `getalldata`'s tx list (`dx:rpcwallet.cpp:3254`) and `z_listreceivedaddress` (`:3555`). + +**Tests** +- Extend the verbatim oracle `dx:src/gtest/test_wallet_history.cpp`: cold, warm and pre-filled runs must match byte for byte. +- A key added after the mask was filled, which opens a foreign output, must appear in the output. +- More than 64 keys; locked and encrypted wallets; `-witnessbuildthreads=1` vs 24; a `DEBUG_LOCKORDER` build. +- The mainnet A/B harness: `cmp` cold and warm replies against the captured 26,233,144-byte reply (`/home/d/dragonx-ab/mainnet-ab/out/B/cap/023_z_listsentbyaddress_addr0_full.json`). + +**D2.** Cache `DecryptSent` results per (n, ovk index): the plaintext and address on success, or "no". That's about 650 B per entry. Key the seed ovk by value, so lock state still decides whether the cache is consulted. Use the same oracle tests as D1. + +### 3.2 Smaller items + +**D3: the `getalldata` balance loop** (`dx:rpcwallet.cpp:3108-3151`, under `LOCK2` at `:2998`). +- Today it copies each `SaplingNoteData` by value, witnesses included (`:3110`). For every unspent note and every address it re-derives the spending key and trial-decrypts (`:3117-3121`). +- Fix: + - build `SaplingHistoryKeys` once, and reuse it at `:3233`; + - take note data by `const&`; + - use `OutputDecryptions(wtx, op.n).Get(a.ivk)` over `keys.spendable`, moving `OutputDecryptions` out of its anonymous namespace (`rpchushwallet.cpp:56-120`) into `rpchushwallet.h`. +- Keep two quirks for byte-identity, and fix them later as a separate change with a release note: + - The no-op `txType == 3;` at `:3124` means locked notes are never counted as locked. When fixed, use a per-note local, `noteType = (txType == 0 && IsLockedNote(op)) ? 3 : txType`. `txType` is per tx (`:3043`), so assigning to it would mark every later note of the tx as locked. + - Every address whose ivk opens the note is credited, because the `continue` doesn't exit the address loop. +- Don't switch to `GetFilteredNotes`: that changes the output. +- Test: an oracle gtest with a verbatim copy of the old loop (shared-ivk, locked-note, watch-only and encrypted cases). + +**D4: write the wallet locator on clean shutdown.** +- **Evidence:** the copy, taken from a clean Windows shutdown, was 12 blocks behind its chainstate. So every start rescans those blocks. +- **Cause:** + - `FlushStateToDisk` writes the locator only if `DATABASE_WRITE_INTERVAL` (3600 s, `dx:main.h:145`) has passed, even for `FLUSH_STATE_ALWAYS` (`dx:main.cpp:3838-3841`). + - `Shutdown()` calls it at `dx:init.cpp:259`. + - `pwalletMain->Flush(true)` at `:280` only flushes BDB. +- **Hook:** in `Shutdown()`, between `:259` and `:280`: + + ```cpp + LOCK2(cs_main, pwalletMain->cs_wallet); + const CBlockIndex* tip = pwalletMain->GetWalletTip(); // new accessor: pindexWalletTip is private (wallet.h:988-994) + if (tip && tip != lastStamped) + pwalletMain->SetBestChainINTERNAL(walletdb, chainActive.GetLocator(tip)); + ``` + +- **Don't route it through `CWallet::SetBestChain`.** That swaps in `pindexWalletTip` when set but falls back to the passed locator when it's null (`dx:wallet.cpp:829-834`). The null case is exactly "stamp the tip before the init rescan ran", which would skip the unscanned blocks for good. +- **The failure path needs the lock too.** AppInit2's failure path calls `Shutdown()` without `join_all` (`dx:bitcoind.cpp:232-243`), so the notifier can still be running. Reading the tip under `LOCK2` keeps that safe. +- **Don't drop the interval check globally.** `gettxoutsetinfo` and init also flush with ALWAYS, and each would then rewrite ~2 GB under `cs_main`. +- **Trade-off:** one locator write rewrites every note-bearing `CWalletTx`, ~2 GB here (`SetBestChainINTERNAL`, `dx:wallet.h:939-986`, WriteTx loop `:947-962`). The cost moves to shutdown rather than disappearing; see the D4 row in the table above. +- **Ship it together with a parallel witness-root check at init.** +- **Risk:** the GUI's stop policy (`src/daemon/daemon_controller.h:360`, `evaluateStopWait`) must allow about 16 s plus 3-60 s. Context: the 2026-09-30 Windows kill loop. +- **Tests:** + - Stop within the hour and restart: no "Rescanning last" line, and identical balances. + - Stop during the startup rescan: the locator does not advance. + +**D5: JSON trims.** +- Use `static_cast` at `dx:rpchushwallet.cpp:553`. Today it copies 190.6 MB of tx data per full sweep. +- Drop the `getValues` / reverse / `push_backV` copy (`dx:rpcwallet.cpp:3749-3755`). Collect the selected `wtx` pointers in the existing reverse loop, which keeps `nCount` picking the newest txs, then build `ret` oldest to newest. The same pattern is at `getalldata` `:3259-3265`. + +**Also found; each needs its own design.** +- The hourly `SetBestChain` rewrites ~2 GB while `cs_main` is held (3-47 s gaps in the live log, *inferred from log gaps*). +- `FindMySaplingNotes` tries foreign outputs once per viewing key and again per address (`dx:wallet.cpp:2282-2322`). De-duplicating ivks would probably halve the per-block tries *(inference)*. + +### 3.3 Options that change RPC output (need a GUI opt-in) + +**D6: sends-only mode.** +- **The parameter:** add an optional 6th parameter `returnType` to `z_listsentbyaddress`, default 0 (unchanged output). + - The code's meanings are 1 = spends, 2 = received, 3 = sends. The comment at `dx:rpchushwallet.cpp:561-566` swaps 1 and 3; fix it in the same change. +- **With `3` and no address filter:** + - the spends helper still runs (`:569`), but its output isn't emitted; + - `received`, where all the trial decryption happens, is skipped (`:576-581`); + - `sends` is emitted (`:584-592`), and every tx is still pushed (`:594-598`). + - The GUI gets the same txids and the same `sends.transparentSends`. +- **Daemon changes:** + - Raise the check at `dx:rpcwallet.cpp:3606` from `> 5` to `> 6`, keeping the `== 3` rejection. + - Change `:3693` to `params.size() >= 5`. Today it reads `nCount` only when `size() == 5`, so every 6-parameter call would silently drop it. + - Read `params[5]`, reject anything outside 0-3, and document it in the help text. `dx:src/rpc/client.cpp:165` already converts index 5. +- **GUI side:** send 6 parameters. An old daemon throws its help text for more than 5, so fall back to 5 parameters and remember that per connection. + - Alternative: a new RPC name, probed through "Method not found" the same way as `z_autoshieldstatus` (`app.h:1744-1747`). +- **Owner decision:** a variant that returns only txs with wallet spends would return 0 rows here, but it changes the `knownTxids` set. + +**D7: honour the address argument.** +- The GUI passes `shieldedAddresses.front()` (`network_refresh_service.cpp:1319`) and relies on the daemon using `"*"` (`dx:rpcwallet.cpp:3743`). Honouring the argument by default would silently shrink GUI sweeps to one address. +- It isn't faster either: with a filter set, `fBool=true` runs `zsTxSendsToJSON` for every tx (`:586`). +- Recommendation: document `"*"` as a compatibility contract. If honouring the argument is ever wanted, put it behind a flag. + +**Additive and safe for the GUI: a wallet-tip / history fingerprint.** +- Expose `pindexWalletTip`, for example in `getwalletinfo` or a new `z_getwallethistorystate(maxheight)` returning `{count, txid-hash, wallet_tip}`. +- The GUI ignores unknown JSON fields. +- This would make Part 1's invalidation exact, replacing the E-margin guess and the `txcount` tripwire. + +--- + +## 4. Rollout and compatibility + +| Step | Ships in | Works with | Notes | +|---|---|---|---| +| Part 1: watermark + anchor verify + backlog + sticky rescan + large-window path + banding + safety net | Next GUI release | Every daemon version; it uses only existing `z_listsentbyaddress` parameters, `getblockhash`, `getwalletinfo` and `getinfo` | Old GUI with a new snapshot: the key is ignored. New GUI with an old snapshot: one full sweep, which is today's behaviour. On v1.3.0, seeding the first W needs the banded sweep | +| D1, D2, D3, D5 | Daemon v1.3.1 / v1.3.2 | Invisible to the GUI | Gate each on the byte-identical oracle and the mainnet `cmp` | +| D6 + GUI probe | Daemon first, then GUI | The GUI falls back on old daemons | Makes the safety net and any full sweep cheap | +| D4 | Together with a parallel witness-root check | Invisible apart from shutdown time | Confirm the GUI stop policy first | + +--- + +## 5. Open questions + +1. How big is the backlog after the first full sweep on the owner's wallet? Unmeasured; it sizes the persisted `backlog` field. +2. What is the full-sweep time on the owner's 24-core Windows node? The 113 s was measured on Linux/WSL, 7 s inside the 120 s timeout. +3. How long does the wallet notifier lag after a catch-up on this wallet? This sets E. +4. Should the snapshot store keep more than one wallet, so that switching wallets also benefits? +5. D6 shape: a 6th parameter or a new RPC? Should a "wallet-spends-only" variant exist? +6. Does any client besides ObsidianDragon depend on `returnType 0` output? This sets D1's priority once D6 exists. +7. Are the 5 ivks distinct? Assumed. The per-address note counts sum to exactly 16,206, which fits but doesn't prove it. +8. Does the owner want the additive wallet-tip / fingerprint RPC, which makes Part 1's invalidation exact? diff --git a/src/app.h b/src/app.h index fe37fab..154e6ca 100644 --- a/src/app.h +++ b/src/app.h @@ -1761,7 +1761,10 @@ private: services::ShieldedHistoryPass history_pass_; std::uint64_t history_scan_epoch_ = 0; // bumped when the scan heights are invalidated std::uint64_t history_sent_epoch_ = 0; // bumped when the sent sweep's watermark is reset - int history_sent_scan_height_ = -1; // tip of the last successful z_listsentbyaddress; -1 = none + int history_sent_scan_height_ = -1; // W: tip of the last successful z_listsentbyaddress; -1 = none + int history_sent_anchor_height_ = -1; // A: bottom edge the last sweep's window covered (W - margin) + std::string history_sent_anchor_hash_; // H_A: getblockhash(A) at that sweep; re-verified after a load + bool history_sent_verify_pending_ = false; // a loaded watermark owes an anchor verify before its next sweep int history_full_sweep_failures_ = 0; // whole-wallet sweeps failed in a row (backoff) std::time_t history_full_sweep_retry_at_ = 0; std::unordered_set history_enrich_backlog_; // txids found, not enriched yet diff --git a/src/app_network.cpp b/src/app_network.cpp index 5f1b4ee..274579b 100644 --- a/src/app_network.cpp +++ b/src/app_network.cpp @@ -1888,6 +1888,9 @@ void App::invalidateShieldedHistoryScanProgress(bool persistCache, bool keepSent recent_probe_rotation_.reset(); if (!keepSentScan) { history_sent_scan_height_ = -1; // the next sweep covers the whole wallet again + history_sent_anchor_height_ = -1; + history_sent_anchor_hash_.clear(); + history_sent_verify_pending_ = false; history_enrich_backlog_.clear(); history_full_sweep_failures_ = 0; history_full_sweep_retry_at_ = 0; @@ -2007,11 +2010,26 @@ void App::loadTransactionHistoryCacheIfAvailable() } } confirmed_cache_block_ = loaded.tipHeight; + // Restore the sent-sweep watermark so the first sweep is windowed, not a 400 s full sweep. A verify is + // owed before the window is trusted (the next sweep checks getblockhash(A) == H_A), and the enrichment + // backlog must come back too: a pure z->z send's txid lives only in a sweep result, and would be lost if + // the window moved past an un-enriched one. + history_sent_scan_height_ = loaded.sentSweep.height; + history_sent_anchor_height_ = loaded.sentSweep.anchorHeight; + history_sent_anchor_hash_ = loaded.sentSweep.anchorHash; + history_sent_verify_pending_ = loaded.sentSweep.valid(); + history_enrich_backlog_.clear(); + history_enrich_backlog_.insert(loaded.sentSweep.backlog.begin(), loaded.sentSweep.backlog.end()); + history_full_sweep_failures_ = loaded.sentSweep.fullFailures; + history_full_sweep_retry_at_ = loaded.sentSweep.fullRetryDelaySec > 0 + ? std::time(nullptr) + loaded.sentSweep.fullRetryDelaySec : 0; + ++history_sent_epoch_; // the loaded watermark replaces any in-flight sweep's result transaction_history_cache_loaded_ = true; // What is on disk is exactly what was just loaded: the first identical refresh must not rewrite it. history_store_.sig = data::historySnapshotSignature( state_.transactions, {}, data::TransactionHistoryCache::walletIdentityHash(walletIdentity), - transaction_history_cache_.keyId(), shielded_history_scan_heights_, loaded.tipHeight, loaded.tipHash); + transaction_history_cache_.keyId(), shielded_history_scan_heights_, loaded.tipHeight, loaded.tipHash, + history_sent_scan_height_, history_sent_anchor_height_, history_sent_anchor_hash_, history_enrich_backlog_); history_store_.have = true; history_store_.lastSubmit = std::chrono::steady_clock::now(); history_store_pending_ = false; @@ -2040,7 +2058,9 @@ void App::storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode mode) state_.transactions, pendingOpids, data::TransactionHistoryCache::walletIdentityHash(walletIdentity), transaction_history_cache_.keyId(), shielded_history_scan_heights_, - state_.sync.blocks, state_.sync.best_blockhash); + state_.sync.blocks, state_.sync.best_blockhash, + history_sent_scan_height_, history_sent_anchor_height_, history_sent_anchor_hash_, + history_enrich_backlog_); if (sig.rows == 0) return; const auto now = std::chrono::steady_clock::now(); @@ -2077,6 +2097,18 @@ void App::storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode mode) if (pendingOpids.find(transaction.txid) != pendingOpids.end()) continue; job->transactions.push_back(transaction); } + // The sent-sweep watermark + enrichment backlog travel with the rows (writeSnapshot emits them only + // when self-consistent). The backoff is stored as a delay relative to now (retry_at is absolute). + job->sentSweep.height = history_sent_scan_height_; + job->sentSweep.anchorHeight = history_sent_anchor_height_; + job->sentSweep.anchorHash = history_sent_anchor_hash_; + job->sentSweep.backlog.assign(history_enrich_backlog_.begin(), history_enrich_backlog_.end()); + job->sentSweep.fullFailures = history_full_sweep_failures_; + { + const std::time_t nowSec = std::time(nullptr); + job->sentSweep.fullRetryDelaySec = history_full_sweep_retry_at_ > nowSec + ? static_cast(history_full_sweep_retry_at_ - nowSec) : 0; + } auto* cache = &transaction_history_cache_; // App member: outlives every job (see app.h) history_writer_.submit("tx-history", [cache, job]() { return cache->writeSnapshot(std::move(*job)); }); @@ -2592,6 +2624,9 @@ void App::refreshAddressData(services::ScanTicket ticket) } if (newUnscannedZ && (!previousZ.empty() || !history_listed_z_.empty())) { history_sent_scan_height_ = -1; + history_sent_anchor_height_ = -1; + history_sent_anchor_hash_.clear(); + history_sent_verify_pending_ = false; history_full_sweep_failures_ = 0; history_full_sweep_retry_at_ = 0; ++history_sent_epoch_; @@ -2777,6 +2812,9 @@ void App::startTransactionCycle(bool full, bool routineAllowed, services::ScanTi } } transactionSnapshot.sentScanFromHeight = history_sent_scan_height_; + transactionSnapshot.sentVerifyPending = history_sent_verify_pending_; + transactionSnapshot.sentVerifyAnchorHeight = history_sent_anchor_height_; + transactionSnapshot.sentVerifyAnchorHash = history_sent_anchor_hash_; // A windowed sweep mid-pass too, now and then, so an untracked send (the Console, another wallet on // the same seed) doesn't wait for a whole pass on a large wallet: at most every quarter of the pass // cadence, and only in cycles the throttle allows. @@ -2844,6 +2882,9 @@ void App::startTransactionCycle(bool full, bool routineAllowed, services::ScanTi if (sameSentEpoch) { if (result.sentSweepOk && result.sentSweepHeight > 0) { history_sent_scan_height_ = result.sentSweepHeight; + history_sent_anchor_height_ = result.sentSweepAnchorHeight; + history_sent_anchor_hash_ = result.sentSweepAnchorHash; + history_sent_verify_pending_ = false; // this sweep covered the window; nothing owed history_full_sweep_failures_ = 0; history_full_sweep_retry_at_ = 0; } else if (result.sentSweepRan && result.sentSweepFull && @@ -6562,6 +6603,9 @@ void App::restoreOriginalWallet() connection_status_ = TR("sb_restarting_daemon"); abortMaintenanceCalls(); // a long call holds the curl lock disconnect() needs if (rpc_ && rpc_->isConnected()) rpc_->disconnect(); + // Swapping the wallet + restarting with -rescan changes the history: invalidate the sent-sweep watermark + // (and persist that) before the disconnect, as rescanBlockchain() and the zap path already do. + invalidateShieldedHistoryScanProgress(true); onDisconnected("Restoring original wallet"); ui::Notifications::instance().info(TR("wallet_restore_started"), 12.0f); diff --git a/src/data/transaction_history_cache.cpp b/src/data/transaction_history_cache.cpp index 4c9e589..bc7e5c7 100644 --- a/src/data/transaction_history_cache.cpp +++ b/src/data/transaction_history_cache.cpp @@ -366,6 +366,27 @@ TransactionHistoryCache::LoadResult TransactionHistoryCache::load( } } } + // Sent-sweep watermark (optional). Drop it unless W/A are self-consistent and W is not above the + // stored tip (a replay after an unclean stop would otherwise use a stale watermark); the anchor + // hash is re-verified against the live chain on the next sweep regardless. + if (payload.contains("sent_sweep") && payload["sent_sweep"].is_object()) { + const auto& s = payload["sent_sweep"]; + SentSweepState sw; + sw.height = s.value("height", -1); + sw.anchorHeight = s.value("anchor_height", -1); + sw.anchorHash = s.value("anchor_hash", std::string()); + sw.fullFailures = s.value("full_failures", 0); + sw.fullRetryDelaySec = static_cast(s.value("full_retry_delay_s", std::int64_t{0})); + sw.daemonId = s.value("daemon_id", std::string()); + if (s.contains("backlog") && s["backlog"].is_array()) { + for (const auto& txid : s["backlog"]) + if (txid.is_string() && !txid.get().empty()) + sw.backlog.push_back(txid.get()); + } + const bool consistent = sw.height >= 0 && sw.anchorHeight > 0 && + sw.anchorHeight <= sw.height && sw.height <= tipHeight; + if (consistent) result.sentSweep = std::move(sw); + } result.tipHeight = tipHeight; result.tipHash = tipHash; result.updatedAt = updatedAt; @@ -383,7 +404,8 @@ bool TransactionHistoryCache::replace(const std::string& walletIdentity, const std::string& tipHash, const std::vector& transactions, std::time_t updatedAt, - const std::unordered_map& shieldedScanHeights) + const std::unordered_map& shieldedScanHeights, + const SentSweepState& sentSweep) { // The synchronous form (tests, tools): the same write the background writer does. SnapshotJob job; @@ -394,6 +416,7 @@ bool TransactionHistoryCache::replace(const std::string& walletIdentity, job.transactions = transactions; job.updatedAt = updatedAt; job.shieldedScanHeights = shieldedScanHeights; + job.sentSweep = sentSweep; return writeSnapshot(std::move(job)); } @@ -426,6 +449,23 @@ bool TransactionHistoryCache::writeSnapshot(SnapshotJob&& job) } } + // Sent-sweep watermark, only when it is self-consistent (0 < A <= W <= tip). An unknown key means + // W = -1 on load, which is the pre-watermark behaviour, so older builds read a new payload safely. + const SentSweepState& sw = job.sentSweep; + if (sw.height >= 0 && sw.anchorHeight > 0 && sw.anchorHeight <= sw.height && sw.height <= job.tipHeight) { + json s; + s["height"] = sw.height; + s["anchor_height"] = sw.anchorHeight; + s["anchor_hash"] = sw.anchorHash; + s["backlog"] = json::array(); + for (const auto& txid : sw.backlog) + if (!txid.empty()) s["backlog"].push_back(txid); + s["full_failures"] = sw.fullFailures; + s["full_retry_delay_s"] = static_cast(sw.fullRetryDelaySec); + s["daemon_id"] = sw.daemonId; + payload["sent_sweep"] = std::move(s); + } + std::string plainText = payload.dump(); std::vector nonce; std::vector cipherText; diff --git a/src/data/transaction_history_cache.h b/src/data/transaction_history_cache.h index 29719e0..2597842 100644 --- a/src/data/transaction_history_cache.h +++ b/src/data/transaction_history_cache.h @@ -15,6 +15,24 @@ struct sqlite3; namespace dragonx { namespace data { +// Persisted z_listsentbyaddress watermark, so a relaunch runs a windowed sweep instead of a full one. +// height (W) = tip of the last successful sweep; anchorHeight (A) = W - margin, the bottom edge the window +// covered; anchorHash (H_A) = getblockhash(A) at that time, verified on the next load. backlog = txids found +// but not yet enriched (must persist WITH W, or a pure z->z send whose txid lives only in a sweep result is +// lost once the window moves past it). The backoff fields keep a relaunch from immediately re-running a full +// sweep that keeps failing; daemonId resets the backoff on a version change. (Namespace-scope, not nested, so +// it is a complete type for the default argument of replace().) +struct SentSweepState { + int height = -1; + int anchorHeight = -1; + std::string anchorHash; + std::vector backlog; + int fullFailures = 0; + long fullRetryDelaySec = 0; + std::string daemonId; + bool valid() const { return height >= 0; } +}; + class TransactionHistoryCache { public: struct LoadResult { @@ -25,6 +43,7 @@ public: std::time_t updatedAt = 0; std::vector transactions; std::unordered_map shieldedScanHeights; + SentSweepState sentSweep; }; // Everything one snapshot write needs, BY VALUE, so it can run on the history writer thread after the UI @@ -38,6 +57,7 @@ public: std::vector transactions; std::time_t updatedAt = 0; std::unordered_map shieldedScanHeights; + SentSweepState sentSweep; SnapshotJob() = default; SnapshotJob(const SnapshotJob&) = delete; @@ -81,7 +101,8 @@ public: const std::string& tipHash, const std::vector& transactions, std::time_t updatedAt, - const std::unordered_map& shieldedScanHeights = {}); + const std::unordered_map& shieldedScanHeights = {}, + const SentSweepState& sentSweep = SentSweepState{}); // Thread-safe snapshot write (the history writer thread; replace() wraps it): JSON + AEAD with no lock // held, then INSERT OR REPLACE + prune under the DB lock. Always wipes job.key. False for a keyless job. bool writeSnapshot(SnapshotJob&& job); diff --git a/src/data/transaction_history_store_policy.cpp b/src/data/transaction_history_store_policy.cpp index 72fb396..a3cefe7 100644 --- a/src/data/transaction_history_store_policy.cpp +++ b/src/data/transaction_history_store_policy.cpp @@ -65,7 +65,11 @@ HistorySnapshotSig historySnapshotSignature(const std::vector& std::uint64_t keyId, const std::unordered_map& shieldedScanHeights, int tipHeight, - const std::string& tipHash) + const std::string& tipHash, + int sentHeight, + int sentAnchorHeight, + const std::string& sentAnchorHash, + const std::unordered_set& sentBacklog) { HistorySnapshotSig sig; std::uint64_t rowSum = 0; // sums: commutative, so row order does not matter; duplicates still count @@ -89,11 +93,19 @@ HistorySnapshotSig historySnapshotSignature(const std::vector& exactSum += finalize(e); ++sig.rows; } + std::uint64_t backlogSum = 0; // order-independent: a set, so sum is commutative + for (const auto& txid : sentBacklog) { + if (txid.empty()) continue; + std::uint64_t h = kOffset; + mixStr(h, txid); + backlogSum += finalize(h); + } std::uint64_t c = kOffset; mixU64(c, rowSum); mixU64(c, sig.rows); mixStr(c, walletHash); mixU64(c, keyId); + mixU64(c, backlogSum); sig.content = finalize(c); std::uint64_t heightSum = 0; @@ -109,6 +121,9 @@ HistorySnapshotSig historySnapshotSignature(const std::vector& mixU64(s, exactSum); mixU64(s, static_cast(static_cast(tipHeight))); mixStr(s, tipHash); + mixU64(s, static_cast(static_cast(sentHeight))); + mixU64(s, static_cast(static_cast(sentAnchorHeight))); + mixStr(s, sentAnchorHash); sig.slow = finalize(s); return sig; } diff --git a/src/data/transaction_history_store_policy.h b/src/data/transaction_history_store_policy.h index a23f69b..f5b1dad 100644 --- a/src/data/transaction_history_store_policy.h +++ b/src/data/transaction_history_store_policy.h @@ -38,7 +38,15 @@ HistorySnapshotSig historySnapshotSignature(const std::vector& std::uint64_t keyId, const std::unordered_map& shieldedScanHeights, int tipHeight, - const std::string& tipHash); + const std::string& tipHash, + // Sent-sweep watermark: W/A/H_A move per sweep (folded into `slow`, + // written on the slow cadence and always on disconnect/lock/exit), + // the enrichment backlog into `content` (written within ~20 s), so an + // invalidation that only changes the watermark still reaches disk. + int sentHeight = -1, + int sentAnchorHeight = -1, + const std::string& sentAnchorHash = std::string(), + const std::unordered_set& sentBacklog = {}); enum class HistoryStoreMode { Coalesce, // routine refresh: content changes at most every kHistoryMinStoreIntervalSec, slow drift every diff --git a/src/services/history_scan_policy.cpp b/src/services/history_scan_policy.cpp index 249f04d..87f3470 100644 --- a/src/services/history_scan_policy.cpp +++ b/src/services/history_scan_policy.cpp @@ -37,18 +37,34 @@ bool sentSweepIsFull(int tip, int fromHeight) return tip <= 0 || fromHeight < 0 || fromHeight > tip; } -nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight) +nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight, int margin) { nlohmann::json params = nlohmann::json::array(); params.push_back(anyZAddress); params.push_back(0); if (sentSweepIsFull(tip, fromHeight)) return params; - const int maxDepth = (tip - fromHeight) + kSentSweepReorgMarginBlocks; + const int maxDepth = (tip - fromHeight) + std::max(0, margin); params.push_back(2); // filter type 2: depth params.push_back(maxDepth); // at most this deep (mempool txs, depth 0, always pass) return params; } +int sentSweepMargin(bool firstAfterLoad) +{ + return firstAfterLoad ? (kSentSweepReorgMarginBlocks + kSentSweepFirstLoadExtraBlocks) + : kSentSweepReorgMarginBlocks; +} + +int sentSweepAnchor(int fromHeight, bool firstAfterLoad) +{ + return std::max(1, fromHeight - sentSweepMargin(firstAfterLoad)); +} + +bool sentSweepNeedsLongCall(int tip, int fromHeight) +{ + return tip > 0 && fromHeight >= 0 && (tip - fromHeight) > kSentSweepLongCallBlocks; +} + long fullSentSweepRetryDelaySec(int consecutiveFailures) { if (consecutiveFailures <= 0) return 0; diff --git a/src/services/history_scan_policy.h b/src/services/history_scan_policy.h index a665739..39c1569 100644 --- a/src/services/history_scan_policy.h +++ b/src/services/history_scan_policy.h @@ -28,6 +28,9 @@ constexpr double kHistoryScanDuty = 0.10; // the same 10% as the othe constexpr int kHistoryMinStaleBlocks = 8; // re-scan an address at most every 8 blocks... constexpr int kHistoryMaxStaleBlocks = 500; // ...and at least every 500 (a safety net) constexpr int kSentSweepReorgMarginBlocks = 100; // > MAX_REORG_LENGTH (99), plus tip drift +constexpr int kSentSweepFirstLoadExtraBlocks = 1000; // E: extra margin on the first sweep after a load, + // to absorb wallet-notifier lag at the last save +constexpr int kSentSweepLongCallBlocks = 2400; // tip - W beyond this: use callLong + count failures constexpr long kFullSentSweepTimeoutSec = 120; // a whole-wallet sweep must be able to finish constexpr int kTransparentPageMin = 1000; constexpr int kTransparentPageMax = 5000; @@ -50,8 +53,17 @@ int transparentPageSize(std::size_t previousRowCount); // tip after a reorg), else only txs at most (tip - fromHeight) + kSentSweepReorgMarginBlocks deep. bool sentSweepIsFull(int tip, int fromHeight); // [z, 0] for the full sweep, [z, 0, 2, maxDepth] for the window: JSON integers, and never 3 params (the -// daemon rejects that count). -nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight); +// daemon rejects that count). `margin` widens the window (the first sweep after a load uses a bigger one). +nlohmann::json sentSweepParams(const std::string& anyZAddress, int tip, int fromHeight, + int margin = kSentSweepReorgMarginBlocks); +// Window margin: steady-state kSentSweepReorgMarginBlocks, bigger on the first sweep after a load. +int sentSweepMargin(bool firstAfterLoad); +// A: the bottom edge the window covers (W - margin), clamped to >= 1. The anchor whose block hash is stored +// (H_A) and re-verified on the next load. +int sentSweepAnchor(int fromHeight, bool firstAfterLoad); +// A watermark saved long ago makes a huge "windowed" call: past this gap, use callLong and count failures +// toward the backoff, so it doesn't time out on every block while the daemon keeps working. +bool sentSweepNeedsLongCall(int tip, int fromHeight); // Backoff after full sweeps that failed in a way that cost the daemon the work: 0, 10 min, 1 h, then 6 h. long fullSentSweepRetryDelaySec(int consecutiveFailures); // Whether a failed full sweep counts toward that backoff: only when the daemon did the work (reply timeout, diff --git a/src/services/network_refresh_service.cpp b/src/services/network_refresh_service.cpp index 6c58290..10ff9fc 100644 --- a/src/services/network_refresh_service.cpp +++ b/src/services/network_refresh_service.cpp @@ -1314,32 +1314,61 @@ NetworkRefreshService::TransactionRefreshResult NetworkRefreshService::collectTr const bool sweepIsFull = sentSweepIsFull(currentBlockHeight, snapshot.sentScanFromHeight); if (!snapshot.shieldedAddresses.empty() && !snapshot.skipSentSweep && (result.shieldedScanComplete || (snapshot.allowMidPassSentSweep && !sweepIsFull))) { - result.sentSweepRan = true; - result.sentSweepFull = sweepIsFull; - const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight, - snapshot.sentScanFromHeight); - const auto sweepStart = std::chrono::steady_clock::now(); - try { - json sent = result.sentSweepFull - ? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec) - : rpc.call("z_listsentbyaddress", params); - if (sent.is_array()) { - for (const auto& sentTx : sent) { - auto txid = readOptional(sentTx, "txid"); - if (!txid || txid->empty()) continue; - appendSentTransparentOutputs(result.transactions, *txid, sentTx); - knownTxids.insert(*txid); // shielded (z->z) legs enriched via z_viewtransaction - } - result.sentSweepOk = true; - result.sentSweepHeight = currentBlockHeight; - } - } catch (const std::exception& e) { - transactionRpcError = true; - result.sentSweepErrKind = rpc::rpcFailureKindOf(e); - result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout; - DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what()); + int effectiveFromHeight = snapshot.sentScanFromHeight; + bool effectiveFull = sweepIsFull; + // Anchor verify: a watermark restored from disk is trusted only if the block at A still matches the + // stored H_A (no reorg deeper than the margin, same chain/datadir). A mismatch, or an inability to + // check, drops the watermark and forces a full sweep. Checked only for a windowed sweep. + if (snapshot.sentVerifyPending && !sweepIsFull && snapshot.sentVerifyAnchorHeight > 0) { + bool anchorOk = false; + try { + json h = rpc.call("getblockhash", json::array({snapshot.sentVerifyAnchorHeight})); + if (h.is_string()) anchorOk = (h.get() == snapshot.sentVerifyAnchorHash); + } catch (...) { anchorOk = false; } + if (!anchorOk) { effectiveFromHeight = -1; effectiveFull = true; } + } + // A verify that turned a windowed mid-pass sweep into a full one must wait for a completed pass, + // rather than run a whole-wallet sweep in the middle of one. + if (!(effectiveFull && !result.shieldedScanComplete)) { + result.sentSweepRan = true; + result.sentSweepFull = effectiveFull; + // Bigger margin on the first windowed sweep after a load (absorbs wallet-notifier lag at save). + const int margin = sentSweepMargin(snapshot.sentVerifyPending && !effectiveFull); + const json params = sentSweepParams(snapshot.shieldedAddresses.front(), currentBlockHeight, + effectiveFromHeight, margin); + // A watermark saved long ago makes a huge "windowed" call; give it the long timeout too. + const bool longCall = effectiveFull || sentSweepNeedsLongCall(currentBlockHeight, effectiveFromHeight); + const auto sweepStart = std::chrono::steady_clock::now(); + try { + json sent = longCall + ? rpc.callLong("z_listsentbyaddress", params, kFullSentSweepTimeoutSec) + : rpc.call("z_listsentbyaddress", params); + if (sent.is_array()) { + for (const auto& sentTx : sent) { + auto txid = readOptional(sentTx, "txid"); + if (!txid || txid->empty()) continue; + appendSentTransparentOutputs(result.transactions, *txid, sentTx); + knownTxids.insert(*txid); // shielded (z->z) legs enriched via z_viewtransaction + } + result.sentSweepOk = true; + result.sentSweepHeight = currentBlockHeight; + // Anchor for the NEXT load's verify: A = W - steady margin, H_A = getblockhash(A). An + // empty H_A (fetch failed) just means the next load can't verify and does a full sweep. + const int newAnchor = sentSweepAnchor(currentBlockHeight, false); + result.sentSweepAnchorHeight = newAnchor; + try { + json ha = rpc.call("getblockhash", json::array({newAnchor})); + if (ha.is_string()) result.sentSweepAnchorHash = ha.get(); + } catch (...) {} + } + } catch (const std::exception& e) { + transactionRpcError = true; + result.sentSweepErrKind = rpc::rpcFailureKindOf(e); + result.replyTimedOut |= result.sentSweepErrKind == rpc::RpcFailureKind::ReplyTimeout; + DEBUG_LOGF("z_listsentbyaddress error: %s\n", e.what()); + } + result.sentSweepMs = msSince(sweepStart); } - result.sentSweepMs = msSince(sweepStart); } for (const auto& txid : snapshot.sendTxids) { diff --git a/src/services/network_refresh_service.h b/src/services/network_refresh_service.h index 7f61214..266d424 100644 --- a/src/services/network_refresh_service.h +++ b/src/services/network_refresh_service.h @@ -254,7 +254,13 @@ public: std::unordered_map priorityScanFloors; bool skipShieldedScans = false; // a Light cycle (or a probe-less recent poll): no receive scans bool skipSentSweep = false; - int sentScanFromHeight = -1; // the sweep's watermark; -1 = sweep the whole wallet + int sentScanFromHeight = -1; // the sweep's watermark (W); -1 = sweep the whole wallet + // A loaded watermark owes an anchor verify before its first windowed sweep: getblockhash(A) must + // still equal H_A (no reorg deeper than the margin). A mismatch forces a full sweep. While pending, + // the window also uses the bigger first-after-load margin. + bool sentVerifyPending = false; + int sentVerifyAnchorHeight = -1; // A + std::string sentVerifyAnchorHash; // H_A std::size_t transparentRowHint = 0; // listtransactions rows last time: sizes the first page // Same tip and nothing changed since the rows on screen were fetched: reuse their transparent rows // instead of paging listtransactions again. @@ -291,7 +297,9 @@ public: bool sentSweepRan = false; bool sentSweepOk = false; bool sentSweepFull = false; - int sentSweepHeight = -1; // the tip a successful sweep covered + int sentSweepHeight = -1; // W: the tip a successful sweep covered + int sentSweepAnchorHeight = -1; // A = W - margin, the bottom edge the window covered + std::string sentSweepAnchorHash; // H_A = getblockhash(A), fetched after a successful sweep double sentSweepMs = 0.0; // time in the sweep (part of scanMs) rpc::RpcFailureKind sentSweepErrKind = rpc::RpcFailureKind::None; // Txids found but not enriched (over the per-cycle cap, or the view failed transiently): the caller diff --git a/tests/test_phase4.cpp b/tests/test_phase4.cpp index 8054e2a..efad5bb 100644 --- a/tests/test_phase4.cpp +++ b/tests/test_phase4.cpp @@ -2157,13 +2157,13 @@ void testNetworkRefreshRpcCollectors() auto transactionResult = Refresh::collectTransactionRefreshResult(transactionRpc, snapshot, 321, 4); EXPECT_TRUE(transactionRpc.methodNames() == std::vector({ "listtransactions", "z_listreceivedbyaddress", "z_listreceivedbyaddress", - "z_listsentbyaddress", "z_viewtransaction", "gettransaction" + "z_listsentbyaddress", "getblockhash", "z_viewtransaction", "gettransaction" })); EXPECT_EQ(transactionRpc.calls[1].params, json::array({"zs-one", 0})); EXPECT_EQ(transactionRpc.calls[2].params, json::array({"zs-two", 0})); EXPECT_EQ(transactionRpc.calls[3].params, json::array({"zs-one", 0})); - EXPECT_EQ(transactionRpc.calls[4].params, json::array({"pending-send"})); EXPECT_EQ(transactionRpc.calls[5].params, json::array({"pending-send"})); + EXPECT_EQ(transactionRpc.calls[6].params, json::array({"pending-send"})); EXPECT_EQ(transactionResult.blockHeight, 321); EXPECT_EQ(transactionResult.newViewTxEntries.size(), static_cast(1)); EXPECT_EQ(transactionResult.newViewTxEntries.count("pending-send"), static_cast(1)); @@ -2201,7 +2201,7 @@ void testNetworkRefreshRpcCollectors() auto zzResult = Refresh::collectTransactionRefreshResult(zzRpc, zzSnapshot, 400, 4); EXPECT_TRUE(zzRpc.methodNames() == std::vector({ "listtransactions", "z_listreceivedbyaddress", "z_listsentbyaddress", - "z_viewtransaction", "gettransaction" + "getblockhash", "z_viewtransaction", "gettransaction" })); EXPECT_EQ(zzResult.transactions.size(), static_cast(1)); EXPECT_EQ(zzResult.transactions[0].txid, std::string("zz-send")); @@ -2490,7 +2490,7 @@ void testNetworkRefreshRpcCollectors() auto tolerantResult = Refresh::collectTransactionRefreshResult(tolerantRpc, tolerantSnapshot, 102, 4); EXPECT_EQ(tolerantResult.shieldedAddressesScanned, static_cast(0)); // both skipped EXPECT_TRUE(tolerantResult.shieldedScanComplete); - EXPECT_TRUE(tolerantRpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress"})); + EXPECT_TRUE(tolerantRpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress", "getblockhash"})); } Refresh::TransactionRefreshSnapshot recentSnapshot; @@ -2580,7 +2580,7 @@ void testNetworkRefreshRpcCollectors() auto finalShielded = Refresh::collectTransactionRefreshResult( finalShieldedRpc, partialShieldedSnapshot, 400, 0); EXPECT_TRUE(finalShieldedRpc.methodNames() == std::vector({ - "listtransactions", "z_listreceivedbyaddress", "z_listsentbyaddress" + "listtransactions", "z_listreceivedbyaddress", "z_listsentbyaddress", "getblockhash" })); EXPECT_EQ(finalShieldedRpc.calls[1].params, json::array({"zs-two", 0})); EXPECT_TRUE(finalShielded.shieldedScanComplete); @@ -2600,7 +2600,7 @@ void testNetworkRefreshRpcCollectors() cachedShieldedRpc.addResponse("z_listsentbyaddress", json::array()); auto cachedShielded = Refresh::collectTransactionRefreshResult( cachedShieldedRpc, cachedShieldedSnapshot, 500, 0); - EXPECT_TRUE(cachedShieldedRpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress"})); + EXPECT_TRUE(cachedShieldedRpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress", "getblockhash"})); EXPECT_TRUE(cachedShielded.shieldedScanComplete); EXPECT_EQ(cachedShielded.shieldedAddressesScanned, static_cast(0)); EXPECT_EQ(cachedShielded.transactions.size(), static_cast(1)); @@ -9199,6 +9199,79 @@ void testTransactionHistoryCache() removeTree(dir); } +// Sent-sweep watermark persistence (SENT-HISTORY-SWEEP Part 1): the watermark + enrichment backlog +// round-trip through the encrypted snapshot, and a missing / inconsistent one reads as "no watermark". +void testSentSweepPersistence() +{ + using dragonx::TransactionInfo; + using dragonx::data::TransactionHistoryCache; + using dragonx::data::SentSweepState; + + fs::path dir = makeTempDir(); + fs::path dbPath = dir / "th.sqlite"; + std::string id = "mainnet|R|zs"; + std::string pass = "pw"; + std::vector txs{ TransactionInfo{"tx1", "send", -1.0, 1700000000, 5, "zdst", "zsrc", "m"} }; + + { // valid watermark round-trips, backlog + backoff included + TransactionHistoryCache cache(dbPath.string()); + EXPECT_TRUE(cache.ensureOpen()); + EXPECT_TRUE(cache.unlockWithPassphrase(id, pass)); + SentSweepState sw; + sw.height = 500; sw.anchorHeight = 380; sw.anchorHash = "ha380"; + sw.backlog = {"bt1", "bt2"}; sw.fullFailures = 2; sw.fullRetryDelaySec = 3600; sw.daemonId = "v1.3.0|t"; + EXPECT_TRUE(cache.replace(id, 500, "tip", txs, 1700000000, {}, sw)); + auto loaded = cache.load(id, 500, "tip"); + EXPECT_TRUE(loaded.loaded); + EXPECT_TRUE(loaded.sentSweep.valid()); + EXPECT_EQ(loaded.sentSweep.height, 500); + EXPECT_EQ(loaded.sentSweep.anchorHeight, 380); + EXPECT_EQ(loaded.sentSweep.anchorHash, std::string("ha380")); + EXPECT_EQ(loaded.sentSweep.backlog.size(), static_cast(2)); + EXPECT_EQ(loaded.sentSweep.fullFailures, 2); + EXPECT_EQ(loaded.sentSweep.fullRetryDelaySec, 3600L); + EXPECT_EQ(loaded.sentSweep.daemonId, std::string("v1.3.0|t")); + } + { // no watermark passed -> reads invalid (W = -1), i.e. today's full-sweep behaviour + TransactionHistoryCache cache(dbPath.string()); + EXPECT_TRUE(cache.unlockWithPassphrase(id, pass)); + EXPECT_TRUE(cache.replace(id, 500, "tip", txs, 1700000001)); + auto loaded = cache.load(id, 500, "tip"); + EXPECT_TRUE(loaded.loaded); + EXPECT_FALSE(loaded.sentSweep.valid()); + EXPECT_EQ(loaded.sentSweep.height, -1); + } + { // inconsistent watermarks are refused (W above tip, or A > W) + TransactionHistoryCache cache(dbPath.string()); + EXPECT_TRUE(cache.unlockWithPassphrase(id, pass)); + SentSweepState aboveTip; aboveTip.height = 9999; aboveTip.anchorHeight = 100; aboveTip.anchorHash = "h"; + EXPECT_TRUE(cache.replace(id, 500, "tip", txs, 1700000002, {}, aboveTip)); + EXPECT_FALSE(cache.load(id, 500, "tip").sentSweep.valid()); + SentSweepState anchorAbove; anchorAbove.height = 400; anchorAbove.anchorHeight = 450; anchorAbove.anchorHash = "h"; + EXPECT_TRUE(cache.replace(id, 500, "tip", txs, 1700000003, {}, anchorAbove)); + EXPECT_FALSE(cache.load(id, 500, "tip").sentSweep.valid()); + } + removeTree(dir); +} + +// Sent-sweep pure policy helpers (anchor, margin, long-call threshold, windowed params). +void testSentSweepPolicyHelpers() +{ + using namespace dragonx::services; + EXPECT_EQ(sentSweepMargin(false), kSentSweepReorgMarginBlocks); + EXPECT_EQ(sentSweepMargin(true), kSentSweepReorgMarginBlocks + kSentSweepFirstLoadExtraBlocks); + EXPECT_EQ(sentSweepAnchor(5000, false), 5000 - kSentSweepReorgMarginBlocks); + EXPECT_EQ(sentSweepAnchor(50, false), 1); // clamped: W - margin < 1 + EXPECT_EQ(sentSweepAnchor(5000, true), 5000 - (kSentSweepReorgMarginBlocks + kSentSweepFirstLoadExtraBlocks)); + EXPECT_FALSE(sentSweepNeedsLongCall(1000, 999)); // tiny window + EXPECT_FALSE(sentSweepNeedsLongCall(1000, -1)); // no watermark (full-sweep path) + EXPECT_TRUE(sentSweepNeedsLongCall(10000, 10000 - kSentSweepLongCallBlocks - 1)); + auto windowed = sentSweepParams("z1", 2000, 1500, sentSweepMargin(true)); + EXPECT_EQ(windowed.size(), static_cast(4)); + EXPECT_EQ(windowed[3].get(), (2000 - 1500) + sentSweepMargin(true)); + EXPECT_EQ(sentSweepParams("z1", 2000, -1).size(), static_cast(2)); // full sweep +} + // GC: generating a new address changes the wallet-identity hash; the prior hash's snapshot // + salt must not accumulate forever. A save under a new identity prunes the old rows. void testTransactionHistoryCachePrunesOldWallets() @@ -16174,7 +16247,7 @@ void testHistoryCollectorSweepAndEnrichment() rpc.addResponse("listtransactions", json::array()); rpc.addResponse("z_listsentbyaddress", json::array()); auto result = Refresh::collectTransactionRefreshResult(rpc, snap, 110, 4); - EXPECT_TRUE(rpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress"})); + EXPECT_TRUE(rpc.methodNames() == std::vector({"listtransactions", "z_listsentbyaddress", "getblockhash"})); EXPECT_EQ(rpc.calls[1].params.dump(), std::string("[\"zs-one\",0,2,110]")); EXPECT_EQ(rpc.longCalls, 0); EXPECT_TRUE(result.sentSweepRan && result.sentSweepOk && !result.sentSweepFull); @@ -16191,6 +16264,63 @@ void testHistoryCollectorSweepAndEnrichment() EXPECT_EQ(rpc.lastLongTimeout, 120L); EXPECT_TRUE(result.sentSweepFull && result.sentSweepOk); } + { // Restored watermark, anchor verify MATCHES: a windowed sweep (bigger first-after-load margin), and + // the next anchor (A = W - steady margin, H_A) is fetched for the following load. + auto snap = freshSnapshot(); + snap.sentScanFromHeight = 100; + snap.sentVerifyPending = true; + snap.sentVerifyAnchorHeight = 50; + snap.sentVerifyAnchorHash = "anchor-hash"; + MockRefreshRpc rpc; + rpc.addResponse("listtransactions", json::array()); + rpc.addResponse("getblockhash", "anchor-hash"); // verify: matches + rpc.addResponse("z_listsentbyaddress", json::array()); + rpc.addResponse("getblockhash", "new-anchor-hash"); // post-success anchor fetch + auto result = Refresh::collectTransactionRefreshResult(rpc, snap, 110, 4); + EXPECT_TRUE(result.sentSweepRan && result.sentSweepOk && !result.sentSweepFull); + const int wantDepth = (110 - 100) + dragonx::services::sentSweepMargin(true); + bool sawWindowed = false; + for (const auto& c : rpc.calls) + if (c.method == "z_listsentbyaddress") + sawWindowed = (c.params.dump() == std::string("[\"zs-one\",0,2,") + std::to_string(wantDepth) + "]"); + EXPECT_TRUE(sawWindowed); + EXPECT_EQ(result.sentSweepAnchorHeight, dragonx::services::sentSweepAnchor(110, false)); + EXPECT_EQ(result.sentSweepAnchorHash, std::string("new-anchor-hash")); + } + { // Anchor verify MISMATCH (reorg deeper than the margin / different chain): drop the watermark and + // sweep the whole wallet instead of a window that would miss the moved blocks. + auto snap = freshSnapshot(); + snap.sentScanFromHeight = 100; + snap.sentVerifyPending = true; + snap.sentVerifyAnchorHeight = 50; + snap.sentVerifyAnchorHash = "old-hash"; + MockRefreshRpc rpc; + rpc.addResponse("listtransactions", json::array()); + rpc.addResponse("getblockhash", "reorged-hash"); // verify: mismatch + rpc.addResponse("z_listsentbyaddress", json::array()); + rpc.addResponse("getblockhash", "anchor"); + auto result = Refresh::collectTransactionRefreshResult(rpc, snap, 110, 4); + EXPECT_TRUE(result.sentSweepRan && result.sentSweepOk && result.sentSweepFull); + bool sawFull = false; + for (const auto& c : rpc.calls) + if (c.method == "z_listsentbyaddress") sawFull = (c.params.dump() == std::string("[\"zs-one\",0]")); + EXPECT_TRUE(sawFull); + } + { // A watermark saved long ago (tip - W > 2400): still a windowed call, but given the long timeout so + // it doesn't time out on every block. + Refresh::TransactionRefreshSnapshot snap; + snap.shieldedAddresses = {"zs-one"}; + snap.shieldedScanHeights = {{"zs-one", 3000}}; // scan complete at the tip + snap.sentScanFromHeight = 400; // tip - W = 2600 > 2400 + MockRefreshRpc rpc; + rpc.addResponse("listtransactions", json::array()); + rpc.addResponse("z_listsentbyaddress", json::array()); + rpc.addResponse("getblockhash", "anchor"); + auto result = Refresh::collectTransactionRefreshResult(rpc, snap, 3000, 4); + EXPECT_TRUE(result.sentSweepRan && result.sentSweepOk && !result.sentSweepFull); + EXPECT_TRUE(rpc.longCalls >= 1); // the big windowed call uses callLong + EXPECT_EQ(rpc.lastLongTimeout, 120L); + } { // A failed sweep reports why (the caller's backoff), and keeps every previous row. auto snap = freshSnapshot(); snap.previousTransactions.push_back(dragonx::TransactionInfo{"r-old", "receive", 1.0, 50, 9, "R-one", "", ""}); @@ -16371,7 +16501,8 @@ void testHistoryCollectorSweepAndEnrichment() auto result = Refresh::collectTransactionRefreshResult(rpc, midPass, 110, 4); EXPECT_FALSE(result.shieldedScanComplete); EXPECT_TRUE(result.sentSweepRan && !result.sentSweepFull); - EXPECT_EQ(rpc.calls.back().params.dump(), std::string("[\"zs-a\",0,2,110]")); + // The sweep is the 2nd-to-last call now (a getblockhash for the anchor follows a successful sweep). + EXPECT_EQ(rpc.calls[rpc.calls.size() - 2].params.dump(), std::string("[\"zs-a\",0,2,110]")); midPass.allowMidPassSentSweep = false; MockRefreshRpc offRpc; offRpc.addResponse("listtransactions", json::array()); @@ -19072,6 +19203,8 @@ int main() testBalanceAddressListModel(); testExplorerBlockCache(); testTransactionHistoryCache(); + testSentSweepPersistence(); + testSentSweepPolicyHelpers(); testTransactionHistoryCachePrunesOldWallets(); testHistorySnapshotSignature(); testHistoryStorePolicy();