audit: fix ADDR send amplification and the IVK-only witness deref (UNTESTED)

Two more findings from the general audit. Compile-verified only, like the rest of
this branch.

main.cpp SendMessages / ADDR: the per-address loop called
    pto->PushAddrMessage(msgMaker.Make(flags, msg_type, pto->vAddrToSend))
i.e. it pushed the ENTIRE vector once per accepted address. One 24-byte getaddr
therefore produced N messages of N addresses each instead of one message of N --
on a live addrman (GetAddr returning ~494) that is ~494 x ~14.8 KB = ~7.3 MB of
upstream for a 24-byte request, and up to ~30 MB with a full addrman. All of it
serialized, with a double-SHA256 checksum per message, while cs_main is held, so
each burst also stalls block validation and RPC.

The locally built vAddr was accumulated and then discarded, and the
`vAddr.resize(MAX_ADDR_TO_SEND)` sat exactly where upstream has `vAddr.clear()` --
it can never fire, because vAddr.size() already equals MAX_ADDR_TO_SEND there.
This is a local regression, not inherited: 512da314a rewrote the correct upstream
form into this one.

Restores the upstream idiom (accumulate, flush in MAX_ADDR_TO_SEND batches, send
the remainder after the loop) and hoists the msg_type/make_flags selection out of
the loop, since neither varies per address.

wallet.cpp VerifyAndSetInitialWitness: five sites dereferenced
`*item.second.nullifier` on a boost::optional that can legitimately be unset. A
Sapling note discovered through an imported INCOMING viewing key has no nullifier
-- computing one requires the full viewing key, and z_importviewingkey calls only
AddSaplingIncomingViewingKey. Dereferencing it aborts the daemon with a boost
assertion rather than an RPC error, at the end of the import's own rescan; and
because the IVK-only note data is already persisted in the wallet transaction and
BuildWitnessCache is re-driven from ChainTip on every connected block, the node
then fails the same way on every restart.

The codebase already had the right pattern in the two neighbouring functions --
DecrementNoteWitnesses guards with `if (nd->nullifier && ...)` and BuildWitnessCache
with `if (!nd->nullifier) continue;` -- so only VerifyAndSetInitialWitness was
missing it. Those guards also establish the intended semantics: a note whose spend
depth cannot be computed is treated as UNSPENT, keeping its witness rather than
pruning a note we cannot prove spent. Adds a boost::optional overload of
SaplingWitnessMinimumHeight doing exactly that, and routes all five sites through it.

NOT fixed here, deliberately, because none can be done responsibly without tests:
  - the inline TLS handshake on ThreadSocketHandler (architectural: one slow
    unauthenticated connection can freeze all P2P I/O for up to 60s);
  - the getblocktemplate function-static leaking a CBlockTemplate per concurrent
    caller, assigned with cs_main released;
  - z_sendmany's pre-flight size estimate omitting Sapling spends, so a wallet with
    ~499+ notes builds an oversize, unbroadcastable transaction after minutes of
    proving;
  - z_shieldcoinbase's opt-in donation paying a hardcoded upstream-Hush z-address
    that no DragonX party can spend (a policy decision, not a code fix).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo

doc/man/dragonx-cli.1

@@ -3,7 +3,7 @@
 .SH NAME
 dragonx-cli \- manual page for dragonx-cli v1.3.0
 .SH DESCRIPTION
-DragonX RPC client version v1.3.0
+DragonX RPC client version v1.3.0\-af7d9e230
 .PP
 In order to ensure you are adequately protecting your privacy when using
 DragonX, please see <https://dragonx.is/security/>.

doc/man/dragonx-tx.1

@@ -3,7 +3,7 @@
 .SH NAME
 dragonx-tx \- manual page for dragonx-tx v1.3.0
 .SH DESCRIPTION
-hush\-tx utility version v1.3.0
+hush\-tx utility version v1.3.0\-af7d9e230
 .SS "Usage:"
 .TP
 hush\-tx [options] <hex\-tx> [commands]

doc/man/dragonxd.1

@@ -3,7 +3,7 @@
 .SH NAME
 dragonxd \- manual page for dragonxd v1.3.0
 .SH DESCRIPTION
-DragonX Daemon version v1.3.0
+DragonX Daemon version v1.3.0\-af7d9e230
 .PP
 In order to ensure you are adequately protecting your privacy when using
 DragonX, please see <https://dragonx.is/security/>.

qa/fd-guard/.gitignore

@@ -1,4 +0,0 @@
-shim/libfdguard.so
-shim/fdguard_test
-*.o
-shim/t/

qa/fd-guard/README.md

@@ -1,55 +0,0 @@
-# fd-guard: catch stray close() of files a thread doesn't own
-
-This is the harness that found the wolfSSL `CloseSocket` macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into `blocks/*.dat`.
-
-It has two parts:
-
-- `shim/`: an `LD_PRELOAD` guard around `close`, `fclose`, `dup2` and `dup3`.
-  - Before each close it resolves `/proc/self/fd/N`.
-  - It logs a backtrace when a non-owner closes `wallet.dat`, `database/log.*` or `blocks/*.dat`, and for every close that fails with `EBADF` (the second half of a double close).
-  - `FDGUARD_ABORT=1` makes the first hit abort.
-- `repro/`: isolated regtest runs that put a shimmed `dragonxd` under load:
-  - **rpc:** impatient and RST clients, half-open connections, work-queue bursts and long-polls;
-  - **p2p:** TLS peers churned by `disconnectnode`/ban/kill, plus misbehaving raw TCP/TLS clients;
-  - **tor:** a fake control port;
-  - **mining:** with `blocknotify` and autoshield.
-
-  Each run executes in its own unprivileged network namespace: loopback only, no route.
-
-## Requirements
-
-- Linux.
-- `unshare -rn` (unprivileged user namespaces), `nsenter`, `ip`, python3, gcc, and `addr2line` for symbolizing.
-- A `dragonxd`/`dragonx-cli` built with `-g`. Adding `-rdynamic` makes backtraces readable without `addr2line`.
-- A regtest datadir with a grown wallet to copy from (`GOLDEN`). It is never modified.
-
-## Running
-
-```sh
-qa/fd-guard/shim/build.sh                      # builds libfdguard.so
-qa/fd-guard/shim/run_tests.sh                  # self-test: double close, foreign close of a guarded file
-cd qa/fd-guard/repro
-export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src   # RUNS defaults to ~/fd-guard-runs
-LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300      # 1 peer, disconnectnode every 5 s
-LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200  # everything, production-like work queue
-python3 tools/tally.py "$RUNS/<name>/fdguard.log"              # who closed what
-python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
-python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat   # BDB pages inside block files
-```
-
-Each run writes `run.txt` (full log plus summary), `fdguard.log` and the target's `debug.log` and `db.log` under `$RUNS/<name>/`.
-
-## What to expect
-
-Measured on a 2041-note regtest wallet, before and after the fix (`net: stop wolfSSL's CloseSocket macro ...`), under the same loads:
-
-| Run | Binary | EBADF closes | Non-owner closes (wallet/log/blk) | libevent `Epoll` failures | `db.log` |
-|---|---|---|---|---|---|
-| `p2pmin`, 300 s | before | 51 (≈ 1 per disconnect) | 1 | 0 | empty |
-| | after | 0 | 0 | 0 | empty |
-| `full`, 1200 s, `WQ=64 RT=8` | before | 930 | 186 (37) | 111 | 632 KB: 7500 × `BDB0060 PANIC`, `write failed for page N` |
-| | after | 0 | 0 | 0 | empty |
-
-- RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
-- `scan_bdb.py` found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
-- `SSL_write ... closing connection` lines are expected under `tcpchurn`. They are how the load triggers disconnects, not a symptom.

qa/fd-guard/repro/common.sh

@@ -1,72 +0,0 @@
-# shellcheck shell=bash
-# Shared config + helpers for the fd-double-close REPRODUCTION harness.
-# Regtest only. Meant to run INSIDE an unprivileged network namespace (see ns.sh),
-# so every port is private to the namespace and nothing can reach a real network.
-set +e
-
-REPRO=${REPRO:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}
-SRCROOT=$(cd "$REPRO/../../.." && pwd)
-BIN=${BIN:-$SRCROOT/src}                    # dir with dragonxd + dragonx-cli (build with -g -rdynamic for backtraces)
-DRAGONXD=$BIN/dragonxd
-CLI=$BIN/dragonx-cli
-RUNS=${RUNS:-$HOME/fd-guard-runs}           # run output (block files, logs): keep it out of the repo
-GOLDEN=${GOLDEN:?set GOLDEN to a regtest datadir with a grown wallet (it is only ever copied)}
-ASMAP=${ASMAP:-$SRCROOT/asmap.dat}
-SP=${SP:-$HOME/.zcash-params/sapling-spend.params}
-OP=${OP:-$HOME/.zcash-params/sapling-output.params}
-SHIM=${SHIM:-$REPRO/../shim/libfdguard.so}
-
-RPCUSER=ab
-RPCPASS=abpass
-RPC_T=18901 ; P2P_T=18911
-RPC_H1=18902; P2P_H1=18912
-RPC_H2=18903; P2P_H2=18913
-TORPORT=18950
-
-clin() { local dd=$1 rp=$2; shift 2; "$CLI" -regtest -datadir="$dd" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp "$@"; }
-
-write_conf() { printf 'rpcuser=%s\nrpcpassword=%s\nrpcport=%s\n' "$RPCUSER" "$RPCPASS" "$2" > "$1/DRAGONX.conf"; }
-
-# prep_golden <dd> : copy of the golden wallet datadir (2041 notes, txindex=1, height 286)
-prep_golden() {
-  local dd=$1
-  rm -rf "$dd"; cp -a "$GOLDEN" "$dd"
-  cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
-  write_conf "$dd" "$RPC_T"
-  : > "$dd/regtest/debug.log"; rm -f "$dd/regtest/db.log"
-}
-
-# prep_fresh <dd> <rpcport> : empty regtest datadir (helpers)
-prep_fresh() {
-  local dd=$1 rp=$2
-  rm -rf "$dd"; mkdir -p "$dd"
-  cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
-  write_conf "$dd" "$rp"
-}
-
-start_helper() {   # <dd> <p2p> <rpc> <logfile>
-  local dd=$1 p2p=$2 rp=$3 lg=$4
-  ( cd "$dd" && exec nice -n 10 "$DRAGONXD" -regtest -datadir="$dd" -testnode=1 -checkpoints=0 \
-      -listen=1 -bind=127.0.0.1 -port=$p2p -connect=127.0.0.1:1 -addnode=127.0.0.1:$P2P_T \
-      -dnsseed=0 -dns=0 -asmap="$dd/asmap.dat" -autoshield=0 \
-      -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp -listenonion=0 -maxinboundfromip=100 \
-      -debug=net -debug=tls -daemon ) >> "$lg" 2>&1
-}
-
-waitrpc() {        # <dd> <rpc> [secs]
-  local dd=$1 rp=$2 n=${3:-300} i
-  for i in $(seq 1 "$n"); do clin "$dd" "$rp" getblockcount >/dev/null 2>&1 && return 0; sleep 1; done
-  return 1
-}
-
-stop_daemon() {    # <dd> <rpc>   (helpers, started with -daemon)
-  local dd=$1 rp=$2 pid i
-  pid=$(cat "$dd/regtest/hushd.pid" 2>/dev/null)
-  clin "$dd" "$rp" stop >/dev/null 2>&1
-  for i in $(seq 1 90); do { [ -z "$pid" ] || ! kill -0 "$pid" 2>/dev/null; } && return 0; sleep 1; done
-  [ -n "$pid" ] && kill -9 "$pid" 2>/dev/null
-}
-
-MARKERS=('libevent: Epoll' 'Socket close failed' 'Bad file descriptor' 'Operation not permitted' 'Illegal seek'
-         'SSL_write' 'Unable to seek' 'socket select error' 'write failed for page' 'PANIC' 'error scanning DB'
-         'terminate' 'Assertion' 'EXCEPTION')

qa/fd-guard/repro/go.sh

@@ -1,14 +0,0 @@
-#!/usr/bin/env bash
-# go.sh <name> <seconds>  -- create a private network namespace (loopback only, no route),
-# run run.sh inside it, then tear the namespace down. Env knobs pass through to run.sh.
-export REPRO=${REPRO:-$(cd "$(dirname "$0")" && pwd)}
-export RUNS=${RUNS:-$HOME/fd-guard-runs}
-OUT=$RUNS/$1; mkdir -p "$OUT"
-setsid unshare -rn sh -c "ip link set lo up && echo \$\$ > $OUT/ns.pid && exec sleep infinity" </dev/null >/dev/null 2>&1 &
-for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s "$OUT/ns.pid" ] && break; sleep 0.3; done
-NSP=$(cat "$OUT/ns.pid")
-cp "$REPRO/run.sh" "$OUT/run.sh.snapshot"   # run a frozen copy: editing run.sh cannot disturb a live run
-nsenter -t "$NSP" -U -n --preserve-credentials bash "$OUT/run.sh.snapshot" "$@" > "$OUT/go.txt" 2>&1
-rc=$?
-kill "$NSP" 2>/dev/null
-echo "go.sh: run.sh rc=$rc, namespace holder $NSP killed" >> "$OUT/go.txt"

qa/fd-guard/repro/run.sh

@@ -1,156 +0,0 @@
-#!/usr/bin/env bash
-# run.sh <name> <seconds>   -- run INSIDE the network namespace (go.sh does that).
-# Env knobs:
-#   LOADS=rpc,p2p,tor,mine   which load generators run (default all four); p2pmin = 1 helper + disconnectnode every DISC_INT s
-#   WQ=4 RT=2                -rpcworkqueue / -rpcthreads of the target
-#   GUARD=all                FDGUARD_GUARD (db,blk | all)
-#   SHIMON=1                 run the target under LD_PRELOAD=libfdguard.so
-#   STRACE=0                 1 = also wrap the target in strace (failed fd syscalls only)
-#   TRACE=0                  1 = FDGUARD_TRACE_ALL (one line per open/close; big)
-#   MINE_INT=3               seconds between generate 1
-#   HAMMER_ARGS=...          override the hammer thread mix
-#   STOP_ON_HIT=0            1 = end the run 60 s after the first NON-OWNER db/blk close
-#   EXTRA=...                extra target args
-source "$REPRO/common.sh"
-NAME=$1; DUR=${2:-600}
-OUT=$RUNS/$NAME
-LOADS=${LOADS:-rpc,p2p,tor,mine}; WQ=${WQ:-4}; RT=${RT:-2}; GUARD=${GUARD:-all}
-SHIMON=${SHIMON:-1}; STRACE=${STRACE:-0}; TRACE=${TRACE:-0}; MINE_INT=${MINE_INT:-3}
-STOP_ON_HIT=${STOP_ON_HIT:-0}
-HAMMER_ARGS=${HAMMER_ARGS:---normal 8 --impatient 8 --slowfire 2 --halfopen 3 --pipeline 3 --garbage 2 --slowread 2 --longpoll 4 --burst-size 16 --burst-every 2}
-has() { case ",$LOADS," in *",$1,"*) return 0;; esac; return 1; }
-
-mkdir -p "$OUT/logs"; rm -f "$OUT/STOP"
-DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
-exec > >(tee -a "$OUT/run.txt") 2>&1
-echo "=== repro run $NAME  $(date -u +%FT%TZ)  dur=${DUR}s loads=$LOADS wq=$WQ rt=$RT guard=$GUARD shim=$SHIMON strace=$STRACE"
-echo "netns check (must be ENETUNREACH=101): $(python3 -c 'import socket;print(socket.socket().connect_ex(("1.1.1.1",80)))')"
-echo "binary: $DRAGONXD  build-id: $(readelf -n "$DRAGONXD" | awk '/Build ID/{print $3}')"
-echo "dragonxd already running in this namespace? $(pgrep -a -x dragonxd | wc -l) (other agents' nodes live in other namespaces)"
-
-PIDS=()
-cleanup() {
-  echo "--- cleanup $(date -u +%T)"
-  touch "$OUT/STOP"
-  for p in "${PIDS[@]}"; do kill "$p" 2>/dev/null; done
-  stop_daemon "$DD_H2" "$RPC_H2"; stop_daemon "$DD_H1" "$RPC_H1"
-  if [ -n "$TPID" ] && kill -0 "$TPID" 2>/dev/null; then
-    timeout 20 "$CLI" -regtest -datadir="$DD_T" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T stop >/dev/null 2>&1
-    for _ in $(seq 1 60); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
-    # abandoned longpoll getblocktemplate calls can hold every -rpcthreads worker, so RPC stop may never run:
-    # fall back to SIGTERM (same clean StartShutdown path), SIGKILL only as a last resort
-    kill -0 "$TPID" 2>/dev/null && { echo "RPC stop did not finish in 60 s -> SIGTERM"; kill -TERM "$TPID"; }
-    for _ in $(seq 1 180); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
-    kill -0 "$TPID" 2>/dev/null && { echo "target did not stop -> SIGKILL"; kill -9 "$TPID"; }
-  fi
-  [ -n "$WPID" ] && wait "$WPID" 2>/dev/null; echo "target exit status: $?"
-  for p in "${PIDS[@]}"; do kill -9 "$p" 2>/dev/null; done
-  summarize
-}
-summarize() {
-  local DL=$DD_T/regtest/debug.log
-  echo "=== markers in target debug.log"
-  for m in "${MARKERS[@]}"; do printf '  [%6d] %s\n' "$(grep -c -- "$m" "$DL" 2>/dev/null)" "$m"; done
-  echo "  db.log bytes: $(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null || echo absent)"
-  grep -q 'Shutdown: done' "$DL" && echo "  clean shutdown: yes" || echo "  clean shutdown: NO"
-  if [ "$SHIMON" = 1 ]; then
-    echo "=== fdguard reports"; grep -o '=== FDGUARD #[0-9]* [A-Z-]*' "$OUT/fdguard.log" | awk '{print $4}' | sort | uniq -c
-    grep '^S ' "$OUT/fdguard.log" | tail -1
-    python3 "$REPRO/../shim/symbolize.py" --summary "$OUT/fdguard.log" > "$OUT/fdguard.summary.txt" 2>&1
-    head -60 "$OUT/fdguard.summary.txt"
-  fi
-  echo "=== BDB-page-in-blockfile detector"
-  python3 "$REPRO/tools/scan_bdb.py" "$DD_T"/regtest/blocks/blk*.dat "$DD_T"/regtest/blocks/rev*.dat 2>&1 | tail -15
-  echo "=== end $(date -u +%FT%TZ)"
-}
-trap cleanup EXIT
-trap 'exit 130' INT TERM
-
-echo "--- prep datadirs"
-prep_golden "$DD_T"
-prep_fresh "$DD_H1" "$RPC_H1"; prep_fresh "$DD_H2" "$RPC_H2"
-cat > "$OUT/blocknotify.sh" <<EOF
-#!/bin/sh
-# pool-like blocknotify: a couple of dragonx-cli calls per block
-$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblock "\$1" >/dev/null 2>&1
-$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblocktemplate >/dev/null 2>&1
-echo "\$(date +%s) \$1" >> $OUT/logs/blocknotify.log
-EOF
-chmod +x "$OUT/blocknotify.sh"
-
-if has tor; then
-  nice -n 10 python3 "$REPRO/tools/faketor.py" --port $TORPORT > "$OUT/logs/faketor.txt" 2>&1 & PIDS+=($!)
-  TORARGS="-listenonion=1 -torcontrol=127.0.0.1:$TORPORT -debug=tor"
-else
-  TORARGS="-listenonion=0"
-fi
-if has p2p || has p2pmin; then P2PARGS="-listen=1 -bind=127.0.0.1 -port=$P2P_T -allowlist=127.0.0.0/8 -maxinboundfromip=100"
-else P2PARGS="-listen=0 -port=$P2P_T"; fi
-
-TARGS=(-regtest -datadir="$DD_T" -testnode=1 -checkpoints=0 $P2PARGS -connect=127.0.0.1:1 -dnsseed=0 -dns=0
-       -asmap="$DD_T/asmap.dat" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T
-       -txindex=1 -autoshield=1 -rpcworkqueue=$WQ -rpcthreads=$RT
-       "-blocknotify=$OUT/blocknotify.sh %s" $TORARGS -debug=net -debug=tls -printtoconsole=0 -daemon=0 $EXTRA)
-echo "target args: ${TARGS[*]}" | sed "s/$RPCPASS/***/g"
-ENVV=()
-if [ "$SHIMON" = 1 ]; then
-  ENVV=(FDGUARD_LOG="$OUT/fdguard.log" FDGUARD_GUARD="$GUARD" FDGUARD_ABORT=0 FDGUARD_STATS_SEC=60 FDGUARD_MAXREPORTS=2000 LD_PRELOAD="$SHIM")
-  [ "$TRACE" = 1 ] && ENVV+=(FDGUARD_TRACE_ALL=1)
-fi
-WRAP=()
-[ "$STRACE" = 1 ] && WRAP=(strace -f -tt -qq -e trace=close,openat,accept4,socket,dup2,dup3 -e status=failed -o "$OUT/strace.txt")
-( cd "$DD_T" && exec env "${ENVV[@]}" nice -n 10 "${WRAP[@]}" "$DRAGONXD" "${TARGS[@]}" ) > "$OUT/logs/target_stdout.txt" 2>&1 &
-WPID=$!
-echo "--- target launched (wrapper pid $WPID); waiting for RPC"
-waitrpc "$DD_T" "$RPC_T" 600 || { echo "target never answered RPC"; exit 1; }
-TPID=$(cat "$DD_T/regtest/hushd.pid" 2>/dev/null); [ -z "$TPID" ] && TPID=$WPID
-echo "target up pid=$TPID height=$(clin "$DD_T" "$RPC_T" getblockcount) $(date -u +%T)"
-
-T0=$(date +%s)
-if has p2p; then
-  start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"
-  start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
-  waitrpc "$DD_H1" "$RPC_H1" 120; waitrpc "$DD_H2" "$RPC_H2" 120
-  nice -n 10 python3 "$REPRO/tools/tcpchurn.py" --port $P2P_T --duration "$DUR" --concurrency 8 > "$OUT/logs/tcpchurn.txt" 2>&1 & PIDS+=($!)
-  nice -n 10 bash "$REPRO/tools/rpcchurn.sh" "$OUT" "$DUR" > "$OUT/logs/rpcchurn.txt" 2>&1 & PIDS+=($!)
-fi
-if has p2pmin; then
-  # minimal P2P: ONE well-behaved helper peer, and an ordinary "disconnectnode" every DISC_INT s
-  # (the helper's -addnode re-dials the target). No raw TCP/TLS churn, no bans, no kills.
-  start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"; waitrpc "$DD_H1" "$RPC_H1" 120
-  ( k=0; while [ ! -e "$OUT/STOP" ]; do sleep "${DISC_INT:-5}"; for a in $(clin "$DD_T" "$RPC_T" getpeerinfo 2>/dev/null | python3 -c 'import json,sys; [print(p["addr"]) for p in json.load(sys.stdin)]' 2>/dev/null); do
-      clin "$DD_T" "$RPC_T" disconnectnode "$a" >/dev/null 2>&1 && { k=$((k+1)); echo "$(date -u +%T.%N) disconnectnode $a" >> "$OUT/logs/disconnects.log"; }; done
-    sleep 1; clin "$DD_H1" "$RPC_H1" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1   # helper re-dials now (its -addnode retry is ~2 min)
-    echo "$k" > "$OUT/logs/disconnects.txt"; done ) & PIDS+=($!)
-fi
-if has rpc; then
-  MINEARG=""; has mine && MINEARG="--mine --mine-interval $MINE_INT"
-  nice -n 10 python3 "$REPRO/tools/hammer.py" --port $RPC_T --user $RPCUSER --password $RPCPASS --duration "$DUR" \
-      $HAMMER_ARGS $MINEARG --pid "$TPID" --logfile "$OUT/logs/hammer.log" > "$OUT/logs/hammer_stdout.txt" 2>&1 & PIDS+=($!)
-elif has mine; then
-  ( while [ ! -e "$OUT/STOP" ]; do clin "$DD_T" "$RPC_T" generate 1 >/dev/null 2>&1; sleep "$MINE_INT"; done ) & PIDS+=($!)
-fi
-
-# ---- poll
-end=$(( T0 + DUR )); n=0; hit_at=0
-while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
-  sleep 30; n=$((n+1))
-  alive=no; kill -0 "$TPID" 2>/dev/null && alive=yes
-  DL=$DD_T/regtest/debug.log
-  h=$(clin "$DD_T" "$RPC_T" getblockcount 2>/dev/null); c=$(clin "$DD_T" "$RPC_T" getconnectioncount 2>/dev/null)
-  rep=$(grep -c '^=== FDGUARD' "$OUT/fdguard.log" 2>/dev/null); noc=$(grep -c 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null)
-  ebf=$(grep -c 'EBADF-CLOSE' "$OUT/fdguard.log" 2>/dev/null); epl=$(grep -c '=== FDGUARD #[0-9]* EPOLL' "$OUT/fdguard.log" 2>/dev/null)
-  dbnoc=$(grep -A3 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null | grep -c 'class=\(db\|blk\)')
-  printf '[poll %3d %s +%4ds] alive=%s h=%s conns=%s | fdguard reports=%s nonowner=%s(db/blk %s) ebadf=%s epoll=%s | Epoll=%s SockCloseFail=%s EBADF=%s EPERM=%s SSL_write=%s seek=%s sel=%s | db.log=%s\n' \
-    $n "$(date -u +%T)" $(( $(date +%s) - T0 )) $alive "$h" "$c" "$rep" "$noc" "$dbnoc" "$ebf" "$epl" \
-    "$(grep -c 'libevent: Epoll' "$DL")" "$(grep -c 'Socket close failed' "$DL")" "$(grep -c 'Bad file descriptor' "$DL")" \
-    "$(grep -c 'Operation not permitted' "$DL")" "$(grep -c 'SSL_write' "$DL")" "$(grep -c 'Unable to seek' "$DL")" \
-    "$(grep -c 'socket select error' "$DL")" "$(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null)" | tee -a "$OUT/poll.txt"
-  [ "$alive" = no ] && { echo "!!! target died"; break; }
-  if [ "$STOP_ON_HIT" = 1 ] && [ "${dbnoc:-0}" -gt 0 ]; then
-    [ "$hit_at" = 0 ] && hit_at=$(date +%s)
-    [ $(( $(date +%s) - hit_at )) -ge 60 ] && { echo "--- stop-on-hit"; break; }
-  fi
-done
-cat "$OUT/logs/tcpchurn.txt" "$OUT/logs/rpcchurn.txt" 2>/dev/null
-grep -E 'TOTAL|totals' -A12 "$OUT/logs/hammer.log" 2>/dev/null | tail -14

qa/fd-guard/repro/tools/faketor.py

@@ -1,98 +0,0 @@
-#!/usr/bin/env python3
-"""
-Fake Tor control port for the fd-bug harness.
-
-dragonxd's torcontrol.cpp opens a libevent bufferevent to -torcontrol=<host:port>,
-speaks the control protocol (PROTOCOLINFO / AUTHCHALLENGE / AUTHENTICATE / ADD_ONION),
-and on EOF/ERROR reconnects with exponential backoff. We give it a server on a
-loopback port that behaves badly in several ways so the reconnect/teardown loop
-(bufferevent_free with BEV_OPT_CLOSE_ON_FREE, the event_base, the reconnect timer)
-runs constantly. Each accepted connection picks a behaviour at random:
-
-  close   : accept, then immediately close (EOF right away)
-  rst     : accept, set SO_LINGER 0, close -> sends RST
-  stall   : accept, read nothing, hold the socket open then drop it
-  garbage : send non-control-protocol bytes then close
-  halfok  : answer PROTOCOLINFO plausibly, then drop before AUTHENTICATE completes
-
-Loopback only. Run with --port. Ctrl-C / SIGTERM stops it.
-"""
-import argparse, random, socket, struct, threading, time, sys, signal
-
-STOP = threading.Event()
-
-def handle(conn, mode):
-    try:
-        conn.settimeout(5.0)
-        if mode == "close":
-            conn.close(); return
-        if mode == "rst":
-            conn.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
-                            struct.pack("ii", 1, 0))
-            conn.close(); return
-        if mode == "stall":
-            # read whatever the controller sends but never reply; hold 2-4s
-            try:
-                conn.recv(4096)
-            except Exception:
-                pass
-            time.sleep(random.uniform(2.0, 4.0))
-            conn.close(); return
-        if mode == "garbage":
-            try:
-                conn.recv(4096)
-            except Exception:
-                pass
-            conn.sendall(b"\x00\xff\x01garbage not a tor reply\r\n" * 4)
-            conn.close(); return
-        if mode == "halfok":
-            # read the PROTOCOLINFO line, give a plausible 250 reply advertising
-            # NULL auth, then drop the connection before AUTHENTICATE finishes.
-            try:
-                conn.recv(4096)
-            except Exception:
-                pass
-            conn.sendall(b"250-PROTOCOLINFO 1\r\n"
-                         b"250-AUTH METHODS=NULL\r\n"
-                         b'250-VERSION Tor="0.4.7.13"\r\n'
-                         b"250 OK\r\n")
-            time.sleep(random.uniform(0.2, 1.0))
-            conn.close(); return
-    except Exception:
-        try: conn.close()
-        except Exception: pass
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--port", type=int, required=True)
-    ap.add_argument("--host", default="127.0.0.1")
-    args = ap.parse_args()
-
-    modes = ["close", "rst", "stall", "garbage", "halfok"]
-    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    srv.bind((args.host, args.port))
-    srv.listen(16)
-    srv.settimeout(1.0)
-    print(f"faketor: listening on {args.host}:{args.port}", flush=True)
-
-    def onsig(*_): STOP.set()
-    signal.signal(signal.SIGTERM, onsig)
-    signal.signal(signal.SIGINT, onsig)
-
-    n = 0
-    while not STOP.is_set():
-        try:
-            conn, _ = srv.accept()
-        except socket.timeout:
-            continue
-        except Exception:
-            break
-        mode = modes[n % len(modes)]
-        n += 1
-        threading.Thread(target=handle, args=(conn, mode), daemon=True).start()
-    srv.close()
-    print(f"faketor: stopped after {n} connections", flush=True)
-
-if __name__ == "__main__":
-    main()

qa/fd-guard/repro/tools/hammer.py

@@ -1,796 +0,0 @@
-#!/usr/bin/env python3
-# hammer.py -- JSON-RPC robustness/load harness for a LOCAL regtest dragonxd.
-#
-# Scope: connects only to the --host/--port you pass (meant to be 127.0.0.1 and a
-# regtest RPC port). Its job is to reproduce a known file-descriptor bug in the
-# libevent-2.1.8 HTTP RPC server (src/httpserver.cpp) by exercising the same
-# request patterns the production pool box sees: many concurrent clients, clients
-# that disconnect mid-request/mid-response (the explorer indexer's 120 s client
-# timeout), work-queue overflow, keep-alive pipelining, and malformed input.
-# stdlib only.
-#
-# Correctness oracle: every request carries a unique JSON-RPC id; if a reply comes
-# back with a different id on the same socket, that is cross-talk (the server wrote
-# one client's reply onto another client's fd) and is logged + counted.
-#
-# See README.txt for the exact invocation.
-
-import argparse
-import base64
-import collections
-import json
-import os
-import random
-import socket
-import struct
-import threading
-import time
-
-# ------------------------------------------------------------------ config/state
-
-class C:
-    host = "127.0.0.1"
-    port = 19461
-    auth = ""
-    ctimeout = 15.0
-
-CFG = C()
-STOP = threading.Event()
-SHARED = {"hashes": [], "lpid": None, "zaddrs": [], "rawblock": None}
-SHARED_LOCK = threading.Lock()
-_ID = [random.SystemRandom().randrange(1 << 40)]
-_ID_LOCK = threading.Lock()
-LOGLOCK = threading.Lock()
-LOGF = [None]
-
-def next_id():
-    with _ID_LOCK:
-        _ID[0] += 1
-        return _ID[0]
-
-def log(msg):
-    line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
-    with LOGLOCK:
-        print(line, flush=True)
-        if LOGF[0]:
-            LOGF[0].write(line + "\n"); LOGF[0].flush()
-
-# ------------------------------------------------------------------ stats
-
-KEYS = ["sent", "ok", "rpcerr", "wq500", "http401", "http4xx", "httpother",
-        "early_rst", "early_fin", "srvreset", "srveof", "ctimeout", "connfail",
-        "idmismatch", "malformed", "mined", "half_open", "half_srvclose"]
-
-class Stats:
-    def __init__(self):
-        self.lk = threading.Lock()
-        self.total = collections.Counter()
-        self.win = collections.Counter()
-        self.anom = []
-    def inc(self, k, n=1):
-        with self.lk:
-            self.total[k] += n; self.win[k] += n
-    def anomaly(self, kind, detail):
-        with self.lk:
-            if len(self.anom) < 300:
-                self.anom.append((time.time(), kind, detail))
-        log("ANOMALY %s %s" % (kind, detail))
-    def swap(self):
-        with self.lk:
-            w = self.win; self.win = collections.Counter(); return w
-
-ST = Stats()
-
-# ------------------------------------------------------------------ socket helpers
-
-def auth_header(u, p):
-    return "Basic " + base64.b64encode(("%s:%s" % (u, p)).encode()).decode()
-
-def mkconn(timeout=None, rcvbuf=None):
-    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    if rcvbuf:
-        s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, rcvbuf)
-    s.settimeout(timeout if timeout is not None else CFG.ctimeout)
-    s.connect((CFG.host, CFG.port))
-    return s
-
-def end_rst(s):
-    # linger 0 -> TCP RST on close (models a client that dies / resets abruptly)
-    try:
-        s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
-    except OSError:
-        pass
-    try:
-        s.close()
-    except OSError:
-        pass
-
-def end_fin(s):
-    try:
-        s.shutdown(socket.SHUT_RDWR)
-    except OSError:
-        pass
-    try:
-        s.close()
-    except OSError:
-        pass
-
-def end_early(s, how=None):
-    how = how or random.choice(("rst", "fin"))
-    if how == "rst":
-        end_rst(s); ST.inc("early_rst")
-    else:
-        end_fin(s); ST.inc("early_fin")
-
-def req_bytes(method, params, rid, keepalive=True, auth=None, uri="/",
-              httpmethod="POST", body=None, clen=None):
-    if body is None:
-        body = json.dumps({"jsonrpc": "1.0", "id": rid,
-                           "method": method, "params": params}).encode()
-    head = [
-        "%s %s HTTP/1.1" % (httpmethod, uri),
-        "Host: %s:%d" % (CFG.host, CFG.port),
-        "Authorization: %s" % (auth if auth is not None else CFG.auth),
-        "Content-Type: application/json",
-        "Content-Length: %d" % (clen if clen is not None else len(body)),
-        "Connection: %s" % ("keep-alive" if keepalive else "close"),
-        "", "",
-    ]
-    return "\r\n".join(head).encode() + body
-
-class Reader:
-    """Incremental HTTP/1.1 reply reader over one socket."""
-    def __init__(self, s):
-        self.s = s; self.buf = b""
-    def _fill(self):
-        d = self.s.recv(65536)
-        if not d:
-            raise EOFError()
-        self.buf += d
-    def read(self):
-        while b"\r\n\r\n" not in self.buf:
-            self._fill()
-            if len(self.buf) > (1 << 20):
-                raise ValueError("oversize header")
-        head, self.buf = self.buf.split(b"\r\n\r\n", 1)
-        lines = head.split(b"\r\n")
-        p = lines[0].split(b" ", 2)
-        if len(p) < 2 or not p[0].startswith(b"HTTP/1."):
-            raise ValueError("bad status line %r" % lines[0][:80])
-        status = int(p[1])
-        hdr = {}
-        for ln in lines[1:]:
-            if b":" in ln:
-                k, v = ln.split(b":", 1)
-                hdr[k.strip().lower()] = v.strip()
-        if hdr.get(b"transfer-encoding", b"").lower() == b"chunked":
-            body = b""
-            while True:
-                while b"\r\n" not in self.buf:
-                    self._fill()
-                ln, self.buf = self.buf.split(b"\r\n", 1)
-                n = int(ln.split(b";")[0], 16)
-                while len(self.buf) < n + 2:
-                    self._fill()
-                body += self.buf[:n]; self.buf = self.buf[n + 2:]
-                if n == 0:
-                    break
-        elif b"content-length" in hdr:
-            n = int(hdr[b"content-length"])
-            while len(self.buf) < n:
-                self._fill()
-            body, self.buf = self.buf[:n], self.buf[n:]
-        else:
-            try:
-                while True:
-                    self._fill()
-            except EOFError:
-                pass
-            body, self.buf = self.buf, b""
-        return status, hdr, body
-
-def account(status, body, rid, method, expect401=False):
-    if status == 401:
-        ST.inc("http401")
-        if not expect401:
-            ST.anomaly("unexpected401", (method,))
-        return None
-    if status == 500 and b"Work queue depth exceeded" in body:
-        ST.inc("wq500"); return None
-    j = None
-    if body[:1] in (b"{", b"["):
-        try:
-            j = json.loads(body)
-        except ValueError:
-            j = None
-    if j is None:
-        if 400 <= status < 500:
-            ST.inc("http4xx")
-        else:
-            ST.inc("httpother")
-            if status == 200:
-                ST.anomaly("non-json-200", (method, body[:120].decode("latin1")))
-        return None
-    if isinstance(j, dict) and rid is not None and j.get("id") != rid:
-        ST.inc("idmismatch")
-        ST.anomaly("idmismatch", {"method": method, "sent": rid, "got": j.get("id"),
-                                  "body": body[:200].decode("latin1")})
-    if isinstance(j, dict) and j.get("error"):
-        ST.inc("rpcerr")
-    else:
-        ST.inc("ok")
-    return j
-
-def call(s, rd, method, params, keepalive=True, expect401=False, auth=None):
-    rid = next_id()
-    try:
-        s.sendall(req_bytes(method, params, rid, keepalive=keepalive, auth=auth))
-        ST.inc("sent")
-        status, hdr, body = rd.read()
-    except socket.timeout:
-        ST.inc("ctimeout"); return None, False
-    except (ConnectionResetError, BrokenPipeError):
-        ST.inc("srvreset"); return None, False
-    except EOFError:
-        ST.inc("srveof"); return None, False
-    except ValueError as e:
-        ST.inc("malformed"); ST.anomaly("malformed", (method, str(e))); return None, False
-    except OSError:
-        ST.inc("srvreset"); return None, False
-    j = account(status, body, rid, method, expect401)
-    alive = keepalive and hdr.get(b"connection", b"").lower() != b"close"
-    return j, alive
-
-# ------------------------------------------------------------------ menus
-
-def a_hash():
-    with SHARED_LOCK:
-        return random.choice(SHARED["hashes"]) if SHARED["hashes"] else None
-
-def a_zaddr():
-    with SHARED_LOCK:
-        return random.choice(SHARED["zaddrs"]) if SHARED["zaddrs"] else None
-
-def normal_menu():
-    # Mostly fast calls (sub-ms) so a well-behaved client interleaves with mining.
-    # Heavy wallet calls (z_gettotalbalance ~2.3s on the golden wallet) appear but
-    # are rare, so they don't permanently peg the small worker pool on their own.
-    fast = [("getinfo", []), ("getblockcount", []), ("getbestblockhash", []),
-            ("getrawmempool", []), ("getmininginfo", []), ("getdifficulty", []),
-            ("getblockchaininfo", []), ("getpeerinfo", []),
-            ("getblocktemplate", []), ("getnetworkinfo", [])]
-    h = a_hash()
-    if h:
-        fast.append(("getblock", [h, 2]))
-        fast.append(("getblockheader", [h]))
-    # "slow" here is only mildly-larger fast calls; the genuinely heavy wallet calls
-    # live in the paced slow-fire class so a normal dashboard poller never pegs the
-    # worker pool.
-    slow = [("coinsupply", []), ("listtransactions", ["*", 10]),
-            ("listunspent", [0]), ("getblock", [a_hash() or "", 2])]
-    return random.choice(slow) if random.random() < 0.1 else random.choice(fast)
-
-def medium_menu():
-    # Fast to compute (~4 ms on this node) but a non-trivial reply to write, so they
-    # are good targets for a mid-response disconnect. The fd-reuse race fires on the
-    # disconnect, not on call duration, so firing these at a HIGH rate drives the
-    # race hard while the worker frees in milliseconds (mining keeps its slots).
-    m = [("listtransactions", ["*", 10000]), ("getrawmempool", []),
-         ("getrawmempool", [True]), ("getblockchaininfo", []),
-         ("getblocktemplate", []), ("getchaintips", []), ("getinfo", []),
-         ("coinsupply", []), ("listunspent", [0])]
-    h = a_hash()
-    if h:
-        m.append(("getblock", [h, 2]))
-        m.append(("getblock", [h, 1]))
-    return random.choice(m)
-
-def heavy_menu():
-    # Genuinely slow calls (z_gettotalbalance ~1.7 s, z_listunspent ~5.7 s on the
-    # golden wallet). Fired only by the small, PACED slow-fire class so a worker
-    # keeps running a multi-second call long after the client disconnected (the
-    # production indexer/coinsupply scenario that widens the fd-reuse window),
-    # without the fire-and-disconnect backlog pinning the whole pool.
-    m = [("z_gettotalbalance", [0]), ("z_listunspent", [0])]
-    z = a_zaddr()
-    if z:
-        m.append(("z_listreceivedbyaddress", [z, 0]))
-    return random.choice(m)
-
-# ------------------------------------------------------------------ clients
-
-def client_normal(cid):
-    while not STOP.is_set():
-        keepalive = random.random() < 0.7
-        try:
-            s = mkconn()
-        except OSError:
-            ST.inc("connfail"); time.sleep(0.05); continue
-        rd = Reader(s)
-        n = random.randint(1, 8) if keepalive else 1
-        try:
-            for _ in range(n):
-                if STOP.is_set():
-                    break
-                method, params = normal_menu()
-                j, alive = call(s, rd, method, params, keepalive=keepalive)
-                if not alive:
-                    break
-        finally:
-            (end_fin if random.random() < 0.8 else end_rst)(s)
-        time.sleep(random.uniform(0.01, 0.15))
-
-def client_impatient(cid):
-    # High-rate fast-disconnect churn: send a (medium) request, then disconnect
-    # after 1-500 ms at a random phase. This is the main driver of the fd-reuse
-    # race -- a disconnect concurrent with the server writing the reply / re-arming
-    # the socket -- and the worker frees in ms so mining is unaffected.
-    while not STOP.is_set():
-        method, params = medium_menu()
-        rid = next_id()
-        body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
-                           "params": params}).encode()
-        phase = random.choice(("mid_request", "after_send", "mid_response"))
-        try:
-            s = mkconn(timeout=2.0)
-        except OSError:
-            ST.inc("connfail"); continue
-        try:
-            if phase == "mid_request":
-                full = req_bytes(method, params, rid, keepalive=False, body=body)
-                cut = random.randint(len(full) - len(body), len(full) - 1)
-                s.sendall(full[:cut])
-                time.sleep(random.uniform(0.001, 0.5))
-                end_early(s); continue
-            s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
-            ST.inc("sent")
-            time.sleep(random.uniform(0.001, 0.5))
-            if phase == "mid_response":
-                try:
-                    s.recv(random.randint(0, 64))  # read a few bytes, then bail
-                except OSError:
-                    pass
-            end_early(s)
-        except (OSError, socket.timeout):
-            ST.inc("srvreset")
-            end_rst(s)
-        time.sleep(random.uniform(0.05, 0.4))
-
-def client_slowfire(cid, pace):
-    # Paced indexer/coinsupply client: fire ONE genuinely-slow request, disconnect
-    # after 1-500 ms (so the worker runs the multi-second call with no client left
-    # == the production fd-reuse window), then wait `pace` s before the next so a
-    # fire-and-disconnect backlog cannot pin the whole worker pool. Phase: cut the
-    # request mid-body, or send it whole and abort while the worker is busy, or read
-    # a few bytes of the reply then abort.
-    while not STOP.is_set():
-        method, params = heavy_menu()
-        rid = next_id()
-        body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
-                           "params": params}).encode()
-        phase = random.choice(("mid_request", "after_send", "mid_response"))
-        try:
-            s = mkconn(timeout=2.0)
-        except OSError:
-            ST.inc("connfail"); time.sleep(pace); continue
-        try:
-            if phase == "mid_request":
-                full = req_bytes(method, params, rid, keepalive=False, body=body)
-                cut = random.randint(len(full) - len(body), len(full) - 1)
-                s.sendall(full[:cut])
-                time.sleep(random.uniform(0.001, 0.5)); end_early(s)
-            else:
-                s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
-                ST.inc("sent")
-                time.sleep(random.uniform(0.001, 0.5))
-                if phase == "mid_response":
-                    try:
-                        s.recv(random.randint(0, 64))
-                    except OSError:
-                        pass
-                end_early(s)
-        except (OSError, socket.timeout):
-            ST.inc("srvreset"); end_rst(s)
-        time.sleep(random.uniform(pace * 0.5, pace * 1.5))
-
-def client_half_open(cid):
-    # connect, send nothing / partial headers, trickle, then vanish
-    while not STOP.is_set():
-        try:
-            s = mkconn(timeout=1.0)
-        except OSError:
-            ST.inc("connfail"); time.sleep(0.1); continue
-        ST.inc("half_open")
-        style = random.choice(("silent", "partial_headers", "full_headers_no_body"))
-        try:
-            if style == "partial_headers":
-                s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\n")
-            elif style == "full_headers_no_body":
-                s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\nAuthorization: " +
-                          CFG.auth.encode() + b"\r\nContent-Length: 1000\r\n\r\n")
-            deadline = time.monotonic() + random.uniform(0.1, 20.0)
-            while time.monotonic() < deadline and not STOP.is_set():
-                time.sleep(random.uniform(0.2, 1.5))
-                try:
-                    s.sendall(b" ")  # one byte, keep it half-alive
-                except OSError:
-                    ST.inc("half_srvclose"); break
-                try:
-                    s.settimeout(0.01)
-                    if s.recv(1) == b"":
-                        ST.inc("half_srvclose"); break
-                except socket.timeout:
-                    pass
-                except OSError:
-                    ST.inc("half_srvclose"); break
-        finally:
-            end_early(s)
-
-def client_burst(cid, size, every):
-    while not STOP.is_set():
-        batch = []
-        for _ in range(size):
-            try:
-                s = mkconn(timeout=3.0)
-            except OSError:
-                ST.inc("connfail"); continue
-            batch.append(s)
-        for s in batch:
-            try:
-                method, params = medium_menu()
-                s.sendall(req_bytes(method, params, next_id(), keepalive=False))
-                ST.inc("sent")
-            except OSError:
-                pass
-        # most close immediately (before the 500/reply is written); a few read it
-        keep = random.sample(batch, max(0, len(batch) // 4)) if batch else []
-        for s in batch:
-            if s in keep:
-                continue
-            end_early(s)
-        for s in keep:
-            try:
-                rd = Reader(s); st, hd, bd = rd.read(); account(st, bd, None, "burst")
-            except Exception:
-                ST.inc("srvreset")
-            finally:
-                end_fin(s)
-        for _ in range(int(every * 10)):
-            if STOP.is_set():
-                break
-            time.sleep(0.1)
-
-def client_pipeline(cid):
-    while not STOP.is_set():
-        try:
-            s = mkconn()
-        except OSError:
-            ST.inc("connfail"); time.sleep(0.05); continue
-        n = random.randint(2, 8)
-        ids = []
-        blob = b""
-        for _ in range(n):
-            rid = next_id(); ids.append(rid)
-            method, params = normal_menu()
-            blob += req_bytes(method, params, rid, keepalive=True)
-        try:
-            s.sendall(blob); ST.inc("sent", n)
-            rd = Reader(s)
-            toread = random.randint(0, n)  # sometimes leave replies unread, then close
-            for k in range(toread):
-                st, hd, bd = rd.read()
-                account(st, bd, ids[k], "pipeline")
-        except (OSError, EOFError, ValueError):
-            ST.inc("srvreset")
-        finally:
-            end_early(s)
-        time.sleep(random.uniform(0, 0.03))
-
-def client_garbage(cid):
-    probes = [
-        ("submitblock", lambda: ["ab" * random.randint(1, 4000)]),
-        ("submitblock", lambda: [os.urandom(random.randint(1, 200)).hex()]),
-        ("submitblock", lambda: ["zzzz"]),
-        ("submitblock", lambda: [""]),
-    ]
-    while not STOP.is_set():
-        pick = random.random()
-        try:
-            s = mkconn(timeout=5.0)
-        except OSError:
-            ST.inc("connfail"); continue
-        try:
-            if pick < 0.5:
-                method, mk = random.choice(probes)
-                j, _ = call(s, Reader(s), method, mk(), keepalive=False)
-            elif pick < 0.65:
-                # malformed JSON body
-                s.sendall(req_bytes("x", [], next_id(), keepalive=False,
-                                    body=b'{"jsonrpc":"1.0","method":'))
-                ST.inc("sent")
-                try:
-                    account(*((lambda r: (r[0], r[2]))(Reader(s).read())), rid=None, method="badjson")
-                except Exception:
-                    ST.inc("srvreset")
-            elif pick < 0.8:
-                # unknown method / wrong URI
-                s.sendall(req_bytes("getinfo", [], next_id(), keepalive=False, uri="/nope"))
-                ST.inc("sent")
-                try:
-                    st, hd, bd = Reader(s).read(); account(st, bd, None, "wronguri")
-                except Exception:
-                    ST.inc("srvreset")
-            else:
-                # Content-Length larger than the body we send, then bail
-                body = b'{"id":1}'
-                s.sendall(req_bytes("getinfo", [], 1, keepalive=False, body=body, clen=len(body) + 5000))
-                ST.inc("sent")
-                time.sleep(random.uniform(0.001, 0.3))
-                end_early(s)
-                continue
-        except (OSError, socket.timeout):
-            ST.inc("srvreset")
-        finally:
-            end_early(s)
-        time.sleep(random.uniform(0.02, 0.2))
-
-def client_slowread(cid):
-    # ask for a big reply with a tiny recv buffer and do not read it, stalling the
-    # server write side, then close abruptly
-    while not STOP.is_set():
-        try:
-            s = mkconn(timeout=3.0, rcvbuf=512)
-        except OSError:
-            ST.inc("connfail"); continue
-        try:
-            s.sendall(req_bytes("listtransactions", ["*", 10000], next_id(), keepalive=False))
-            ST.inc("sent")
-            time.sleep(random.uniform(0.05, 1.0))
-        except OSError:
-            ST.inc("srvreset")
-        finally:
-            end_early(s)
-        time.sleep(random.uniform(0, 0.05))
-
-def _stale_lpid():
-    # A longpollid whose hash != the current tip makes getblocktemplate return at
-    # once (the server's wait loop condition is immediately false), so it exercises
-    # the long-poll request path (lpval parsing, cs_main leave/enter) and the
-    # mid-response disconnect WITHOUT parking a worker. On a tiny -rpcthreads pool a
-    # genuinely parking long-poll + RPC-driven mining self-deadlocks (both workers
-    # wait for a block that only a third worker could mine); that is a pool-size
-    # artifact, not the fd bug, so the harness avoids it by default.
-    with SHARED_LOCK:
-        lpid = SHARED["lpid"]
-    if lpid and len(lpid) > 64:
-        h = lpid[:64]
-        flip = "0" if h[0] != "0" else "1"
-        return flip + h[1:] + lpid[64:]
-    return ("0" * 64) + "0"
-
-def client_longpoll(cid):
-    while not STOP.is_set():
-        try:
-            s = mkconn(timeout=10.0)
-        except OSError:
-            ST.inc("connfail"); time.sleep(0.2); continue
-        rd = Reader(s)
-        params = [{"longpollid": _stale_lpid()}]
-        rid = next_id()
-        try:
-            s.sendall(req_bytes("getblocktemplate", params, rid, keepalive=False))
-            ST.inc("sent")
-            if random.random() < 0.5:
-                # impatient pool worker: abandon the request after 1-500 ms
-                time.sleep(random.uniform(0.001, 0.5))
-                end_early(s); continue
-            st, hd, bd = rd.read()
-            j = account(st, bd, rid, "getblocktemplate(lp)")
-            if j and isinstance(j, dict) and j.get("result"):
-                with SHARED_LOCK:
-                    SHARED["lpid"] = j["result"].get("longpollid")
-        except (OSError, socket.timeout, EOFError, ValueError):
-            ST.inc("srvreset")
-        finally:
-            end_fin(s)
-        time.sleep(random.uniform(0.02, 0.2))
-
-def miner(interval):
-    # Mine a block every `interval` seconds. Retry through work-queue rejections so
-    # blocks still land under heavy load (the production box mines steadily too).
-    while not STOP.is_set():
-        got = False
-        deadline = time.monotonic() + interval + 30
-        while not got and not STOP.is_set() and time.monotonic() < deadline:
-            try:
-                s = mkconn(timeout=60.0)
-                j, _ = call(s, Reader(s), "generate", [1], keepalive=False)
-                end_fin(s)
-                if j and not j.get("error") and j.get("result"):
-                    ST.inc("mined"); got = True
-                    h = j["result"][0] if isinstance(j["result"], list) else None
-                    if h:
-                        with SHARED_LOCK:
-                            SHARED["hashes"].append(h)
-                            SHARED["hashes"] = SHARED["hashes"][-200:]
-                else:
-                    time.sleep(0.2)  # likely work-queue 500; back off and retry
-            except Exception:
-                time.sleep(0.2)
-        for _ in range(int(interval * 10)):
-            if STOP.is_set():
-                break
-            time.sleep(0.1)
-
-# ------------------------------------------------------------------ bootstrap shared state
-
-def bootstrap():
-    try:
-        s = mkconn(timeout=30.0); rd = Reader(s)
-        j, _ = call(s, rd, "getbestblockhash", [], keepalive=True)
-        if j and j.get("result"):
-            with SHARED_LOCK:
-                SHARED["hashes"].append(j["result"])
-        j, _ = call(s, rd, "getblockcount", [], keepalive=True)
-        cnt = j.get("result") if j else None
-        if isinstance(cnt, int):
-            for hh in range(max(0, cnt - 30), cnt + 1):
-                jj, _ = call(s, rd, "getblockhash", [hh], keepalive=True)
-                if jj and jj.get("result"):
-                    with SHARED_LOCK:
-                        SHARED["hashes"].append(jj["result"])
-        j, _ = call(s, rd, "listaddresses", [], keepalive=True)
-        j, _ = call(s, rd, "z_listaddresses", [], keepalive=True)
-        if j and isinstance(j.get("result"), list):
-            with SHARED_LOCK:
-                SHARED["zaddrs"] = j["result"][:16]
-        j, _ = call(s, rd, "getblocktemplate", [{}], keepalive=True)
-        if j and isinstance(j.get("result"), dict):
-            with SHARED_LOCK:
-                SHARED["lpid"] = j["result"].get("longpollid")
-        end_fin(s)
-        log("bootstrap: %d hashes, %d zaddrs, lpid=%s" %
-            (len(SHARED["hashes"]), len(SHARED["zaddrs"]), bool(SHARED["lpid"])))
-    except Exception as e:
-        log("bootstrap failed: %s" % e)
-
-# ------------------------------------------------------------------ fd accounting
-
-def count_fds(pid):
-    if not pid:
-        return (None, None, None)
-    base = "/proc/%d/fd" % pid
-    try:
-        fds = os.listdir(base)
-    except OSError:
-        return (None, None, None)
-    sock = files = 0
-    for fd in fds:
-        try:
-            tgt = os.readlink(os.path.join(base, fd))
-        except OSError:
-            continue
-        if tgt.startswith("socket:"):
-            sock += 1
-        elif tgt.startswith("/"):
-            files += 1
-    return (len(fds), sock, files)
-
-def pid_alive(pid):
-    if not pid:
-        return True
-    try:
-        os.kill(pid, 0); return True
-    except OSError:
-        return False
-
-# ------------------------------------------------------------------ main
-
-def reporter(pid):
-    t0 = time.monotonic()
-    while not STOP.is_set():
-        for _ in range(10):
-            if STOP.is_set():
-                break
-            time.sleep(0.1)
-        w = ST.swap()
-        tot, sk, fl = count_fds(pid)
-        fdstr = ""
-        if tot is not None:
-            fdstr = " | fds=%d sock=%d file=%d" % (tot, sk, fl)
-        log("t=%4ds ok=%d rpcerr=%d wq500=%d 4xx=%d other=%d | "
-            "earlyRST=%d earlyFIN=%d srvRST=%d srvEOF=%d ctmo=%d connfail=%d | "
-            "idmis=%d malf=%d | mined=%d halfopen=%d%s" % (
-                int(time.monotonic() - t0),
-                w["ok"], w["rpcerr"], w["wq500"], w["http4xx"], w["httpother"],
-                w["early_rst"], w["early_fin"], w["srvreset"], w["srveof"],
-                w["ctimeout"], w["connfail"], w["idmismatch"], w["malformed"],
-                w["mined"], w["half_open"], fdstr))
-        if pid and not pid_alive(pid):
-            log("!!! NODE PROCESS %d IS GONE -- stopping harness" % pid)
-            STOP.set()
-            break
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--host", default="127.0.0.1")
-    ap.add_argument("--port", type=int, default=19461)
-    ap.add_argument("--user", default="ab")
-    ap.add_argument("--password", default="abpass")
-    ap.add_argument("--duration", type=int, default=300)
-    ap.add_argument("--ctimeout", type=float, default=15.0)
-    ap.add_argument("--normal", type=int, default=24)
-    ap.add_argument("--impatient", type=int, default=10)
-    ap.add_argument("--slowfire", type=int, default=3,
-                    help="paced clients that fire a genuinely-slow call then disconnect")
-    ap.add_argument("--slowfire-pace", type=float, default=2.0)
-    ap.add_argument("--halfopen", type=int, default=6)
-    ap.add_argument("--pipeline", type=int, default=6)
-    ap.add_argument("--garbage", type=int, default=4)
-    ap.add_argument("--slowread", type=int, default=2)
-    ap.add_argument("--longpoll", type=int, default=6)
-    # burst-size must exceed (rpcthreads + rpcworkqueue) to overflow the queue even
-    # when calls are fast; with -rpcthreads=8 -rpcworkqueue=4 that threshold is 12.
-    ap.add_argument("--burst-threads", type=int, default=1)
-    ap.add_argument("--burst-size", type=int, default=24)
-    ap.add_argument("--burst-every", type=float, default=1.5)
-    ap.add_argument("--mine", action="store_true", help="run a miner thread")
-    ap.add_argument("--mine-interval", type=float, default=4.0)
-    ap.add_argument("--pid", type=int, default=0, help="node pid, for /proc fd + liveness")
-    ap.add_argument("--logfile", default="")
-    args = ap.parse_args()
-
-    CFG.host = args.host; CFG.port = args.port
-    CFG.auth = auth_header(args.user, args.password); CFG.ctimeout = args.ctimeout
-    if args.logfile:
-        LOGF[0] = open(args.logfile, "a")
-
-    log("hammer start host=%s port=%d duration=%ds pid=%s" %
-        (CFG.host, CFG.port, args.duration, args.pid or "-"))
-    bootstrap()
-
-    threads = []
-    def spawn(fn, n, *a):
-        for i in range(n):
-            t = threading.Thread(target=fn, args=(i,) + a, daemon=True)
-            t.start(); threads.append(t)
-
-    spawn(client_normal, args.normal)
-    spawn(client_impatient, args.impatient)
-    spawn(client_slowfire, args.slowfire, args.slowfire_pace)
-    spawn(client_half_open, args.halfopen)
-    spawn(client_pipeline, args.pipeline)
-    spawn(client_garbage, args.garbage)
-    spawn(client_slowread, args.slowread)
-    spawn(client_longpoll, args.longpoll)
-    for i in range(args.burst_threads):
-        t = threading.Thread(target=client_burst, args=(i, args.burst_size, args.burst_every), daemon=True)
-        t.start(); threads.append(t)
-    if args.mine:
-        t = threading.Thread(target=miner, args=(args.mine_interval,), daemon=True)
-        t.start(); threads.append(t)
-
-    rt = threading.Thread(target=reporter, args=(args.pid,), daemon=True)
-    rt.start()
-
-    end = time.monotonic() + args.duration
-    try:
-        while time.monotonic() < end and not STOP.is_set():
-            time.sleep(0.2)
-    except KeyboardInterrupt:
-        pass
-    STOP.set()
-    time.sleep(1.0)
-
-    log("=== TOTALS ===")
-    for k in KEYS:
-        if ST.total[k]:
-            log("  %-14s %d" % (k, ST.total[k]))
-    log("anomalies: %d" % len(ST.anom))
-    for t, kind, detail in ST.anom[:40]:
-        log("  %s %s %s" % (time.strftime("%H:%M:%S", time.localtime(t)), kind, detail))
-    if args.pid:
-        log("node alive at end: %s" % pid_alive(args.pid))
-
-if __name__ == "__main__":
-    main()

qa/fd-guard/repro/tools/rpcchurn.sh

@@ -1,29 +0,0 @@
-#!/usr/bin/env bash
-# RPC-driven P2P peer churn on the target (same mix as fdbug/p2p/rpcchurn.sh):
-# disconnectnode / addnode onetry / setban+clearbanned / kill -9 + restart of helper2.
-# usage: rpcchurn.sh <run dir> <seconds>
-source "$REPRO/common.sh"
-OUT=$1; DUR=${2:-300}
-DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
-end=$(( $(date +%s) + DUR )); i=0; restarts=0
-while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
-  i=$((i+1))
-  case $(( i % 5 )) in
-    0) clin "$DD_T" "$RPC_T" disconnectnode "127.0.0.1:$P2P_H1" >/dev/null 2>&1 ;;
-    1) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H1" onetry >/dev/null 2>&1 ;;
-    2) clin "$DD_T" "$RPC_T" setban "127.0.0.1" add 1 >/dev/null 2>&1; sleep 0.3
-       clin "$DD_T" "$RPC_T" clearbanned >/dev/null 2>&1
-       clin "$DD_H2" "$RPC_H2" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 ;;
-    3) pid=$(cat "$DD_H2/regtest/hushd.pid" 2>/dev/null)
-       if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
-         kill -9 "$pid" 2>/dev/null
-         for _ in $(seq 1 20); do kill -0 "$pid" 2>/dev/null || break; sleep 0.5; done
-         start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
-         waitrpc "$DD_H2" "$RPC_H2" 120 >/dev/null 2>&1
-         restarts=$((restarts+1))
-       fi ;;
-    4) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H2" onetry >/dev/null 2>&1 ;;
-  esac
-  sleep 0.7
-done
-echo "rpcchurn: $i cycles, $restarts helper2 restarts"

qa/fd-guard/repro/tools/scan_bdb.py

@@ -1,113 +0,0 @@
-#!/usr/bin/env python3
-"""
-scan_bdb.py -- find Berkeley DB pages written into files that must not contain them
-(blocks/blk*.dat, blocks/rev*.dat).  Stand-in for the production incident tool.
-
-BDB writes page P of a database at file offset P*pagesize through pwrite(fd, ...).  If
-fd has been closed and its number reused by a block file, the page lands in the block
-file at exactly P*pagesize.  So at every pagesize-aligned offset we decode the PAGE
-header (db_page.h, all little-endian on x86):
-    0  u32 lsn.file     4  u32 lsn.offset   8  u32 pgno   12 u32 prev_pgno
-   16  u32 next_pgno   20  u16 entries     22  u16 hf_offset
-   24  u8  level       25  u8  type
-and a metadata page (pgno 0 / any meta type) also carries magic at 12 and pagesize at 20.
-
-A hit needs ALL of:
-  * pgno == offset // pagesize                       (the self-locating property)
-  * type is a real BDB page type (1..16 except 0)
-  * level consistent with type (overflow/meta/hash: 0; btree leaf: 1; internal: 2..32)
-  * hf_offset <= pagesize; entries <= pagesize/2
-  * for overflow (7): entries (refcount) >= 1 and 0 < hf_offset (data length) <= pagesize-26
-Strength: "STRONG" if lsn == (0,1) (pages rewritten by lsn_reset -- the production
-signature) or a meta page with a valid magic, else "weak".
-Also reports any aligned BDB meta magic (btree 0x053162, hash 0x061561, queue 0x042253,
-heap 0x074582) even when pgno does not match.
-
-usage: scan_bdb.py [--pagesize 4096] [--all-offsets] files...
-exit 0 = clean, 2 = hits.
-"""
-import argparse, struct, sys
-
-TYPES = {1: "duplicate", 2: "hash_unsorted", 3: "ibtree", 4: "irecno", 5: "lbtree", 6: "lrecno",
-         7: "overflow", 8: "hashmeta", 9: "btreemeta", 10: "qammeta", 11: "qamdata", 12: "ldup",
-         13: "hash", 14: "heapmeta", 15: "heap", 16: "iheap"}
-MAGIC = {0x053162: "btree", 0x061561: "hash", 0x042253: "queue", 0x074582: "heap"}
-META = {8, 9, 10, 14}
-
-
-def level_ok(t, lvl):
-    if t in (5, 6, 12):           # leaf pages
-        return lvl == 1
-    if t in (3, 4):               # internal pages
-        return 2 <= lvl <= 32
-    return lvl == 0               # overflow / meta / hash / queue / heap
-
-
-def check(buf, off, ps):
-    if len(buf) < 26:
-        return None
-    lf, lo, pgno, prv, nxt, ent, hfo, lvl, typ = struct.unpack_from("<IIIIIHHBB", buf, 0)
-    if typ not in TYPES or pgno != off // ps or not level_ok(typ, lvl):
-        return None
-    if typ in META:
-        magic, ver, psz = struct.unpack_from("<III", buf, 12)
-        if magic not in MAGIC:
-            return None
-        return ("STRONG", f"{TYPES[typ]} magic={MAGIC[magic]} ver={ver} pagesize={psz} lsn=({lf},{lo})")
-    if hfo > ps or ent > ps // 2:
-        return None
-    if typ == 7 and not (ent >= 1 and 0 < hfo <= ps - 26):
-        return None
-    strength = "STRONG" if (lf, lo) == (0, 1) else "weak"
-    return (strength, f"{TYPES[typ]} lsn=({lf},{lo}) prev={prv} next={nxt} entries={ent} hf_offset={hfo} level={lvl}")
-
-
-def scan(path, ps):
-    hits, magic_only = [], []
-    with open(path, "rb") as f:
-        off = 0
-        while True:
-            buf = f.read(ps)
-            if not buf:
-                break
-            r = check(buf, off, ps)
-            if r:
-                hits.append((off, off // ps) + r)
-            elif len(buf) >= 16:
-                m = struct.unpack_from("<I", buf, 12)[0]
-                if m in MAGIC and buf[25] in META:
-                    magic_only.append((off, MAGIC[m]))
-            off += len(buf)
-    return hits, magic_only, off
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--pagesize", type=int, default=4096)
-    ap.add_argument("--quiet", action="store_true")
-    ap.add_argument("files", nargs="+")
-    a = ap.parse_args()
-    total = strong = 0
-    for p in a.files:
-        try:
-            hits, mo, size = scan(p, a.pagesize)
-        except OSError as e:
-            print(f"{p}: ERROR {e}")
-            continue
-        s = sum(1 for h in hits if h[2] == "STRONG")
-        total += len(hits); strong += s
-        if hits or mo:
-            print(f"{p}: {len(hits)} BDB page(s) ({s} strong), {len(mo)} stray meta magic, size={size}")
-            if not a.quiet:
-                for h in hits[:20]:
-                    print(f"    off={h[0]} pgno={h[1]} {h[2]} {h[3]}")
-                for m in mo[:5]:
-                    print(f"    meta-magic off={m[0]} {m[1]}")
-        else:
-            print(f"{p}: clean ({size} bytes)")
-    print(f"TOTAL BDB pages found: {total} ({strong} strong)")
-    return 2 if total else 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

qa/fd-guard/repro/tools/tally.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-"""tally.py <fdguard.log> -- attribute every fdguard report to (type, closer function, target class)
-and, for EBADF/NON-OWNER reports, to the code that made the previous close of that fd number.
-
-The point: show that the stray closes ORIGINATE in CNode code (net.cpp CloseSocketDisconnect /
-~CNode closing a number they already closed) and that libevent / BDB / CreateNodeFromAcceptedSocket /
-GetTransaction only appear as victims-turned-amplifiers downstream of that.
-"""
-import collections, re, sys
-
-ORIGIN = ("_ZN5CNode21CloseSocketDisconnectEv", "_ZN5CNodeD1Ev", "_ZN5CNodeD2Ev", "_ZN11CNetCleanupD2Ev")
-NAMES = {"_ZN5CNode21CloseSocketDisconnectEv": "CNode::CloseSocketDisconnect (net.cpp:572)",
-         "_ZN5CNodeD1Ev": "CNode::~CNode (net.cpp:2819)",
-         "_ZN11CNetCleanupD2Ev": "CNetCleanup::~CNetCleanup (net.cpp:2465, shutdown)",
-         "evhttp_connection_free": "libevent evhttp_connection_free (http.c:1244)",
-         "__os_closehandle": "BDB __os_closehandle",
-         "_Z28CreateNodeFromAcceptedSocketjbRK8CAddressS1_": "CreateNodeFromAcceptedSocket (net.cpp:1336)",
-         "_Z14GetTransactionRK7uint256R12CTransactionRS_b": "GetTransaction fclose (main.cpp)",
-         "epoll_nochangelist_add": "libevent epoll ADD", "epoll_nochangelist_del": "libevent epoll DEL/MOD"}
-
-
-def fn(frame_line):
-    m = re.search(r"\+0x[0-9a-f]+ (\S+?)\+0x", frame_line)
-    return m.group(1) if m else "?"
-
-
-def main(path):
-    reps, cur = [], None
-    for line in open(path, errors="replace"):
-        if line.startswith("=== FDGUARD #"):
-            cur = {"hdr": line.strip(), "lines": []}
-            reps.append(cur)
-        elif cur is not None:
-            if line.startswith("=== END"):
-                cur = None
-            else:
-                cur["lines"].append(line.rstrip("\n"))
-    tally = collections.Counter()
-    prev_tally = collections.Counter()
-    db_blk = []
-    for r in reps:
-        typ = r["hdr"].split()[3]
-        L = r["lines"]
-        cls = next((re.search(r"class=(\S+)", l).group(1) for l in L if " target=" in l and "class=" in l), "?")
-        try:
-            i = next(k for k, l in enumerate(L) if l.strip() == "closer stack:")
-            closer = fn(L[i + 1])
-        except StopIteration:
-            closer = "?"
-        prev = "-"
-        for k, l in enumerate(L):
-            if "previous successful close of this fd number" in l:
-                # first frame line after the target= line
-                for l2 in L[k + 1:k + 4]:
-                    if l2.strip().startswith("#0"):
-                        prev = fn(l2)
-                        break
-                break
-        tally[(typ, NAMES.get(closer, closer), cls)] += 1
-        if typ in ("EBADF-CLOSE", "NON-OWNER-CLOSE"):
-            prev_tally[NAMES.get(prev, prev)] += 1
-        if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk"):
-            tgt = next((l.split("target=")[1].split()[0] for l in L if " target=" in l), "?")
-            db_blk.append((r["hdr"].split()[2], r["hdr"].split()[4], NAMES.get(closer, closer), tgt.rsplit("/", 2)[-2:]))
-    print(f"{len(reps)} reports in {path}")
-    print("\n count  type              class  closer")
-    for (typ, closer, cls), n in tally.most_common():
-        print(f"{n:6d}  {typ:17s} {cls:6s} {closer}")
-    print("\nEBADF + NON-OWNER reports by the code that made the PREVIOUS close of the same number:")
-    for k, n in prev_tally.most_common():
-        print(f"{n:6d}  {k}")
-    o = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk") and c.startswith(("CNode", "CNetCleanup")))
-    a = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk")) - o
-    print(f"\nNON-OWNER closes of wallet/BDB-log/block files: {o + a}  (by CNode code directly: {o}; by an amplifier: {a})")
-    for d in db_blk[:40]:
-        print("   ", d[0], d[1], "closer=" + d[2], "target=" + "/".join(d[3]))
-
-
-if __name__ == "__main__":
-    main(sys.argv[1])

qa/fd-guard/repro/tools/tcpchurn.py

@@ -1,147 +0,0 @@
-#!/usr/bin/env python3
-"""
-Raw TCP / TLS client churn against the target node's P2P port.
-
-dragonxd P2P is TLS 1.3 only (wolfSSL), self-signed, no cert verification. The
-accept path is net.cpp CreateNodeFromAcceptedSocket -> tlsmanager.accept()
-(wolfSSL_accept), and teardown is CloseSocketDisconnect / ~CNode
-(wolfSSL_shutdown + wolfSSL_free + CloseSocket). This driver pokes the accept and
-TLS-teardown/error paths (SSL_read/SSL_write at net.cpp ~1005 and tlsmanager.cpp
-~660-680) by opening many short-lived connections that each misbehave:
-
-  rst      : connect, then abort with RST (SO_LINGER 0) before any TLS bytes
-  fin      : connect, then close cleanly (FIN) before any TLS bytes
-  garbage  : connect, send non-ClientHello bytes, then drop
-  stall    : connect, send a few bytes, hold mid-handshake, drop
-  tlsjunk  : complete a real TLS 1.3 handshake (verify off), send junk, drop
-  tlshalf  : start TLS, send a partial ClientHello record, drop
-
-Loopback only, against a port this harness started. --duration seconds.
-"""
-import argparse, random, socket, ssl, struct, sys, time, os
-
-def mk():
-    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    s.settimeout(3.0)
-    return s
-
-def rst_close(s):
-    try:
-        s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
-    except Exception:
-        pass
-    try: s.close()
-    except Exception: pass
-
-def do_rst(host, port):
-    s = mk()
-    try:
-        s.connect((host, port)); rst_close(s)
-    except Exception:
-        rst_close(s)
-
-def do_fin(host, port):
-    s = mk()
-    try:
-        s.connect((host, port)); s.close()
-    except Exception:
-        try: s.close()
-        except Exception: pass
-
-def do_garbage(host, port):
-    s = mk()
-    try:
-        s.connect((host, port))
-        s.sendall(os.urandom(random.randint(16, 400)))
-        time.sleep(random.uniform(0.0, 0.3))
-    except Exception:
-        pass
-    rst_close(s)
-
-def do_stall(host, port):
-    s = mk()
-    try:
-        s.connect((host, port))
-        # a couple of bytes that look like the start of a TLS record, then wait
-        s.sendall(b"\x16\x03\x01")
-        time.sleep(random.uniform(0.5, 2.0))
-    except Exception:
-        pass
-    rst_close(s)
-
-def _ctx():
-    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
-    ctx.check_hostname = False
-    ctx.verify_mode = ssl.CERT_NONE
-    try:
-        ctx.minimum_version = ssl.TLSVersion.TLSv1_3
-    except Exception:
-        pass
-    return ctx
-
-def do_tlsjunk(host, port):
-    raw = mk()
-    try:
-        raw.connect((host, port))
-        tls = _ctx().wrap_socket(raw, server_hostname=None)
-        # handshake completed; the node now expects a P2P version message.
-        # send junk inside the TLS channel, then tear down.
-        try:
-            tls.sendall(os.urandom(random.randint(8, 128)))
-        except Exception:
-            pass
-        time.sleep(random.uniform(0.0, 0.4))
-        try:
-            tls.close()
-        except Exception:
-            rst_close(raw)
-    except Exception:
-        rst_close(raw)
-
-def do_tlshalf(host, port):
-    s = mk()
-    try:
-        s.connect((host, port))
-        # a well-formed TLS record header claiming a long handshake, but we send
-        # only part of it, then drop -> server sits in wolfSSL_accept WANT_READ.
-        body = os.urandom(40)
-        rec = b"\x16\x03\x03" + struct.pack(">H", 512) + body
-        s.sendall(rec)
-        time.sleep(random.uniform(0.3, 1.5))
-    except Exception:
-        pass
-    rst_close(s)
-
-HANDLERS = [do_rst, do_fin, do_garbage, do_stall, do_tlsjunk, do_tlshalf]
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--host", default="127.0.0.1")
-    ap.add_argument("--port", type=int, required=True)
-    ap.add_argument("--duration", type=float, default=300.0)
-    ap.add_argument("--concurrency", type=int, default=6)
-    ap.add_argument("--rate", type=float, default=0.03, help="sleep between bursts")
-    args = ap.parse_args()
-
-    end = time.time() + args.duration
-    counts = {h.__name__: 0 for h in HANDLERS}
-    import threading
-    lock = threading.Lock()
-
-    def worker():
-        while time.time() < end:
-            h = random.choice(HANDLERS)
-            h(args.host, args.port)
-            with lock:
-                counts[h.__name__] += 1
-            time.sleep(random.uniform(0.0, args.rate))
-
-    ts = [threading.Thread(target=worker, daemon=True) for _ in range(args.concurrency)]
-    for t in ts: t.start()
-    for t in ts: t.join()
-
-    total = sum(counts.values())
-    print(f"tcpchurn: {total} connections in {args.duration:.0f}s -> {counts}", flush=True)
-
-if __name__ == "__main__":
-    main()

qa/fd-guard/shim/build.sh

@@ -1,15 +0,0 @@
-#!/bin/sh
-# Build the fdguard LD_PRELOAD shim and its self-test program.
-#   ./build.sh            -> libfdguard.so, fdguard_test
-set -e
-cd "$(dirname "$0")"
-CC=${CC:-gcc}
-$CC -O2 -g -fPIC -shared -Wall -Wextra -Wno-unused-parameter -fvisibility=default \
-    -o libfdguard.so.tmp fdguard.c -ldl -lpthread
-# rename, never overwrite in place: a running process may have the old one mapped
-mv -f libfdguard.so.tmp libfdguard.so
-# the test program keeps its symbols (no -rdynamic on purpose: like dragonxd,
-# owner detection must work from .symtab alone) and frame pointers for clarity
-$CC -O1 -g -fno-omit-frame-pointer -fno-inline -Wall -o fdguard_test.tmp fdguard_test.c -lpthread
-mv -f fdguard_test.tmp fdguard_test
-echo "built: $(pwd)/libfdguard.so $(pwd)/fdguard_test"

qa/fd-guard/shim/fdguard_test.c

@@ -1,206 +0,0 @@
-/*
- * fdguard_test.c -- exercises libfdguard.so.
- *
- * The function names imitate the real owners inside dragonxd so the shim's
- * built-in domain patterns apply unchanged:
- *   __os_openhandle / __os_closehandle   Berkeley DB os layer   (domain bdb)
- *   OpenDiskFile / FlushBlockFile        block-file code         (domain blk)
- *   ConnectSocketDirectly / CloseSocket  P2P sockets             (domain p2p)
- *   foreign_stale_close / foreign_*      somebody who holds a stale number
- *
- * usage: fdguard_test <dir> <case>   (see main() for the list)
- */
-#define _GNU_SOURCE
-#include <errno.h>
-#include <fcntl.h>
-#include <pthread.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/epoll.h>
-#include <sys/socket.h>
-#include <sys/syscall.h>
-#include <time.h>
-#include <unistd.h>
-
-#define NI __attribute__((noinline))
-static char D[512];
-static char *P(const char *rel) { static char b[4][700]; static int i; i = (i + 1) & 3; snprintf(b[i], sizeof b[i], "%s/%s", D, rel); return b[i]; }
-static void say(const char *m) { fprintf(stderr, "[test] %s\n", m); }
-
-/* --- "Berkeley DB" --- */
-NI int __os_openhandle(const char *path) { return open(path, O_RDWR | O_CREAT, 0600); }
-NI int __os_closehandle(int fd) { return close(fd); }
-/* --- "block files" --- */
-NI FILE *OpenDiskFile(const char *path) { FILE *f = fopen(path, "rb+"); if (!f) f = fopen(path, "wb+"); return f; }
-NI int FlushBlockFile(FILE *f) { fflush(f); return fclose(f); }
-/* --- "P2P" --- */
-NI int ConnectSocketDirectly(void) { return socket(AF_INET, SOCK_STREAM, 0); }
-NI int CloseSocket(int *s) { int r = close(*s); if (r) fprintf(stderr, "[test] Socket close failed: %d. Error: %s\n", *s, strerror(errno)); return r; }
-/* --- foreigners --- */
-NI int foreign_stale_close(int fd) { return close(fd); }
-NI int foreign_stale_fclose(FILE *f) { return fclose(f); }
-NI int foreign_dup2(int fd) { int dn = open("/dev/null", O_RDWR); int r = dup2(dn, fd); close(dn); return r; }
-NI int foreign_close_range(int fd) { return close_range((unsigned)fd, (unsigned)fd, 0); }
-NI void foreign_closefrom(int fd) { closefrom(fd); }
-/* --- plain code that opens and closes in the same function --- */
-NI int copy_like(const char *src) { int fd = open(src, O_RDONLY); char b[16]; if (fd >= 0) { (void)!read(fd, b, sizeof b); close(fd); } return fd; }
-NI int plain_open(const char *p) { return open(p, O_RDWR | O_CREAT, 0600); }
-NI int plain_close(int fd) { return close(fd); }
-
-static int fd_valid(int fd) { return fcntl(fd, F_GETFD) != -1; }
-
-static int c_double(void)
-{
-    int fd = plain_open(P("plain.txt"));
-    plain_close(fd);
-    int r = plain_close(fd);  /* EBADF */
-    return r == -1 && errno == EBADF ? 0 : 1;
-}
-static int c_dbowner(void) { int fd = __os_openhandle(P("regtest/wallet.dat")); int r = __os_closehandle(fd); return r; }
-static int c_dbforeign(void)
-{
-    int fd = __os_openhandle(P("regtest/wallet.dat"));
-    foreign_stale_close(fd);              /* HIT */
-    int r = __os_closehandle(fd);         /* BDB's own close then fails: EBADF (unless deny) */
-    (void)r;
-    return 0;
-}
-static int c_deny(void)
-{
-    int fd = __os_openhandle(P("regtest/wallet.dat"));
-    foreign_stale_close(fd);              /* HIT + DENIED */
-    if (!fd_valid(fd)) { say("deny FAILED: fd closed"); return 1; }
-    if (write(fd, "x", 1) != 1) { say("deny FAILED: write"); return 1; }
-    return __os_closehandle(fd);
-}
-static int c_blkowner(void) { FILE *f = OpenDiskFile(P("regtest/blocks/blk00000.dat")); fputs("blk", f); return FlushBlockFile(f); }
-static int c_blkraw(void)
-{
-    FILE *f = OpenDiskFile(P("regtest/blocks/blk00001.dat"));
-    foreign_stale_close(fileno(f));       /* HIT: raw close under a live FILE */
-    FlushBlockFile(f);                     /* EBADF from fclose */
-    return 0;
-}
-static int c_stalefile(void)
-{
-    FILE *f = OpenDiskFile(P("regtest/blocks/blk00002.dat"));
-    int n = fileno(f);
-    foreign_stale_close(n);               /* HIT #1 */
-    int w = __os_openhandle(P("regtest/wallet.dat"));
-    if (w != n) fprintf(stderr, "[test] note: wallet got fd %d, FILE had %d\n", w, n);
-    FlushBlockFile(f);                    /* HIT #2: fclose of a stale FILE whose number is now wallet.dat */
-    if (w == n) __os_closehandle(w);      /* EBADF: BDB lost its fd */
-    return 0;
-}
-static int c_reuse(void)
-{
-    int s = ConnectSocketDirectly();
-    int keep = s;
-    CloseSocket(&s);                      /* legit */
-    int w = __os_openhandle(P("regtest/wallet.dat"));   /* reuses the number */
-    if (w != keep) fprintf(stderr, "[test] note: wallet got fd %d, socket was %d\n", w, keep);
-    s = keep;
-    CloseSocket(&s);                      /* HIT: stale socket number now = wallet.dat */
-    __os_closehandle(w);                  /* EBADF */
-    return 0;
-}
-static int c_dup2(void) { int w = __os_openhandle(P("regtest/wallet.dat")); foreign_dup2(w); __os_closehandle(w); return 0; }
-static int c_range(void)
-{
-    int w = __os_openhandle(P("regtest/wallet.dat"));
-    foreign_close_range(w);                /* HIT */
-    w = __os_openhandle(P("regtest/database/log.0000000001"));
-    int hi = fcntl(w, F_DUPFD, 600);       /* move it high so closefrom only hits it */
-    close(w);
-    foreign_closefrom(hi);                /* HIT (domain mismatch: opened by test main via fcntl dup? -> fallback db) */
-    return 0;
-}
-static int c_owner_misc(void)
-{
-    /* nothing here may be flagged */
-    copy_like(P("regtest/wallet.dat"));           /* common-caller */
-    int fd = __os_openhandle(P("regtest/blocks/blk00003.dat"));
-    FILE *f = fdopen(fd, "rb+");                   /* ownership moves to FILE */
-    FlushBlockFile(f);
-    int s = ConnectSocketDirectly(); CloseSocket(&s);
-    fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR);  /* untracked open */
-    __os_closehandle(fd);                          /* fallback:bdb */
-    return 0;
-}
-static int c_fallback_hit(void)
-{
-    int fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR);  /* untracked open */
-    foreign_stale_close(fd);                       /* HIT via fallback rule */
-    return 0;
-}
-static int c_epoll(void)
-{
-    int ep = epoll_create1(0);
-    int fd = plain_open(P("plain2.txt"));
-    struct epoll_event ev = { .events = EPOLLIN };
-    int r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev);  /* EPERM: regular file */
-    close(fd);
-    r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev);      /* EBADF */
-    close(ep);
-    return r == -1 ? 0 : 1;
-}
-static int c_selfclose(void)
-{
-    /* close every fd from 3 to 1100 by number, like a buggy daemonizer */
-    for (int i = 3; i < 1100; i++) close(i);
-    return 0;
-}
-
-/* concurrency + micro-benchmark: open/close churn in many threads */
-static int nthr = 8, iters = 20000;
-static void *worker(void *arg)
-{
-    long id = (long)arg;
-    char p[700]; snprintf(p, sizeof p, "%s/thr%ld.txt", D, id);
-    for (int i = 0; i < iters; i++) {
-        int s = ConnectSocketDirectly(); CloseSocket(&s);
-        int fd = plain_open(p); plain_close(fd);
-        FILE *f = fopen(p, "r"); if (f) fclose(f);
-        if ((i & 255) == 0) { int w = __os_openhandle(P("regtest/wallet.dat")); __os_closehandle(w); }
-    }
-    return NULL;
-}
-static int c_threads(void)
-{
-    const char *e = getenv("T_ITERS"); if (e) iters = atoi(e);
-    pthread_t t[64];
-    struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
-    for (long i = 0; i < nthr; i++) pthread_create(&t[i], NULL, worker, (void *)i);
-    for (int i = 0; i < nthr; i++) pthread_join(t[i], NULL);
-    clock_gettime(CLOCK_MONOTONIC, &b);
-    double ns = (b.tv_sec - a.tv_sec) * 1e9 + (b.tv_nsec - a.tv_nsec);
-    long ops = (long)nthr * iters * 3;
-    printf("threads=%d iters=%d ops=%ld (socket/open/fopen + close each) wall=%.3fs  %.0f ns/op-pair/thread\n",
-           nthr, iters, ops, ns / 1e9, ns * nthr / ops);
-    return 0;
-}
-
-int main(int argc, char **argv)
-{
-    if (argc < 3) { fprintf(stderr, "usage: %s <dir> <case>\n", argv[0]); return 2; }
-    snprintf(D, sizeof D, "%s", argv[1]);
-    const char *c = argv[2];
-    if (!strcmp(c, "double")) return c_double();
-    if (!strcmp(c, "dbowner")) return c_dbowner();
-    if (!strcmp(c, "dbforeign")) return c_dbforeign();
-    if (!strcmp(c, "deny")) return c_deny();
-    if (!strcmp(c, "blkowner")) return c_blkowner();
-    if (!strcmp(c, "blkraw")) return c_blkraw();
-    if (!strcmp(c, "stalefile")) return c_stalefile();
-    if (!strcmp(c, "reuse")) return c_reuse();
-    if (!strcmp(c, "dup2")) return c_dup2();
-    if (!strcmp(c, "range")) return c_range();
-    if (!strcmp(c, "ownermisc")) return c_owner_misc();
-    if (!strcmp(c, "fallback")) return c_fallback_hit();
-    if (!strcmp(c, "epoll")) return c_epoll();
-    if (!strcmp(c, "selfclose")) return c_selfclose();
-    if (!strcmp(c, "threads")) return c_threads();
-    fprintf(stderr, "unknown case %s\n", c);
-    return 2;
-}

qa/fd-guard/shim/run_tests.sh

@@ -1,97 +0,0 @@
-#!/bin/bash
-# Self-test for libfdguard.so.  Every case lists the report types it must
-# produce (exactly, in order) -- an empty list means "no report at all".
-#   ./run_tests.sh            (builds first)
-set -u
-cd "$(dirname "$0")"
-./build.sh >/dev/null || { echo "build failed"; exit 1; }
-SO=$PWD/libfdguard.so
-T=$PWD/t
-rm -rf "$T"; mkdir -p "$T/regtest/blocks" "$T/regtest/database"
-pass=0; fail=0
-
-run() {  # run <case> [ENV=VAL ...]  -> sets RC, LOG
-  local c=$1; shift
-  LOG=$T/$c${SUFFIX:-}.log; rm -f "$LOG"
-  env FDGUARD_EXE=fdguard_test FDGUARD_LOG="$LOG" "$@" LD_PRELOAD="$SO" ./fdguard_test "$T" "$c" >"$T/$c${SUFFIX:-}.out" 2>&1
-  RC=$?
-}
-types() { grep '^=== FDGUARD' "$LOG" 2>/dev/null | awk '{print $4}' | tr '\n' ' ' | sed 's/ $//'; }
-reasons() { grep -o 'reason=[^ ]*' "$LOG" 2>/dev/null | sed 's/reason=//' | tr '\n' ' ' | sed 's/ $//'; }
-check() {  # check <name> <cond-description> <actual> <expected>
-  if [ "$3" == "$4" ]; then pass=$((pass+1)); printf 'PASS %-12s %s\n' "$1" "$2";
-  else fail=$((fail+1)); printf 'FAIL %-12s %s\n      got:  [%s]\n      want: [%s]\n' "$1" "$2" "$3" "$4"; fi
-}
-
-# case | expected report types | expected reasons
-while IFS='|' read -r c want wantr; do
-  [ -z "$c" ] && continue
-  run "$c"
-  check "$c" "reports" "$(types)" "$want"
-  check "$c" "reasons" "$(reasons)" "$wantr"
-  check "$c" "exit 0" "$RC" "0"
-done <<'EOF'
-double|EBADF-CLOSE|
-dbowner||
-dbforeign|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
-blkowner||
-blkraw|NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE
-stalefile|NON-OWNER-CLOSE NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE stdio:STALE-FILE
-reuse|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
-dup2|NON-OWNER-CLOSE|DOMAIN-MISMATCH
-range|NON-OWNER-CLOSE NON-OWNER-CLOSE NON-OWNER-CLOSE|DOMAIN-MISMATCH DOMAIN-MISMATCH fallback:DB-FILE-CLOSED-OUTSIDE-BDB
-ownermisc||
-fallback|NON-OWNER-CLOSE|fallback:DB-FILE-CLOSED-OUTSIDE-BDB
-epoll|EPOLL-CTL-EPERM EPOLL-CTL-EBADF|epoll_ctl(ADD)-failed epoll_ctl(ADD)-failed
-EOF
-
-# the previous close of the reused number must be the first CloseSocket (proves the double close)
-run reuse
-check reuse "history shows 1st CloseSocket on a socket" \
-  "$(grep -A2 'previous successful close' "$LOG" | head -3 | grep -c -e 'class=sock dom=p2p' -e 'CloseSocket+')" "2"
-# the stale FILE report must name wallet.dat as the victim and OpenDiskFile as the FILE's opener
-run stalefile
-check stalefile "victim=wallet.dat, FILE opened in OpenDiskFile" \
-  "$(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -m1 -c 'target=.*wallet.dat class=db') $(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -A3 'which had been opened by: fopen' | grep -c 'OpenDiskFile+')" "1 1"
-
-# the log fd survives a close-everything loop and the attempt is reported
-run selfclose
-check selfclose "log-fd close refused + reported" "$(grep -c 'CLOSE-OF-FDGUARD-LOG-FD' "$LOG")" "1"
-check selfclose "log still writable afterwards (final stats)" "$(grep -c '^S .* final' "$LOG")" "1"
-
-# abort mode: first non-owner close -> SIGABRT (rc 134), report already written
-SUFFIX=_abort run dbforeign FDGUARD_ABORT=1
-check abort "rc=134 (SIGABRT)" "$RC" "134"
-check abort "report written before abort" "$(types)" "NON-OWNER-CLOSE"
-# abort=2 also aborts on EBADF
-SUFFIX=_abort2 run double FDGUARD_ABORT=2
-check abort2 "rc=134 on EBADF" "$RC" "134"
-
-# deny mode: wallet fd survives the foreign close; no EBADF afterwards
-SUFFIX=_deny run deny FDGUARD_DENY=1
-check deny "exit 0 (fd still valid + writable)" "$RC" "0"
-check deny "reported as DENIED, no EBADF" "$(types) $(grep -c 'ACTION=DENIED' "$LOG")" "NON-OWNER-CLOSE 1"
-
-# extra classes: guard=all with the owner cases must still be silent
-SUFFIX=_all run ownermisc FDGUARD_GUARD=all
-check guard-all "ownermisc silent with FDGUARD_GUARD=all" "$(types)" ""
-
-# trace-all writes one T line per open/close
-SUFFIX=_trace run dbowner FDGUARD_TRACE_ALL=1
-check trace "T lines for open+close" "$(grep -c '^T ' "$LOG")" "2"
-
-# passive in non-matching processes
-LOG=$T/passive.log; env FDGUARD_EXE=dragonxd FDGUARD_LOG="$LOG" LD_PRELOAD="$SO" ./fdguard_test "$T" dbforeign >/dev/null 2>&1
-check passive "no log when exe does not match" "$( [ -e "$LOG" ] && echo exists || echo none)" "none"
-
-# concurrency: 8 threads x 5000 x (socket, open, fopen) churn, guard=all -> no reports
-SUFFIX=_thr run threads T_ITERS=5000 FDGUARD_GUARD=all
-check threads "no reports under 8-thread churn (guard=all)" "$(types)" ""
-check threads "stats: HITS=0 EBADF=0" "$(grep '^S .*final' "$LOG" | grep -o 'HITS=[0-9]* EBADF=[0-9]*')" "HITS=0 EBADF=0"
-
-# symbolizer resolves file:line for the test binary
-check symbolize "addr2line resolved CloseSocket to fdguard_test.c" \
-  "$(./symbolize.py "$T/reuse.log" | grep -m1 -o 'in CloseSocket at .*fdguard_test.c:[0-9]*' | sed 's/.*fdguard_test.c:[0-9]*/ok/')" "ok"
-
-echo "== $pass passed, $fail failed"
-[ $fail -eq 0 ]

qa/fd-guard/shim/symbolize.py

@@ -1,212 +0,0 @@
-#!/usr/bin/env python3
-"""
-symbolize.py -- turn fdguard log frames into function + file:line.
-
-Every stack frame in an fdguard report looks like
-    #3 0x5b96b433c991 /path/to/module+0x1991 c_dbforeign+0x2d [bdb]
-where "+0x1991" is the ELF virtual address inside that module (runtime address
-minus the module's load bias, so PIE executables and shared libraries need no
-further adjustment; for a non-PIE executable it equals the absolute address).
-Frames are return addresses, so we ask addr2line about vaddr-1 (the call
-instruction), with -f -C -i (function, demangle, inline chain).
-
-backtrace_symbols_fd lines "module(+0x1991)[0x...]" are symbolized too.
-
-usage:
-  symbolize.py [options] LOG [LOG...]          annotated log on stdout
-  symbolize.py --summary LOG                    one line per distinct report
-options:
-  --map OLD=NEW      read module OLD from path NEW instead (repeatable),
-                     e.g. --map /path/to/deployed/dragonxd=./dragonxd.unstripped
-  --addr2line PATH   addr2line binary (default: addr2line)
-  --only-reports     drop T/S lines (trace + stats) from the output
-"""
-import os
-import re
-import subprocess
-import sys
-from collections import defaultdict, OrderedDict
-
-FRAME_RE = re.compile(r'^(\s*#\d+ 0x[0-9a-f]+ )(\S+)\+0x([0-9a-f]+)(.*)$')
-BSFD_RE = re.compile(r'^(\S+)\(\+0x([0-9a-f]+)\)\[0x[0-9a-f]+\]\s*$')
-HDR_RE = re.compile(r'^=== FDGUARD #(\d+) (\S+)')
-
-
-def parse_args(argv):
-    opts = {'map': {}, 'addr2line': 'addr2line', 'summary': False, 'only': False, 'logs': []}
-    it = iter(argv)
-    for a in it:
-        if a == '--map':
-            old, new = next(it).split('=', 1)
-            opts['map'][old] = new
-        elif a.startswith('--map='):
-            old, new = a[6:].split('=', 1)
-            opts['map'][old] = new
-        elif a == '--addr2line':
-            opts['addr2line'] = next(it)
-        elif a == '--summary':
-            opts['summary'] = True
-        elif a == '--only-reports':
-            opts['only'] = True
-        elif a in ('-h', '--help'):
-            print(__doc__)
-            sys.exit(0)
-        else:
-            opts['logs'].append(a)
-    if not opts['logs']:
-        opts['logs'] = ['-']
-    return opts
-
-
-def read_lines(paths):
-    out = []
-    for p in paths:
-        f = sys.stdin if p == '-' else open(p, errors='replace')
-        out.extend(f.read().splitlines())
-        if f is not sys.stdin:
-            f.close()
-    return out
-
-
-def resolve_module(mod, opts, known):
-    if mod in opts['map']:
-        return opts['map'][mod]
-    if os.path.exists(mod):
-        return mod
-    base = os.path.basename(mod)
-    for k in known:
-        if os.path.basename(k) == base and os.path.exists(opts['map'].get(k, k)):
-            return opts['map'].get(k, k)
-    return None
-
-
-def run_addr2line(binary, addrs, a2l):
-    """addrs: list of int vaddrs (already -1'd). returns {addr: [(func, loc), ...]}"""
-    res = {}
-    if not addrs:
-        return res
-    uniq = sorted(set(addrs))
-    try:
-        p = subprocess.run([a2l, '-a', '-f', '-C', '-i', '-e', binary] + ['0x%x' % a for a in uniq],
-                           stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, check=False)
-    except FileNotFoundError:
-        return res
-    cur = None
-    lines = p.stdout.splitlines()
-    i = 0
-    while i < len(lines):
-        ln = lines[i]
-        if re.match(r'^0x[0-9a-f]+$', ln):
-            cur = int(ln, 16)
-            res[cur] = []
-            i += 1
-            continue
-        func = ln
-        loc = lines[i + 1] if i + 1 < len(lines) else '??:0'
-        if cur is not None:
-            res[cur].append((func, loc))
-        i += 2
-    return res
-
-
-def main():
-    opts = parse_args(sys.argv[1:])
-    lines = read_lines(opts['logs'])
-
-    if opts['summary']:
-        summary(lines)
-        return
-
-    # pass 1: collect modules + addresses
-    want = defaultdict(set)
-    known = OrderedDict()
-    for ln in lines:
-        m = FRAME_RE.match(ln)
-        if m:
-            known[m.group(2)] = 1
-            want[m.group(2)].add(max(int(m.group(3), 16) - 1, 0))
-    for ln in lines:
-        m = BSFD_RE.match(ln)
-        if m:
-            want[m.group(1)].add(max(int(m.group(2), 16) - 1, 0))
-
-    table = {}
-    for mod, addrs in want.items():
-        path = resolve_module(mod, opts, known)
-        if not path:
-            continue
-        table[mod] = run_addr2line(path, list(addrs), opts['addr2line'])
-
-    def fmt(mod, vaddr):
-        r = table.get(mod, {}).get(vaddr)
-        if not r:
-            return []
-        out = []
-        for j, (func, loc) in enumerate(r):
-            if func == '??' and loc.startswith('??'):
-                continue
-            # addr2line -i lists the innermost (inlined) frame first
-            tag = 'inlined:' if j < len(r) - 1 else 'in'
-            out.append('%s %s at %s' % (tag, func, loc))
-        return out
-
-    w = sys.stdout.write
-    for ln in lines:
-        if opts['only'] and (ln.startswith('T ') or ln.startswith('S ')):
-            continue
-        w(ln + '\n')
-        m = FRAME_RE.match(ln)
-        if m:
-            v = max(int(m.group(3), 16) - 1, 0)
-            for s in fmt(m.group(2), v):
-                w('         -> ' + s + '\n')
-            continue
-        m = BSFD_RE.match(ln)
-        if m:
-            v = max(int(m.group(2), 16) - 1, 0)
-            for s in fmt(m.group(1), v):
-                w('         -> ' + s + '\n')
-
-
-def summary(lines):
-    """one line per (type, reason, class, first exe closer frame)"""
-    groups = OrderedDict()
-    cur = None
-    in_closer = False
-    for ln in lines:
-        h = HDR_RE.match(ln)
-        if h:
-            cur = {'type': h.group(2), 'reason': '', 'cls': '', 'top': '', 'call': ''}
-            in_closer = False
-            continue
-        if cur is None:
-            continue
-        s = ln.strip()
-        if s.startswith('call:'):
-            cur['call'] = s.split('(')[0][6:]
-        elif s.startswith('target=') and ' class=' in s and not cur['cls']:
-            cur['cls'] = s.rsplit('class=', 1)[-1]
-        elif s.startswith('verdict='):
-            cur['reason'] = s.split('reason=', 1)[-1].split(' ')[0]
-        elif s == 'closer stack:':
-            in_closer = True
-        elif in_closer and FRAME_RE.match(ln):
-            parts = s.split(' ')
-            if len(parts) >= 4 and not cur['top'] and '/libc.so' not in parts[2] and '/libstdc++' not in parts[2]:
-                cur['top'] = parts[3]
-        elif in_closer and not s.startswith('#'):
-            in_closer = False
-        if s.startswith('=== END'):
-            k = (cur['type'], cur['call'], cur['reason'], cur['cls'], cur['top'])
-            groups[k] = groups.get(k, 0) + 1
-            cur = None
-    print('%6s  %-22s %-12s %-6s %-40s %s' % ('count', 'type', 'call', 'class', 'reason', 'first closer frame'))
-    for (t, c, r, cl, top), n in groups.items():
-        print('%6d  %-22s %-12s %-6s %-40s %s' % (n, t[:22], c, cl, r[:40], top))
-    stats = [ln for ln in lines if ln.startswith('S ')]
-    if stats:
-        print('\nlast stats line:\n  ' + stats[-1])
-
-
-if __name__ == '__main__':
-    main()

src/Makefile.gtest.include

@@ -14,13 +14,7 @@ hush_gtest_SOURCES = \
 	gtest/test_randomx_preverify.cpp \
 	gtest/test_hdtransparent.cpp \
 	gtest/test_mnemonic_compat.cpp \
-	gtest/test_stratum_jobid.cpp \
+	gtest/test_stratum_jobid.cpp
-	gtest/test_wallet_notedata.cpp \
-	gtest/test_net_closesocket.cpp \
-	gtest/test_fd_defense.cpp \
-	gtest/test_wallet_dbenv.cpp \
-	gtest/test_wallet_notecache.cpp \
-	gtest/test_wallet_history.cpp
 
 hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES)
 hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)

src/gtest/test_fd_defense.cpp

@@ -1,83 +0,0 @@
-// Copyright (c) 2016-2024 The Hush developers
-// Distributed under the GPLv3 software license, see the accompanying
-// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
-
-#include <gtest/gtest.h>
-
-#include <atomic>
-#include <errno.h>
-#include <stdexcept>
-#include <stdio.h>
-#include <unistd.h>
-
-#include <boost/filesystem.hpp>
-#include <boost/thread.hpp>
-
-#include "chainparams.h"
-#include "init.h"
-#include "main.h"
-#include "noui.h"
-#include "util.h"
-#include "validationinterface.h"
-
-extern std::atomic<bool> fRequestShutdown;
-
-// A block file opened for reading must not be writable through its fd: a stray write through a reused fd number
-// (the wolfSSL CloseSocket macro let the wallet's BDB fd be closed behind its back) used to land in it.
-TEST(FdDefense, ReadOnlyBlockFileIsNotWritable) {
-    SelectParams(CBaseChainParams::MAIN);
-    const boost::filesystem::path dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("fddefense-%%%%%%%%");
-    const bool hadDatadir = mapArgs.count("-datadir") > 0;
-    const std::string oldDatadir = hadDatadir ? mapArgs["-datadir"] : "";
-    boost::filesystem::create_directories(dir);
-    mapArgs["-datadir"] = dir.string();
-    ClearDatadirCache();
-
-    const CDiskBlockPos pos(0, 0);
-    const boost::filesystem::path blk = GetBlockPosFilename(pos, "blk");
-    boost::filesystem::create_directories(blk.parent_path());
-    {
-        FILE* f = fopen(blk.string().c_str(), "wb");
-        ASSERT_TRUE(f != nullptr);
-        fputs("block bytes", f);
-        fclose(f);
-    }
-
-    FILE* ro = OpenBlockFile(pos, true);
-    ASSERT_TRUE(ro != nullptr);
-    char c = 0;
-    EXPECT_EQ(1u, fread(&c, 1, 1, ro)) << "still readable";
-    errno = 0;
-    EXPECT_EQ(-1, pwrite(fileno(ro), "X", 1, 0)) << "a write through a read-only block file's fd must fail";
-    EXPECT_EQ(EBADF, errno);
-    fclose(ro);
-
-    FILE* rw = OpenBlockFile(pos, false);
-    ASSERT_TRUE(rw != nullptr);
-    EXPECT_EQ(1, pwrite(fileno(rw), "b", 1, 0)) << "the writable open still writes";
-    fclose(rw);
-
-    if (hadDatadir) mapArgs["-datadir"] = oldDatadir; else mapArgs.erase("-datadir");
-    ClearDatadirCache();
-    boost::filesystem::remove_all(dir);
-}
-
-// A wallet notification that throws (a wallet DB failure) requests a clean shutdown instead of escaping the
-// notifier thread into std::terminate; a successful one doesn't; thread interruption still propagates.
-TEST(FdDefense, WalletNotifyFailureShutsDownCleanly) {
-    noui_connect();   // the daemon connects these UI signals at startup; the error path shows a message box
-    fRequestShutdown = false;
-    EXPECT_TRUE(NotifyWalletsOrShutdown([] {}));
-    EXPECT_FALSE(ShutdownRequested());
-
-    EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw std::runtime_error("CWalletDB::ListAccountCreditDebit(): error scanning DB"); }));
-    EXPECT_TRUE(ShutdownRequested());
-    fRequestShutdown = false;
-
-    EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw 42; }));
-    EXPECT_TRUE(ShutdownRequested());
-    fRequestShutdown = false;
-
-    EXPECT_THROW(NotifyWalletsOrShutdown([] { throw boost::thread_interrupted(); }), boost::thread_interrupted);
-    EXPECT_FALSE(ShutdownRequested());
-}

src/gtest/test_net_closesocket.cpp

@@ -1,50 +0,0 @@
-// Copyright (c) 2016-2024 The Hush developers
-// Distributed under the GPLv3 software license, see the accompanying
-// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
-
-#include <gtest/gtest.h>
-
-#include <fcntl.h>
-#include <sys/socket.h>
-#include <unistd.h>
-
-#include "net.h"
-#include "util/sock.h"
-
-// Regression test for the wolfSSL CloseSocket macro (see net.h): closing a CNode's socket must invalidate
-// hSocket, so neither a repeated CloseSocketDisconnect nor ~CNode can close that fd number again once another
-// owner (the wallet's BDB files, a block file, an RPC socket) has been handed it.
-TEST(NetCloseSocket, ClosedNodeNeverClosesAReusedFdNumber) {
-    int pair[2];
-    ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
-    const int number = pair[0];
-
-    CNode* node = new CNode(number, CAddress(), "", true);   // inbound: sends no version message
-    node->CloseSocketDisconnect();
-    EXPECT_EQ(INVALID_SOCKET, node->hSocket) << "CloseSocketDisconnect left the closed fd number in hSocket";
-    EXPECT_EQ(-1, fcntl(number, F_GETFD)) << "the node's socket wasn't closed";
-
-    // Another owner gets the freed number, as the kernel's lowest-free-fd rule does in a busy node.
-    const int other = open("/dev/null", O_RDONLY);
-    ASSERT_GE(other, 0);
-    ASSERT_EQ(number, dup2(other, number));
-
-    node->CloseSocketDisconnect();   // e.g. a send error after the disconnect was already handled
-    delete node;                     // ~CNode
-    EXPECT_NE(-1, fcntl(number, F_GETFD)) << "a closed CNode closed the fd number's new owner";
-
-    close(number);
-    close(other);
-    close(pair[1]);
-}
-
-// util/sock.cpp's CloseSocket is the one that must be called: it closes once and invalidates.
-TEST(NetCloseSocket, CloseSocketInvalidates) {
-    int pair[2];
-    ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
-    SOCKET s = pair[0];
-    EXPECT_TRUE(CloseSocket(s));
-    EXPECT_EQ(INVALID_SOCKET, s);
-    EXPECT_FALSE(CloseSocket(s)) << "a second close of an invalidated handle must be a no-op";
-    close(pair[1]);
-}

src/gtest/test_wallet_dbenv.cpp

@@ -1,58 +0,0 @@
-// Copyright (c) 2016-2024 The Hush developers
-// Distributed under the GPLv3 software license, see the accompanying
-// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
-
-#include <gtest/gtest.h>
-
-#include <boost/filesystem.hpp>
-
-#include "wallet/db.h"
-
-namespace {
-
-// A fresh environment in a temp dir with one committed record in "wallet.dat", so database/ holds a log file.
-class WalletDbEnv : public ::testing::Test {
-protected:
-    boost::filesystem::path dir;
-    CDBEnv env;
-
-    void SetUp() override {
-        dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("dbenv-%%%%%%%%");
-        boost::filesystem::create_directories(dir);
-        ASSERT_TRUE(env.Open(dir));
-        Db db(env.dbenv, DB_CXX_NO_EXCEPTIONS);
-        ASSERT_EQ(0, db.open(NULL, "wallet.dat", "main", DB_BTREE, DB_CREATE | DB_AUTO_COMMIT, 0));
-        char k[] = "key", v[] = "value";
-        Dbt key(k, sizeof k), value(v, sizeof v);
-        ASSERT_EQ(0, db.put(NULL, &key, &value, 0));
-        ASSERT_EQ(0, db.close(0));
-        ASSERT_TRUE(boost::filesystem::exists(Log()));
-    }
-    void TearDown() override {
-        env.Close();
-        boost::system::error_code ec;
-        boost::filesystem::remove_all(dir, ec);
-    }
-    boost::filesystem::path Log() const { return dir / "database" / "log.0000000001"; }
-};
-
-} // namespace
-
-TEST_F(WalletDbEnv, CleanShutdownRemovesTheLogs) {
-    env.Flush(true);
-    EXPECT_FALSE(boost::filesystem::exists(dir / "database"));
-}
-
-// BDB0060 PANIC (a page write failed, e.g. through an fd closed behind BDB's back): the logs may hold committed
-// changes wallet.dat never got, and the next start's DB_RECOVER needs them.
-TEST_F(WalletDbEnv, PanickedEnvironmentKeepsTheLogs) {
-    ASSERT_EQ(0, env.dbenv->set_flags(DB_PANIC_ENVIRONMENT, 1));
-    env.Flush(true);
-    EXPECT_TRUE(boost::filesystem::exists(Log()));
-}
-
-TEST_F(WalletDbEnv, FailedCheckpointKeepsTheLogs) {
-    env.CheckpointLSN("no-such-file.dat");   // lsn_reset fails
-    env.Flush(true);
-    EXPECT_TRUE(boost::filesystem::exists(Log()));
-}

src/gtest/test_wallet_history.cpp

@@ -1,816 +0,0 @@
-// Copyright (c) 2016-2024 The Hush developers
-// Distributed under the GPLv3 software license, see the accompanying
-// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
-
-#include <gtest/gtest.h>
-
-#include <deque>
-#include <utf8.h>
-
-#include "chain.h"
-#include "chainparams.h"
-#include "coins.h"
-#include "init.h"
-#include "key_io.h"
-#include "main.h"
-#include "primitives/block.h"
-#include "primitives/transaction.h"
-#include "random.h"
-#include "rpc/server.h"
-#include "script/standard.h"
-#include "utilmoneystr.h"
-#include "wallet/wallet.h"
-#include "wallet/rpchushwallet.h"
-#include "zcash/Address.hpp"
-#include "zcash/Note.hpp"
-#include "zcash/NoteEncryption.hpp"
-#include "zcash/zip32.h"
-
-using namespace std;
-using namespace libzcash;
-
-// The history-RPC helpers (src/wallet/rpchushwallet.cpp) as of 69bb035e7, verbatim apart from the Legacy_
-// prefix and static linkage: the reference the optimized helpers must reproduce byte for byte.
-static void Legacy_zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
-
-  LOCK2(cs_main, pwalletMain->cs_wallet);
-
-  //Used to identify incomplete key sets
-  int vinCount = 0;
-  int vinSpendCount = 0;
-  int shieldedSpendCount = 0;
-  int shieldedSpendsSpentCount = 0;
-
-  //Check address
-  bool isTAddress = false;
-  bool isZsAddress = false;
-
-  CTxDestination tAddress = DecodeDestination(strAddress);
-  auto zAddress = DecodePaymentAddress(strAddress);
-  SaplingPaymentAddress zsAddress;
-
-  if (filterByAddress) {
-
-    if (IsValidDestination(tAddress))
-      isTAddress = true;
-
-    if (IsValidPaymentAddress(zAddress)) {
-      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
-          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
-          isZsAddress = true;
-      }
-    }
-  }
-
-
-  // Transparent Inputs belonging to the wallet
-  UniValue tSpends(UniValue::VARR);
-  if (isTAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vin.size(); i++) {
-      vinCount++;
-      const CTxIn& txin = wtx.vin[i];
-      UniValue obj(UniValue::VOBJ);
-      CTxDestination address;
-      const CWalletTx* parent = pwalletMain->GetWalletTx(txin.prevout.hash);
-      if (parent != NULL) {
-        const CTxOut& parentOut = parent->vout[txin.prevout.n];
-        ExtractDestination(parentOut.scriptPubKey, address);
-        if(IsMine(*pwalletMain, address)){
-          vinSpendCount++;
-          totalSpends += CAmount(-parentOut.nValue);
-          obj.push_back(Pair("address",EncodeDestination(address)));
-          obj.push_back(Pair("scriptPubKey",HexStr(parentOut.scriptPubKey.begin(), parentOut.scriptPubKey.end())));
-          obj.push_back(Pair("amount",ValueFromAmount(-parentOut.nValue)));
-          obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
-          obj.push_back(Pair("spendVout",(int)txin.prevout.n));
-          CTxDestination filterAddress = DecodeDestination(strAddress);
-          if (address == tAddress || !filterByAddress) {
-            filteredSpends += CAmount(-parentOut.nValue);
-            tSpends.push_back(obj);
-          }
-        }
-      }
-    }
-  }
-  spends.push_back(Pair("transparentSpends",tSpends));
-
-  // Sapling Inputs belonging to the wallet
-  UniValue zsSpends(UniValue::VARR);
-  if (isZsAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vShieldedSpend.size(); i++) {
-      shieldedSpendCount++;
-      const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
-      UniValue obj(UniValue::VOBJ);
-
-      SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
-
-      if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
-        const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
-        const OutputDescription& output = parent->vShieldedOutput[op.n];
-        auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
-        auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
-
-        if (pt) {
-          auto note = pt.get();
-          auto pa = nd.ivk.address(note.d);
-          auto address = pa.get();
-          shieldedSpendsSpentCount++;
-          totalSpends += CAmount(-note.value());
-          obj.push_back(Pair("address",EncodePaymentAddress(address)));
-          obj.push_back(Pair("amount", ValueFromAmount(CAmount(-note.value()))));
-          obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
-          obj.push_back(Pair("spendshieldedOutputIndex",(int)op.n));
-          if (address == zsAddress || !filterByAddress) {
-            filteredSpends += CAmount(-note.value());
-            zsSpends.push_back(obj);
-          }
-        }
-      }
-    }
-  }
-  spends.push_back(Pair("saplingSpends",zsSpends));
-
- 
-  spends.push_back(Pair("totalSpends",ValueFromAmount(filteredSpends)));
-
-  if (!filterByAddress) {
-    if (filteredSpends != 0 && (vinCount != vinSpendCount || shieldedSpendCount != shieldedSpendsSpentCount)) {
-      spends.push_back(Pair("missingSpendingKeys", true));
-      spends.push_back(Pair("vinCount", vinCount));
-      spends.push_back(Pair("vinSpendCount", vinSpendCount));
-      spends.push_back(Pair("shieldedSpendCount", shieldedSpendCount));
-      spends.push_back(Pair("shieldedSpendsSpentCount", shieldedSpendsSpentCount));
-    } else {
-        spends.push_back(Pair("missingSpendingKeys", false));
-    }
-  }
-}
-
-
-static void Legacy_zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
-
-  if(fZdebug)
-    fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
-
-  LOCK2(cs_main, pwalletMain->cs_wallet);
-
-  //Check address
-  bool isTAddress = false;
-  bool isZsAddress = false;
-
-  CTxDestination tAddress = DecodeDestination(strAddress);
-  auto zAddress = DecodePaymentAddress(strAddress);
-  SaplingPaymentAddress zsAddress;
-
-  if (filterByAddress) {
-
-    if (IsValidDestination(tAddress))
-      isTAddress = true;
-
-    if (IsValidPaymentAddress(zAddress)) {
-      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
-          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
-          isZsAddress = true;
-      }
-    }
-  }
-
-
-  //Transparent Received txos belonging to the wallet
-  UniValue tReceived(UniValue::VARR);
-  if (isTAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vout.size(); i++) {
-      const CTxOut& txout = wtx.vout[i];
-      UniValue obj(UniValue::VOBJ);
-      CTxDestination address;
-      ExtractDestination(txout.scriptPubKey, address);
-      if(IsMine(*pwalletMain, address)){
-        obj.push_back(Pair("address",EncodeDestination(address)));
-        obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
-        obj.push_back(Pair("amount",ValueFromAmount(txout.nValue)));
-        obj.push_back(Pair("vout", i));
-        if (address == tAddress || !filterByAddress) {
-          totalReceived += CAmount(txout.nValue);
-          tReceived.push_back(obj);
-        }
-      }
-    }
-  }
-  received.push_back(Pair("transparentReceived",tReceived));
-
-
-  //Sapling Sends belonging to the wallet
-  UniValue zsReceived(UniValue::VARR);
-  if (isZsAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
-      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
-      UniValue obj(UniValue::VOBJ);
-      bool changeTx = false;
-      //Decrypt sapling incoming commitments using IVK
-      std::set<libzcash::SaplingPaymentAddress> addresses;
-      pwalletMain->GetSaplingPaymentAddresses(addresses);
-      for (auto addr : addresses) {
-        libzcash::SaplingExtendedSpendingKey extsk;
-        if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-          auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                  outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
-
-          if (pt) {
-            auto note = pt.get();
-            auto pt_unwrapped = pt.get();
-            auto memo = pt_unwrapped.memo();
-            obj.push_back(Pair("address",EncodePaymentAddress(addr)));
-            obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
-            obj.push_back(Pair("memo", HexStr(memo)));
-
-              if (memo[0] <= 0xf4) {
-                  auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
-                  std::string memoStr(memo.begin(), end.base());
-                if (utf8::is_valid(memoStr)) {
-                obj.push_back(Pair("memoStr", memoStr));
-            }
-        }
-            obj.push_back(Pair("shieldedOutputIndex",i));
-
-            //Check Change Status
-            if (wtx.vShieldedSpend.size()!=0) {
-              std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
-              nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
-              BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
-                    changeTx = true;
-                }
-              }
-            }
-            obj.push_back(Pair("change",changeTx));
-            if (addr == zsAddress || !filterByAddress) {
-              totalReceived += CAmount(note.value());
-              zsReceived.push_back(obj);
-            }
-          }
-        }
-      }
-    }
-  }
-  received.push_back(Pair("saplingReceived",zsReceived));
-
-  received.push_back(Pair("totalReceived",ValueFromAmount(totalReceived)));
-}
-
-
-static void Legacy_zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
-
-  LOCK2(cs_main, pwalletMain->cs_wallet);
-
-  //Used to identify incomplete key sets
-  int shieldedOutputCount = 0;
-  int shieldedOutputDecryptedCount = 0;
-
-  //Check address
-  bool isTAddress = false;
-  bool isZsAddress = false;
-
-  CTxDestination tAddress = DecodeDestination(strAddress);
-  auto zAddress = DecodePaymentAddress(strAddress);
-  SaplingPaymentAddress zsAddress;
-
-  if (filterByAddress) {
-
-    if (IsValidDestination(tAddress))
-      isTAddress = true;
-
-    if (IsValidPaymentAddress(zAddress)) {
-      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
-          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
-          isZsAddress = true;
-      }
-    }
-  }
-
-  //All Transparent Sends in the transaction
-
-  UniValue tSends(UniValue::VARR);
-  if (isTAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vout.size(); i++) {
-      const CTxOut& txout = wtx.vout[i];
-      UniValue obj(UniValue::VOBJ);
-      CTxDestination address;
-      ExtractDestination(txout.scriptPubKey, address);
-      obj.push_back(Pair("address",EncodeDestination(address)));
-      obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
-      obj.push_back(Pair("amount",ValueFromAmount(-txout.nValue)));
-      obj.push_back(Pair("vout", i));
-      if (address == tAddress || !filterByAddress) {
-        totalSends += CAmount(-txout.nValue);
-        tSends.push_back(obj);
-      }
-    }
-  }
-  sends.push_back(Pair("transparentSends",tSends));
-
-  //All Shielded Sends in the transaction
-  UniValue zsSends(UniValue::VARR);
-  if (isZsAddress || !filterByAddress) {
-    for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
-      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
-      shieldedOutputCount++;
-      UniValue obj(UniValue::VOBJ);
-      bool changeTx = false;
-
-      //Decrypt sapling outgoing t to z transaction using HDseed
-      if (wtx.vShieldedSpend.size()==0) {
-        HDSeed seed;
-        if (pwalletMain->GetHDSeedForDerivation(seed)) {
-          auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-                  outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
-
-          if (opt) {
-            auto opt_unwrapped = opt.get();
-            auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                    outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
-
-            if (pt) {
-              shieldedOutputDecryptedCount++;
-              auto pt_unwrapped = pt.get();
-              libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
-              obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
-              obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
-              obj.push_back(Pair("shieldedOutputIndex",i));
-              obj.push_back(Pair("change",false));
-              if (sentAddr == zsAddress || !filterByAddress) {
-                totalSends += CAmount(-pt_unwrapped.value());
-                zsSends.push_back(obj);
-              }
-            }
-          }
-        }
-
-      //attempt Decryption of Outgoing Sapling using wallet extended spending keys
-      } else {
-        std::set<libzcash::SaplingPaymentAddress> addresses;
-        pwalletMain->GetSaplingPaymentAddresses(addresses);
-        for (auto addr : addresses) {
-          libzcash::SaplingExtendedSpendingKey extsk;
-          if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-            auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-                    outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
-
-            if (opt) {
-              auto opt_unwrapped = opt.get();
-              auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                      outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
-
-              if (pt) {
-                auto pt_unwrapped = pt.get();
-                auto memo = pt_unwrapped.memo();
-                shieldedOutputDecryptedCount++;
-                libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
-                obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
-                obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
-                obj.push_back(Pair("memo", HexStr(memo)));
-
-              if (memo[0] <= 0xf4) {
-                  auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
-                  std::string memoStr(memo.begin(), end.base());
-                if (utf8::is_valid(memoStr)) {
-                obj.push_back(Pair("memoStr", memoStr));
-            }
-        }
-                obj.push_back(Pair("shieldedOutputIndex",i));
-
-                //Check Change Status
-                if (wtx.vShieldedSpend.size()!=0) {
-                  std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
-                  nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
-                  BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                    if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
-                        changeTx = true;
-                    }
-                  }
-                }
-                obj.push_back(Pair("change",changeTx));
-                if (sentAddr == zsAddress || !filterByAddress) {
-                  totalSends += CAmount(-pt_unwrapped.value());
-                  zsSends.push_back(obj);
-                }
-              }
-            }
-          }
-        }
-      }
-    }
-  }
-  sends.push_back(Pair("saplingSends",zsSends));
-
-  if (shieldedOutputCount != shieldedOutputDecryptedCount) {
-    sends.push_back(Pair("missingSaplingOVK", true));
-  } else {
-    sends.push_back(Pair("missingSaplingOVK", false));
-  }
-
-  sends.push_back(Pair("totalSends",ValueFromAmount(totalSends)));
-}
-
-
-static void Legacy_zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
-
-  LOCK2(cs_main, pwalletMain->cs_wallet);
-
-  //Track total wallet spend and received
-  CAmount totalSpends = 0;
-  CAmount filteredSpends = 0;
-  CAmount totalReceived = 0;
-  CAmount totalSends = 0;
-
-  //Various Univalue to be added to the final transaction
-  UniValue spends(UniValue::VOBJ);
-  UniValue received(UniValue::VOBJ);
-  UniValue sends(UniValue::VOBJ);
-  UniValue tx(UniValue::VOBJ);
-
-  //Begin Compiling the Decrypted Transaction
-  tx.push_back(Pair("txid", wtx.GetHash().ToString()));
-
-  if(fZdebug)
-    fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
-
-  if (wtx.IsCoinBase())
-  {
-      tx.push_back(Pair("coinbase", true));
-      if (wtx.GetDepthInMainChain() < 1)
-          tx.push_back(Pair("category", "orphan"));
-      else if (wtx.GetBlocksToMaturity() > 0)
-          tx.push_back(Pair("category", "immature"));
-      else
-          tx.push_back(Pair("category", "generate"));
-  } else {
-    tx.push_back(Pair("coinbase", false));
-    tx.push_back(Pair("category", "standard"));
-  }
-
-  tx.push_back(Pair("blockhash", wtx.hashBlock.GetHex()));
-  tx.push_back(Pair("blockindex", wtx.nIndex));
-  int confirms = wtx.GetDepthInMainChain();
-  if(confirms > 0)
-  {
-      tx.push_back(Pair("blocktime", mapBlockIndex[wtx.hashBlock]->GetBlockTime()));
-  } else {
-      tx.push_back(Pair("blocktime", 0));
-  }
-  tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
-  tx.push_back(Pair("confirmations", confirms));
-  tx.push_back(Pair("time", wtx.GetTxTime()));
-  tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
-
-  //Wallet Conflicts
-  UniValue conflicts(UniValue::VARR);
-  BOOST_FOREACH(const uint256& conflict, wtx.GetConflicts())
-      conflicts.push_back(conflict.GetHex());
-  tx.push_back(Pair("walletconflicts", conflicts));
-
-  // Return Type used to determine what is included in the transaction
-  // 0 Spends, Received and spends
-  // 1 Sends
-  // 2 Received
-  // 3 Spends
-  // TODO - Setup Enum, maybe...
-
-  // Add Spends retrieved earlier
-  if (returnType != 2) {
-    Legacy_zsTxSpendsToJSON(wtx, spends, totalSpends, filteredSpends, strAddress, fBool);
-    if ((!fBool || filteredSpends != 0) && (returnType == 0 || returnType == 1)) {
-      tx.push_back(Pair("spends",spends));
-    }
-  }
-  // Get Received
-  if (returnType == 0 || returnType == 2) {
-    Legacy_zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
-    if (!fBool || totalReceived != 0) {
-      tx.push_back(Pair("received",received));
-    }
-  }
-
-  // Get Sends
-  if (returnType == 0 || returnType == 3) {
-    //Only include sends if there are spends that belong to the wallet.
-    if (totalSpends != 0 || fBool) {
-      Legacy_zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
-    }
-    if (!fBool || totalSends != 0) {
-      tx.push_back(Pair("sends",sends));
-    }
-  }
-
-  if ((returnType == 0 && (!fBool || filteredSpends != 0 || totalReceived != 0 || totalSends != 0))
-   || (returnType == 1 && (!fBool || filteredSpends != 0))
-   || (returnType == 2 && (!fBool || totalReceived != 0))
-   || (returnType == 3 && (!fBool || totalSends != 0))) {
-    ret.push_back(tx);
-  }
-
-}
-
-namespace {
-
-std::array<unsigned char, HUSH_MEMO_SIZE> Memo(const std::string& text)
-{
-    std::array<unsigned char, HUSH_MEMO_SIZE> memo;
-    memo.fill(0);
-    if (text == "empty")            // the "no memo" encoding: 0xF6 then zeros
-        memo[0] = 0xF6;
-    else if (text == "binary")      // starts below 0xF5 but isn't UTF-8
-        memo[0] = 0xC3, memo[1] = 0x28;
-    else
-        memcpy(memo.data(), text.data(), text.size());
-    return memo;
-}
-
-// A real encrypted Sapling output, and when ovk is given, an outCiphertext the sender can decrypt with it
-// (the wallet never checks proofs or value commitments).
-OutputDescription Output(const SaplingPaymentAddress& to, uint64_t value, const std::string& memo,
-                         const boost::optional<uint256>& ovk = boost::none)
-{
-    SaplingNote note(to, value);
-    auto enc = SaplingNotePlaintext(note, Memo(memo)).encrypt(note.pk_d);
-    assert(static_cast<bool>(enc));
-    OutputDescription od;
-    od.cv = GetRandHash();
-    od.cm = note.cm().get();
-    od.ephemeralKey = enc->second.get_epk();
-    od.encCiphertext = enc->first;
-    if (ovk) {
-        SaplingOutgoingPlaintext out(note.pk_d, enc->second.get_esk());
-        od.outCiphertext = out.encrypt(*ovk, od.cv, od.cm, enc->second);
-    } else {
-        GetRandBytes(od.outCiphertext.begin(), od.outCiphertext.size());
-    }
-    return od;
-}
-
-CTransaction SaplingTx(const std::vector<OutputDescription>& outputs, const std::vector<uint256>& spentNullifiers,
-                       const std::vector<COutPoint>& vin = {})
-{
-    CMutableTransaction mtx;
-    mtx.fOverwintered = true;
-    mtx.nVersion = SAPLING_TX_VERSION;
-    mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
-    mtx.vShieldedOutput = outputs;
-    for (const uint256& nf : spentNullifiers) {
-        SpendDescription sd;
-        sd.nullifier = nf;
-        mtx.vShieldedSpend.push_back(sd);
-    }
-    for (const COutPoint& prev : vin)
-        mtx.vin.push_back(CTxIn(prev));
-    return CTransaction(mtx);
-}
-
-SaplingExtendedSpendingKey Key(uint32_t account)
-{
-    std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x6b);
-    HDSeed seed(raw);
-    return SaplingExtendedSpendingKey::Master(seed).Derive(account | ZIP32_HARDENED_KEY_LIMIT);
-}
-
-}
-
-// A wallet whose txs exercise every branch of the history helpers: notes received on default and diversified
-// addresses, by a viewing-key-only address and by a key imported afterwards; z->z sends with change and memos;
-// a t->z shielding send; spends of our notes and of someone else's. pwalletMain points at it during the test.
-class HistoryWallet : public CWallet {
-public:
-    bool EncryptKeys(CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::EncryptKeys(vMasterKeyIn); }
-    bool Unlock(const CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::Unlock(vMasterKeyIn); }
-};
-
-class WalletHistory : public ::testing::Test {
-protected:
-    HistoryWallet wallet;
-    CWallet* savedWallet = nullptr;
-    SaplingPaymentAddress paA, paA2, paB, paC, paD, paF;
-    CTxDestination taddr;
-    std::map<uint256, int> depthOf;
-    std::vector<uint256> txids;
-    CCoinsView coinsBase;
-    CCoinsViewCache coins{&coinsBase};
-    CCoinsViewCache* savedCoinsTip = nullptr;
-    CBlockIndex* savedTip = nullptr;
-    std::deque<CBlock> blocks;
-    std::deque<CBlockIndex> index;
-    std::vector<uint256> insertedBlocks;
-
-    void SetUp() override {
-        SelectParams(CBaseChainParams::MAIN);
-        savedWallet = pwalletMain;
-        pwalletMain = &wallet;
-        savedCoinsTip = pcoinsTip;
-        pcoinsTip = &coins;
-        LOCK2(cs_main, wallet.cs_wallet);
-        savedTip = chainActive.Tip();
-
-        std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x2d);
-        ASSERT_TRUE(wallet.SetHDSeed(HDSeed(raw)));
-        HDSeed seed;
-        ASSERT_TRUE(wallet.GetHDSeedForDerivation(seed));
-        const uint256 seedOvk = ovkForShieldingFromTaddr(seed);
-
-        auto skA = Key(0), skB = Key(1), skC = Key(2), skD = Key(3), skF = Key(9);
-        paA = skA.DefaultAddress();
-        diversifier_index_t j;
-        j.begin()[0] = 50;
-        paA2 = skA.ToXFVK().Address(j)->second;
-        paB = skB.DefaultAddress();
-        paC = skC.DefaultAddress();
-        paD = skD.DefaultAddress();
-        paF = skF.DefaultAddress();
-        const uint256 ovkA = skA.expsk.full_viewing_key().ovk, ovkC = skC.expsk.full_viewing_key().ovk;
-        ASSERT_TRUE(wallet.AddSaplingZKey(skA, paA));
-        ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skA.expsk.full_viewing_key().in_viewing_key(), paA2));
-        ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skB.expsk.full_viewing_key().in_viewing_key(), paB));
-        ASSERT_TRUE(wallet.AddSaplingZKey(skC, paC));
-
-        CKey tkey;
-        tkey.MakeNewKey(true);
-        ASSERT_TRUE(wallet.AddKeyPubKey(tkey, tkey.GetPubKey()));
-        taddr = tkey.GetPubKey().GetID();
-
-        const uint256 nfA = uint256S("a1"), nfA2 = uint256S("a2"), nfC = uint256S("c1"), nfForeign = uint256S("ff");
-
-        // T0: transparent funds to our taddr.
-        CMutableTransaction t0;
-        t0.vin.push_back(CTxIn(COutPoint(GetRandHash(), 0)));
-        t0.vout.push_back(CTxOut(10 * COIN, GetScriptForDestination(taddr)));
-        AddTx(CTransaction(t0), 30);
-        // T1: received by A, A2, B (viewing key only), C and someone else; no spends.
-        uint256 t1 = AddTx(SaplingTx({Output(paA, 1000, "for A"), Output(paA2, 2000, "empty"), Output(paB, 3000, "for B"),
-                                      Output(paC, 4000, "binary"), Output(paF, 5000, "not ours")}, {}),
-                           25, {{0, nfA}, {1, nfA2}, {3, nfC}});
-        (void)t1;
-        // T2: A sends to F with change back to A, spending A's note: outCiphertexts under A's ovk.
-        AddTx(SaplingTx({Output(paF, 600, "payment", ovkA), Output(paA, 399, "change", ovkA)}, {nfA}), 20);
-        // T3: t->z shielding of T0's output to A, outCiphertext under the seed's shielding ovk.
-        AddTx(SaplingTx({Output(paA, 9 * COIN, "empty", seedOvk)}, {}, {COutPoint(txids[0], 0)}), 15);
-        // T4: C spends its note: to A2 (another of our keys) and change to C.
-        AddTx(SaplingTx({Output(paA2, 1500, "to a diversified A", ovkC), Output(paC, 2499, "binary", ovkC)}, {nfC}), 10);
-        // T5: someone else's spend paying A.
-        AddTx(SaplingTx({Output(paA, 777, "from a stranger")}, {nfForeign}), 8);
-        // T6: received by D before D's key is in the wallet (an import without rescan), then mined.
-        AddTx(SaplingTx({Output(paD, 4242, "for D, found later")}, {}), 5);
-        // T8: spends one of our notes and one of someone else's (a coinjoin-like tx): missingSpendingKeys.
-        AddTx(SaplingTx({Output(paF, 100, "mixed", ovkA)}, {nfA, uint256S("fe")}), 3);
-        // T9: spends A2's note, but its one output's outCiphertext is under nobody's ovk we know: missingSaplingOVK.
-        AddTx(SaplingTx({Output(paF, 50, "unreadable send")}, {nfA2}), 2);
-        // T7: spends A2's note (A and A2 share A's key); never mined and not in the mempool (depth -1).
-        AddTx(SaplingTx({Output(paF, 2900, "", ovkA)}, {nfA2}), 0);
-        ASSERT_TRUE(wallet.AddSaplingZKey(skD, paD));
-
-        Mine();
-    }
-
-    void TearDown() override {
-        LOCK(cs_main);
-        chainActive.SetTip(savedTip);
-        for (const uint256& h : insertedBlocks)
-            mapBlockIndex.erase(h);
-        pcoinsTip = savedCoinsTip;
-        pwalletMain = savedWallet;
-    }
-
-    uint256 AddTx(const CTransaction& tx, int depth, const std::map<uint32_t, uint256>& nullifiers = {}) {
-        CWalletTx wtx(&wallet, tx);
-        mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
-        for (const auto& nf : nullifiers)
-            found.at(SaplingOutPoint(tx.GetHash(), nf.first)).nullifier = nf.second;
-        wtx.SetSaplingNoteData(found);
-        EXPECT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
-        if (depth > 0)
-            depthOf[tx.GetHash()] = depth;
-        txids.push_back(tx.GetHash());
-        return tx.GetHash();
-    }
-
-    void Mine() {
-        const int tip = 40;
-        for (int h = 0; h <= tip; h++) {
-            blocks.emplace_back();
-            CBlock& block = blocks.back();
-            block.nTime = 1700000000 + 36 * h;
-            block.nNonce = uint256S(strprintf("%x", 0x900 + h));
-            if (h > 0)
-                block.hashPrevBlock = blocks[h - 1].GetHash();
-            for (const auto& d : depthOf)
-                if (tip - d.second + 1 == h)
-                    block.vtx.push_back(wallet.mapWallet.at(d.first));
-            block.hashMerkleRoot = block.BuildMerkleTree();
-            index.emplace_back(block);
-            CBlockIndex& bi = index.back();
-            bi.SetHeight(h);
-            bi.pprev = h > 0 ? &index[h - 1] : nullptr;
-            auto inserted = mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi));
-            ASSERT_TRUE(inserted.second);
-            insertedBlocks.push_back(block.GetHash());
-            bi.phashBlock = &inserted.first->first;
-        }
-        chainActive.SetTip(&index[tip]);
-        for (const auto& d : depthOf)
-            wallet.mapWallet.at(d.first).SetMerkleBranch(blocks[tip - d.second + 1]);
-    }
-
-    std::string NewJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
-        UniValue ret(UniValue::VARR);
-        zsWalletTxJSON(wtx, ret, addr, fBool, returnType, GetSaplingHistoryKeys());
-        return ret.write();
-    }
-    std::string LegacyJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
-        UniValue ret(UniValue::VARR);
-        Legacy_zsWalletTxJSON(wtx, ret, addr, fBool, returnType);
-        return ret.write();
-    }
-    void ExpectMatchesLegacy(bool checkCoverage = true);
-};
-
-// Every tx, every return type, unfiltered and filtered by each kind of address: byte-identical JSON. Run with
-// the wallet's plaintext cache warm (as after the startup warm-up) and cold (every output trial-decrypted).
-class WalletHistoryCold : public WalletHistory {};
-
-TEST_F(WalletHistoryCold, MatchesLegacy) {
-    LOCK2(cs_main, wallet.cs_wallet);
-    for (auto& p : wallet.mapWallet)
-        for (auto& item : p.second.mapSaplingNoteData)
-            item.second.cachedPlaintext = boost::none;
-    ExpectMatchesLegacy();
-}
-
-TEST_F(WalletHistory, MatchesLegacy) {
-    LOCK2(cs_main, wallet.cs_wallet);
-    ExpectMatchesLegacy();
-}
-
-void WalletHistory::ExpectMatchesLegacy(bool checkCoverage) {
-    const std::vector<std::pair<std::string, bool>> filters = {
-        {"*", false}, {"*", true}, {EncodePaymentAddress(paA), true}, {EncodePaymentAddress(paA2), true},
-        {EncodePaymentAddress(paB), true}, {EncodePaymentAddress(paC), true}, {EncodePaymentAddress(paD), true},
-        {EncodePaymentAddress(paF), true}, {EncodeDestination(taddr), true},
-    };
-    std::string all;
-    for (const uint256& txid : txids) {
-        const CWalletTx& wtx = wallet.mapWallet.at(txid);
-        for (int returnType = 0; returnType <= 3; returnType++) {
-            for (const auto& f : filters) {
-                // New first: the legacy spends helper default-inserts into wallet maps.
-                std::string got = NewJSON(wtx, f.first, f.second, returnType);
-                std::string want = LegacyJSON(wtx, f.first, f.second, returnType);
-                EXPECT_EQ(want, got) << "tx " << txid.ToString() << " returnType " << returnType << " filter " << f.first << "/" << f.second;
-                all += got;
-            }
-        }
-    }
-    // The fixture really reaches the interesting branches.
-    if (!checkCoverage)
-        return;
-    for (const char* needle : {"\"change\":true", "\"memoStr\"", "\"saplingSpends\":[{", "\"transparentSpends\":[{",
-                               "\"saplingSends\":[{", "\"saplingReceived\":[{", "\"missingSaplingOVK\":true",
-                               "\"missingSpendingKeys\":true", "for D, found later"}) {
-        EXPECT_NE(std::string::npos, all.find(needle)) << needle;
-    }
-}
-
-// The same comparison on an encrypted wallet, locked (no spending keys or HD seed available: no shielded
-// received or sent entries, as before) and then unlocked.
-class WalletHistoryEncrypted : public WalletHistory {};
-
-TEST_F(WalletHistoryEncrypted, MatchesLegacyLockedAndUnlocked) {
-    LOCK2(cs_main, wallet.cs_wallet);
-    CKeyingMaterial masterKey(32, 0x5e);
-    ASSERT_TRUE(wallet.EncryptKeys(masterKey));
-    ASSERT_TRUE(wallet.IsLocked());
-    ASSERT_TRUE(GetSaplingHistoryKeys().spendable.empty());
-    ExpectMatchesLegacy(false);
-    ASSERT_TRUE(wallet.Unlock(masterKey));
-    ASSERT_FALSE(GetSaplingHistoryKeys().spendable.empty());
-    ExpectMatchesLegacy();
-}
-
-// The history helpers only read the wallet. The old spends helper's operator[] lookups inserted an entry into
-// mapSaplingNullifiersToNotes for every other wallet's nullifier they met, which made a later
-// z_viewtransaction of that tx fail (map::at on a null outpoint).
-TEST_F(WalletHistory, DoesNotModifyTheWallet) {
-    LOCK2(cs_main, wallet.cs_wallet);
-    const size_t nullifierEntries = wallet.mapSaplingNullifiersToNotes.size();
-    std::map<uint256, size_t> noteEntries;
-    for (const auto& p : wallet.mapWallet)
-        noteEntries[p.first] = p.second.mapSaplingNoteData.size();
-
-    for (const uint256& txid : txids)
-        for (int returnType = 0; returnType <= 3; returnType++)
-            NewJSON(wallet.mapWallet.at(txid), "*", false, returnType);
-    EXPECT_EQ(nullifierEntries, wallet.mapSaplingNullifiersToNotes.size());
-    EXPECT_EQ(0u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff"))) << "someone else's spend";
-    for (const auto& p : wallet.mapWallet)
-        EXPECT_EQ(noteEntries.at(p.first), p.second.mapSaplingNoteData.size());
-
-    // What the old helpers did, for the record.
-    for (const uint256& txid : txids)
-        LegacyJSON(wallet.mapWallet.at(txid), "*", false, 0);
-    EXPECT_EQ(1u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff")));
-}

src/gtest/test_wallet_notedata.cpp

@@ -1,272 +0,0 @@
-// Copyright (c) 2016-2024 The Hush developers
-// Distributed under the GPLv3 software license, see the accompanying
-// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
-
-#include <gtest/gtest.h>
-
-#include <deque>
-
-#include "chain.h"
-#include "coins.h"
-#include "main.h"
-#include "primitives/block.h"
-#include "primitives/transaction.h"
-#include "random.h"
-#include "uint256.h"
-#include "wallet/wallet.h"
-#include "zcash/Note.hpp"
-#include "zcash/zip32.h"
-
-namespace {
-
-class NoteDataWallet : public CWallet {
-public:
-    bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock = true) {
-        return CWallet::UpdatedNoteData(wtxIn, wtx, fSameBlock);
-    }
-    int VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessOnly) {
-        return CWallet::VerifyAndSetInitialWitness(pindex, witnessOnly);
-    }
-};
-
-libzcash::SaplingIncomingViewingKey TestIvk()
-{
-    return libzcash::SaplingIncomingViewingKey(uint256S("0101010101010101010101010101010101010101010101010101010101010101"));
-}
-
-// A note as the wallet holds it once witnessed: nullifier set, witnesses cached at a height.
-SaplingNoteData WitnessedNote(const SaplingMerkleTree& tree, int height, const uint256& nullifier)
-{
-    SaplingNoteData nd(TestIvk(), nullifier);
-    nd.witnesses.push_front(tree.witness());
-    nd.witnessHeight = height;
-    return nd;
-}
-
-// A note as FindMySaplingNotes returns it during a replay or rescan: no nullifier, no witnesses.
-SaplingNoteData FreshNote()
-{
-    return SaplingNoteData(TestIvk());
-}
-
-}
-
-// A replayed or rescanned tx must keep the witnesses AND the nullifier it already had, and must not
-// report an update: dropping the nullifier made the next block clear and reseed every carried witness.
-TEST(WalletNoteData, ReplayKeepsNullifierAndWitnesses) {
-    NoteDataWallet wallet;
-    CWalletTx wtx(&wallet, CTransaction());
-    uint256 txid = uint256S("aa");
-    SaplingOutPoint op0(txid, 0);
-
-    SaplingMerkleTree tree;
-    tree.append(uint256S("01"));
-    const uint256 nf = uint256S("f00d");
-    wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 100, nf);
-    wtx.mapSaplingNoteData[op0].witnessRootValidated = true;
-    auto before = wtx.mapSaplingNoteData;
-
-    CWalletTx replay(&wallet, CTransaction());
-    replay.mapSaplingNoteData[op0] = FreshNote();
-
-    EXPECT_FALSE(wallet.UpdatedNoteData(replay, wtx));
-    ASSERT_EQ(1u, wtx.mapSaplingNoteData.size());
-    const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
-    ASSERT_TRUE(static_cast<bool>(nd.nullifier));
-    EXPECT_EQ(nf, *nd.nullifier);
-    EXPECT_EQ(100, nd.witnessHeight);
-    EXPECT_TRUE(nd.witnesses == before.at(op0).witnesses);
-    // ...but the kept witness is re-checked against the active chain once (in-memory flag only).
-    EXPECT_FALSE(nd.witnessRootValidated);
-}
-
-// A tx re-mined in a different block (reorg) has its notes at new tree positions: the old nullifier
-// must not be carried, so VerifyAndSetInitialWitness reseeds the note at its new position.
-TEST(WalletNoteData, MovedTxDropsNullifier) {
-    NoteDataWallet wallet;
-    CWalletTx wtx(&wallet, CTransaction());
-    SaplingOutPoint op0(uint256S("ee"), 0);
-
-    SaplingMerkleTree tree;
-    tree.append(uint256S("04"));
-    wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 500, uint256S("cafe"));
-
-    CWalletTx moved(&wallet, CTransaction());
-    moved.mapSaplingNoteData[op0] = FreshNote();
-
-    EXPECT_TRUE(wallet.UpdatedNoteData(moved, wtx, false));
-    const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
-    EXPECT_FALSE(static_cast<bool>(nd.nullifier));
-}
-
-// When the incoming data really adds something (a note for a newly added key), the tx is updated,
-// and the notes it already had keep their witnesses, witness height and nullifier together.
-TEST(WalletNoteData, NewNoteUpdatesAndKeepsExistingNoteState) {
-    NoteDataWallet wallet;
-    CWalletTx wtx(&wallet, CTransaction());
-    uint256 txid = uint256S("bb");
-    SaplingOutPoint op0(txid, 0);
-    SaplingOutPoint op1(txid, 1);
-
-    SaplingMerkleTree tree;
-    tree.append(uint256S("02"));
-    const uint256 nf = uint256S("beef");
-    wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 200, nf);
-
-    CWalletTx incoming(&wallet, CTransaction());
-    incoming.mapSaplingNoteData[op0] = FreshNote();
-    incoming.mapSaplingNoteData[op1] = FreshNote();
-
-    EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
-    ASSERT_EQ(2u, wtx.mapSaplingNoteData.size());
-    const SaplingNoteData& nd0 = wtx.mapSaplingNoteData.at(op0);
-    ASSERT_TRUE(static_cast<bool>(nd0.nullifier));
-    EXPECT_EQ(nf, *nd0.nullifier);
-    EXPECT_EQ(200, nd0.witnessHeight);
-    EXPECT_EQ(1u, nd0.witnesses.size());
-    const SaplingNoteData& nd1 = wtx.mapSaplingNoteData.at(op1);
-    EXPECT_FALSE(static_cast<bool>(nd1.nullifier));
-    EXPECT_TRUE(nd1.witnesses.empty());
-    EXPECT_EQ(-1, nd1.witnessHeight);
-}
-
-// Nothing is carried for a note that has no witnesses (PART B2): its height and nullifier describe
-// no witness, so they must not survive onto the incoming data.
-TEST(WalletNoteData, UnwitnessedNoteCarriesNothing) {
-    NoteDataWallet wallet;
-    CWalletTx wtx(&wallet, CTransaction());
-    SaplingOutPoint op0(uint256S("cc"), 0);
-
-    SaplingNoteData stale(TestIvk(), uint256S("dead"));
-    stale.witnessHeight = 300;
-    wtx.mapSaplingNoteData[op0] = stale;
-
-    CWalletTx incoming(&wallet, CTransaction());
-    incoming.mapSaplingNoteData[op0] = FreshNote();
-
-    EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
-    const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
-    EXPECT_FALSE(static_cast<bool>(nd.nullifier));
-    EXPECT_EQ(-1, nd.witnessHeight);
-    EXPECT_TRUE(nd.witnesses.empty());
-}
-
-// Incoming data with no Sapling notes never touches the wallet's note data.
-TEST(WalletNoteData, EmptyIncomingIsNoUpdate) {
-    NoteDataWallet wallet;
-    CWalletTx wtx(&wallet, CTransaction());
-    SaplingOutPoint op0(uint256S("dd"), 0);
-    SaplingMerkleTree tree;
-    tree.append(uint256S("03"));
-    wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 400, uint256S("abcd"));
-
-    CWalletTx incoming(&wallet, CTransaction());
-    EXPECT_FALSE(wallet.UpdatedNoteData(incoming, wtx));
-    EXPECT_EQ(1u, wtx.mapSaplingNoteData.size());
-    EXPECT_EQ(400, wtx.mapSaplingNoteData.at(op0).witnessHeight);
-}
-
-namespace {
-
-// A real encrypted Sapling output to pa (the wallet never checks its proof or value commitment).
-OutputDescription EncryptedOutput(const libzcash::SaplingPaymentAddress& pa, uint64_t value)
-{
-    libzcash::SaplingNote note(pa, value);
-    std::array<unsigned char, HUSH_MEMO_SIZE> memo;
-    memo.fill(0);
-    auto enc = libzcash::SaplingNotePlaintext(note, memo).encrypt(note.pk_d);
-    assert(static_cast<bool>(enc));
-    OutputDescription od;
-    od.cv = GetRandHash();
-    od.cm = note.cm().get();
-    od.ephemeralKey = enc->second.get_epk();
-    od.encCiphertext = enc->first;
-    return od;
-}
-
-// A three-block active chain (heights 0-2) whose block 1 holds the given txs, undone on destruction:
-// chainActive, mapBlockIndex and pcoinsTip are process-wide and every gtest suite shares the process.
-struct FakeActiveChain {
-    std::deque<CBlock> blocks;
-    std::deque<CBlockIndex> index;
-    CBlockIndex* savedTip;
-    CCoinsView coinsBase;
-    CCoinsViewCache coins{&coinsBase};
-    CCoinsViewCache* savedCoinsTip;
-
-    explicit FakeActiveChain(const std::vector<CTransaction>& inBlock1) {
-        LOCK(cs_main);
-        savedTip = chainActive.Tip();
-        savedCoinsTip = pcoinsTip;
-        pcoinsTip = &coins;
-        for (int h = 0; h <= 2; h++) {
-            blocks.emplace_back();
-            CBlock& block = blocks.back();
-            block.nTime = 1700000000 + h;
-            block.nNonce = uint256S(strprintf("%x", 0x700 + h));
-            if (h > 0)
-                block.hashPrevBlock = blocks[h - 1].GetHash();
-            if (h == 1)
-                block.vtx = inBlock1;
-            block.hashMerkleRoot = block.BuildMerkleTree();
-            index.emplace_back(block);
-            CBlockIndex& bi = index.back();
-            bi.SetHeight(h);
-            bi.pprev = h > 0 ? &index[h - 1] : nullptr;
-            bi.phashBlock = &mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi)).first->first;
-        }
-        chainActive.SetTip(&index.back());
-    }
-    ~FakeActiveChain() {
-        LOCK(cs_main);
-        chainActive.SetTip(savedTip);
-        for (const CBlock& block : blocks)
-            mapBlockIndex.erase(block.GetHash());
-        pcoinsTip = savedCoinsTip;
-    }
-};
-
-}
-
-// A note found with an incoming viewing key only (z_importviewingkey) can't be spent and never gets
-// a nullifier, because deriving one takes the full viewing key. VerifyAndSetInitialWitness must leave
-// it alone. It used to clear it and reseed its witness from the block on disk on every call, then
-// abort on the missing nullifier the first time the note was confirmed. Here there are no block
-// files, so the old reseed path throws.
-TEST(WalletNoteData, ViewingKeyOnlyNoteIsNotWitnessed) {
-    NoteDataWallet wallet;
-    std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x3c);
-    HDSeed seed(raw);
-    auto sk = libzcash::SaplingExtendedSpendingKey::Master(seed);
-    auto pa = sk.DefaultAddress();
-
-    LOCK2(cs_main, wallet.cs_wallet);
-    ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(sk.expsk.full_viewing_key().in_viewing_key(), pa));
-    ASSERT_FALSE(wallet.HaveSaplingFullViewingKey(sk.expsk.full_viewing_key().in_viewing_key()));
-
-    CMutableTransaction mtx;
-    mtx.fOverwintered = true;
-    mtx.nVersion = SAPLING_TX_VERSION;
-    mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
-    mtx.vShieldedOutput.push_back(EncryptedOutput(pa, 5000));
-    CTransaction tx(mtx);
-
-    CWalletTx wtx(&wallet, tx);
-    mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
-    ASSERT_EQ(1u, found.size()) << "the viewing key finds the note";
-    wtx.SetSaplingNoteData(found);
-    ASSERT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
-
-    FakeActiveChain chain({tx});
-    CWalletTx& stored = wallet.mapWallet.at(tx.GetHash());
-    stored.SetMerkleBranch(chain.blocks[1]);
-    ASSERT_EQ(2, stored.GetDepthInMainChain());
-
-    int minHeight = -1;
-    ASSERT_NO_THROW(minHeight = wallet.VerifyAndSetInitialWitness(chainActive.Tip(), false));
-    EXPECT_EQ(chainActive.Height(), minHeight) << "it must not pull the witness rebuild down to its height";
-    const SaplingNoteData& nd = stored.mapSaplingNoteData.at(SaplingOutPoint(tx.GetHash(), 0));
-    EXPECT_TRUE(nd.witnesses.empty());
-    EXPECT_FALSE(static_cast<bool>(nd.nullifier));
-    EXPECT_EQ(-1, nd.witnessHeight);
-}

src/httpserver.cpp

@@ -420,6 +420,10 @@ static void libevent_log_cb(int severity, const char *msg)
         LogPrint("libevent", "libevent: %s\n", msg);
 }
 
+/** Cap on the combined size of an HTTP request line + headers. libevent's default is EV_SIZE_MAX,
+ *  i.e. unbounded, and it buffers before any ACL or auth check runs. */
+static const size_t MAX_HEADERS_SIZE = 8192;
+
 bool InitHTTPServer()
 {
     struct evhttp* http = 0;
@@ -467,6 +471,10 @@ bool InitHTTPServer()
 
     evhttp_set_timeout(http, GetArg("-rpcservertimeout", DEFAULT_HTTP_SERVER_TIMEOUT));
     evhttp_set_max_body_size(http, MAX_SIZE);
+    // libevent defaults max_headers_size to EV_SIZE_MAX, so without this a single connection can
+    // stream an unbounded request line / header block and grow RSS ~1:1 with bytes sent, BEFORE the
+    // -rpcallowip ACL or auth check runs (both happen after libevent has parsed the request).
+    evhttp_set_max_headers_size(http, MAX_HEADERS_SIZE);
     evhttp_set_gencb(http, http_request_cb, NULL);
 
     if (!HTTPBindAddresses(http)) {

src/init.cpp

@@ -513,10 +513,6 @@ std::string HelpMessage(HelpMessageMode mode)
     // If this is used incorrectly (-rescanheight too large), then the local wallet may attempt to spend funds which it does not have witness data about
     // which will cause a "missing inputs" error when added to the mempool. Rescanning from correct height will fix this.
     strUsage += HelpMessageOpt("-keepnotewitnesscache", _("Keep partial Sapling Note Witness cache. Must be used with -rescanheight to find missing cache items."));
-    if (showDebug)
-        strUsage += HelpMessageOpt("-notecachewarmup", "Decrypt all wallet notes into memory at startup, so the first balance call after a restart doesn't (default: 1)");
-    if (showDebug)
-        strUsage += HelpMessageOpt("-checknotecache", "On every balance or note-listing call, decrypt each cached note it considers again and log and repair any difference from the cache (slow; default: 0)");
     strUsage += HelpMessageOpt("-rescan", _("Rescan the block chain for missing wallet transactions") + " " + _("on startup"));
     strUsage += HelpMessageOpt("-rescanheight", _("Rescan from specified height when rescan=1 on startup"));
     strUsage += HelpMessageOpt("-salvagewallet", _("Attempt to recover private keys from a corrupt wallet.dat") + " " + _("on startup"));
@@ -1020,8 +1016,7 @@ extern int32_t HUSH_REWIND;
 
 // --- Adaptive coins-cache sizing -------------------------------------------------------------
 // The in-memory UTXO/coins cache (nCoinCacheUsage) is the biggest lever on IBD speed: a bigger
-// cache means fewer size-triggered chainstate flushes to disk (the chainstate is also flushed at
+// cache means far fewer chainstate flushes to disk. We size it to use most of RAM, but a scheduled
-// least every DATABASE_FLUSH_INTERVAL / DATABASE_FLUSH_BLOCKS, see main.h). We size it to use most of RAM, but a scheduled
 // background task (AdjustCoinCacheForMemoryPressure, registered in AppInit2) shrinks the target when
 // free system memory runs low — e.g. the user opens other apps — and grows it back when memory frees
 // up, always leaving a reserve free for the rest of the system. The existing per-block flush
@@ -2234,9 +2229,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
 
     // ********************************************************* Step 8: load wallet
 #ifdef ENABLE_WALLET
-    // Set when the startup rescan below is interrupted: the wallet's state then ends at the scan's
-    // own checkpoint, not at the tip the notifier starts from.
-    bool fWalletScanIncomplete = false;
     if (fDisableWallet) {
         pwalletMain = NULL;
         LogPrintf("Wallet disabled!\n");
@@ -2508,14 +2500,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
             }
         }
 
-        // Sweep and consolidation pick notes with GetFilteredNotes at minDepth > 1. Older versions
-        // located each note's block by re-reading its tx through the txindex, so on a -txindex=0 node
-        // every fully shielded tx was skipped and these never moved those notes. Now they will.
-        if (!fTxIndex && (pwalletMain->fSweepEnabled || pwalletMain->fSaplingConsolidationEnabled)) {
-            LogPrintf("%s: -txindex=0 with -zsweep/-consolidation: fully shielded notes, which versions before "
-                      "v1.3.1 never saw on such nodes, are now included in sweeps and consolidations\n", __func__);
-        }
-
         //Set Automatic Coinbase Shielding (default ON, conditional: self-guards
         //on nodes where it cannot act - no owned coinbase, external mineraddress,
         //or locked wallet). Closes the transparent-coinbase leak for miners.
@@ -2682,7 +2666,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
             } else {
                 LogPrintf("Rescan did not complete; leaving the best-block locator at the scan's own "
                           "checkpoint so the remaining range is rescanned on the next start\n");
-                fWalletScanIncomplete = true;
             }
 
             // Restore wallet transaction metadata after -zapwallettxes=1
@@ -2710,15 +2693,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
                 }
             }
         }
-        // Decrypt the wallet's notes into the in-memory plaintext cache now, while RPC is still in warmup
-        // and before the wallet notification thread (started below) can want cs_wallet. Otherwise the
-        // first balance call after a restart decrypts them all under cs_main.
-        fCheckNoteCache = GetBoolArg("-checknotecache", false);
-        if (GetBoolArg("-notecachewarmup", true) && !ShutdownRequested()) {
-            uiInterface.InitMessage(_("Decrypting wallet notes..."));
-            pwalletMain->WarmSaplingNoteCache();
-        }
-
         pwalletMain->SetBroadcastTransactions(GetBoolArg("-walletbroadcast", true));
     } // (!fDisableWallet)
 #else // ENABLE_WALLET
@@ -2762,21 +2736,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
         {
             LOCK(cs_main);
             pindexLastTip = chainActive.Tip();
-#ifdef ENABLE_WALLET
-            // The wallet's state reflects the block the notifier starts from, unless the startup
-            // rescan was interrupted: then it ends at the scan's checkpoint, and the hourly
-            // SetBestChain must keep stamping that so the next start rescans the rest.
-            if (pwalletMain) {
-                const CBlockIndex* pindexWallet = pindexLastTip;
-                if (fWalletScanIncomplete) {
-                    CWalletDB walletdb(pwalletMain->strWalletFile);
-                    CBlockLocator locator;
-                    if (walletdb.ReadBestBlock(locator))
-                        pindexWallet = FindForkInGlobalIndex(chainActive, locator);
-                }
-                pwalletMain->SetWalletTip(pindexWallet);
-            }
-#endif
         }
         boost::function<void()> threadnotifywallets = boost::bind(&ThreadNotifyWallets, pindexLastTip);
         threadGroup.create_thread(
@@ -2818,24 +2777,8 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
         uiInterface.InitMessage(_("Activating best chain..."));
         // scan for better chains in the block chain database, that are not yet connected in the active best chain
         CValidationState state;
-        const CBlockIndex* pindexBeforeActivate = NULL;
-        {
-            LOCK(cs_main);
-            pindexBeforeActivate = chainActive.Tip();
-        }
         if ( !ActivateBestChain(true,state))
             strErrors << "Failed to connect best block";
-        else {
-            // Persist a startup replay now: the wallet's witness rebuild runs next and can hold
-            // cs_main for a long time, and a node killed before any later flush would replay it all again.
-            bool fTipMoved;
-            {
-                LOCK(cs_main);
-                fTipMoved = chainActive.Tip() != pindexBeforeActivate;
-            }
-            if (fTipMoved)
-                FlushStateToDisk();
-        }
     } else {
         LogPrintf("HUSH_REWIND < 0\n");
     }

src/main.cpp

@@ -3738,7 +3738,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
     static int64_t nLastWrite = 0;
     static int64_t nLastFlush = 0;
     static int64_t nLastSetChain = 0;
-    static int nLastFlushHeight = -1;
     std::set<int> setFilesToPrune;
     bool fFlushForPrune = false;
     try {
@@ -3764,9 +3763,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
         if (nLastSetChain == 0) {
             nLastSetChain = nNow;
         }
-        if (nLastFlushHeight < 0) {
-            nLastFlushHeight = chainActive.Height();
-        }
         size_t cacheSize = pcoinsTip->DynamicMemoryUsage();
         // The cache is large and close to the limit, but we have time now (not in the middle of a block processing).
         bool fCacheLarge = mode == FLUSH_STATE_PERIODIC && cacheSize * (10.0/9) > nCoinCacheUsage;
@@ -3776,10 +3772,8 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
         bool fPeriodicWrite = mode == FLUSH_STATE_PERIODIC && nNow > nLastWrite + (int64_t)DATABASE_WRITE_INTERVAL * 1000000;
         // It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
         bool fPeriodicFlush = mode == FLUSH_STATE_PERIODIC && nNow > nLastFlush + (int64_t)DATABASE_FLUSH_INTERVAL * 1000000;
-        // Many blocks connected since the last flush, e.g. a long replay inside one ActivateBestChain call.
-        bool fBlockCountFlush = (mode == FLUSH_STATE_IF_NEEDED || mode == FLUSH_STATE_PERIODIC) && chainActive.Height() - nLastFlushHeight >= DATABASE_FLUSH_BLOCKS;
         // Combine all conditions that result in a full cache flush.
-        bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune || fBlockCountFlush;
+        bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
         // Write blocks and block index to disk.
         if (fDoFullFlush || fPeriodicWrite) {
             // Depend on nMinDiskSpace to ensure we can write block index
@@ -3833,7 +3827,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
             if (!pcoinsTip->Flush())
                 return AbortNode(state, "Failed to write to coin database");
             nLastFlush = nNow;
-            nLastFlushHeight = chainActive.Height();
         }
         if ((mode == FLUSH_STATE_ALWAYS || mode == FLUSH_STATE_PERIODIC) && nNow > nLastSetChain + (int64_t)DATABASE_WRITE_INTERVAL * 1000000) {
             // Update best block in wallet (so we can detect restored wallets).
@@ -5922,10 +5915,7 @@ FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)
         return NULL;
     boost::filesystem::path path = GetBlockPosFilename(pos, prefix);
     boost::filesystem::create_directories(path.parent_path());
-    // Read-only opens get a read-only FILE (every fReadOnly caller only reads). With "rb+", a write through a
+    FILE* file = fopen(path.string().c_str(), "rb+");
-    // reused fd number landed here: when the wolfSSL CloseSocket macro (see net.h) closed the wallet's BDB fd
-    // behind its back, BDB's next page write went to whatever reopened that number -- a block file opened here.
-    FILE* file = fopen(path.string().c_str(), fReadOnly ? "rb" : "rb+");
     if (!file && !fReadOnly)
         file = fopen(path.string().c_str(), "wb+");
     if (!file) {
@@ -8346,20 +8336,13 @@ bool SendMessages(CNode* pto, bool fSendTrickle)
         // Message: addr
         if (fSendTrickle)
         {
-            vector<CAddress> vAddr;
+            // Accumulate into vAddr and send it ONCE (or in MAX_ADDR_TO_SEND-sized batches).
-            vAddr.reserve(pto->vAddrToSend.size());
+            // This loop previously pushed the ENTIRE pto->vAddrToSend on every accepted address,
-            BOOST_FOREACH(const CAddress& addr, pto->vAddrToSend)
+            // so a single 24-byte getaddr produced N messages of N addresses each instead of one
-            {
+            // message of N -- ~500x the intended bandwidth on a typical addrman, all serialized
-                if (pto->AddAddressIfNotAlreadyKnown(addr))
+            // (with per-message double-SHA256 checksums) while cs_main is held. The locally built
-                {
+            // vAddr was accumulated and then discarded, and the vAddr.resize(MAX_ADDR_TO_SEND) was
-                    vAddr.push_back(addr);
+            // a no-op standing where upstream has vAddr.clear().
-
-                    if (vAddr.size() >= MAX_ADDR_TO_SEND)
-                    {
-                      // Should be impossible since we always check size before adding to
-                      // vAddrToSend. Recover by trimming the vector.
-                      vAddr.resize(MAX_ADDR_TO_SEND);
-                    }
             const char* msg_type;
             int make_flags;
             if (pto->m_wants_addrv2) {
@@ -8369,13 +8352,26 @@ bool SendMessages(CNode* pto, bool fSendTrickle)
                 msg_type = NetMsgType::ADDR;
                 make_flags = 0;
             }
-                    pto->PushAddrMessage(CNetMsgMaker(std::min(pto->nVersion, PROTOCOL_VERSION)).Make(make_flags, msg_type, pto->vAddrToSend));
+            const CNetMsgMaker msgMaker(std::min(pto->nVersion, PROTOCOL_VERSION));
 
+            vector<CAddress> vAddr;
+            vAddr.reserve(pto->vAddrToSend.size());
+            BOOST_FOREACH(const CAddress& addr, pto->vAddrToSend)
+            {
+                if (pto->AddAddressIfNotAlreadyKnown(addr))
+                {
+                    vAddr.push_back(addr);
+                    if (vAddr.size() >= MAX_ADDR_TO_SEND)
+                    {
+                        pto->PushAddrMessage(msgMaker.Make(make_flags, msg_type, vAddr));
+                        vAddr.clear();
+                    }
                 }
             }
 
             pto->vAddrToSend.clear();
-            vAddr.clear();
+            if (!vAddr.empty())
+                pto->PushAddrMessage(msgMaker.Make(make_flags, msg_type, vAddr));
         }
 
         CNodeState &state = *State(pto->GetId());

src/main.h

@@ -143,15 +143,8 @@ static const unsigned int MAX_LOCATOR_SZ = 101;
 static unsigned int BLOCK_DOWNLOAD_WINDOW = 1024;
 /** Time to wait (in seconds) between writing blocks/block index to disk. */
 static const unsigned int DATABASE_WRITE_INTERVAL = 60 * 60;
-/** Time to wait (in seconds) between flushing chainstate to disk. Upstream uses 24h to keep a large
+/** Time to wait (in seconds) between flushing chainstate to disk. */
- *  coins cache warm, but DragonX's cache stays a few MiB (private chain, few transparent outputs), so
+static const unsigned int DATABASE_FLUSH_INTERVAL = 24 * 60 * 60;
- *  the size triggers never fire and this timer is all that persists the chainstate while a node runs:
- *  a node that never reached 24h uptime or a clean shutdown replayed weeks of blocks on every start. */
-static const unsigned int DATABASE_FLUSH_INTERVAL = 60 * 60;
-/** Also flush chainstate once this many blocks were connected since the last flush. A long
- *  ActivateBestChain (startup replay, catching up after downtime) only flushes IF_NEEDED until it
- *  returns, so without this a node killed part-way through loses the whole replay. */
-static const int DATABASE_FLUSH_BLOCKS = 5000;
 /** Maximum length of reject messages. */
 static const unsigned int MAX_REJECT_MESSAGE_LENGTH = 111;
 static const int64_t DEFAULT_MAX_TIP_AGE = 24 * 60 * 60;

src/net.cpp

@@ -45,13 +45,6 @@
 #include <wolfssl/options.h>
 #include <wolfssl/ssl.h>
 #include <hush/tlsmanager.cpp>
-
-// CloseSocket must be util/sock.cpp's CloseSocket(SOCKET&), which invalidates the handle it closes. If
-// wolfSSL's `#define CloseSocket(s) close(s)` survived into this file (see net.h), each close here would leave
-// its stale fd number behind to be closed again after another thread reused it.
-#ifdef CloseSocket
-#error "CloseSocket is a macro here (wolfssl/wolfio.h): it would bypass CloseSocket(SOCKET&) in util/sock.cpp"
-#endif
 using namespace hush;
 extern int32_t HUSH_TESTNODE;
 extern uint8_t ASSETCHAINS_CLEARNET;

src/net.h

@@ -57,14 +57,6 @@
 #define byte unsigned char
 #include <wolfssl/ssl.h>
 #undef byte
-// wolfssl/wolfio.h (pulled in by ssl.h) does `#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Left
-// defined, it turns every CloseSocket(hSocket) after this point -- all 15 calls in net.cpp, including the ones in
-// CloseSocketDisconnect and ~CNode -- into a bare close() that never sets hSocket = INVALID_SOCKET, so util/sock.cpp's
-// CloseSocket(SOCKET&) is never called. Every P2P disconnect then closed its fd number again later (a repeated
-// CloseSocketDisconnect, then ~CNode), by which time another thread often owned that number: wallet.dat or a
-// database/log.* fd (BDB then wrote wallet pages into whatever got the number next, e.g. a blocks/*.dat file
-// opened "rb+"), a block file, or an HTTP RPC socket. wolfio.h is include-guarded, so it can't come back.
-#undef CloseSocket
 
 class CAddrMan;
 class CBlockIndex;
@@ -615,7 +607,11 @@ public:
         // Known checking here is only to save space from duplicates.
         // SendMessages will filter it again for knowns that were added
         // after addresses were pushed.
-        if (_addr.IsValid() && !IsAddressKnown(addr) && addr_format_supported) {
+        // NOTE: _addr (the address being queued), NOT addr (this peer's own address, net.h ~416).
+        // Testing the member made the filter constant for the connection's lifetime: once the peer's
+        // own address entered its addrKnown -- routine, via the remote's AdvertizeLocal -- every
+        // relay path to it silently no-opped until the daily addrKnown.reset().
+        if (_addr.IsValid() && !IsAddressKnown(_addr) && addr_format_supported) {
 
             if (vAddrToSend.size() >= MAX_ADDR_TO_SEND) {
                 vAddrToSend[insecure_rand() % vAddrToSend.size()] = _addr;

src/rpc/blockchain.cpp

@@ -534,6 +534,10 @@ UniValue getblockdeltas(const UniValue& params, bool fHelp, const CPubKey& mypk)
     if (fHelp || params.size() != 1)
         throw runtime_error("");
 
+    // Reads mapBlockIndex / chainActive (and, below, pcoinsTip's mutable anchor cache),
+    // all of which are cs_main-guarded. Every sibling RPC in this file locks; this one did not.
+    LOCK(cs_main);
+
     std::string strHash = params[0].get_str();
     uint256 hash(uint256S(strHash));
 
@@ -602,12 +606,20 @@ UniValue getblockhashes(const UniValue& params, bool fHelp, const CPubKey& mypk)
 
     std::vector<std::pair<uint256, unsigned int> > blockHashes;
 
-    if (fActiveOnly)
+    {
+        // The lock must SPAN GetTimestampIndex: with fActiveOnly it calls blockOnchainActive() for
+        // every row, which reads mapBlockIndex and chainActive. The previous form was
+        //     if (fActiveOnly)
+        //         LOCK(cs_main);
+        // and LOCK() declares a scoped object, so as an unbraced substatement it was constructed
+        // and destroyed on that line -- the walk then ran completely unsynchronised. Taken
+        // unconditionally here: this RPC is explorer-only and not hot, and a conditional lock is
+        // exactly the shape that produced the bug.
         LOCK(cs_main);
-
         if (!GetTimestampIndex(high, low, fActiveOnly, blockHashes)) {
             throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "No information available for block hashes");
         }
+    }
 
     UniValue result(UniValue::VARR);
 
@@ -877,6 +889,10 @@ UniValue getblockmerkletree(const UniValue& params, bool fHelp, const CPubKey& m
             + HelpExampleRpc("getblockmerkletree", "290000")
         );
 
+    // Reads mapBlockIndex / chainActive (and, below, pcoinsTip's mutable anchor cache),
+    // all of which are cs_main-guarded. Every sibling RPC in this file locks; this one did not.
+    LOCK(cs_main);
+
     CBlockIndex* phushblockindex;
     uint256 blockRoot;
     SaplingMerkleTree tree;

src/rpc/mining.cpp

@@ -759,9 +759,25 @@ UniValue getblocktemplate(const UniValue& params, bool fHelp, const CPubKey& myp
 #ifdef ENABLE_WALLET
         CReserveKey reservekey(pwalletMain);
         LEAVE_CRITICAL_SECTION(cs_main);
+        // MUST re-enter cs_main before letting an exception escape. The enclosing LOCK(cs_main) is
+        // a scoped CMutexLock whose owns_lock is still true, so if CreateNewBlockWithKey throws
+        // (any wallet/BDB fault: disk full, EMFILE, a corrupt wallet.dat) its destructor unlocks an
+        // already-unlocked mutex during unwinding -> BOOST_VERIFY -> SIGABRT. Asserts cannot be
+        // compiled out here (main.cpp #errors on NDEBUG), so this aborts the daemon instead of
+        // returning the actionable error, and the abort happens inside unwinding so nothing is logged.
+        try {
             pblocktemplate = CreateNewBlockWithKey(reservekey,pindexPrevNew->GetHeight()+1,HUSH_MAXGPUCOUNT,false);
+        } catch (...) {
+            ENTER_CRITICAL_SECTION(cs_main);
+            throw;
+        }
 #else
+        try {
             pblocktemplate = CreateNewBlockWithKey();
+        } catch (...) {
+            ENTER_CRITICAL_SECTION(cs_main);
+            throw;
+        }
 #endif
         ENTER_CRITICAL_SECTION(cs_main);
         if (!pblocktemplate)

src/rpc/server.cpp

@@ -26,7 +26,6 @@
 #include "util.h"
 #include "util/strencodings.h"
 #include "asyncrpcqueue.h"
-#include <chrono>
 #include <memory>
 #include <univalue.h>
 #include <boost/bind.hpp>
@@ -641,30 +640,6 @@ std::string JSONRPCExecBatch(const UniValue& vReq)
     return ret.write() + "\n";
 }
 
-namespace {
-// Logs an RPC whose wall time reached 500ms, or every RPC under -debug=bench. That time includes
-// waiting for cs_main / cs_wallet behind block connection or another RPC, which the callee's own
-// timers can't see because they start once the locks are held. A long-poll getblocktemplate waits
-// for a new block with cs_main released, so it is only logged under -debug=bench. The method is
-// logged as method=<name>: a bare " <name> " token would match the GUI's " rescan " log heuristic.
-class RPCCallTimer {
-    const std::string& strMethod;
-    const bool fLongPoll;
-    const std::chrono::steady_clock::time_point start;
-public:
-    RPCCallTimer(const std::string& method, const UniValue& params)
-        : strMethod(method),
-          fLongPoll(method == "getblocktemplate" && params.size() > 0 && params[0].isObject() &&
-                    !find_value(params[0], "longpollid").isNull()),
-          start(std::chrono::steady_clock::now()) {}
-    ~RPCCallTimer() {
-        const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
-        if ((nMs >= 500 && !fLongPoll) || LogAcceptCategory("bench"))
-            LogPrintf("RPC method=%s took %dms\n", strMethod, nMs);
-    }
-};
-}
-
 UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params) const
 {
     // Find method
@@ -699,7 +674,6 @@ UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params
     try
     {
         // Execute
-        RPCCallTimer timer(strMethod, params);
         return pcmd->actor(params, false, CPubKey());
     }
     catch (const std::exception& e)

src/validationinterface.cpp

@@ -86,30 +86,6 @@ struct CachedBlockData {
         pindex(pindex), oldTrees(oldTrees), txConflicted(txConflicted) {}
 };
 
-bool NotifyWalletsOrShutdown(const std::function<void()>& notify)
-{
-    // The wallet can throw here on a database failure, e.g. CWalletDB::ListAccountCreditDebit's "error scanning
-    // DB" after BDB lost its file. Escaping this thread, that std::terminate()d the node (TraceThread logs and
-    // rethrows) with nothing flushed. Stop cleanly instead: going on would leave the wallet behind the chain,
-    // missing the block it failed on.
-    try {
-        notify();
-        return true;
-    } catch (const boost::thread_interrupted&) {
-        throw;
-    } catch (const std::exception& e) {
-        PrintExceptionContinue(&e, "ThreadNotifyWallets()");
-    } catch (...) {
-        PrintExceptionContinue(NULL, "ThreadNotifyWallets()");
-    }
-    LogPrintf("*** %s\n", "Wallet failed to process a block notification; shutting down");
-    StartShutdown();
-    uiInterface.ThreadSafeMessageBox(
-        _("Error: A fatal internal error occurred, see debug.log for details"),
-        "", CClientUIInterface::MSG_ERROR);
-    return false;
-}
-
 void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
 {
     // If pindexLastTip == nullptr, the wallet is at genesis.
@@ -227,12 +203,8 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
                     _("Error: A fatal internal error occurred, see debug.log for details"),
                     "", CClientUIInterface::MSG_ERROR);
                 StartShutdown();
-                // Don't hand the wallet an empty block as if it had been processed.
-                fDisconnectIncomplete = true;
-                break;
             }
 
-            if (!NotifyWalletsOrShutdown([&] {
             // Let wallets know transactions went from 1-confirmed to
             // 0-confirmed or conflicted:
             for (const CTransaction &tx : block.vtx) {
@@ -240,8 +212,6 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
             }
             // Update cached incremental witnesses
             GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
-            }))
-                return;
 
             // On to the next block!
             pindexLastTip = pindexLastTip->pprev;
@@ -269,11 +239,8 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
                     _("Error: A fatal internal error occurred, see debug.log for details"),
                     "", CClientUIInterface::MSG_ERROR);
                 StartShutdown();
-                // Don't hand the wallet an empty block as if it had been processed.
-                break;
             }
 
-            if (!NotifyWalletsOrShutdown([&] {
             // Tell wallet about transactions that went from mempool
             // to conflicted:
             for (const CTransaction &tx : blockData.txConflicted) {
@@ -285,8 +252,6 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
             }
             // Update cached incremental witnesses
             GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
-            }))
-                return;
 
             // This block is done!
             pindexLastTip = blockData.pindex;

src/validationinterface.h

@@ -8,7 +8,6 @@
 #define HUSH_VALIDATIONINTERFACE_H
 
 #include <boost/signals2/signal.hpp>
-#include <functional>
 #include "zcash/IncrementalMerkleTree.hpp"
 
 class CBlock;
@@ -77,8 +76,4 @@ CMainSignals& GetMainSignals();
 
 void ThreadNotifyWallets(CBlockIndex *pindexLastTip);
 
-/** Runs one block's wallet notifications for ThreadNotifyWallets. Returns false, after logging the error and
- *  requesting a shutdown, if they threw (e.g. a wallet database failure); thread interruption propagates. */
-bool NotifyWalletsOrShutdown(const std::function<void()>& notify);
-
 #endif // HUSH_VALIDATIONINTERFACE_H

src/version.h

@@ -22,13 +22,7 @@
 
 // network protocol versioning
 // DragonX 1.0.0 - bumped to separate from old HUSH/DragonX nodes with RandomX bug
-// DragonX 1.3.0 - bumped for the consensus changes in this release (BLOCK_VALID_CONTEXT
+static const int PROTOCOL_VERSION = 2000000;
-//                 no longer overwrites the validity level, the dead CBOPRET coinbase
-//                 price check is gone, and pow.cpp lost 157 lines). MIN_PEER_PROTO_VERSION
-//                 is deliberately NOT raised: it currently equals this value, and raising
-//                 it would disconnect every node still on v1.0.3 -- which is all users,
-//                 since no release has ever been installable (unsigned archives).
-static const int PROTOCOL_VERSION = 2000001;
 //! initial proto version, to be increased after version/verack negotiation
 static const int INIT_PROTO_VERSION = 209;
 //! In this version, 'getheaders' was introduced.

src/wallet/crypter.cpp

@@ -591,7 +591,7 @@ bool CCryptoKeyStore::GetSaplingSpendingKey(const libzcash::SaplingFullViewingKe
         if (!IsCrypted())
             return CBasicKeyStore::GetSaplingSpendingKey(fvk, skOut);
 
-        for (const auto& entry : mapCryptedSaplingSpendingKeys) {
+        for (auto entry : mapCryptedSaplingSpendingKeys) {
             if (entry.first.fvk == fvk) {
                 const std::vector<unsigned char> &vchCryptedSecret = entry.second;
                 return DecryptSaplingSpendingKey(vMasterKey, vchCryptedSecret, entry.first, skOut);

src/wallet/crypter.h

@@ -265,7 +265,7 @@ public:
             LOCK(cs_SpendingKeyStore);
             if (!IsCrypted())
                 return CBasicKeyStore::HaveSaplingSpendingKey(fvk);
-            for (const auto& entry : mapCryptedSaplingSpendingKeys) {
+            for (auto entry : mapCryptedSaplingSpendingKeys) {
                 if (entry.first.fvk == fvk) {
                     return true;
                 }

src/wallet/db.cpp

@@ -61,7 +61,6 @@ void CDBEnv::Reset()
     dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS);
     fDbEnvInit = false;
     fMockDb = false;
-    fKeepLogs = false;
 }
 
 CDBEnv::CDBEnv() : dbenv(NULL)
@@ -277,12 +276,10 @@ bool CDBEnv::Salvage(const std::string& strFile, bool fAggressive, std::vector<C
 
 void CDBEnv::CheckpointLSN(const std::string& strFile)
 {
-    if (dbenv->txn_checkpoint(0, 0, 0) != 0)
+    dbenv->txn_checkpoint(0, 0, 0);
-        fKeepLogs = true;
     if (fMockDb)
         return;
-    if (dbenv->lsn_reset(strFile.c_str(), 0) != 0)
+    dbenv->lsn_reset(strFile.c_str(), 0);
-        fKeepLogs = true;
 }
 
 
@@ -501,16 +498,10 @@ void CDBEnv::Flush(bool fShutdown)
                 // Move log data to the dat file
                 CloseDb(strFile);
                 LogPrint("db", "CDBEnv::Flush: %s checkpoint\n", strFile);
-                int ret = dbenv->txn_checkpoint(0, 0, 0);
+                dbenv->txn_checkpoint(0, 0, 0);
-                if (ret != 0) {
-                    LogPrintf("CDBEnv::Flush: %s checkpoint failed: %s\n", strFile, DbEnv::strerror(ret));
-                    fKeepLogs = true;
-                }
                 LogPrint("db", "CDBEnv::Flush: %s detach\n", strFile);
-                if (!fMockDb && (ret = dbenv->lsn_reset(strFile.c_str(), 0)) != 0) {
+                if (!fMockDb)
-                    LogPrintf("CDBEnv::Flush: %s lsn_reset failed: %s\n", strFile, DbEnv::strerror(ret));
+                    dbenv->lsn_reset(strFile.c_str(), 0);
-                    fKeepLogs = true;
-                }
                 LogPrint("db", "CDBEnv::Flush: %s closed\n", strFile);
                 mapFileUseCount.erase(mi++);
             } else
@@ -520,19 +511,6 @@ void CDBEnv::Flush(bool fShutdown)
         if (fShutdown) {
             char** listp;
             if (mapFileUseCount.empty()) {
-                // After a failed flush, or once the environment has panicked (BDB0060, e.g. after a page
-                // write failed), wallet.dat can be missing committed changes that only the logs hold.
-                // Removing them would turn a wallet the next start recovers (DB_RECOVER) into one that
-                // silently lost them. If the logs are unusable, the next start moves database/ aside.
-                // (get_flags itself fails with DB_RUNRECOVERY once the environment has panicked.)
-                u_int32_t envFlags = 0;
-                if (dbenv->get_flags(&envFlags) != 0 || (envFlags & DB_PANIC_ENVIRONMENT))
-                    fKeepLogs = true;
-                if (fKeepLogs && !fMockDb) {
-                    LogPrintf("CDBEnv::Flush: the wallet database was not flushed cleanly; keeping %s for recovery at the next start\n",
-                              (boost::filesystem::path(strPath) / "database").string());
-                    Close();
-                } else {
                 dbenv->log_archive(&listp, DB_ARCH_REMOVE);
                 Close();
                 if (!fMockDb)
@@ -540,5 +518,4 @@ void CDBEnv::Flush(bool fShutdown)
             }
         }
     }
-    }
 }

src/wallet/db.h

@@ -28,7 +28,6 @@
 #include "sync.h"
 #include "version.h"
 
-#include <atomic>
 #include <map>
 #include <string>
 #include <vector>
@@ -54,9 +53,6 @@ class CDBEnv
 private:
     bool fDbEnvInit;
     bool fMockDb;
-    // A checkpoint or lsn_reset failed: the logs in database/ may hold the only copy of committed
-    // wallet changes, so the shutdown flush must leave them for DB_RECOVER at the next start.
-    std::atomic<bool> fKeepLogs;
     // Don't change into boost::filesystem::path, as that can result in
     // shutdown problems/crashes caused by a static initialized internal pointer.
     std::string strPath;

src/wallet/rpcdump.cpp

@@ -299,7 +299,10 @@ UniValue importprivkey(const UniValue& params, bool fHelp, const CPubKey& mypk)
     bool fRescan = true;
     if (params.size() > 2)
         fRescan = params[2].get_bool();
-    if ( fRescan && params.size() == 4 )
+    // '> 3', not '== 4': with the optional 5th (secret_key) argument present the equality test
+    // failed and height silently stayed 0, rescanning from genesis. Every sibling RPC in this file
+    // already uses the '>' form.
+    if ( fRescan && params.size() > 3 )
         height = params[3].get_int();
 
 

src/wallet/rpchushwallet.cpp

@@ -16,109 +16,6 @@ using namespace libzcash;
 
 bool EnsureWalletIsAvailable(bool avoidException);
 
-SaplingHistoryKeys GetSaplingHistoryKeys()
-{
-  AssertLockHeld(pwalletMain->cs_wallet);
-  SaplingHistoryKeys keys;
-  std::set<libzcash::SaplingPaymentAddress> addresses;
-  pwalletMain->GetSaplingPaymentAddresses(addresses);
-  for (const auto& addr : addresses) {
-    libzcash::SaplingExtendedSpendingKey extsk;
-    if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-      const libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
-      SaplingHistoryKeys::SpendableAddress a;
-      a.addr = addr;
-      a.ivk = fvk.in_viewing_key();
-      a.ovk = fvk.ovk;
-      // Left zero when the wallet doesn't know addr, exactly as GetNullifiersForAddresses does.
-      pwalletMain->GetSaplingIncomingViewingKey(addr, a.walletIvk);
-      keys.spendable.push_back(a);
-    }
-  }
-  return keys;
-}
-
-bool SaplingHistoryKeys::IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const
-{
-  AssertLockHeld(pwalletMain->cs_wallet);
-  if (!noteNullifiers) {
-    noteNullifiers = std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>();
-    for (const auto& p : pwalletMain->mapWallet) {
-      for (const auto& item : p.second.mapSaplingNoteData) {
-        if (item.second.nullifier)
-          noteNullifiers->insert(std::make_pair(item.second.ivk, *item.second.nullifier));
-      }
-    }
-  }
-  return noteNullifiers->count(std::make_pair(addrIvk, nullifier)) > 0;
-}
-
-namespace {
-
-// One output's decryptions under incoming viewing keys, each key tried at most once. Two distinct ivks can't
-// both decrypt an output: the cm check recomputes pk_d = [ivk]g_d, and ivks are canonical scalars below the
-// order of g_d (ka_agree and ivk_to_pkd reject the rest), so a second one would need a NoteCommit collision.
-// So once it is known which ivk decrypts the output (the wallet's note data records the one FindMySaplingNotes
-// found, with its plaintext normally cached), no other ivk is tried.
-class OutputDecryptions {
-  const OutputDescription& output;
-  const SaplingNoteData* nd;
-  std::map<libzcash::SaplingIncomingViewingKey, boost::optional<SaplingNoteCache>> tried;
-
-  boost::optional<SaplingNoteCache> Decrypt(const libzcash::SaplingIncomingViewingKey& ivk) const {
-    auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
-    if (!both)
-      return boost::none;
-    return SaplingNoteCache{ both->second, both->first.memo() };
-  }
-  // The plaintext under the ivk the wallet recorded for this output, decrypting it if it isn't cached.
-  const boost::optional<SaplingNoteCache>& Recorded() {
-    auto it = tried.find(nd->ivk);
-    if (it == tried.end())
-      it = tried.insert(std::make_pair(nd->ivk, nd->cachedPlaintext ? nd->cachedPlaintext : Decrypt(nd->ivk))).first;
-    return it->second;
-  }
-public:
-  OutputDecryptions(const CWalletTx& wtx, uint32_t n) : output(wtx.vShieldedOutput[n]), nd(nullptr) {
-    auto it = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), n));
-    if (it != wtx.mapSaplingNoteData.end())
-      nd = &it->second;
-  }
-  const SaplingNoteCache* Get(const libzcash::SaplingIncomingViewingKey& ivk) {
-    auto it = tried.find(ivk);
-    if (it == tried.end()) {
-      boost::optional<SaplingNoteCache> result;
-      if (nd && nd->ivk == ivk) {
-        result = Recorded();
-      } else if (!(nd && Recorded())) {
-        // Not a wallet note, or one whose recorded ivk doesn't decrypt it: try this key.
-        result = Decrypt(ivk);
-      }
-      it = tried.insert(std::make_pair(ivk, result)).first;
-    }
-    return it->second ? &*it->second : nullptr;
-  }
-};
-
-// What an outCiphertext tells the sender holding ovk: the note's plaintext and the address it was sent to.
-boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>> DecryptSent(
-    const OutputDescription& outputDesc, const uint256& ovk)
-{
-  auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-          outputDesc.outCiphertext,ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
-  if (!opt)
-    return boost::none;
-  auto opt_unwrapped = opt.get();
-  auto pt = libzcash::SaplingNotePlaintext::decrypt(
-          outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
-  if (!pt)
-    return boost::none;
-  auto pt_unwrapped = pt.get();
-  return std::make_pair(pt_unwrapped, libzcash::SaplingPaymentAddress(pt_unwrapped.d, opt_unwrapped.pk_d));
-}
-
-}
-
 void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -190,29 +87,18 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
       const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
       UniValue obj(UniValue::VOBJ);
 
-      // Lookups by find(), not operator[]: that inserted an entry for every other wallet's nullifier it met
+      SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
-      // (and could insert an empty note into a wallet tx).
+
       if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
-        const SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes.at(spendDesc.nullifier);
+        const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
-        const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash);
-        auto ndIt = parent->mapSaplingNoteData.find(op);
-        // The spent note's plaintext, from the wallet's cache when it has it.
-        boost::optional<SaplingNoteCache> pt;
-        if (ndIt != parent->mapSaplingNoteData.end() && op.n < parent->vShieldedOutput.size()) {
-          const SaplingNoteData& nd = ndIt->second;
-          if (nd.cachedPlaintext) {
-            pt = nd.cachedPlaintext;
-          } else {
         const OutputDescription& output = parent->vShieldedOutput[op.n];
-            auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
+        auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
-            if (both)
+        auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
-              pt = SaplingNoteCache{ both->second, both->first.memo() };
-          }
-        }
 
         if (pt) {
-          const libzcash::SaplingNote& note = pt->note;
+          auto note = pt.get();
-          const libzcash::SaplingPaymentAddress address(note.d, note.pk_d);
+          auto pa = nd.ivk.address(note.d);
+          auto address = pa.get();
           shieldedSpendsSpentCount++;
           totalSpends += CAmount(-note.value());
           obj.push_back(Pair("address",EncodePaymentAddress(address)));
@@ -246,7 +132,7 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
 }
 
 
-void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
+void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
 
   if(fZdebug)
     fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
@@ -302,18 +188,22 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
   UniValue zsReceived(UniValue::VARR);
   if (isZsAddress || !filterByAddress) {
     for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
+      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
       UniValue obj(UniValue::VOBJ);
       bool changeTx = false;
-      //Decrypt sapling incoming commitments using IVK: once per key, not once per address
+      //Decrypt sapling incoming commitments using IVK
-      OutputDecryptions decryptions(wtx, i);
+      std::set<libzcash::SaplingPaymentAddress> addresses;
-      for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
+      pwalletMain->GetSaplingPaymentAddresses(addresses);
-        const libzcash::SaplingPaymentAddress& addr = a.addr;
+      for (auto addr : addresses) {
-        {
+        libzcash::SaplingExtendedSpendingKey extsk;
-          const SaplingNoteCache* pt = decryptions.Get(a.ivk);
+        if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
+          auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                  outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
 
           if (pt) {
-            const libzcash::SaplingNote& note = pt->note;
+            auto note = pt.get();
-            auto memo = pt->memo;
+            auto pt_unwrapped = pt.get();
+            auto memo = pt_unwrapped.memo();
             obj.push_back(Pair("address",EncodePaymentAddress(addr)));
             obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
             obj.push_back(Pair("memo", HexStr(memo)));
@@ -329,8 +219,10 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
 
             //Check Change Status
             if (wtx.vShieldedSpend.size()!=0) {
+              std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
+              nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
               BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                if (keys.IsNullifierOfAddress(a.walletIvk, spendDesc.nullifier)) {
+                if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
                     changeTx = true;
                 }
               }
@@ -351,7 +243,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
 }
 
 
-void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
+void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
 
@@ -404,13 +296,6 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
   //All Shielded Sends in the transaction
   UniValue zsSends(UniValue::VARR);
   if (isZsAddress || !filterByAddress) {
-    //The t to z shielding ovk comes from the HD seed: derive it once per tx, not once per output
-    boost::optional<uint256> seedOvk;
-    if (wtx.vShieldedSpend.size()==0 && wtx.vShieldedOutput.size()!=0) {
-      HDSeed seed;
-      if (pwalletMain->GetHDSeedForDerivation(seed))
-        seedOvk = ovkForShieldingFromTaddr(seed);
-    }
     for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
       const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
       shieldedOutputCount++;
@@ -419,14 +304,20 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 
       //Decrypt sapling outgoing t to z transaction using HDseed
       if (wtx.vShieldedSpend.size()==0) {
-        if (seedOvk) {
+        HDSeed seed;
-          {
+        if (pwalletMain->GetHDSeedForDerivation(seed)) {
-            auto sent = DecryptSent(outputDesc, *seedOvk);
+          auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
+                  outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
 
-            if (sent) {
+          if (opt) {
+            auto opt_unwrapped = opt.get();
+            auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                    outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
+
+            if (pt) {
               shieldedOutputDecryptedCount++;
-              const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
+              auto pt_unwrapped = pt.get();
-              const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
+              libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
               obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
               obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
               obj.push_back(Pair("shieldedOutputIndex",i));
@@ -439,22 +330,26 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
           }
         }
 
-      //attempt Decryption of Outgoing Sapling using wallet extended spending keys: once per ovk, not once per address
+      //attempt Decryption of Outgoing Sapling using wallet extended spending keys
       } else {
-        std::map<uint256, boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>>> sentByOvk;
+        std::set<libzcash::SaplingPaymentAddress> addresses;
-        for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
+        pwalletMain->GetSaplingPaymentAddresses(addresses);
-          {
+        for (auto addr : addresses) {
-            auto tried = sentByOvk.find(a.ovk);
+          libzcash::SaplingExtendedSpendingKey extsk;
-            if (tried == sentByOvk.end())
+          if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-              tried = sentByOvk.insert(std::make_pair(a.ovk, DecryptSent(outputDesc, a.ovk))).first;
+            auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-            const auto& sent = tried->second;
+                    outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
 
-            {
+            if (opt) {
-              if (sent) {
+              auto opt_unwrapped = opt.get();
-                const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
+              auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                      outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
+
+              if (pt) {
+                auto pt_unwrapped = pt.get();
                 auto memo = pt_unwrapped.memo();
                 shieldedOutputDecryptedCount++;
-                const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
+                libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
                 obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
                 obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
                 obj.push_back(Pair("memo", HexStr(memo)));
@@ -470,10 +365,10 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 
                 //Check Change Status
                 if (wtx.vShieldedSpend.size()!=0) {
-                  libzcash::SaplingIncomingViewingKey sentIvk;
+                  std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
-                  pwalletMain->GetSaplingIncomingViewingKey(sentAddr, sentIvk);
+                  nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
                   BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                    if (keys.IsNullifierOfAddress(sentIvk, spendDesc.nullifier)) {
+                    if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
                         changeTx = true;
                     }
                   }
@@ -502,7 +397,7 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 }
 
 
-void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys) {
+void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
 
@@ -574,7 +469,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
   }
   // Get Received
   if (returnType == 0 || returnType == 2) {
-    zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, keys);
+    zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
     if (!fBool || totalReceived != 0) {
       tx.push_back(Pair("received",received));
     }
@@ -584,7 +479,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
   if (returnType == 0 || returnType == 3) {
     //Only include sends if there are spends that belong to the wallet.
     if (totalSpends != 0 || fBool) {
-      zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, keys);
+      zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
     }
     if (!fBool || totalSends != 0) {
       tx.push_back(Pair("sends",sends));

src/wallet/rpchushwallet.h

@@ -13,35 +13,10 @@ struct balancestruct {
   CAmount immature;
 };
 
-/** The wallet's Sapling keys as the history helpers use them, gathered once per RPC call. The helpers used to
- *  look them up for every output and every address, re-deriving each full viewing key (an elliptic-curve
- *  operation), and walked every note in the wallet for each change check. Build it with pwalletMain->cs_wallet
- *  held, and keep holding it while the helpers run: that keeps the wallet's keys and notes fixed. The one thing
- *  it can't stop is the walletpassphrase timer relocking the wallet mid-call; the call then reports the keys as
- *  they were when it started, where the old per-output lookups switched over partway. */
-struct SaplingHistoryKeys {
-    struct SpendableAddress {
-        libzcash::SaplingPaymentAddress addr;
-        libzcash::SaplingIncomingViewingKey ivk;         //!< from its spending key: what received outputs are decrypted with
-        uint256 ovk;                                     //!< from its spending key: what outCiphertexts are decrypted with
-        libzcash::SaplingIncomingViewingKey walletIvk;   //!< GetSaplingIncomingViewingKey(addr), as the change check looked it up
-    };
-    //! Every wallet address whose spending key is available (never while the wallet is locked), in
-    //! GetSaplingPaymentAddresses order.
-    std::vector<SpendableAddress> spendable;
-    //! Whether GetNullifiersForAddresses({addr}) would contain (addr, nullifier), given addr's
-    //! GetSaplingIncomingViewingKey result (zero when the wallet doesn't know addr): nullifier belongs to a
-    //! wallet note received with that ivk. Walks the wallet's notes once, on first use.
-    bool IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const;
-private:
-    mutable boost::optional<std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>> noteNullifiers;
-};
-SaplingHistoryKeys GetSaplingHistoryKeys();
-
 void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress);
-void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
+void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress);
-void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
+void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress);
-void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys);
+void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType);
 
 
 #endif //HUSH_WALLET_RPCWALLET_H

src/wallet/rpcwallet.cpp

@@ -3032,13 +3032,11 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
     }
 
     //Create Ordered List
-    // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
+    map<int64_t,CWalletTx> orderedTxs;
-    // As before, the first tx inserted for an nOrderPos wins.
-    map<int64_t,const CWalletTx*> orderedTxs;
     for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
       const uint256& wtxid = it->first;
       const CWalletTx& wtx = (*it).second;
-      orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
+      orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
 
       unsigned int txType = 0;
       // 0 Unassigend
@@ -3230,11 +3228,10 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
         }
 
 
-        const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
         uint64_t t = GetTime();
-        for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
+        for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
         {
-            const CWalletTx& wtx = *(*it).second;
+            const CWalletTx& wtx = (*it).second;
 
             if (!CheckFinalTx(wtx))
                 continue;
@@ -3251,7 +3248,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
               continue;
             }
 
-            zsWalletTxJSON(wtx, trans, "*", false, 0, historyKeys);
+            zsWalletTxJSON(wtx, trans, "*", false, 0);
             if (trans.size() >= nCount) break;
 
         }
@@ -3518,20 +3515,17 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
         throw runtime_error("Filter must be greater that 0.");
 
     //Created Ordered Transaction Map
-    // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
+    map<int64_t,CWalletTx> orderedTxs;
-    // As before, the first tx inserted for an nOrderPos wins.
-    map<int64_t,const CWalletTx*> orderedTxs;
     for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
       const CWalletTx& wtx = (*it).second;
-      orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
+      orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
     }
 
 
-    const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
     uint64_t t = GetTime();
     //Reverse Iterate thru transactions
-    for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
+    for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
-        const CWalletTx& wtx = *(*it).second;
+        const CWalletTx& wtx = (*it).second;
         bool includeTransaction = true;
 
         //Excude transactions with less confirmations than required
@@ -3552,7 +3546,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
         }
 
         if (includeTransaction) {
-          zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, historyKeys);
+          zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2);
         }
 
         if (ret.size() > nCount) break;
@@ -3704,20 +3698,17 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
         throw runtime_error("Filter must be greater that 0.");
 
     //Created Ordered Transaction Map
-    // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
+    map<int64_t,CWalletTx> orderedTxs;
-    // As before, the first tx inserted for an nOrderPos wins.
-    map<int64_t,const CWalletTx*> orderedTxs;
     for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
       const CWalletTx& wtx = (*it).second;
-      orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
+      orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
     }
 
 
-    const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
     uint64_t t = GetTime();
     //Reverse Iterate thru transactions
-    for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
+    for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
-      const CWalletTx& wtx = *(*it).second;
+      const CWalletTx& wtx = (*it).second;
 
       if (!CheckFinalTx(wtx))
           continue;
@@ -3740,7 +3731,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
       if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter)
           continue;
 
-      zsWalletTxJSON(wtx, ret, "*", false, 0, historyKeys);
+      zsWalletTxJSON(wtx, ret, "*", false, 0);
 
 
       if (ret.size() >= nCount) break;
@@ -4318,7 +4309,7 @@ UniValue z_listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
         for (auto & entry : saplingEntries) {
             UniValue obj(UniValue::VOBJ);
 
-            int nHeight   = HeightFromDepth(entry.confirmations);
+            int nHeight   = tx_height(entry.op.hash);
             int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
 
             // Only return notarized results when minconf>1
@@ -4651,7 +4642,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
     if (boost::get<libzcash::SaplingPaymentAddress>(&zaddr) != nullptr) {
         for (SaplingNoteEntry & entry : saplingEntries) {
             UniValue obj(UniValue::VOBJ);
-            int nHeight   = HeightFromDepth(entry.confirmations);
+            int nHeight   = tx_height(entry.op.hash);
             int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
             // Only return notarized results when minconf>1
             if (nMinDepth > 1 && dpowconfs == 1)
@@ -4671,7 +4662,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
             }
             obj.push_back(Pair("outindex", (int)entry.op.n));
             obj.push_back(Pair("rawconfirmations", entry.confirmations));
-            const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash);
+            auto wtx = pwalletMain->mapWallet.at(entry.op.hash); //.ToString());
             obj.push_back(Pair("time", wtx.GetTxTime()));
 
             obj.push_back(Pair("confirmations", dpowconfs));
@@ -5385,7 +5376,7 @@ UniValue z_sendmany(const UniValue& params, bool fHelp, const CPubKey& mypk)
     // sort notes from largest to smallest, which means
     // we will spend the largest first
     std::sort(saplingEntries.begin(), saplingEntries.end(),
-        [](const SaplingNoteEntry& i, const SaplingNoteEntry& j) -> bool {
+        [](SaplingNoteEntry i, SaplingNoteEntry j) -> bool {
             return i.note.value() > j.note.value();
         });
 

src/wallet/wallet.cpp

@@ -43,9 +43,7 @@
 #include "wallet/asyncrpcoperation_autoshieldcoinbase.h"
 #include <random>
 #include <limits>
-#include <system_error>
 #include <thread>
-#include <chrono>
 #include <atomic>
 #include <mutex>
 #include "zcash/zip32.h"
@@ -77,8 +75,8 @@ bool fPayAtLeastCustomFee = true;
 CBlockIndex *hush_chainactive(int32_t height);
 extern int32_t HUSH_LOADINGBLOCKS;
 int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
+int tx_height( const uint256 &hash );
 bool fTxDeleteEnabled = false;
-bool fCheckNoteCache = false;
 bool fTxConflictDeleteEnabled = false;
 int fDeleteInterval = DEFAULT_TX_DELETE_INTERVAL;
 unsigned int fDeleteTransactionsAfterNBlocks = DEFAULT_TX_RETENTION_BLOCKS;
@@ -577,23 +575,6 @@ void CWallet::ChainTip(const CBlockIndex *pindex,
         DecrementNoteWitnesses(pindex);
         UpdateNullifierNoteMapForBlock(pblock);
     }
-
-    // Track the block the wallet's state reflects, advancing only along blocks delivered in order:
-    // a gap (the startup rescan was interrupted, so the wallet sits at its checkpoint behind the
-    // notifier) pins the pointer, and SetBestChain keeps stamping the checkpoint.
-    LOCK(cs_wallet);
-    if (added) {
-        if (pindexWalletTip == nullptr || pindex->pprev == pindexWalletTip)
-            pindexWalletTip = pindex;
-    } else if (pindex == pindexWalletTip) {
-        pindexWalletTip = pindex->pprev;
-    }
-}
-
-void CWallet::SetWalletTip(const CBlockIndex* pindex)
-{
-    LOCK(cs_wallet);
-    pindexWalletTip = pindex;
 }
 
 void CWallet::RunSaplingSweep(int blockHeight) {
@@ -820,22 +801,10 @@ bool CWallet::CommitAutomatedTx(const CTransaction& tx) {
 
 void CWallet::SetBestChain(const CBlockLocator& loc)
 {
-    // FlushStateToDisk passes the chain tip, but ThreadNotifyWallets delivers blocks to the wallet
-    // asynchronously and can lag it. Stamping a locator ahead of what the wallet has processed makes
-    // the next start skip those blocks for good (no rescan, no notification): notes received in them
-    // go missing and spends in them go unseen. So stamp the block the wallet's state reflects (seeded
-    // by init before the notifier starts, then advanced by ChainTip). Init's own calls, made before
-    // it is seeded, pass the right locator themselves.
-    CBlockLocator locator = loc;
-    {
-        LOCK2(cs_main, cs_wallet);
-        if (pindexWalletTip != nullptr)
-            locator = chainActive.GetLocator(pindexWalletTip);
-    }
     // Default ctor => fFlushOnClose=true => ~CDB runs a full BDB txn_checkpoint over the entire
     // cache. Fine for the hourly/shutdown callers; ruinous inside a scan (see SetBestChainNoFlush).
     CWalletDB walletdb(strWalletFile);
-    SetBestChainINTERNAL(walletdb, locator);
+    SetBestChainINTERNAL(walletdb, loc);
 }
 
 bool CWallet::SetBestChainNoFlush(const CBlockLocator& loc)
@@ -1263,16 +1232,24 @@ void ClearSingleNoteWitnessCache(NoteData* nd)
     nd->witnessRootValidated = false;
 }
 
-// A note without a nullifier is never extended by BuildWitnessCache (its pending-note gate requires one),
+int CWallet::SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight)
-// so it doesn't pull the rebuild's start height down.
-int CWallet::SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight)
 {
-    if (nullifier && GetSaplingSpendDepth(*nullifier) <= WITNESS_CACHE_SIZE) {
+    if (GetSaplingSpendDepth(nullifier) <= WITNESS_CACHE_SIZE) {
         nMinimumHeight = min(nWitnessHeight, nMinimumHeight);
     }
     return nMinimumHeight;
 }
 
+int CWallet::SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight)
+{
+    // No nullifier => an incoming-viewing-key-only note (z_importviewingkey without the full
+    // viewing key). Spend depth is unknowable, so treat it as unspent and keep its witness.
+    if (!nullifier) {
+        return min(nWitnessHeight, nMinimumHeight);
+    }
+    return SaplingWitnessMinimumHeight(*nullifier, nWitnessHeight, nMinimumHeight);
+}
+
 int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessOnly)
 {
   LOCK2(cs_main, cs_wallet);
@@ -1299,14 +1276,6 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
         uint256 blockRoot;
         uint256 witnessRoot;
 
-        //A note found with an incoming viewing key only (z_importviewingkey) can't be spent, so it needs
-        //no witness, and it never gets a nullifier: deriving one takes the full viewing key, which
-        //UpdateSaplingNullifierNoteMapWithTx skips without. Seeding it here meant clearing and reseeding
-        //it from disk on every call (the no-nullifier rule below), then aborting on the missing nullifier
-        //the first time it was confirmed. If its spending key is imported later, the rule below seeds it.
-        if (!HaveSaplingFullViewingKey(nd->ivk))
-          continue;
-
         if (!nd->nullifier)
           ::ClearSingleNoteWitnessCache(nd);
 
@@ -1518,25 +1487,17 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
   work.reserve(pending.size());
   for (const PendingNote& p : pending) work.push_back(p.nd);
 
-  // Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is logged and has its
+  // Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is worth parallelizing
-  // anchors checked; routine 1-block tip extension stays quiet to avoid per-block log spam.
+  // and logging; routine 1-block tip extension runs serially to avoid per-block thread-spawn
+  // overhead and log spam.
   const bool bulkRebuild = (rangeLen > 100);
 
-  // Parallelize whenever the extension itself is expensive, not only for a bulk range: a routine
-  // 1-block extension appends each of the block's commitments to every lagging note, so a large
-  // wallet and a busy block (13k notes x 20 commitments) is seconds of hashing under
-  // cs_main/cs_wallet. Spawning the threads costs ~1ms, so only small jobs stay serial.
-  uint64_t nCms = 0;
-  for (const std::vector<uint256>& cms : blockCms)
-    nCms += cms.size();
-  const bool fParallel = bulkRebuild || (uint64_t)work.size() * nCms >= 2000;
-
   int nPar = (int)GetArg("-witnessbuildthreads", 0);
   if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
   if (nPar <= 0) nPar = 1;
   if (nPar > (int)work.size()) nPar = (int)work.size();
   if (nPar < 1) nPar = 1;
-  if (!fParallel) nPar = 1;
+  if (!bulkRebuild) nPar = 1;
 
   std::atomic<bool> failed(false);
   std::mutex failMtx;
@@ -1548,27 +1509,14 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
   // clone-every-block path for A/B verification.
   bool fastRebuild = GetBoolArg("-witnessfastrebuild", true);
 
-  // A bulk rebuild reports how many notes failed to reach the canonical anchor. root() walks the
-  // full depth-32 path (~2.4ms), so the check runs here on the workers and never on the routine
-  // per-block path, where it used to hold cs_main/cs_wallet for ~30s per block on a ~13k-note
-  // wallet purely to feed a log line that path never prints.
-  const uint256 canonicalRoot = pindex->hashFinalSaplingRoot;
-  std::atomic<int> nUnrecoverable(0);
-  auto checkRoot = [&](const SaplingNoteData* nd) {
-    if (bulkRebuild && (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot))
-      nUnrecoverable++;
-  };
-
   auto worker = [&](int tid) {
     try {
       for (size_t k = (size_t)tid; k < work.size(); k += (size_t)nPar) {
         SaplingNoteData* nd = work[k];
         int startH = nd->witnessHeight;
         int nBlocks = tipHeight - startH;
-        if (nBlocks <= 0) {
+        if (nBlocks <= 0)
-          checkRoot(nd);
           continue;
-        }
 
         if (!fastRebuild || nBlocks <= CACHE) {
           // Reference / shallow path: clone every block (preserves pre-existing older snapshots).
@@ -1609,7 +1557,6 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
             nd->witnesses.pop_back();
           nd->witnessHeight = tipHeight;
         }
-        checkRoot(nd);
       }
     } catch (const std::exception& e) {
       std::lock_guard<std::mutex> lk(failMtx);
@@ -1624,18 +1571,9 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
   if (nPar <= 1) {
     worker(0);
   } else {
-    // If the OS refuses a thread (task limit, memory), run the remaining shards here instead of
-    // letting the exception unwind past joinable threads (std::terminate). Each shard still runs
-    // exactly once, so the result is unchanged.
     std::vector<std::thread> threads;
     threads.reserve(nPar);
-    int t = 0;
+    for (int t = 0; t < nPar; t++) threads.emplace_back(worker, t);
-    try {
-      for (; t < nPar; t++) threads.emplace_back(worker, t);
-    } catch (const std::system_error& e) {
-      LogPrintf("%s: could only start %d of %d witness threads (%s); running the rest inline\n", __func__, t, nPar, e.what());
-    }
-    for (int u = t; u < nPar; u++) worker(u);
     for (std::thread& th : threads) th.join();
   }
   int64_t tExtend = GetTimeMicros() - e0;
@@ -1650,16 +1588,20 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
   // is unrecoverable here: it is left flagged (the GetSaplingNoteWitnesses majority-anchor guard
   // skips it for spends) and reported, rather than spun on indefinitely. witnessRootValidated is
   // in-memory only, so a fresh validation pass still runs on each restart and after any reorg
-  // (DecrementNoteWitnesses clears it), keeping the heal self-correcting. The unrecoverable count
+  // (DecrementNoteWitnesses clears it), keeping the heal self-correcting.
-  // itself comes from checkRoot() on the workers, and only for a bulk rebuild.
   if (!work.empty()) {
-    for (SaplingNoteData* nd : work)
+    const uint256& canonicalRoot = pindex->hashFinalSaplingRoot;
+    int nUnrecoverable = 0;
+    for (SaplingNoteData* nd : work) {
+      if (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot)
+        nUnrecoverable++;
       nd->witnessRootValidated = true;
+    }
     if (bulkRebuild) {
       LogPrintf("%s: rebuilt %u note witness cache(s) to height %d in %ldms using %d thread(s)%s\n",
                 __func__, (unsigned)work.size(), tipHeight, (long)((tRead + tExtend) / 1000), nPar,
-                nUnrecoverable.load()
+                nUnrecoverable
-                  ? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable.load()).c_str()
+                  ? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable).c_str()
                   : "");
     }
   }
@@ -1981,8 +1923,6 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
         bool fUpdated = false;
         if (!fInsertedNew)
         {
-            // Re-mined in a different block (a reorg): its notes sit at new tree positions.
-            bool fSameBlock = wtx.hashBlock.IsNull() || wtxIn.hashBlock.IsNull() || wtxIn.hashBlock == wtx.hashBlock;
             // Merge
             if (!wtxIn.hashBlock.IsNull() && wtxIn.hashBlock != wtx.hashBlock)
             {
@@ -1995,7 +1935,7 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
                 wtx.nIndex = wtxIn.nIndex;
                 fUpdated = true;
             }
-            if (UpdatedNoteData(wtxIn, wtx, fSameBlock)) {
+            if (UpdatedNoteData(wtxIn, wtx)) {
                 fUpdated = true;
             }
             if (wtxIn.fFromMe && wtxIn.fFromMe != wtx.fFromMe)
@@ -2032,52 +1972,14 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
     return true;
 }
 
-// True if both maps hold the same notes with identical persisted state (every field SaplingNoteData
+bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx)
-// serializes). SaplingNoteData::operator== ignores the witnesses, so it cannot tell a real change
-// from a redundant update.
-// Copies the decrypted plaintext of each incoming note into the matching (same outpoint and ivk) entry that
-// lacks one. UpdatedNoteData uses it when it keeps the wallet's entries: a replay or rescan has just
-// decrypted the notes (FindMySaplingNotes), and entries loaded from wallet.dat have no plaintext. The cache
-// is in memory only and outside every comparison, so this never makes a tx look changed.
-static void KeepNotePlaintexts(const mapSaplingNoteData_t& incoming, mapSaplingNoteData_t& wallet)
-{
-    for (const mapSaplingNoteData_t::value_type& in : incoming) {
-        if (!in.second.cachedPlaintext)
-            continue;
-        mapSaplingNoteData_t::iterator it = wallet.find(in.first);
-        if (it != wallet.end() && !it->second.cachedPlaintext && it->second.ivk == in.second.ivk)
-            it->second.cachedPlaintext = in.second.cachedPlaintext;
-    }
-}
-
-static bool SaplingNoteDataIdentical(const mapSaplingNoteData_t& a, const mapSaplingNoteData_t& b)
-{
-    if (a.size() != b.size())
-        return false;
-    for (const mapSaplingNoteData_t::value_type& item : a) {
-        mapSaplingNoteData_t::const_iterator it = b.find(item.first);
-        if (it == b.end())
-            return false;
-        const SaplingNoteData& x = item.second;
-        const SaplingNoteData& y = it->second;
-        if (!(x.ivk == y.ivk) || x.nullifier != y.nullifier || x.witnessHeight != y.witnessHeight || x.witnesses != y.witnesses)
-            return false;
-    }
-    return true;
-}
-
-bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock)
 {
     bool unchangedSaplingFlag = (wtxIn.mapSaplingNoteData.empty() || wtxIn.mapSaplingNoteData == wtx.mapSaplingNoteData);
-    if (unchangedSaplingFlag) {
+    if (!unchangedSaplingFlag) {
-        KeepNotePlaintexts(wtxIn.mapSaplingNoteData, wtx.mapSaplingNoteData);
-        return false;
-    }
-
         auto tmp = wtxIn.mapSaplingNoteData;
         // Ensure we keep any cached witnesses we may already have
 
-    for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
+        for (const std::pair <SaplingOutPoint, SaplingNoteData> nd : wtx.mapSaplingNoteData) {
             if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
                 tmp.at(nd.first).witnesses.assign(
                         nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
@@ -2085,39 +1987,14 @@ bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSame
                 //Copying it unconditionally (when witnesses are NOT copied) advances the height past
                 //the actual witness state for a whole tx's notes at once = the batch desync originator.
                 tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
-            //The nullifier is derived from the note and its witness position, so it travels with the
-            //witnesses too. wtxIn comes from FindMySaplingNotes, which never sets it; dropping it made
-            //VerifyAndSetInitialWitness clear and reseed the witnesses just carried over, so replaying
-            //or rescanning already-witnessed txs forced a rebuild of every one of their notes. A tx
-            //re-mined in another block is the exception: its notes moved, so the old nullifier is
-            //wrong and dropping it still reseeds them at the new position.
-            if (nd.second.nullifier && fSameBlock)
-                tmp.at(nd.first).nullifier = nd.second.nullifier;
             }
         }
 
-    // Keep a decrypted plaintext the wallet already holds (in memory only; a pure function of the
-    // outpoint and ivk) when the incoming note data lacks one.
-    for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
-        mapSaplingNoteData_t::iterator it = tmp.find(nd.first);
-        if (it != tmp.end() && !it->second.cachedPlaintext && nd.second.cachedPlaintext && it->second.ivk == nd.second.ivk)
-            it->second.cachedPlaintext = nd.second.cachedPlaintext;
-    }
-
-    // A replayed or rescanned tx usually carries nothing new; don't rewrite it to disk. Do have
-    // VerifyAndSetInitialWitness re-check the kept witnesses against the active chain once (the
-    // flag is in memory only): the chain may have moved under them, and it keeps a rescan able to
-    // find and reseed a witness that went bad, without rebuilding the healthy ones.
-    if (SaplingNoteDataIdentical(tmp, wtx.mapSaplingNoteData)) {
-        for (mapSaplingNoteData_t::value_type& item : wtx.mapSaplingNoteData)
-            item.second.witnessRootValidated = false;
-        KeepNotePlaintexts(tmp, wtx.mapSaplingNoteData);
-        return false;
-    }
-
         // Now copy over the updated note data
         wtx.mapSaplingNoteData = tmp;
-    return true;
+    }
+
+    return !unchangedSaplingFlag;
 }
 
 /**
@@ -2290,24 +2167,21 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
 
     // Protocol Spec: 4.19 Block Chain Scanning (Sapling)
     for (uint32_t i = 0; i < nZouts; ++i) {
-        const OutputDescription& output = tx.vShieldedOutput[i];
+        const OutputDescription output = tx.vShieldedOutput[i];
         bool found = false;
         for (auto it = mapSaplingFullViewingKeys.begin(); it != mapSaplingFullViewingKeys.end(); ++it) {
             SaplingIncomingViewingKey ivk = it->first;
-            auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
+            auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
             if (result) {
-                // The note's (d, pk_d) is the address it was sent to, exactly what ivk.address(d) would
+                auto address = ivk.address(result.get().d);
-                // derive again: decrypt_with_note already computed pk_d from this ivk and d.
+                if (address && mapSaplingIncomingViewingKeys.count(address.get()) == 0) {
-                const libzcash::SaplingPaymentAddress address(result->second.d, result->second.pk_d);
+                    viewingKeysToAdd[address.get()] = ivk;
-                if (mapSaplingIncomingViewingKeys.count(address) == 0) {
-                    viewingKeysToAdd[address] = ivk;
                 }
                 // We don't cache the nullifier here as computing it requires knowledge of the note position
                 // in the commitment tree, which can only be determined when the transaction has been mined.
                 SaplingOutPoint op {hash, i};
                 SaplingNoteData nd;
                 nd.ivk = ivk;
-                nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
                 noteData.insert(std::make_pair(op, nd));
                 found = true;
                 break;
@@ -2316,7 +2190,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
         if (!found) {
             for (auto it = mapSaplingIncomingViewingKeys.begin(); it != mapSaplingIncomingViewingKeys.end(); ++it) {
                 SaplingIncomingViewingKey ivk = it-> second;
-                auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
+                auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
                 if (!result) {
                     continue;
                 }
@@ -2325,7 +2199,6 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
                 SaplingOutPoint op {hash, i};
                 SaplingNoteData nd;
                 nd.ivk = ivk;
-                nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
                 noteData.insert(std::make_pair(op, nd));
                 break;
             }
@@ -5746,137 +5619,6 @@ bool CMerkleTx::AcceptToMemoryPool(bool fLimitFree, bool fRejectAbsurdFee)
     return ::AcceptToMemoryPool(mempool, state, *this, fLimitFree, NULL, fRejectAbsurdFee);
 }
 
-unsigned int CWallet::FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes)
-{
-    AssertLockHeld(cs_wallet);
-
-    // Each decryption is independent: it reads its own output and writes only its own note's
-    // cachedPlaintext. So a big batch (a whole wallet after a restart, ~0.5ms a note) is split across
-    // threads the way BuildWitnessCache's Phase 2 is, while this thread keeps cs_wallet held. Workers
-    // must not lock, or even assert a lock: under DEBUG_LOCKORDER a fresh thread has no lock stack.
-    // Spawning costs ~1ms, so small batches stay serial.
-    int nPar = 1;
-    if (notes.size() >= 128) {
-        nPar = (int)GetArg("-witnessbuildthreads", 0);
-        if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
-        nPar = std::max(1, std::min(nPar, (int)(notes.size() / 64)));
-    }
-
-    std::vector<unsigned int> filled(nPar, 0);
-    auto worker = [&](int tid) {
-        for (size_t k = (size_t)tid; k < notes.size(); k += (size_t)nPar) {
-            const SaplingNoteToDecrypt& n = notes[k];
-            try {
-                auto result = SaplingNotePlaintext::decrypt_with_note(
-                    n.output->encCiphertext, n.nd->ivk, n.output->ephemeralKey, n.output->cm);
-                if (result) {
-                    n.nd->cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
-                    filled[tid]++;
-                }
-            } catch (...) {
-                // Left unfilled, exactly like a note that doesn't decrypt; callers handle that. Nothing may
-                // escape a std::thread.
-            }
-        }
-    };
-
-    if (nPar <= 1) {
-        worker(0);
-    } else {
-        // If the OS refuses a thread, run the remaining shards here instead of letting the exception
-        // unwind past joinable threads (std::terminate). Each shard still runs exactly once.
-        std::vector<std::thread> threads;
-        threads.reserve(nPar);
-        int t = 0;
-        try {
-            for (; t < nPar; t++) threads.emplace_back(worker, t);
-        } catch (const std::exception& e) {
-            LogPrintf("%s: could only start %d of %d threads (%s); decrypting the rest inline\n", __func__, t, nPar, e.what());
-        }
-        for (int u = t; u < nPar; u++) worker(u);
-        for (std::thread& th : threads) th.join();
-    }
-
-    unsigned int nFilled = 0;
-    for (unsigned int f : filled)
-        nFilled += f;
-    nSaplingNoteCacheFills += nFilled;
-    nSaplingNoteCacheFillThreads = nPar;
-    LogPrint("bench", "%s: decrypted %u of %u notes using %d thread(s)\n", __func__, nFilled, (unsigned int)notes.size(), nPar);
-    return nFilled;
-}
-
-// wallet.dat doesn't store the plaintext, so without this the first balance call after a restart decrypts
-// the whole wallet under cs_main (~0.5ms a note). init runs it before RPC leaves warmup and before the
-// wallet notification thread starts; only the few RPCs allowed during warmup (key export, address
-// listing) can wait on cs_wallet meanwhile. Unlike GetFilteredNotes it never asserts, so a corrupt note
-// can't stop the node from starting (or from serving those RPCs); GetFilteredNotes still asserts on such
-// a note, as before.
-CWallet::SaplingNoteCacheWarmup CWallet::WarmSaplingNoteCache()
-{
-    const std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
-    LOCK(cs_wallet);
-    SaplingNoteCacheWarmup result;
-    std::vector<SaplingNoteToDecrypt> uncached;
-    for (auto& p : mapWallet) {
-        CWalletTx& wtx = p.second;
-        for (auto& item : wtx.mapSaplingNoteData) {
-            result.notes++;
-            if (item.second.cachedPlaintext)
-                continue;
-            if (item.first.n >= wtx.vShieldedOutput.size()) {
-                LogPrint("zrpcunsafe", "%s: note %s:%u has no such output\n", __func__, item.first.hash.ToString(), item.first.n);
-                result.noSuchOutput++;
-                continue;
-            }
-            uncached.push_back(SaplingNoteToDecrypt { &item.second, &wtx.vShieldedOutput[item.first.n] });
-        }
-    }
-    result.filled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
-    for (const SaplingNoteToDecrypt& n : uncached) {
-        if (!n.nd->cachedPlaintext) {
-            LogPrint("zrpcunsafe", "%s: a note of output %s does not decrypt under its ivk\n", __func__, n.output->cm.ToString());
-            result.undecryptable++;
-        }
-    }
-    const unsigned int nBad = result.noSuchOutput + result.undecryptable;
-    const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
-    LogPrintf("%s: decrypted %u of %u Sapling notes in %dms%s\n", __func__, result.filled, result.notes, nMs,
-              nBad ? strprintf("; %u could not be decrypted (-debug=zrpcunsafe lists them)", nBad) : "");
-    return result;
-}
-
-// -checknotecache: decrypts a cached note again, and logs and repairs any difference. GetFilteredNotes runs
-// it on every note that passes its tx, spent and locked filters, before the address filter, so a corrupt
-// cached address can't wrongly drop a note unchecked. A diagnostic for the cache: it costs the full
-// decryption the cache exists to avoid.
-static void CheckSaplingNoteCache(const SaplingOutPoint& op, SaplingNoteData& nd, const OutputDescription& output)
-{
-    auto fresh = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, nd.ivk, output.ephemeralKey, output.cm);
-    if (!fresh) {
-        LogPrintf("%s: note %s:%u has a cached plaintext but no longer decrypts\n", __func__, op.hash.ToString(), op.n);
-        return;
-    }
-    const SaplingNoteCache& cached = *nd.cachedPlaintext;
-    const libzcash::SaplingNote& note = fresh->second;
-    if (!(cached.note.d == note.d && cached.note.pk_d == note.pk_d && cached.note.r == note.r &&
-          cached.note.value() == note.value() && cached.memo == fresh->first.memo())) {
-        LogPrintf("%s: cached plaintext of note %s:%u differs from a fresh decryption; using the fresh one\n",
-                  __func__, op.hash.ToString(), op.n);
-        nd.cachedPlaintext = SaplingNoteCache{ note, fresh->first.memo() };
-    }
-}
-
-// Replaces tx_height() on the wallet's note paths. tx_height() re-read the tx from disk through
-// GetTransaction, once per wallet tx (or per returned note) under LOCK2, and on a -txindex=0 node it
-// never found a fully shielded tx, so those notes were silently skipped. The depth already locates
-// the block: GetDepthInMainChain is chainActive.Height() - height + 1 for an active-chain tx.
-int HeightFromDepth(int nDepth)
-{
-    AssertLockHeld(cs_main);
-    return nDepth > 0 ? chainActive.Height() - nDepth + 1 : 0;
-}
-
 /**
  * Find notes in the wallet filtered by payment address, min depth and ability to spend.
  * These notes are decrypted and added to the output parameter vector, outEntries.
@@ -5913,99 +5655,57 @@ void CWallet::GetFilteredNotes(
 {
     LOCK2(cs_main, cs_wallet);
 
-    // Every caller already holds LOCK2, so the lock above never waits and this times the walk alone.
+    LogPrintf("%s ignoreLocked=%d\n", __func__, ignoreLocked);
-    // Time spent waiting for the locks shows up in the RPC's own "took" line (rpc/server.cpp).
-    const std::chrono::steady_clock::time_point walkStart = std::chrono::steady_clock::now();
-    const size_t nEntriesBefore = saplingEntries.size();
-    unsigned int nTxs = 0, nNotes = 0, nSpent = 0;
 
-    // Pass 1: the notes that pass every filter not needing the plaintext, in wallet order.
-    struct Candidate {
-        const SaplingOutPoint* op;
-        SaplingNoteData* nd;
-        const OutputDescription* output;
-        int nDepth;
-    };
-    std::vector<Candidate> candidates;
-    std::vector<SaplingNoteToDecrypt> uncached;
     for (auto & p : mapWallet) {
-        // By reference: a copy deep-copies every note's witness list (up to WITNESS_CACHE_SIZE
+        CWalletTx wtx = p.second;
-        // witnesses each), and it was made before any filter ran. Not const: a missing plaintext is
-        // cached onto the wallet's own entry.
-        CWalletTx& wtx = p.second;
-
-        // Most wallet txs (coinbase, sends from other wallets) carry no notes of ours.
-        if (wtx.mapSaplingNoteData.empty())
-            continue;
 
         // Filter the transactions before checking for notes
         if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
             continue;
 
-        // cs_main is held, so the depth can't change during the walk.
-        const int nDepth = wtx.GetDepthInMainChain();
         if (minDepth > 1) {
-            int nHeight    = HeightFromDepth(nDepth);
+            int nHeight    = tx_height(wtx.GetHash());
             if ( nHeight == 0 ) {
                 continue;
             }
+            int nDepth     = wtx.GetDepthInMainChain();
             int dpowconfs  = hush_dpowconfs(nHeight,nDepth);
             if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
                 continue;
             }
         } else {
-            if ( nDepth < minDepth || nDepth > maxDepth) {
+            if ( wtx.GetDepthInMainChain() < minDepth ||
+                wtx.GetDepthInMainChain() > maxDepth) {
                 continue;
             }
         }
-        nTxs++;
 
         for (auto & pair : wtx.mapSaplingNoteData) {
-            const SaplingOutPoint& op = pair.first;
+            SaplingOutPoint op = pair.first;
-            SaplingNoteData& nd = pair.second;
+            SaplingNoteData nd = pair.second;
-            nNotes++;
 
-            // The cheap filters run before any decryption, which costs ~0.5ms of Jubjub arithmetic
+            auto maybe_pt = SaplingNotePlaintext::decrypt(
-            // per note. Spent notes stay in the wallet (-deletetx is off by default).
+                wtx.vShieldedOutput[op.n].encCiphertext,
-            if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
+                nd.ivk,
-                nSpent++;
+                wtx.vShieldedOutput[op.n].ephemeralKey,
-                continue;
+                wtx.vShieldedOutput[op.n].cm);
-            }
+            assert(static_cast<bool>(maybe_pt));
+            auto notePt = maybe_pt.get();
 
-            // skip locked notes
+            auto maybe_pa = nd.ivk.address(notePt.d);
-            if (ignoreLocked && IsLockedNote(op)) {
+            assert(static_cast<bool>(maybe_pa));
-                LogPrint("zrpcunsafe", "%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
+            auto pa = maybe_pa.get();
-                continue;
-            }
-
-            assert(op.n < wtx.vShieldedOutput.size());
-            const OutputDescription* output = &wtx.vShieldedOutput[op.n];
-            candidates.push_back(Candidate { &op, &nd, output, nDepth });
-            if (!nd.cachedPlaintext)
-                uncached.push_back(SaplingNoteToDecrypt { &nd, output });
-        }
-    }
-
-    // Decrypt what the cache lacks: normally nothing, since FindMySaplingNotes fills new notes and the
-    // startup warm-up fills the ones loaded from wallet.dat.
-    const unsigned int nFilled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
-
-    // Pass 2: the filters that need the note's address.
-    if (filterAddresses.empty())
-        saplingEntries.reserve(saplingEntries.size() + candidates.size());
-    for (const Candidate& c : candidates) {
-        // A wallet note that doesn't decrypt under its own ivk is corrupt; as before, that's fatal here.
-        assert(static_cast<bool>(c.nd->cachedPlaintext));
-        if (fCheckNoteCache)
-            CheckSaplingNoteCache(*c.op, *c.nd, *c.output);
-        const SaplingNoteCache& plaintext = *c.nd->cachedPlaintext;
-        const libzcash::SaplingPaymentAddress pa(plaintext.note.d, plaintext.note.pk_d);
 
             // skip notes which belong to a different payment address in the wallet
             if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
                 continue;
             }
 
+            if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
+                continue;
+            }
+
             // skip notes which cannot be spent
             if (requireSpendingKey) {
                 libzcash::SaplingIncomingViewingKey ivk;
@@ -6017,21 +5717,16 @@ void CWallet::GetFilteredNotes(
                 }
             }
 
-        saplingEntries.push_back(SaplingNoteEntry {
+            // skip locked notes
-            *c.op, pa, plaintext.note, plaintext.memo, c.nDepth });
+             if (ignoreLocked && IsLockedNote(op)) {
+                 LogPrintf("%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
+                 continue;
              }
 
-    // A slow walk holds cs_main long enough to stall block connection, so it is always logged; a
+            auto note = notePt.note(nd.ivk).get();
-    // fast one only with -debug=bench (the balance RPCs run it on every GUI poll). The wallet-size
+            saplingEntries.push_back(SaplingNoteEntry {
-    // counts go to the default log only with -debug=bench too.
+                op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() });
-    const int64_t nWalkMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - walkStart).count();
+        }
-    if (LogAcceptCategory("bench")) {
-        LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d txs=%u/%u notes=%u spent=%u decrypted=%u entries=%u walk=%dms\n",
-                  __func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nTxs, (unsigned int)mapWallet.size(),
-                  nNotes, nSpent, nFilled, (unsigned int)(saplingEntries.size() - nEntriesBefore), nWalkMs);
-    } else if (nWalkMs >= 500) {
-        LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d walk=%dms\n",
-                  __func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nWalkMs);
     }
 }
 

src/wallet/wallet.h

@@ -62,8 +62,6 @@ extern bool bSpendZeroConfChange;
 extern bool fSendFreeTransactions;
 extern bool fPayAtLeastCustomFee;
 extern bool fTxDeleteEnabled;
-//! -checknotecache: re-decrypt cached Sapling notes in GetFilteredNotes and repair any difference
-extern bool fCheckNoteCache;
 extern bool fTxConflictDeleteEnabled;
 extern int fDeleteInterval;
 extern unsigned int fDeleteTransactionsAfterNBlocks;
@@ -303,14 +301,6 @@ class SproutNoteData
 };
 
 
-/** A Sapling note's decrypted plaintext as the wallet uses it: the note (value, rcm, and the address it
- *  was sent to as (d, pk_d)) and its memo. */
-struct SaplingNoteCache
-{
-    libzcash::SaplingNote note;
-    std::array<unsigned char, HUSH_MEMO_SIZE> memo;
-};
-
 class SaplingNoteData
 {
 public:
@@ -338,15 +328,6 @@ public:
     // read true and short-circuit the witness self-heal in VerifyAndSetInitialWitness.
     bool witnessRootValidated = false;
 
-    //In Memory Only
-    // The note's decrypted plaintext, so the balance and note-listing calls don't redo the Sapling
-    // decryption (several Jubjub scalar multiplications) for every note on every call. It is a pure
-    // function of (txid, output index, ivk), so nothing invalidates it. Never serialized, and left out
-    // of operator== and SaplingNoteDataIdentical on purpose: a filled and an unfilled entry persist the
-    // same state, so filling one must never make a tx look changed and get rewritten. Filled by
-    // FindMySaplingNotes when the note is found, otherwise by FillSaplingNoteCache.
-    boost::optional<SaplingNoteCache> cachedPlaintext;
-
     ADD_SERIALIZE_METHODS;
 
     template <typename Stream, typename Operation>
@@ -385,11 +366,6 @@ struct SaplingNoteEntry
     int confirmations;
 };
 
-/** Height of the active-chain block holding a wallet tx that is nDepth deep (as GetDepthInMainChain
- *  returns it, and SaplingNoteEntry::confirmations holds it), or 0 when the tx is unconfirmed or not
- *  in the active chain. Caller holds cs_main. */
-int HeightFromDepth(int nDepth);
-
 /** A transaction with a merkle branch linking it to the block chain. */
 class CMerkleTx : public CTransaction
 {
@@ -919,6 +895,12 @@ public:
 
 protected:
 
+    int SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight);
+    //! Overload for a note whose nullifier may be unset. A note discovered through an imported
+    //! INCOMING viewing key has no nullifier (computing one needs the full viewing key), so
+    //! dereferencing the optional aborts the daemon. Treats such a note as unspent, which is the
+    //! conservative direction: it keeps the witness alive rather than pruning a note we cannot
+    //! prove spent.
     int SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight);
 
     /**
@@ -989,12 +971,8 @@ private:
     template <class T>
     void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>);
 
-    // Last block ThreadNotifyWallets delivered to ChainTip, i.e. the block the wallet's state
-    // reflects (guarded by cs_wallet). SetBestChain stamps this instead of the chain tip.
-    const CBlockIndex* pindexWalletTip = nullptr;
-
 protected:
-    bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock);
+    bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx);
     void MarkAffectedTransactionsDirty(const CTransaction& tx);
 
     /* the hd chain data model (chain counters) */
@@ -1322,8 +1300,6 @@ public:
     bool CommitAutomatedTx(const CTransaction& tx);
     /** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */
     void SetBestChain(const CBlockLocator& loc);
-    // Seed the block the wallet's state reflects, before ThreadNotifyWallets starts delivering.
-    void SetWalletTip(const CBlockIndex* pindex);
     /** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false
      *  so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment
      *  at CWallet::SetBestChain), and reports whether the write actually committed. */
@@ -1488,29 +1464,6 @@ public:
     bool LoadCryptedMnemonicEntropy(const uint256& entropyFp, const std::vector<unsigned char>& vchCryptedSecret);
     
     /* Find notes filtered by payment address, min depth, ability to spend */
-    /** A wallet note whose decrypted plaintext is not cached yet, with the output it decrypts. */
-    struct SaplingNoteToDecrypt {
-        SaplingNoteData* nd;
-        const OutputDescription* output;
-    };
-    /** Decrypts each note into nd->cachedPlaintext and returns how many it filled. A note that doesn't
-     *  decrypt under its ivk is left unfilled. Caller holds cs_wallet. */
-    unsigned int FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes);
-    /** Notes FillSaplingNoteCache has decrypted since startup (FindMySaplingNotes' fills not counted), and
-     *  the threads its last call used. Guarded by cs_wallet. */
-    uint64_t nSaplingNoteCacheFills = 0;
-    int nSaplingNoteCacheFillThreads = 0;
-
-    struct SaplingNoteCacheWarmup {
-        unsigned int notes = 0;          //!< notes in the wallet
-        unsigned int filled = 0;         //!< decrypted by this call
-        unsigned int noSuchOutput = 0;   //!< naming an output their tx doesn't have
-        unsigned int undecryptable = 0;  //!< not decrypting under their own ivk
-    };
-    /** Decrypts every wallet note whose plaintext isn't cached yet (all of them after wallet.dat is loaded).
-     *  Never asserts: a note that can't be decrypted is counted, logged and left unfilled. Run at startup. */
-    SaplingNoteCacheWarmup WarmSaplingNoteCache();
-
     void GetFilteredNotes(std::vector<SaplingNoteEntry>& saplingEntries,
                           std::string address,
                           int minDepth=1,

src/zcash/Note.cpp

@@ -115,16 +115,11 @@ boost::optional<SaplingOutgoingPlaintext> SaplingOutgoingPlaintext::decrypt(
     }
 }
 
-namespace {
+boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
-
-// SaplingNotePlaintext::decrypt with an incoming viewing key, also handing back the pk_d = [ivk]g_d
-// it derives to check cmu.
-boost::optional<SaplingNotePlaintext> DecryptWithIvk(
     const SaplingEncCiphertext &ciphertext,
     const uint256 &ivk,
     const uint256 &epk,
-    const uint256 &cmu,
+    const uint256 &cmu
-    uint256 &pk_d
 )
 {
     auto pt = AttemptSaplingEncDecryption(ciphertext, ivk, epk);
@@ -145,6 +140,7 @@ boost::optional<SaplingNotePlaintext> DecryptWithIvk(
         return boost::none;
     }
 
+    uint256 pk_d;
     if (!librustzcash_ivk_to_pkd(ivk.begin(), ret.d.data(), pk_d.begin())) {
         return boost::none;
     }
@@ -168,36 +164,6 @@ boost::optional<SaplingNotePlaintext> DecryptWithIvk(
     return ret;
 }
 
-}
-
-boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
-    const SaplingEncCiphertext &ciphertext,
-    const uint256 &ivk,
-    const uint256 &epk,
-    const uint256 &cmu
-)
-{
-    uint256 pk_d;
-    return DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
-}
-
-boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> SaplingNotePlaintext::decrypt_with_note(
-    const SaplingEncCiphertext &ciphertext,
-    const uint256 &ivk,
-    const uint256 &epk,
-    const uint256 &cmu
-)
-{
-    uint256 pk_d;
-    auto pt = DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
-    if (!pt) {
-        return boost::none;
-    }
-    // ivk_to_pkd succeeding implies d is a valid diversifier, which is all note(ivk) checks on top.
-    SaplingNote note(pt->d, pk_d, pt->value(), pt->rcm);
-    return std::make_pair(*pt, note);
-}
-
 boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
     const SaplingEncCiphertext &ciphertext,
     const uint256 &epk,

src/zcash/Note.hpp

@@ -85,17 +85,6 @@ public:
         const uint256 &cmu
     );
 
-    // decrypt(ciphertext, ivk, epk, cmu) that also returns the note. Building the note needs
-    // pk_d = [ivk]g_d, which the decryption derives anyway to check cmu; note(ivk) would derive it
-    // again (another Jubjub scalar multiplication). A distinct name rather than an overload, so it
-    // can never capture a call meant for the (epk, esk, pk_d, cmu) one.
-    static boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> decrypt_with_note(
-        const SaplingEncCiphertext &ciphertext,
-        const uint256 &ivk,
-        const uint256 &epk,
-        const uint256 &cmu
-    );
-
     boost::optional<SaplingNote> note(const SaplingIncomingViewingKey& ivk) const;
 
     virtual ~SaplingNotePlaintext() {}